fix(cutover): gate rollback stages B/C on --accept-post-cutover-write-loss

Address Baz #919 (high, rollback safety). Promoting the frozen pre-cutover backup
makes traces the successor accepted after cutover_start non-live. They are not
destroyed — the successor is parked as traces_local_v2 until finalize.sh, recoverable
during the soak — but the live table stops serving them. This is inherent to promoting
a point-in-time backup and is deliberately not auto-repaired (merging the successor's
post-cutover writes back would re-import the data the rollback exists to discard).

- rollback.sh: stages B/C now require --accept-post-cutover-write-loss and print the
  recovery pointer before the promote. Named --accept-* (not --confirm-*) because it
  acknowledges an unavoidable consequence rather than a precondition the operator fixes.
- README: scope the "no data loss" claim to "no data-bearing table dropped"; add the
  post-cutover-write caveat with the recovery path; show the flag in the B/C examples.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: andrescrz

apps/opik-backend/data-migrations/traces-local-v2-cutover/README.md

@@ -419,26 +419,36 @@ as one file per stage (`000004_rollback_stage_a_discard_shadow.sql`, `…_stage_
 `…_stage_c_promote_original.sql`, and the shared `000004_rollback_reverse_replay.sql`) and driven by
 [`scripts/rollback.sh`](scripts/rollback.sh), so no one authors it under pressure.
 
-**No data loss by construction.** The stages are mutually exclusive, so each lives in its **own file** — no single file
-mixes the `TRUNCATE` (stage A only) with the `EXCHANGE`/`DROP` of the others, and running any file does exactly one
-stage. No statement drops a data-bearing table: swaps are atomic `EXCHANGE`/`RENAME`, and the only `DROP` targets the
-`Distributed` wrapper, which stores no data (it is a routing definition over `traces_local`). Before running, `rollback.sh`
-**asserts the live `traces` topology matches the requested stage and aborts otherwise** — so a wrong-stage run (the only
-way a `TRUNCATE`/`DROP` could hit the wrong table) makes no change. Every stage lands in the same **canonical state**:
-`traces` = the original data (live), `traces_local_v2` = the successor data (parked backup). No leftover `*_new` names.
-The parked backup is dropped only later, by `finalize.sh`, after the soak.
+**No data-bearing table is dropped by construction.** The stages are mutually exclusive, so each lives in its **own
+file** — no single file mixes the `TRUNCATE` (stage A only) with the `EXCHANGE`/`DROP` of the others, and running any
+file does exactly one stage. No statement drops a data-bearing table: swaps are atomic `EXCHANGE`/`RENAME`, and the only
+`DROP` targets the `Distributed` wrapper, which stores no data (it is a routing definition over `traces_local`). Before
+running, `rollback.sh` **asserts the live `traces` topology matches the requested stage and aborts otherwise** — so a
+wrong-stage run (the only way a `TRUNCATE`/`DROP` could hit the wrong table) makes no change. Every stage lands in the
+same **canonical state**: `traces` = the original data (live), `traces_local_v2` = the successor data (parked backup).
+No leftover `*_new` names. The parked backup is dropped only later, by `finalize.sh`, after the soak.
+
+> **Stages B/C make post-cutover writes non-live — an accepted, acknowledged trade-off.** Promoting the frozen
+> `traces_pre_cutover_backup` means traces the successor accepted **after** `cutover_start` stop being served by the live
+> table (the reverse-replay carries post-cutover *deletes* forward, but not *writes*). They are **not destroyed**: the
+> successor is parked as `traces_local_v2` and retained until `finalize.sh`, so recover them from there during the soak
+> if the rollback is later judged unnecessary. This is inherent to promoting a point-in-time backup and is *not* auto-repaired
+> — merging the successor's post-cutover writes back would re-import the very data the rollback exists to discard. Because
+> it is irreversible in the moment, stages B/C require `--accept-post-cutover-write-loss`, and `rollback.sh` prints the
+> recovery pointer before the promote.
 
 Pick the stage by how far the cutover got (`cutover_start` is the value `exchange_and_wrap.sh` printed):
 
 - **Stage A — before EXCHANGE:** `./scripts/rollback.sh --database opik --stage A`. Discards the disposable shadow
   `traces_local_v2`; the live `traces` was never touched. (Guarded: aborts unless `traces` is still the original schema.)
-- **Stage B — after EXCHANGE, before wrap:** `./scripts/rollback.sh --database opik --stage B --cutover-start '<ts>'`.
-  `EXCHANGE` `traces_pre_cutover_backup` back to live `traces`, rename the now-parked successor back to
-  `traces_local_v2`, then the reverse replay. (Guarded: aborts if `traces` is `Distributed` — use C.)
-- **Stage C — after wrap:** `./scripts/rollback.sh --database opik --stage C --cutover-start '<ts>'`. Drops the
-  `Distributed` wrapper, then one atomic `RENAME` promotes the original (`traces_pre_cutover_backup`) back to `traces`
-  and parks the successor under `traces_local_v2`, then the reverse replay. (Guarded: aborts unless `traces` is
-  `Distributed`.)
+- **Stage B — after EXCHANGE, before wrap:** `./scripts/rollback.sh --database opik --stage B --cutover-start '<ts>'
+  --confirm-retention-paused --accept-post-cutover-write-loss`. `EXCHANGE` `traces_pre_cutover_backup` back to live
+  `traces`, rename the now-parked successor back to `traces_local_v2`, then the reverse replay. (Guarded: aborts if
+  `traces` is `Distributed` — use C.)
+- **Stage C — after wrap:** `./scripts/rollback.sh --database opik --stage C --cutover-start '<ts>'
+  --confirm-retention-paused --accept-post-cutover-write-loss`. Drops the `Distributed` wrapper, then one atomic
+  `RENAME` promotes the original (`traces_pre_cutover_backup`) back to `traces` and parks the successor under
+  `traces_local_v2`, then the reverse replay. (Guarded: aborts unless `traces` is `Distributed`.)
 
 After a stage B or C rollback, `traces` is the Nullable original again — **revert `traceColumnsNonNullable` to `false`
 AND roll-restart every backend instance**. The flag is read from a **startup snapshot** of `OpikConfiguration` (bound via

apps/opik-backend/data-migrations/traces-local-v2-cutover/scripts/rollback.sh

@@ -7,9 +7,17 @@
 #   --stage A   backfill/delta ran but the EXCHANGE did not — discard the shadow (live `traces` is untouched).
 #   --stage B   the EXCHANGE ran but not the wrap — swap the tables back, then reverse-replay.
 #   --stage C   the wrap ran — drop the wrapper, promote the parked original, then reverse-replay.
-# Stages B and C need --cutover-start (printed by exchange_and_wrap.sh) to bound the reverse-replay, and
+# Stages B and C need --cutover-start (printed by exchange_and_wrap.sh) to bound the reverse-replay,
 # --confirm-retention-paused (retention deletes bypass the bridge, so a retention sweep in the rollback window would
-# resurrect a deleted row from the backup). Keep the deletion bridge enabled through the rollback so no delete is lost.
+# resurrect a deleted row from the backup), and --accept-post-cutover-write-loss (see below). Keep the deletion bridge
+# enabled through the rollback so no delete is lost.
+#
+# POST-CUTOVER WRITES: stages B/C promote the frozen pre-cutover backup back to live `traces`, so traces WRITTEN to the
+# successor after cutover_start stop being live. They are NOT destroyed — the successor is parked as traces_local_v2 and
+# retained until finalize.sh, so they can be recovered from there during the soak — but the live table no longer serves
+# them. This is inherent to promoting a point-in-time backup and cannot be "fixed" (auto-merging the successor's writes
+# would re-import the very data the rollback is discarding); --accept-post-cutover-write-loss makes the operator
+# acknowledge it before the promote.
 #
 # SAFETY: the stages are mutually exclusive and each lives in its OWN file, so no single file mixes a TRUNCATE with an
 # EXCHANGE/DROP — running any file does exactly one stage. Before running, this asserts the live `traces` topology matches
@@ -28,13 +36,15 @@ DATABASE=""
 STAGE=""
 CUTOVER_START=""
 CONFIRM_RETENTION_PAUSED=0
+ACCEPT_WRITE_LOSS=0
 
 while [[ $# -gt 0 ]]; do
     case "$1" in
         --database) DATABASE="${2:?"$1 requires a value"}"; shift 2 ;;
         --stage) STAGE="${2:?"$1 requires a value"}"; shift 2 ;;
         --cutover-start) CUTOVER_START="${2:?"$1 requires a value"}"; shift 2 ;;
         --confirm-retention-paused) CONFIRM_RETENTION_PAUSED=1; shift ;;
+        --accept-post-cutover-write-loss) ACCEPT_WRITE_LOSS=1; shift ;;
         *) echo "Unknown argument: $1" >&2; exit 2 ;;
     esac
 done
@@ -57,6 +67,16 @@ if [[ ( "$STAGE" == "B" || "$STAGE" == "C" ) && "$CONFIRM_RETENTION_PAUSED" != "
     echo "       Pause retention (RETENTION_ENABLED=false on every backend), then re-run with the flag." >&2
     exit 2
 fi
+# Stages B/C promote the frozen pre-cutover backup, so writes the successor accepted after cutover_start stop being live
+# (they are preserved in the parked traces_local_v2 until finalize.sh, recoverable during the soak). This is unavoidable
+# when promoting a point-in-time backup — require the operator to acknowledge it, unlike a precondition they could fix.
+if [[ ( "$STAGE" == "B" || "$STAGE" == "C" ) && "$ACCEPT_WRITE_LOSS" != "1" ]]; then
+    echo "ERROR: rollback --stage $STAGE requires --accept-post-cutover-write-loss. Promoting the frozen backup makes" >&2
+    echo "       traces written to the successor after cutover_start non-live. They are NOT destroyed — the successor is" >&2
+    echo "       parked as traces_local_v2 until finalize.sh, so recover them from there during the soak — but the live" >&2
+    echo "       table will no longer serve them. Re-run with the flag once you accept this." >&2
+    exit 2
+fi
 
 ch() {
     clickhouse-client --database "$DATABASE" --log_comment 'traces_local_v2_rollback' --query "$1"
@@ -124,6 +144,11 @@ run_file() {
 
 assert_topology
 
+if [[ "$STAGE" == "B" || "$STAGE" == "C" ]]; then
+    echo "NOTE: promoting the frozen backup now. Traces the successor accepted after cutover_start ($CUTOVER_START) will" >&2
+    echo "      stop being live; recover them from the parked traces_local_v2 (kept until finalize.sh) if needed." >&2
+fi
+
 case "$STAGE" in
     A)
         run_file 000004_rollback_stage_a_discard_shadow.sql