fix(cutover): gate EXCHANGE/rollback on retention-paused; make delete_traffic best-effort

Address Baz round:
- exchange_and_wrap.sh / rollback.sh: require --confirm-retention-paused on every
  EXCHANGE path and on rollback stages B/C. Retention deletes bypass the deletion
  bridge, so a sweep in the cutover/rollback window leaks live across the swap
  (forward) or resurrects a deleted row from the backup (reverse-replay). It is an
  operator assertion — a backend setting the script can't read — so the README
  Go/No-Go keeps the real "actually paused on every backend" verification.
- delete_traffic.py (#851): larger newest-page refill and more generous empty-refill
  tolerance so mask-visibility lag doesn't end the run early; docstring states it is a
  best-effort traffic generator, not a guaranteed full-drain.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: andrescrz

apps/opik-backend/data-migrations/traces-local-v2-cutover/README.md

@@ -144,10 +144,13 @@ new table before the EXCHANGE. The replay matches the **full key**, not `id` alo
    leaving `traces` a `MergeTree` where deletes still work); the `RENAME` + `Distributed` wrap runs only with
    `--with-wrap`. Restore the buffer ceiling and verify.
    ```bash
-   CLICKHOUSE_HOST=<host> CLICKHOUSE_PASSWORD=<pw> ./scripts/exchange_and_wrap.sh --database opik --confirm-buffer-raised
+   CLICKHOUSE_HOST=<host> CLICKHOUSE_PASSWORD=<pw> ./scripts/exchange_and_wrap.sh --database opik \
+       --backfill-start '<anchor from backfill.sh>' --confirm-buffer-raised --confirm-retention-paused
    ```
-   `--confirm-buffer-raised` is required on every EXCHANGE path: it asserts the async-insert buffer is raised so writes in
-   the final window survive the swap (add `--with-wrap --confirm-daos-retargeted` only once the DAOs target `traces_local`).
+   Every EXCHANGE path requires: `--backfill-start` (for the final deletion replay), `--confirm-buffer-raised` (writes in
+   the final window survive the swap), and `--confirm-retention-paused` (retention deletes bypass the bridge, so a
+   retention sweep in the window would leak across the swap). Add `--with-wrap --confirm-daos-retargeted` only once the
+   DAOs target `traces_local`.
 
 > **HARD PREREQUISITE for the wrap (step 4, part 2): the delete/mutation DAO must target `traces_local` first.** A
 > `Distributed` table supports `SELECT` and `INSERT` but **not** mutations. Verified on ClickHouse 26.3:
@@ -613,7 +616,10 @@ cheap (stage A); the bridge stays enabled so nothing is lost on a retry.
 - [ ] **Async-insert ceiling confirmed** — raising `asyncInsertBusyTimeoutMaxMs` demonstrably widens the adaptive buffer
       under load, not just the cap. `exchange_and_wrap.sh` enforces the acknowledgment via `--confirm-buffer-raised`, but
       that is an assertion only — this checklist item is the actual "it took effect under load" verification.
-- [ ] **Data Retention confirmed disabled** for the cutover window (`RETENTION_ENABLED=false`).
+- [ ] **Data Retention confirmed disabled** for the cutover window (`RETENTION_ENABLED=false`). Retention deletes bypass
+      the deletion bridge, so a sweep in the window would leak/resurrect across the swap; `exchange_and_wrap.sh` and
+      `rollback.sh` (stages B/C) enforce `--confirm-retention-paused`, but that is an assertion — this item is the real
+      "it is actually paused on every backend" verification.
 - [ ] **Reconciliation clean** — per-window source/dest counts within 0.01% across the whole backfill.
 - [ ] **Replication settled before the EXCHANGE** — `replication_queue` empty and the deletion-replay mutation
       `is_done` on **all** replicas (`exchange_and_wrap.sh` gates on this; do not `--force` past it in production).

apps/opik-backend/data-migrations/traces-local-v2-cutover/scripts/exchange_and_wrap.sh

@@ -47,6 +47,10 @@
 #                     writes across the swap so they land on the new table; at the default, a write in the final window
 #                     can commit to the old table and be lost after the EXCHANGE. It's a backend setting the script can't
 #                     read, so the operator must assert it.
+#   --confirm-retention-paused  REQUIRED for every EXCHANGE path. Retention deletes (deleteForRetention*) bypass the
+#                     deletion bridge and are never replayed onto the successor, so a retention sweep during the cutover
+#                     window leaks live across the swap. Asserts retention is paused (RETENTION_ENABLED=false on every
+#                     backend) for the whole window — a backend setting the script can't read.
 
 set -euo pipefail
 
@@ -63,6 +67,7 @@ FORCE=0
 CONFIRM_MAINTENANCE=0
 CONFIRM_DAOS_RETARGETED=0
 CONFIRM_BUFFER_RAISED=0
+CONFIRM_RETENTION_PAUSED=0
 
 while [[ $# -gt 0 ]]; do
     case "$1" in
@@ -75,6 +80,7 @@ while [[ $# -gt 0 ]]; do
         --confirm-maintenance) CONFIRM_MAINTENANCE=1; shift ;;
         --confirm-daos-retargeted) CONFIRM_DAOS_RETARGETED=1; shift ;;
         --confirm-buffer-raised) CONFIRM_BUFFER_RAISED=1; shift ;;
+        --confirm-retention-paused) CONFIRM_RETENTION_PAUSED=1; shift ;;
         *) echo "Unknown argument: $1" >&2; exit 2 ;;
     esac
 done
@@ -124,6 +130,16 @@ if [[ "$WRAP_ONLY" != "1" && -z "$BACKFILL_START" ]]; then
     echo "ERROR: the EXCHANGE requires --backfill-start (the anchor printed by backfill.sh) for the final deletion replay." >&2
     exit 2
 fi
+# Retention deletes (TraceDAO.deleteForRetention*) bypass the deletion bridge, so they are never replayed onto the
+# successor — if any backend still has RETENTION_ENABLED=true, a retention sweep in the cutover window leaks live across
+# the swap. Retention is a backend setting the script can't read, so require the operator to assert it is paused for the
+# whole window. Applies to every EXCHANGE path (not --wrap-only, which does no data cutover).
+if [[ "$WRAP_ONLY" != "1" && "$CONFIRM_RETENTION_PAUSED" != "1" ]]; then
+    echo "ERROR: the EXCHANGE requires --confirm-retention-paused. Retention deletes bypass the deletion bridge, so a" >&2
+    echo "       retention sweep during the cutover window would leak live across the swap. Pause retention" >&2
+    echo "       (RETENTION_ENABLED=false on every backend) for the whole window, then re-run with the flag." >&2
+    exit 2
+fi
 
 ch() {
     clickhouse-client --database "$DATABASE" --log_comment 'traces_local_v2_cutover:exchange_and_wrap' --query "$1"

apps/opik-backend/data-migrations/traces-local-v2-cutover/scripts/rollback.sh

@@ -7,8 +7,9 @@
 #   --stage A   backfill/delta ran but the EXCHANGE did not — discard the shadow (live `traces` is untouched).
 #   --stage B   the EXCHANGE ran but not the wrap — swap the tables back, then reverse-replay.
 #   --stage C   the wrap ran — drop the wrapper, promote the parked original, then reverse-replay.
-# Stages B and C need --cutover-start (printed by exchange_and_wrap.sh) to bound the reverse-replay. Keep the deletion
-# bridge enabled through the rollback so no delete is lost.
+# Stages B and C need --cutover-start (printed by exchange_and_wrap.sh) to bound the reverse-replay, and
+# --confirm-retention-paused (retention deletes bypass the bridge, so a retention sweep in the rollback window would
+# resurrect a deleted row from the backup). Keep the deletion bridge enabled through the rollback so no delete is lost.
 #
 # SAFETY: the stages are mutually exclusive and each lives in its OWN file, so no single file mixes a TRUNCATE with an
 # EXCHANGE/DROP — running any file does exactly one stage. Before running, this asserts the live `traces` topology matches
@@ -26,12 +27,14 @@ SQL_DIR="$SCRIPT_DIR/db-app-analytics"
 DATABASE=""
 STAGE=""
 CUTOVER_START=""
+CONFIRM_RETENTION_PAUSED=0
 
 while [[ $# -gt 0 ]]; do
     case "$1" in
         --database) DATABASE="${2:?"$1 requires a value"}"; shift 2 ;;
         --stage) STAGE="${2:?"$1 requires a value"}"; shift 2 ;;
         --cutover-start) CUTOVER_START="${2:?"$1 requires a value"}"; shift 2 ;;
+        --confirm-retention-paused) CONFIRM_RETENTION_PAUSED=1; shift ;;
         *) echo "Unknown argument: $1" >&2; exit 2 ;;
     esac
 done
@@ -44,6 +47,16 @@ case "$STAGE" in
     A|B|C) ;;
     *) echo "ERROR: --stage must be A, B or C" >&2; exit 2 ;;
 esac
+# Stages B/C run the reverse-replay, which — like the forward replay — only re-applies bridged deletes. Retention deletes
+# (deleteForRetention*) bypass the bridge, so a retention sweep during the rollback window would restore a legitimately
+# deleted row from the backup and resurrect it. Retention is a backend setting the script can't read; require the
+# operator to assert it is paused. Stage A does no reverse-replay, so it is exempt.
+if [[ ( "$STAGE" == "B" || "$STAGE" == "C" ) && "$CONFIRM_RETENTION_PAUSED" != "1" ]]; then
+    echo "ERROR: rollback --stage $STAGE requires --confirm-retention-paused. The reverse-replay only re-applies bridged" >&2
+    echo "       deletes; a retention sweep in the rollback window would resurrect a deleted row from the backup." >&2
+    echo "       Pause retention (RETENTION_ENABLED=false on every backend), then re-run with the flag." >&2
+    exit 2
+fi
 
 ch() {
     clickhouse-client --database "$DATABASE" --log_comment 'traces_local_v2_rollback' --query "$1"

tests_load/tests/traces-local-v2-cutover/delete_traffic.py

@@ -7,6 +7,11 @@
 It pulls a pool of existing trace ids via search and deletes them at the target rate, refilling as it drains. Run it
 during or after the backfill so the traces it deletes have already been copied — that is the leak the bridge prevents.
 
+This is a best-effort TRAFFIC GENERATOR (deletes newest-first for the run's --duration), NOT a guaranteed full-drain:
+search returns the newest page, so during delete-mask visibility lag a refill can transiently return only ids already
+in `seen`/`pool`; the loop tolerates a few such empty refills (EMPTY_REFILL_LIMIT) before concluding the project is
+drained. It does not assert every trace was deleted — its job is to exercise the deletion bridge, not to empty the table.
+
 Prerequisites: `OPIK_URL_OVERRIDE` pointing at the local install. Run `python delete_traffic.py --help` for options.
 """
 
@@ -20,8 +25,12 @@
 _stop = False
 
 # Consecutive empty refills tolerated before concluding the project is drained. An empty refill can be a transient
-# delete-mask-visibility lag (the just-deleted top ids not yet hidden from search), not a truly empty project.
-EMPTY_REFILL_LIMIT = 3
+# delete-mask-visibility lag (the just-deleted top ids not yet hidden from search), not a truly empty project — so give
+# the mask several beats to propagate before stopping.
+EMPTY_REFILL_LIMIT = 5
+# Newest-page size to pull per refill. Larger reaches past the just-deleted (still-visible) top ids to undeleted ones
+# during mask lag, so the run keeps finding work instead of stopping early.
+REFILL_FETCH = 2000
 
 
 def _handle_sigint(_signum, _frame):
@@ -61,7 +70,7 @@ def main(project, tps, duration, batch):
         tick = time.time()
         if len(pool) < batch:
             # Exclude both already-deleted ids and those still queued in `pool`, so a refill can't requeue an in-flight id.
-            fetched = _fetch_ids(client, project, want=500, exclude=seen | set(pool))
+            fetched = _fetch_ids(client, project, want=REFILL_FETCH, exclude=seen | set(pool))
             if fetched is None:
                 # transient search failure — back off and retry rather than mistaking it for "no more traces".
                 time.sleep(interval if interval > 0 else 0.5)