Skip to content

[#33444] DocDB: Add retention-pin observability for index-backfill ordering generations - #33657

Open
jmeehan16 wants to merge 1 commit into
feature-stack/Shopify/uniq-idx-7-fail-closed-gatefrom
feature-stack/Shopify/uniq-idx-8a-retention-observability
Open

[#33444] DocDB: Add retention-pin observability for index-backfill ordering generations#33657
jmeehan16 wants to merge 1 commit into
feature-stack/Shopify/uniq-idx-7-fail-closed-gatefrom
feature-stack/Shopify/uniq-idx-8a-retention-observability

Conversation

@jmeehan16

@jmeehan16 jmeehan16 commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

Retention-pin observability for deferred unique-index verification (#33444). With #33655
enforcing the retention barrier as a history read fence, an orphaned active ordering
generation pins ALL history GC on its tablet -- previously findable only by reading tablet
superblocks. This was called out as a measurement-gate requirement in the read-fence review:
the rollout's retained-history measurements need these signals. Two server-level tserver
gauges, updated by the existing 1s EmitMetrics poller:

  • ts_index_backfill_pinned_tablets -- tablet peers whose history GC is pinned by an active
    index-backfill ordering generation. Counts peers in any state: the pin is on-disk
    metadata, not runtime state.
  • ts_index_backfill_oldest_retention_barrier_age_ms -- age of the oldest active barrier,
    clamped at zero (the barrier derives from a master-chosen read time, which clock skew can
    put slightly ahead of this server's clock right after activation). 0 when no generation is
    active or no active generation carries a barrier hybrid time -- that missing-barrier state
    blocks all history GC on its tablet and already logs a warning every 30s.

Alerting guidance: key alerts on ts_index_backfill_pinned_tablets > 0 persisting
longer than the expected build+verify duration -- never on age alone. The worst state (an
active generation with a missing barrier) blocks all history GC yet reports age 0; the
pinned count plus the tserver's periodic missing-barrier warning are the signals that cover
it.

The tserver /tablet detail page additionally shows a held generation's fields (index
table, base op index, write-ID floor version, retention barrier + age) for per-tablet
triage.

Stacked on #33403, #33404, #33484, #33544, #33553, #33580, #33584, #33601, #33602, #33654,
#33655, and #33656, based on feature-stack/Shopify/uniq-idx-7-fail-closed-gate per the
feature-stack workflow, so the diff is exactly this PR's own commit. A failing pr-title
check on stacked PRs is a known gap in that check; the stack merges bottom-up and retargets
as parents merge.

Upgrade/Rollback safety

A non-event: no wire, persisted-format, RPC, or flag changes -- two new server-level gauges
and a web-UI section, all reading existing tablet metadata. Old and new tservers coexist
freely; rollback simply stops exporting the metrics.

Test plan

macOS arm64, release -- all green:

  • New: FixedHybridTimeWriteIdITest.RetentionPinMetricsTrackActiveGenerations --
    deterministic gauge semantics via directly-set generations, on every replica: baseline
    zero, pinned count with a barrier placed 5s in the past (age floor asserted with
    slack), the missing-barrier edge (pinned = 1 with age = 0), and release clearing both.
  • New: PgIndexBackfillVerifier.RetentionPinMetricsReflectHeldGeneration/0 -- end-to-end
    wiring under the real activation path: with the funnel's release blocked, all three
    tservers report the pin and a positive age.
  • New: PgIndexBackfillVerifierReleased.RetentionPinMetricsClearOnRelease/0 -- gauges
    return to zero after the production release path runs.
  • Regressions: FixedHybridTimeWriteIdITest.MissingBarrierBlocksAllHistoryGC,
    FixedHybridTimeWriteIdITest.RetentionBarrierPreservesSupersededWindowVersions. All
    three new tests re-run after the review round (HTML escaping + shared age helper).

AlmaLinux (x86_64, clang21) -- all green (this PR is the stack tip, so every run below
is at its own commit c69686cd0d):

  • TSAN/ASAN/debug: fixed_hybrid_time_write_id-itest (carries
    RetentionPinMetricsTrackActiveGenerations), tablet_peer-test,
    unique_index_verifier-test; TSAN non_transactional_batch_writer-test.
  • ASAN + debug: both new pg_index_backfill-test tests, both params, isolated
    (RetentionPinMetricsReflectHeldGeneration, RetentionPinMetricsClearOnRelease).
  • ASAN + debug + release: full pg_index_backfill-test.
    The single debug-build pg_index_backfill-test failure across these runs is
    PgIndexBackfillBackendsManager.NoAbortTxn/1 (GetSafeTime 60s timeout), the pre-
    existing upstream flake: it fails 3/3 at a base commit carrying none of this stack's
    code, and has upstream hardening lineage ([YSQL] test: harden YSQL backends manager tests #19351).
  • Metrics-emission blast radius (EmitMetrics now walks every peer's generation state):
    ASAN remote_bootstrap-itest, clone-tablet-itest, tablet_bootstrap-test,
    docdb-test, pg_txn-test; debug snapshot-test, tablet_snapshots-test,
    remote_bootstrap-itest, xcluster_external_apply_bootstrap-test.

…dering generations

With the verification retention barrier enforced as a history read fence,
an orphaned active generation pins ALL history GC on its tablet --
previously findable only by reading tablet superblocks. Two tserver
gauges (updated by the existing 1s metrics poller) make a stuck barrier
visible; retention-pin observability is also a measurement-gate
requirement for the deferred-verification rollout (its retained-history
measurements need these signals):

  ts_index_backfill_pinned_tablets
    Tablet peers whose history GC is pinned by an active generation.
  ts_index_backfill_oldest_retention_barrier_age_ms
    Age of the oldest active barrier; 0 when no generation is active or
    no active generation carries a barrier hybrid time (that state blocks
    all GC on its tablet and already logs a warning every 30s).

The count includes peers in any state -- the pin is on-disk metadata, not
runtime state. The tserver /tablet detail page additionally shows a held
generation's fields (index table, base op index, write-ID floor version,
retention barrier + age) for per-tablet triage.

Test Plan:
./yb_build.sh release --cxx-test fixed_hybrid_time_write_id-itest --gtest_filter 'FixedHybridTimeWriteIdITest.RetentionPinMetricsTrackActiveGenerations'
./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillVerifier.RetentionPinMetricsReflectHeldGeneration/0'
./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillVerifierReleased.RetentionPinMetricsClearOnRelease/0'
The itest pins gauge semantics deterministically: count over directly-set
generations, age at least a barrier placed 5s in the past, missing
barrier = pinned with zero age, release clears both. The e2e tests pin
the wiring under the production activation/release lifecycle (gauges
nonzero while the funnel's release is blocked; zero after release).
Regressions: MissingBarrierBlocksAllHistoryGC,
RetentionBarrierPreservesSupersededWindowVersions.

Assisted-By: devx/08801cf6-2854-4c6f-a378-1d98d239b22d

---
_automated · Claude Fable 5 (opencode)_
@jmeehan16
jmeehan16 marked this pull request as ready for review September 1, 2026 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants