[#33444] DocDB: Add retention-pin observability for index-backfill ordering generations - #33657
Open
jmeehan16 wants to merge 1 commit into
Conversation
…dering generations
With the verification retention barrier enforced as a history read fence,
an orphaned active generation pins ALL history GC on its tablet --
previously findable only by reading tablet superblocks. Two tserver
gauges (updated by the existing 1s metrics poller) make a stuck barrier
visible; retention-pin observability is also a measurement-gate
requirement for the deferred-verification rollout (its retained-history
measurements need these signals):
ts_index_backfill_pinned_tablets
Tablet peers whose history GC is pinned by an active generation.
ts_index_backfill_oldest_retention_barrier_age_ms
Age of the oldest active barrier; 0 when no generation is active or
no active generation carries a barrier hybrid time (that state blocks
all GC on its tablet and already logs a warning every 30s).
The count includes peers in any state -- the pin is on-disk metadata, not
runtime state. The tserver /tablet detail page additionally shows a held
generation's fields (index table, base op index, write-ID floor version,
retention barrier + age) for per-tablet triage.
Test Plan:
./yb_build.sh release --cxx-test fixed_hybrid_time_write_id-itest --gtest_filter 'FixedHybridTimeWriteIdITest.RetentionPinMetricsTrackActiveGenerations'
./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillVerifier.RetentionPinMetricsReflectHeldGeneration/0'
./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillVerifierReleased.RetentionPinMetricsClearOnRelease/0'
The itest pins gauge semantics deterministically: count over directly-set
generations, age at least a barrier placed 5s in the past, missing
barrier = pinned with zero age, release clears both. The e2e tests pin
the wiring under the production activation/release lifecycle (gauges
nonzero while the funnel's release is blocked; zero after release).
Regressions: MissingBarrierBlocksAllHistoryGC,
RetentionBarrierPreservesSupersededWindowVersions.
Assisted-By: devx/08801cf6-2854-4c6f-a378-1d98d239b22d
---
_automated · Claude Fable 5 (opencode)_
jmeehan16
marked this pull request as ready for review
September 1, 2026 16:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Retention-pin observability for deferred unique-index verification (#33444). With #33655
enforcing the retention barrier as a history read fence, an orphaned active ordering
generation pins ALL history GC on its tablet -- previously findable only by reading tablet
superblocks. This was called out as a measurement-gate requirement in the read-fence review:
the rollout's retained-history measurements need these signals. Two server-level tserver
gauges, updated by the existing 1s
EmitMetricspoller:ts_index_backfill_pinned_tablets-- tablet peers whose history GC is pinned by an activeindex-backfill ordering generation. Counts peers in any state: the pin is on-disk
metadata, not runtime state.
ts_index_backfill_oldest_retention_barrier_age_ms-- age of the oldest active barrier,clamped at zero (the barrier derives from a master-chosen read time, which clock skew can
put slightly ahead of this server's clock right after activation). 0 when no generation is
active or no active generation carries a barrier hybrid time -- that missing-barrier state
blocks all history GC on its tablet and already logs a warning every 30s.
Alerting guidance: key alerts on
ts_index_backfill_pinned_tablets > 0persistinglonger than the expected build+verify duration -- never on age alone. The worst state (an
active generation with a missing barrier) blocks all history GC yet reports age 0; the
pinned count plus the tserver's periodic missing-barrier warning are the signals that cover
it.
The tserver
/tabletdetail page additionally shows a held generation's fields (indextable, base op index, write-ID floor version, retention barrier + age) for per-tablet
triage.
Stacked on #33403, #33404, #33484, #33544, #33553, #33580, #33584, #33601, #33602, #33654,
#33655, and #33656, based on
feature-stack/Shopify/uniq-idx-7-fail-closed-gateper thefeature-stack workflow, so the diff is exactly this PR's own commit. A failing pr-title
check on stacked PRs is a known gap in that check; the stack merges bottom-up and retargets
as parents merge.
Upgrade/Rollback safety
A non-event: no wire, persisted-format, RPC, or flag changes -- two new server-level gauges
and a web-UI section, all reading existing tablet metadata. Old and new tservers coexist
freely; rollback simply stops exporting the metrics.
Test plan
macOS arm64, release -- all green:
FixedHybridTimeWriteIdITest.RetentionPinMetricsTrackActiveGenerations--deterministic gauge semantics via directly-set generations, on every replica: baseline
zero, pinned count with a barrier placed 5s in the past (age floor asserted with
slack), the missing-barrier edge (pinned = 1 with age = 0), and release clearing both.
PgIndexBackfillVerifier.RetentionPinMetricsReflectHeldGeneration/0-- end-to-endwiring under the real activation path: with the funnel's release blocked, all three
tservers report the pin and a positive age.
PgIndexBackfillVerifierReleased.RetentionPinMetricsClearOnRelease/0-- gaugesreturn to zero after the production release path runs.
FixedHybridTimeWriteIdITest.MissingBarrierBlocksAllHistoryGC,FixedHybridTimeWriteIdITest.RetentionBarrierPreservesSupersededWindowVersions. Allthree new tests re-run after the review round (HTML escaping + shared age helper).
AlmaLinux (x86_64, clang21) -- all green (this PR is the stack tip, so every run below
is at its own commit
c69686cd0d):fixed_hybrid_time_write_id-itest(carriesRetentionPinMetricsTrackActiveGenerations),tablet_peer-test,unique_index_verifier-test; TSANnon_transactional_batch_writer-test.pg_index_backfill-testtests, both params, isolated(
RetentionPinMetricsReflectHeldGeneration,RetentionPinMetricsClearOnRelease).pg_index_backfill-test.The single debug-build
pg_index_backfill-testfailure across these runs isPgIndexBackfillBackendsManager.NoAbortTxn/1(GetSafeTime60s timeout), the pre-existing upstream flake: it fails 3/3 at a base commit carrying none of this stack's
code, and has upstream hardening lineage ([YSQL] test: harden YSQL backends manager tests #19351).
EmitMetricsnow walks every peer's generation state):ASAN
remote_bootstrap-itest,clone-tablet-itest,tablet_bootstrap-test,docdb-test,pg_txn-test; debugsnapshot-test,tablet_snapshots-test,remote_bootstrap-itest,xcluster_external_apply_bootstrap-test.