[#33444] DocDB: Add persisted index-backfill ordering-generation record and tablet fences - #33580
Open
jmeehan16 wants to merge 1 commit into
Conversation
This was referenced Aug 27, 2026
Open
✅ Deploy Preview for infallible-bardeen-164bc9 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
…rd and tablet fences
Deferred uniqueness verification scans the marked versions that unique-index
backfill wrote at the fixed backfill hybrid time. That scan is only sound
against a stable tablet set and a validated write-ID sequence, so each index
tablet needs a durable record of the in-flight backfill's ordering state --
one that survives restart, remote bootstrap, and inheritance by split
children.
The record (`IndexBackfillOrderingGenerationPB`, superblock field 39,
persistence following the `cdc_sdk_safe_time` pattern): `active`, the owning
index `table_id`, `base_op_index`, `retention_barrier_ht`, and
`write_id_floor_version` (numbering pinned in the proto: 1 = the initial
floor, 0 = absent/unknown). The full combined shape is defined here;
`retention_barrier_ht` is consumed by the verification read fence and
identity/activation by the master orchestration, both in later parts.
The durable form of an inactive generation is field absence:
`LoadFromSuperBlock` clears the in-memory record when the field is absent
(so a remote-bootstrap superblock replacement from a source without a
generation cannot resurrect a released one), and the setter normalizes
inactive input to the default -- the caller contract is "set an active
generation or set {}", keeping in-memory and durable state identical.
While a generation is active on a tablet:
- Marked fixed-hybrid-time writes must carry Raft indexes strictly above
`base_op_index` (`WriteOperation::ValidateLeaderOpId`). The base is the
activation operation's own Raft index, so an index at or below it cannot
belong to this generation -- it would indicate misrouting or replay of a
foreign sequence. (Rejecting marked writes when *no* generation is active
arrives with the master activation flow; until then marked writes remain
production-unreachable.)
- Tablet splitting is rejected before Raft append
(`SplitOperation::ValidateLeaderOpId`) -- the fail-closed layer under the
master-side split fence (next part). The master's split path retries and
succeeds once the generation is released.
- Tablet cloning is rejected the same way. Clone targets build fresh metadata
(`RaftGroupMetadata::CreateNew`), not a superblock copy, so a clone would
carry hard-linked snapshot data containing generation-scoped marked write
IDs with no generation record describing them and no backfill job tracking
their lifecycle.
`TEST_bypass_index_backfill_ordering_generation_split_fence` forces the
split fence open to demonstrate the property the fence is *not* needed for:
split children inherit the parent's marked entries and continue drawing
write IDs from their own Raft indexes, so per-key write-ID uniqueness holds
even across a split (the fence exists for verification-scan stability and
lifecycle tracking, not key correctness).
Test Plan:
./yb_build.sh release --cxx-test tablet_peer-test --gtest_filter 'TabletPeerTest.FixedHybridTimeWriteRejectedAtOrBelowGenerationBase'
./yb_build.sh release --cxx-test tablet_peer-test --gtest_filter 'TabletPeerTest.IndexBackfillOrderingGenerationPersistsAcrossReload'
./yb_build.sh release --cxx-test tablet_peer-test --gtest_filter 'TabletPeerTest.IndexBackfillOrderingGenerationClearedBySuperblockReplacement'
./yb_build.sh release --cxx-test tablet_peer-test --gtest_filter 'TabletPeerTest.SplitRejectedWhileOrderingGenerationActive'
./yb_build.sh release --cxx-test tablet_peer-test --gtest_filter 'TabletPeerTest.CloneRejectedWhileOrderingGenerationActive'
./yb_build.sh release --cxx-test fixed_hybrid_time_write_id-itest --gtest_filter 'FixedHybridTimeWriteIdITest.MasterSplitRefusedWhileOrderingGenerationActive'
./yb_build.sh release --cxx-test fixed_hybrid_time_write_id-itest --gtest_filter 'FixedHybridTimeWriteIdITest.OrderingGenerationSurvivesRemoteBootstrap'
./yb_build.sh release --cxx-test fixed_hybrid_time_write_id-itest --gtest_filter 'FixedHybridTimeWriteIdITest.SplitWithFenceBypassedPreservesPerKeyWriteIdUniqueness'
Regressions: 3a fixed-hybrid-time tablet_peer tests and leader-change itest,
0a/0b abort tests, PgIndexBackfillSkipAllRaftOrdering e2e -- all green.
Assisted-By: devx/08801cf6-2854-4c6f-a378-1d98d239b22d
---
_automated · Claude Fable 5 (opencode)_
jmeehan16
force-pushed
the
feature-stack/Shopify/uniq-idx-3b-i-generation-record
branch
from
August 31, 2026 18:36
11ccf5d to
cc9ba40
Compare
jmeehan16
added a commit
that referenced
this pull request
Aug 31, 2026
…fecycle from the master The persisted ordering generation (#33580) had no production lifecycle: nothing activated it before marked backfill writes, nothing released it, and the master-side split suppression protecting a backfill is in-memory only (lost on failover). This part wires the full lifecycle and flips marked-write gating fail-closed. Activation rides ChangeMetadataOperation (a new `ChangeMetadataRequestPB.index_backfill_ordering_generation` variant, applied via `Tablet::UpdateIndexBackfillOrderingGeneration` exactly like `mark_backfill_done`): Raft-replicated, follower-applied, WAL-replayed, and `base_op_index` is the activation operation's own Raft index, so every replica and every replay derives the same base with no master-side bookkeeping. Re-activation (failover resume) is idempotent -- the base only moves up. Master flow, SKIP_ALL jobs on PGSQL indexed tables only (the mode rides YSQL chunk requests; YCQL backfill writes are never marked, so generations would only fence their splits without protecting anything): - Before the first chunk (`DoBackfill`), the job disables and drains index-table splitting, then fans out one waited `UpdateOrderingGenerationForTablet` task per index tablet (the `GetSafeTimeForTablet` join pattern). All acks -> chunks launch; any failure -> the job aborts before a single marked write exists, and CREATE INDEX fails cleanly. The drain closes the activation/split TOCTOU: consensus already rejects CHANGE_METADATA_OP while a split is pending, the generation fence rejects splits after activation applies, and a split appended in the append-to-apply window can at worst fail the activation task -- never corrupt a scan set that marked writes exist in. - Release is fire-and-forget from the terminal funnel (`UpdateIndexPermissionsForIndexes`), which every success/failure/abort path traverses. The tserver metadata validator converges stragglers, mirroring the retain_delete_markers machinery: tablets with an active generation join its existing GetBackfillStatus poll, and the generation is released locally (no Raft) once the master reports a terminal state. `IndexStatusPB` gains `BACKFILL_FAILED` (removal-path index permissions map to it) so a failed SKIP_ALL job whose funnel release was lost cannot leave an orphaned invalid index permanently split-fenced and retention-pinned; the retain_delete_markers heal itself stays success-only. - The tablet-split manager refuses to split an index whose indexed table has a durable SKIP_ALL backfill job. The manager's table validation takes no catalog locks itself -- the indexed table is a caller-resolved parameter, because the manual-split path enters through ValidateSplitCandidateUnlocked with the catalog mutex already held (a recursive shared acquisition is fatal under lock_debug) (`SysTablesEntryPB.backfill_jobs`, survives failover, cleared in the funnel), with an INFO-level skip reason -- today's two in-memory suppressions cover only the indexed table and evaporate on failover. The tablet-side generation fence stays as the fail-closed layer. Gating flip: `WriteOperation::ValidateLeaderOpId` now rejects marked writes with *no* active generation (previously deferred) -- a marked write outside a generation would store versions nothing tracks or releases (e.g. a stale chunk retry after the job's terminal state). Tests that drive marked writes directly now activate a generation first. `write_id_floor_version` numbering lands as `kIndexBackfillWriteIdFloorVersion = 1`. The mode selector remains hardcoded to CHECK_ALL (#33484), so this whole flow stays production-unreachable; tests drive it through the master-side TEST override. Test Plan: ./yb_build.sh release --cxx-test tablet_peer-test --gtest_filter 'TabletPeerTest.OrderingGenerationChangeMetadataOpSetsBaseFromOwnRaftIndex' ./yb_build.sh release --cxx-test tablet_peer-test --gtest_filter 'TabletPeerTest.FixedHybridTimeWriteRejectedAtOrBelowGenerationBase' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillSkipAllRaftOrdering.OrderingGenerationActivatedAndReleased/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillSkipAllActivationFailure.ActivationFailureFailsCreateIndexCleanly/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillSkipAllBlocked.SplitFencedDuringBackfill/0' Retrofitted (marked writes now require an active generation): ./yb_build.sh release --cxx-test tablet_peer-test --gtest_filter 'TabletPeerTest.FixedHybridTimeWrite*' ./yb_build.sh release --cxx-test fixed_hybrid_time_write_id-itest Regressions: skip_all e2e + marker canary through the real activation flow, uniq-idx-1 mode tests, DuplicatesExistBeforeBackfill, 3b-i generation tests, 0a/0b abort tests, and drop-path tests for the permissions mapping change (PgIndexBackfillTest.Drop, PgIndexBackfillFastClientTimeout.DropWhileBackfilling) -- all green. Assisted-By: devx/08801cf6-2854-4c6f-a378-1d98d239b22d --- _automated · Claude Fable 5 (opencode)_
This was referenced Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Deferred uniqueness verification (#33444) scans the marked versions that unique-index
backfill wrote at the fixed backfill hybrid time (#33553). That scan is only sound against a
stable tablet set and a validated write-ID sequence, so each index tablet needs a durable
record of the in-flight backfill's ordering state -- one that survives restart, remote
bootstrap, and inheritance by split children and clones.
The record (
IndexBackfillOrderingGenerationPB, superblock field 39, persistence followingthe
cdc_sdk_safe_timepattern):active, the owning indextable_id,base_op_index,retention_barrier_ht, andwrite_id_floor_version. The full combined shape is definedhere;
retention_barrier_htis consumed by the verification read fence andidentity/activation by the master orchestration, both in later parts. The durable form of an
inactive generation is field absence, and the setter normalizes to keep in-memory and
durable state identical;
LoadFromSuperBlockclears the in-memory record when the field isabsent, so a remote-bootstrap superblock replacement cannot resurrect a released generation.
Enforcement while a generation is active, all before WAL append via the
ValidateLeaderOpIdhook (#33553):
strictly above
base_op_index(the activation operation's own index, assigned by themaster flow in the next part): an index at or below the base cannot belong to this
generation and would indicate misrouting or replay of a foreign sequence. Rejecting marked
writes when no generation is active also arrives with the master activation flow.
SplitOperationis rejected while active: verification must runagainst a stable tablet set. This is the fail-closed layer under the master-side split
fence (next part; today's master-side suppression is in-memory only and lost on failover).
TEST_bypass_index_backfill_ordering_generation_split_fenceexists to prove the fence isfor scan stability and lifecycle tracking, not key correctness: with the fence forced
open, split children inherit the parent's marked entries, continue drawing write IDs from
their own Raft indexes, and per-key write-ID uniqueness still holds (tested).
CloneOperationis rejected while active: the clone would inherit anactive generation with no backfill job attached, permanently fencing its splits and
pinning its retention.
With no generation active (the only production state until the master orchestration lands),
the hooks reduce to reading a defaulted struct and returning OK for every split, clone, and
marked write.
Stacked on #33403, #33404, #33484, #33544, and #33553, based on
feature-stack/Shopify/uniq-idx-3a-marked-write-storageper the feature-stack workflow, sothe diff is exactly this PR's own commit. A failing pr-title check on stacked PRs is a known
gap in that check; the stack merges bottom-up and retargets as parents merge.
Upgrade/Rollback safety
Proto change:
RaftGroupReplicaSuperBlockPB.index_backfill_ordering_generation(newoptional message, field 39). Additive:
error and ignore the field -- no crash, no load failure. However, the superblock is
regenerated from in-memory state on every metadata flush (not round-tripped), so the first
flush on an old binary drops the record; a subsequent re-upgrade would see no generation
and silently stop enforcing the base validation and the split/clone fences. This is
production-safe today because nothing can activate a generation: the setter's only callers
are tests, and the master activation flow (next part) is gated behind the same
one-release-stack constraint as [#33444] DocDB: Add marked fixed-hybrid-time write path with Raft-index write IDs #33553 -- AutoFlag promotion, which requires all nodes
upgraded, is what authorizes the first activation.
their own copy when the (Raft-replicated) activation operation applies. The only wire path
for the field itself is a remote-bootstrap superblock copy, where an old-binary recipient
drops the unknown field on its first flush -- the same benign behavior as rollback. Until
the activation operation exists, mixed clusters never contain the field at all.
Behavioral surface pre-activation: the split/clone/write hooks run unconditionally on
leaders but return OK with the defaulted (inactive) record, so splits, clones, and writes
are unaffected -- covered by the full
tablet-split-itest(TSAN+debug),clone-tablet-itest(ASAN+debug),remote_bootstrap-itest(ASAN), and a full releasepg_index_backfill-testregression, all green.No gflag default changes; the new flag is TEST-only.
Test plan
macOS arm64, release -- all green:
TabletPeerTest.IndexBackfillOrderingGenerationPersistsAcrossReload-- fullcombined record round-trips through the superblock; releasing persists as field
absence.
TabletPeerTest.IndexBackfillOrderingGenerationClearedBySuperblockReplacement--remote-bootstrap-style
ReplaceSuperBlockwith an absent field clears a stale localgeneration.
TabletPeerTest.FixedHybridTimeWriteRejectedAtOrBelowGenerationBase-- markedwrite with index at/below the base rejected before WAL append; index above the base
admitted; released generation does not constrain.
TabletPeerTest.SplitRejectedWhileOrderingGenerationActive/CloneRejectedWhileOrderingGenerationActive-- both rejected pre-WAL, nothing loggedor committed.
FixedHybridTimeWriteIdITest.MasterSplitRefusedWhileOrderingGenerationActive--end-to-end through the master split path: split refused via the tablet fence while
active; releasing the generation lets the same split complete (positive control).
FixedHybridTimeWriteIdITest.OrderingGenerationSurvivesRemoteBootstrap--tombstoned follower is remote-bootstrapped back by the leader; the recovered replica
carries the generation.
FixedHybridTimeWriteIdITest.SplitWithFenceBypassedPreservesPerKeyWriteIdUniqueness-- fence forced open, tablet split; children inherit the generation and the parent's
marked entries; a marked rewrite of an inherited key at the same fixed hybrid time
lands as a second distinct physical version; per-key (key, HT, write ID) uniqueness
holds on both children.
TabletPeerTest.FixedHybridTimeWritesUseRaftIndexWriteIdand the [#33444] DocDB: Add marked fixed-hybrid-time write path with Raft-index write IDs #33553suites.
AlmaLinux (x86_64, clang21) -- all green:
tablet_peer-test,fixed_hybrid_time_write_id-itest,non_transactional_batch_writer-test, fulltablet-split-itest(the split-fence hookruns for every split).
tablet_peer-test,fixed_hybrid_time_write_id-itest,non_transactional_batch_writer-test, fullclone-tablet-itest, fullremote_bootstrap-itest(superblock replacement surface).tablet_peer-test,fixed_hybrid_time_write_id-itest,non_transactional_batch_writer-test,tablet_bootstrap-test, fullclone-tablet-itest, fulltablet-split-itest.pg_index_backfill-test(181/181) -- inertness regression; this changeis invisible to the pg layer until the master orchestration lands.