Skip to content

[BACKPORT 2025.2][#31220] CDC: Add cleanup_stale_cdc_streams yb-admin command (#32025) - #33598

Open
egladysh wants to merge 1 commit into
yugabyte:2025.2from
Shopify:backport-c02cbb55f-2025.2
Open

[BACKPORT 2025.2][#31220] CDC: Add cleanup_stale_cdc_streams yb-admin command (#32025)#33598
egladysh wants to merge 1 commit into
yugabyte:2025.2from
Shopify:backport-c02cbb55f-2025.2

Conversation

@egladysh

Copy link
Copy Markdown
Collaborator

Summary

Adds a CleanupStaleCDCStreams RPC to the MasterReplication master
service and a corresponding cleanup_stale_cdc_streams yb-admin
command.

The command scans the cdc_state table and identifies stale entries —
rows whose CDC stream no longer exists, or whose tablet no longer
exists. It supports:

  • A dry_run flag to report stale entries without deleting them.

Example (dry run):

./bin/yb-admin \
    cleanup_stale_cdc_streams dry-run

Found 2 stale cdc_state entries (dry run).
  tablet_id: 8f3b9b3a8b2c4e2b9c7d6a5f4e3d2c1b, stream_id: 3d9f1e2a4b6c4d8e9f0123456789abcd, table_id: 000033e8000030008000000000004000, table_name: orders, reason: stream not found
  tablet_id: missing_tablet_id, stream_id: 7a8b9c0d1e2f34567890abcdef123456, reason: tablet not found

Implementation notes:

  • The cdc_state table is scanned once and materialized so both the
    tablet-collection and classification loops share one consistent
    snapshot.
  • Tablet table metadata is resolved under a single SharedLock before
    the classification loop to avoid repeated lock acquisitions.
  • xCluster streams that carry an empty namespace_id have their
    namespace resolved via their first table_id.

Upgrade/Rollback safety

The proto change is purely additive: new messages
(CleanupStaleCDCStreamsRequestPB, CleanupStaleCDCStreamsResponsePB)
and a new RPC method (CleanupStaleCDCStreams) are added to
MasterReplication. No existing message fields are modified.

On a mixed-version cluster, an older master that does not have this RPC
will return UNIMPLEMENTED when cleanup_stale_cdc_streams is invoked;
all other CDC and xCluster operations are unaffected.

Rolling back removes the yb-admin command and the RPC handler. No
on-disk state, catalog schema, or gflag defaults are changed.

Original commit: c02cbb5 / #32025

Test plan

  • CDCServiceTest.TestCleanupStaleCDCStreamsWithoutCDCStateTable
    verifies OBJECT_NOT_FOUND when the cdc_state table does not exist.
  • CDCServiceTest.TestCleanupStaleCDCStreamsDryRunAndDelete
    dry-run returns stale entries without deleting; live run deletes exactly
    those entries and leaves the valid entry untouched.
  • CDCServiceTest.TestCleanupStaleCDCStreamsNamespaceFilter — with
    a namespace filter, only entries attributable to the selected namespace
    are deleted; cross-namespace and unattributable (both stream and tablet
    missing) entries are preserved.

Co-authored-by: Sumukh-Phalgaonkar 61342752+Sumukh-Phalgaonkar@users.noreply.github.com

…yb-admin command (yugabyte#32025)

## Summary

Adds a `CleanupStaleCDCStreams` RPC to the `MasterReplication` master
service and a corresponding `cleanup_stale_cdc_streams` `yb-admin`
command.

The command scans the `cdc_state` table and identifies stale entries —
rows whose CDC stream no longer exists, or whose tablet no longer
exists. It supports:

- A `dry_run` flag to report stale entries without deleting them.

Example (dry run):

```
./bin/yb-admin \
    cleanup_stale_cdc_streams dry-run

Found 2 stale cdc_state entries (dry run).
  tablet_id: 8f3b9b3a8b2c4e2b9c7d6a5f4e3d2c1b, stream_id: 3d9f1e2a4b6c4d8e9f0123456789abcd, table_id: 000033e8000030008000000000004000, table_name: orders, reason: stream not found
  tablet_id: missing_tablet_id, stream_id: 7a8b9c0d1e2f34567890abcdef123456, reason: tablet not found
```

Implementation notes:
- The `cdc_state` table is scanned once and materialized so both the
tablet-collection and classification loops share one consistent
snapshot.
- Tablet table metadata is resolved under a single `SharedLock` before
the classification loop to avoid repeated lock acquisitions.
- xCluster streams that carry an empty `namespace_id` have their
namespace resolved via their first `table_id`.

## Upgrade/Rollback safety

The proto change is purely additive: new messages
(`CleanupStaleCDCStreamsRequestPB`, `CleanupStaleCDCStreamsResponsePB`)
and a new RPC method (`CleanupStaleCDCStreams`) are added to
`MasterReplication`. No existing message fields are modified.

On a mixed-version cluster, an older master that does not have this RPC
will return `UNIMPLEMENTED` when `cleanup_stale_cdc_streams` is invoked;
all other CDC and xCluster operations are unaffected.

Rolling back removes the `yb-admin` command and the RPC handler. No
on-disk state, catalog schema, or gflag defaults are changed.

Original commit: c02cbb5 / yugabyte#32025

## Test plan

- [ ] `CDCServiceTest.TestCleanupStaleCDCStreamsWithoutCDCStateTable` —
verifies `OBJECT_NOT_FOUND` when the `cdc_state` table does not exist.
- [ ] `CDCServiceTest.TestCleanupStaleCDCStreamsDryRunAndDelete` —
dry-run returns stale entries without deleting; live run deletes exactly
those entries and leaves the valid entry untouched.
- [ ] `CDCServiceTest.TestCleanupStaleCDCStreamsNamespaceFilter` — with
a namespace filter, only entries attributable to the selected namespace
are deleted; cross-namespace and unattributable (both stream and tablet
missing) entries are preserved.

---------

Co-authored-by: Sumukh-Phalgaonkar <61342752+Sumukh-Phalgaonkar@users.noreply.github.com>
@egladysh

Copy link
Copy Markdown
Collaborator Author

Trigger Jenkins

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants