[Kernel] Allow time travel across catalogManaged transitions by PatrickJin-db · Pull Request #7645 · delta-io/delta

PatrickJin-db · 2026-09-11T09:13:00Z

Which Delta project/connector is this regarding?

Description

Previously, the validateMaxCatalogVersionPresence check was too restrictive. If a table was upgraded from path-based to catalog-managed, then using Kernel to time travel to a version before the upgrade would fail because maxCatalogVersion would be present but catalogManaged is missing in that snapshot's Protocol. Similarly, if a table was downgraded from catalog-managed to path-based, then using Kernel to time travel to a version before the downgrade would fail because maxCatalogVersion would be missing but catalogManaged was present in that snapshot's Protocol.

In principle, we need to relax those checks to validate maxCatalogVersion presence against the latest protocol, not the target snapshot protocol. In other words, enforce maxCatalogVersionPresent == latestSnapshot.isCatalogManaged. However, for version-based time travel, this would require extra I/O to load the P&M for the latest version. To reduce unnecessary P&M loads, version-based time travel will only load the latest protocol if maxCatalogVersion presence disagrees with the target version's P&M. In other words:

Query Type	When to fetch latest P&M
Latest version	We are already loading the latest snapshot anyways
Timestamp travel	We need to load the latest snapshot to resolve the version anyways
Version travel	Only when `maxCatalogVersionPresent != targetSnapshot.isCatalogManaged`

The resulting behavior for time travel queries is:

Target protocol	Latest protocol	Path-based access	Catalog-based access
Filesystem	Filesystem	Success	Failure
catalogManaged	catalogManaged	Failure	Success
Filesystem	catalogManaged	Succeeds for version-based, fails for timestamp-based	Succeeds
catalogManaged	Filesystem	Success	Succeeds for version-based, fails for timestamp-based

How was this patch tested?

Added unit tests and e2e tests for time travel across catalogManaged upgrade/downgrade versions

Does this PR introduce any user-facing changes?

No

rliao147 · 2026-09-15T21:10:10Z

+   * path-based read of a current catalogManaged table. In that case only, load the latest protocol
+   * to distinguish the two.
+   */
+  private void validateMaxCatalogVersionPresence(Engine engine, Protocol snapshotProtocol) {


is snapshotProtocol really the target version's protocol? maybe targetProtocol would be better.

rliao147 · 2026-09-15T21:26:56Z

+
+    SnapshotQueryContext latestContext =
+        SnapshotQueryContext.forLatestSnapshot(tablePath.toString());
+    Protocol latestProtocol =


The validation's latest protocol load only reads the on disk delta log and ignores the caller's staged commits, so it can misjudge whether the table is catalogManaged.

fixed by supplying ctx.logDatas

rliao147 · 2026-09-15T21:32:59Z

+    SnapshotQueryContext latestContext =
+        SnapshotQueryContext.forLatestSnapshot(tablePath.toString());
+    Protocol latestProtocol =
+        new SnapshotManager(tablePath).buildLatestSnapshot(engine, latestContext).getProtocol();


also, do we need to load the full snapshot? Don't we only need the latest log segment + P&M here?

IIUC, this method only reads P&M, but i refactored to make it more explicit that we are only loading P&M here

Keeps the maxCatalogVersion presence rule in one place instead of splitting timestamp queries into SnapshotBuilderImpl. Co-authored-by: Isaac <no-reply@databricks.com>

…ck-jin/relax-max-catalog-version-time-travel Co-authored-by: Isaac <no-reply@databricks.com>

rliao147 · 2026-10-10T01:10:49Z

-  private void validateMaxCatalogVersionPresence(Protocol protocol) {
-    boolean isCatalogManaged = TableFeatures.isCatalogManagedSupported(protocol);
+  /**
+   * Validates maxCatalogVersion presence for the current query.


I'm questioning the design decision here: when we're time traveling to a non-catalogManaged historical version, why do we need to validate the max catalog version?

Example:
v0 - not catalog managed
v1 - upgrade to catalog managed
v2 - catalog managed

Say we're trying to time travel to v0, do we really need to validate v2 (max catalog version)? This opens up some issues:

scenario 1) attempt to time travel to v0
v0 - not catalog managed, published
v1 - upgrade to catalog managed (ratified, not published)
v2 - catalog managed (ratified, not published)

scenario 2) attempt to time travel to v0
v0 - not catalog managed, published
v1 - upgrade to catalog managed (ratified, not published)
v2 - catalog managed -- introduces a table feature that kernel does not understand (ratified, not published)

Will the above fail? I think we need to think more about whether or not we want to load the protocol for the max catalog version.

Good point, in that case I think it makes sense not to validate maxCatalogVersion if the target protocol is not CCv2.

That said, the issue of Kernel failing to load a past version if it cannot read the current protocol already exists today for timestamp-based time travel right? Since timestamp-based time travel requires loading the latest snapshot.

rliao147 · 2026-10-10T01:14:04Z


 package io.delta.kernel.defaults.catalogManaged

+import java.io.File


Can we please add the following test cases?

A later change above the catalog limit:
• v0: non-catalog-managed.
• v1: catalog-managed.
• v2: published downgrade to non-catalog-managed.
• Request v0 with maxCatalogVersion= 1.

An unpublished downgrade
• v0: catalog-managed.
• v1: downgrade to non-catalog-managed, approved but unpublished.
• Supply v1’s commit; request v0 without a maxCatalogVersion

Only supply history needed for the requested version

Request published v0 with maxCatalogVersion=2, supplying only unpublished v1.

A newer version has an unsupported reader feature

Request readable v0; v2 introduces a feature Kernel cannot read.

Could we add an upgrade → downgrade → upgrade sequence, checking historical targets from each period with and without maxCatalogVersion? This would verify that an earlier downgrade doesn’t incorrectly allow a missing limit after a subsequent upgrade.

v0 — filesystem-managed v1 — upgrade to catalog-managed v2 — downgrade to filesystem-managed v3 — upgrade to catalog-managed again ← latest

Expected reads:

v0/v2: filesystem-managed targets: succeeds without maxCatalogVersion
v1: catalog-managed target: fails without maxCatalogVersion

A file-system managed target version is fully published and below any catalog ratified version, so maxCatalogVersion cannot change what is read. Skipping it avoids loading the latest version, which failed when catalog commits above the target were not provided or the latest version had a reader feature Kernel does not support. A catalogManaged target without maxCatalogVersion still checks the latest version so that versions from before a downgrade can be read by path. That check uses a new ProtocolMetadataLogReplay.isCatalogManaged, which stops at the latest protocol and skips the readability check. Adds the requested tests for later downgrades, unpublished downgrades, partial catalog commits, and unreadable later versions. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Patrick Jin <patrick.jin@databricks.com>

rliao147

Many comments around test coverage. Could you also update the PR description? thanks!

rliao147 · 2026-10-20T18:44:30Z

+
+Version time travel across a catalog-managed upgrade or downgrade is now allowed. `atVersion`
+queries no longer reject `maxCatalogVersion` for filesystem-managed versions, and require it for
+catalog-managed versions only while the table is still catalog-managed. Latest and timestamp


Latest and timestamp queries are unchanged

Could we update this sentence?

"For timestamp-based time travel, provide maxCatalogVersion if the supplied latest snapshot is catalog-managed; omit it otherwise."

Also, the user guide should not have comments like X no longer does Y. That information should be in a changlog, and we should have the user guide reflect the current state of the codebase.

rliao147 · 2026-10-20T18:47:57Z

+      val tablePath = tempDir.getCanonicalPath
+      // inCommitTimestamp of the catalog-owned-preview table's version 0
+      val v0Timestamp = 1749830855993L
+      val parsedLogData = writeCommit(


Could we also test a downgrade with checkpoint protection? This helper only removes catalogManaged. Please add coverage preserving checkpointProtection, delta.requireCheckpointProtectionBeforeVersion, and protected checkpoints, then verify latest and pre-downgrade reads return the expected data and schema.

rliao147 · 2026-10-20T19:02:13Z

+    } else if (!ctx.versionOpt.isPresent()) {
+      validateMaxCatalogVersionPresenceAgainst(targetProtocol);
+    } else if (TableFeatures.isCatalogManagedSupported(targetProtocol)
+        && !ctx.maxCatalogVersion.isPresent()) {


Could we add comments here? A published catalogManaged downgrade follows publication of earlier commits, allowing catalogManaged Delta Log history to be read without invoking the catalog.

rliao147 · 2026-10-20T19:03:35Z


 package io.delta.kernel.defaults.catalogManaged

+import java.io.File


Could we add an upgrade → downgrade → upgrade sequence, checking historical targets from each period with and without maxCatalogVersion? This would verify that an earlier downgrade doesn’t incorrectly allow a missing limit after a subsequent upgrade.

rliao147 · 2026-10-20T19:05:08Z


 package io.delta.kernel.defaults.catalogManaged

+import java.io.File


v0 — filesystem-managed v1 — upgrade to catalog-managed v2 — downgrade to filesystem-managed v3 — upgrade to catalog-managed again ← latest

Expected reads:

v0/v2: filesystem-managed targets: succeeds without maxCatalogVersion
v1: catalog-managed target: fails without maxCatalogVersion

rliao147 · 2026-10-20T19:08:16Z

+    }
+  }
+
+  test("time travel across unpublished catalogManaged downgrade") {


Can we extend this test?

v0 — catalog-managed, published v1 — catalog-managed data write, published v2 — downgrade, ratified but unpublished

We should also check:

v2 is recognized as the latest, filesystem-managed snapshot.

using that latest snapshot, a timestamp selecting v1 returns v1’s expected data without maxCatalogVersion.

rliao147 · 2026-10-20T19:15:57Z

+    }
+  }
+
+  test("time travel across catalogManaged upgrade") {


Could we also test reading commit ranges across an upgrade and downgrade? For example:

v0: filesystem-managed v1: upgrade to catalog-managed v2: catalog-managed data write v3: downgrade to filesystem-managed v4: filesystem-managed data write

Before writing v3: supply v2’s ratified commit and maxCatalogVersion=2. Request v0–v2 and verify all three commits are returned.

After writing v4: without catalog inputs, request v3 onward with no ending version. Verify v3 and v4 are returned.

Request v0–v4 and verify every version is returned in order, once each.

[Kernel] Allow time travel across catalogManaged transitions

5ddb2a5

rliao147 requested review from dengsh12 and rliao147 September 15, 2026 16:59

rliao147 suggested changes Sep 15, 2026

View reviewed changes

PatrickJin-db and others added 2 commits October 6, 2026 03:15

Always load latest P&M in version time travel queries

41926dc

Validate timestamp queries in SnapshotFactory with the other query types

4c17a33

Keeps the maxCatalogVersion presence rule in one place instead of splitting timestamp queries into SnapshotBuilderImpl. Co-authored-by: Isaac <no-reply@databricks.com>

PatrickJin-db force-pushed the patrick-jin/relax-max-catalog-version-time-travel branch from 91214eb to 4c17a33 Compare October 6, 2026 04:38

PatrickJin-db requested a review from rliao147 October 6, 2026 13:19

Merge branch 'master' of https://github.com/delta-io/delta into patri…

a24a2ff

…ck-jin/relax-max-catalog-version-time-travel Co-authored-by: Isaac <no-reply@databricks.com>

rliao147 suggested changes Oct 10, 2026

View reviewed changes

PatrickJin-db requested a review from rliao147 October 19, 2026 18:11

rliao147 suggested changes Oct 20, 2026

View reviewed changes


		package io.delta.kernel.defaults.catalogManaged

		import java.io.File

Conversation

PatrickJin-db commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which Delta project/connector is this regarding?

Description

How was this patch tested?

Does this PR introduce any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PatrickJin-db Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rliao147 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

PatrickJin-db commented Sep 11, 2026 •

edited

Loading

PatrickJin-db Oct 6, 2026 •

edited

Loading