[SPARK] Support SHOW PARTITIONS on managed Delta tables resolved as V1Table by pratham76 · Pull Request #7757 · delta-io/delta

pratham76 · 2026-09-29T06:00:25Z

Which Delta project/connector is this regarding?

Description

As part of this PR we now support SHOW PARTITIONS command on managed Delta table that resolve as V1Table instead of DeltaTableV2

DeltaAnalysis already had a rule rewriting ShowPartitions(ResolvedTable(..., V1Table, ...)) into ShowDeltaPartitionsCommand for exactly this shape. However, Spark's built-in ResolveSessionCatalog analyzer rule also matches ShowPartitions on any table resolved through the session catalog, and eagerly downgrades it to the V1 ShowPartitionsCommand first, since both rules run in the same fixed-point resolution batch, with Spark's rule ordered ahead of Delta's extended resolution rules. As a result, the plan was already converted to ShowPartitionsCommand before DeltaAnalysis's rule ever saw it, and Spark's built-in command doesn't understand Delta's log-based partitioning, failing with PARTITION_SCHEMA_IS_EMPTY on managed Delta tables reached via this path.

Adds a second case in DeltaAnalysis that catches the already-downgraded ShowPartitionsCommand, re-resolves the table identifier, and reroutes it to ShowDeltaPartitionsCommand when the target actually is a Delta table; non-Delta tables fall through unchanged to Spark's built-in command as before.

How was this patch tested?

Added a new test to ShowDeltaPartitionsSuite, which installs a delegating catalog (DummySessionCatalogInner) as spark_catalog so the table loads as V1Table, then verifies SHOW PARTITIONS still returns Delta's partition values.

Does this PR introduce any user-facing changes?

Yes, Now SHOW PARTITIONS command would be supported on managed Delta tables as well

pratham76 · 2026-09-29T06:02:53Z

@timothyw553 Could you PTAL? Thanks!

timothyw553

thanks @pratham76 for the PR! I've left two comments

timothyw553 · 2026-10-14T18:58:37Z

+      val resolvedTable = session.sessionState.analyzer.ResolveRelations(
+        UnresolvedTable(s.tableName.nameParts, "SHOW PARTITIONS"))
+      DeltaTableV2.maybeExtractFrom(resolvedTable) match {
+        case Some(_) => ShowDeltaPartitionsCommand(resolvedTable, s.spec.getOrElse(Map.empty))
+        case None => s
+      }


ShowPartitionsCommand already targets Spark’s session catalog. However, with spark.sql.legacy.v1IdentifierNoCatalog=true, its table identifier can contain only db.t, without the catalog name.

The new lookup treats db.t as a fresh name and resolves it against the current catalog. If another catalog is current and contains a Delta table named db.t, the command selects that table instead.

my suggestion:

case s: ShowPartitionsCommand => + val tableName = s.tableName.copy( + catalog = s.tableName.catalog.orElse(Some(CatalogManager.SESSION_CATALOG_NAME))) val resolvedTable = session.sessionState.analyzer.ResolveRelations( - UnresolvedTable(s.tableName.nameParts, "SHOW PARTITIONS")) + UnresolvedTable(tableName.nameParts, "SHOW PARTITIONS"))

Thanks. Updated to pin the catalog to CatalogManager.SESSION_CATALOG_NAME when tableName.catalog is absent, exactly as you suggested.

timothyw553 · 2026-10-14T18:59:16Z

+        checkAnswer(
+          sql("SHOW PARTITIONS delta_table"),
+          Seq(
+            Row("2024-01-01"),
+            Row("2024-01-02")
+          )
+        )


nit: The test checks only an unfiltered, successful request. It would still pass if the new interception code discarded s.spec and always listed every partition.

Extend the same dummy-catalog block instead of creating another table fixture:

checkAnswer( sql("SHOW PARTITIONS delta_table PARTITION (date = '2024-01-01')"), Seq(Row("2024-01-01")) )

Agreed, added a second assertion in the same withSQLConf block, SHOW PARTITIONS delta_table PARTITION (date = '2024-01-01') now checked to return only that one row, so the test would catch partitionSpec being dropped.

pratham76 · 2026-10-15T02:52:14Z

Thanks for taking a look @timothyw553, Have addressed both the comments. Could you PTAL? Thanks!

timothyw553

small nits

timothyw553 · 2026-10-15T04:44:39Z

+      sql("INSERT INTO delta_table VALUES (2, '2024-01-02', 200.0)")
+
+      spark.sessionState.catalogManager.reset()
+      withSQLConf("spark.sql.catalog.spark_catalog" -> classOf[DummySessionCatalogInner].getName) {


should this along with reset be in a try {} finally {} ?

fixed — wrapped the withSQLConf block in try/finally so the trailing catalogManager.reset() always runs even if an assertion inside fails

timothyw553 · 2026-10-15T04:45:32Z

+      spark.sessionState.catalogManager.reset()
+      withSQLConf("spark.sql.catalog.spark_catalog" -> classOf[DummySessionCatalogInner].getName) {
+        checkAnswer(
+          sql("SHOW PARTITIONS delta_table"),


[ai assisted] This fixture exercises the fallback that handles Spark’s already-converted ShowPartitionsCommand. It does not verify:

The new direct ShowPartitions(... V1Table ...) branch.

The explicit session-catalog lookup when another catalog is active and the legacy identifier setting is enabled.

The fallback’s promise to leave non-Delta commands unchanged.

These are coverage gaps

Thanks, all three addressed with new tests:

Added a non-Delta (parquet) table case proving the ShowPartitionsCommand fallback is a no-op when the target isn't Delta.

Added a regression test for the session-catalog-pinning fix: a second catalog is made current, spark.sql.legacy.v1IdentifierNoCatalog is enabled, and a same-named table exists in both — confirms SHOW PARTITIONS still returns the session catalog's data.

Added a DummyV1TableCatalog (registered under a non-spark_catalog name, returning V1Table for Delta tables) to exercise the original ShowPartitions(V1Table) rule directly, since it's otherwise unreachable under the session catalog — ResolveSessionCatalog always intercepts there first.

pratham76 · 2026-10-15T16:11:30Z

Thanks @timothyw553 , have addressed these nits as well. PTAL again.

Spark's ResolveSessionCatalog analyzer rule eagerly rewrites SHOW PARTITIONS on a table resolved through the session catalog into the V1 ShowPartitionsCommand, running before DeltaAnalysis's existing ShowPartitions -> ShowDeltaPartitionsCommand rule gets a chance to intercept it. This left managed Delta tables that resolve as V1Table (e.g. through a delegating catalog) falling through to Spark's built-in command, which doesn't understand Delta's log-based partitioning and fails with PARTITION_SCHEMA_IS_EMPTY. Add a second rule that catches the already-downgraded ShowPartitionsCommand, re-resolves the table identifier, and reroutes it to ShowDeltaPartitionsCommand when the target is actually a Delta table.

timothyw553

LGTM. Thank you for the contribution!

pratham76 · 2026-10-15T19:04:53Z

Thanks @timothyw553 for the approval, Could we check this in if no additional changes required ? CI has passed. I will create a separate PR for doc update.

pratham76 force-pushed the show-partitions-managed-table-support branch from 8c21b83 to 17b90fb Compare September 29, 2026 06:01

foss-contributor added bug Something isn't working delta-datalayout delta-spark Issues and PRs related to general Delta-Spark issues labels Sep 29, 2026

seewishnew requested a review from openinx September 30, 2026 09:29

timothyw553 reviewed Oct 14, 2026

View reviewed changes

pratham76 force-pushed the show-partitions-managed-table-support branch from 17b90fb to 3e30379 Compare October 15, 2026 02:49

timothyw553 reviewed Oct 15, 2026

View reviewed changes

pratham76 force-pushed the show-partitions-managed-table-support branch from 3e30379 to 0128dd2 Compare October 15, 2026 16:07

pratham76 force-pushed the show-partitions-managed-table-support branch from 0128dd2 to 16f0694 Compare October 15, 2026 16:11

timothyw553 approved these changes Oct 15, 2026

View reviewed changes

timothyw553 merged commit 7e5d70d into delta-io:master Oct 15, 2026
43 checks passed

pratham76 mentioned this pull request Oct 16, 2026

[Spark][Docs] Document SHOW PARTITIONS support for managed tables #7884

Open

1 of 5 tasks

Conversation

pratham76 commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which Delta project/connector is this regarding?

Description

How was this patch tested?

Does this PR introduce any user-facing changes?

Uh oh!

pratham76 commented Sep 29, 2026

Uh oh!

timothyw553 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pratham76 commented Oct 15, 2026

Uh oh!

timothyw553 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pratham76 commented Oct 15, 2026

Uh oh!

timothyw553 left a comment

Choose a reason for hiding this comment

Uh oh!

pratham76 commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pratham76 commented Sep 29, 2026 •

edited

Loading