Skip to content

Convert DataFrameSnapshotCache from Scala to Java - #7873

Closed
HaoSun1993 wants to merge 2 commits into
delta-io:masterfrom
HaoSun1993:v2-read-batch1-cache-20261015
Closed

HaoSun1993 wants to merge 2 commits into
delta-io:masterfrom
HaoSun1993:v2-read-batch1-cache-20261015

Conversation

@HaoSun1993

@HaoSun1993 HaoSun1993 commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Withdrawn: the DataFrameSnapshotCache conversion is no longer pursued in this series. The description and validation below document the withdrawn proposal.

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Description

Visibility review

Make the class private[v2]. The public repository caller, subclass, and exposed-type inventory found no direct consumers requiring broader access. There is no companion.

The connector lexical boundary is io.delta.spark.internal.v2. The JVM classes and required Java entry points retain their public access. Scala source access is checked separately against that package scope.

Convert DataFrameSnapshotCache from Java to Scala. Use the Scala DataFrame alias for
Dataset[Row] while preserving the public constructor, getters, and AutoCloseable contract.
Every close() call continues to invoke unpersist() on the stored DataFrame.

No callers, subclasses, or direct tests of this cache currently exist in the repository.
JVM inspection confirms the same fields and signatures, with matching getter and close
instructions. Performance has not been benchmarked.

How was this patch tested?

Visibility revision validation

Spark 4.0/4.1/4.2 compilation and changed-source scalafmtCheck passed on cumulative source 21fa13e45c5431f8786cb3a957dc7c15328e4adc. All 72 existing Spark 4.2 tests passed with zero failures/errors/skips; baseline case names and outcomes match. Each converted source in this PR is identical to its corresponding validated source. No tests, cases, assertions, or skips were changed.

Current revision: 0ba63c99f6c9dc04dddfad94a4994636a507d592. The following earlier conversion checks and performance observations remain applicable.

Original conversion validation

  • build/sbt -DsparkVersion=4.0 sparkV2/compile
  • build/sbt -DsparkVersion=4.1 sparkV2/compile
  • build/sbt -DsparkVersion=4.2 sparkV2/compile 'sparkV2/testOnly *CDCDataFileTest *ColumnReorderReadFunctionTest *IndexedFileTest'
  • sparkV2/Compile/scalafmtCheck, restricted to the converted Scala source.

All commands passed. No tests, assertions, expected results, or skips were changed.

Does this PR introduce any user-facing changes?

No.

Signed-off-by: Hao Sun <86338940+HaoSun1993@users.noreply.github.com>
Signed-off-by: Hao Sun <hao.sun@databricks.com>
@HaoSun1993 HaoSun1993 closed this Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant