Skip to content

[Spark] Reuse DeltaSourceSnapshot for V2 streaming initial snapshots - #7798

Merged
timothyw553 merged 4 commits into
delta-io:masterfrom
PorridgeSwim:you-zhou_data/dsv2-delta-source-snapshot
Oct 5, 2026
Merged

timothyw553 merged 4 commits into
delta-io:masterfrom
PorridgeSwim:you-zhou_data/dsv2-delta-source-snapshot

Conversation

@PorridgeSwim

Copy link
Copy Markdown
Collaborator

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Description

DeltaV2MicroBatchStream currently has two initial-snapshot implementations: a driver-side InitialSnapshotCache that materializes the full file list, and a flag-gated DataFrame-based snapshot cache (streaming.distributedInitialSnapshot). The V1 DeltaSource already handles initial-snapshot ordering and lifecycle through DeltaSourceSnapshot.

This PR makes DeltaV2MicroBatchStream use DeltaSourceSnapshot exclusively:

  • Lazily create one DeltaSourceSnapshot for the initial snapshot version and reuse it while the stream reads that snapshot.
  • Follow the same lifecycle as the V1 source: release the snapshot when planning moves past the initial snapshot and when the stream stops.
  • Use the same snapshot path for initial change-data-feed reads.
  • Remove the driver-side InitialSnapshotCache, the DataFrame snapshot cache routing, and their helpers from DeltaV2MicroBatchStream.
  • Add KernelActionUtils.toKernelScanAddFile, which converts a V1 AddFile to the Kernel scan AddFile. It keeps deletion vectors, row-tracking fields, tags, and stats.
  • Remove the initial-snapshot file-count limit (streaming.initialSnapshotMaxFiles) and its DELTA_STREAMING_INITIAL_SNAPSHOT_TOO_LARGE error. DeltaSourceSnapshot does not collect the whole file list on the driver, so the limit no longer applies.

How was this patch tested?

  • New KernelActionUtilsSuite test round-trips a V1 AddFile through toKernelScanAddFile and addFileFromKernel, covering partition values, tags, deletion vector, row-tracking fields, and stats.
  • Existing V2 streaming suites cover initial snapshot and initial CDF reads.

Does this PR introduce any user-facing changes?

No. The removed config and error class were internal and applied only to the V2 streaming source.

@PorridgeSwim
PorridgeSwim force-pushed the you-zhou_data/dsv2-delta-source-snapshot branch from fec9670 to a29a860 Compare October 5, 2026 18:05

@timothyw553 timothyw553 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@timothyw553
timothyw553 merged commit 71e9845 into delta-io:master Oct 5, 2026
49 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants