Intra-fragment ZoneMap pruning for Spark reads

[majin1102]

I'm working on improving ZoneMap pruning for Spark reads.

Background

lance-spark already loads ZoneMap statistics on the driver, but currently reduces the result to surviving fragment IDs. More precise information—physical row ranges inside a fragment—is discarded.

As a result, Spark cannot pass intra-fragment pruning results to executors. The native scanner may also query the same ZoneMap again when processing the original filter.

Proposal

Preserve the driver-side ZoneMap result as fragment-local physical slices:

Spark predicates
    → driver queries ZoneMap
    → FragmentSlice(fragmentId, rowOffset, rowCount)
    → LanceSplit
    → executor scanner
    → physical range reads

FragmentSlice describes a physical row interval within a fragment. It is called a slice rather than a split because it narrows an existing Spark task instead of creating a new task.

The original filter is always evaluated by Lance. Unsupported predicates, incomplete index coverage, or invalid ZoneMap metadata conservatively fall back to a full-fragment scan.

Related work

• lance#8951 adds the Java/JNI FragmentSlice scanner API. Lance intersects physical slices with deletion vectors, scalar-index results, and the original filter.
• lance-spark#807 evaluates ZoneMap statistics on the driver and passes matching ranges through LanceSplit to normal reads and COUNT(*).
• lance#9034 addresses the limitation of use_scalar_index = false, which disables all scalar indices. It adds ignoredScalarIndices so executors can skip only the ZoneMap already consumed by the driver while continuing to use other indices, such as BTree.

[majin1102]

@hamersaw @jackye1995 @LuciferYang Could you take a look to the dependent lance-format/lance#8951 and lance-format/lance#9034 when you have a chance? Would love to hear your thoughts.