Convert DeltaV2TableShims from Java to Scala

[HaoSun1993]

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Description

Convert the Spark 4.2 DeltaV2TableShims implementation from Java to Scala and move it to the Scala shim source directory. Preserve schema-evolution decisions, metadata-only DELETE dispatch and predicate short-circuiting.

Use Scala Option for the capability helper and catalog hook. Update both Scala callers and Spark 4.0/4.1 Java shim variants together. Reuse the existing query-context option and adapt it with ScalaUtils.toJavaOptional at the Java metadata-only DELETE executor boundary. Keep Spark-required arrays, TableChange/TableCapability/predicate hierarchies and Supports* interfaces for their Spark contracts. KernelEngine and protocol/metadata objects are existing library/domain interfaces with no equivalent representation to migrate. Direct Arrays.stream traversal avoids collection copies and extra adapters.

Keep the Scala class private[v2] for its class-type consumers/subclasses within io.delta.spark.internal.v2. Keep the companion public because InMemorySparkTable in org.apache.spark.sql.delta.catalog calls its helper across separate package trees. Class and constructor remain JVM-public.

Compatibility: the internal helper/hook return descriptors change from Optional to Option; repository callers are migrated and already-compiled consumers must be rebuilt. No required stable external contract was found. Scala protected hooks emit JVM-public methods, so Java subclasses with protected overrides must widen them when rebuilt (protocol, metadata, kernelEngine, initialSnapshot, catalogTable).

The adapter may allocate one Optional per catalog-backed metadata-only DELETE, outside predicate traversal; empty uses Optional.empty. Capability Some replaces the prior Optional.of wrapper. No persistent fields, collection copies or per-row adapters are added. Performance is unmeasured; allocation elimination is not assumed. No tests, cases, assertions or coverage are added.

How was this patch tested?

• Fresh build/sbt -DsparkVersion=<version> 'sparkV2/compile' passed for Spark 4.0, 4.1 and 4.2 on commit 418750bcd35b067d3619da391aa29de80b8f4ebf (tree 15e935babefcf7a947af891c93e65d75a051e6be).
• Targeted Spark 4.2 converted-shim scalafmtCheck (3.8.6) passed. A previous full sparkV2/scalastyle run failed on two existing deltahadoopconfiguration violations at unchanged lines KernelContext.scala:39 and DeltaV2Table.scala:151; no full module lint pass is claimed.
• No tests, cases, assertions or coverage added. Broader CI remains pending. Performance has not been benchmarked; the scalar cost is described above.

Does this PR introduce any user-facing changes?

No intended query behavior changes. Scala class access is intentionally restricted to V2; the companion remains public. The Java subclass source-compatibility caveat is described above.