Skip to content

Commit 61399f8

Browse files
committed
[KYUUBI #7789] Bump Paimon 2.1.0 and enable Paimon tests for Spark 4.1
### Why are the changes needed? Paimon 1.4.2 has no Spark 4.1 build, so the Spark 4.1 profile had to depend on the Spark 4.0 artifact (`paimon-spark-4.0_2.13`) and keep `PaimonTest` excluded. Paimon 2.1.0 ships `paimon-spark-4.1_2.13`, which unblocks running the Paimon authz suite on Spark 4.1. - `paimon.version` 1.4.2 -> 2.1.0 and `paimon.artifact` -> `paimon-spark-${spark.binary.version}_${scala.binary.version}` for the Spark 4.1/4.2/4.3 profiles - Spark 4.1: drop `org.apache.kyuubi.tags.PaimonTest` from `maven.plugin.scalatest.exclude.tags` - Spark 4.2/4.3: `PaimonTest` stays excluded; both profiles alias `spark.binary.version` to 4.1, so they keep resolving `paimon-spark-4.1_2.13` - Spark 4.0 keeps Paimon 1.4.2: the Spark 4.0 build of Paimon 2.x declares Spark 4.0.3, but its classes reference the Spark 4.1-only `org.apache.spark.sql.connector.catalog.constraints.Constraint` (`Table.constraints()`), so it cannot run on Spark 4.0 ### How was this patch tested? `PaimonCatalogRangerSparkExtensionSuite`, Java 17 and Scala 2.13: - Spark 4.1.2 + Paimon 2.1.0: 37/37 passed - Spark 4.0.3 + Paimon 1.4.2: 37/37 passed - Spark 4.0.3 + Paimon 2.0.0 and 2.1.0: fails with `NoClassDefFoundError: org.apache.spark.sql.connector.catalog.constraints.Constraint`, raised inside the authz reflection helper - hence the Spark 4.0 profile stays on 1.4.2 `HiveCatalogRangerSparkExtensionSuite` on Spark 4.1.2 + Paimon 2.1.0: 53/53 passed. ``` build/mvn -B test -pl extensions/spark/kyuubi-spark-authz -am -Pspark-4.1 -Pscala-2.13 -Pspark-provided \ -Dtest=none -DwildcardSuites=org.apache.kyuubi.plugin.spark.authz.ranger.PaimonCatalogRangerSparkExtensionSuite ``` ### Was this patch assisted by generative AI tooling? Assisted-by: Codex with gemini-4-argon Closes #7789 from pan3793/paimon-2.1. Closes #7789 c84ee87 [Cheng Pan] nit: fix comment wording 25f329c [Cheng Pan] nit c24af36 [Cheng Pan] Bump Paimon 2.1.0 and enable Paimon tests for Spark 4.1 Authored-by: Cheng Pan <pan3793@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org>
1 parent 564627e commit 61399f8

1 file changed

Lines changed: 9 additions & 8 deletions

File tree

‎pom.xml‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2080,6 +2080,8 @@
20802080
<delta.version>4.3.1</delta.version>
20812081
<delta.artifact>delta-spark_${spark.binary.version}_${scala.binary.version}</delta.artifact>
20822082
<hudi.version>1.2.0</hudi.version>
2083+
<!-- Paimon 2.x references the Spark 4.1+ Table.constraints() API in its Spark 4.0
2084+
build, causing java.lang.NoClassDefFoundError at runtime. -->
20832085
<paimon.version>1.4.2</paimon.version>
20842086
<paimon.artifact>paimon-spark-${spark.binary.version}_${scala.binary.version}</paimon.artifact>
20852087
<!-- Exclude SparkLocalClusterTest due to Scala 2.13.17 changes serialVersionUID of scala.collection.immutable.ArraySeq -->
@@ -2103,10 +2105,9 @@
21032105
<delta.version>4.3.1</delta.version>
21042106
<delta.artifact>delta-spark_${spark.binary.version}_${scala.binary.version}</delta.artifact>
21052107
<hudi.version>1.2.0</hudi.version>
2106-
<!-- TODO: update once Paimon support Spark 4.1. -->
2107-
<paimon.version>1.4.2</paimon.version>
2108-
<paimon.artifact>paimon-spark-4.0_${scala.binary.version}</paimon.artifact>
2109-
<maven.plugin.scalatest.exclude.tags>org.scalatest.tags.Slow,org.apache.kyuubi.tags.PaimonTest</maven.plugin.scalatest.exclude.tags>
2108+
<paimon.version>2.1.0</paimon.version>
2109+
<paimon.artifact>paimon-spark-${spark.binary.version}_${scala.binary.version}</paimon.artifact>
2110+
<maven.plugin.scalatest.exclude.tags>org.scalatest.tags.Slow</maven.plugin.scalatest.exclude.tags>
21102111
<spark.archive.name>spark-${spark.version}-bin-hadoop3.tgz</spark.archive.name>
21112112
</properties>
21122113
</profile>
@@ -2132,8 +2133,8 @@
21322133
<delta.artifact>delta-spark_${spark.binary.version}_${scala.binary.version}</delta.artifact>
21332134
<hudi.version>1.2.0</hudi.version>
21342135
<!-- TODO: update once Paimon support Spark 4.2. -->
2135-
<paimon.version>1.4.2</paimon.version>
2136-
<paimon.artifact>paimon-spark-4.0_${scala.binary.version}</paimon.artifact>
2136+
<paimon.version>2.1.0</paimon.version>
2137+
<paimon.artifact>paimon-spark-${spark.binary.version}_${scala.binary.version}</paimon.artifact>
21372138
<maven.plugin.scalatest.exclude.tags>org.scalatest.tags.Slow,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.IcebergTest,org.apache.kyuubi.tags.PaimonTest,org.apache.kyuubi.tags.HudiTest</maven.plugin.scalatest.exclude.tags>
21382139
<spark.archive.name>spark-${spark.version}-bin-hadoop3.tgz</spark.archive.name>
21392140
</properties>
@@ -2161,8 +2162,8 @@
21612162
<delta.artifact>delta-spark_${spark.binary.version}_${scala.binary.version}</delta.artifact>
21622163
<hudi.version>1.2.0</hudi.version>
21632164
<!-- TODO: update once Paimon support Spark 4.3. -->
2164-
<paimon.version>1.4.2</paimon.version>
2165-
<paimon.artifact>paimon-spark-4.0_${scala.binary.version}</paimon.artifact>
2165+
<paimon.version>2.1.0</paimon.version>
2166+
<paimon.artifact>paimon-spark-4.1_${scala.binary.version}</paimon.artifact>
21662167
<maven.plugin.scalatest.exclude.tags>org.scalatest.tags.Slow,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.IcebergTest,org.apache.kyuubi.tags.PaimonTest,org.apache.kyuubi.tags.HudiTest</maven.plugin.scalatest.exclude.tags>
21672168
<!-- TODO remove once Spark 4.3.0 GA -->
21682169
<spark.archive.mirror>https://dist.apache.org/repos/dist/dev/spark/v4.3.0-rc2-bin</spark.archive.mirror>

0 commit comments

Comments
 (0)