NullPointerException on Write with Pyspark 3.5.2 (Scala 2.12)

[duongphannamhung]

Try to use pyspark 3.5.2 to write but not work with connector 3.6->3.8 (have tried all). Have tried read and worked well.

Describe the bug (required)
This is code to reproduce

from pyspark.sql.types import StringType, LongType, StructType, StructField

data = [("u_001", 1, "user1"), ("u_002", 2, "user2")]
schema = StructType([
    StructField("_vertexId", StringType(), False),
    StructField("user_id", LongType(), True),
    StructField("username", StringType(), True),
])
df = spark.createDataFrame(data, schema)

(
    df.write
    .format("com.vesoft.nebula.connector.NebulaDataSource")
    .option("type", "vertex")
    .option("operateType", "write")
    .option("writeMode", "insert")
    .option("spaceName", "test_spark")
    .option("label", "test_user")
    .option("vertexField", "_vertexId")
    .option("batch", 1)
    .option("metaAddress", "metad0:9559")
    .option("graphAddress", "graphd:9669")
    .option("user", "root")
    .option("passwd", "nebula")
    .mode("overwrite")
    .save()
)

Here is log:

java.lang.NullPointerException
	at com.vesoft.nebula.connector.writer.NebulaVertexWriter.<init>(NebulaVertexWriter.scala:34)
	at com.vesoft.nebula.connector.writer.NebulaVertexWriterFactory.createWriter(NebulaSourceWriter.scala:27)
	at org.apache.spark.sql.execution.datasources.v2.WritingSparkTask.run(WriteToDataSourceV2Exec.scala:436)
	at org.apache.spark.sql.execution.datasources.v2.WritingSparkTask.run$(WriteToDataSourceV2Exec.scala:425)
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:491)
	at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:388)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
	at org.apache.spark.scheduler.Task.run(Task.scala:141)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

[duongphannamhung]

Does it because we haven't supported Spark 3.5 @wey-gu @Nicole00