Skip to content

Commit 8edd88f

Browse files
authored
docs(spark): refresh the user guide against the shipped connector (#9607)
The user guide still named `0.78.0`, seven releases behind the `0.85.0` on Maven Central, so copying the `--jars` URL or the dependency snippet pulled a stale JAR. `java/vortex-spark/README.md` had drifted too. Three things the page omits although they ship: `MapType` (covered by `testWriteAndReadMapColumns`), filter pushdown (`SupportsPushDownV2Filters` since #7785), and Azure paths (`fs.azure.account.key*` is forwarded from the Hadoop config). ## Tests `make -C docs html` succeeds under the repo's `--fail-on-warning`, same warning count as develop; the documented 2.13 URL returns HTTP 200. ## AI assistance Written with agentic AI assistance; I verified each claim against the code it describes. Signed-off-by: jackylee <qcsd2011@gmail.com>
1 parent b92cee5 commit 8edd88f

2 files changed

Lines changed: 31 additions & 11 deletions

File tree

docs/user-guide/spark.md

Lines changed: 26 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ connector is published to Maven Central in two flavors:
66
- `dev.vortex:vortex-spark_2.13` for Spark 4.x (Scala 2.13)
77
- `dev.vortex:vortex-spark_2.12` for Spark 3.5.x (Scala 2.12)
88

9-
Use the `all` classifier JAR (e.g. `vortex-spark_2.13-0.78.0-all.jar`). It is self-contained:
9+
Use the `all` classifier JAR (e.g. `vortex-spark_2.13-0.85.0-all.jar`). It is self-contained:
1010
it bundles the Vortex JNI bindings, native libraries for Linux (x86_64 and aarch64) and macOS
1111
(aarch64), and relocates its Arrow, Guava, and Jackson dependencies to avoid classpath
1212
conflicts with Spark. The thin (unclassified) JAR does not work on its own because it
@@ -18,21 +18,21 @@ For `spark-shell`, `spark-submit`, or `pyspark`, pass the `all` JAR with `--jars
1818
accepts either a local path or a URL, so you can point directly at Maven Central:
1919

2020
```shell
21-
spark-shell --jars https://repo1.maven.org/maven2/dev/vortex/vortex-spark_2.13/0.78.0/vortex-spark_2.13-0.78.0-all.jar
21+
spark-shell --jars https://repo1.maven.org/maven2/dev/vortex/vortex-spark_2.13/0.85.0/vortex-spark_2.13-0.85.0-all.jar
2222
```
2323

2424
Or equivalently when building a session programmatically, e.g. in PySpark:
2525

2626
```python
2727
spark = (
2828
SparkSession.builder
29-
.config("spark.jars", "/path/to/vortex-spark_2.13-0.78.0-all.jar")
29+
.config("spark.jars", "/path/to/vortex-spark_2.13-0.85.0-all.jar")
3030
.getOrCreate()
3131
)
3232
```
3333

3434
```{note}
35-
`--packages dev.vortex:vortex-spark_2.13:0.78.0` does not work: `--packages` cannot select
35+
`--packages dev.vortex:vortex-spark_2.13:0.85.0` does not work: `--packages` cannot select
3636
the `all` classifier and resolves the thin JAR, which fails at runtime with
3737
`NoClassDefFoundError: dev/vortex/relocated/...`.
3838
```
@@ -47,7 +47,7 @@ To depend on the connector from a JVM project, add the `all` classifier to the d
4747
Gradle (Kotlin):
4848

4949
```kotlin
50-
implementation("dev.vortex:vortex-spark_2.13:0.78.0:all")
50+
implementation("dev.vortex:vortex-spark_2.13:0.85.0:all")
5151
```
5252

5353
Maven:
@@ -56,7 +56,7 @@ Maven:
5656
<dependency>
5757
<groupId>dev.vortex</groupId>
5858
<artifactId>vortex-spark_2.13</artifactId>
59-
<version>0.78.0</version>
59+
<version>0.85.0</version>
6060
<classifier>all</classifier>
6161
</dependency>
6262
```
@@ -79,6 +79,11 @@ partition per file.
7979

8080
Column pruning is pushed down — only the columns referenced by the query are read from the file.
8181

82+
Filter pushdown is also supported. Comparisons (`=`, `<>`, `>`, `>=`, `<`, `<=`), `IS NULL`,
83+
`IS NOT NULL`, `IN`, the string predicates `STARTS_WITH`, `ENDS_WITH` and `CONTAINS`, and a bare
84+
boolean column are evaluated during the native scan. Anything else, including predicates on
85+
partition columns, is returned to Spark for post-scan evaluation.
86+
8287
## Writing Vortex Files
8388

8489
```java
@@ -187,6 +192,7 @@ INSERT INTO vortex.`/path/to/data` VALUES (1, 'Alice', 20);
187192
| `TimestampNTZType` | Timestamp (microseconds, no timezone) |
188193
| `ArrayType` | List |
189194
| `StructType` | Struct |
195+
| `MapType` | Map |
190196

191197
## S3 Support
192198

@@ -198,3 +204,17 @@ Dataset<Row> df = spark.read()
198204
.option("path", "s3://bucket/path/to/data")
199205
.load();
200206
```
207+
208+
## Azure Support
209+
210+
Azure Data Lake Storage paths work the same way:
211+
212+
```java
213+
Dataset<Row> df = spark.read()
214+
.format("vortex")
215+
.option("path", "abfss://container@account.dfs.core.windows.net/path/to/data")
216+
.load();
217+
```
218+
219+
Account keys are read from the Hadoop configuration: any `fs.azure.account.key*` entry is
220+
forwarded to the native reader as the storage account key.

java/vortex-spark/README.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ Two flavors are published to Maven Central:
1111
| `dev.vortex:vortex-spark_2.13` | Spark 4.x | 2.13 |
1212
| `dev.vortex:vortex-spark_2.12` | Spark 3.5.x | 2.12 |
1313

14-
Use the `all` classifier JAR (e.g. `vortex-spark_2.13-0.83.0-all.jar`). It is self-contained:
14+
Use the `all` classifier JAR (e.g. `vortex-spark_2.13-0.85.0-all.jar`). It is self-contained:
1515
it bundles the Vortex JNI bindings, native libraries for Linux (x86_64 and aarch64) and macOS
1616
(aarch64), and relocates its Arrow, Guava, and Jackson dependencies to avoid classpath
1717
conflicts with Spark. The thin (unclassified) JAR does not work on its own because it
@@ -23,28 +23,28 @@ Pass the `all` JAR to `spark-shell`, `spark-submit`, or `pyspark` with `--jars`.
2323
either a local path or a URL, so you can point directly at Maven Central:
2424

2525
```shell
26-
spark-shell --jars https://repo1.maven.org/maven2/dev/vortex/vortex-spark_2.13/0.83.0/vortex-spark_2.13-0.83.0-all.jar
26+
spark-shell --jars https://repo1.maven.org/maven2/dev/vortex/vortex-spark_2.13/0.85.0/vortex-spark_2.13-0.85.0-all.jar
2727
```
2828

2929
Or configure it on the session builder, e.g. in PySpark:
3030

3131
```python
3232
spark = (
3333
SparkSession.builder
34-
.config("spark.jars", "/path/to/vortex-spark_2.13-0.83.0-all.jar")
34+
.config("spark.jars", "/path/to/vortex-spark_2.13-0.85.0-all.jar")
3535
.getOrCreate()
3636
)
3737
```
3838

39-
Note that `--packages dev.vortex:vortex-spark_2.13:0.83.0` does not work: `--packages` cannot
39+
Note that `--packages dev.vortex:vortex-spark_2.13:0.85.0` does not work: `--packages` cannot
4040
select the `all` classifier and resolves the thin JAR, which fails at runtime with
4141
`NoClassDefFoundError: dev/vortex/relocated/...`.
4242

4343
To depend on the connector from a JVM project instead, add the `all` classifier to the
4444
dependency:
4545

4646
```kotlin
47-
implementation("dev.vortex:vortex-spark_2.13:0.83.0:all")
47+
implementation("dev.vortex:vortex-spark_2.13:0.85.0:all")
4848
```
4949

5050
## Usage

0 commit comments

Comments
 (0)