Support the Vortex file format

[alexey-milovidov]

Closes: #87327
Related: ClickHouse/rust_vendor#74
Related: ClickHouse/rust_vendor#79
Related: #115096
Related: #114373
Related: #116021

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Added support for reading and writing the Vortex columnar file format (the Vortex input and output format). This closes #87327.

Documentation entry for user-facing changes

The implementation uses the Rust vortex crate (v0.84.0) through a new C FFI crate rust/workspace/vortex (_ch_rust_vortex), following the same pattern as prql and polyglot. Data crosses the FFI boundary through the Arrow C Data Interface and is converted with the same ArrowColumnToCHColumn/CHColumnToArrowColumn code as the Arrow format. IO is delegated back to ClickHouse through callbacks: reads go through ClickHouse's own read buffers (range reads for seekable inputs, whole-file buffering otherwise), and the produced file is streamed into the output buffer. The library spawns no threads of its own: a scan is split into tasks that ClickHouse runs on its own thread pools (max_parsing_threads for decoding, max_download_threads for reads), writing is driven on the calling thread, and Rust panics are caught at the FFI boundary and turned into exceptions.

Features:

• Reading with projection pushdown: only the columns used by the query are read from the file.
• Parallel reading (merged from feat: parallel reading for vortex #115096 by @m7kss1): the file is split into row ranges that are read, filtered, and decoded concurrently; nearby segment reads are coalesced into one request. Chunks are returned as soon as they are decoded, so the row order is not guaranteed unless input_format_vortex_preserve_order is set.
• Schema inference and count()-only queries answered from file metadata without reading data.
• Writing with the library's default adaptive compression (BtrBlocks-style cascading encodings + zstd), including a valid empty file for empty results.
• Graceful errors on malformed and truncated files (fuzzer-friendly: no aborts, Rust panics become exceptions).

Limitations (documented in docs/reference/formats/Vortex.mdx):

• Map, Int128/UInt128/Int256/UInt256, IPv6, and Interval columns cannot be written (no corresponding Vortex type).
• String and FixedString are written as Vortex Binary (ClickHouse strings are arbitrary bytes, while Vortex requires Utf8 to be valid UTF-8).
• The format is disabled in MSan builds: the MSan-instrumented library (with origin tracking) is so large that linking unit_tests_dbms overflows the 2 GiB R_X86_64_PC32 relocation range (same approach as wasmtime and delta-kernel-rs).
• Writing is single-threaded in this first version: the adaptive compressor samples many encodings per column, so writes are slower than Parquet — parallelism can be added later. Reading is parallel since feat: parallel reading for vortex #115096 was merged into this PR: on ClickBench, reads went from geomean ~26.8x slower than Parquet to ~1.2x, winning 9 of 43 queries (see the benchmark results). Filter pushdown (added in Enhancement for Vortex format #114373, input_format_vortex_filter_push_down, on by default) lets the scan skip the statistics zones the predicate rules out and decode only the matching rows of the rest: on a file of 1 000 000 sorted values, WHERE n = 500000 reads 66 KiB with the pushdown against 7.5 MiB without it. The most selective ClickBench queries are nevertheless still 1.5-5.5x slower than Parquet. The pushdown was reworked in ref: rework filter pushdown: ActionsDAG converter, temporal types and… #116021 by @m7kss1 (merged into this PR): the filter is translated directly from the query's ActionsDAG instead of going through a KeyCondition, which adds Bool, Date, Date32 and DateTime64 columns and LIKE/NOT LIKE/startsWith prefix ranges, and reports what reached the scan through the VortexFilterPushdownConjunctsPushed, VortexFilterPushdownConjunctsDropped, VortexScanSplits and VortexScanEmptySplits profile events.

The 127 new vendored Rust crates were added in ClickHouse/rust_vendor#74, and ClickHouse/rust_vendor#79 moved them to 0.84.0. Both are merged, so contrib/rust_vendor points at main rather than at a pull request branch.

---

Workflow [PR]
Sync PR [sync-upstream/pr/112950]

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
ClickHouse-docs	🟢 Ready	View Preview	Aug 1, 2026, 10:03 PM

[clickhouse-gh]

Workflow [PR], commit [aa8fba4]

Summary: ❌

job_name	test_name	status	info	comment
Integration tests (amd_asan_ubsan, db disk, old analyzer, 6/6)		FAIL
	test_s3_plain_rewritable/test.py::test[s3_plain_rewritable_with_metadata_cache-data_with_cache/]	FAIL	cidb
	test_s3_plain_rewritable/test.py::test_projections[s3_plain_rewritable]	FAIL	cidb
	test_s3_plain_rewritable/test.py::test_projections[cache_s3_plain_rewritable]	FAIL	cidb
	test_s3_plain_rewritable/test.py::test_projections[s3_plain_rewritable_with_metadata_cache]	FAIL	cidb
Fast test (arm_darwin)		DROPPED
Build (amd_darwin)		DROPPED
Build (arm_darwin)		DROPPED
Build (arm_v80compat)		DROPPED
Build (amd_freebsd)		DROPPED
Build (ppc64le)		DROPPED
Build (amd_compat)		DROPPED
Build (amd_musl)		DROPPED
Build (riscv64)		DROPPED

---

AI Review

Summary

This PR adds native Vortex read/write support, filter pushdown, parallel scan execution, and the Rust FFI bridge. The implementation is close, but two contract mismatches still need to be fixed before merge: the scan callback API documents behavior that the current implementation cannot uphold, and the public pushdown contract overstates Date support on normal vortex.date columns.

Findings

⚠️ Majors

• [rust/workspace/vortex/include/vortex_ffi.h:201-217] The scan callback contract is internally inconsistent. VortexBlockInputFormat already relies on onChunk being callable before vortex_ffi_scan_create returns (src/Processors/Formats/Impl/Vortex/VortexBlockInputFormat.cpp:465-472), so the first callbacks do not have an FFI_VortexScan * yet and cannot actually call vortex_ffi_scan_cancel even though the header says that is allowed from on_chunk. Once the handle is published, the C++ failure path calls vortex_ffi_scan_cancel before returning non-zero (src/Processors/Formats/Impl/Vortex/VortexBlockInputFormat.cpp:276-285), which suppresses on_finish even though the same block says a non-zero return is reported there. The terminal callback sequence therefore depends on timing. Suggested fix: either publish the scan handle before any on_chunk can fire, or narrow/specify the contract so on_chunk does not promise cancellation and the cancel + non-zero path has defined semantics.
• [src/Processors/Formats/Impl/Vortex/VortexExpressionConverter.cpp:213-214] The implementation still rejects Date headers on the real vortex.date / Arrow DATE32 carrier, but input_format_vortex_filter_push_down and the Vortex docs now advertise Date pushdown. The writer emits ClickHouse Date as Arrow DATE32 (src/Processors/Formats/Impl/CHColumnToArrowColumn.cpp:137), so file(..., 'Vortex', 'd Date') silently loses pushdown on normal Vortex date columns and only the special Date-over-raw-UInt16 case from 05045_vortex_pushdown_types.sh is covered. Suggested fix: either teach this DATE32 branch how to preserve Date overflow semantics for Date headers, or narrow the public contract back to Date32 and add a focused regression test for the real vortex.date path.

Final Verdict

• Status: ⚠️ Request changes
• Minimum required actions: fix the scan callback contract mismatch, and align Date pushdown behavior with the documented contract, either in code or by narrowing the docs/settings/tests.

[alexey-milovidov]

🕵 @groeneai, investigate the two Hung check failed, possible deadlock found stress-test failures and provide a fix in a separate PR (if one is already in progress, link it here):

• Stress test (amd_msan): a fuzzer query is stuck in ActionsDAG::evaluatePartialResult / ActionsDAG::updateHeader while building an ExpressionStep in the planner — uninterruptible CPU-bound expression evaluation, possibly the same class as Uninterruptible CPU-bound function evaluation over a large value #112203.
• Stress test (arm_release): a fuzzer query over merge(REGEXP('.+'), '.*') joined to a constant subquery is stuck in ReadFromMerge::filterTablesAndCreateChildrenPlans during QueryPlan::optimize (optimizePrimaryKeyConditionAndLimit → applyFilters), single-threaded, tens of seconds in plan optimization over the stress database's many tables.

Both hangs are unrelated to this PR: the hung queries do not involve the Vortex format at all, and the amd_msan binary does not even include it (Vortex is disabled under MSan).

[groeneai]

Both hangs are the same defect, and it is not the one the amd_msan half is described as. Neither is caused by this PR.

amd_msan is not ActionsDAG::evaluatePartialResult. The hung query is

SELECT * FROM one AS A INNER JOIN system.one AS one ON A.dummy = one.dummy
             INNER JOIN merge('^') AS two ON A.dummy = two.dummy

and its stack (hung_check.log, thread 5193) is:

AnalysisTableExpressionData::ensureColumnMembershipSetsArePopulated
IdentifierResolver::tryResolveIdentifierFromTableExpression
... QueryAnalysisPass::run
ReadFromMerge::getModifiedQueryInfo
ReadFromMerge::createChildrenPlans          StorageMerge.cpp:849
ReadFromMerge::filterTablesAndCreateChildrenPlans
ReadFromMerge::applyFilters                 StorageMerge.cpp:1790
QueryPlanOptimizations::optimizePrimaryKeyConditionAndLimit
QueryPlan::optimize

The evaluatePartialResult / updateHeader / ExpressionStep frames do appear in that artifact, but in the Top messages not matching their format strings section, as the stack of a completed sleepEachRow exception (Code: 44. The argument of function sleepEachRow must be constant) raised on an unrelated HTTPHandler query. The reason they look like the hung query is that the CI report only renders the last 32 KiB of hung_check.log, and the hung row lives in the excluded head: 84% of the amd_msan log and 91% of the arm_release log are outside that window, and ReadFromMerge appears 0 times inside it. Reading the artifact directly instead of the report body is what separates the two.

So both rows are ReadFromMerge child-plan construction under QueryPlan::optimize, differing only in the leaf: amd_msan in ensureColumnMembershipSetsArePopulated, arm_release in FunctionFactory::tryGet under resolveFunction.

Cost vs interruptibility are two separate things, and only the first has a fix.

Cost: your #113140 is the right fix for it, and the amd_msan leaf is precisely the function its comment names as being rebuilt once per identifier. That accounts for the wall time.

Interruptibility: #113140 does not address it. Its 87 added lines contain no cancellation check (0 occurrences of isCancelled, checkTimeLimit, QueryStatus, ProcessList), and StorageMerge.cpp has none anywhere in its 2046 lines. Both rows show is_cancelled: 1 with max_execution_time = 10, yet ran 1757 s and 3994 s single-threaded. That is the gap: cancellation is observed between pipeline steps, and ReadFromMerge::applyFilters builds every child plan inside one optimizePrimaryKeyConditionAndLimit call with no checkpoint, so a cancelled query cannot leave. #113140 shrinks the window a lot but does not close it; a wide enough merge still exceeds any timeout uninterruptibly.

The phase counters confirm the hang is in the still-running phase rather than anything already finished: recorded phases sum to 6.6 s of 1757 s (amd_msan) and 29.4 s of 3994 s (arm_release), because QueryPlanOptimizeMicroseconds is only committed when its scope exits.

I am authoring the cancellation-checkpoint fix and will link the PR here when it opens. #112203 is a different mechanism (uninterruptible evaluation of one large value); this is uninterruptible plan construction, closer to #113003 and covered in general by #47272.

Unrelated to this PR, confirmed rather than assumed. Your MSan argument holds: rust/workspace/CMakeLists.txt in this PR sets ENABLE_VORTEX OFF when SANITIZE STREQUAL "memory", so the amd_msan binary cannot contain the format. For arm_release the diff touches 23 files, none of them under Storages/, Analyzer/, Planner/ or QueryPlan/, so it cannot appear on this stack; neither hung query references the format, and both shapes are recorded on other PRs.

[groeneai]

Fixing PR for the ReadFromMerge::createChildrenPlans plan-time cancellation hung check on my
CI finish ledger: #113415 (external, open).

I had a fix for this on a local branch, but @alexey-milovidov opened #113415 with the identical
guard at the identical call site, so I dropped mine rather than open a duplicate. My analysis and
the one coverage gap I measured are in
#113415 (comment).

[alexey-milovidov]

🕵 CI triage for the reds on a56a056dafe (all unrelated to this PR; origin/master has been merged in and CI restarted on e5634f18090):

• Stress test (arm_release) and Stress test (amd_tsan): the server stopped responding mid-stress — the ReadFromMerge::createChildrenPlans plan-time cancellation hang diagnosed earlier in this PR; the fix Honor query cancellation while building child plans of a Merge table #113415 merged to master after these runs and is now pulled into this branch by the master merge.
• Stress test (arm_tsan): Logical error: 'Not-ready Set is passed as the second argument for function 'notNullIn'' — known open issue Logical error: Not-ready Set is passed as the second argument for function 'A (STID: 0250-41a5) #107619 (fixes in flight: Fix _path / _file filtering for object storage, file and url #112968, Fix logical error for a single-argument or in tryExtractCommonExpressions #113412).
• AST fuzzer (amd_tsan): Logical error: 'Invalid number of columns in chunk pushed to OutputPort. Expected 0, found 1' (STID: 2270-2caa) on a window-function query over remoteSecure with SAMPLE + QUALIFY. The existing open issues Logical error: Invalid number of columns in chunk pushed to OutputPort. Expected A, found B (STID: 2270-3452) #96656 and Logical error: Invalid number of columns in chunk pushed to OutputPort. Expected A, found B (STID: 2270-32ca) #103695 are the same error family but different stack IDs. @groeneai, investigate this failure: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=112950&sha=a56a056dafe6b02647111a53b27d34cf38b0579a&name_0=PR&name_1=AST%20fuzzer%20%28amd_tsan%29 and provide a fix in a separate PR. If a fix is already in progress, link it here.

[groeneai]

The AST fuzzer STID 2270-2caa failure on a56a056dafe is covered by a fix I already have open: #113469

I confirmed that on your exact query rather than by inference. A pre-fix binary reproduces your log line byte for byte (Expected 0, found 1, empty Header:, Chunk: Nullable(size = 2, UInt8(size = 2), UInt8(size = 2)), aborting in RemoteSource::prepare); the branch head returns the 8 correct rows and no error. Both arms asserted by build id.

Mechanism: cume_dist() OVER () references no column of the table expression, and defaultValueOfArgumentType(k) is constant-folded, so the outer query reads nothing from the shard. prepareBuildQueryPlanForTableExpression then synthesizes one column purely to learn the row count, and the window function is deferred to the initiator along with the projection that would have consumed that column, so nothing keeps it and it is pruned. The initiator's mergeable header ends up empty while the shard still streams the surviving Nullable(UInt8) constant, hence 1 column where 0 are expected.

The crash is the visible half. When no constant survives there is no error and rows are silently dropped instead: over two shards of 1000 rows, SELECT count(*) OVER () FROM remote(...) returned 1000 rows pre-fix, and 0 with prefer_localhost_replica = 0, against 2000 correct. The locally-read shard survives, every remote one is lost.

SAMPLE, QUALIFY, the four shards, and remoteSecure rather than remote are all incidental: dropping each of them still aborts pre-fix. What is load-bearing is more than one shard plus a window function referencing no column of the table. count(c0) OVER (), a single shard, and enable_analyzer = 0 are all clean pre-fix, so the fix is scoped to the analyzer path.

113469 covers this carrier; I am not claiming it clears the whole message class. Agreed that 96656 and 103695 are the same family with different stack IDs. One caveat on keying: within this family a single STID pools more than one query shape, so the shape identifies a carrier more reliably than the STID does.

[alexey-milovidov]

🕵 Rebased onto master and closed out the 0.84 loose ends.

Conflict with master. The only real one was base/base/sanitizer_options.h: master widened the Keeper suppression from race_top:^DB::getRequestBytesCost to race: (the race_top: form only moved the report one frame down, into bytesSize), while this branch added the two race:oneshot:: entries for the Vortex segment-read handover. Both sides are kept — master's wording and race: form for getRequestBytesCost, plus the oneshot pair and its paragraph. Everything else auto-merged, and each of the five other shared files (src/CMakeLists.txt, SettingsChangesHistory.cpp, CHColumnToArrowColumn.cpp, BuzzHouse/Generator/SessionSettings.cpp, navigation.json) took master's change in a region this PR does not touch.

The 0.84 bump had never been through CI — it was pushed after the last report, and the pull request was CONFLICTING, so no workflow ran on it. Verified locally against the vendored 0.84 crates:

• cargo check -p _ch_rust_vortex — clean, zero warnings, so the ScanBuilder::with_projection / with_filter change to BoundExpression is correctly adapted.
• cargo test -p _ch_rust_vortex — 9/9 FFI tests pass, including ffi_roundtrip, ffi_read_coalescing and the cancel/error paths, so bind is exercised for real and not just type-checked.
• All 10 *_vortex_* stateless tests pass, including the two that were red on 4c209785: 04893_vortex_schema_cache and 04927_vortex_parallel_read (its new EXPLAIN PIPELINE pair reproduces exactly — File 0 → 1 with input_format_vortex_preserve_order, Resize 1 → 2 with ExpressionTransform × 2 without it).
• various_checks.sh and the new clickhouse_spelling check from master are clean on this branch's files. Style check is green in CI too.

One thing worth recording about the new bind: it cannot turn a valid query into an error, because makeVortexLiteral builds the literal in the file column's own Arrow type and vortex_ffi_expr_literal_int/_uint return null through try_from when the value does not fit — so an out-of-range or mistyped comparison drops out of the pushdown on our side before Vortex ever sees it, instead of failing to bind.

Vendoring. ClickHouse/rust_vendor#79 is merged, so contrib/rust_vendor now points at main (810b5603) instead of into a pull request branch. The merge commit's tree is byte-identical to the branch tip it replaces — git diff ab24d7175 810b5603c is empty — so no vendored crate changes; only the commit the submodule names does.

Remaining review threads are both agreed follow-ups, not blockers: extension dtypes for the currently unwritable types (Map, the wide integers, IPv6, Interval), and the bulk-copy path for out-of-line view arrays behind @m7kss1's q20 flamegraph.

[m7kss1]

@alexey-milovidov i believe that once we merge #116021 we’ll be on par with parquet on clickbench, though we still need to do some work on missing pushdowns and late materialization. soon i’ll write a comment here about all the follow ups that need to be done after we merge this to create separate issue for that

[m7kss1]

@alexey-milovidov i believe that once we merge #116021 we’ll be on par with parquet on clickbench, though we still need to do some work on missing pushdowns and late materialization. soon i’ll write a comment here about all the follow ups that need to be done after we merge this to create separate issue for that

after we merge this, let's also run clickbench on parquet vs vortex. from what i see on my vm, geomean on 3 hot runs is 1.1 (vortex is faster) and we win in 21/43 queries

[alexey-milovidov]

🕵 Merged master (2e987058) to clear the only red on 78d75c9a: Stress test (arm_release) aborted with Logical error: 'The size of reverse_flags (1) does not match the size of KeyDescription 3' in KeyDescription::getKeyFromAST via getPatchPartMetadataV2 during a lightweight UPDATE over a descending sorting key. Unrelated to this PR: the branch carried the pre-revert lightweight-updates code, and the re-landed #116007 fixes exactly this (buildKeyColumns now appends a reverse_flags entry for each additional system key column), so the merge pulls the fix in. The merge was conflict-free and correctly kept contrib/rust_vendor at 810b5603c (the vortex-0.84 merge, a descendant of master's pointer). All 9 Vortex stateless tests pass against the merged build; the AI review verdict on 78d75c9a was ✅ Approve and both open threads are agreed follow-ups.

[m7kss1]

@alexey-milovidov hey, is it possible for me to get commit rights to this branch so i can prepare this pr for merge? except for #116021, only small cosmetic things are left

[alexey-milovidov]

🕵 Merged master and fixed the Style check red on 2e987058: the settings_changes_history check now demands the current version block, so the input_format_vortex_filter_push_down / input_format_vortex_preserve_order records moved from the 26.8 to the 26.9 block of SettingsChangesHistory.cpp (master rolled the version over). The merge itself had one trivial conflict (a comment line in 03251_insert_sparse_all_formats.sh). Verified locally: incremental build clean, all 9 Vortex stateless tests pass. Pushed 291eeed6.

[alexey-milovidov]

@m7kss1, it's not possible to do it, as the branch is in ClickHouse/ClickHouse. But please send fixes in the other PR, and I will merge them ASAP.

[m7kss1]

@alexey-milovidov oh, i see. but what do you think about #116021? translating filter expressions directly from the ActionsDAG, rather than from KeyCondition is an important building block for adding more pushdown optimizations later

[alexey-milovidov]

🕵 Merged master (conflict-free; contrib/rust_vendor correctly stays at 810b5603c) and pushed aa8fba4f.

Build (arm_tidy) was the only red - everything else was DROPPED behind it. clang-tidy flagged cppcoreguidelines-init-variables on bool_value in VortexExpressionConverter::makeLiteral; initialized it.

AI review Major (Bool pushdown) does not reproduce - answered on the thread. Bool is DataTypeUInt8 with a custom name, so WhichDataType::isUInt8 is true for it; that is exactly why the arrow::Type::UINT8 case has to write which.isUInt8() && !isBool(cmp_type). Measured on a local build: a Bool predicate reports VortexFilterPushdownConjunctsPushed = 1 and drops whole splits.

AI review test request was valid and is addressed: 05053_vortex_pushdown_bool asserts the scan ProfileEvents for b = true, a bare b and NOT b over a real Bool column, which result equivalence alone cannot show.

All 12 Vortex stateless tests pass locally (aarch64 RelWithDebInfo).

Unrelated finding while verifying, worth a separate look: a local build dir configured with NO_ARMV81_OR_HIGHER=ON adds -C target_feature=+crc,-neon to RUSTFLAGS (cmake/cpu_features.cmake). rustc warns that the target ABI cannot be implemented correctly without neon, and it is not just a warning - f64 values are exchanged as raw bit patterns across crate boundaries, so alp's cut-point search returns an empty ALP-RD dictionary and writing any Float32/Float64 column aborts with panic: codes lookup table must be populated before RD encoding. It disappears with NO_ARMV81_OR_HIGHER=OFF, and CI does not set it, so it does not affect this PR - but that build configuration looks broken for Rust in general.

[clickhouse-gh]

VortexBlockInputFormat already relies on onChunk being callable before vortex_ffi_scan_create returns (src/Processors/Formats/Impl/Vortex/VortexBlockInputFormat.cpp:465), so the first callbacks do not have an FFI_VortexScan * yet and cannot actually call vortex_ffi_scan_cancel even though this contract says that is allowed from on_chunk.

Once the handle is published, the current C++ failure path calls vortex_ffi_scan_cancel before returning non-zero (src/Processors/Formats/Impl/Vortex/VortexBlockInputFormat.cpp:276), which suppresses on_finish even though the next lines say a non-zero return surfaces there. So the terminal callback sequence depends on timing.

We need either to publish the scan handle before any on_chunk can fire, or to narrow/specify this API so on_chunk does not promise cancellation and the cancel + non-zero path has defined semantics.

[clickhouse-gh]

The public contract now says input_format_vortex_filter_push_down supports Date, but the real vortex.date / Arrow DATE32 carrier still lands here and rejects a Date header outright.

That is observable on normal Vortex files, not just on synthetic schemas: CHColumnToArrowColumn writes ClickHouse Date as Arrow DATE32 (src/Processors/Formats/Impl/CHColumnToArrowColumn.cpp:137), so file(..., 'Vortex', 'd Date') silently loses pushdown on actual Vortex date columns. The only current coverage in 05045_vortex_pushdown_types.sh:151 is the special Date-over-raw-UInt16 case.

Either teach this DATE32 branch how to preserve the Date overflow semantics for Date headers, or narrow the docs/settings/tests back to Date32.