Repository navigation
feat(benchmarks): add dict_encoding_sweep harness - #26250
Draft
Rich-T-kid wants to merge 5 commits into
Draft
Rich-T-kid wants to merge 5 commits into
Rich-T-kid wants to merge 5 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #26250 +/- ##
==========================================
- Coverage 82.89% 82.78% -0.12%
==========================================
Files 1154 1156 +2
Lines 457620 458504 +884
Branches 457620 458504 +884
==========================================
+ Hits 379352 379569 +217
- Misses 55262 55910 +648
- Partials 23006 23025 +19 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
Author
|
will keep this PR in draft mode and wait for the next arrow-rs bump as apache/arrow-rs#11407 should close the gap for the sort operator, and #26186 & #25874 will remove the un-needed cast in the project operator. Good results overall, once the parquet scan speed up is in I expect dictionarys to be faster than utf8View for low to medium queries |
This was referenced Oct 13, 2026
Introduces `dict_encoding_sweep`, a focused benchmark driver that compares `Dictionary<Int32, Utf8>` against `Utf8View` for the same operator across a cardinality sweep. For each cell it generates two in-memory tables sharing identical values, runs the same SQL against each, and reports median / min / p25 / p75 runtimes to CSV so the dict vs view crossover is directly visible. The harness covers scan, filter, project, aggregation, sort, join, limit, repartition, coalesce_batches, coalesce_partitions, and spm groups. The README section documents operator groups, flags, and CSV schema so other contributors can reproduce the numbers without touching the harness. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Extend dict_encoding_sweep with scan-level Parquet cases that vary physical PLAIN/RLE_DICTIONARY encoding independently from Utf8View/dictionary Arrow output. Verify both source pages and decoded output types so reported cells cannot silently measure the wrong path.
Rich-T-kid
force-pushed
the
dict-encoding-bench
branch
from
October 17, 2026 01:26
2b497c8 to
fdf73af
Compare
Pin apache/arrow-rs#11462 at 1a272a83 so the benchmark bot can compare the released reader with the proposed key-bounds optimization. Update the FixedSizeList offset call required by that Arrow revision.
Contributor
Author
|
run benchmark parquet_rle_dictionary_scan baseline: |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to 6096dab diff Run configurationrun benchmark parquet_rle_dictionary_scan
baseline:
ref: "6096dab24154d1a954e86004d5ac5a9a067f26e1"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 72e7d68 (72e7d68) to 6096dab diff Run configurationrun benchmark parquet_rle_dictionary_scan
baseline:
ref: "6096dab24154d1a954e86004d5ac5a9a067f26e1"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"CPU Details (lscpu)Details
Resource Usageparquet_rle_dictionary_scan — base (merge-base)
parquet_rle_dictionary_scan — branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark parquet_rle_dictionary_scan baseline: |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to 6096dab diff Run configurationrun benchmark parquet_rle_dictionary_scan
baseline:
ref: "6096dab24154d1a954e86004d5ac5a9a067f26e1"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 72e7d68 (72e7d68) to 6096dab diff Run configurationrun benchmark parquet_rle_dictionary_scan
baseline:
ref: "6096dab24154d1a954e86004d5ac5a9a067f26e1"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"CPU Details (lscpu)Details
Resource Usageparquet_rle_dictionary_scan — base (merge-base)
parquet_rle_dictionary_scan — branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to fbf7737 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "fbf77371a0512765ebf81fdc60e4a6cdceb8101f"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to fbf7737 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "fbf77371a0512765ebf81fdc60e4a6cdceb8101f"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to fbf7737 diff Run configurationrun benchmark tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "fbf77371a0512765ebf81fdc60e4a6cdceb8101f"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 72e7d68 (72e7d68) to fbf7737 diff Run configurationrun benchmark tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "fbf77371a0512765ebf81fdc60e4a6cdceb8101f"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 72e7d68 (72e7d68) to fbf7737 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "fbf77371a0512765ebf81fdc60e4a6cdceb8101f"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 72e7d68 (72e7d68) to fbf7737 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "fbf77371a0512765ebf81fdc60e4a6cdceb8101f"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark parquet_rle_dictionary_scan baseline: |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to 6096dab diff Run configurationrun benchmark parquet_rle_dictionary_scan
baseline:
ref: "6096dab24154d1a954e86004d5ac5a9a067f26e1"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"Results will be posted here when complete File an issue against this benchmark runner |
Contributor
Author
|
run benchmark tpch tpcds clickbench_partitioned env: baseline: changed: |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to 72e7d68 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to 72e7d68 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to 72e7d68 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
Hi @Rich-T-kid, thanks for the request (#26250 (comment)). Usage: Any benchmark name is accepted: Per-side configuration ( env:
# shared env is inherited by BOTH the build and the run, so build
# flags go here. Builds default to no debuginfo for speed; opt back
# in for hung-job gdb dumps and cap jobs to stay within memory:
CARGO_PROFILE_RELEASE_DEBUG: "1"
CARGO_BUILD_JOBS: "1"
baseline:
ref: v45.0.0
env:
# per-side env only reaches the benchmark run, not the build
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1G
changed:
ref: v46.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2GPod size and CPU architecture (each key is optional and keeps the default when absent; requests and limits are set equal): resources:
cpu: "16"
memory: "128Gi"
arch: arm64 # arm64 or amd64Independent benchmark shards (optional; 1–8, default 1): run benchmark arrow_writer
shards: 4Each shard builds both revisions, measures its assigned cases, and posts its own report. Resources and the job deadline apply per shard. DataFusion requests specifying a shard count greater than 1 fail with an error. Omit shards or use shards: 1 for DataFusion. File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to 72e7d68 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to 72e7d68 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
None. This is benchmark and reproducibility tooling for choosing between
dictionary and
Utf8Viewstring representations.Why
Dictionary-versus-
Utf8Viewdecisions depend on both NDV and the physicalsource encoding. In particular, Parquet
RLE_DICTIONARYinput can preserve itsdictionary directly, but the released Arrow reader performs an additional
per-batch key-bounds scan. That made earlier DataFusion scan measurements show
Utf8Viewwinning cases that should favor dictionary output onceapache/arrow-rs#11462 lands.
This PR makes the comparison reproducible at the operator and Parquet scan
levels, and adds a focused Criterion target that the DataFusion benchmark bot
can run on consistent hardware.
What changed
dict_encoding_sweep, a deterministic benchmark driver comparingDictionary<Int32, Utf8>andUtf8Viewacross filters, aggregations, sorts,projections, joins, repartitioning, limits, and coalescing operators.
Utf8schema, their data-page encoding is verified, and explicit target schemas
request dictionary or
Utf8Viewoutput directly fromDataSourceExec.Physical plans containing a representation cast are rejected.
parquet_rle_dictionary_scanCriterion benchmark for the benchmarkbot. It scans 1,000,000 rows from verified
RLE_DICTIONARYpages at NDVs1, 10, 100, 1,000, 10,000, 100,000, and 500,000.
1a272a83469707276e7e393201da9ba75262a968. The benchmark target is committedseparately from the pin so the bot can compare the released reader against
the proposed optimization.
FixedSizeListArrayoffset call required by the pinned Arrowrevision.
Parquet physical-plan construction remains outside the timed region. Every
timed drain receives a fresh physical plan because Parquet scan plans contain
one-shot stream state.
Results
RLE_DICTIONARYParquet scan with Arrow PR #11462The benchmark-bot run
used a dedicated Neoverse-V2 host and compared:
6096dab24: Criterion target with released Arrow 60.0.0.72e7d6871: the same target pinned to Arrow PR integrate consumer tests, implement tpch query 18 to 22 #11462 at1a272a83.Each cell scans 1,000,000 rows of 16-byte strings from two 500,000-row Parquet
row groups.
dict/viewis changed-revision dictionary time divided bychanged-revision
Utf8Viewtime.Utf8ViewUtf8ViewThe benchmark bot's leading ratios compare the two Git revisions for the same
benchmark row; they do not compare Dictionary with
Utf8View. For example, theNDV 10 Dictionary row reports baseline
1.86versus changed1.00, meaning theArrow PR makes that Dictionary benchmark 1.86x faster. The NDV 10
Utf8Viewrow reports baseline
1.00versus changed1.13, meaning the changed revisionis 13% slower for that row. To compare the two output representations, compare
their absolute times within the same revision column.
The Arrow change reduces dictionary scan latency by approximately 46%, 43%,
39%, 34%, and 17% at NDVs 10, 100, 1,000, 10,000, and 100,000 respectively.
NDV 1 and 500,000 are effectively unchanged on the benchmark host.
On the benchmark host, dictionary output is 34-41% lower latency from NDV 10
through 100,000 and still 7% lower at NDV 500,000.
Utf8Viewwins only the NDV1 case.
Benchmark-bot rerun
A fresh run on the same Neoverse-V2 instance class
repeated the exact immutable-ref comparison.
Dict deltais the reduction inDictionary latency from released Arrow to Arrow PR #11462;
PR Dict vs Viewisthe Dictionary latency reduction relative to
Utf8Viewwithin the changedrevision.
Utf8ViewThe rerun reproduces the strong NDV 10-10,000 result: the PR cuts Dictionary
latency by 37-46%, and Dictionary is 33-41% lower latency than
Utf8View.The exact Arrow-PR improvement at NDV 100,000 and 500,000 varies more between
bot runs, but both bot runs agree that the changed Dictionary output is faster
than
Utf8Viewat those cardinalities. NDV 1 consistently favorsUtf8View.Local Apple M4 Max cross-check
The local comparison used the same two commits and full-LTO benchmark profile.
Each value is the geometric mean of two order-balanced, measurement-only
Criterion runs with 20 samples per cell. Negative deltas mean the Arrow PR is
faster than released Arrow.
Utf8View(Dict +87.5%)Utf8View(Dict +52.0%)Locally, the PR leaves
Utf8Viewessentially unchanged through NDV 100,000while cutting dictionary latency by 22-55% from NDV 10 through 100,000.
Dictionary therefore flips from losing every baseline cell to winning NDV
10-100,000 after the Arrow change. The 500,000-NDV local cell was noisier and
still favors
Utf8View; the benchmark host instead has Dictionary ahead by 7%,so the high-cardinality crossover is hardware-sensitive.
The tested NDVs 10 through 500,000 are not powers of two, so their RLE bit-width
bound is larger than the dictionary length and the new bounds-scan skip does not
fire. Their improvement primarily measures the PR's boolean-fold-to-min/max
validation rewrite. Power-of-two cardinalities such as 16, 128, and 1,024 are
the cases that directly exercise removal of the scan.
In-memory operators
The general sweep remains useful for downstream operator selection. The
existing measurements show that dictionaries are especially effective for
grouping, dictionary-aware projection functions, and low-cardinality filters;
the crossover varies by operator, so scan representation should not be chosen
from NDV alone when downstream work is known.
Test coverage
cargo test -p datafusion-benchmarks parquet_scan_matrix_uses_requested_encodings --libverifies physical
PLAINandRLE_DICTIONARYpages, both native outputrepresentations, row counts, and the absence of casts.
cargo check -p datafusion-benchmarks --bench parquet_rle_dictionary_scancargo clippy -p datafusion-benchmarks --bench parquet_rle_dictionary_scan -- -D warningscargo fmt --all -- --checkci/scripts/doc_prettier_check.shArrow PR pin.
The benchmark-bot run completed successfully on its dedicated 12-vCPU host.
Are there any user-facing changes?
No. This PR contains benchmark tooling plus a temporary Arrow revision pin for
the before/after measurement.