Skip to content

Commit 27e6b40

Browse files
robert3005claude
andauthored
feat(array): builders skip child canonicalization and append nested values in bulk [builders-child-stack] (#9064)
Collapses the first four PRs of the builders-child-stack into one (#8964, #8966, and #9131 were closed and folded in here): With this change all builders lose set_validity method as it's no longer used in the codebase and there are always better ways of expressing the desired operation - **Append repeated and viewed lists in bulk, not a value at a time** — bulk-append paths for repeated and viewed lists, and sparse array canonicalisation no longer goes through scalars. - **Builders no longer canonicalize their children** (#8964). - **Accumulate nested builder validity without a null buffer** (#8966). - **Append constant runs and sparse patches without slicing or canonicalizing** (#9131) — constant structs, extensions, and fixed-size lists append directly; sparse patches and fills append without slicing the patch array; the fixed-size-list fill tile is built once, not once per gap. --------- Signed-off-by: Robert Kruszewski <github@robertk.io> Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent b748502 commit 27e6b40

20 files changed

Lines changed: 2053 additions & 412 deletions

File tree

encodings/sparse/src/canonical.rs

Lines changed: 260 additions & 54 deletions
Large diffs are not rendered by default.

vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/mod.rs

Lines changed: 17 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -338,6 +338,7 @@ mod tests {
338338

339339
use crate::ArrayRef;
340340
use crate::IntoArray;
341+
use crate::RecursiveCanonical;
341342
use crate::VortexSessionExecute;
342343
use crate::aggregate_fn::Accumulator;
343344
use crate::aggregate_fn::AggregateFnVTable;
@@ -376,15 +377,26 @@ mod tests {
376377
use crate::scalar::ScalarValue;
377378
use crate::validity::Validity;
378379

380+
/// The size the array occupies once rebuilt through the canonical builders, which is the
381+
/// layout [`UncompressedSizeInBytes`] is defined against: the builders normalize physical
382+
/// widths that the input array is free to choose differently, picking the smallest decimal
383+
/// value type for a precision and `u64` list-view offsets and sizes.
384+
///
385+
/// Builders no longer canonicalize their children, so the finished array is only canonical at
386+
/// the top level - recursively canonicalize it before measuring.
379387
fn materialized_uncompressed_size_in_bytes(array: &ArrayRef) -> u64 {
388+
let mut ctx = array_session().create_execution_ctx();
380389
let mut builder = builder_with_capacity(array.dtype(), array.len());
381390
array
382-
.append_to_builder(
383-
builder.as_mut(),
384-
&mut array_session().create_execution_ctx(),
385-
)
391+
.append_to_builder(builder.as_mut(), &mut ctx)
386392
.vortex_expect("appended");
387-
builder.finish().nbytes()
393+
builder
394+
.finish()
395+
.execute::<RecursiveCanonical>(&mut ctx)
396+
.vortex_expect("recursively canonicalized")
397+
.0
398+
.into_array()
399+
.nbytes()
388400
}
389401

390402
fn aggregate(array: &ArrayRef) -> VortexResult<u64> {

0 commit comments

Comments
 (0)