Skip to content

.CLAUDE.md

Latest commit

 

History

History
160 lines (129 loc) · 4.83 KB

.CLAUDE.md

File metadata and controls

160 lines (129 loc) · 4.83 KB
Raw

parquet-go Project Context

Overview

High-performance Go library for reading and writing Apache Parquet files. Originally developed by Twilio Segment, now community-maintained at github.com/parquet-go/parquet-go.

  • Go Version: 1.22+
  • Current Version: v0.26.0+ (pre-v1, breaking changes possible)
  • Codebase Size: ~71,700 lines across 210 Go files

Project Structure

parquet-go/
├── Root Package          # Core API: Reader/Writer/File/Schema
├── encoding/             # 7 encoding formats (plain, rle, delta, etc.)
├── compress/             # 6 compression codecs (snappy, gzip, zstd, etc.)
├── bloom/                # Split-block Bloom filter (SIMD-optimized)
├── sparse/               # Sparse array utilities with gather ops
├── hashprobe/            # Hash-based dictionary operations
├── format/               # Generated Thrift definitions (parquet spec)
└── internal/             # Memory mgmt, byte algorithms, unsafe casts

Key Files by Function

Purpose Files
Core Types parquet.go, file.go, schema.go, node.go, type.go
Reading reader.go, column.go, page.go
Writing writer.go, column_buffer.go, buffer.go
Logical Types type_boolean.go through type_variant.go (40+ files)
Column Buffers column_buffer_*.go (20+ files for each physical type)
Page Impl page_*.go (15+ files)
Operations convert.go, merge.go, sorting.go
Configuration config.go (900+ lines)

Core Types & Interfaces

Schema System

  • Schema - Immutable, thread-safe parquet schema
  • Node - Interface for schema nodes
  • Field - Named schema node interface
  • Column - Concrete column representation
  • Type - Logical type interface
  • Kind - Physical type enum (Boolean, Int32, Int64, Float, Double, ByteArray, FixedLenByteArray)

Reader/Writer System

  • GenericReader[T] - Type-safe reader (preferred)
  • GenericWriter[T] - Type-safe writer (preferred)
  • SortingWriter[T] - Writer with integrated sorting
  • GenericBuffer[T] - In-memory row group buffer (implements sort.Interface)

Row Group Abstractions

  • RowGroup - Interface for row group collections
  • ColumnChunk - Interface for column data
  • Page - Interface for page data with stats
  • Pages - Sequential page reader

Common Patterns

Writing Data

// One-shot
parquet.WriteFile[T](path, rows, options...)

// Streaming
writer := parquet.NewGenericWriter[T](w, options...)
writer.Write(rows)
writer.Close()

// With sorting
sortWriter := parquet.NewSortingWriter[T](w, rowCount, options...)

Reading Data

// One-shot
rows, _ := parquet.ReadFile[T](path)

// Streaming
reader := parquet.NewGenericReader[T](r, options...)
n, _ := reader.Read(rows)

// Low-level
file, _ := parquet.OpenFile(r, size)
for _, rg := range file.RowGroups() { ... }

Build Commands

make test     # Run tests with -race and coverage
make format   # Go fmt + modernize tool
make tools    # Install development tools

Testing Patterns

  • Unit tests: *_test.go throughout (100+ files)
  • Examples: example_test.go
  • Property tests: internal/quick package
  • Key test files:
    • parquet_test.go - Core functionality
    • writer_test.go - Writer specifics (91KB)
    • reader_test.go - Reader specifics
    • merge_test.go - Merge operations
    • convert_test.go - Schema conversion

Debugging

Set environment variable:

PARQUETGODEBUG=1

Important Implementation Details

Performance Optimizations

  • SIMD assembly for AMD64 (dictionary ops, page bounds, ordering)
  • Zero-copy via interface-based design
  • Memory pooling with BufferPool
  • Async reading with ReadModeAsync

Schema Tags

Struct field tags control parquet behavior:

type Record struct {
    ID   int64  `parquet:"id"`
    Name string `parquet:"name,optional"`
    Data []byte `parquet:"data,snappy"`
}

Encoding Options

  • plain, rle, delta (binary-packed, length byte array, byte array)
  • bytestreamsplit (float optimization)
  • dictionary encoding

Compression Codecs

snappy, gzip, brotli, zstd, lz4, uncompressed

Recent Development Focus

  • Bug fixes: panic in Group.GoType(), json.RawMessage handling, repetition levels
  • New features: Geometry/Geography types, VARIANT logical type
  • Performance: GenericWriter optimizations
  • Stability: SortingWriter improvements

Common Bug Areas

  • Schema conversion edge cases (convert.go)
  • Nested structure handling (repetition/definition levels)
  • Memory management in streaming writes
  • Page statistics and index handling
  • Dictionary encoding with nulls
  • Interface type handling in dynamic value mapping (value.go, row.go)

Dependencies (Runtime)

  • Compression: brotli, gzip, lz4, zstd libraries
  • Encoding: bitpack, jsonlite
  • Types: google/uuid, go-geom (geometry)
  • Serialization: protobuf