|
| 1 | +--- |
| 2 | +id: f5707f0f4f230a276a3428b1eb1e6469 |
| 3 | +author: Lemon Mint |
| 4 | +title: 'Schottky: Zero-Allocation, Order-Preserving Byte-Key Encoding for Go' |
| 5 | +description: Schottky is a high-performance Go library designed to encode multi-type composite values into order-preserving byte keys. Featuring zero-allocation execution, independent NULL ordering, PostgreSQL 18 type compatibility, and prefix-scan upper bounding, it is built for modern storage engines and index pipelines. |
| 6 | +language: en |
| 7 | +date: 2026-08-31T07:47:42.614875839Z |
| 8 | +path: /schottky |
| 9 | +go_package: gosuda.org/schottky |
| 10 | +go_repourl: https://github.com/gosuda/schottky.git |
| 11 | +--- |
| 12 | +When implementing LSM-tree or B-tree based key-value stores and database indexes, composite fields often need to be combined into a single byte key. |
| 13 | + |
| 14 | +Standard serialization formats like JSON or Protocol Buffers are not designed for this purpose because their serialized byte outputs do not preserve the natural sorting order required by unsigned bytewise comparisons (`bytes.Compare` or `memcmp`). |
| 15 | + |
| 16 | +Schottky is a Go library designed to encode multi-type composite tuples into order-preserving byte keys. |
| 17 | + |
| 18 | +```bash |
| 19 | +go get gosuda.org/schottky@latest |
| 20 | +``` |
| 21 | + |
| 22 | +### Serialization vs. Sort Keys |
| 23 | + |
| 24 | +Guaranteeing correct bytewise sorting requires addressing several low-level data representation details: |
| 25 | + |
| 26 | +- **Integers**: Standard two's complement big-endian encoding breaks natural ordering due to the most significant sign bit. Inverting the sign bit is necessary for correct unsigned byte comparisons. |
| 27 | +- **Floating-point numbers**: Requires sign bit adjustments, inverted ordering for negative values, and consistent handling of `NaN` and `-0`. |
| 28 | +- **Variable-length strings and byte slices**: Field boundaries must be preserved without breaking prefix sort orders. |
| 29 | +- **Composite key requirements**: Support for independent ASC/DESC ordering per field, decoupled NULLS FIRST/LAST rules, strict lexicographical precedence (earlier fields determine order), and compatibility with prefix scanning. |
| 30 | + |
| 31 | +Schottky converts each value into a canonical payload before applying presence tags and directional orientation. For DESC fields, each byte of the ASC payload is bitwise inverted (`^b`). NULL placement is handled via dedicated presence tags and operates independently of sort direction. |
| 32 | + |
| 33 | +### Basic Usage |
| 34 | + |
| 35 | +The following example builds a composite key consisting of an Account ID (`ASC, NULLS LAST`) and a Name (`DESC, NULLS FIRST`): |
| 36 | + |
| 37 | +```go |
| 38 | +package main |
| 39 | + |
| 40 | +import ( |
| 41 | + "fmt" |
| 42 | + |
| 43 | + "gosuda.org/schottky" |
| 44 | +) |
| 45 | + |
| 46 | +func main() { |
| 47 | + storage := make([]byte, 0, 128) |
| 48 | + builder := schottky.NewBuilder(storage) |
| 49 | + |
| 50 | + builder.Int64(42, schottky.AscNullsLast) |
| 51 | + accountPrefixLen := builder.Len() |
| 52 | + |
| 53 | + builder.String("Ada", schottky.DescNullsFirst) |
| 54 | + key, err := builder.Key() |
| 55 | + |
| 56 | + if err != nil { |
| 57 | + panic(err) |
| 58 | + } |
| 59 | + accountPrefix := key[:accountPrefixLen] |
| 60 | + fmt.Printf("key=%x\nprefix=%x\n", key, accountPrefix) |
| 61 | +} |
| 62 | +``` |
| 63 | + |
| 64 | +Schottky provides four explicit sort order configurations: |
| 65 | + |
| 66 | +- `AscNullsFirst` |
| 67 | +- `AscNullsLast` |
| 68 | +- `DescNullsFirst` |
| 69 | +- `DescNullsLast` |
| 70 | + |
| 71 | +NULL positioning is never implicitly inferred. If an invalid `Order` value is passed, the builder records `ErrInvalidOrder`, which is returned when calling `Key()` or `Err()`. |
| 72 | + |
| 73 | +### Prefix Scanning and Range Bounds |
| 74 | + |
| 75 | +Schottky composite keys contain no global headers, field count metadata, type tags, or trailers. Assuming the schema is known ahead of time, field encodings are simply concatenated. |
| 76 | + |
| 77 | +Because of this layout, the encoded bytes of leading fields form a valid prefix for range scans. In the example above, `accountPrefix` can be directly used as a prefix filter to scan all records where `Account ID == 42`. |
| 78 | + |
| 79 | +To compute the exclusive upper bound for half-open `[prefix, upper)` range scans, use `PrefixUpperBound`: |
| 80 | + |
| 81 | +```go |
| 82 | +upperStorage := make([]byte, 0, len(accountPrefix)) |
| 83 | +upper, finite, err := schottky.PrefixUpperBound(upperStorage, accountPrefix) |
| 84 | + |
| 85 | +if err != nil { |
| 86 | + panic(err) |
| 87 | +} |
| 88 | + |
| 89 | +if finite { |
| 90 | + // Half-open [accountPrefix, upper) range scan |
| 91 | +} else { |
| 92 | + // Unbounded open range scan |
| 93 | +} |
| 94 | +``` |
| 95 | + |
| 96 | +*Note: `Builder.Len()` must be measured at clean field boundaries. Slicing inside a field's internal byte stream produces an invalid prefix.* |
| 97 | + |
| 98 | +### Zero-Allocation and Buffer Management |
| 99 | + |
| 100 | +Key generation frequently runs on critical database paths. To eliminate heap allocations and buffer resizing overhead, `Builder` works strictly within the capacity of the caller-provided slice and will not reallocate internally. |
| 101 | + |
| 102 | +If the buffer runs out of capacity, `ErrShortBuffer` is recorded without writing partial bytes. Field writes are atomic, and the first error encountered is preserved until checked via `Key()` or `Err()`. Providing sufficient capacity upfront ensures zero-allocation encoding. |
| 103 | + |
| 104 | +Buffer sizes can be calculated in advance using helper functions such as `EncodedBytesSize`, `EncodedStringSize`, and `EncodedDecimalSize`, or via fixed-size constants. The returned key references the provided buffer directly, leaving memory lifecycle management to the caller. |
| 105 | + |
| 106 | +The `Decoder` works symmetrically: it borrows directly from the input key, requires caller-provided destination buffers for variable-length fields, and provides `Remaining() == 0` to detect trailing bytes or schema mismatches. |
| 107 | + |
| 108 | +### Supported Data Types |
| 109 | + |
| 110 | +- **Integers**: Signed and Unsigned (8-bit to 64-bit), `Int128` |
| 111 | +- **Floating-Point & Numerics**: `Float32`, `Float64`, Decimal Text |
| 112 | +- **Basic Types**: Binary String, Byte Slice, Boolean, Enum Rank |
| 113 | +- **Date & Time**: Date, Time, Zoned Time, Timestamp, Duration, Calendar Interval |
| 114 | +- **Network & Identifiers**: UUID, MAC, IP, IP Prefix, Canonical Network Prefix, LSN |
| 115 | +- **Composite Structures**: Nested Tuples, Ranges, and raw structural encodings |
| 116 | +- **Collation**: Unicode Collation Keys and external canonical tokens |
| 117 | + |
| 118 | +SQL type mapping is aligned with PostgreSQL 18 B-tree sorting rules. Types dependent on database catalogs or internal engine state are handled by passing external canonical tokens. |
| 119 | + |
| 120 | +### String Collation |
| 121 | + |
| 122 | +`Builder.String` defaults to raw UTF-8 binary order. For locale-aware sorting, Schottky provides a concurrent-safe, immutable `Collator`: |
| 123 | + |
| 124 | +- **Deterministic Collation**: Encodes the collation key alongside raw UTF-8 bytes to provide a tie-breaker when collation weights are identical. |
| 125 | +- **Nondeterministic Collation**: Treats collation-equal strings as identical, omitting the raw byte tie-breaker. |
| 126 | + |
| 127 | +Unicode and profile versions should be tracked in the metadata schema. If collation providers or profile settings change, existing keys must be rebuilt. |
| 128 | + |
| 129 | +### Schema Management |
| 130 | + |
| 131 | +Because Schottky keys are raw, headerless byte sequences, the schema layer must track: |
| 132 | + |
| 133 | +1. Field sequence and data types. |
| 134 | +2. Sort directions (`ASC`/`DESC`) and NULL ordering (`NULLS FIRST`/`LAST`). |
| 135 | +3. String collation and normalization rules. |
| 136 | +4. Schottky and Collation profile versions. |
| 137 | + |
| 138 | +Comparing keys generated with different schemas or decoding against a mismatched schema breaks ordering guarantees. |
| 139 | + |
| 140 | +### Performance and Links |
| 141 | + |
| 142 | +On Go 1.27+, experimental portable SIMD acceleration can be enabled using `GOEXPERIMENT=simd`. Scalar and SIMD paths produce byte-identical keys. |
| 143 | + |
| 144 | +- **GitHub Repository**: https://github.com/gosuda/schottky |
| 145 | +- **Key Layout Specification**: https://github.com/gosuda/schottky/blob/main/docs/03-key-layout.md |
| 146 | +- **SQL Type Mapping Guide**: https://github.com/gosuda/schottky/blob/main/docs/17-sql-type-map.md |
| 147 | +- **Go API Reference**: https://github.com/gosuda/schottky/blob/main/docs/18-api.md |
| 148 | + |
0 commit comments