Skip to content

Commit 2dd0aa1

Browse files
authored
Merge pull request #260 from gosuda/blog/schottky-release
chore(blog): add schottky package && blog
2 parents b9a903c + 315ed20 commit 2dd0aa1

3 files changed

Lines changed: 151 additions & 2 deletions

File tree

root/blog/ai-native-development.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ title: AI 네이티브 개발 방법론
55
description: AI 시대, 개발자와 AI가 상호작용하며 함께 성장하는 'AI 네이티브 개발' 방법론을 소개하고, 기존 개발 방식의 한계를 분석합니다.
66
language: ko
77
date: 2026-02-01T01:18:36.450880849Z
8-
path: /blog/posts/ai-네이티브-개발-방법론-z86704c1a
8+
path: /blog/posts/ai-native-development-z86704c1a
99
---
1010

1111
## AI 와 개발자
@@ -35,4 +35,5 @@ AI 네이티브 개발은 AI 를 신뢰하거나 도구로 쓰는 단계를 넘
3535

3636
AI 네이티브 개발은 특정 개인의 숙련도나 요령에 의존하지 않는다. 구성원이 바뀌어도 AI 컨텍스트가 유지된다면 같은 문제에 대해 유사한 관점의 질문을 던질 수 있고, 과거의 선택과 그 이유를 기반으로 더 나은 결정을 이어갈 수 있다. 개발 역량은 개인에게 귀속되지 않고, 조직 전체에 누적되고 재현 가능해진다.
3737

38-
결국 AI 네이티브 개발이 지향하는 것은 특정한 아키텍쳐나 개발 방법론이 아닌, AI 와 인간의 판단과 학습이 지속적으로 강화되는 개발 프로세스 자체를 의미한다. 이 구조 안에서 개발자와 AI는 서로를 대체하지 않는다. 그 대신 같은 문제 공간을 공유하며, 함께 더 나은 판단과 더 견고한 구조를 만들어가는 관계로 진화한다.
38+
결국 AI 네이티브 개발이 지향하는 것은 특정한 아키텍쳐나 개발 방법론이 아닌, AI 와 인간의 판단과 학습이 지속적으로 강화되는 개발 프로세스 자체를 의미한다. 이 구조 안에서 개발자와 AI는 서로를 대체하지 않는다. 그 대신 같은 문제 공간을 공유하며, 함께 더 나은 판단과 더 견고한 구조를 만들어가는 관계로 진화한다.
39+

root/packages/schottky.md

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
---
2+
id: f5707f0f4f230a276a3428b1eb1e6469
3+
author: Lemon Mint
4+
title: 'Schottky: Zero-Allocation, Order-Preserving Byte-Key Encoding for Go'
5+
description: Schottky is a high-performance Go library designed to encode multi-type composite values into order-preserving byte keys. Featuring zero-allocation execution, independent NULL ordering, PostgreSQL 18 type compatibility, and prefix-scan upper bounding, it is built for modern storage engines and index pipelines.
6+
language: en
7+
date: 2026-08-31T07:47:42.614875839Z
8+
path: /schottky
9+
go_package: gosuda.org/schottky
10+
go_repourl: https://github.com/gosuda/schottky.git
11+
---
12+
When implementing LSM-tree or B-tree based key-value stores and database indexes, composite fields often need to be combined into a single byte key.
13+
14+
Standard serialization formats like JSON or Protocol Buffers are not designed for this purpose because their serialized byte outputs do not preserve the natural sorting order required by unsigned bytewise comparisons (`bytes.Compare` or `memcmp`).
15+
16+
Schottky is a Go library designed to encode multi-type composite tuples into order-preserving byte keys.
17+
18+
```bash
19+
go get gosuda.org/schottky@latest
20+
```
21+
22+
### Serialization vs. Sort Keys
23+
24+
Guaranteeing correct bytewise sorting requires addressing several low-level data representation details:
25+
26+
- **Integers**: Standard two's complement big-endian encoding breaks natural ordering due to the most significant sign bit. Inverting the sign bit is necessary for correct unsigned byte comparisons.
27+
- **Floating-point numbers**: Requires sign bit adjustments, inverted ordering for negative values, and consistent handling of `NaN` and `-0`.
28+
- **Variable-length strings and byte slices**: Field boundaries must be preserved without breaking prefix sort orders.
29+
- **Composite key requirements**: Support for independent ASC/DESC ordering per field, decoupled NULLS FIRST/LAST rules, strict lexicographical precedence (earlier fields determine order), and compatibility with prefix scanning.
30+
31+
Schottky converts each value into a canonical payload before applying presence tags and directional orientation. For DESC fields, each byte of the ASC payload is bitwise inverted (`^b`). NULL placement is handled via dedicated presence tags and operates independently of sort direction.
32+
33+
### Basic Usage
34+
35+
The following example builds a composite key consisting of an Account ID (`ASC, NULLS LAST`) and a Name (`DESC, NULLS FIRST`):
36+
37+
```go
38+
package main
39+
40+
import (
41+
"fmt"
42+
43+
"gosuda.org/schottky"
44+
)
45+
46+
func main() {
47+
storage := make([]byte, 0, 128)
48+
builder := schottky.NewBuilder(storage)
49+
50+
builder.Int64(42, schottky.AscNullsLast)
51+
accountPrefixLen := builder.Len()
52+
53+
builder.String("Ada", schottky.DescNullsFirst)
54+
key, err := builder.Key()
55+
56+
if err != nil {
57+
panic(err)
58+
}
59+
accountPrefix := key[:accountPrefixLen]
60+
fmt.Printf("key=%x\nprefix=%x\n", key, accountPrefix)
61+
}
62+
```
63+
64+
Schottky provides four explicit sort order configurations:
65+
66+
- `AscNullsFirst`
67+
- `AscNullsLast`
68+
- `DescNullsFirst`
69+
- `DescNullsLast`
70+
71+
NULL positioning is never implicitly inferred. If an invalid `Order` value is passed, the builder records `ErrInvalidOrder`, which is returned when calling `Key()` or `Err()`.
72+
73+
### Prefix Scanning and Range Bounds
74+
75+
Schottky composite keys contain no global headers, field count metadata, type tags, or trailers. Assuming the schema is known ahead of time, field encodings are simply concatenated.
76+
77+
Because of this layout, the encoded bytes of leading fields form a valid prefix for range scans. In the example above, `accountPrefix` can be directly used as a prefix filter to scan all records where `Account ID == 42`.
78+
79+
To compute the exclusive upper bound for half-open `[prefix, upper)` range scans, use `PrefixUpperBound`:
80+
81+
```go
82+
upperStorage := make([]byte, 0, len(accountPrefix))
83+
upper, finite, err := schottky.PrefixUpperBound(upperStorage, accountPrefix)
84+
85+
if err != nil {
86+
panic(err)
87+
}
88+
89+
if finite {
90+
// Half-open [accountPrefix, upper) range scan
91+
} else {
92+
// Unbounded open range scan
93+
}
94+
```
95+
96+
*Note: `Builder.Len()` must be measured at clean field boundaries. Slicing inside a field's internal byte stream produces an invalid prefix.*
97+
98+
### Zero-Allocation and Buffer Management
99+
100+
Key generation frequently runs on critical database paths. To eliminate heap allocations and buffer resizing overhead, `Builder` works strictly within the capacity of the caller-provided slice and will not reallocate internally.
101+
102+
If the buffer runs out of capacity, `ErrShortBuffer` is recorded without writing partial bytes. Field writes are atomic, and the first error encountered is preserved until checked via `Key()` or `Err()`. Providing sufficient capacity upfront ensures zero-allocation encoding.
103+
104+
Buffer sizes can be calculated in advance using helper functions such as `EncodedBytesSize`, `EncodedStringSize`, and `EncodedDecimalSize`, or via fixed-size constants. The returned key references the provided buffer directly, leaving memory lifecycle management to the caller.
105+
106+
The `Decoder` works symmetrically: it borrows directly from the input key, requires caller-provided destination buffers for variable-length fields, and provides `Remaining() == 0` to detect trailing bytes or schema mismatches.
107+
108+
### Supported Data Types
109+
110+
- **Integers**: Signed and Unsigned (8-bit to 64-bit), `Int128`
111+
- **Floating-Point & Numerics**: `Float32`, `Float64`, Decimal Text
112+
- **Basic Types**: Binary String, Byte Slice, Boolean, Enum Rank
113+
- **Date & Time**: Date, Time, Zoned Time, Timestamp, Duration, Calendar Interval
114+
- **Network & Identifiers**: UUID, MAC, IP, IP Prefix, Canonical Network Prefix, LSN
115+
- **Composite Structures**: Nested Tuples, Ranges, and raw structural encodings
116+
- **Collation**: Unicode Collation Keys and external canonical tokens
117+
118+
SQL type mapping is aligned with PostgreSQL 18 B-tree sorting rules. Types dependent on database catalogs or internal engine state are handled by passing external canonical tokens.
119+
120+
### String Collation
121+
122+
`Builder.String` defaults to raw UTF-8 binary order. For locale-aware sorting, Schottky provides a concurrent-safe, immutable `Collator`:
123+
124+
- **Deterministic Collation**: Encodes the collation key alongside raw UTF-8 bytes to provide a tie-breaker when collation weights are identical.
125+
- **Nondeterministic Collation**: Treats collation-equal strings as identical, omitting the raw byte tie-breaker.
126+
127+
Unicode and profile versions should be tracked in the metadata schema. If collation providers or profile settings change, existing keys must be rebuilt.
128+
129+
### Schema Management
130+
131+
Because Schottky keys are raw, headerless byte sequences, the schema layer must track:
132+
133+
1. Field sequence and data types.
134+
2. Sort directions (`ASC`/`DESC`) and NULL ordering (`NULLS FIRST`/`LAST`).
135+
3. String collation and normalization rules.
136+
4. Schottky and Collation profile versions.
137+
138+
Comparing keys generated with different schemas or decoding against a mismatched schema breaks ordering guarantees.
139+
140+
### Performance and Links
141+
142+
On Go 1.27+, experimental portable SIMD acceleration can be enabled using `GOEXPERIMENT=simd`. Scalar and SIMD paths produce byte-identical keys.
143+
144+
- **GitHub Repository**: https://github.com/gosuda/schottky
145+
- **Key Layout Specification**: https://github.com/gosuda/schottky/blob/main/docs/03-key-layout.md
146+
- **SQL Type Mapping Guide**: https://github.com/gosuda/schottky/blob/main/docs/17-sql-type-map.md
147+
- **Go API Reference**: https://github.com/gosuda/schottky/blob/main/docs/18-api.md
148+

zdata/data.json.zstd

80.7 KB
Binary file not shown.

0 commit comments

Comments
 (0)