Skip to content

Commit 7840c3a

Browse files
committed
feat(stack-encrypt)!: order text, encrypting every order term through cllw-ore's vitaminc schemes
Text could not be ordered: no ordering operator was answered on a text field, and the ORE and OPE terms stored for text went through cllw-ore's own collation, which keeps case, so Zoe ordered before alice and é after z. ADR-0002 defines text's order key as its fold (Unicode 16 NFC with case and accents folded), and vitaminc 0.6 ships it in vitaminc-ore. cllw-ore 0.5.2-rc.1 implements vitaminc-ore's `OrderScheme` behind its `vitaminc` feature, so Stack Encrypt encodes nothing for ordering itself. An ORE or OPE term is now cllw-ore's `vitaminc::Ore` or `vitaminc::Ope` encryption of the value: the engine names the value's order target through the new sealed `sem::Orderable` (integers at their own width, `f32`/`f64` as `F64`, text as `Text`, bytes as `Bytes`) and passes the value straight through, on the typed path and for each scalar of a dynamic value alike; cllw-ore encodes it with vitaminc-ore, text by the fold. Café, cafe, CAFÉ and cafe with a combining accent are one term. Text holding a code point Unicode 16 does not assign has no fold and no order term, as it has no equality or match term; ADR-0002 is amended to say vitaminc 0.6 refuses such a code point rather than passing it through. The per-index key derivation is kept: the PRF block keys the scheme inside the visitor and is wiped there, and the column is opened with the value's target and the empty context, since the key is already the index's own. The key labels move to `/v2` because every order term changes bytes (new column salt, cllw-ore's marker byte for text and bytes). Every source type's term is now one ciphertext type (`OreCllw8VariableV1`, `OpeCllw8VariableV1`); `OreTerm<T>`/`OpeTerm<T>` keep `T` as a marker so typed callers compile unchanged, and `sem::CllwOreEncrypt`/`CllwOpeEncrypt` are no longer re-exported. The term pins and the record, query and value-kinds fixtures are re-recorded. op::Ordered covers String, and the data path answers the orderings on text too, refusing order text with no fold at its boundary, for one value and for each bound of a range, through `vitaminc_ore::text`, the encoding the `Text` target uses. The Go generator writes Lt, Lte, Gt, Gte and Between on a string field with ore or ope; its notice is left for order indexes on bytes or a float, and for a second order index. Closes #1234.
1 parent d1bd14a commit 7840c3a

58 files changed

Lines changed: 1464 additions & 812 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.lock‎

Lines changed: 20 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,10 @@ stack-profile = { path = "./packages/stack-profile", version = "0.43.0" }
8181
cts-common = { version = "0.43.0", default-features = false }
8282
zerokms-protocol = "0.12.31"
8383
recipher = "=0.3.1"
84-
cllw-ore = { version = "=0.5.0", default-features = false }
84+
# cllw-ore's `vitaminc` feature implements vitaminc-ore's `OrderScheme`,
85+
# which Stack Encrypt's order terms are encrypted through. It first ships
86+
# in a pre-release, and Cargo resolves a pre-release only when named.
87+
cllw-ore = { version = "=0.5.2-rc.1", default-features = false, features = ["vitaminc"] }
8588

8689
# External dependencies, with the suite's feature lists.
8790
base64 = "0.22.0"
@@ -130,5 +133,8 @@ vitaminc-hmac = "0.6.1"
130133
# Normalized equality terms (Unicode 16 NFC text, one integer encoding for
131134
# every width), which `vitaminc-prf`'s `normalize` feature derives over.
132135
vitaminc-normalize = "0.6.1"
136+
# The text order key: the system ADR-0002's fold, pinned to Unicode 16,
137+
# which text ORE and OPE terms are derived over (ADR-0009).
138+
vitaminc-ore = "0.6.1"
133139
vitaminc-prf = { version = "0.6.1", features = ["normalize"] }
134140
vitaminc-protected = "0.6.1"

‎docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md‎

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -203,9 +203,18 @@ transform and its version is named in the domain label, for example
203203
UTC calendar day.
204204
- **Text fold, v1**: NFD, then full Unicode case folding (`CaseFolding.txt`
205205
statuses C and F, so `ß` becomes `ss`), then NFD again, then removal of
206-
every code point with General_Category `Mn`, then NFC. The order of the
207-
steps is part of the transform, because folding can produce text that is
208-
no longer normalised and the steps do not commute.
206+
every accent, then NFC. An accent is a mark (General_Category `Mn`, `Mc`
207+
or `Me`) with a non-zero canonical combining class: the acute in `é`, the
208+
umlaut in `ö`, a Devanagari virama. A mark of class zero is a letter of
209+
its own in its script, such as a Thai or Devanagari vowel sign (U+0E31,
210+
U+0901), and is kept. The order of the steps is part of the transform,
211+
because folding can produce text that is no longer normalised and the
212+
steps do not commute.
213+
*Amended 2026-10-19:* this step said "every code point with
214+
General_Category `Mn`", which would also drop class-zero vowel signs and
215+
so tie different words. The fold that ships, `vitaminc-ore`'s
216+
`vitaminc/ore/value/text-folded/unicode-16/v1`, keeps them, and this
217+
bullet now describes it.
209218
- **Prefix truncation**: a number of code points kept after any other
210219
transform. Alphabet packing, if ever built, is a transform of the same
211220
kind.
@@ -219,6 +228,13 @@ write. The cost is that two strings using characters added after the pin,
219228
which a later Unicode would treat as equivalent, may not match. Moving the
220229
pin is a new label: existing columns keep the label they were written under
221230
and stay queryable, and a column moves only by re-encryption.
231+
*Amended 2026-10-18:* vitaminc 0.6 refuses a code point unassigned in the
232+
pinned version instead of passing it through. `vitaminc-normalize` (equality
233+
and match terms) and `vitaminc-ore`'s fold (order terms) both return
234+
`UnassignedCodePoint`, so text holding one has no term, and writing it to an
235+
indexed column or querying with it is refused. The refusal is the same on
236+
every path and in every language, so the transform is still a fixed
237+
function of the text it accepts, and moving the pin is still a new label.
222238

223239
Order terms fold case and accents because code-point order puts `é` after
224240
`z`; a fixed fold approximates the first level of Unicode collation without

‎languages/golang/CHANGELOG.md‎

Lines changed: 22 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,20 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
3535
implemented outside this module (a wrapper, or a fake in a generator's
3636
tests) changes its `Check` to return them; `enginetest.Static` shows the
3737
table.
38+
- **An `int8`, `uint8` or `byte` field takes no `ore` or `ope` index; drop
39+
the index and run stashgen again.** With 256 values, an order term ranks
40+
every row among 256, which a few known rows resolve to the values, the
41+
leak a `bool`'s order term is already refused for. The engine refuses the
42+
declaration as it refuses one on a `bool`, so stashgen stops at it. The
43+
field keeps `equality` and `Eq`; a value that must order declares a wider
44+
type.
45+
- **Every ORE and OPE term changes bytes; encrypt indexed rows again.**
46+
Order terms now encrypt through cllw-ore's vitaminc schemes, for every
47+
type: integers of every width, floats, text and bytes. A term v0.1.0-alpha.1
48+
stored does not compare with one this release derives, and nothing
49+
refuses it: `Lt`, `Between` or an `ORDER BY` over a column holding the
50+
alpha's terms silently misses or misplaces rows. Re-derive a column's
51+
order terms (encrypt its rows again) before querying it.
3852

3953
### Added
4054

@@ -47,8 +61,8 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
4761
`Decimal` are vitaminc's `vcvalue` types, with `encrypt.DateOf`,
4862
`ParseInt128`, `ParseUint128` and `ParseDecimal` (which refuses NaN and
4963
the infinities with `ErrNonFiniteDecimal`). Equality applies to all of
50-
them, ORE and OPE to every integer width. stashgen refuses an order index
51-
on a decimal, date or timestamp, naming #1236. The value-kinds fixture
64+
them, ORE and OPE to every integer width but 8 bits. stashgen refuses an
65+
order index on a decimal, date or timestamp, naming #1236. The value-kinds fixture
5266
test opens a record the Rust lowering sealed and derives the same terms.
5367
- **Any built-in Go integer field takes an integer EQL type** (ADR-0002):
5468
an `int16` or a `uint16` into `Integer`, an `int` into `Bigint`. The
@@ -74,9 +88,13 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
7488
type, range checked as for `Integer`) tagged `encrypt_into=` one of them
7589
seals at 16 bits; `SmallintEq` gets `Eq`, with the same equality term as
7690
an integer of the same value under the same column.
91+
- **Ordering methods on text.** A string field with `index=ore` or
92+
`index=ope` gets `Lt`, `Lte`, `Gt`, `Gte` and `Between`. Text orders by its
93+
folded form (ADR-0002: case and accents folded, under Unicode 16), so
94+
`Café` and `cafe` tie and `alice` orders before `Zoe`.
7795
- **stashgen notices an index no generated method queries**: an `ore` or
78-
`ope` index on text, bytes or a float (the engine answers the orderings
79-
over integers only), or the second of two order indexes on an integer.
96+
`ope` index on bytes or a float (the engine answers the orderings over
97+
integers and text only), or the second of two order indexes on a field.
8098
The index is still stored; the notice is on stderr and in the file.
8199

82100
### Changed

‎languages/golang/cmd/stashgen/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -124,7 +124,7 @@ For `-type User`, the file `user_stash.go` holds:
124124
- `Fields`, with one entry for each sealed field.
125125
An opaque struct, or a struct with no sealed field outside an opaque one, gets no `Fields`: nothing in it is sealed on its own, so nothing can be queried.
126126
An entry encrypts one value, and has one query method for each operator the engine answers on the field: `Eq`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` or `Matches`.
127-
stashgen asks the embedded engine which operators each field answers, and which index answers each one, so it holds no copy of those rules: an equality index answers `Eq`, a match index on a string `Matches`, and an ORE or OPE index on an integer the four orderings and `Between`, through ORE when the field declares both.
127+
stashgen asks the embedded engine which operators each field answers, and which index answers each one, so it holds no copy of those rules: an equality index answers `Eq`, a match index on a string `Matches`, and an ORE or OPE index on an integer or a string the four orderings and `Between`, through ORE when the field declares both. A string orders by its folded form: case and accents folded, under Unicode 16.
128128
`encrypt_into=TextEq` answers `Eq`, which returns the type's query value, `eql.TextEqQuery`.
129129
`encrypt_into=TextMatch` answers `Matches`, which returns an `encrypt.EQLMatchQuery`: the `eql.TextMatchQuery` operand, with `Verify` and `TargetLimit`.
130130
The query of `Matches` carries `Verify` and `TargetLimit`, because its answer can hold rows that do not match; an exact operator's query carries neither.

‎languages/golang/encrypt/README.md‎

Lines changed: 27 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -229,20 +229,21 @@ or a type defined over one (`type Email string`):
229229
230230
| Go field | Seals as | Indexes it can store | Query methods |
231231
|---|---|---|---|
232-
| `string` | text | `equality`, `match`, `ore`, `ope` | `Eq`, `Matches` |
232+
| `string` | text | `equality`, `match`, `ore`, `ope` | `Eq`, `Matches`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` |
233233
| `[]byte` | bytes | `equality`, `ore`, `ope` | `Eq` |
234234
| `bool` | a boolean | none | none |
235-
| `int8`, `int16`, `int32`, `int64`, `int` | an integer of the field's own width (`int` is 64-bit) | `equality`, `ore`, `ope` | `Eq`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` |
236-
| `uint8` (`byte`), `uint16`, `uint32`, `uint64`, `uint` | the same, unsigned | `equality`, `ore`, `ope` | the same |
235+
| `int16`, `int32`, `int64`, `int` | an integer of the field's own width (`int` is 64-bit) | `equality`, `ore`, `ope` | `Eq`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` |
236+
| `uint16`, `uint32`, `uint64`, `uint` | the same, unsigned | `equality`, `ore`, `ope` | the same |
237+
| `int8`, `uint8` (`byte`) | an 8-bit integer | `equality` | `Eq` |
237238
| `encrypt.Int128`, `encrypt.Uint128` | a 128-bit integer | `equality`, `ore`, `ope` | the same |
238239
| `float32`, `float64` | a float of its width | `equality`, `ore`, `ope` | `Eq` |
239240
| `encrypt.Decimal` | a decimal: a 96-bit coefficient and a scale up to 28 | `equality` | `Eq` |
240241
| `encrypt.Date` | a calendar date | `equality` | `Eq` |
241242
| `time.Time` | a timestamp, to the nanosecond, in UTC | `equality` | `Eq` |
242243
243-
Each query method needs its index declared. An `ore` or `ope` index on text,
244-
bytes or a float stores its term, but the engine answers no ordering through
245-
it yet, so such a field has no `Lt` (see [Queries](#queries)).
244+
Each query method needs its index declared. An `ore` or `ope` index on bytes
245+
or a float stores its term, but the engine answers no ordering through it
246+
yet, so such a field has no `Lt` (see [Queries](#queries)).
246247
247248
`encrypt.Date`, `Int128`, `Uint128` and `Decimal` are vitaminc's `vcvalue`
248249
types; build them with `encrypt.DateOf`, `ParseInt128`, `ParseUint128` and
@@ -253,8 +254,11 @@ sealed with, but a zero has no sign: `-0.00` opens as `0.00`. The zero
253254
`ErrEncoding`, naming the field; so is a decimal or a `time.Time` outside
254255
vitaminc's range. A sealed field cannot be a pointer, so an optional date has
255256
no sealed form yet: keep it in an opaque struct, or use a sentinel date. Each integer type seals at its own
256-
width, and each width is passed to the order scheme as itself, so an `int8`
257-
field's terms are an `int8`'s. Equality follows vitaminc's normalisation: an
257+
width, and each width is passed to the order scheme as itself, so an `int16`
258+
field's terms are an `int16`'s. An `int8` or `uint8` field has no order
259+
index: with 256 values, an order term would rank every row among 256, which
260+
a few known rows resolve to the values, as a `bool`'s would say which rows
261+
are true. Equality follows vitaminc's normalisation: an
258262
integer equals itself at every width, decimals compare at any scale
259263
(`1.50 = 1.5`), and a timestamp compares to the microsecond. An order index on
260264
a decimal, date or timestamp is refused for now: those order terms wait for
@@ -298,17 +302,27 @@ rules.
298302
| The field declares | Methods | Each returns |
299303
|---|---|---|
300304
| `index=equality` | `Eq` | `encrypt.EqualityTerm` |
301-
| `index=ore` or `index=ope`, on an integer of any width | `Lt`, `Lte`, `Gt`, `Gte` | `encrypt.OreTerm`, or `encrypt.OpeTerm` when the field declares no `ore` |
305+
| `index=ore` or `index=ope`, on an integer of 16 bits or wider or a string | `Lt`, `Lte`, `Gt`, `Gte` | `encrypt.OreTerm`, or `encrypt.OpeTerm` when the field declares no `ore` |
302306
| the same | `Between(ctx, cipher, lo, hi)` | `encrypt.Range[encrypt.OreTerm]` or `encrypt.Range[encrypt.OpeTerm]`, the two bounds' terms |
303307
| `index=match`, on a string | `Matches` | `encrypt.MatchQuery[T]` |
304308
| `encrypt_into=TextEq` | `Eq` | `eql.TextEqQuery` |
305309
| `encrypt_into=TextMatch` | `Matches` | `encrypt.EQLMatchQuery[eql.TextMatchQuery, T]` |
306310
307311
A method takes the field's own Go type, so `Fields.Age.Lt` on a `uint32`
308-
field takes a `uint32`. An order index on text, bytes or a float stores its
309-
term but answers no ordering operator yet, so such a field has no `Lt`; nor
310-
does the `ope` index of a field that declares `ore` too, since `ore` answers
311-
the orderings. stashgen prints a notice for each such index, on stderr and in
312+
field takes a `uint32`.
313+
314+
Text orders by its folded form (ADR-0002): case and accents are folded and
315+
`ß` becomes `ss`, under Unicode 16. So `Café`, `cafe` and `CAFÉ` are one
316+
value and tie, `alice` orders before `Zoe`, and `é` orders with `e`, not
317+
after `z`. `Between` is inclusive at both ends and, as SQL's `BETWEEN`,
318+
does not refuse a range whose low bound is above its high bound: such a
319+
range matches nothing. So `Between("b", "A")` matches nothing, since `A`
320+
folds to `a`, below `b`. Text holding a code point Unicode 16 does not assign has no
321+
fold, and its query or write is refused with `encrypt.ErrTerm`.
322+
323+
An order index on bytes or a float stores its term but answers no ordering
324+
operator, so such a field has no `Lt`; nor does the `ope` index of a field
325+
that declares `ore` too, since `ore` answers the orderings. stashgen prints a notice for each such index, on stderr and in
312326
the generated file: the terms are stored, and only your own code can compare
313327
them. A match index on an index-only field (`index=match` with no `encrypt`)
314328
is refused, because `Verify` checks a match query's rows by decrypting them.

‎languages/golang/encrypt/doc.go‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,10 @@
153153
//
154154
// Eq returns an [EqualityTerm] (or, for a field with encrypt_into, the EQL
155155
// type's query value), an ordering an [OreTerm] or [OpeTerm], and Between a
156-
// [Range] of two. Their answers are exact. Matches returns a [MatchQuery],
156+
// [Range] of two. Their answers are exact. The orderings are answered on
157+
// integers and on text, which orders by its folded form (case and accents
158+
// folded, under Unicode 16), so Café and cafe tie and alice orders before
159+
// Zoe. Matches returns a [MatchQuery],
157160
// whose answer can hold rows that do not match: size the fetch with
158161
// [MatchQuery.TargetLimit], then decrypt the rows and keep those
159162
// [MatchQuery.Verify] reports true, one call into the engine for the page.

‎languages/golang/encrypt/guest/Cargo.lock‎

Lines changed: 20 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎languages/golang/encrypt/guest/tests/native_ops.rs‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -330,15 +330,16 @@ fn guest_terms_match_the_native_sem_derivations() {
330330
.expect("native");
331331
assert_eq!(eq_bytes, native_bytes.as_bytes());
332332

333-
// Variable-width CLLW output for strings.
333+
// Variable-width CLLW output for strings: the text behind cllw-ore's
334+
// one marker byte, eight bytes a plaintext byte.
334335
let ore_s = block_on(ops::term(
335336
&cipher.default_keyset(),
336337
&encode(s("alice")),
337338
&encode(s("users/name")),
338339
TERM_ORE,
339340
))
340341
.expect("string ore");
341-
assert_eq!(ore_s.len(), 5 * 8);
342+
assert_eq!(ore_s.len(), 6 * 8);
342343

343344
// No ZeroKMS traffic for any of it.
344345
assert_eq!(cipher.kms().generate_calls.load(Ordering::SeqCst), 0);

‎languages/golang/encrypt/internal/testusers/kinds.go‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ type Kinds struct {
4545
S string `stash:"s,encrypt,index=equality;match"`
4646
E Email `stash:"e,encrypt,index=equality;match;ore"`
4747
Bo bool `stash:"bo,encrypt"`
48-
I8 int8 `stash:"i8,encrypt,index=equality;ore"`
48+
I8 int8 `stash:"i8,encrypt,index=equality"`
4949
I16 int16 `stash:"i16,encrypt,index=ope"`
5050
I32 int32 `stash:"i32,encrypt,index=equality"`
5151
I int `stash:"i,encrypt,index=ore"`
@@ -79,13 +79,14 @@ type Kinds struct {
7979
// context, identities and indexes of the plan in
8080
// packages/stack-encrypt/tests/fixtures/value_kinds.json, so the generated
8181
// code opens the record Rust sealed there and derives the same terms. Every
82-
// integer seals at its own width; the order indexes of a date, timestamp or
83-
// decimal wait for #1236, so those three carry equality alone.
82+
// integer seals at its own width; an 8-bit integer has no order term, and
83+
// the order indexes of a date, timestamp or decimal wait for #1236, so
84+
// those five carry equality alone.
8485
type ValueKinds struct {
8586
_ struct{} `stash:"context=value_kinds"`
86-
I8 int8 `stash:"i8,encrypt,index=equality;ore"`
87+
I8 int8 `stash:"i8,encrypt,index=equality"`
8788
I16 int16 `stash:"i16,encrypt,index=equality;ore"`
88-
U8 uint8 `stash:"u8,encrypt,index=equality;ore"`
89+
U8 uint8 `stash:"u8,encrypt,index=equality"`
8990
U16 uint16 `stash:"u16,encrypt,index=equality;ope"`
9091
I128 encrypt.Int128 `stash:"i128,encrypt,index=equality;ore"`
9192
U128 encrypt.Uint128 `stash:"u128,encrypt,index=equality;ope"`

0 commit comments

Comments
 (0)