Skip to content

Commit b98bad2

Browse files
committed
feat(stack-encrypt)!: order text, encrypting every order term through cllw-ore's vitaminc schemes
Text could not be ordered: no ordering operator was answered on a text field, and the ORE and OPE terms stored for text went through cllw-ore's own collation, which keeps case, so Zoe ordered before alice and é after z. ADR-0002 defines text's order key as its fold (Unicode 16 NFC with case and accents folded), and vitaminc 0.6 ships it in vitaminc-ore. cllw-ore 0.5.2-rc.1 implements vitaminc-ore's `OrderScheme` behind its `vitaminc` feature, so Stack Encrypt encodes nothing for ordering itself. An ORE or OPE term is now cllw-ore's `vitaminc::Ore` or `vitaminc::Ope` encryption of the value: the engine names the value's order target through the new sealed `sem::Orderable` (integers at their own width, `f32`/`f64` as `F64`, text as `Text`, bytes as `Bytes`) and passes the value straight through, on the typed path and for each scalar of a dynamic value alike; cllw-ore encodes it with vitaminc-ore, text by the fold. Café, cafe, CAFÉ and cafe with a combining accent are one term. Text holding a code point Unicode 16 does not assign has no fold and no order term, as it has no equality or match term; ADR-0002 is amended to say vitaminc 0.6 refuses such a code point rather than passing it through. The per-index key derivation is kept: the PRF block keys the scheme inside the visitor and is wiped there, and the column is opened with the value's target and the empty context, since the key is already the index's own. The key labels move to `/v2` because every order term changes bytes (new column salt, cllw-ore's marker byte for text and bytes). Every source type's term is now one ciphertext type (`OreCllw8VariableV1`, `OpeCllw8VariableV1`); `OreTerm<T>`/`OpeTerm<T>` keep `T` as a marker so typed callers compile unchanged, and `sem::CllwOreEncrypt`/`CllwOpeEncrypt` are no longer re-exported. The term pins and the record, query and value-kinds fixtures are re-recorded. op::Ordered covers String, and the data path answers the orderings on text too, refusing order text with no fold at its boundary, for one value and for each bound of a range, through `vitaminc_ore::text`, the encoding the `Text` target uses. The Go generator writes Lt, Lte, Gt, Gte and Between on a string field with ore or ope; its notice is left for order indexes on bytes or a float, and for a second order index. Closes #1234.
1 parent b178803 commit b98bad2

47 files changed

Lines changed: 1215 additions & 483 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.lock‎

Lines changed: 20 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,10 @@ stack-profile = { path = "./packages/stack-profile", version = "0.43.0" }
8181
cts-common = { version = "0.43.0", default-features = false }
8282
zerokms-protocol = "0.12.31"
8383
recipher = "=0.3.1"
84-
cllw-ore = { version = "=0.5.0", default-features = false }
84+
# cllw-ore's `vitaminc` feature implements vitaminc-ore's `OrderScheme`,
85+
# which Stack Encrypt's order terms are encrypted through. It first ships
86+
# in a pre-release, and Cargo resolves a pre-release only when named.
87+
cllw-ore = { version = "=0.5.2-rc.1", default-features = false, features = ["vitaminc"] }
8588

8689
# External dependencies, with the suite's feature lists.
8790
base64 = "0.22.0"
@@ -130,5 +133,8 @@ vitaminc-hmac = "0.6.1"
130133
# Normalized equality terms (Unicode 16 NFC text, one integer encoding for
131134
# every width), which `vitaminc-prf`'s `normalize` feature derives over.
132135
vitaminc-normalize = "0.6.1"
136+
# The text order key: the system ADR-0002's fold, pinned to Unicode 16,
137+
# which text ORE and OPE terms are derived over (ADR-0009).
138+
vitaminc-ore = "0.6.1"
133139
vitaminc-prf = { version = "0.6.1", features = ["normalize"] }
134140
vitaminc-protected = "0.6.1"

‎docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md‎

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -203,9 +203,18 @@ transform and its version is named in the domain label, for example
203203
UTC calendar day.
204204
- **Text fold, v1**: NFD, then full Unicode case folding (`CaseFolding.txt`
205205
statuses C and F, so `ß` becomes `ss`), then NFD again, then removal of
206-
every code point with General_Category `Mn`, then NFC. The order of the
207-
steps is part of the transform, because folding can produce text that is
208-
no longer normalised and the steps do not commute.
206+
every accent, then NFC. An accent is a mark (General_Category `Mn`, `Mc`
207+
or `Me`) with a non-zero canonical combining class: the acute in `é`, the
208+
umlaut in `ö`, a Devanagari virama. A mark of class zero is a letter of
209+
its own in its script, such as a Thai or Devanagari vowel sign (U+0E31,
210+
U+0901), and is kept. The order of the steps is part of the transform,
211+
because folding can produce text that is no longer normalised and the
212+
steps do not commute.
213+
*Amended 2026-10-19:* this step said "every code point with
214+
General_Category `Mn`", which would also drop class-zero vowel signs and
215+
so tie different words. The fold that ships, `vitaminc-ore`'s
216+
`vitaminc/ore/value/text-folded/unicode-16/v1`, keeps them, and this
217+
bullet now describes it.
209218
- **Prefix truncation**: a number of code points kept after any other
210219
transform. Alphabet packing, if ever built, is a transform of the same
211220
kind.
@@ -219,6 +228,13 @@ write. The cost is that two strings using characters added after the pin,
219228
which a later Unicode would treat as equivalent, may not match. Moving the
220229
pin is a new label: existing columns keep the label they were written under
221230
and stay queryable, and a column moves only by re-encryption.
231+
*Amended 2026-10-18:* vitaminc 0.6 refuses a code point unassigned in the
232+
pinned version instead of passing it through. `vitaminc-normalize` (equality
233+
and match terms) and `vitaminc-ore`'s fold (order terms) both return
234+
`UnassignedCodePoint`, so text holding one has no term, and writing it to an
235+
indexed column or querying with it is refused. The refusal is the same on
236+
every path and in every language, so the transform is still a fixed
237+
function of the text it accepts, and moving the pin is still a new label.
222238

223239
Order terms fold case and accents because code-point order puts `é` after
224240
`z`; a fixed fold approximates the first level of Unicode collation without

‎languages/golang/CHANGELOG.md‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,13 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
3535
implemented outside this module (a wrapper, or a fake in a generator's
3636
tests) changes its `Check` to return them; `enginetest.Static` shows the
3737
table.
38+
- **Every ORE and OPE term changes bytes; encrypt indexed rows again.**
39+
Order terms now encrypt through cllw-ore's vitaminc schemes, for every
40+
type: integers of every width, floats, text and bytes. A term v0.1.0-alpha.1
41+
stored does not compare with one this release derives, and nothing
42+
refuses it: `Lt`, `Between` or an `ORDER BY` over a column holding the
43+
alpha's terms silently misses or misplaces rows. Re-derive a column's
44+
order terms (encrypt its rows again) before querying it.
3845

3946
### Added
4047

@@ -74,9 +81,13 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
7481
type, range checked as for `Integer`) tagged `encrypt_into=` one of them
7582
seals at 16 bits; `SmallintEq` gets `Eq`, with the same equality term as
7683
an integer of the same value under the same column.
84+
- **Ordering methods on text.** A string field with `index=ore` or
85+
`index=ope` gets `Lt`, `Lte`, `Gt`, `Gte` and `Between`. Text orders by its
86+
folded form (ADR-0002: case and accents folded, under Unicode 16), so
87+
`Café` and `cafe` tie and `alice` orders before `Zoe`.
7788
- **stashgen notices an index no generated method queries**: an `ore` or
78-
`ope` index on text, bytes or a float (the engine answers the orderings
79-
over integers only), or the second of two order indexes on an integer.
89+
`ope` index on bytes or a float (the engine answers the orderings over
90+
integers and text only), or the second of two order indexes on a field.
8091
The index is still stored; the notice is on stderr and in the file.
8192

8293
### Changed

‎languages/golang/cmd/stashgen/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -124,7 +124,7 @@ For `-type User`, the file `user_stash.go` holds:
124124
- `Fields`, with one entry for each sealed field.
125125
An opaque struct, or a struct with no sealed field outside an opaque one, gets no `Fields`: nothing in it is sealed on its own, so nothing can be queried.
126126
An entry encrypts one value, and has one query method for each operator the engine answers on the field: `Eq`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` or `Matches`.
127-
stashgen asks the embedded engine which operators each field answers, and which index answers each one, so it holds no copy of those rules: an equality index answers `Eq`, a match index on a string `Matches`, and an ORE or OPE index on an integer the four orderings and `Between`, through ORE when the field declares both.
127+
stashgen asks the embedded engine which operators each field answers, and which index answers each one, so it holds no copy of those rules: an equality index answers `Eq`, a match index on a string `Matches`, and an ORE or OPE index on an integer or a string the four orderings and `Between`, through ORE when the field declares both. A string orders by its folded form: case and accents folded, under Unicode 16.
128128
`encrypt_into=TextEq` answers `Eq`, which returns the type's query value, `eql.TextEqQuery`.
129129
`encrypt_into=TextMatch` answers `Matches`, which returns an `encrypt.EQLMatchQuery`: the `eql.TextMatchQuery` operand, with `Verify` and `TargetLimit`.
130130
The query of `Matches` carries `Verify` and `TargetLimit`, because its answer can hold rows that do not match; an exact operator's query carries neither.

‎languages/golang/encrypt/README.md‎

Lines changed: 19 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -229,7 +229,7 @@ or a type defined over one (`type Email string`):
229229
230230
| Go field | Seals as | Indexes it can store | Query methods |
231231
|---|---|---|---|
232-
| `string` | text | `equality`, `match`, `ore`, `ope` | `Eq`, `Matches` |
232+
| `string` | text | `equality`, `match`, `ore`, `ope` | `Eq`, `Matches`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` |
233233
| `[]byte` | bytes | `equality`, `ore`, `ope` | `Eq` |
234234
| `bool` | a boolean | none | none |
235235
| `int8`, `int16`, `int32`, `int64`, `int` | an integer of the field's own width (`int` is 64-bit) | `equality`, `ore`, `ope` | `Eq`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` |
@@ -240,9 +240,9 @@ or a type defined over one (`type Email string`):
240240
| `encrypt.Date` | a calendar date | `equality` | `Eq` |
241241
| `time.Time` | a timestamp, to the nanosecond, in UTC | `equality` | `Eq` |
242242
243-
Each query method needs its index declared. An `ore` or `ope` index on text,
244-
bytes or a float stores its term, but the engine answers no ordering through
245-
it yet, so such a field has no `Lt` (see [Queries](#queries)).
243+
Each query method needs its index declared. An `ore` or `ope` index on bytes
244+
or a float stores its term, but the engine answers no ordering through it
245+
yet, so such a field has no `Lt` (see [Queries](#queries)).
246246
247247
`encrypt.Date`, `Int128`, `Uint128` and `Decimal` are vitaminc's `vcvalue`
248248
types; build them with `encrypt.DateOf`, `ParseInt128`, `ParseUint128` and
@@ -298,17 +298,27 @@ rules.
298298
| The field declares | Methods | Each returns |
299299
|---|---|---|
300300
| `index=equality` | `Eq` | `encrypt.EqualityTerm` |
301-
| `index=ore` or `index=ope`, on an integer of any width | `Lt`, `Lte`, `Gt`, `Gte` | `encrypt.OreTerm`, or `encrypt.OpeTerm` when the field declares no `ore` |
301+
| `index=ore` or `index=ope`, on an integer of any width or a string | `Lt`, `Lte`, `Gt`, `Gte` | `encrypt.OreTerm`, or `encrypt.OpeTerm` when the field declares no `ore` |
302302
| the same | `Between(ctx, cipher, lo, hi)` | `encrypt.Range[encrypt.OreTerm]` or `encrypt.Range[encrypt.OpeTerm]`, the two bounds' terms |
303303
| `index=match`, on a string | `Matches` | `encrypt.MatchQuery[T]` |
304304
| `encrypt_into=TextEq` | `Eq` | `eql.TextEqQuery` |
305305
| `encrypt_into=TextMatch` | `Matches` | `encrypt.EQLMatchQuery[eql.TextMatchQuery, T]` |
306306
307307
A method takes the field's own Go type, so `Fields.Age.Lt` on a `uint32`
308-
field takes a `uint32`. An order index on text, bytes or a float stores its
309-
term but answers no ordering operator yet, so such a field has no `Lt`; nor
310-
does the `ope` index of a field that declares `ore` too, since `ore` answers
311-
the orderings. stashgen prints a notice for each such index, on stderr and in
308+
field takes a `uint32`.
309+
310+
Text orders by its folded form (ADR-0002): case and accents are folded and
311+
`ß` becomes `ss`, under Unicode 16. So `Café`, `cafe` and `CAFÉ` are one
312+
value and tie, `alice` orders before `Zoe`, and `é` orders with `e`, not
313+
after `z`. `Between` is inclusive at both ends and, as SQL's `BETWEEN`,
314+
does not refuse a range whose low bound is above its high bound: such a
315+
range matches nothing. So `Between("b", "A")` matches nothing, since `A`
316+
folds to `a`, below `b`. Text holding a code point Unicode 16 does not assign has no
317+
fold, and its query or write is refused with `encrypt.ErrTerm`.
318+
319+
An order index on bytes or a float stores its term but answers no ordering
320+
operator, so such a field has no `Lt`; nor does the `ope` index of a field
321+
that declares `ore` too, since `ore` answers the orderings. stashgen prints a notice for each such index, on stderr and in
312322
the generated file: the terms are stored, and only your own code can compare
313323
them. A match index on an index-only field (`index=match` with no `encrypt`)
314324
is refused, because `Verify` checks a match query's rows by decrypting them.

‎languages/golang/encrypt/doc.go‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,10 @@
153153
//
154154
// Eq returns an [EqualityTerm] (or, for a field with encrypt_into, the EQL
155155
// type's query value), an ordering an [OreTerm] or [OpeTerm], and Between a
156-
// [Range] of two. Their answers are exact. Matches returns a [MatchQuery],
156+
// [Range] of two. Their answers are exact. The orderings are answered on
157+
// integers and on text, which orders by its folded form (case and accents
158+
// folded, under Unicode 16), so Café and cafe tie and alice orders before
159+
// Zoe. Matches returns a [MatchQuery],
157160
// whose answer can hold rows that do not match: size the fetch with
158161
// [MatchQuery.TargetLimit], then decrypt the rows and keep those
159162
// [MatchQuery.Verify] reports true, one call into the engine for the page.

‎languages/golang/encrypt/guest/Cargo.lock‎

Lines changed: 20 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎languages/golang/encrypt/guest/tests/native_ops.rs‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -330,15 +330,16 @@ fn guest_terms_match_the_native_sem_derivations() {
330330
.expect("native");
331331
assert_eq!(eq_bytes, native_bytes.as_bytes());
332332

333-
// Variable-width CLLW output for strings.
333+
// Variable-width CLLW output for strings: the text behind cllw-ore's
334+
// one marker byte, eight bytes a plaintext byte.
334335
let ore_s = block_on(ops::term(
335336
&cipher.default_keyset(),
336337
&encode(s("alice")),
337338
&encode(s("users/name")),
338339
TERM_ORE,
339340
))
340341
.expect("string ore");
341-
assert_eq!(ore_s.len(), 5 * 8);
342+
assert_eq!(ore_s.len(), 6 * 8);
342343

343344
// No ZeroKMS traffic for any of it.
344345
assert_eq!(cipher.kms().generate_calls.load(Ordering::SeqCst), 0);

‎languages/golang/encrypt/internal/testusers/kinds_stash.go‎

Lines changed: 27 additions & 7 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)