Skip to content

Commit e086fcf

Browse files
committed
feat(stack-encrypt)!: order text, encrypting every order term through cllw-ore's vitaminc schemes
Text could not be ordered: no ordering operator was answered on a text field, and the ORE and OPE terms stored for text went through cllw-ore's own collation, which keeps case, so Zoe ordered before alice and é after z. ADR-0002 defines text's order key as its fold (Unicode 16 NFC with case and accents folded), and vitaminc 0.6 ships it in vitaminc-ore. cllw-ore 0.5.2-rc.1 implements vitaminc-ore's `OrderScheme` behind its `vitaminc` feature, so Stack Encrypt encodes nothing for ordering itself. An ORE or OPE term is now cllw-ore's `vitaminc::Ore` or `vitaminc::Ope` encryption of the value: the engine names the value's order target through the new sealed `sem::Orderable` (integers at their own width, `f32`/`f64` as `F64`, text as `Text`, bytes as `Bytes`) and passes the value straight through, on the typed path and for each scalar of a dynamic value alike; cllw-ore encodes it with vitaminc-ore, text by the fold. Café, cafe, CAFÉ and cafe with a combining accent are one term. Text holding a code point Unicode 16 does not assign has no fold and no order term, as it has no equality or match term; ADR-0002 is amended to say vitaminc 0.6 refuses such a code point rather than passing it through. The per-index key derivation is kept: the PRF block keys the scheme inside the visitor and is wiped there, and the column is opened with the value's target and the empty context, since the key is already the index's own. The key labels move to `/v2` because every order term changes bytes (new column salt, cllw-ore's marker byte for text and bytes). Every source type's term is now one ciphertext type (`OreCllw8VariableV1`, `OpeCllw8VariableV1`); `OreTerm<T>`/`OpeTerm<T>` keep `T` as a marker so typed callers compile unchanged, and `sem::CllwOreEncrypt`/`CllwOpeEncrypt` are no longer re-exported. The term pins and the record, query and value-kinds fixtures are re-recorded. op::Ordered covers String, and the data path answers the orderings on text too, refusing order text with no fold at its boundary, for one value and for each bound of a range, through `vitaminc_ore::text`, the encoding the `Text` target uses. The Go generator writes Lt, Lte, Gt, Gte and Between on a string field with ore or ope; its notice is left for order indexes on bytes or a float, and for a second order index. Closes #1234.
1 parent 5024415 commit e086fcf

47 files changed

Lines changed: 1214 additions & 482 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.lock‎

Lines changed: 20 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,10 @@ stack-profile = { path = "./packages/stack-profile", version = "0.43.0" }
8181
cts-common = { version = "0.43.0", default-features = false }
8282
zerokms-protocol = "0.12.31"
8383
recipher = "=0.3.1"
84-
cllw-ore = { version = "=0.5.0", default-features = false }
84+
# cllw-ore's `vitaminc` feature implements vitaminc-ore's `OrderScheme`,
85+
# which Stack Encrypt's order terms are encrypted through. It first ships
86+
# in a pre-release, and Cargo resolves a pre-release only when named.
87+
cllw-ore = { version = "=0.5.2-rc.1", default-features = false, features = ["vitaminc"] }
8588

8689
# External dependencies, with the suite's feature lists.
8790
base64 = "0.22.0"
@@ -131,5 +134,8 @@ vitaminc-hmac = "0.6.0"
131134
# Normalized equality terms (Unicode 16 NFC text, one integer encoding for
132135
# every width), which `vitaminc-prf`'s `normalize` feature derives over.
133136
vitaminc-normalize = "0.6.0"
137+
# The text order key: the system ADR-0002's fold, pinned to Unicode 16,
138+
# which text ORE and OPE terms are derived over (ADR-0009).
139+
vitaminc-ore = "0.6.0"
134140
vitaminc-prf = { version = "0.6.0", features = ["normalize"] }
135141
vitaminc-protected = "0.6.0"

‎docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md‎

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -203,9 +203,18 @@ transform and its version is named in the domain label, for example
203203
UTC calendar day.
204204
- **Text fold, v1**: NFD, then full Unicode case folding (`CaseFolding.txt`
205205
statuses C and F, so `ß` becomes `ss`), then NFD again, then removal of
206-
every code point with General_Category `Mn`, then NFC. The order of the
207-
steps is part of the transform, because folding can produce text that is
208-
no longer normalised and the steps do not commute.
206+
every accent, then NFC. An accent is a mark (General_Category `Mn`, `Mc`
207+
or `Me`) with a non-zero canonical combining class: the acute in `é`, the
208+
umlaut in `ö`, a Devanagari virama. A mark of class zero is a letter of
209+
its own in its script, such as a Thai or Devanagari vowel sign (U+0E31,
210+
U+0901), and is kept. The order of the steps is part of the transform,
211+
because folding can produce text that is no longer normalised and the
212+
steps do not commute.
213+
*Amended 2026-10-19:* this step said "every code point with
214+
General_Category `Mn`", which would also drop class-zero vowel signs and
215+
so tie different words. The fold that ships, `vitaminc-ore`'s
216+
`vitaminc/ore/value/text-folded/unicode-16/v1`, keeps them, and this
217+
bullet now describes it.
209218
- **Prefix truncation**: a number of code points kept after any other
210219
transform. Alphabet packing, if ever built, is a transform of the same
211220
kind.
@@ -219,6 +228,13 @@ write. The cost is that two strings using characters added after the pin,
219228
which a later Unicode would treat as equivalent, may not match. Moving the
220229
pin is a new label: existing columns keep the label they were written under
221230
and stay queryable, and a column moves only by re-encryption.
231+
*Amended 2026-10-18:* vitaminc 0.6 refuses a code point unassigned in the
232+
pinned version instead of passing it through. `vitaminc-normalize` (equality
233+
and match terms) and `vitaminc-ore`'s fold (order terms) both return
234+
`UnassignedCodePoint`, so text holding one has no term, and writing it to an
235+
indexed column or querying with it is refused. The refusal is the same on
236+
every path and in every language, so the transform is still a fixed
237+
function of the text it accepts, and moving the pin is still a new label.
222238

223239
Order terms fold case and accents because code-point order puts `é` after
224240
`z`; a fixed fold approximates the first level of Unicode collation without

‎languages/golang/CHANGELOG.md‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,13 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
3535
implemented outside this module (a wrapper, or a fake in a generator's
3636
tests) changes its `Check` to return them; `enginetest.Static` shows the
3737
table.
38+
- **Every ORE and OPE term changes bytes; encrypt indexed rows again.**
39+
Order terms now encrypt through cllw-ore's vitaminc schemes, for every
40+
type: integers of every width, floats, text and bytes. A term v0.1.0-alpha.1
41+
stored does not compare with one this release derives, and nothing
42+
refuses it: `Lt`, `Between` or an `ORDER BY` over a column holding the
43+
alpha's terms silently misses or misplaces rows. Re-derive a column's
44+
order terms (encrypt its rows again) before querying it.
3845

3946
### Added
4047

@@ -74,9 +81,13 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
7481
type, range checked as for `Integer`) tagged `encrypt_into=` one of them
7582
seals at 16 bits; `SmallintEq` gets `Eq`, with the same equality term as
7683
an integer of the same value under the same column.
84+
- **Ordering methods on text.** A string field with `index=ore` or
85+
`index=ope` gets `Lt`, `Lte`, `Gt`, `Gte` and `Between`. Text orders by its
86+
folded form (ADR-0002: case and accents folded, under Unicode 16), so
87+
`Café` and `cafe` tie and `alice` orders before `Zoe`.
7788
- **stashgen notices an index no generated method queries**: an `ore` or
78-
`ope` index on text, bytes or a float (the engine answers the orderings
79-
over integers only), or the second of two order indexes on an integer.
89+
`ope` index on bytes or a float (the engine answers the orderings over
90+
integers and text only), or the second of two order indexes on a field.
8091
The index is still stored; the notice is on stderr and in the file.
8192

8293
### Changed

‎languages/golang/cmd/stashgen/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -124,7 +124,7 @@ For `-type User`, the file `user_stash.go` holds:
124124
- `Fields`, with one entry for each sealed field.
125125
An opaque struct, or a struct with no sealed field outside an opaque one, gets no `Fields`: nothing in it is sealed on its own, so nothing can be queried.
126126
An entry encrypts one value, and has one query method for each operator the engine answers on the field: `Eq`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` or `Matches`.
127-
stashgen asks the embedded engine which operators each field answers, and which index answers each one, so it holds no copy of those rules: an equality index answers `Eq`, a match index on a string `Matches`, and an ORE or OPE index on an integer the four orderings and `Between`, through ORE when the field declares both.
127+
stashgen asks the embedded engine which operators each field answers, and which index answers each one, so it holds no copy of those rules: an equality index answers `Eq`, a match index on a string `Matches`, and an ORE or OPE index on an integer or a string the four orderings and `Between`, through ORE when the field declares both. A string orders by its folded form: case and accents folded, under Unicode 16.
128128
`encrypt_into=TextEq` answers `Eq`, which returns the type's query value, `eql.TextEqQuery`.
129129
`encrypt_into=TextMatch` answers `Matches`, which returns an `encrypt.EQLMatchQuery`: the `eql.TextMatchQuery` operand, with `Verify` and `TargetLimit`.
130130
The query of `Matches` carries `Verify` and `TargetLimit`, because its answer can hold rows that do not match; an exact operator's query carries neither.

‎languages/golang/encrypt/README.md‎

Lines changed: 19 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -210,7 +210,7 @@ or a type defined over one (`type Email string`):
210210
211211
| Go field | Seals as | Indexes it can store | Query methods |
212212
|---|---|---|---|
213-
| `string` | text | `equality`, `match`, `ore`, `ope` | `Eq`, `Matches` |
213+
| `string` | text | `equality`, `match`, `ore`, `ope` | `Eq`, `Matches`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` |
214214
| `[]byte` | bytes | `equality`, `ore`, `ope` | `Eq` |
215215
| `bool` | a boolean | none | none |
216216
| `int8`, `int16`, `int32`, `int64`, `int` | an integer of the field's own width (`int` is 64-bit) | `equality`, `ore`, `ope` | `Eq`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` |
@@ -221,9 +221,9 @@ or a type defined over one (`type Email string`):
221221
| `encrypt.Date` | a calendar date | `equality` | `Eq` |
222222
| `time.Time` | a timestamp, to the nanosecond, in UTC | `equality` | `Eq` |
223223
224-
Each query method needs its index declared. An `ore` or `ope` index on text,
225-
bytes or a float stores its term, but the engine answers no ordering through
226-
it yet, so such a field has no `Lt` (see [Queries](#queries)).
224+
Each query method needs its index declared. An `ore` or `ope` index on bytes
225+
or a float stores its term, but the engine answers no ordering through it
226+
yet, so such a field has no `Lt` (see [Queries](#queries)).
227227
228228
`encrypt.Date`, `Int128`, `Uint128` and `Decimal` are vitaminc's `vcvalue`
229229
types; build them with `encrypt.DateOf`, `ParseInt128`, `ParseUint128` and
@@ -279,17 +279,27 @@ rules.
279279
| The field declares | Methods | Each returns |
280280
|---|---|---|
281281
| `index=equality` | `Eq` | `encrypt.EqualityTerm` |
282-
| `index=ore` or `index=ope`, on an integer of any width | `Lt`, `Lte`, `Gt`, `Gte` | `encrypt.OreTerm`, or `encrypt.OpeTerm` when the field declares no `ore` |
282+
| `index=ore` or `index=ope`, on an integer of any width or a string | `Lt`, `Lte`, `Gt`, `Gte` | `encrypt.OreTerm`, or `encrypt.OpeTerm` when the field declares no `ore` |
283283
| the same | `Between(ctx, cipher, lo, hi)` | `encrypt.Range[encrypt.OreTerm]` or `encrypt.Range[encrypt.OpeTerm]`, the two bounds' terms |
284284
| `index=match`, on a string | `Matches` | `encrypt.MatchQuery[T]` |
285285
| `encrypt_into=TextEq` | `Eq` | `eql.TextEqQuery` |
286286
| `encrypt_into=TextMatch` | `Matches` | `encrypt.EQLMatchQuery[eql.TextMatchQuery, T]` |
287287
288288
A method takes the field's own Go type, so `Fields.Age.Lt` on a `uint32`
289-
field takes a `uint32`. An order index on text, bytes or a float stores its
290-
term but answers no ordering operator yet, so such a field has no `Lt`; nor
291-
does the `ope` index of a field that declares `ore` too, since `ore` answers
292-
the orderings. stashgen prints a notice for each such index, on stderr and in
289+
field takes a `uint32`.
290+
291+
Text orders by its folded form (ADR-0002): case and accents are folded and
292+
`ß` becomes `ss`, under Unicode 16. So `Café`, `cafe` and `CAFÉ` are one
293+
value and tie, `alice` orders before `Zoe`, and `é` orders with `e`, not
294+
after `z`. `Between` is inclusive at both ends and, as SQL's `BETWEEN`,
295+
does not refuse a range whose low bound is above its high bound: such a
296+
range matches nothing. So `Between("b", "A")` matches nothing, since `A`
297+
folds to `a`, below `b`. Text holding a code point Unicode 16 does not assign has no
298+
fold, and its query or write is refused with `encrypt.ErrTerm`.
299+
300+
An order index on bytes or a float stores its term but answers no ordering
301+
operator, so such a field has no `Lt`; nor does the `ope` index of a field
302+
that declares `ore` too, since `ore` answers the orderings. stashgen prints a notice for each such index, on stderr and in
293303
the generated file: the terms are stored, and only your own code can compare
294304
them. A match index on an index-only field (`index=match` with no `encrypt`)
295305
is refused, because `Verify` checks a match query's rows by decrypting them.

‎languages/golang/encrypt/doc.go‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,10 @@
153153
//
154154
// Eq returns an [EqualityTerm] (or, for a field with encrypt_into, the EQL
155155
// type's query value), an ordering an [OreTerm] or [OpeTerm], and Between a
156-
// [Range] of two. Their answers are exact. Matches returns a [MatchQuery],
156+
// [Range] of two. Their answers are exact. The orderings are answered on
157+
// integers and on text, which orders by its folded form (case and accents
158+
// folded, under Unicode 16), so Café and cafe tie and alice orders before
159+
// Zoe. Matches returns a [MatchQuery],
157160
// whose answer can hold rows that do not match: size the fetch with
158161
// [MatchQuery.TargetLimit], then decrypt the rows and keep those
159162
// [MatchQuery.Verify] reports true, one call into the engine for the page.

‎languages/golang/encrypt/guest/Cargo.lock‎

Lines changed: 20 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎languages/golang/encrypt/guest/tests/native_ops.rs‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -327,15 +327,16 @@ fn guest_terms_match_the_native_sem_derivations() {
327327
.expect("native");
328328
assert_eq!(eq_bytes, native_bytes.as_bytes());
329329

330-
// Variable-width CLLW output for strings.
330+
// Variable-width CLLW output for strings: the text behind cllw-ore's
331+
// one marker byte, eight bytes a plaintext byte.
331332
let ore_s = block_on(ops::term(
332333
&cipher.default_keyset(),
333334
&encode(s("alice")),
334335
&encode(s("users/name")),
335336
TERM_ORE,
336337
))
337338
.expect("string ore");
338-
assert_eq!(ore_s.len(), 5 * 8);
339+
assert_eq!(ore_s.len(), 6 * 8);
339340

340341
// No ZeroKMS traffic for any of it.
341342
assert_eq!(cipher.kms().generate_calls.load(Ordering::SeqCst), 0);

‎languages/golang/encrypt/internal/testusers/kinds_stash.go‎

Lines changed: 27 additions & 7 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)