Skip to content

Commit 9c65e7e

Browse files
authored
Merge pull request #1235 from cipherstash/feat/stack-encrypt-text-ordering
feat(stack-encrypt): order text over ADR-0002's folded order key
2 parents 0800375 + 3004048 commit 9c65e7e

62 files changed

Lines changed: 1563 additions & 1013 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.lock‎

Lines changed: 20 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,10 @@ stack-profile = { path = "./packages/stack-profile", version = "0.43.0" }
8181
cts-common = { version = "0.43.0", default-features = false }
8282
zerokms-protocol = "0.12.31"
8383
recipher = "=0.3.1"
84-
cllw-ore = { version = "=0.5.0", default-features = false }
84+
# cllw-ore's `vitaminc` feature implements vitaminc-ore's `OrderScheme`,
85+
# which Stack Encrypt's order terms are encrypted through. It first ships
86+
# in a pre-release, and Cargo resolves a pre-release only when named.
87+
cllw-ore = { version = "=0.5.2-rc.1", default-features = false, features = ["vitaminc"] }
8588

8689
# External dependencies, with the suite's feature lists.
8790
base64 = "0.22.0"
@@ -130,5 +133,8 @@ vitaminc-hmac = "0.6.1"
130133
# Normalized equality terms (Unicode 16 NFC text, one integer encoding for
131134
# every width), which `vitaminc-prf`'s `normalize` feature derives over.
132135
vitaminc-normalize = "0.6.1"
136+
# The text order key: the system ADR-0002's fold, pinned to Unicode 16,
137+
# which text ORE and OPE terms are derived over (ADR-0009).
138+
vitaminc-ore = "0.6.1"
133139
vitaminc-prf = { version = "0.6.1", features = ["normalize"] }
134140
vitaminc-protected = "0.6.1"

‎docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md‎

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -203,9 +203,18 @@ transform and its version is named in the domain label, for example
203203
UTC calendar day.
204204
- **Text fold, v1**: NFD, then full Unicode case folding (`CaseFolding.txt`
205205
statuses C and F, so `ß` becomes `ss`), then NFD again, then removal of
206-
every code point with General_Category `Mn`, then NFC. The order of the
207-
steps is part of the transform, because folding can produce text that is
208-
no longer normalised and the steps do not commute.
206+
every accent, then NFC. An accent is a mark (General_Category `Mn`, `Mc`
207+
or `Me`) with a non-zero canonical combining class: the acute in `é`, the
208+
umlaut in `ö`, a Devanagari virama. A mark of class zero is a letter of
209+
its own in its script, such as a Thai or Devanagari vowel sign (U+0E31,
210+
U+0901), and is kept. The order of the steps is part of the transform,
211+
because folding can produce text that is no longer normalised and the
212+
steps do not commute.
213+
*Amended 2026-10-19:* this step said "every code point with
214+
General_Category `Mn`", which would also drop class-zero vowel signs and
215+
so tie different words. The fold that ships, `vitaminc-ore`'s
216+
`vitaminc/ore/value/text-folded/unicode-16/v1`, keeps them, and this
217+
bullet now describes it.
209218
- **Prefix truncation**: a number of code points kept after any other
210219
transform. Alphabet packing, if ever built, is a transform of the same
211220
kind.
@@ -219,6 +228,13 @@ write. The cost is that two strings using characters added after the pin,
219228
which a later Unicode would treat as equivalent, may not match. Moving the
220229
pin is a new label: existing columns keep the label they were written under
221230
and stay queryable, and a column moves only by re-encryption.
231+
*Amended 2026-10-18:* vitaminc 0.6 refuses a code point unassigned in the
232+
pinned version instead of passing it through. `vitaminc-normalize` (equality
233+
and match terms) and `vitaminc-ore`'s fold (order terms) both return
234+
`UnassignedCodePoint`, so text holding one has no term, and writing it to an
235+
indexed column or querying with it is refused. The refusal is the same on
236+
every path and in every language, so the transform is still a fixed
237+
function of the text it accepts, and moving the pin is still a new label.
222238

223239
Order terms fold case and accents because code-point order puts `é` after
224240
`z`; a fixed fold approximates the first level of Unicode collation without

‎languages/golang/CHANGELOG.md‎

Lines changed: 22 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,20 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
3535
implemented outside this module (a wrapper, or a fake in a generator's
3636
tests) changes its `Check` to return them; `enginetest.Static` shows the
3737
table.
38+
- **An `int8`, `uint8` or `byte` field takes no `ore` or `ope` index; drop
39+
the index and run stashgen again.** With 256 values, an order term ranks
40+
every row among 256, which a few known rows resolve to the values, the
41+
leak a `bool`'s order term is already refused for. The engine refuses the
42+
declaration as it refuses one on a `bool`, so stashgen stops at it. The
43+
field keeps `equality` and `Eq`; a value that must order declares a wider
44+
type.
45+
- **Every ORE and OPE term changes bytes; encrypt indexed rows again.**
46+
Order terms now encrypt through cllw-ore's vitaminc schemes, for every
47+
type: integers of every width, floats, text and bytes. A term v0.1.0-alpha.1
48+
stored does not compare with one this release derives, and nothing
49+
refuses it: `Lt`, `Between` or an `ORDER BY` over a column holding the
50+
alpha's terms silently misses or misplaces rows. Re-derive a column's
51+
order terms (encrypt its rows again) before querying it.
3852

3953
### Added
4054

@@ -47,8 +61,8 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
4761
`Decimal` are vitaminc's `vcvalue` types, with `encrypt.DateOf`,
4862
`ParseInt128`, `ParseUint128` and `ParseDecimal` (which refuses NaN and
4963
the infinities with `ErrNonFiniteDecimal`). Equality applies to all of
50-
them, ORE and OPE to every integer width. stashgen refuses an order index
51-
on a decimal, date or timestamp, naming #1236. The value-kinds fixture
64+
them, ORE and OPE to every integer width but 8 bits. stashgen refuses an
65+
order index on a decimal, date or timestamp, naming #1236. The value-kinds fixture
5266
test opens a record the Rust lowering sealed and derives the same terms.
5367
- **Any built-in Go integer field takes an integer EQL type** (ADR-0002):
5468
an `int16` or a `uint16` into `Integer`, an `int` into `Bigint`. The
@@ -74,9 +88,13 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
7488
type, range checked as for `Integer`) tagged `encrypt_into=` one of them
7589
seals at 16 bits; `SmallintEq` gets `Eq`, with the same equality term as
7690
an integer of the same value under the same column.
91+
- **Ordering methods on text.** A string field with `index=ore` or
92+
`index=ope` gets `Lt`, `Lte`, `Gt`, `Gte` and `Between`. Text orders by its
93+
folded form (ADR-0002: case and accents folded, under Unicode 16), so
94+
`Café` and `cafe` tie and `alice` orders before `Zoe`.
7795
- **stashgen notices an index no generated method queries**: an `ore` or
78-
`ope` index on text, bytes or a float (the engine answers the orderings
79-
over integers only), or the second of two order indexes on an integer.
96+
`ope` index on bytes or a float (the engine answers the orderings over
97+
integers and text only), or the second of two order indexes on a field.
8098
The index is still stored; the notice is on stderr and in the file.
8199

82100
### Changed

‎languages/golang/cmd/stashgen/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -124,7 +124,7 @@ For `-type User`, the file `user_stash.go` holds:
124124
- `Fields`, with one entry for each sealed field.
125125
An opaque struct, or a struct with no sealed field outside an opaque one, gets no `Fields`: nothing in it is sealed on its own, so nothing can be queried.
126126
An entry encrypts one value, and has one query method for each operator the engine answers on the field: `Eq`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` or `Matches`.
127-
stashgen asks the embedded engine which operators each field answers, and which index answers each one, so it holds no copy of those rules: an equality index answers `Eq`, a match index on a string `Matches`, and an ORE or OPE index on an integer the four orderings and `Between`, through ORE when the field declares both.
127+
stashgen asks the embedded engine which operators each field answers, and which index answers each one, so it holds no copy of those rules: an equality index answers `Eq`, a match index on a string `Matches`, and an ORE or OPE index on an integer or a string the four orderings and `Between`, through ORE when the field declares both. A string orders by its folded form: case and accents folded, under Unicode 16.
128128
`encrypt_into=TextEq` answers `Eq`, which returns the type's query value, `eql.TextEqQuery`.
129129
`encrypt_into=TextMatch` answers `Matches`, which returns an `encrypt.EQLMatchQuery`: the `eql.TextMatchQuery` operand, with `Verify` and `TargetLimit`.
130130
The query of `Matches` carries `Verify` and `TargetLimit`, because its answer can hold rows that do not match; an exact operator's query carries neither.

‎languages/golang/encrypt/README.md‎

Lines changed: 27 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -229,20 +229,21 @@ or a type defined over one (`type Email string`):
229229
230230
| Go field | Seals as | Indexes it can store | Query methods |
231231
|---|---|---|---|
232-
| `string` | text | `equality`, `match`, `ore`, `ope` | `Eq`, `Matches` |
232+
| `string` | text | `equality`, `match`, `ore`, `ope` | `Eq`, `Matches`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` |
233233
| `[]byte` | bytes | `equality`, `ore`, `ope` | `Eq` |
234234
| `bool` | a boolean | none | none |
235-
| `int8`, `int16`, `int32`, `int64`, `int` | an integer of the field's own width (`int` is 64-bit) | `equality`, `ore`, `ope` | `Eq`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` |
236-
| `uint8` (`byte`), `uint16`, `uint32`, `uint64`, `uint` | the same, unsigned | `equality`, `ore`, `ope` | the same |
235+
| `int16`, `int32`, `int64`, `int` | an integer of the field's own width (`int` is 64-bit) | `equality`, `ore`, `ope` | `Eq`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` |
236+
| `uint16`, `uint32`, `uint64`, `uint` | the same, unsigned | `equality`, `ore`, `ope` | the same |
237+
| `int8`, `uint8` (`byte`) | an 8-bit integer | `equality` | `Eq` |
237238
| `encrypt.Int128`, `encrypt.Uint128` | a 128-bit integer | `equality`, `ore`, `ope` | the same |
238239
| `float32`, `float64` | a float of its width | `equality`, `ore`, `ope` | `Eq` |
239240
| `encrypt.Decimal` | a decimal: a 96-bit coefficient and a scale up to 28 | `equality` | `Eq` |
240241
| `encrypt.Date` | a calendar date | `equality` | `Eq` |
241242
| `time.Time` | a timestamp, to the nanosecond, in UTC | `equality` | `Eq` |
242243
243-
Each query method needs its index declared. An `ore` or `ope` index on text,
244-
bytes or a float stores its term, but the engine answers no ordering through
245-
it yet, so such a field has no `Lt` (see [Queries](#queries)).
244+
Each query method needs its index declared. An `ore` or `ope` index on bytes
245+
or a float stores its term, but the engine answers no ordering through it
246+
yet, so such a field has no `Lt` (see [Queries](#queries)).
246247
247248
`encrypt.Date`, `Int128`, `Uint128` and `Decimal` are vitaminc's `vcvalue`
248249
types; build them with `encrypt.DateOf`, `ParseInt128`, `ParseUint128` and
@@ -253,8 +254,11 @@ sealed with, but a zero has no sign: `-0.00` opens as `0.00`. The zero
253254
`ErrEncoding`, naming the field; so is a decimal or a `time.Time` outside
254255
vitaminc's range. A sealed field cannot be a pointer, so an optional date has
255256
no sealed form yet: keep it in an opaque struct, or use a sentinel date. Each integer type seals at its own
256-
width, and each width is passed to the order scheme as itself, so an `int8`
257-
field's terms are an `int8`'s. Equality follows vitaminc's normalisation: an
257+
width, and each width is passed to the order scheme as itself, so an `int16`
258+
field's terms are an `int16`'s. An `int8` or `uint8` field has no order
259+
index: with 256 values, an order term would rank every row among 256, which
260+
a few known rows resolve to the values, as a `bool`'s would say which rows
261+
are true. Equality follows vitaminc's normalisation: an
258262
integer equals itself at every width, decimals compare at any scale
259263
(`1.50 = 1.5`), and a timestamp compares to the microsecond. An order index on
260264
a decimal, date or timestamp is refused for now: those order terms wait for
@@ -298,17 +302,27 @@ rules.
298302
| The field declares | Methods | Each returns |
299303
|---|---|---|
300304
| `index=equality` | `Eq` | `encrypt.EqualityTerm` |
301-
| `index=ore` or `index=ope`, on an integer of any width | `Lt`, `Lte`, `Gt`, `Gte` | `encrypt.OreTerm`, or `encrypt.OpeTerm` when the field declares no `ore` |
305+
| `index=ore` or `index=ope`, on an integer of 16 bits or wider or a string | `Lt`, `Lte`, `Gt`, `Gte` | `encrypt.OreTerm`, or `encrypt.OpeTerm` when the field declares no `ore` |
302306
| the same | `Between(ctx, cipher, lo, hi)` | `encrypt.Range[encrypt.OreTerm]` or `encrypt.Range[encrypt.OpeTerm]`, the two bounds' terms |
303307
| `index=match`, on a string | `Matches` | `encrypt.MatchQuery[T]` |
304308
| `encrypt_into=TextEq` | `Eq` | `eql.TextEqQuery` |
305309
| `encrypt_into=TextMatch` | `Matches` | `encrypt.EQLMatchQuery[eql.TextMatchQuery, T]` |
306310
307311
A method takes the field's own Go type, so `Fields.Age.Lt` on a `uint32`
308-
field takes a `uint32`. An order index on text, bytes or a float stores its
309-
term but answers no ordering operator yet, so such a field has no `Lt`; nor
310-
does the `ope` index of a field that declares `ore` too, since `ore` answers
311-
the orderings. stashgen prints a notice for each such index, on stderr and in
312+
field takes a `uint32`.
313+
314+
Text orders by its folded form (ADR-0002): case and accents are folded and
315+
`ß` becomes `ss`, under Unicode 16. So `Café`, `cafe` and `CAFÉ` are one
316+
value and tie, `alice` orders before `Zoe`, and `é` orders with `e`, not
317+
after `z`. `Between` is inclusive at both ends and, as SQL's `BETWEEN`,
318+
does not refuse a range whose low bound is above its high bound: such a
319+
range matches nothing. So `Between("b", "A")` matches nothing, since `A`
320+
folds to `a`, below `b`. Text holding a code point Unicode 16 does not assign has no
321+
fold, and its query or write is refused with `encrypt.ErrTerm`.
322+
323+
An order index on bytes or a float stores its term but answers no ordering
324+
operator, so such a field has no `Lt`; nor does the `ope` index of a field
325+
that declares `ore` too, since `ore` answers the orderings. stashgen prints a notice for each such index, on stderr and in
312326
the generated file: the terms are stored, and only your own code can compare
313327
them. A match index on an index-only field (`index=match` with no `encrypt`)
314328
is refused, because `Verify` checks a match query's rows by decrypting them.

‎languages/golang/encrypt/doc.go‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,10 @@
153153
//
154154
// Eq returns an [EqualityTerm] (or, for a field with encrypt_into, the EQL
155155
// type's query value), an ordering an [OreTerm] or [OpeTerm], and Between a
156-
// [Range] of two. Their answers are exact. Matches returns a [MatchQuery],
156+
// [Range] of two. Their answers are exact. The orderings are answered on
157+
// integers and on text, which orders by its folded form (case and accents
158+
// folded, under Unicode 16), so Café and cafe tie and alice orders before
159+
// Zoe. Matches returns a [MatchQuery],
157160
// whose answer can hold rows that do not match: size the fetch with
158161
// [MatchQuery.TargetLimit], then decrypt the rows and keep those
159162
// [MatchQuery.Verify] reports true, one call into the engine for the page.

‎languages/golang/encrypt/guest/Cargo.lock‎

Lines changed: 20 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎languages/golang/encrypt/guest/src/abi.rs‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -465,8 +465,9 @@ pub unsafe extern "C" fn se_plan_check(plan_ptr: *const u8, plan_len: u32) -> u6
465465
/// term's codec-encoded object out. An operator the field does not answer
466466
/// is `STATUS_ENCODING` naming the field, as a mistyped value is, before
467467
/// the cipher is consulted; the refusals the typed path makes before a term
468-
/// exists (a range the wrong way round, match text that cuts no token or
469-
/// does not normalize) are `STATUS_TERM`, also before the cipher. The input
468+
/// exists (match text that cuts no token or does not normalize) are
469+
/// `STATUS_TERM`, also before the cipher. A range the wrong way round is
470+
/// not refused: it matches no row, as SQL's `BETWEEN` does. The input
470471
/// buffer is wiped in place.
471472
///
472473
/// # Safety

‎languages/golang/encrypt/guest/tests/native_ops.rs‎

Lines changed: 3 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -330,15 +330,16 @@ fn guest_terms_match_the_native_sem_derivations() {
330330
.expect("native");
331331
assert_eq!(eq_bytes, native_bytes.as_bytes());
332332

333-
// Variable-width CLLW output for strings.
333+
// Variable-width CLLW output for strings: the text behind cllw-ore's
334+
// one marker byte, eight bytes a plaintext byte.
334335
let ore_s = block_on(ops::term(
335336
&cipher.default_keyset(),
336337
&encode(s("alice")),
337338
&encode(s("users/name")),
338339
TERM_ORE,
339340
))
340341
.expect("string ore");
341-
assert_eq!(ore_s.len(), 5 * 8);
342+
assert_eq!(ore_s.len(), 6 * 8);
342343

343344
// No ZeroKMS traffic for any of it.
344345
assert_eq!(cipher.kms().generate_calls.load(Ordering::SeqCst), 0);
@@ -1556,13 +1557,6 @@ fn query_term_validation_refuses_what_the_query_op_refuses() {
15561557
"between",
15571558
STATUS_ENCODING,
15581559
),
1559-
(
1560-
"inverted range",
1561-
Tree::Array(vec![Tree::UInt32(65), Tree::UInt32(18)]),
1562-
"age",
1563-
"between",
1564-
STATUS_TERM,
1565-
),
15661560
("no token", s("al"), "name", "matches", STATUS_TERM),
15671561
] {
15681562
let input = encode(input);

0 commit comments

Comments
 (0)