feat(stack-encrypt)!: order text, encrypting every order term through cllw-ore's vitaminc schemes

Text could not be ordered: no ordering operator was answered on a text
field, and the ORE and OPE terms stored for text went through cllw-ore's
own collation, which keeps case, so Zoe ordered before alice and é after
z. ADR-0002 defines text's order key as its fold (Unicode 16 NFC with
case and accents folded), and vitaminc 0.6 ships it in vitaminc-ore.

cllw-ore 0.5.2-rc.1 implements vitaminc-ore's `OrderScheme` behind its
`vitaminc` feature, so Stack Encrypt encodes nothing for ordering itself.
An ORE or OPE term is now cllw-ore's `vitaminc::Ore` or `vitaminc::Ope`
encryption of the value: the engine names the value's order target
through the new sealed `sem::Orderable` (integers at their own width,
`f32`/`f64` as `F64`, text as `Text`, bytes as `Bytes`) and passes the
value straight through, on the typed path and for each scalar of a
dynamic value alike; cllw-ore encodes it with vitaminc-ore, text by the
fold. Café, cafe, CAFÉ and cafe with a combining accent are one term.
Text holding a code point Unicode 16 does not assign has no fold and no
order term, as it has no equality or match term; ADR-0002 is amended to
say vitaminc 0.6 refuses such a code point rather than passing it
through.

The per-index key derivation is kept: the PRF block keys the scheme
inside the visitor and is wiped there, and the column is opened with the
value's target and the empty context, since the key is already the
index's own. The key labels move to `/v2` because every order term
changes bytes (new column salt, cllw-ore's marker byte for text and
bytes). Every source type's term is now one ciphertext type
(`OreCllw8VariableV1`, `OpeCllw8VariableV1`); `OreTerm<T>`/`OpeTerm<T>`
keep `T` as a marker so typed callers compile unchanged, and
`sem::CllwOreEncrypt`/`CllwOpeEncrypt` are no longer re-exported. The
term pins and the record, query and value-kinds fixtures are re-recorded.

op::Ordered covers String, and the data path answers the orderings on
text too, refusing order text with no fold at its boundary, for one
value and for each bound of a range, through `vitaminc_ore::text`, the
encoding the `Text` target uses. The Go generator writes Lt, Lte, Gt,
Gte and Between on a string field with ore or ope; its notice is left
for order indexes on bytes or a float, and for a second order index.

Closes #1234.

Author: coderdan

Cargo.lock

@@ -81,7 +81,10 @@ stack-profile = { path = "./packages/stack-profile", version = "0.43.0" }
 cts-common = { version = "0.43.0", default-features = false }
 zerokms-protocol = "0.12.31"
 recipher = "=0.3.1"
-cllw-ore = { version = "=0.5.0", default-features = false }
+# cllw-ore's `vitaminc` feature implements vitaminc-ore's `OrderScheme`,
+# which Stack Encrypt's order terms are encrypted through. It first ships
+# in a pre-release, and Cargo resolves a pre-release only when named.
+cllw-ore = { version = "=0.5.2-rc.1", default-features = false, features = ["vitaminc"] }
 
 # External dependencies, with the suite's feature lists.
 base64 = "0.22.0"
@@ -131,5 +134,8 @@ vitaminc-hmac = "0.6.0"
 # Normalized equality terms (Unicode 16 NFC text, one integer encoding for
 # every width), which `vitaminc-prf`'s `normalize` feature derives over.
 vitaminc-normalize = "0.6.0"
+# The text order key: the system ADR-0002's fold, pinned to Unicode 16,
+# which text ORE and OPE terms are derived over (ADR-0009).
+vitaminc-ore = "0.6.0"
 vitaminc-prf = { version = "0.6.0", features = ["normalize"] }
 vitaminc-protected = "0.6.0"

Cargo.toml

@@ -219,6 +219,13 @@ write. The cost is that two strings using characters added after the pin,
 which a later Unicode would treat as equivalent, may not match. Moving the
 pin is a new label: existing columns keep the label they were written under
 and stay queryable, and a column moves only by re-encryption.
+*Amended 2026-10-18:* vitaminc 0.6 refuses a code point unassigned in the
+pinned version instead of passing it through. `vitaminc-normalize` (equality
+and match terms) and `vitaminc-ore`'s fold (order terms) both return
+`UnassignedCodePoint`, so text holding one has no term, and writing it to an
+indexed column or querying with it is refused. The refusal is the same on
+every path and in every language, so the transform is still a fixed
+function of the text it accepts, and moving the pin is still a new label.
 
 Order terms fold case and accents because code-point order puts `é` after
 `z`; a fixed fold approximates the first level of Unicode collation without

docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md

@@ -70,9 +70,14 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
   `float32` or `float64` field tagged `encrypt_into=` one of them gets `Eq`.
   `=` compares as PostgreSQL compares the plain types: an integer has one
   term whatever its width, and for floats `-0.0 = 0.0` and `NaN = NaN`.
+- **Ordering methods on text.** A string field with `index=ore` or
+  `index=ope` gets `Lt`, `Lte`, `Gt`, `Gte` and `Between`. Text orders by its
+  folded form (ADR-0002: case and accents folded, under Unicode 16), so
+  `Café` and `cafe` tie and `alice` orders before `Zoe`. Text order terms
+  change bytes from the alpha's; none was deployed.
 - **stashgen notices an index no generated method queries**: an `ore` or
-  `ope` index on text, bytes or a float (the engine answers the orderings
-  over integers only), or the second of two order indexes on an integer.
+  `ope` index on bytes or a float (the engine answers the orderings over
+  integers and text only), or the second of two order indexes on a field.
   The index is still stored; the notice is on stderr and in the file.
 
 ### Changed

languages/golang/CHANGELOG.md

@@ -124,7 +124,7 @@ For `-type User`, the file `user_stash.go` holds:
 - `Fields`, with one entry for each sealed field.
   An opaque struct, or a struct with no sealed field outside an opaque one, gets no `Fields`: nothing in it is sealed on its own, so nothing can be queried.
   An entry encrypts one value, and has one query method for each operator the engine answers on the field: `Eq`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` or `Matches`.
-  stashgen asks the embedded engine which operators each field answers, and which index answers each one, so it holds no copy of those rules: an equality index answers `Eq`, a match index on a string `Matches`, and an ORE or OPE index on an integer the four orderings and `Between`, through ORE when the field declares both.
+  stashgen asks the embedded engine which operators each field answers, and which index answers each one, so it holds no copy of those rules: an equality index answers `Eq`, a match index on a string `Matches`, and an ORE or OPE index on an integer or a string the four orderings and `Between`, through ORE when the field declares both. A string orders by its folded form: case and accents folded, under Unicode 16.
   `encrypt_into=TextEq` answers `Eq`, which returns the type's query value, `eql.TextEqQuery`.
   `encrypt_into=TextMatch` answers `Matches`, which returns an `encrypt.EQLMatchQuery`: the `eql.TextMatchQuery` operand, with `Verify` and `TargetLimit`.
   The query of `Matches` carries `Verify` and `TargetLimit`, because its answer can hold rows that do not match; an exact operator's query carries neither.

languages/golang/cmd/stashgen/README.md

@@ -210,7 +210,7 @@ or a type defined over one (`type Email string`):
 
 | Go field | Seals as | Indexes it can store | Query methods |
 |---|---|---|---|
-| `string` | text | `equality`, `match`, `ore`, `ope` | `Eq`, `Matches` |
+| `string` | text | `equality`, `match`, `ore`, `ope` | `Eq`, `Matches`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` |
 | `[]byte` | bytes | `equality`, `ore`, `ope` | `Eq` |
 | `bool` | a boolean | none | none |
 | `int8`, `int16`, `int32`, `int64`, `int` | an integer of the field's own width (`int` is 64-bit) | `equality`, `ore`, `ope` | `Eq`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` |
@@ -221,9 +221,9 @@ or a type defined over one (`type Email string`):
 | `encrypt.Date` | a calendar date | `equality` | `Eq` |
 | `time.Time` | a timestamp, to the nanosecond, in UTC | `equality` | `Eq` |
 
-Each query method needs its index declared. An `ore` or `ope` index on text,
-bytes or a float stores its term, but the engine answers no ordering through
-it yet, so such a field has no `Lt` (see [Queries](#queries)).
+Each query method needs its index declared. An `ore` or `ope` index on bytes
+or a float stores its term, but the engine answers no ordering through it
+yet, so such a field has no `Lt` (see [Queries](#queries)).
 
 `encrypt.Date`, `Int128`, `Uint128` and `Decimal` are vitaminc's `vcvalue`
 types; build them with `encrypt.DateOf`, `ParseInt128`, `ParseUint128` and
@@ -279,17 +279,27 @@ rules.
 | The field declares | Methods | Each returns |
 |---|---|---|
 | `index=equality` | `Eq` | `encrypt.EqualityTerm` |
-| `index=ore` or `index=ope`, on an integer of any width | `Lt`, `Lte`, `Gt`, `Gte` | `encrypt.OreTerm`, or `encrypt.OpeTerm` when the field declares no `ore` |
+| `index=ore` or `index=ope`, on an integer of any width or a string | `Lt`, `Lte`, `Gt`, `Gte` | `encrypt.OreTerm`, or `encrypt.OpeTerm` when the field declares no `ore` |
 | the same | `Between(ctx, cipher, lo, hi)` | `encrypt.Range[encrypt.OreTerm]` or `encrypt.Range[encrypt.OpeTerm]`, the two bounds' terms |
 | `index=match`, on a string | `Matches` | `encrypt.MatchQuery[T]` |
 | `encrypt_into=TextEq` | `Eq` | `eql.TextEqQuery` |
 | `encrypt_into=TextMatch` | `Matches` | `encrypt.EQLMatchQuery[eql.TextMatchQuery, T]` |
 
 A method takes the field's own Go type, so `Fields.Age.Lt` on a `uint32`
-field takes a `uint32`. An order index on text, bytes or a float stores its
-term but answers no ordering operator yet, so such a field has no `Lt`; nor
-does the `ope` index of a field that declares `ore` too, since `ore` answers
-the orderings. stashgen prints a notice for each such index, on stderr and in
+field takes a `uint32`.
+
+Text orders by its folded form (ADR-0002): case and accents are folded and
+`ß` becomes `ss`, under Unicode 16. So `Café`, `cafe` and `CAFÉ` are one
+value and tie, `alice` orders before `Zoe`, and `é` orders with `e`, not
+after `z`. `Between` is inclusive at both ends and, as SQL's `BETWEEN`,
+does not refuse a range whose low bound is above its high bound: such a
+range matches nothing. So `Between("b", "A")` matches nothing, since `A`
+folds to `a`, below `b`. Text holding a code point Unicode 16 does not assign has no
+fold, and its query or write is refused with `encrypt.ErrTerm`.
+
+An order index on bytes or a float stores its term but answers no ordering
+operator, so such a field has no `Lt`; nor does the `ope` index of a field
+that declares `ore` too, since `ore` answers the orderings. stashgen prints a notice for each such index, on stderr and in
 the generated file: the terms are stored, and only your own code can compare
 them. A match index on an index-only field (`index=match` with no `encrypt`)
 is refused, because `Verify` checks a match query's rows by decrypting them.

languages/golang/encrypt/README.md

@@ -152,7 +152,10 @@
 //
 // Eq returns an [EqualityTerm] (or, for a field with encrypt_into, the EQL
 // type's query value), an ordering an [OreTerm] or [OpeTerm], and Between a
-// [Range] of two. Their answers are exact. Matches returns a [MatchQuery],
+// [Range] of two. Their answers are exact. The orderings are answered on
+// integers and on text, which orders by its folded form (case and accents
+// folded, under Unicode 16), so Café and cafe tie and alice orders before
+// Zoe. Matches returns a [MatchQuery],
 // whose answer can hold rows that do not match: size the fetch with
 // [MatchQuery.TargetLimit], then decrypt the rows and keep those
 // [MatchQuery.Verify] reports true, one call into the engine for the page.

languages/golang/encrypt/doc.go

@@ -327,15 +327,16 @@ fn guest_terms_match_the_native_sem_derivations() {
     .expect("native");
     assert_eq!(eq_bytes, native_bytes.as_bytes());
 
-    // Variable-width CLLW output for strings.
+    // Variable-width CLLW output for strings: the text behind cllw-ore's
+    // one marker byte, eight bytes a plaintext byte.
     let ore_s = block_on(ops::term(
         &cipher.default_keyset(),
         &encode(s("alice")),
         &encode(s("users/name")),
         TERM_ORE,
     ))
     .expect("string ore");
-    assert_eq!(ore_s.len(), 5 * 8);
+    assert_eq!(ore_s.len(), 6 * 8);
 
     // No ZeroKMS traffic for any of it.
     assert_eq!(cipher.kms().generate_calls.load(Ordering::SeqCst), 0);