Skip to content

Commit 1c71db5

Browse files
committed
feat(stack-encrypt)!: order text by its folded order key, and answer the orderings on text
Text could not be ordered: no ordering operator was answered on a text field, and the ORE and OPE terms stored for text went through cllw-ore's own collation, which keeps case, so Zoe ordered before alice and é after z. ADR-0002 defines text's order key as its fold (Unicode 16 NFC with case and accents folded), and vitaminc 0.6 ships it in vitaminc-ore. Text ORE and OPE terms now derive over that key on the typed and the data-plan paths alike: the text is folded with vitaminc_ore::text and the folded bytes go to cllw-ore's byte encryption, whose ciphertext type is the one it gives text, so OreTerm<String> and OpeTerm<String> keep their type. The released cllw-ore and ore-rs do not implement vitaminc-ore's OrderScheme yet, so the CLLW schemes the engine already uses encrypt the key. A text term's key is derived under a framing that adds the fold's frozen label; integer, float and bytes order terms keep their bytes and framing. Café, cafe, CAFÉ and cafe with a combining accent are one term. Text holding a code point Unicode 16 does not assign has no fold and no order term. op::Ordered covers String, whose Between judges an inverted range on the folded keys through a new sealed method, Ordered::inverted: ("b", "A") is inverted, ("A", "a") is one value. The data path answers the orderings on text too, refuses an inverted text range and unfoldable order text at its boundary, and the Go generator now writes Lt, Lte, Gt, Gte and Between on a string field with ore or ope. Its notice is left for order indexes on bytes or a float, and for a second order index. Closes #1234.
1 parent 433720f commit 1c71db5

30 files changed

Lines changed: 693 additions & 157 deletions

‎Cargo.lock‎

Lines changed: 16 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,5 +131,8 @@ vitaminc-hmac = "0.6.0"
131131
# Normalized equality terms (Unicode 16 NFC text, one integer encoding for
132132
# every width), which `vitaminc-prf`'s `normalize` feature derives over.
133133
vitaminc-normalize = "0.6.0"
134+
# The text order key: the system ADR-0002's fold, pinned to Unicode 16,
135+
# which text ORE and OPE terms are derived over (ADR-0009, #1234).
136+
vitaminc-ore = "0.6.0"
134137
vitaminc-prf = { version = "0.6.0", features = ["normalize"] }
135138
vitaminc-protected = "0.6.0"

‎languages/golang/CHANGELOG.md‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,9 +41,14 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
4141
NULL.
4242
- **`encrypt.Range`**: what `Between` returns, the low and high bound's
4343
terms.
44+
- **Ordering methods on text.** A string field with `index=ore` or
45+
`index=ope` gets `Lt`, `Lte`, `Gt`, `Gte` and `Between`. Text orders by its
46+
folded form (ADR-0002: case and accents folded, under Unicode 16), so
47+
`Café` and `cafe` tie and `alice` orders before `Zoe`. Text order terms
48+
change bytes from the alpha's; none was deployed.
4449
- **stashgen notices an index no generated method queries**: an `ore` or
45-
`ope` index on text, bytes or a float (the engine answers the orderings
46-
over integers only), or the second of two order indexes on an integer.
50+
`ope` index on bytes or a float (the engine answers the orderings over
51+
integers and text only), or the second of two order indexes on a field.
4752
The index is still stored; the notice is on stderr and in the file.
4853

4954
### Changed

‎languages/golang/cmd/stashgen/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -124,7 +124,7 @@ For `-type User`, the file `user_stash.go` holds:
124124
- `Fields`, with one entry for each sealed field.
125125
An opaque struct, or a struct with no sealed field outside an opaque one, gets no `Fields`: nothing in it is sealed on its own, so nothing can be queried.
126126
An entry encrypts one value, and has one query method for each operator the engine answers on the field: `Eq`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` or `Matches`.
127-
stashgen asks the embedded engine which operators each field answers, and which index answers each one, so it holds no copy of those rules: an equality index answers `Eq`, a match index on a string `Matches`, and an ORE or OPE index on an integer the four orderings and `Between`, through ORE when the field declares both.
127+
stashgen asks the embedded engine which operators each field answers, and which index answers each one, so it holds no copy of those rules: an equality index answers `Eq`, a match index on a string `Matches`, and an ORE or OPE index on an integer or a string the four orderings and `Between`, through ORE when the field declares both. A string orders by its folded form: case and accents folded, under Unicode 16.
128128
`encrypt_into=TextEq` answers `Eq`, which returns the type's query value, `eql.TextEqQuery`.
129129
`encrypt_into=TextMatch` answers `Matches`, which returns an `encrypt.EQLMatchQuery`: the `eql.TextMatchQuery` operand, with `Verify` and `TargetLimit`.
130130
The query of `Matches` carries `Verify` and `TargetLimit`, because its answer can hold rows that do not match; an exact operator's query carries neither.

‎languages/golang/encrypt/README.md‎

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -243,17 +243,25 @@ rules.
243243
| The field declares | Methods | Each returns |
244244
|---|---|---|
245245
| `index=equality` | `Eq` | `encrypt.EqualityTerm` |
246-
| `index=ore` or `index=ope`, on an integer | `Lt`, `Lte`, `Gt`, `Gte` | `encrypt.OreTerm`, or `encrypt.OpeTerm` when the field declares no `ore` |
246+
| `index=ore` or `index=ope`, on an integer or a string | `Lt`, `Lte`, `Gt`, `Gte` | `encrypt.OreTerm`, or `encrypt.OpeTerm` when the field declares no `ore` |
247247
| the same | `Between(ctx, cipher, lo, hi)` | `encrypt.Range[encrypt.OreTerm]` or `encrypt.Range[encrypt.OpeTerm]`, the two bounds' terms |
248248
| `index=match`, on a string | `Matches` | `encrypt.MatchQuery[T]` |
249249
| `encrypt_into=TextEq` | `Eq` | `eql.TextEqQuery` |
250250
| `encrypt_into=TextMatch` | `Matches` | `encrypt.EQLMatchQuery[eql.TextMatchQuery, T]` |
251251
252252
A method takes the field's own Go type, so `Fields.Age.Lt` on a `uint32`
253-
field takes a `uint32`. An order index on text, bytes or a float stores its
254-
term but answers no ordering operator yet, so such a field has no `Lt`; nor
255-
does the `ope` index of a field that declares `ore` too, since `ore` answers
256-
the orderings. stashgen prints a notice for each such index, on stderr and in
253+
field takes a `uint32`.
254+
255+
Text orders by its folded form (ADR-0002): case and accents are folded and
256+
`ß` becomes `ss`, under Unicode 16. So `Café`, `cafe` and `CAFÉ` are one
257+
value and tie, `alice` orders before `Zoe`, and `é` orders with `e`, not
258+
after `z`. `Between("b", "A")` is refused as an inverted range, since `A`
259+
folds to `a`. Text holding a code point Unicode 16 does not assign has no
260+
fold, and its query or write is refused with `encrypt.ErrTerm`.
261+
262+
An order index on bytes or a float stores its term but answers no ordering
263+
operator, so such a field has no `Lt`; nor does the `ope` index of a field
264+
that declares `ore` too, since `ore` answers the orderings. stashgen prints a notice for each such index, on stderr and in
257265
the generated file: the terms are stored, and only your own code can compare
258266
them. A match index on an index-only field (`index=match` with no `encrypt`)
259267
is refused, because `Verify` checks a match query's rows by decrypting them.

‎languages/golang/encrypt/doc.go‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -137,7 +137,10 @@
137137
//
138138
// Eq returns an [EqualityTerm] (or, for a field with encrypt_into, the EQL
139139
// type's query value), an ordering an [OreTerm] or [OpeTerm], and Between a
140-
// [Range] of two. Their answers are exact. Matches returns a [MatchQuery],
140+
// [Range] of two. Their answers are exact. The orderings are answered on
141+
// integers and on text, which orders by its folded form (case and accents
142+
// folded, under Unicode 16), so Café and cafe tie and alice orders before
143+
// Zoe. Matches returns a [MatchQuery],
141144
// whose answer can hold rows that do not match: size the fetch with
142145
// [MatchQuery.TargetLimit], then decrypt the rows and keep those
143146
// [MatchQuery.Verify] reports true, one call into the engine for the page.

‎languages/golang/encrypt/guest/Cargo.lock‎

Lines changed: 16 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎languages/golang/encrypt/internal/testusers/kinds_stash.go‎

Lines changed: 27 additions & 7 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎languages/golang/encrypt/internal/testusers/probe_stash.go‎

Lines changed: 32 additions & 12 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎languages/golang/encrypt/order_test.go‎

Lines changed: 71 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,10 @@ import (
44
"bytes"
55
"cmp"
66
"context"
7+
"errors"
78
"math/rand"
89
"reflect"
10+
"slices"
911
"testing"
1012
"testing/quick"
1113

@@ -17,7 +19,7 @@ import (
1719
// stores for them through the generated Probe fields' Encrypt, comparison
1820
// in Go. A query method derives the same bytes, but only for the index
1921
// that answers its operator (ORE, where a field declares both), and only
20-
// over integers, so the stored terms reach every index and kind. ORE is
22+
// over integers and text, so the stored terms reach every index and kind. ORE is
2123
// probabilistic — a wrong comparator or a wrong derivation still agrees
2224
// with plaintext order on many pairs — so a fixed vector set says little;
2325
// hundreds of random pairs per type say more. The index key is the
@@ -121,24 +123,31 @@ func adjacentProperty[T any](t *testing.T, cipher *encrypt.Cipher, ope bool, der
121123
}
122124
}
123125

124-
// collatedAlphabet holds characters that the ORE and OPE string encodings
125-
// keep as they are. Before deriving a term, cllw-ore's orderize_string
126-
// decomposes each character canonically and drops anything that is not
127-
// alphanumeric, whitespace or ASCII punctuation. So two strings order by
128-
// their UTF-8 bytes only when that collation leaves both unchanged: a
129-
// private-use character is dropped, and a precomposed Hangul syllable or
130-
// an accented letter decomposes. The non-ASCII letters here have no
131-
// canonical decomposition, so multi-byte UTF-8 ordering is still covered.
132-
var collatedAlphabet = []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789 !\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~ßжω中")
126+
// foldedAlphabet holds characters that text's order key keeps as they are.
127+
// Text orders by its fold (ADR-0002): NFD, full case folding, NFD again,
128+
// accents dropped, NFC. So two strings order by their UTF-8 bytes when the
129+
// fold leaves both unchanged: lower-case letters with no accent, digits,
130+
// space and ASCII punctuation. The non-ASCII letters here are already
131+
// folded and have no decomposition, so multi-byte UTF-8 ordering is still
132+
// covered. Upper case, accents and the sharp s are checked as ties below.
133+
var foldedAlphabet = []rune("abcdefghijklmnopqrstuvwxyz0123456789 !\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~\u0436\u03c9\u4e2d")
133134

134-
func collatedString(r *rand.Rand) string {
135+
func foldedString(r *rand.Rand) string {
135136
out := make([]rune, r.Intn(24))
136137
for i := range out {
137-
out[i] = collatedAlphabet[r.Intn(len(collatedAlphabet))]
138+
out[i] = foldedAlphabet[r.Intn(len(foldedAlphabet))]
138139
}
139140
return string(out)
140141
}
141142

143+
// inOrder orders strings by their place in values, for a list written in
144+
// the order the fold gives it.
145+
func inOrder(values []string) func(a, b string) int {
146+
return func(a, b string) int {
147+
return cmp.Compare(slices.Index(values, a), slices.Index(values, b))
148+
}
149+
}
150+
142151
func TestTermOrderIsPlaintextOrder(t *testing.T) {
143152
c := deterministicClient(t)
144153
cipher := c.DefaultKeyset()
@@ -176,16 +185,20 @@ func TestTermOrderIsPlaintextOrder(t *testing.T) {
176185
d := storedTerm(f.S.Encrypt, ope,
177186
func(e testusers.EncryptedProbeS) []byte { return e.Ore },
178187
func(e testusers.EncryptedProbeS) []byte { return e.Ope })
179-
// Strings order by the UTF-8 bytes of their collated form; a
180-
// prefix orders before its extensions. See collatedAlphabet.
181-
orderProperty(t, cipher, ope, d, byteOrder, collatedString)
182-
adjacentProperty(t, cipher, ope, d, []string{"", "a", "aa", "ab", "b", "ba", "ß", "ßa", "中"}, byteOrder)
183-
// Collation drops a control or private-use character, and
184-
// strips the accent from a decomposed letter, so the terms
185-
// cannot tell these pairs apart.
186-
adjacentProperty(t, cipher, ope, d, []string{"", "\x7f"}, same)
187-
adjacentProperty(t, cipher, ope, d, []string{"ab", "ab"}, same)
188-
adjacentProperty(t, cipher, ope, d, []string{"e", "é"}, same)
188+
// Strings order by the UTF-8 bytes of their folded form; a
189+
// prefix orders before its extensions. See foldedAlphabet.
190+
orderProperty(t, cipher, ope, d, byteOrder, foldedString)
191+
adjacentProperty(t, cipher, ope, d, []string{"", "a", "aa", "ab", "b", "ba", "ж", "ω", "中"}, byteOrder)
192+
// The fold drops case and accents and expands ß, so the terms
193+
// cannot tell these apart.
194+
adjacentProperty(t, cipher, ope, d, []string{"a", "A"}, same)
195+
adjacentProperty(t, cipher, ope, d, []string{"e", "\u00e9", "\u00c9", "e\u0301"}, same)
196+
adjacentProperty(t, cipher, ope, d, []string{"ss", "\u00df", "SS"}, same)
197+
adjacentProperty(t, cipher, ope, d, []string{"caf\u00e9", "CAFE", "Cafe\u0301"}, same)
198+
// Mixed case and accents order by the fold: Zoe after alice,
199+
// and an accented letter beside its base letter.
200+
names := []string{"alice", "Bob", "eclair", "\u00c9lan", "emu", "Zoe"}
201+
adjacentProperty(t, cipher, ope, d, names, inOrder(names))
189202
})
190203
t.Run("bytes", func(t *testing.T) {
191204
d := storedTerm(f.B.Encrypt, ope,
@@ -197,3 +210,39 @@ func TestTermOrderIsPlaintextOrder(t *testing.T) {
197210
})
198211
}
199212
}
213+
214+
// The generated ordering methods on a text field derive the term its
215+
// Encrypt stores for the folded value, and Between refuses a range that
216+
// is inverted once folded.
217+
func TestTextOrderingMethodsFoldTheValue(t *testing.T) {
218+
c := deterministicClient(t)
219+
cipher := c.DefaultKeyset()
220+
ctx := context.Background()
221+
f := testusers.ProbeFields
222+
stored, err := f.S.Encrypt(ctx, cipher, "cafe")
223+
if err != nil {
224+
t.Fatal(err)
225+
}
226+
for _, v := range []string{"cafe", "Caf\u00e9", "CAFE", "cafe\u0301"} {
227+
lt, err := f.S.Lt(ctx, cipher, v)
228+
if err != nil {
229+
t.Fatal(err)
230+
}
231+
if !bytes.Equal(lt, stored.Ore) {
232+
t.Errorf("Lt(%q) is not the stored term of cafe", v)
233+
}
234+
}
235+
r, err := f.S.Between(ctx, cipher, "a", "Zoe")
236+
if err != nil {
237+
t.Fatal(err)
238+
}
239+
if r.Low.Compare(r.High) >= 0 {
240+
t.Errorf("a should order below Zoe")
241+
}
242+
if _, err := f.S.Between(ctx, cipher, "b", "A"); !errors.Is(err, encrypt.ErrTerm) {
243+
t.Errorf("Between(b, A) = %v, want ErrTerm: A folds to a, below b", err)
244+
}
245+
if _, err := f.S.Between(ctx, cipher, "A", "a"); err != nil {
246+
t.Errorf("Between(A, a) is one folded value: %v", err)
247+
}
248+
}

0 commit comments

Comments
 (0)