Skip to content

Commit 06ba56e

Browse files
authored
Merge pull request #1378 from cipherstash/feat/go-json-typed-documents
feat(go): encrypted JSON documents of Go types, with typed needles
2 parents f9b93ac + c8c6a39 commit 06ba56e

25 files changed

Lines changed: 2161 additions & 118 deletions

‎languages/golang/CHANGELOG.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,37 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
1212

1313
### Added
1414

15+
- **Encrypted JSON documents of Go types** (#1351). A struct, map or slice
16+
field tagged `encrypt_into=JsonSearch` is a document: the generated code
17+
marshals it with `encoding/json` when it seals it, and reads the opened
18+
document back into the declared type (`gensupport.GetDocument`,
19+
`GetDocumentPtr`), so 64-bit integers and finite `float64`s round-trip
20+
exactly. A pointer to one takes `null=sql`. The rest of a document comes
21+
back as `encoding/json` reads it: invalid UTF-8 in a string becomes
22+
U+FFFD, a nil map or slice is stored as JSON `null` (not `{}` or `[]`),
23+
and a `time.Time` inside a document is stored as RFC 3339 text in its
24+
own offset, which `EqAt` and the orderings compare as text. `stashgen` refuses, when it
25+
runs, a type `encoding/json` cannot carry both ways: an interface such as
26+
`any`, a struct field `encoding/json` drops (unexported, tagged
27+
`json:"-"`, or two writing one member), a field tagged
28+
`json:",string"` (stored as a JSON string, which a query by its number
29+
does not match), an embedded pointer to an
30+
unexported struct (written, never read back), a one-way `MarshalJSON` or
31+
`MarshalText`, a channel, a function, a scalar, or a `[]byte`. The `Fields` entry's operands are Go
32+
values now: `Contains`, `ContainedBy`, `EqAt`, `Lt`, `Lte`, `Gt`, `Gte`
33+
and `Between` take `any`,
34+
a `json.RawMessage` crossing as the JSON text it is and anything else
35+
marshalled (`gensupport.JSONText`); a `[]byte`, a type defined over
36+
one, or a pointer to either is refused, and so is a containment needle
37+
that is a Go string, a pointer to one, or writes JSON `null`. A zero
38+
struct field without `omitempty` narrows a containment query, so
39+
`Contains` on a field whose type holds a struct refuses a needle of that
40+
type (`gensupport.NewDocumentField`); the generated doc comments say how
41+
to write a needle. `encrypt.DecodeJSON[T]` reads a value a selection or
42+
subtree read opened into a Go type, numbers in an interface as
43+
`json.Number`. An error from either direction names a Go type and never
44+
the value.
45+
1546
- **Encrypted JSON documents with `stashgen`** (#1351, #1352). A
1647
`json.RawMessage` field tagged `encrypt_into=JsonSearch` (options:
1748
`ordered=$.a;$.b[*].c`, `sealed_root=false`) is stored as an

‎languages/golang/cmd/stashgen/README.md‎

Lines changed: 62 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -74,7 +74,7 @@ The first part of a tag is the field's name, which is the column name in a datab
7474
| `stash:"notes,encrypt"` | seal the field, with no index |
7575
| `stash:"email,encrypt,index=equality;match"` | seal the field, and derive each index beside it |
7676
| `stash:"score,index=ore"` | derive the index alone; no ciphertext is stored, so `Decrypt` leaves the field at its zero value |
77-
| `stash:"profile,encrypt_into=JsonSearch(ordered=$.age)"` | on a `json.RawMessage`: store the field as an encrypted JSON document; see "Encrypted JSON documents" |
77+
| `stash:"profile,encrypt_into=JsonSearch(ordered=$.age)"` | on a `json.RawMessage`, or a struct, map or slice type: store the field as an encrypted JSON document; see "Encrypted JSON documents" |
7878
| `stash:"attrs,index=json"` | refused: a JSON document is stored with `encrypt_into=JsonSearch`, not as an index beside a ciphertext |
7979
| `stash:"id,passthrough"` | store the field as it is |
8080
| `stash:"tenant,context_field"` | the field's value is the context of every other field in the struct, in place of a `context=` tag; stored as it is |
@@ -113,12 +113,55 @@ These words are the same as the Rust API's words for the same behaviour, with tw
113113

114114
### Encrypted JSON documents
115115

116-
A `json.RawMessage` field with `encrypt_into=JsonSearch` is stored as one encrypted JSON document, an `eql.JSONSearch` in a `public.eql_v4_json_search` column.
117-
The field holds the document's JSON text; the engine parses it, so an object that names a member twice, an integer outside the 64-bit range and text that is not JSON are refused with `encrypt.ErrEncoding`.
118-
Values come back as JSON text holding the same values; key order and the spelling of numbers are not kept.
119-
A `*json.RawMessage` field is optional and must say `null=sql`: a document has no fixed length to hide a missing one behind. A non-nil pointer to `null` is a stored JSON `null`.
116+
A field with `encrypt_into=JsonSearch` is stored as one encrypted JSON document, an `eql.JSONSearch` in a `public.eql_v4_json_search` column.
117+
Declare it as a struct, map or slice type, or as a `json.RawMessage` that holds the document's JSON text:
118+
119+
```go
120+
type Profile struct {
121+
Name string `json:"name"`
122+
Age int64 `json:"age"`
123+
Visits []Visit `json:"visits"`
124+
}
125+
126+
type Customer struct {
127+
_ struct{} `stash:"context=customers"`
128+
ID int64 `stash:"id,passthrough"`
129+
Profile Profile `stash:"profile,encrypt_into=JsonSearch(ordered=$.age)"`
130+
Notes json.RawMessage `stash:"notes,encrypt_into=JsonSearch"`
131+
}
132+
133+
// A needle for Contains: a field left nil is not in the query.
134+
type ProfileNeedle struct {
135+
Name *string `json:"name,omitempty"`
136+
Age *int64 `json:"age,omitempty"`
137+
}
138+
139+
q, err := customers.Fields.Profile.Contains(ctx, cipher, ProfileNeedle{Age: &age})
140+
older, err := customers.Fields.Profile.At.Age.Gt(ctx, cipher, 40)
141+
people, err := customers.Decrypt(ctx, cipher, rows) // each Profile is a Profile again
142+
```
143+
144+
The generated code marshals a struct, map or slice type with `encoding/json` when it seals the field, and unmarshals the opened document into the declared type.
145+
It calls `encoding/json` directly, never through `map[string]any`, so an `int64`, a `uint64` and a finite `float64` come back exactly.
146+
The rest of the document comes back as `encoding/json` reads what it wrote, which is not always the value sealed:
147+
148+
- Invalid UTF-8 in a Go string is written as U+FFFD, with no error: `"a\xffb"` comes back as `"a\uFFFDb"`, the replacement character in place of the byte. `stashgen` cannot refuse it, because it is a property of a value, not of a type, and finding it would need a reflective walk of each value at run time, which the generated code does not do. Check text that may not be UTF-8 with `utf8.ValidString` before you seal it.
149+
- A nil map or slice is stored as JSON `null`, and an empty one as `{}` or `[]`. Each reads back as it was (a field with `null=sql` stores a nil pointer as SQL `NULL` instead), but a query tells them apart: `Contains` with an empty map does not match a stored `null`.
150+
- A `time.Time` field is refused, since it is one value, not a document, but a `time.Time` inside a document is accepted. It is stored as the text its `MarshalJSON` writes: RFC 3339, in the time's own offset, with trailing zero nanoseconds left out. One instant in two zones is two strings, so `EqAt` with a time in another zone matches nothing, and an ordering on that path orders the text, which is not time order: `…T10:00:00+10:00` sorts after `…T01:00:00Z`, an earlier instant, and `…:00.5Z` sorts before `…:00Z`. If you query a time, store it as an integer (`t.UnixNano()`) or as UTC text of a fixed width. Any type with its own `MarshalJSON` or `MarshalText` is stored as the text it writes, and queried as that text.
151+
152+
`stashgen` refuses a type that `encoding/json` cannot carry both ways: an interface such as `any` anywhere in it (it reads numbers back as `float64`); a struct field that `encoding/json` drops (unexported, tagged `json:"-"`, or one of two that write the same member); a `MarshalJSON` or `MarshalText` with no matching unmarshal method; a channel or a function.
153+
It refuses a field tagged `json:",string"` anywhere in a document: `encoding/json` stores its number or bool as a JSON string (`"age":"40"`), so `EqAt`, an ordering or a needle by the number would match nothing, and an ordered path would order the text. A document's numbers cross as their exact digits without it.
154+
An unexported embedded struct is fine: its fields are promoted and stored.
155+
An embedded pointer to an unexported struct is refused: `encoding/json` writes its fields but cannot read them back, since it cannot allocate the struct.
156+
An embedded pointer to an exported struct is accepted, but whether it was nil is not stored: it comes back non-nil when the document holds a member it promotes, and nil when it holds none, so a non-nil pointer whose promoted fields are all left out by `omitempty` comes back nil.
157+
It refuses a string, a number and a bool too: a document is a struct, map or slice.
158+
A `[]byte` is refused, since `encoding/json` writes it as base64: declare JSON text as `json.RawMessage`.
159+
The engine parses the JSON text, so an object that names a member twice and an integer outside the 64-bit range are refused with `encrypt.ErrEncoding`.
160+
A `json.RawMessage` comes back as JSON text that holds the same values; key order and the spelling of numbers are not kept.
161+
A pointer field (`*Profile`, `*json.RawMessage`) is optional and must say `null=sql`: a document has no fixed length to hide a missing one behind.
162+
A nil pointer is SQL `NULL`; a non-nil `*json.RawMessage` holding `null` is a stored JSON `null`.
120163
A `json.RawMessage` sealed any other way (`encrypt`, `index=equality`) is bytes, as any type defined over `[]byte` is.
121-
`JsonSearch` takes `json.RawMessage` itself, or an alias of it (`type Doc = json.RawMessage`), and refuses a type defined over it (`type Doc json.RawMessage`): `stashgen` knows a document by the `json.RawMessage` type, where a scalar type is accepted with any type defined over it.
164+
`JsonSearch` takes `json.RawMessage` itself, or an alias of it (`type Doc = json.RawMessage`), and refuses a type defined over it (`type Doc json.RawMessage`): such a type has no `MarshalJSON`, so `encoding/json` would write it as a base64 string, not as the JSON text it holds, where a scalar type is accepted with any type defined over it.
122165

123166
The type takes options in parentheses: `ordered=` names the paths whose numbers and strings carry an ordering term, separated by `;`, and `sealed_root=false` turns off the sealed root (on by default).
124167
`encrypt_into=JsonSearch(ordered=$.age;$.visits[*].at)` orders two paths.
@@ -150,7 +193,19 @@ SELECT id FROM customers ORDER BY (
150193

151194
with `$1` from `At.VisitsAt.Selector(ctx, c)` and `$2` from `At.VisitsAt.Gt(ctx, c, v)`; for the latest visit, order by `ORDER BY 1 DESC NULLS LAST` inside and `DESC` outside.
152195
`OpenOne` reads one value and refuses a read whose path has a wildcard before anything is opened; open a wildcard read with `Open`.
153-
The needle and the values are `json.RawMessage`: write the JSON you mean, such as `json.RawMessage(`+"`"+`{"active":true}`+"`"+`)`.
196+
A needle and a value are Go values: a `json.RawMessage` of JSON text crosses as it is, and anything else is marshalled with `encoding/json`.
197+
`EqAt` and the orderings take a scalar: a string, a number, a bool, or `nil` for JSON `null`.
198+
A `[]byte`, a type defined over one, or a pointer to either, is refused rather than written as base64.
199+
A Go string, or a pointer to one, is refused as a `Contains` or `ContainedBy` needle, since it would be one JSON string: pass JSON text as `json.RawMessage`.
200+
So is a needle that writes JSON `null` (`nil`, or a nil pointer, map or slice), which only a null document contains.
201+
For `EqAt` and the orderings a Go string is the string value itself.
202+
A document contains a needle when it holds every member the needle writes, and `encoding/json` writes a struct field that has no `omitempty` even when it is zero.
203+
So a zero field narrows the query to the documents that hold that zero: `Profile{Name: "Alice"}` also asks for `"age": 0`.
204+
Give a needle struct pointer fields with `omitempty`, as `ProfileNeedle` above, or use a map.
205+
`Contains` refuses a needle of the field's own type when that type holds a struct, since that is the likeliest way to make the mistake; a type defined over it (`type wholeProfile Profile`) states that you mean every field.
206+
That includes a slice or map of structs: a `[]Visit` document refuses a `[]Visit` needle, since each `Visit` writes its zero fields; pass a `[]VisitNeedle`, a slice of a needle type with pointer fields and `omitempty`.
207+
`ContainedBy` takes the field's own type: there a zero field widens the query.
208+
Each value a selection or read opens is JSON text; `encrypt.DecodeJSON[T]` reads one into a Go type, with the same rules as a whole document.
154209
A position in a path is refused for `EqAt` and `Selector`, whose selector reads every position as the wildcard; read a position with `Subtree`.
155210

156211
### Optional fields

0 commit comments

Comments
 (0)