You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 06ba56e
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: languages/golang/cmd/stashgen/README.md
+62-7Lines changed: 62 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -74,7 +74,7 @@ The first part of a tag is the field's name, which is the column name in a datab
74
74
|`stash:"notes,encrypt"`| seal the field, with no index |
75
75
|`stash:"email,encrypt,index=equality;match"`| seal the field, and derive each index beside it |
76
76
|`stash:"score,index=ore"`| derive the index alone; no ciphertext is stored, so `Decrypt` leaves the field at its zero value |
77
-
|`stash:"profile,encrypt_into=JsonSearch(ordered=$.age)"`| on a `json.RawMessage`: store the field as an encrypted JSON document; see "Encrypted JSON documents" |
77
+
|`stash:"profile,encrypt_into=JsonSearch(ordered=$.age)"`| on a `json.RawMessage`, or a struct, map or slice type: store the field as an encrypted JSON document; see "Encrypted JSON documents" |
78
78
|`stash:"attrs,index=json"`| refused: a JSON document is stored with `encrypt_into=JsonSearch`, not as an index beside a ciphertext |
79
79
|`stash:"id,passthrough"`| store the field as it is |
80
80
|`stash:"tenant,context_field"`| the field's value is the context of every other field in the struct, in place of a `context=` tag; stored as it is |
@@ -113,12 +113,55 @@ These words are the same as the Rust API's words for the same behaviour, with tw
113
113
114
114
### Encrypted JSON documents
115
115
116
-
A `json.RawMessage` field with `encrypt_into=JsonSearch` is stored as one encrypted JSON document, an `eql.JSONSearch` in a `public.eql_v4_json_search` column.
117
-
The field holds the document's JSON text; the engine parses it, so an object that names a member twice, an integer outside the 64-bit range and text that is not JSON are refused with `encrypt.ErrEncoding`.
118
-
Values come back as JSON text holding the same values; key order and the spelling of numbers are not kept.
119
-
A `*json.RawMessage` field is optional and must say `null=sql`: a document has no fixed length to hide a missing one behind. A non-nil pointer to `null` is a stored JSON `null`.
116
+
A field with `encrypt_into=JsonSearch` is stored as one encrypted JSON document, an `eql.JSONSearch` in a `public.eql_v4_json_search` column.
117
+
Declare it as a struct, map or slice type, or as a `json.RawMessage` that holds the document's JSON text:
people, err:= customers.Decrypt(ctx, cipher, rows) // each Profile is a Profile again
142
+
```
143
+
144
+
The generated code marshals a struct, map or slice type with `encoding/json` when it seals the field, and unmarshals the opened document into the declared type.
145
+
It calls `encoding/json` directly, never through `map[string]any`, so an `int64`, a `uint64` and a finite `float64` come back exactly.
146
+
The rest of the document comes back as `encoding/json` reads what it wrote, which is not always the value sealed:
147
+
148
+
- Invalid UTF-8 in a Go string is written as U+FFFD, with no error: `"a\xffb"` comes back as `"a\uFFFDb"`, the replacement character in place of the byte. `stashgen` cannot refuse it, because it is a property of a value, not of a type, and finding it would need a reflective walk of each value at run time, which the generated code does not do. Check text that may not be UTF-8 with `utf8.ValidString` before you seal it.
149
+
- A nil map or slice is stored as JSON `null`, and an empty one as `{}` or `[]`. Each reads back as it was (a field with `null=sql` stores a nil pointer as SQL `NULL` instead), but a query tells them apart: `Contains` with an empty map does not match a stored `null`.
150
+
- A `time.Time` field is refused, since it is one value, not a document, but a `time.Time` inside a document is accepted. It is stored as the text its `MarshalJSON` writes: RFC 3339, in the time's own offset, with trailing zero nanoseconds left out. One instant in two zones is two strings, so `EqAt` with a time in another zone matches nothing, and an ordering on that path orders the text, which is not time order: `…T10:00:00+10:00` sorts after `…T01:00:00Z`, an earlier instant, and `…:00.5Z` sorts before `…:00Z`. If you query a time, store it as an integer (`t.UnixNano()`) or as UTC text of a fixed width. Any type with its own `MarshalJSON` or `MarshalText` is stored as the text it writes, and queried as that text.
151
+
152
+
`stashgen` refuses a type that `encoding/json` cannot carry both ways: an interface such as `any` anywhere in it (it reads numbers back as `float64`); a struct field that `encoding/json` drops (unexported, tagged `json:"-"`, or one of two that write the same member); a `MarshalJSON` or `MarshalText` with no matching unmarshal method; a channel or a function.
153
+
It refuses a field tagged `json:",string"` anywhere in a document: `encoding/json` stores its number or bool as a JSON string (`"age":"40"`), so `EqAt`, an ordering or a needle by the number would match nothing, and an ordered path would order the text. A document's numbers cross as their exact digits without it.
154
+
An unexported embedded struct is fine: its fields are promoted and stored.
155
+
An embedded pointer to an unexported struct is refused: `encoding/json` writes its fields but cannot read them back, since it cannot allocate the struct.
156
+
An embedded pointer to an exported struct is accepted, but whether it was nil is not stored: it comes back non-nil when the document holds a member it promotes, and nil when it holds none, so a non-nil pointer whose promoted fields are all left out by `omitempty` comes back nil.
157
+
It refuses a string, a number and a bool too: a document is a struct, map or slice.
158
+
A `[]byte` is refused, since `encoding/json` writes it as base64: declare JSON text as `json.RawMessage`.
159
+
The engine parses the JSON text, so an object that names a member twice and an integer outside the 64-bit range are refused with `encrypt.ErrEncoding`.
160
+
A `json.RawMessage` comes back as JSON text that holds the same values; key order and the spelling of numbers are not kept.
161
+
A pointer field (`*Profile`, `*json.RawMessage`) is optional and must say `null=sql`: a document has no fixed length to hide a missing one behind.
162
+
A nil pointer is SQL `NULL`; a non-nil `*json.RawMessage` holding `null` is a stored JSON `null`.
120
163
A `json.RawMessage` sealed any other way (`encrypt`, `index=equality`) is bytes, as any type defined over `[]byte` is.
121
-
`JsonSearch` takes `json.RawMessage` itself, or an alias of it (`type Doc = json.RawMessage`), and refuses a type defined over it (`type Doc json.RawMessage`): `stashgen` knows a document by the `json.RawMessage` type, where a scalar type is accepted with any type defined over it.
164
+
`JsonSearch` takes `json.RawMessage` itself, or an alias of it (`type Doc = json.RawMessage`), and refuses a type defined over it (`type Doc json.RawMessage`): such a type has no `MarshalJSON`, so `encoding/json` would write it as a base64 string, not as the JSON text it holds, where a scalar type is accepted with any type defined over it.
122
165
123
166
The type takes options in parentheses: `ordered=` names the paths whose numbers and strings carry an ordering term, separated by `;`, and `sealed_root=false` turns off the sealed root (on by default).
124
167
`encrypt_into=JsonSearch(ordered=$.age;$.visits[*].at)` orders two paths.
@@ -150,7 +193,19 @@ SELECT id FROM customers ORDER BY (
150
193
151
194
with `$1` from `At.VisitsAt.Selector(ctx, c)` and `$2` from `At.VisitsAt.Gt(ctx, c, v)`; for the latest visit, order by `ORDER BY 1 DESC NULLS LAST` inside and `DESC` outside.
152
195
`OpenOne` reads one value and refuses a read whose path has a wildcard before anything is opened; open a wildcard read with `Open`.
153
-
The needle and the values are `json.RawMessage`: write the JSON you mean, such as `json.RawMessage(`+"`"+`{"active":true}`+"`"+`)`.
196
+
A needle and a value are Go values: a `json.RawMessage` of JSON text crosses as it is, and anything else is marshalled with `encoding/json`.
197
+
`EqAt` and the orderings take a scalar: a string, a number, a bool, or `nil` for JSON `null`.
198
+
A `[]byte`, a type defined over one, or a pointer to either, is refused rather than written as base64.
199
+
A Go string, or a pointer to one, is refused as a `Contains` or `ContainedBy` needle, since it would be one JSON string: pass JSON text as `json.RawMessage`.
200
+
So is a needle that writes JSON `null` (`nil`, or a nil pointer, map or slice), which only a null document contains.
201
+
For `EqAt` and the orderings a Go string is the string value itself.
202
+
A document contains a needle when it holds every member the needle writes, and `encoding/json` writes a struct field that has no `omitempty` even when it is zero.
203
+
So a zero field narrows the query to the documents that hold that zero: `Profile{Name: "Alice"}` also asks for `"age": 0`.
204
+
Give a needle struct pointer fields with `omitempty`, as `ProfileNeedle` above, or use a map.
205
+
`Contains` refuses a needle of the field's own type when that type holds a struct, since that is the likeliest way to make the mistake; a type defined over it (`type wholeProfile Profile`) states that you mean every field.
206
+
That includes a slice or map of structs: a `[]Visit` document refuses a `[]Visit` needle, since each `Visit` writes its zero fields; pass a `[]VisitNeedle`, a slice of a needle type with pointer fields and `omitempty`.
207
+
`ContainedBy` takes the field's own type: there a zero field widens the query.
208
+
Each value a selection or read opens is JSON text; `encrypt.DecodeJSON[T]` reads one into a Go type, with the same rules as a whole document.
154
209
A position in a path is refused for `EqAt` and `Selector`, whose selector reads every position as the wildcard; read a position with `Subtree`.
0 commit comments