Skip to content

Commit 7f60d98

Browse files
committed
docs(go): show Match in the references and the JSON example
The stashgen README's "Encrypted JSON documents" table gains a Match row with both SQL forms, and a section on the grammar, the refusals and what Diagnostic's Reason, Construct and At give, the four places the answer parts from jsonb_path_match, the 22023 bound, and the index the per-operator form engages that json_tree_match does not. encrypt/README.md and doc.go say the same in short. The typed-document example finds Bea by one predicate that combines an ordering with the visit filter. The Go module's CHANGELOG records Match, the Diagnostic accessors and the redacted EQL types (#1443).
1 parent df6bb14 commit 7f60d98

6 files changed

Lines changed: 186 additions & 3 deletions

File tree

‎languages/golang/CHANGELOG.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,37 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
1212

1313
### Added
1414

15+
- **Bind a whole jsonpath filter on a `JsonSearch` field as one parameter
16+
(#1443).** Every `JsonSearch` field gets
17+
`Fields.<Field>.Match(ctx, cipher, predicate, vars, opts)`, which takes
18+
what plaintext `jsonb_path_match(col, $1, $2)` takes: a jsonpath predicate
19+
and its variables (any value `encoding/json` writes as an object, a
20+
`json.RawMessage` of one, or `nil` for none). It returns an
21+
`eql.JSONSearchPredicateQuery` for
22+
`col @@ $1::eql_v4.query_json_predicate` or
23+
`eql_v4.json_tree_match(col, $1::eql_v4.query_json_predicate)`. The engine
24+
parses the predicate and compiles it into a tree of the operands `EqAt`,
25+
`Selector`, `At.<Path>` and `Elements.<Array>.Match` derive; the Go side
26+
reads none of it. The grammar is paths of `.key`, `."quoted key"` and
27+
`[*]`; `==`, `!=`, `<`, `<=`, `>`, `>=` against a variable or a JSON
28+
literal; `exists(path)` and `exists(path ? (@.a == $x && @.b > $y))`; and
29+
`&&`, `||`, `!(…)`. A construct it cannot answer is refused as an
30+
`*encrypt.Diagnostic` wrapping `encrypt.ErrEncoding`, never compiled to a
31+
tree that drops a leaf; an ordering at a path the tag does not order is
32+
`encrypt.ErrTerm`. Each row answers as `jsonb_path_match` does over its
33+
plaintext, save four documented cases (lax array unwrapping and wrapping,
34+
a string compared with a number, and folded ordering terms). The tree is
35+
evaluated inside a function, so no index engages: `EqAt`, `Contains` and
36+
`At.<Path>` stay the indexed form. The embedded EQL guest names the
37+
`eql_v4.query_json_predicate` domain, so `Checker.Operators` lists `match`
38+
(input `predicate`) on every `JsonSearch` field, and the guest reports a
39+
refused predicate as a malformed input rather than an internal failure.
40+
41+
- **`Diagnostic.Construct` and `Diagnostic.At` (#1443).** A refused `Match`
42+
predicate names the construct in the grammar's words (`` `like_regex` ``,
43+
`a position (`[2]`)`) and the byte offset in the predicate where it
44+
starts. Neither carries text from the predicate.
45+
1546
- **Test several members of one array element of a `JsonSearch` field
1647
(#1442).** A field whose `ordered=` paths name members beneath an array's
1748
wildcard (`$.contacts[*].relationship;$.contacts[*].calls`) gets
@@ -148,6 +179,18 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
148179

149180
### Breaking
150181

182+
- **Every `encrypt/eql` type prints and logs redacted (#1443).** `String`,
183+
`Format` (every verb, `%#v` included) and `LogValue` write the type's
184+
name and `[redacted]` (`eql.TextEq{[redacted]}`, `eql.TextEq{}` when
185+
empty), never the bytes: a stored value holds ciphertext and terms, and a
186+
query value terms. Before, each type was a bare `[]byte` to `fmt` and
187+
`slog`: `%s` and `%q` gave its JSON, and `fmt.Sprint(v)`, `%v` and `%x`
188+
its bytes as numbers or hex. They now give the redacted text, with no
189+
error. A caller that read the JSON with `%s` converts the value instead, `string(v)` or `json.RawMessage(v)`, or
190+
calls `v.MarshalJSON()`; `Value` is unchanged too. There is no named
191+
accessor method for the JSON: the value is the JSON. Generated by
192+
eql-codegen for every type.
193+
151194
- **`eql.SteVecDocument` and `eql.SteVecQuery` are gone.** They named
152195
EQL v4's SteVec domains, which the engine never produced and the v4
153196
bundle no longer carries: in v4, `public.eql_v4_json_search` is the

‎languages/golang/cmd/stashgen/README.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -179,6 +179,7 @@ The field's `Fields` entry has a method per operation:
179179
| `Subtree(ctx, c, path, opts)` | `encrypt.JSONRead`: `Tokens()`, `Open` and `OpenOne` | `eql_v4.json_tree_subtree(col, $1::text[], 64)` |
180180
| `Children(ctx, c, path)`, `ChildCount(ctx, c, path)` | `encrypt.JSONToken` | `eql_v4.json_tree_children(col, $1::text)`, `eql_v4.json_tree_child_count(col, $1::text)` |
181181
| `Elements.Visits.Match(ctx, c, predicates, opts)` | `eql.JSONSearchElementsQuery` | `eql_v4.json_tree_element_match(col, $1::eql_v4.query_json_elements)` |
182+
| `Match(ctx, c, predicate, vars, opts)` | `eql.JSONSearchPredicateQuery` | `col @@ $1::eql_v4.query_json_predicate`, or `eql_v4.json_tree_match(col, $1::eql_v4.query_json_predicate)` |
182183

183184
`At` has one accessor per ordered path, named for its keys (`$.visits[*].at` is `At.VisitsAt`, `$.a.1` is `At.A1`); two paths with one name, or one path given twice, stop `stashgen`.
184185
An ordering compares the value `->` extracts at its path, and EQL's `<` compares ordering terms alone, so take the selector from the same accessor (`At.Age.Selector`) rather than spelling the path again.
@@ -220,6 +221,64 @@ Each comparison is of the member's ordering term, so `MemberEq` is an equality o
220221
`EqAt` at the member's full path is exact, of any element.
221222
`Match` reads each array to `opts.WildcardBound` elements, 64 when it is 0, and a row whose array is longer fails the query with SQLSTATE 22023 rather than answer over part of it: set the bound past the longest array.
222223
The function runs on each row a scan gives it, so narrow the rows first with an indexed predicate, such as `@>` on the same values.
224+
225+
`Match` on the field itself takes a whole filter as one parameter: a jsonpath predicate and its variables, the two things plaintext `jsonb_path_match(col, $1, $2)` takes.
226+
It is generated on every `JsonSearch` field, ordered paths or none: the engine reads the predicate when `Match` is called, and refuses what the field cannot answer then.
227+
228+
```go
229+
q, err := entities.Fields.CustomFieldValues.Match(ctx, cipher,
230+
`$."extra"."country" == $val && $."weight" < $max`,
231+
map[string]any{"val": "nz", "max": 9.5}, encrypt.JSONReadOptions{})
232+
```
233+
234+
```sql
235+
SELECT id FROM entity WHERE custom_field_values @@ $1::eql_v4.query_json_predicate
236+
-- or, the same rows
237+
SELECT id FROM entity WHERE eql_v4.json_tree_match(custom_field_values, $1::eql_v4.query_json_predicate)
238+
```
239+
240+
`vars` are any value `encoding/json` writes as a JSON object (a map, a struct, or a `json.RawMessage` of one), or `nil` for none; a Go string is refused, as it is for a `Contains` needle.
241+
The Go side reads nothing of the predicate: the engine parses it and compiles it into a tree of the operands the methods above derive, one leaf per comparison.
242+
The grammar is the subset of SQL/JSON path that the stored terms can answer, as PostgreSQL reads it:
243+
244+
- paths: `$`, then `.key`, `."quoted key"` and `[*]` steps;
245+
- values: a variable (`$v`, `$"v"`) or a JSON literal (a string, a number, `true`, `false`, `null`);
246+
- comparisons: `==`, `!=` (or `<>`), `<`, `<=`, `>`, `>=` between a path and a value, either side first;
247+
- `exists(path)`, and `exists(path ? (@.m == $v && @.n > $w))`, conditions on `@` or `@.member` joined by `&&`;
248+
- `&&`, `||`, `!(…)`, parentheses and `/* */` comments. Keywords are read in any case.
249+
250+
An ordering (`<` and the rest) needs its path in `ordered=`, and so does every member of a filter of more than one condition, which compiles to the element match above.
251+
Anything else is refused before any term is derived, as an `*encrypt.Diagnostic` wrapping `encrypt.ErrEncoding`.
252+
`Construct()` names the construct in the grammar's own words (`` `like_regex` ``, `a position (`[2]`)`), and `At()` is the byte offset in the predicate where it starts; neither says anything of the predicate's text or the variables' values.
253+
`Reason()` says why:
254+
255+
- `json_predicate_syntax`: text the grammar does not read.
256+
- `json_predicate_unsupported`: a jsonpath construct the stored terms cannot answer, such as `like_regex`, arithmetic, a method, `.**`, or a filter outside `exists`.
257+
- `json_predicate_unknown_variable`: a `$name` that `vars` does not hold.
258+
- `json_predicate_value_type`: an object or array as a value, or anything but a number or a string under an ordering or in a filter of more than one condition.
259+
- `json_predicate_not_equal_at_wildcard`: `!=` at a path with `[*]`.
260+
- `json_predicate_too_deep`: more than 32 levels of nesting, or a path deeper than a document.
261+
- `json_path_position`: a position (`[2]`) in a path.
262+
- `json_path_no_wildcard`: a filter on a path with no `[*]`.
263+
- `json_path_wildcard`: a `[*]` in a filter's member.
264+
- `json_syntax`: JSON text that does not parse, a literal in the predicate (`1e999`) or `vars` (`{`, or an empty or whitespace-only `json.RawMessage`); with `At()` for a literal, without it for `vars`.
265+
- `json_input`: `vars` that are not a JSON object (`[1]`, `1`).
266+
267+
An ordering at a path the tag does not order is `encrypt.ErrTerm` with the code `stack_encrypt::json_path_not_ordered`.
268+
269+
Each row answers as `jsonb_path_match` answers over its plaintext, true, false or NULL, save four cases:
270+
271+
- Lax mode's unwrapping of an array the path names no `[*]` for is not applied: `$.tags == "a"` over `{"tags": ["a"]}` is false here. Write `[*]`.
272+
- Nor is its wrapping of a value that is not an array under `[*]`: `$.tags[*] == "a"` over `{"tags": "a"}` is false here. Keep an array an array.
273+
- A string compared with a number, which jsonpath calls unknown, is false under `==` and true under `!=`. Keep one kind of value at a path.
274+
- Orderings, and filters of more than one condition, compare the folded ordering terms: text case- and accent-folded, numbers as float64. So `exists($.c[*] ? (@.r == "sibling" && @.n > 5))` finds a `"Sibling"`; `==` anywhere else is exact.
275+
276+
A filter of more than one condition reads each array to `opts.WildcardBound` elements, 64 when it is 0; a row whose array is longer fails the query with SQLSTATE 22023, as the element match does.
277+
278+
`Match` costs an index: `eql_v4.json_tree_match` evaluates the tree inside a function, so it filters every row a scan gives it and no index engages.
279+
The per-operator form engages the GIN index for `@>` (`EqAt`, `Contains`) and a btree on an ordered path's term (`At.<Path>`).
280+
Where a query needs an index, write those leaves with the per-operator methods, or put one beside `Match` to narrow the rows: `WHERE col @> $1::eql_v4.query_json AND col @@ $2::eql_v4.query_json_predicate`.
281+
223282
`OpenOne` reads one value and refuses a read whose path has a wildcard before anything is opened; open a wildcard read with `Open`.
224283
A needle and a value are Go values: a `json.RawMessage` of JSON text crosses as it is, and anything else is marshalled with `encoding/json`.
225284
`EqAt` and the orderings take a scalar: a string, a number, a bool, or `nil` for JSON `null`.

‎languages/golang/encrypt/README.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -678,6 +678,36 @@ asked, naming it. A row whose array is longer than
678678
`JSONReadOptions.WildcardBound` (64 by default) fails the query with
679679
SQLSTATE 22023.
680680

681+
A whole filter binds as one parameter with the field's `Match`: a jsonpath
682+
predicate and its variables, as plaintext `jsonb_path_match(col, $1, $2)`
683+
takes them. The engine reads the predicate and compiles it into a tree of
684+
the operands above; the Go side reads none of it.
685+
686+
```go
687+
q, err := users.Fields.Profile.Match(ctx, cipher,
688+
`$."plan" == $plan && $."age" < $max`,
689+
map[string]any{"plan": "pro", "max": 40}, encrypt.JSONReadOptions{})
690+
rows, err := db.QueryContext(ctx, `SELECT id FROM users WHERE profile @@ $1::eql_v4.query_json_predicate`, q)
691+
```
692+
693+
`eql_v4.json_tree_match(profile, $1::eql_v4.query_json_predicate)` returns
694+
the same rows. The variables are any value `encoding/json` writes as an
695+
object, or `nil` for none. The grammar is paths of `.key`, `."key"` and
696+
`[*]`; `==`, `!=`, `<`, `<=`, `>`, `>=` with a variable or a JSON literal;
697+
`exists(path)` and `exists(path ? (@.a == $x && @.b > $y))`; `&&`, `||` and
698+
`!(…)`. An ordering, and each member of a filter of more than one condition,
699+
needs its path in `ordered=`. Anything else is refused as a `*Diagnostic`
700+
wrapping `ErrEncoding`, whose `Reason()`, `Construct()` and `At()` give the
701+
reason, the construct in the grammar's words and the byte offset in the
702+
predicate, never its text or the values. Each row answers as
703+
`jsonb_path_match` does over its plaintext, save four cases the stashgen
704+
README lists: lax unwrapping and wrapping of arrays are not applied, a
705+
string compared with a number is false under `==`, and orderings and
706+
multi-condition filters compare folded terms. The tree is evaluated inside a
707+
function, so no index engages: where a query needs one, write its leaves
708+
with `EqAt`, `Contains` and `At.<Path>`, whose `@>` and `->` comparisons
709+
use the GIN and btree indexes, or put one beside `Match` to narrow the rows.
710+
681711
## Building
682712

683713
The package embeds `wasm/stack_encrypt_guest.wasm`, a build artefact of the

‎languages/golang/encrypt/doc.go‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -258,6 +258,22 @@
258258
// array is longer than [JSONReadOptions.WildcardBound] (64 by default)
259259
// fails the query with SQLSTATE 22023.
260260
//
261+
// A whole filter binds as one parameter with the field's Match: a jsonpath
262+
// predicate and its variables, as plaintext jsonb_path_match takes them.
263+
// The engine parses the predicate and compiles it into a tree of the
264+
// operands above; nothing in Go reads it.
265+
//
266+
// q, err := customers.Fields.Profile.Match(ctx, cipher,
267+
// `$."name" == $name && $."age" < $max`,
268+
// map[string]any{"name": "Alice", "max": 40}, encrypt.JSONReadOptions{})
269+
// // WHERE profile @@ $1::eql_v4.query_json_predicate
270+
//
271+
// A predicate the engine cannot answer is refused as a [Diagnostic]
272+
// wrapping [ErrEncoding], whose Reason, Construct and At name the refusal
273+
// and the byte offset in the predicate where it starts. The tree is
274+
// evaluated inside eql_v4.json_tree_match, so no index engages: a query
275+
// that needs one writes its leaves with EqAt, Contains and At.<Path>.
276+
//
261277
// # Cancellation
262278
//
263279
// Every call takes a [context.Context], and a call whose context is

‎languages/golang/encrypt/example/json/README.md‎

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,8 @@
22

33
This program stores a Go `Profile` struct as one encrypted EQL v4 JSON column.
44
It finds a row by containment, compares an ordered path, finds a row where one
5-
visit passes two tests, decrypts a whole profile, and opens only the visit
6-
times from another query.
5+
visit passes two tests, finds a row by one jsonpath predicate, decrypts a whole
6+
profile, and opens only the visit times from another query.
77

88
## Run
99

@@ -51,7 +51,23 @@ q, err := Fields.Profile.Elements.Visits.Match(ctx, cipher, encrypt.ElementPredi
5151
```
5252

5353
Ada has a visit at home and a visit after 15, but no one visit that is both,
54-
so only Bea's row matches. Each comparison is of the ordering term, so text
54+
so only Bea's row matches.
55+
56+
`Fields.Profile.Match` takes a whole filter as one jsonpath predicate and its
57+
variables, as plaintext `jsonb_path_match` does, and binds as one parameter:
58+
59+
```go
60+
q, err := Fields.Profile.Match(ctx, cipher,
61+
`$."age" < $max && exists($."visits"[*] ? (@."place" == $place && @."at" > $after))`,
62+
map[string]any{"max": 30, "place": "home", "after": 15}, encrypt.JSONReadOptions{})
63+
// SELECT id FROM customers WHERE profile @@ $1::eql_v4.query_json_predicate
64+
```
65+
66+
The engine compiles the predicate into the operands the other methods give:
67+
the age comparison is `At.Age`'s, and the filter is the visit match above.
68+
It refuses a construct it cannot answer, naming it and its byte offset in the
69+
predicate. The tree is evaluated inside a function, so no index engages; the
70+
per-operator queries are the fast path. Each comparison is of the ordering term, so text
5571
compares case- and accent-folded. A row with more visits than
5672
`JSONReadOptions.WildcardBound` (64 by default) fails the query with SQLSTATE
5773
22023. A containment needle made from a

‎languages/golang/encrypt/example/json/main.go‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,25 @@ func run(ctx context.Context) error {
121121
}
122122
fmt.Printf("customer with a visit at home after 15: %d\n", homeLateID)
123123

124+
// The same test and an age ordering as one jsonpath predicate and its
125+
// variables, bound as one parameter. The engine compiles the predicate
126+
// into the operands above, so the age is the ordering's term and the
127+
// filter is the visit match. The tree is evaluated in a function, which
128+
// engages no index: the per-operator queries above are the fast path.
129+
predicate, err := Fields.Profile.Match(ctx, cipher,
130+
`$."age" < $max && exists($."visits"[*] ? (@."place" == $place && @."at" > $after))`,
131+
map[string]any{"max": 30, "place": "home", "after": 15}, encrypt.JSONReadOptions{})
132+
if err != nil {
133+
return fmt.Errorf("prepare predicate: %w", err)
134+
}
135+
var matchedID int64
136+
if err := conn.QueryRowContext(ctx,
137+
`SELECT id FROM customers WHERE profile @@ $1::eql_v4.query_json_predicate`, predicate,
138+
).Scan(&matchedID); err != nil {
139+
return fmt.Errorf("find by predicate: %w", err)
140+
}
141+
fmt.Printf("customer under 30 with a visit at home after 15: %d\n", matchedID)
142+
124143
// Read only the visit times of one row. The database returns encrypted
125144
// entries; Open checks them against the prepared read and returns JSON text.
126145
read, err := Fields.Profile.Subtree(ctx, cipher, "$.visits[*].at", encrypt.JSONReadOptions{})

0 commit comments

Comments
 (0)