Skip to content

Commit 586a67e

Browse files
committed
feat(go): test several members of one array element of a JsonSearch field
A filter such as "a contact who is a sibling and was called more than five times" had no Go method. An EqAt and an ordering at the contacts' paths can each be answered by a different contact, so the application had to read the array back and re-apply the predicate itself. The engine and EQL already answer it: element_match derives one eql_v4.query_json_elements operand, and json_tree_element_match is true of a row where one element passes every predicate. This wires it into the Go SDK. The EQL guest's resolver now names the elements domain, so the engine lists element_match (input path_predicates) on a field that orders a path, and the checker and stashgen read the operator. The compiled predicate domain stays unnamed: match is #1443's. stashgen groups the tag's ordered paths by the array their last wildcard names (record.TargetOptions.ElementArrays) and writes one accessor per array under Elements, named as At names a path, with Match(ctx, cipher, opts, predicates...). An array with no ordered member beneath it gets none, and a root array ($[*]) is Root. The members are compile-time names. stashgen writes Paths (NPaths with -name), a tree nested as the JSON nests it, of encrypt.Path[E, T] values: Paths.Profile.Visits.Place. E is a type stashgen writes for each array, and T is the member's Go type, read from the document's struct as encoding/json reads it, or any where the type names none (a json.RawMessage, say). encrypt.AtPath(path) gives Eq, Lt, Lte, Gt, Gte and Between, each taking a T and returning an encrypt.ElementPredicate[E], and an array's Match takes only its own E. So a value of another type, or a predicate on another array's member, does not compile. Paths holds fields and no methods, so no JSON key is reserved. For a filter built at run time, the array accessor's Member("place") gives the same comparisons with any values. An array gets no accessor, and a notice naming its ordered paths, when its keys give no accessor name, when another array's keys give the same one, or when Paths cannot name its members: a key that gives no Go name on its own, or one name that would stand for two paths. A tag that generated before still generates, with its At unchanged. Match writes each value with gensupport.JSONText as the orderings do, and returns an eql.JSONSearchElementsQuery. The member check runs in the SDK, against the members the declaration orders beneath the array, before the guest is called: the engine's json_path_not_ordered names no member, so the first error a caller sees names the member, the array and the members it orders. It wraps ErrTerm, as the engine's refusal does; no predicates, a zero predicate, a zero Path and a member with a wildcard wrap ErrEncoding. Every other refusal is the engine's Diagnostic. A predicate prints and logs its member and comparison, never its values, and copies the bytes of a json.RawMessage or []byte value. Tests: the hermetic suite pins the operand against the subtree reads and the orderings of the same members, the run-time form against the typed one, every pre-engine refusal, and each engine refusal's sentinel and code through the guest. A type-checking test loads the generated testusers package with a probe file and holds the two compile errors, a value of another type and another array's predicate; the stashgen test does the same against a generated module, and pins the member types and every case that gets no accessor. The PostgreSQL suite (go:test:postgres) finds the row where one contact passes both predicates, by path and by name, and not the row where two contacts each pass one, which the EqAt and ordering superset returns, and raises SQLSTATE 22023 past the bound. The guest test now pins the elements domain named and match not listed. Closes #1442 Claude-Session: https://claude.ai/code/session_01MrxG73fnX1p7oiizh3GRG5
1 parent 3dfbe7f commit 586a67e

23 files changed

Lines changed: 2324 additions & 46 deletions

File tree

‎languages/golang/cmd/stashgen/main.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ func run(args []string, dir string, stdout, stderr io.Writer, newEngine func(con
5959
req := stashgen.Request{Dir: dir}
6060
var models modelFlags
6161
fs.StringVar(&req.Type, "type", "", "the struct that carries the stash tags (required)")
62-
fs.StringVar(&req.Name, "name", "", "write EncryptN, DecryptN and NFields instead of Encrypt, Decrypt and Fields")
62+
fs.StringVar(&req.Name, "name", "", "write EncryptN, DecryptN, NFields and NPaths instead of Encrypt, Decrypt, Fields and Paths")
6363
fs.StringVar(&req.For, "for", "", "P.F: -type declares the tags for F, a type in package P")
6464
fs.Var(&models, "model", "Name=R or Name=R:D: a model R for separate columns; writes EncryptName and DecryptName (repeatable)")
6565
fs.BoolVar(&req.Redact, "redact", false, "write String, GoString and LogValue methods on the -type struct")

‎languages/golang/cmd/stashgen/main_test.go‎

Lines changed: 192 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import (
44
"bytes"
55
"context"
66
"os"
7+
"os/exec"
78
"path/filepath"
89
"strings"
910
"testing"
@@ -442,6 +443,117 @@ func TestRunGeneratesJSONSearchFields(t *testing.T) {
442443
if !strings.Contains(written, "func (p ProfileFieldPath) Selector(ctx context.Context, c *encrypt.Cipher) (encrypt.JSONSelection[eql.JSONEntry], error)") {
443444
t.Error("an ordered path has no Selector of its own")
444445
}
446+
// The array with an ordered member gets an accessor under Elements,
447+
// named as At names its path, whose Match derives the element match.
448+
for _, want := range []string{
449+
"Elements ProfileFieldElements",
450+
"Visits ProfileFieldArray[ProfileFieldElementsVisits] // $.visits[*]: at",
451+
`Visits: ProfileFieldArray[ProfileFieldElementsVisits]{field: f, path: "$.visits[*]"},`,
452+
"type ProfileFieldElementsVisits struct{}",
453+
"type ProfileFieldArray[E any] struct {",
454+
"func (a ProfileFieldArray[E]) Match(ctx context.Context, c *encrypt.Cipher, opts encrypt.JSONReadOptions, predicates ...encrypt.ElementPredicate[E]) (eql.JSONSearchElementsQuery, error)",
455+
"gensupport.JSONElementMatch[eql.JSONSearchElementsQuery](ctx, a.field, c, a.path, opts, predicates)",
456+
"func (a ProfileFieldArray[E]) Member(member string) encrypt.MemberTest[E, any]",
457+
"return encrypt.AtPath(encrypt.NewPath[E, any](a.path, member))",
458+
// Paths names the array's members, nested as the JSON nests them;
459+
// a json.RawMessage document names no Go type, so each is any.
460+
"var Paths = func() (paths struct {",
461+
"At encrypt.Path[ProfileFieldElementsVisits, any] // $.visits[*].at",
462+
`paths.Profile.Visits.At = encrypt.NewPath[ProfileFieldElementsVisits, any]("$.visits[*]", "at")`,
463+
} {
464+
if !strings.Contains(written, want) {
465+
t.Errorf("the generated file lacks %q", want)
466+
}
467+
}
468+
if strings.Contains(written, "PrefsFieldElements") {
469+
t.Error("Prefs orders no member, and got Elements")
470+
}
471+
// The members of one array, named as At names their paths; an ordered
472+
// path that ends at its wildcard, or has none, is no array's member.
473+
written, stderr, code = generate("\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=$.extra.emergency_contacts[*].relationship;$.extra.emergency_contacts[*].previous_call_count;$.tags[*];$.age)\"`")
474+
if code != 0 {
475+
t.Fatalf("exit %d\n%s", code, stderr)
476+
}
477+
// gofmt aligns the struct fields: read each line's words.
478+
words := strings.Join(strings.Fields(written), " ")
479+
for _, want := range []string{
480+
"ExtraEmergencyContactsRelationship ProfileFieldPath // $.extra.emergency_contacts[*].relationship",
481+
"ExtraEmergencyContacts ProfileFieldArray[ProfileFieldElementsExtraEmergencyContacts] // $.extra.emergency_contacts[*]: relationship, previous_call_count",
482+
"Profile struct { Extra struct { EmergencyContacts struct { Relationship encrypt.Path[ProfileFieldElementsExtraEmergencyContacts, any] // $.extra.emergency_contacts[*].relationship PreviousCallCount encrypt.Path[ProfileFieldElementsExtraEmergencyContacts, any] // $.extra.emergency_contacts[*].previous_call_count } } }",
483+
`paths.Profile.Extra.EmergencyContacts.PreviousCallCount = encrypt.NewPath[ProfileFieldElementsExtraEmergencyContacts, any]("$.extra.emergency_contacts[*]", "previous_call_count")`,
484+
} {
485+
if !strings.Contains(words, want) {
486+
t.Errorf("the generated file lacks %q", want)
487+
}
488+
}
489+
if strings.Contains(written, "Tags ProfileFieldArray") {
490+
t.Error("$.tags[*] orders the elements themselves, no member, and got an accessor under Elements")
491+
}
492+
// A field that orders no member beneath an array has no Elements.
493+
written, stderr, code = generate("\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=$.age;$.tags[*])\"`")
494+
if code != 0 {
495+
t.Fatalf("exit %d\n%s", code, stderr)
496+
}
497+
if strings.Contains(written, "Elements") || strings.Contains(written, "JSONElementMatch") || strings.Contains(written, "Paths") {
498+
t.Error("a field with no ordered member beneath an array got Elements or Paths")
499+
}
500+
// A root array's accessor is Root, beside its paths' At accessors.
501+
written, stderr, code = generate("\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=$[*].name;$[*].age)\"`")
502+
if code != 0 {
503+
t.Fatalf("a root array: exit %d\n%s", code, stderr)
504+
}
505+
words = strings.Join(strings.Fields(written), " ")
506+
for _, want := range []string{
507+
"Name ProfileFieldPath // $[*].name",
508+
"Root ProfileFieldArray[ProfileFieldElementsRoot] // $[*]: name, age",
509+
`Root: ProfileFieldArray[ProfileFieldElementsRoot]{field: f, path: "$[*]"},`,
510+
// A root array's members are the field's own in Paths.
511+
"Profile struct { Name encrypt.Path[ProfileFieldElementsRoot, any] // $[*].name Age encrypt.Path[ProfileFieldElementsRoot, any] // $[*].age }",
512+
} {
513+
if !strings.Contains(words, want) {
514+
t.Errorf("a root array: the generated file lacks %q", want)
515+
}
516+
}
517+
// Arrays whose keys give one accessor name, or none, get no accessor:
518+
// the tag still generates, At is as it is without them, and a notice
519+
// in the file and on stderr names the paths as the tag writes them.
520+
for _, c := range []struct{ name, ordered, at, notice string }{
521+
{"two arrays, one name", "$.a_b[*].x;$.aB[*].y;$.c[*].z", "ABX ProfileFieldPath // $.a_b[*].x", "no Elements accessor for the elements of $.a_b[*] (ordered $.a_b[*].x): another array's keys give the same accessor name, Elements.AB"},
522+
{"an array with no name", "$.__[*].x;$.c[*].z", "X ProfileFieldPath // $.__[*].x", "no Elements accessor for the elements of $.__[*] (ordered $.__[*].x): its keys give no Go accessor name"},
523+
// Paths names each key on its own, so a key of digits, which At
524+
// joins to the keys before it, names nothing there.
525+
{"a key with no name under Paths", "$.a.1[*].x;$.c[*].z", "A1X ProfileFieldPath // $.a.1[*].x", `no Elements accessor for the elements of $.a.1[*] (ordered $.a.1[*].x): the key "1" of $.a.1[*].x gives no Go name under Paths that starts with a letter`},
526+
// Paths nests: a member that is also the object or array above
527+
// another member, or two keys that give one name at one place,
528+
// would make one name stand for two paths.
529+
{"a member above a member", "$.a[*].b;$.a[*].b.c;$.c[*].z", "ABC ProfileFieldPath // $.a[*].b.c", "no Elements accessor for the elements of $.a[*] (ordered $.a[*].b, $.a[*].b.c): Paths.Profile.A.B would name both $.a[*].b and $.a[*].b.c, and can name one of them only"},
530+
{"a member above an array", "$.a[*].b;$.a[*].b[*].d;$.c[*].z", "ABD ProfileFieldPath // $.a[*].b[*].d", "no Elements accessor for the elements of $.a[*].b[*] (ordered $.a[*].b[*].d): Paths.Profile.A.B would name both $.a[*].b and $.a[*].b[*].d, and can name one of them only"},
531+
{"two keys, one name in Paths", "$.x.a_b[*].m;$.x.aB.n[*].o;$.c[*].z", "XABM ProfileFieldPath // $.x.a_b[*].m", "no Elements accessor for the elements of $.x.aB.n[*] (ordered $.x.aB.n[*].o): Paths.Profile.X.AB would name both $.x.a_b[*].m and $.x.aB.n[*].o, and can name one of them only"},
532+
} {
533+
written, stderr, code := generate("\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=" + c.ordered + ")\"`")
534+
if code != 0 {
535+
t.Fatalf("%s: exit %d\n%s", c.name, code, stderr)
536+
}
537+
// A wrapped comment reads as one line.
538+
words := strings.Join(strings.Fields(strings.ReplaceAll(written, "\n// ", " ")), " ")
539+
for _, want := range []string{c.at, "C ProfileFieldArray[ProfileFieldElementsC] // $.c[*]: z", "Z encrypt.Path[ProfileFieldElementsC, any] // $.c[*].z", "Profile: " + c.notice} {
540+
if !strings.Contains(words, want) {
541+
t.Errorf("%s: the generated file lacks %q", c.name, want)
542+
}
543+
}
544+
if !strings.Contains(stderr, "stashgen: Customer.Profile: "+c.notice) {
545+
t.Errorf("%s: stderr = %q", c.name, stderr)
546+
}
547+
// $.c[*] alone has an accessor: its field, its initializer and
548+
// its Elements struct field.
549+
if n := strings.Count(written, "ProfileFieldArray[ProfileFieldElements"); n != 2 {
550+
t.Errorf("%s: %d accessors written, want $.c[*]'s alone", c.name, n)
551+
}
552+
}
553+
written, _, code = generate("\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=$.a_b[*].x;$.aB[*].y)\"`")
554+
if code != 0 || strings.Contains(written, "ProfileFieldElements") || strings.Contains(written, "Paths") || !strings.Contains(written, "ABY ProfileFieldPath // $.aB[*].y") {
555+
t.Errorf("two colliding arrays and no other: exit %d, Elements or Paths written, or At changed", code)
556+
}
445557
// A json.RawMessage sealed any other way is bytes, as it was before
446558
// documents; an object key of digits names its accessor.
447559
written, stderr, code = generate("\tData json.RawMessage `stash:\"data,encrypt,index=equality\"`\n\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=$.a.1)\"`")
@@ -473,6 +585,19 @@ func TestRunGeneratesJSONSearchFields(t *testing.T) {
473585
if _, err := os.Stat(filepath.Join(dir, "customer_stash.go")); !os.IsNotExist(err) {
474586
t.Fatal("a file was written after the refusal")
475587
}
588+
// The same for an array whose elements have an ordered member, and an
589+
// engine that answers no element_match.
590+
dir = writeModule(t, map[string]string{"model.go": source("\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=$.visits[*].at)\"`")})
591+
stderrBuf.Reset()
592+
if code := run([]string{"-type", "Customer"}, dir, &bytes.Buffer{}, &stderrBuf, withoutOperator("element_match")); code != 1 {
593+
t.Fatalf("exit %d, want 1\n%s", code, stderrBuf.String())
594+
}
595+
if got := stderrBuf.String(); !strings.Contains(got, "Customer.Profile") || !strings.Contains(got, "the engine answers no element_match") {
596+
t.Fatalf("stderr = %q", got)
597+
}
598+
if _, err := os.Stat(filepath.Join(dir, "customer_stash.go")); !os.IsNotExist(err) {
599+
t.Fatal("a file was written after the refusal")
600+
}
476601
for _, c := range []struct{ name, fields, want string }{
477602
{"a sealed null", "\tPrefs *json.RawMessage `stash:\"prefs,encrypt_into=JsonSearch,null=sealed\"`", "Customer.Prefs: a nil has an equality term and no other, so another index's column would show which values are absent; write null=sql"},
478603
{"a nil document that does not say null=sql", "\tPrefs *json.RawMessage `stash:\"prefs,encrypt_into=JsonSearch\"`", "null=sql"},
@@ -499,6 +624,73 @@ func TestRunGeneratesJSONSearchFields(t *testing.T) {
499624
}
500625
}
501626

627+
// A document of a Go type names each member's Go type in Paths, read as
628+
// encoding/json reads the document: through pointers, a promoted field, a
629+
// map and a slice, with any where the type names none. The compiler then
630+
// refuses a value of another type, and a predicate on another array's
631+
// member, in a call to Match.
632+
func TestRunTypesTheElementMatch(t *testing.T) {
633+
checker, err := encrypt.NewChecker(context.Background())
634+
if err != nil {
635+
t.Skip(err)
636+
}
637+
_ = checker.Close()
638+
model := "package users\n\nimport \"time\"\n\ntype Place string\n\ntype Stamp struct {\n\tWhen time.Time `json:\"when\"`\n}\n\ntype Visit struct {\n\tStamp\n\tAt *int64 `json:\"at\"`\n\tPlace Place `json:\"place\"`\n\tNote string\n}\n\ntype Tag struct {\n\tName string `json:\"name\"`\n}\n\ntype Profile struct {\n\tVisits []Visit `json:\"visits\"`\n\tTags map[string][]Tag `json:\"tags\"`\n}\n\ntype Customer struct {\n\t_ struct{} `stash:\"context=customers\"`\n\tID int64 `stash:\"id,passthrough\"`\n\tProfile Profile `stash:\"profile,encrypt_into=JsonSearch(ordered=$.visits[*].at;$.visits[*].place;$.visits[*].when;$.visits[*].Note;$.visits[*].missing;$.tags.home[*].name)\"`\n}\n"
639+
dir := writeModule(t, map[string]string{"model.go": model})
640+
var stdout, stderr bytes.Buffer
641+
if code := run([]string{"-type", "Customer"}, dir, &stdout, &stderr, stashgen.GuestEngine); code != 0 {
642+
t.Fatalf("exit %d\n%s", code, stderr.String())
643+
}
644+
data, err := os.ReadFile(filepath.Join(dir, "customer_stash.go"))
645+
if err != nil {
646+
t.Fatal(err)
647+
}
648+
words := strings.Join(strings.Fields(string(data)), " ")
649+
for _, want := range []string{
650+
"At encrypt.Path[ProfileFieldElementsVisits, int64] // $.visits[*].at",
651+
"Place encrypt.Path[ProfileFieldElementsVisits, Place] // $.visits[*].place",
652+
"When encrypt.Path[ProfileFieldElementsVisits, time.Time] // $.visits[*].when",
653+
"Note encrypt.Path[ProfileFieldElementsVisits, string] // $.visits[*].Note",
654+
"Missing encrypt.Path[ProfileFieldElementsVisits, any] // $.visits[*].missing",
655+
"Tags struct { Home struct { Name encrypt.Path[ProfileFieldElementsTagsHome, string] // $.tags.home[*].name } }",
656+
`paths.Profile.Visits.At = encrypt.NewPath[ProfileFieldElementsVisits, int64]("$.visits[*]", "at")`,
657+
`paths.Profile.Tags.Home.Name = encrypt.NewPath[ProfileFieldElementsTagsHome, string]("$.tags.home[*]", "name")`,
658+
} {
659+
if !strings.Contains(words, want) {
660+
t.Errorf("the generated file lacks %q", want)
661+
}
662+
}
663+
// What a program writes compiles, and each mistake the types exist to
664+
// catch does not.
665+
build := func(probe string) string {
666+
t.Helper()
667+
src := "package users\n\nimport (\n\t\"context\"\n\n\t\"github.com/cipherstash/stack/languages/golang/encrypt\"\n)\n\nfunc probe(ctx context.Context, c *encrypt.Cipher) {\n" + probe + "}\n"
668+
if err := os.WriteFile(filepath.Join(dir, "probe.go"), []byte(src), 0o600); err != nil {
669+
t.Fatal(err)
670+
}
671+
cmd := exec.Command("go", "build", "./...")
672+
cmd.Dir = dir
673+
cmd.Env = append(os.Environ(), "GOPROXY=off", "GOWORK=off", "GOFLAGS=-mod=mod", "CGO_ENABLED=0")
674+
out, err := cmd.CombinedOutput()
675+
if err == nil {
676+
return ""
677+
}
678+
return string(out)
679+
}
680+
if out := build("\t_, _ = Fields.Profile.Elements.Visits.Match(ctx, c, encrypt.JSONReadOptions{},\n\t\tencrypt.AtPath(Paths.Profile.Visits.Place).Eq(\"home\"),\n\t\tencrypt.AtPath(Paths.Profile.Visits.At).Between(1, 2),\n\t\tFields.Profile.Elements.Visits.Member(\"at\").Gt(\"any value\"),\n\t)\n"); out != "" {
681+
t.Fatalf("the element match as a program writes it does not build:\n%s", out)
682+
}
683+
out := build("\t_, _ = Fields.Profile.Elements.Visits.Match(ctx, c, encrypt.JSONReadOptions{}, encrypt.AtPath(Paths.Profile.Visits.At).Gt(\"15\"))\n\t_, _ = Fields.Profile.Elements.Visits.Match(ctx, c, encrypt.JSONReadOptions{}, encrypt.AtPath(Paths.Profile.Tags.Home.Name).Eq(\"x\"))\n")
684+
for _, want := range []string{
685+
`cannot use "15" (untyped string constant) as int64 value`,
686+
"encrypt.ElementPredicate[ProfileFieldElementsTagsHome]) as encrypt.ElementPredicate[ProfileFieldElementsVisits] value",
687+
} {
688+
if !strings.Contains(out, want) {
689+
t.Errorf("the build does not refuse with %q:\n%s", want, out)
690+
}
691+
}
692+
}
693+
502694
// withoutOperator is the embedded engine with op removed from every field's
503695
// answer: an engine that disagrees with the generator.
504696
func withoutOperator(op string) func(context.Context) (stashgen.Engine, error) {

‎languages/golang/encrypt/checker.go‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -258,7 +258,7 @@ func parseOperators(value any) ([]record.Operator, error) {
258258
op.Index, err = parseOperatorIndex(s)
259259
case "input":
260260
switch s {
261-
case "value", "pair", "json", "path_json", "path_pair", "path", "path_bound":
261+
case "value", "pair", "json", "path_json", "path_pair", "path", "path_bound", "path_predicates":
262262
op.Input = s
263263
default:
264264
return nil, fmt.Errorf("operator %d: unknown input %q", i, s)
@@ -282,14 +282,14 @@ func parseOperators(value any) ([]record.Operator, error) {
282282
// of the names the engine's OpSpec::parse accepts, the scalar operators
283283
// (eq, lt, lte, gt, gte, between, matches) and a document's (contains,
284284
// contained_by, eq_at, the orderings at a path, between_at, selector,
285-
// subtree, children, child_count). The catalog (se_targets) and se_query_term are
285+
// subtree, children, child_count, element_match). The catalog (se_targets) and se_query_term are
286286
// read through this, parseFidelity and parseOperatorIndex, so the two
287287
// readers refuse the same values.
288288
func parseOperatorName(s string) (string, error) {
289289
switch s {
290290
case "eq", "lt", "lte", "gt", "gte", "between", "matches",
291291
"contains", "contained_by", "eq_at", "lt_at", "lte_at", "gt_at", "gte_at", "between_at",
292-
"selector", "subtree", "children", "child_count":
292+
"selector", "subtree", "children", "child_count", "element_match":
293293
return s, nil
294294
}
295295
return "", fmt.Errorf("unknown operator %q", s)

0 commit comments

Comments
 (0)