You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit faa4a41
Browse filesBrowse the repository at this point in the historyBrowse files
refactor(evals): fixtures are spaces with parsed nodes
A suite's fixtures.json5 lists fixtures as space configs (name, path to a JSONL nodes file, schema, plugins), so fixtures share schemas the way spaces do. A nodes file holds parsed nodes as `sctx dump` prints them, with their own ids and link targets; the harness no longer derives either from a title. Every eval names a space: a fixture or a configured space, labelled by node id the same way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019XphpdrUMvqf8fuj8uG7JZ
Copy file name to clipboardExpand all lines: evals/README.md
+14-14Lines changed: 14 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# Semantic evals
2
2
3
-
Semantic evals measure how well model providers judge semantic rules. A case is a node, a rule, and the verdict the node should get. An eval holds the cases on one set of nodes: a fixture, or a configured space.
3
+
Semantic evals measure how well model providers judge semantic rules. A case is a node, a rule, and the verdict the node should get. An eval holds the cases on the nodes of one space. That space is a fixture, which keeps its nodes in the suite, or a configured space.
4
4
5
5
## Run the evals
6
6
@@ -38,13 +38,13 @@ Each run validates through the same pipeline as `sctx validate --semantic`, with
38
38
39
39
## Suites
40
40
41
-
A suite is a directory of evals, with their fixtures in `fixtures/`:
41
+
A suite is a directory of evals and the fixtures they use:
42
42
43
43
```
44
44
evals/semantic/
45
-
opportunity-customer-voice.json5
46
-
fixtures/
47
-
strict-ost.json5
45
+
opportunity-customer-voice.json5 an eval
46
+
fixtures.json5 the fixtures, as space configs
47
+
fixtures/strict-ost.jsonl a fixture's nodes
48
48
```
49
49
50
50
|`--from`| Directory | Schema names resolve from |
@@ -55,30 +55,30 @@ evals/semantic/
55
55
56
56
## Add an eval
57
57
58
-
An eval is a `.json5` file in the suite. It names a `fixture` in the suite, or a configured `space`. `expected` maps a rule id to node ids and their verdicts:
58
+
An eval is a `.json5` file in the suite. It names a `space`: one of the suite's fixtures, or else a space in the config. `expected` maps a rule id to node ids and their verdicts:
59
59
60
60
```json5
61
61
{
62
-
fixture:'strict-ost',
62
+
space:'strict-ost',
63
63
expected: {
64
64
'opportunity-customer-voice': {
65
-
'Build a Slack integration':'fail',
65
+
'Build a Slack integration.md':'fail',
66
66
// A solution in disguise.
67
-
'Add a progress bar to workspace setup':'fail',
67
+
'Add a progress bar to workspace setup.md':'fail',
68
68
},
69
69
},
70
70
}
71
71
```
72
72
73
-
A list of verdicts accepts any of them. On a configured space, node ids are file paths: `{ space: 'work', expected: { 'opportunity-customer-voice': { 'Onboarding.md': 'fail' } } }`.
74
-
75
-
Write a comment on a case whose label is not obvious. We recommend one eval per rule, so each rule's cases are easy to find and to run alone.
73
+
A list of verdicts accepts any of them. Write a comment on a case whose label is not obvious. We recommend one eval per rule, so each rule's cases are easy to find and to run alone.
76
74
77
75
## Add a fixture
78
76
79
-
A fixture is `fixtures/<name>.json5`: a `schema`, optional `plugins` as in a space config, and `nodes`. Each node holds the fields its frontmatter would, with a body in `content`. Its `title` is its node id and link target. `sctx dump <space> | jq '.nodes | map(.schemaData)'` prints notes in this shape.
77
+
A fixture is a space whose nodes are in a JSONL file in the suite. `fixtures.json5` lists the fixtures as a config lists spaces: a `name`, the `path` to the nodes file, a `schema`, and optional `plugins`. Several fixtures can share a schema, and several evals can share a fixture.
78
+
79
+
The nodes file holds one parsed node per line, as `sctx dump` prints them. To make one from markdown notes, configure them as a space and run `sctx dump <space> | jq -c '.nodes[]'`.
80
80
81
-
Several evals can share a fixture. For a rule whose `scope` sends related nodes, such as siblings, give each scenario its own small tree, so that one case's related nodes do not change another's.
81
+
For a rule whose `scope` sends related nodes, such as siblings, give each scenario its own small tree, so that one case's related nodes do not change another's.
{"id":"Increase weekly active teams.md","title":"Increase weekly active teams","type":"outcome","schemaData":{"title":"Increase weekly active teams","type":"outcome","status":"identified","metric":"Increase % of trial teams active in their second week","content":"Teams that use the product every week renew."},"linkTargets":["Increase weekly active teams"]}
2
+
{"id":"Setting up a workspace takes so long that I give up.md","title":"Setting up a workspace takes so long that I give up","type":"opportunity","schemaData":{"title":"Setting up a workspace takes so long that I give up","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Interview with Priya, 2026-09-02","content":"Priya abandoned setup twice before her team lead finished it for her."},"linkTargets":["Setting up a workspace takes so long that I give up"]}
3
+
{"id":"I can't tell which plan fits a team of my size.md","title":"I can't tell which plan fits a team of my size","type":"opportunity","schemaData":{"title":"I can't tell which plan fits a team of my size","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Sales call notes, 2026-09-10","content":"Prospects asked which tier covers twelve seats with guest access."},"linkTargets":["I can't tell which plan fits a team of my size"]}
4
+
{"id":"I want to see what my teammates changed while I was away.md","title":"I want to see what my teammates changed while I was away","type":"opportunity","schemaData":{"title":"I want to see what my teammates changed while I was away","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Interview with Tomas, 2026-09-15","content":"Tomas reads every page again after a holiday to find what is new."},"linkTargets":["I want to see what my teammates changed while I was away"]}
5
+
{"id":"I'm afraid I'll lose my work if my browser crashes.md","title":"I'm afraid I'll lose my work if my browser crashes","type":"opportunity","schemaData":{"title":"I'm afraid I'll lose my work if my browser crashes","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Support tickets, September 2026","content":"Several tickets describe retyping long notes after a crash."},"linkTargets":["I'm afraid I'll lose my work if my browser crashes"]}
6
+
{"id":"I don't know whether my changes have been saved.md","title":"I don't know whether my changes have been saved","type":"opportunity","schemaData":{"title":"I don't know whether my changes have been saved","type":"opportunity","status":"identified","parent":"[[I'm afraid I'll lose my work if my browser crashes]]","source":"Usability test round 3","content":"Four of six participants pressed the save shortcut repeatedly."},"linkTargets":["I don't know whether my changes have been saved"]}
7
+
{"id":"Show unsaved-change indicator.md","title":"Show unsaved-change indicator","type":"solution","schemaData":{"title":"Show unsaved-change indicator","type":"solution","status":"exploring","parent":"[[I don't know whether my changes have been saved]]","content":"A dot on the tab while changes are pending."},"linkTargets":["Show unsaved-change indicator"]}
8
+
{"id":"Add a progress bar to workspace setup.md","title":"Add a progress bar to workspace setup","type":"opportunity","schemaData":{"title":"Add a progress bar to workspace setup","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Product team brainstorm","content":"Show users how many setup steps remain."},"linkTargets":["Add a progress bar to workspace setup"]}
9
+
{"id":"Build a Slack integration.md","title":"Build a Slack integration","type":"opportunity","schemaData":{"title":"Build a Slack integration","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Competitor review","content":"Post page updates to a Slack channel."},"linkTargets":["Build a Slack integration"]}
10
+
{"id":"Offer a discount on annual plans.md","title":"Offer a discount on annual plans","type":"opportunity","schemaData":{"title":"Offer a discount on annual plans","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Pricing workshop","content":"Ten percent off when paying yearly."},"linkTargets":["Offer a discount on annual plans"]}
11
+
{"id":"Increase trial-to-paid conversion by 10%.md","title":"Increase trial-to-paid conversion by 10%","type":"opportunity","schemaData":{"title":"Increase trial-to-paid conversion by 10%","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Quarterly planning","content":"The board wants conversion to rise this quarter."},"linkTargets":["Increase trial-to-paid conversion by 10%"]}
12
+
{"id":"Reduce churn among small teams.md","title":"Reduce churn among small teams","type":"opportunity","schemaData":{"title":"Reduce churn among small teams","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Retention dashboard","content":"Teams of fewer than five seats churn twice as often."},"linkTargets":["Reduce churn among small teams"]}
13
+
{"id":"Onboarding.md","title":"Onboarding","type":"opportunity","schemaData":{"title":"Onboarding","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Interview synthesis","content":"Everything about how new users get started."},"linkTargets":["Onboarding"]}
14
+
{"id":"New users find the editor toolbar confusing.md","title":"New users find the editor toolbar confusing","type":"opportunity","schemaData":{"title":"New users find the editor toolbar confusing","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Usability test round 2","content":"Participants hovered over icons without knowing what they did."},"linkTargets":["New users find the editor toolbar confusing"]}
0 commit comments