Skip to content

Commit faa4a41

Browse files
claudemindsocket
authored andcommitted
refactor(evals): fixtures are spaces with parsed nodes
A suite's fixtures.json5 lists fixtures as space configs (name, path to a JSONL nodes file, schema, plugins), so fixtures share schemas the way spaces do. A nodes file holds parsed nodes as `sctx dump` prints them, with their own ids and link targets; the harness no longer derives either from a title. Every eval names a space: a fixture or a configured space, labelled by node id the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XphpdrUMvqf8fuj8uG7JZ
1 parent e968a30 commit faa4a41

7 files changed

Lines changed: 131 additions & 120 deletions

File tree

‎evals/README.md‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Semantic evals
22

3-
Semantic evals measure how well model providers judge semantic rules. A case is a node, a rule, and the verdict the node should get. An eval holds the cases on one set of nodes: a fixture, or a configured space.
3+
Semantic evals measure how well model providers judge semantic rules. A case is a node, a rule, and the verdict the node should get. An eval holds the cases on the nodes of one space. That space is a fixture, which keeps its nodes in the suite, or a configured space.
44

55
## Run the evals
66

@@ -38,13 +38,13 @@ Each run validates through the same pipeline as `sctx validate --semantic`, with
3838

3939
## Suites
4040

41-
A suite is a directory of evals, with their fixtures in `fixtures/`:
41+
A suite is a directory of evals and the fixtures they use:
4242

4343
```
4444
evals/semantic/
45-
opportunity-customer-voice.json5
46-
fixtures/
47-
strict-ost.json5
45+
opportunity-customer-voice.json5 an eval
46+
fixtures.json5 the fixtures, as space configs
47+
fixtures/strict-ost.jsonl a fixture's nodes
4848
```
4949

5050
| `--from` | Directory | Schema names resolve from |
@@ -55,30 +55,30 @@ evals/semantic/
5555

5656
## Add an eval
5757

58-
An eval is a `.json5` file in the suite. It names a `fixture` in the suite, or a configured `space`. `expected` maps a rule id to node ids and their verdicts:
58+
An eval is a `.json5` file in the suite. It names a `space`: one of the suite's fixtures, or else a space in the config. `expected` maps a rule id to node ids and their verdicts:
5959

6060
```json5
6161
{
62-
fixture: 'strict-ost',
62+
space: 'strict-ost',
6363
expected: {
6464
'opportunity-customer-voice': {
65-
'Build a Slack integration': 'fail',
65+
'Build a Slack integration.md': 'fail',
6666
// A solution in disguise.
67-
'Add a progress bar to workspace setup': 'fail',
67+
'Add a progress bar to workspace setup.md': 'fail',
6868
},
6969
},
7070
}
7171
```
7272

73-
A list of verdicts accepts any of them. On a configured space, node ids are file paths: `{ space: 'work', expected: { 'opportunity-customer-voice': { 'Onboarding.md': 'fail' } } }`.
74-
75-
Write a comment on a case whose label is not obvious. We recommend one eval per rule, so each rule's cases are easy to find and to run alone.
73+
A list of verdicts accepts any of them. Write a comment on a case whose label is not obvious. We recommend one eval per rule, so each rule's cases are easy to find and to run alone.
7674

7775
## Add a fixture
7876

79-
A fixture is `fixtures/<name>.json5`: a `schema`, optional `plugins` as in a space config, and `nodes`. Each node holds the fields its frontmatter would, with a body in `content`. Its `title` is its node id and link target. `sctx dump <space> | jq '.nodes | map(.schemaData)'` prints notes in this shape.
77+
A fixture is a space whose nodes are in a JSONL file in the suite. `fixtures.json5` lists the fixtures as a config lists spaces: a `name`, the `path` to the nodes file, a `schema`, and optional `plugins`. Several fixtures can share a schema, and several evals can share a fixture.
78+
79+
The nodes file holds one parsed node per line, as `sctx dump` prints them. To make one from markdown notes, configure them as a space and run `sctx dump <space> | jq -c '.nodes[]'`.
8080

81-
Several evals can share a fixture. For a rule whose `scope` sends related nodes, such as siblings, give each scenario its own small tree, so that one case's related nodes do not change another's.
81+
For a rule whose `scope` sends related nodes, such as siblings, give each scenario its own small tree, so that one case's related nodes do not change another's.
8282

8383
## Checks
8484

‎evals/load.ts‎

Lines changed: 49 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,14 @@ import { resolvePluginDir } from '../src/plugins/util';
66
import { readSpace } from '../src/read/read-space';
77
import { resolveGraphEdges } from '../src/read/resolve-graph-edges';
88
import { createSpaceContext } from '../src/space-context';
9-
import type { BaseNode, NodeId, SpaceContext } from '../src/types';
9+
import type { BaseNode, SpaceContext } from '../src/types';
1010

1111
/** Where core and each plugin keep their semantic suite. */
1212
export const SUITE_SUBDIR = join('evals', 'semantic');
1313

14+
/** The suite's fixtures: spaces whose nodes are in a JSONL file. */
15+
const FIXTURES_FILE = 'fixtures.json5';
16+
1417
const CORE_SUITE_DIR = join(import.meta.dir, 'semantic');
1518

1619
/** A directory of evals and their fixtures, and the directory schema names resolve from. */
@@ -23,7 +26,7 @@ export interface Suite {
2326
/** Rule id → node id → the verdicts that count as agreeing. */
2427
export type Expected = Record<string, Record<string, string[]>>;
2528

26-
/** Labelled cases on the nodes of one fixture or configured space. */
29+
/** Labelled cases on the nodes of one space: a fixture, or a configured space. */
2730
export interface Eval {
2831
/** `<suite>/<file name>`, e.g. `core/opportunity-customer-voice`. */
2932
name: string;
@@ -50,58 +53,66 @@ export function resolveSuite(source: string, configDirs: string[]): Suite {
5053
throw new Error(`Suite "${source}" is not core, a path, or a plugin found from the config.`);
5154
}
5255

53-
type EvalFile = { fixture?: string; space?: string; expected: Record<string, Record<string, string | string[]>> };
54-
type FixtureFile = Pick<SpaceConfig, 'schema' | 'plugins'> & { nodes: Record<string, unknown>[] };
56+
type EvalFile = { space: string; expected: Record<string, Record<string, string | string[]>> };
57+
type FixturesFile = { spaces: Pick<SpaceConfig, 'name' | 'path' | 'schema' | 'plugins'>[] };
58+
type Space = Pick<Eval, 'context' | 'read'>;
5559

56-
/** Every `*.json5` at the top of each suite is an eval. `config` holds the spaces evals name. */
60+
/**
61+
* Every `*.json5` at the top of each suite, except `fixtures.json5`, is an eval. An eval's `space` names one of
62+
* the suite's fixtures, or else a space in `config`.
63+
*/
5764
export function loadEvals(suites: Suite[], config: Config): Eval[] {
5865
return suites.flatMap((suite) => {
59-
const fixtures = new Map<string, Pick<Eval, 'context' | 'read'>>();
60-
const fixture = (name: string) => {
61-
if (!fixtures.has(name)) fixtures.set(name, loadFixture(suite, name));
62-
return fixtures.get(name)!;
63-
};
66+
const fixtures = loadFixtures(suite);
6467
return readdirSync(suite.dir)
65-
.filter((file) => file.endsWith('.json5'))
68+
.filter((file) => file.endsWith('.json5') && file !== FIXTURES_FILE)
6669
.sort()
6770
.map((file) => {
6871
const name = `${suite.name}/${basename(file, '.json5')}`;
6972
try {
70-
const evalFile = JSON5.parse(readFileSync(join(suite.dir, file), 'utf8')) as EvalFile;
71-
if ((evalFile.fixture === undefined) === (evalFile.space === undefined)) {
72-
throw new Error('needs either "fixture" or "space"');
73-
}
73+
const { space, expected: labels } = JSON5.parse(readFileSync(join(suite.dir, file), 'utf8')) as EvalFile;
74+
if (typeof space !== 'string') throw new Error('needs a "space"');
7475
const expected: Expected = {};
75-
for (const [ruleId, labels] of Object.entries(evalFile.expected)) {
76-
expected[ruleId] = Object.fromEntries(Object.entries(labels).map(([id, v]) => [id, [v].flat()]));
76+
for (const [ruleId, verdicts] of Object.entries(labels)) {
77+
expected[ruleId] = Object.fromEntries(Object.entries(verdicts).map(([id, v]) => [id, [v].flat()]));
7778
}
78-
if (evalFile.fixture !== undefined) return { name, expected, ...fixture(evalFile.fixture) };
79-
const context = createSpaceContext(evalFile.space!, config);
80-
return { name, expected, context, read: () => readSpace(context) };
79+
return { name, expected, ...(fixtures.get(space) ?? configuredSpace(space, config)) };
8180
} catch (err) {
8281
throw new Error(`Eval ${name}: ${err instanceof Error ? err.message : String(err)}`);
8382
}
8483
});
8584
});
8685
}
8786

88-
/** `fixtures/<name>.json5`: a schema and the nodes' schema data, each with its `title` as node id and link target. */
89-
function loadFixture(suite: Suite, name: string): Pick<Eval, 'context' | 'read'> {
90-
const path = join(suite.dir, 'fixtures', `${name}.json5`);
91-
if (!existsSync(path)) throw new Error(`no fixture at ${path}`);
92-
const file = JSON5.parse(readFileSync(path, 'utf8')) as FixtureFile;
93-
const spaceName = `${suite.name}/fixtures/${name}`;
94-
const context = createSpaceContext(
95-
spaceName,
96-
{ spaces: [{ name: spaceName, path: suite.dir, schema: file.schema, plugins: file.plugins }] },
97-
{ configDir: suite.configDir },
87+
function configuredSpace(name: string, config: Config): Space {
88+
const context = createSpaceContext(name, config);
89+
return { context, read: () => readSpace(context) };
90+
}
91+
92+
/** A fixture's nodes file holds one parsed node per line, as `sctx dump` prints them. Resolved fields are dropped, since resolution runs again. */
93+
function loadFixtures(suite: Suite): Map<string, Space> {
94+
const path = join(suite.dir, FIXTURES_FILE);
95+
if (!existsSync(path)) return new Map();
96+
const { spaces } = JSON5.parse(readFileSync(path, 'utf8')) as FixturesFile;
97+
const config: Config = { spaces: spaces.map((s) => ({ ...s, path: resolve(suite.dir, s.path) })) };
98+
return new Map(
99+
config.spaces.map(({ name, path: nodesPath }) => {
100+
const context = createSpaceContext(name, config, { configDir: suite.configDir });
101+
const nodes = readNodes(nodesPath);
102+
return [name, { context, read: async () => resolveGraphEdges(nodes, context.schema.metadata) }];
103+
}),
98104
);
99-
const nodes: BaseNode[] = file.nodes.map((schemaData, i) => {
100-
const { title, type } = schemaData;
101-
if (typeof title !== 'string' || typeof type !== 'string') {
102-
throw new Error(`${path}: node ${i + 1} needs a "title" and a "type"`);
103-
}
104-
return { id: title as NodeId, title, type, linkTargets: [title], schemaData };
105-
});
106-
return { context, read: async () => resolveGraphEdges(nodes, context.schema.metadata) };
105+
}
106+
107+
function readNodes(path: string): BaseNode[] {
108+
return readFileSync(path, 'utf8')
109+
.split('\n')
110+
.flatMap((line, i) => {
111+
if (line.trim() === '') return [];
112+
const { id, title, type, schemaData, linkTargets, content, contentLinks } = JSON.parse(line) as BaseNode;
113+
if (!id || typeof title !== 'string' || typeof type !== 'string' || !schemaData || !linkTargets) {
114+
throw new Error(`${path}:${i + 1}: not a parsed node (needs id, title, type, schemaData and linkTargets)`);
115+
}
116+
return [{ id, title, type, schemaData, linkTargets, content, contentLinks }];
117+
});
107118
}

‎evals/semantic/fixtures.json5‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
spaces: [
3+
// A small product's opportunity solution tree.
4+
{ name: 'strict-ost', path: 'fixtures/strict-ost.jsonl', schema: 'strict_ost.json' },
5+
],
6+
}

‎evals/semantic/fixtures/strict-ost.json5‎

Lines changed: 0 additions & 26 deletions
This file was deleted.
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{"id":"Increase weekly active teams.md","title":"Increase weekly active teams","type":"outcome","schemaData":{"title":"Increase weekly active teams","type":"outcome","status":"identified","metric":"Increase % of trial teams active in their second week","content":"Teams that use the product every week renew."},"linkTargets":["Increase weekly active teams"]}
2+
{"id":"Setting up a workspace takes so long that I give up.md","title":"Setting up a workspace takes so long that I give up","type":"opportunity","schemaData":{"title":"Setting up a workspace takes so long that I give up","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Interview with Priya, 2026-09-02","content":"Priya abandoned setup twice before her team lead finished it for her."},"linkTargets":["Setting up a workspace takes so long that I give up"]}
3+
{"id":"I can't tell which plan fits a team of my size.md","title":"I can't tell which plan fits a team of my size","type":"opportunity","schemaData":{"title":"I can't tell which plan fits a team of my size","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Sales call notes, 2026-09-10","content":"Prospects asked which tier covers twelve seats with guest access."},"linkTargets":["I can't tell which plan fits a team of my size"]}
4+
{"id":"I want to see what my teammates changed while I was away.md","title":"I want to see what my teammates changed while I was away","type":"opportunity","schemaData":{"title":"I want to see what my teammates changed while I was away","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Interview with Tomas, 2026-09-15","content":"Tomas reads every page again after a holiday to find what is new."},"linkTargets":["I want to see what my teammates changed while I was away"]}
5+
{"id":"I'm afraid I'll lose my work if my browser crashes.md","title":"I'm afraid I'll lose my work if my browser crashes","type":"opportunity","schemaData":{"title":"I'm afraid I'll lose my work if my browser crashes","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Support tickets, September 2026","content":"Several tickets describe retyping long notes after a crash."},"linkTargets":["I'm afraid I'll lose my work if my browser crashes"]}
6+
{"id":"I don't know whether my changes have been saved.md","title":"I don't know whether my changes have been saved","type":"opportunity","schemaData":{"title":"I don't know whether my changes have been saved","type":"opportunity","status":"identified","parent":"[[I'm afraid I'll lose my work if my browser crashes]]","source":"Usability test round 3","content":"Four of six participants pressed the save shortcut repeatedly."},"linkTargets":["I don't know whether my changes have been saved"]}
7+
{"id":"Show unsaved-change indicator.md","title":"Show unsaved-change indicator","type":"solution","schemaData":{"title":"Show unsaved-change indicator","type":"solution","status":"exploring","parent":"[[I don't know whether my changes have been saved]]","content":"A dot on the tab while changes are pending."},"linkTargets":["Show unsaved-change indicator"]}
8+
{"id":"Add a progress bar to workspace setup.md","title":"Add a progress bar to workspace setup","type":"opportunity","schemaData":{"title":"Add a progress bar to workspace setup","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Product team brainstorm","content":"Show users how many setup steps remain."},"linkTargets":["Add a progress bar to workspace setup"]}
9+
{"id":"Build a Slack integration.md","title":"Build a Slack integration","type":"opportunity","schemaData":{"title":"Build a Slack integration","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Competitor review","content":"Post page updates to a Slack channel."},"linkTargets":["Build a Slack integration"]}
10+
{"id":"Offer a discount on annual plans.md","title":"Offer a discount on annual plans","type":"opportunity","schemaData":{"title":"Offer a discount on annual plans","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Pricing workshop","content":"Ten percent off when paying yearly."},"linkTargets":["Offer a discount on annual plans"]}
11+
{"id":"Increase trial-to-paid conversion by 10%.md","title":"Increase trial-to-paid conversion by 10%","type":"opportunity","schemaData":{"title":"Increase trial-to-paid conversion by 10%","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Quarterly planning","content":"The board wants conversion to rise this quarter."},"linkTargets":["Increase trial-to-paid conversion by 10%"]}
12+
{"id":"Reduce churn among small teams.md","title":"Reduce churn among small teams","type":"opportunity","schemaData":{"title":"Reduce churn among small teams","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Retention dashboard","content":"Teams of fewer than five seats churn twice as often."},"linkTargets":["Reduce churn among small teams"]}
13+
{"id":"Onboarding.md","title":"Onboarding","type":"opportunity","schemaData":{"title":"Onboarding","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Interview synthesis","content":"Everything about how new users get started."},"linkTargets":["Onboarding"]}
14+
{"id":"New users find the editor toolbar confusing.md","title":"New users find the editor toolbar confusing","type":"opportunity","schemaData":{"title":"New users find the editor toolbar confusing","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Usability test round 2","content":"Participants hovered over icons without knowing what they did."},"linkTargets":["New users find the editor toolbar confusing"]}

0 commit comments

Comments
 (0)