Repository navigation
Commit 822cfc2
committed
test(diagrams): the chat in a browser, the feature in the real editor, and an eval for models
Three scripts for the three things the suites cannot show. None is in the
gate, which is plain Node and stays that way.
scripts/diagram-browser-check.js — the shipped chat.html in headless Chrome,
built the way the host builds it (the same bundle, the same policy) and fed
records made by the real service. 155 checks across both themes: every diagram
is an SVG of the painter's elements only; hostile labels are on the page as
text and nothing ran; no policy violation and no request; the placeholder, the
badge, the banner, the failed view; a click names a node, never a path; SVG
and PNG export pass the host's own check; and a painter that throws, or a page
with no diagram modules at all, both end in the text fallback. Each step waits
for its result, not for a length of time: the first version waited, and failed
one run in three.
scripts/diagram-editor-check.js — the real editor. A second, throwaway instance
of the dev build (its own profile, sessions folder and workspace) loads THIS
checkout's extension in place of the built-in one, talks to a stand-in provider
on localhost, and is driven through the DevTools protocol. 17 checks: the tool
and its rules reach the model; the diagram is painted in the real webview in
the editor's colours; a wider column re-lays it out; a linked node opens its
file at the symbol; the session on disk holds the diagram; and the answer
around it, which quotes a code fence, is one paragraph (the chat fix earlier in
this branch).
It exists because of a mistake worth writing down. The feature was first called
done having only ever run in a browser, with instructions for trying it that
could not have worked: the work sat uncommitted in a git worktree, and a
worktree has no vscode/ — run-dev.sh runs the extensions of the checkout that
does. The editor that was then tried was develop's, with no diagrams in it.
What does work, by hand, from the checkout that has vscode/:
./scripts/run-dev.sh --extensionDevelopmentPath=<checkout>/extensions/levelcode-ai
scripts/diagram-eval.js — the spec's eval: 30 prompts that should produce a
diagram and 10 that should not (fixtures/diagrams/eval-prompts.json), each run
through the real agent loop and scored on format choice, first-pass validity,
fixes by rung, degraded rate, error classes, tokens, node-count overruns, title
quality and answers that drew with characters. --dry-run uses a scripted model
and no network. --run makes billed calls on your key, says how many before the
first one, and is refused without a model and a key; the workspace is an empty
temporary folder and every approval is answered no.
It has not been run against any model. That is the next thing this feature
needs: nothing here says how often a real model's first attempt is valid, or
whether it reaches for the tool at all.
diagramEval (16 tests) holds the harness to its own arithmetic: a scripted
model whose mistakes are known comes out at exactly the numbers its script
implies, and nothing is sent without --run, a model and a key.1 parent ce02534 commit 822cfc2
5 files changed
Lines changed: 1365 additions & 0 deletions
File tree
- extensions/levelcode-ai
- scripts
- test
- fixtures/diagrams
0 commit comments