Skip to content

Commit 822cfc2

Browse files
committed
test(diagrams): the chat in a browser, the feature in the real editor, and an eval for models
Three scripts for the three things the suites cannot show. None is in the gate, which is plain Node and stays that way. scripts/diagram-browser-check.js — the shipped chat.html in headless Chrome, built the way the host builds it (the same bundle, the same policy) and fed records made by the real service. 155 checks across both themes: every diagram is an SVG of the painter's elements only; hostile labels are on the page as text and nothing ran; no policy violation and no request; the placeholder, the badge, the banner, the failed view; a click names a node, never a path; SVG and PNG export pass the host's own check; and a painter that throws, or a page with no diagram modules at all, both end in the text fallback. Each step waits for its result, not for a length of time: the first version waited, and failed one run in three. scripts/diagram-editor-check.js — the real editor. A second, throwaway instance of the dev build (its own profile, sessions folder and workspace) loads THIS checkout's extension in place of the built-in one, talks to a stand-in provider on localhost, and is driven through the DevTools protocol. 17 checks: the tool and its rules reach the model; the diagram is painted in the real webview in the editor's colours; a wider column re-lays it out; a linked node opens its file at the symbol; the session on disk holds the diagram; and the answer around it, which quotes a code fence, is one paragraph (the chat fix earlier in this branch). It exists because of a mistake worth writing down. The feature was first called done having only ever run in a browser, with instructions for trying it that could not have worked: the work sat uncommitted in a git worktree, and a worktree has no vscode/ — run-dev.sh runs the extensions of the checkout that does. The editor that was then tried was develop's, with no diagrams in it. What does work, by hand, from the checkout that has vscode/: ./scripts/run-dev.sh --extensionDevelopmentPath=<checkout>/extensions/levelcode-ai scripts/diagram-eval.js — the spec's eval: 30 prompts that should produce a diagram and 10 that should not (fixtures/diagrams/eval-prompts.json), each run through the real agent loop and scored on format choice, first-pass validity, fixes by rung, degraded rate, error classes, tokens, node-count overruns, title quality and answers that drew with characters. --dry-run uses a scripted model and no network. --run makes billed calls on your key, says how many before the first one, and is refused without a model and a key; the workspace is an empty temporary folder and every approval is answered no. It has not been run against any model. That is the next thing this feature needs: nothing here says how often a real model's first attempt is valid, or whether it reaches for the tool at all. diagramEval (16 tests) holds the harness to its own arithmetic: a scripted model whose mistakes are known comes out at exactly the numbers its script implies, and nothing is sent without --run, a model and a key.
1 parent ce02534 commit 822cfc2

5 files changed

Lines changed: 1365 additions & 0 deletions

File tree

0 commit comments

Comments
 (0)