scenario: fill the corpus until both coverage axes are green - #370
Open
Yaraslaut wants to merge 11 commits into
Open
scenario: fill the corpus until both coverage axes are green#370Yaraslaut wants to merge 11 commits into
Yaraslaut wants to merge 11 commits into
Conversation
Records the plan for filling scripts/scenario/scenarios/ until scenario_coverage.py exits 0 on both axes: a per-rung server driver, 65 named journeys sourced from the rung READMEs, the protocol-axis scenarios, and allowlist entries for the four refusals gated on server configuration no rung installs. Every protocol-axis claim in it was confirmed against a live ladder_pastebin_server and ladder_bookmarks_server rather than inferred from remote.hpp. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
Seven tasks: the server driver, one per rung in ascending workflow-floor order, and the protocol axis. Each rung task opens by probing the real server for wire shapes the DTOs do not state, because that is what the existing ledger scenario's header comment records having needed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
morph_scenario.py owns no server lifecycle by design, which is the right boundary for one scenario and the wrong one for a corpus: a directory of files, each of which mutate_scenario.py reruns once per assertion, cannot be driven by hand. run_scenarios.py starts one server per rung on a fresh SQLite database with its port bound to 0, runs the directory against it, and tears the process group down. --twice reruns the directory against the database the first pass left behind, which is what proves every file is re-runnable; --mutate proves its assertions are load-bearing. Two things the five servers do not agree on, both handled rather than assumed: pastebin announces "listening on port N" where the others announce "listening on ws://127.0.0.1:N", and kanban announces a second, HTTP port for its attachment side channel that must not be mistaken for the WebSocket one. Ledger is seeded with two ledgers rows because no registered action creates one, so OpenAccount against an empty database is refused. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
Eight pastebin journeys and ten polls journeys, each sourced from that
rung's own README -- pastebin's first file is the definition of done's
"create -> list -> open -> burn -> delete" verbatim, and polls' undo file
is the A-votes-B-votes-A-undoes interleaving the README names as the
rung's headline design record.
Both rungs now dispatch every registered action (6/6 and 9/9) and meet
their floors (8/8 and 10/10). Every file was run against a real server,
re-run against the database the first pass left behind, and mutation
tested: no assertion in either rung survives being flipped.
Two defects the corpus found, both fixed here:
* run_scenarios.py never drained the server's stdout after reading the
port line, so a rung server blocked in write(2) once the 64 KiB pipe
filled. The corpus passed and --mutate died with "no data within
10.0s" about sixty reruns in, on a server that was neither crashed
nor refusing. A daemon thread now consumes every line and keeps the
tail for diagnostics.
* paste-lifecycle.scenario asserted a new paste appears in a bare
ListPastes, which is keyset pagination twenty rows at a time -- true
only while the database holds fewer than twenty public pastes with
larger ids. It was passing by luck; repeated runs made it fail about
half the time. It now asks for the page beginning just above its own
captured id, which holds at any database size.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
Twelve journeys covering all seventeen registered actions: sign-in and
CRUD, archive, the atomic bulk edit in both its succeeding and failing
halves, tag rename and merge cascades, the filter axes, the
changes-since poll, the import/export round trip, and two principals
whose collections are isolated but whose shared feed is not.
Every file runs against a real server, twice against the database the
first pass leaves behind, and survives mutation. Four re-runnability
traps the corpus surfaced, each now recorded in the file that hit it:
* a bookmark's tags come back ordered by tag id, not in the order the
create sent them, so every tag assertion is a containment test;
* an ImportOpId is spent permanently and deleting the rows it created
does not release it, so the import file derives its opIds and urls
from a BookmarkId no earlier run had;
* ChangesCursor is (timestampMs, lastId), and consecutive writes over
a loopback socket land inside one millisecond -- so every poll is
asked about a row whose id is above its cursor's lastId, and none
depends on the clock;
* a mutation run aborts files mid-way and leaves their rows behind, so
closing assertions name this run's own ids rather than asserting an
empty listing or an export that mentions nobody.
RecordMetadata is dispatched only to assert its service-principal
refusal; it cannot be driven to success from a socket.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
Fourteen new journeys beside the existing one, covering all seventeen
registered actions: the per-currency zero-sum invariant and the
cross-currency entry that only looks balanced, all four account kinds,
categories and budgets with their month boundary, the rule cascade, the
UpdateRule version conflict the README names as its Scenario B, CSV
import with both of its dedup mechanisms, submit-then-poll reporting,
two books, two editors, and the refusals that leave the books untouched.
Three behaviours the corpus found and now pins, each asserted as it
behaves rather than as it reads:
* a rule firing on a store moves the *account's* whole history, not
just the entry that matched -- categorisation here is account-level,
so one purchase at a newly-ruled merchant reassigns the month;
* StoreTransaction accepts a leg naming another book's account: the
journal files under the named ledger and the foreign balance moves,
while every read stays scoped, so the two books disagree and neither
says so. A leg naming no account at all *is* refused, which is what
shows the lookup runs and simply does not check the book;
* ImportLedgerChunk commits rows as it parses them, so a malformed row
mid-chunk leaves the rows before it posted. Recoverable via the
content hash, which the file then demonstrates.
A wall-clock race of my own making, found by the mutation run: App
sweeps Pending report jobs once a second, so `status == 0` raced the
timer. Both files now assert only what does not race -- the id resolves,
and a job never reaches Failed.
Re-runnability needed a per-run nonce in three files: rules name their
target category by *name*, and the import's content hash is over
description/date/amount with the account id not in it, so fixed strings
made a second run resolve to the first run's rows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
could not reach
kanban could not be driven at all. KanbanAuthorizer inherited
SigningAuthorizer without the anonymous carve-out BookmarksAuthorizer
carries, so `authorize()` demanded a token for `Login` itself -- the
only action that mints one. Every action a fresh remote client could
send was answered `unauthorized`, login included, and
ladder_kanban_server was unusable to anyone not already holding a token
minted out of band. ladder_kanban_headless says so in its own comment:
it takes the token on its command line.
bookmarks documents this exact deadlock, and says it was found by
driving a real server from a client. kanban was written later and
inherited the shape but not the fix. It gets the same carve-out, as
narrow as it can be -- one model type, one action type, compared
exactly -- and its 137 tests still pass. Login still refuses the
reserved `system:` namespace, so an anonymous caller can mint a token
for a name it chooses and nothing more.
Twenty kanban journeys follow, covering all twenty-two actions:
projects and boards, the move that is the rung's centerpiece, WIP
limits, RBAC across three roles with promotion and removal taking
effect on a handler the caller is already holding, rule cascades,
comments, attachments, both event streams, exactly-once moves, and one
integration file that walks a week of a team's work.
Three behaviours pinned as they are:
* an unattached BoardModel answers `stoull` -- the what() of
std::invalid_argument -- to every action. Two of them have a
has_value() guard with the right message that never fires, because
the optional is engaged with an empty string;
* GetRules requires a projectId and then ignores it, answering for
the attached board instead;
* the audit feed includes refused actions and carries no outcome
field, so a rejected move reads exactly like one that happened.
The protocol axis closes with three files: the four instance-management
envelope kinds with their typeId refusals, a malformed frame, and the
256-model cap walked to its limit. Four refusals gated on LimitPolicy
or PayloadCompleteness settings no rung installs are exempted with
written reasons naming the setting and the file.
Two supporting fixes. The driver now gives each run its own working
directory: kanban writes its action log to current_path(), so runs
accumulated it in the caller's directory and GetActivity replayed a
previous run's project 1 as this one's. And morph_scenario accepts the
one reply that legitimately carries no callId -- a decode failure has
no envelope to read one from -- without which that path is unassertable.
scenario_coverage.py now exits 0: 71/71 actions, every rung at its
floor, 8/8 kinds, 11/17 refusals with the rest exempt. Its own canary
test, which pinned exit 1 and said authoring the corpus was later work,
is flipped to 0 with that history recorded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
boundaries in its spec The design spec said "no change to the example applications". Two boundaries were crossed with the author's agreement and are now written down where they contradicted the plan rather than left to be discovered in the diff: kanban's authorizer fix, without which the rung's 22 actions and floor of 20 were unreachable, and the one new rule morph_scenario.py learned about a reply that legitimately carries no callId. Neither the coverage thresholds nor the workflow floors moved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
execute(ImportLedgerChunk) documents the per-row commit as a deliberate decision, with the reason: a SqlTransaction nested around the loop would have storeJournalImpl's inner Commit() end the outer transaction, since SqlTransaction toggles SQL_ATTR_AUTOCOMMIT on the connection directly. The scenario asserted the behaviour correctly but described it as though it were merely observed, which invited a reader to file it. It now names the ruling and says what a client does about it instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
Each file that asserts a defect as it behaves now names the issue that tracks it, so a reader who finds the assertion surprising has somewhere to go and a fix has a place to land: morph#367 (ledger legs are not scoped to the named book), morph#368 (an unattached BoardModel replies "stoull", and the two guards written to prevent that never fire), morph#369 (GetRules requires a projectId and ignores it). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
The coverage report existed and was green, and nothing kept it that
way. drift-guard.yml gains a scenario-coverage job running the runner's
self-test and then scenario_coverage.py.
It belongs in this workflow rather than the build matrix because it
compiles nothing and starts nothing: the report reads remote.hpp,
wire.hpp, the BRIDGE_REGISTER_ACTION sites under examples/, the
scenario files and the allowlist, all as source. Verified on a bare
checkout with no build directory.
The gate is proven able to fail before it is trusted to pass, on the
same terms as the two jobs beside it:
* deleting one kanban scenario -> exit 1, "kanban: 19 workflows,
floor is 20";
* adding an unasserted refusal to remote.hpp -> exit 1, naming it
under UNCOVERED refusals;
* renaming one out from under its allowlist entry -> exit 2, the
reverse audit reporting a stale exemption.
What this does not do is run the corpus: that needs five server
binaries built, which this workflow has none of and would stop being
fast and dependency-free to acquire. The README now says so plainly, so
the gap is stated rather than assumed closed -- a scenario can still
drift from a server's real behaviour without CI noticing, only not from
its surface.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
scripts/scenario/scenario_coverage.pymeasured a surface nothing filled. This fills it: the corpus goes from 4 files to 72, and the report now exits0on both axes.71/71 actions dispatched. 8/8 envelope kinds. 11/17 refusals asserted, the remaining 6 exempt with written reasons.
Every journey is named after what it does and sourced from its rung's own README — pastebin's first file is the definition of done's create → list → open → burn → delete verbatim; polls' undo file is the A-votes-B-votes-A-undoes interleaving the README calls that rung's headline design record.
Verification
Every file was run against a real
ladder_<rung>_server, re-run against the database the first pass left behind, and mutation-tested. 71 files, zero surviving mutants.test_morph_scenario.pypasses 118 cases;ladder_kanban_testspasses 137.scripts/scenario/run_scenarios.pyThe server lifecycle
morph_scenario.pydeliberately does not own, and which the workflow-axis spec already anticipated asrun_all.sh. One server per rung, on a fresh SQLite database in its own working directory with the port bound to0; runs the directory; tears the process group down.--twiceproves every file is re-runnable,--mutateproves its assertions are load-bearing.The blocker: kanban could not be driven at all
KanbanAuthorizerinheritedSigningAuthorizerwithout the anonymous carve-outBookmarksAuthorizercarries, soauthorize()demanded a bearer token forAuthModel/Login— the only action that mints one. Every action a fresh remote client could send was answerederr "unauthorized", login included.ladder_kanban_headlesssays so in its own comment: it takes the token on its command line, because it cannot ask the server for one.bookmarks documents this exact deadlock, and records that it was found by driving a real server from a client. kanban was written later and inherited the shape but not the fix. It gets the identical carve-out — one model type, one action type, compared exactly.
Loginstill refuses the reservedsystem:namespace, so an anonymous caller can mint a token for a name it chooses and nothing more. All 137 kanban tests still pass.Without this the rung's 22 actions and floor of 20 were unreachable.
What the corpus found
Three defects, each asserted as it behaves so the gate stays green, each with its issue named in the file that pins it:
StoreTransactionposts legs onto another book's accounts. The journal files under the named ledger, the foreign balance moves, and every read stays scoped — so the two books disagree and neither says so. A leg naming no account at all is refused, which shows the lookup runs and simply does not check the book.UndoTransactionmakes the equivalent check.BoardModelrepliesstoull, thewhat()ofstd::invalid_argument. Two actions have a guard with exactly the right message that never fires, because the optional is engaged with an empty string.GetRulesrequires aprojectIdand then queries the attached board instead.Four defects in the test apparatus itself, fixed here:
write(2)on a full pipe about sixty mutant reruns in — the corpus passed and--mutatedied with "no data within 10.0s" on a server that was neither crashed nor refusing;current_path(), so runs accumulated it in the caller's directory andGetActivityreplayed a previous run's project 1 as this one's;paste-lifecycle.scenarioasserted a new paste appears in a bareListPastes— true only while the database holds fewer than twenty public pastes with larger ids. It was passing by luck, and failed about half the time under repetition;morph_scenario.pycorrelated strictly bycallId, which made the decode-failure path unassertable: a frame the server cannot decode has no envelope to read acallIdfrom, so it answers with0.Client.rpcnow accepts that in exactly that shape and nowhere else.Deviations from the plan, and where they are recorded
The design spec said "no change to the example applications". Two boundaries were crossed deliberately — the kanban authorizer fix, and the
callIdrule above — and both are now written into the spec where it said otherwise, rather than left to be found in the diff. Neither the coverage thresholds nor the workflow floors moved.scenario_coverage.py's own canary test, which pinned exit 1 and said in its comment that authoring the corpus was later work, is flipped to 0 with that history recorded.Not filed, deliberately
ImportLedgerChunk's per-row commit leaves earlier rows posted when a later row is malformed. That looked like a defect until the code was read:execute(ImportLedgerChunk)records it as a ruling and gives the reason — aSqlTransactionnested around the loop would havestoreJournalImpl's innerCommit()end the outer one, sinceSqlTransactiontogglesSQL_ATTR_AUTOCOMMITon the connection directly. The scenario pins it and now cites that ruling instead of implying a bug.🤖 Generated with Claude Code
https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ