Skip to content

scenario: fill the corpus until both coverage axes are green - #370

Open
Yaraslaut wants to merge 11 commits into
masterfrom
scenario-corpus-both-axes
Open

scenario: fill the corpus until both coverage axes are green#370
Yaraslaut wants to merge 11 commits into
masterfrom
scenario-corpus-both-axes

Conversation

@Yaraslaut

Copy link
Copy Markdown
Member

scripts/scenario/scenario_coverage.py measured a surface nothing filled. This fills it: the corpus goes from 4 files to 72, and the report now exits 0 on both axes.

Rung Files Actions Workflows (floor)
pastebin 12 6/6 8/8
polls 10 9/9 10/10
bookmarks 14 17/17 12/12
ledger 15 17/17 15/15
kanban 21 22/22 20/20

71/71 actions dispatched. 8/8 envelope kinds. 11/17 refusals asserted, the remaining 6 exempt with written reasons.

Every journey is named after what it does and sourced from its rung's own README — pastebin's first file is the definition of done's create → list → open → burn → delete verbatim; polls' undo file is the A-votes-B-votes-A-undoes interleaving the README calls that rung's headline design record.

Verification

Every file was run against a real ladder_<rung>_server, re-run against the database the first pass left behind, and mutation-tested. 71 files, zero surviving mutants. test_morph_scenario.py passes 118 cases; ladder_kanban_tests passes 137.

python3 scripts/scenario/run_scenarios.py --twice     # exit 0
python3 scripts/scenario/run_scenarios.py --mutate    # exit 0, no survivors
python3 scripts/scenario/scenario_coverage.py         # exit 0

scripts/scenario/run_scenarios.py

The server lifecycle morph_scenario.py deliberately does not own, and which the workflow-axis spec already anticipated as run_all.sh. One server per rung, on a fresh SQLite database in its own working directory with the port bound to 0; runs the directory; tears the process group down. --twice proves every file is re-runnable, --mutate proves its assertions are load-bearing.

The blocker: kanban could not be driven at all

KanbanAuthorizer inherited SigningAuthorizer without the anonymous carve-out BookmarksAuthorizer carries, so authorize() demanded a bearer token for AuthModel/Login — the only action that mints one. Every action a fresh remote client could send was answered err "unauthorized", login included. ladder_kanban_headless says so in its own comment: it takes the token on its command line, because it cannot ask the server for one.

bookmarks documents this exact deadlock, and records that it was found by driving a real server from a client. kanban was written later and inherited the shape but not the fix. It gets the identical carve-out — one model type, one action type, compared exactly. Login still refuses the reserved system: namespace, so an anonymous caller can mint a token for a name it chooses and nothing more. All 137 kanban tests still pass.

Without this the rung's 22 actions and floor of 20 were unreachable.

What the corpus found

Three defects, each asserted as it behaves so the gate stays green, each with its issue named in the file that pins it:

Four defects in the test apparatus itself, fixed here:

  • the driver never drained server stdout, so servers wedged in write(2) on a full pipe about sixty mutant reruns in — the corpus passed and --mutate died with "no data within 10.0s" on a server that was neither crashed nor refusing;
  • kanban writes its action log to current_path(), so runs accumulated it in the caller's directory and GetActivity replayed a previous run's project 1 as this one's;
  • the pre-existing paste-lifecycle.scenario asserted a new paste appears in a bare ListPastes — true only while the database holds fewer than twenty public pastes with larger ids. It was passing by luck, and failed about half the time under repetition;
  • morph_scenario.py correlated strictly by callId, which made the decode-failure path unassertable: a frame the server cannot decode has no envelope to read a callId from, so it answers with 0. Client.rpc now accepts that in exactly that shape and nowhere else.

Deviations from the plan, and where they are recorded

The design spec said "no change to the example applications". Two boundaries were crossed deliberately — the kanban authorizer fix, and the callId rule above — and both are now written into the spec where it said otherwise, rather than left to be found in the diff. Neither the coverage thresholds nor the workflow floors moved.

scenario_coverage.py's own canary test, which pinned exit 1 and said in its comment that authoring the corpus was later work, is flipped to 0 with that history recorded.

Not filed, deliberately

ImportLedgerChunk's per-row commit leaves earlier rows posted when a later row is malformed. That looked like a defect until the code was read: execute(ImportLedgerChunk) records it as a ruling and gives the reason — a SqlTransaction nested around the loop would have storeJournalImpl's inner Commit() end the outer one, since SqlTransaction toggles SQL_ATTR_AUTOCOMMIT on the connection directly. The scenario pins it and now cites that ruling instead of implying a bug.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ

Yaraslaut and others added 11 commits August 31, 2026 11:14
Records the plan for filling scripts/scenario/scenarios/ until
scenario_coverage.py exits 0 on both axes: a per-rung server driver,
65 named journeys sourced from the rung READMEs, the protocol-axis
scenarios, and allowlist entries for the four refusals gated on server
configuration no rung installs.

Every protocol-axis claim in it was confirmed against a live
ladder_pastebin_server and ladder_bookmarks_server rather than inferred
from remote.hpp.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
Seven tasks: the server driver, one per rung in ascending workflow-floor
order, and the protocol axis. Each rung task opens by probing the real
server for wire shapes the DTOs do not state, because that is what the
existing ledger scenario's header comment records having needed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
morph_scenario.py owns no server lifecycle by design, which is the right
boundary for one scenario and the wrong one for a corpus: a directory of
files, each of which mutate_scenario.py reruns once per assertion, cannot
be driven by hand.

run_scenarios.py starts one server per rung on a fresh SQLite database
with its port bound to 0, runs the directory against it, and tears the
process group down. --twice reruns the directory against the database the
first pass left behind, which is what proves every file is re-runnable;
--mutate proves its assertions are load-bearing.

Two things the five servers do not agree on, both handled rather than
assumed: pastebin announces "listening on port N" where the others
announce "listening on ws://127.0.0.1:N", and kanban announces a second,
HTTP port for its attachment side channel that must not be mistaken for
the WebSocket one. Ledger is seeded with two ledgers rows because no
registered action creates one, so OpenAccount against an empty database
is refused.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
Eight pastebin journeys and ten polls journeys, each sourced from that
rung's own README -- pastebin's first file is the definition of done's
"create -> list -> open -> burn -> delete" verbatim, and polls' undo file
is the A-votes-B-votes-A-undoes interleaving the README names as the
rung's headline design record.

Both rungs now dispatch every registered action (6/6 and 9/9) and meet
their floors (8/8 and 10/10). Every file was run against a real server,
re-run against the database the first pass left behind, and mutation
tested: no assertion in either rung survives being flipped.

Two defects the corpus found, both fixed here:

  * run_scenarios.py never drained the server's stdout after reading the
    port line, so a rung server blocked in write(2) once the 64 KiB pipe
    filled. The corpus passed and --mutate died with "no data within
    10.0s" about sixty reruns in, on a server that was neither crashed
    nor refusing. A daemon thread now consumes every line and keeps the
    tail for diagnostics.

  * paste-lifecycle.scenario asserted a new paste appears in a bare
    ListPastes, which is keyset pagination twenty rows at a time -- true
    only while the database holds fewer than twenty public pastes with
    larger ids. It was passing by luck; repeated runs made it fail about
    half the time. It now asks for the page beginning just above its own
    captured id, which holds at any database size.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
Twelve journeys covering all seventeen registered actions: sign-in and
CRUD, archive, the atomic bulk edit in both its succeeding and failing
halves, tag rename and merge cascades, the filter axes, the
changes-since poll, the import/export round trip, and two principals
whose collections are isolated but whose shared feed is not.

Every file runs against a real server, twice against the database the
first pass leaves behind, and survives mutation. Four re-runnability
traps the corpus surfaced, each now recorded in the file that hit it:

  * a bookmark's tags come back ordered by tag id, not in the order the
    create sent them, so every tag assertion is a containment test;
  * an ImportOpId is spent permanently and deleting the rows it created
    does not release it, so the import file derives its opIds and urls
    from a BookmarkId no earlier run had;
  * ChangesCursor is (timestampMs, lastId), and consecutive writes over
    a loopback socket land inside one millisecond -- so every poll is
    asked about a row whose id is above its cursor's lastId, and none
    depends on the clock;
  * a mutation run aborts files mid-way and leaves their rows behind, so
    closing assertions name this run's own ids rather than asserting an
    empty listing or an export that mentions nobody.

RecordMetadata is dispatched only to assert its service-principal
refusal; it cannot be driven to success from a socket.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
Fourteen new journeys beside the existing one, covering all seventeen
registered actions: the per-currency zero-sum invariant and the
cross-currency entry that only looks balanced, all four account kinds,
categories and budgets with their month boundary, the rule cascade, the
UpdateRule version conflict the README names as its Scenario B, CSV
import with both of its dedup mechanisms, submit-then-poll reporting,
two books, two editors, and the refusals that leave the books untouched.

Three behaviours the corpus found and now pins, each asserted as it
behaves rather than as it reads:

  * a rule firing on a store moves the *account's* whole history, not
    just the entry that matched -- categorisation here is account-level,
    so one purchase at a newly-ruled merchant reassigns the month;
  * StoreTransaction accepts a leg naming another book's account: the
    journal files under the named ledger and the foreign balance moves,
    while every read stays scoped, so the two books disagree and neither
    says so. A leg naming no account at all *is* refused, which is what
    shows the lookup runs and simply does not check the book;
  * ImportLedgerChunk commits rows as it parses them, so a malformed row
    mid-chunk leaves the rows before it posted. Recoverable via the
    content hash, which the file then demonstrates.

A wall-clock race of my own making, found by the mutation run: App
sweeps Pending report jobs once a second, so `status == 0` raced the
timer. Both files now assert only what does not race -- the id resolves,
and a job never reaches Failed.

Re-runnability needed a per-run nonce in three files: rules name their
target category by *name*, and the import's content hash is over
description/date/amount with the account id not in it, so fixed strings
made a second run resolve to the first run's rows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
 could not reach

kanban could not be driven at all. KanbanAuthorizer inherited
SigningAuthorizer without the anonymous carve-out BookmarksAuthorizer
carries, so `authorize()` demanded a token for `Login` itself -- the
only action that mints one. Every action a fresh remote client could
send was answered `unauthorized`, login included, and
ladder_kanban_server was unusable to anyone not already holding a token
minted out of band. ladder_kanban_headless says so in its own comment:
it takes the token on its command line.

bookmarks documents this exact deadlock, and says it was found by
driving a real server from a client. kanban was written later and
inherited the shape but not the fix. It gets the same carve-out, as
narrow as it can be -- one model type, one action type, compared
exactly -- and its 137 tests still pass. Login still refuses the
reserved `system:` namespace, so an anonymous caller can mint a token
for a name it chooses and nothing more.

Twenty kanban journeys follow, covering all twenty-two actions:
projects and boards, the move that is the rung's centerpiece, WIP
limits, RBAC across three roles with promotion and removal taking
effect on a handler the caller is already holding, rule cascades,
comments, attachments, both event streams, exactly-once moves, and one
integration file that walks a week of a team's work.

Three behaviours pinned as they are:

  * an unattached BoardModel answers `stoull` -- the what() of
    std::invalid_argument -- to every action. Two of them have a
    has_value() guard with the right message that never fires, because
    the optional is engaged with an empty string;
  * GetRules requires a projectId and then ignores it, answering for
    the attached board instead;
  * the audit feed includes refused actions and carries no outcome
    field, so a rejected move reads exactly like one that happened.

The protocol axis closes with three files: the four instance-management
envelope kinds with their typeId refusals, a malformed frame, and the
256-model cap walked to its limit. Four refusals gated on LimitPolicy
or PayloadCompleteness settings no rung installs are exempted with
written reasons naming the setting and the file.

Two supporting fixes. The driver now gives each run its own working
directory: kanban writes its action log to current_path(), so runs
accumulated it in the caller's directory and GetActivity replayed a
previous run's project 1 as this one's. And morph_scenario accepts the
one reply that legitimately carries no callId -- a decode failure has
no envelope to read one from -- without which that path is unassertable.

scenario_coverage.py now exits 0: 71/71 actions, every rung at its
floor, 8/8 kinds, 11/17 refusals with the rest exempt. Its own canary
test, which pinned exit 1 and said authoring the corpus was later work,
is flipped to 0 with that history recorded.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
 boundaries in its spec

The design spec said "no change to the example applications". Two
boundaries were crossed with the author's agreement and are now written
down where they contradicted the plan rather than left to be discovered
in the diff: kanban's authorizer fix, without which the rung's 22
actions and floor of 20 were unreachable, and the one new rule
morph_scenario.py learned about a reply that legitimately carries no
callId.

Neither the coverage thresholds nor the workflow floors moved.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
execute(ImportLedgerChunk) documents the per-row commit as a deliberate
decision, with the reason: a SqlTransaction nested around the loop would
have storeJournalImpl's inner Commit() end the outer transaction, since
SqlTransaction toggles SQL_ATTR_AUTOCOMMIT on the connection directly.

The scenario asserted the behaviour correctly but described it as though
it were merely observed, which invited a reader to file it. It now names
the ruling and says what a client does about it instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
Each file that asserts a defect as it behaves now names the issue that
tracks it, so a reader who finds the assertion surprising has somewhere
to go and a fix has a place to land: morph#367 (ledger legs are not
scoped to the named book), morph#368 (an unattached BoardModel replies
"stoull", and the two guards written to prevent that never fire),
morph#369 (GetRules requires a projectId and ignores it).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
The coverage report existed and was green, and nothing kept it that
way. drift-guard.yml gains a scenario-coverage job running the runner's
self-test and then scenario_coverage.py.

It belongs in this workflow rather than the build matrix because it
compiles nothing and starts nothing: the report reads remote.hpp,
wire.hpp, the BRIDGE_REGISTER_ACTION sites under examples/, the
scenario files and the allowlist, all as source. Verified on a bare
checkout with no build directory.

The gate is proven able to fail before it is trusted to pass, on the
same terms as the two jobs beside it:

  * deleting one kanban scenario -> exit 1, "kanban: 19 workflows,
    floor is 20";
  * adding an unasserted refusal to remote.hpp -> exit 1, naming it
    under UNCOVERED refusals;
  * renaming one out from under its allowlist entry -> exit 2, the
    reverse audit reporting a stale exemption.

What this does not do is run the corpus: that needs five server
binaries built, which this workflow has none of and would stop being
fast and dependency-free to acquire. The README now says so plainly, so
the gap is stated rather than assumed closed -- a scenario can still
drift from a server's real behaviour without CI noticing, only not from
its surface.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GnSnsebJ92Kti1D7m3RwYJ
@codecov

codecov Bot commented Aug 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 50.00000% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...s/kanban/include/kanban/auth/kanban_authorizer.hpp 50.00% 1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant