Skip to content

Commit a5235d6

Browse files
committed
Merge #310: OQL rows by alias; liveness for negative-only spans; APP-DOWN; #302 credential fix; aborted reports; per-BRD conformance in report.json (closes #154 #150, refs #285 #286)
Review: MERGE (initial review + fix round + 3 confirmation rounds) — project-set DEFAULT_PASS runs again (urgent #302 regression fixed); normal report.json shape unchanged except 'aborted': null; per-BRD golden reproduced byte-for-byte from #307's conformance-check.
2 parents a349411 + 84dc6b6 commit a5235d6

32 files changed

Lines changed: 1304 additions & 140 deletions

‎CHANGELOG.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,9 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are.
3232
- fix(token-burn): **usage per `message.id` is each counter's max over its records (in practice the last), not the first record's — subagent output was undercounted; a late empty record cannot erase real numbers.** The header claimed every streamed record of a message carries the same usage; in subagent transcripts the early records carry a partial `output_tokens`. On the captured fixture the subagent model's output went 60 → 1,559 (headline 59,858 → 61,357); on a 2026-10 multi-agent session 37,791 → 84,915 subagent output tokens; a field measurement on harness 2.1.296 saw up to 7×. Main-thread records were identical first to last, so main numbers do not move. **Every existing subagent-heavy token count rises after sync**; fixture expectations re-derived independently (`tests/wave2/fixtures/token-burn/CAPTURE.md`) — found while promoting [#171](https://github.com/MendixMau/mxcli-project-toolkit/issues/171) (measured-claims §7), MendixMau
3333
- new(extraction-report): **the Stage 1 surface now reads the validator's `extraction-quality*.json` — one row per dimension the file declares, a self-graded pass shown as a fault** ([#151](https://github.com/MendixMau/mxcli-project-toolkit/issues/151)). `extractor-quality-loop.md` makes that file the proof of the 95% gate, yet nothing read it: a field build scored its process-model extractor at 100% on 7 dimensions and the report showed "2 pass, 18 skipped" with no quality section. Found at `analysis/<src>/`, `analysis/<src>/extracted/` and the same two inside the KB; dimension keys come from the file. Faults only where the skill's rules show in the file: `selfGraded` + `passed`, `passed` under its own `threshold`, a 0/0 dimension absent from `confirmedZeros`, unparseable JSON. A code-extracted or mixed KB with no score file shows a `manual` section ("not produced yet — no shipped pipeline writes a validator"), not a fault: nothing mandatory produces the file yet; documents KBs get `skipped`; the gate (`gate-check.sh`, presence of the page) and the exit code are unchanged. A score file inside `<kb>/extracted/` is no longer misread as an unparseable item file. No captured validator output exists in this repo (no pipeline ships a validator), so the fixture reads the skill's documented example at run time; `tests/wave2/test-extraction-quality-report.sh` — MendixMau
3434
- new(token-burn): **`--rates`, `--windows`, `--effort`: cost at the project's own per-Mtok rates, agent working time, human prompt count, per-window attribution — aggregates only** ([#296](https://github.com/MendixMau/mxcli-project-toolkit/issues/296)). A field team had to answer "money and time per round and per feature" with ad-hoc transcript scripts. No price is built in (rates file, model globs, first match wins; an unmatched model is UNPRICED and the total marked incomplete, never 0); agent time = main-thread gaps <= `--gap` minutes (default 5) not ending at a human prompt; human prompts = `origin.kind == "human"` user records (task notifications, peer messages, tool results, meta and compact summaries excluded), once per uuid, text never printed. Without the new flags the new code adds nothing to screen, `--json` or `--brief` output (byte-identical to the previous script before the last-record fix below; fixture T9 asserts no new section or key). Field run on a real multi-day transcript: human prompts matched an independent count (89 = 89). The human-prompt fixture records are hand-built from a profile of real record shapes — committing a scrubbed real capture was refused; `tests/wave2/test-token-burn.sh` T9-T14 — jopterhorst
35+
- fix(journeys): **the #302 preflight no longer refuses a password the project set in `project.config.js`; only the template placeholder is refused.** As shipped, #302 refused every run whose `TEST_PASS`/`APP_PASS` was not in the environment unless `ALLOW_CONFIG_CREDENTIALS=1` was set. But `project.config.template.js` tells every project to put its real demo password in `DEFAULT_PASS`, nothing in the toolkit loads an env file before `verify-module.sh` runs the journeys, and `sync-project.sh` refreshes unedited runner/config copies. So after the next sync every journeys rung would have faulted with rc 2. Now `config.js` reports `passSource` (`env` / `project.config.js` / `template placeholder`) and only the template's placeholder (`Demo2025!`) is refused. The env still wins, and `ALLOW_CONFIG_CREDENTIALS=1` still overrides. **Verdict change to know about from #302:** a refused admin credential (HTTP 401/403) used to leave every data rung INVALID with exit 1; the suite now aborts before the first journey with exit 2. A SIGINT/SIGTERM now writes the findings slot as `aborted: "signal SIGINT"` instead of leaving the previous run's file there. `report-normalize.js` keeps `rung: 'setup'` rows as kind `setup` instead of folding them into `ui` (`report-schema.md` enum updated). It also honours `aborted` in a findings file: the report gets a top-level `aborted`, every kept journey row is flagged `notAResult`, and the journey-runner instrument is a fault, so an aborted run never reads as results. It also reads the per-BRD conformance reports `report-<date>-<BRD>.tsv` that conformance-check now writes ([#307](https://github.com/MendixMau/mxcli-project-toolkit/pull/307)). It used to keep only `report-<date>.tsv`, so per-BRD modules never reached report.json and a stale base file could show as measured. It now takes the newest date among `report-<date>[-<any suffix>].tsv` and aggregates every file of that date, including BRD ids that contain a space. A per-BRD pointer `<BRD>:`/x`` keeps its requirement `/x`. Single-file projects are unchanged. Golden input: conformance-check.sh's real per-BRD output on #307's own fixture layout (`tests/wave2/fixtures/conformance-per-brd/`), plus `--selftest` cases and case 3 of `test-report-normalize-two-tree.sh`. Fixtures: `test-journey-preflight.sh` 74 cases (9 new, all failing on the #302 code: project-set password runs, placeholder refuses, env wins, SIGINT writes aborted findings end to end) and `report-normalize.js --selftest` cases (setup kind, aborted run). Refs [#286](https://github.com/MendixMau/mxcli-project-toolkit/issues/286) — MendixMau
36+
- fix(e2e): **`helpers.js` reads OQL rows by SELECT alias, never by position.** `mxcli oql --direct --json` prints keys alphabetically, leaves out a column that is null in every row, and over a view entity adds `ID` beside the alias. `Object.values(r)` therefore shifted columns, and `oqlScalar` returned a view row's ID instead of its count; a string value holding `]` or `}` truncated the brace-counting slice, so a real answer read as "no measurement" ([#154](https://github.com/MendixMau/mxcli-project-toolkit/issues/154), card-disbursement build). Rows are now keyed by the parsed SELECT aliases in SELECT order with explicit nulls, `oqlScalar` reads the first alias (null rather than a guess when the list has no alias), new `oqlRows`, string-aware JSON slicing; the admin preflight uses the same parser. **Behaviour change to check in your project:** `oqlScalar` now returns the **first SELECT alias**, where it used to return the alphabetically-first key. A multi-column seed, scope capture or baseline query whose value flows into `{{vars}}` may now resolve to a different column. Put the column you mean first, or make the query single-column. Golden: the released mxcli v0.26.0 binary's verbatim stdout/stderr for four queries against a stub admin API (`tests/wave2/fixtures/oql-json/`); fixture `tests/wave2/test-oql-rows.sh` (15 cases, 0/15 on the old helper). No live-app field run yet — MendixMau
37+
- fix(e2e): **a negative-only span claim is judged against a liveness reference, and an app that dies mid-suite reads APP-DOWN, not FAIL.** A `mustNotFire`-only step (Cancel on a reset confirmation) captures zero spans, which also happens when Jaeger is down, so it was INVALID on every run and its control mutant was unsupported ([#150](https://github.com/MendixMau/mxcli-project-toolkit/issues/150)). Rung 2 now waits the full window and requires a declared `spans.liveness` span or any service span since the journey began (`otel.js liveness`; the window starts at the journey's own start with a 2 s clock-skew margin); no evidence stays INVALID. An `otel.js` older than the runner, i.e. a hand-edited copy kept by sync, reads INVALID naming the stale file instead of throwing a TypeError. The `trace-negative` mutant injects the forbidden name after that check. A `spans` block declaring neither `ordered` nor `mustNotFire` is now a loud INVALID instead of a silent skip; the array shape `[{mf,status}]` and `{}` are named as such. `examples/validate-journeys.js` accepts a negative-only claim, which it used to reject as "NEVER checked" (contradicting the runner). It also checks the new `spans.liveness` field (only read for a negative-only claim) and a `mustNotFire` that is not a valid regex. `skills/journey-examples.md` documents the liveness rule. Fixture `tests/wave2/test-validate-journeys-spans.sh` (12 cases, 8 fail on the old validator). The shipped `project-tests/e2e/example.journey.json` is rewritten into the shape the runner reads. Its `name`/`role`, `value` seed, `page`/`expectPage`/`expectText`/`click` steps, array-shaped `spans` and SQL-style `<module>$<entity>` queries had drifted from it: the validator reported 3 problems on it, now 0, and all 7 control mutants are supported on it. Between journeys the runner probes `baseUrl` (3 tries); down before a journey → `APP-DOWN` INVALID, not walked; down after one → its FAIL rows are flagged ([#285](https://github.com/MendixMau/mxcli-project-toolkit/issues/285) harness half; a watch rebuild took a runtime down mid-suite on 2026-10-12). `APP_PROBE=0` disables it. Fixture `tests/wave2/test-journey-liveness.sh` (36 cases over the captured Jaeger trace and a real local HTTP server); main-loop wiring not fixtured (needs a browser), no field run yet — MendixMau
3538
- learn(testing-shape, test-result-audit): **three test-strength rules, each with a denominator: refusals probed on the server, every undeclared 4xx a failure, one clean-database final run** ([#172](https://github.com/MendixMau/mxcli-project-toolkit/issues/172)). A field comparison of two builds against one frozen spec found the toolkit build checked permission refusals only in the client (a hidden menu item), logged unexpected 400/422 responses as information, and reached 20/20 by stitching three runs. `testing-shape.md` §7 gains three report lines (`N of N restricted pairs probed server-side`, `undeclared 4xx/5xx: 0 of M requests — each fails its step`, `one clean-DB run <id>: N of N scenarios incl. timers, not stitched`); `test-result-audit.md` gains check 2e, which carries the reasons and the field evidence and asks for those numbers. Both baseline skills stay inside the 80,000-word baseline budget (79,997): the reasons live in the on-demand audit skill, and two redundant sentences in `testing-shape.md` §4's verdict paragraphs were folded (the empty-set corollary is stated in the Trace row, `harness-architecture.md` and `report-schema.md`). The backing inbox note is trimmed to its unpromoted "Keep" items. — MendixMau
3639
- learn(learned-detection-gaps, module-review): **two UI defects every model gate passes: a lazy-loaded combo box filter hides its preset, and a theme that pins row heights by position lets the next row cover an inserted widget** ([#299](https://github.com/MendixMau/mxcli-project-toolkit/issues/299)). Both passed check, mxbuild, lint and the scripted checks on a 127-module production app and were found only in the running app. Two register rows with the first oracle and the pre-build check (pluggable properties that change *when* data loads; grep the theme for fixed `height` on positional row classes before an insert), and two LOOK lines in `module-review.md` §4c with denominators (`N of N filters show their preset on first load`, `N of N widgets inserted into an existing container fully visible`). — jopterhorst
3740
- fix(app-restart, test-stack-up): **a port held by another process before `run --local` is named and reported as "instrument blocked" (exit 4), never killed** ([#262](https://github.com/MendixMau/mxcli-project-toolkit/issues/262)). Studio Pro had started its own runtime on the app/admin ports a restart needed; `mxcli run --local` then failed with "Authentication failed" (the admin API answering was Studio Pro's), the next journey ran against Studio Pro's runtime, and four more runs were NOT RUN. `app-restart.sh` now checks the app port and the admin port (record's `adminPort`, else app + 10) before stopping anything — a holder outside the recorded process tree exits 4 with its name, pid and parent — and again after the stop, where anything still holding a port is by definition not ours. `test-stack-up.sh` runs the same check wherever it would tell you to start `run --local` (`--no-docker`, no Docker daemon, `--check` with a leftover record), on the record's ports, `RUN_LOCAL_APP_PORT`/`RUN_LOCAL_ADMIN_PORT`, or mxcli's defaults 8080/8090. Holder lookup: lsof, then ss, then netstat + tasklist on Windows; unnamed holders still block when nothing of ours is running. Fixtures: `test-app-restart.sh` +2 cases (14 assertions, all fail on the old script), `test-stack-up-local.sh` T7 (4 of 6 fail on the old script). — AlvinnYeo

‎examples/journeys/full.journey.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,8 +39,8 @@
3939
"name": "supplierCode",
4040
"sql": "SELECT SupplierCode AS c FROM Procurement.Supplier WHERE Active = true LIMIT 1",
4141
"_note": [
42-
"Every OQL column must be aliased — helpers.oqlScalar reads the first cell of the first",
43-
"row, and mxcli rejects an unaliased SELECT. A seed that returns null or an empty",
42+
"Every OQL column must be aliased — helpers.oqlScalar reads the first SELECT alias of the",
43+
"first row, and mxcli rejects an unaliased SELECT. A seed that returns null or an empty",
4444
"string aborts the journey as INVALID (instrument absent), never FAIL (feature broken)."
4545
]
4646
},

‎examples/validate-journeys.js‎

Lines changed: 30 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
// rule below encodes a way a journey can be wrong that the runner cannot tell
1515
// apart from a defect: a placeholder that reaches the page as the literal string
1616
// "{{supplierCode}}", an OQL check with no bar that is recorded INVALID forever, a
17-
// `mustNotFire` the runner never evaluates.
17+
// `spans` block that declares nothing the runner reads.
1818
//
1919
// WHAT IT IS NOT
2020
// It cannot tell you whether a selector, entity, microflow or requirement pointer
@@ -51,7 +51,7 @@ function validate(j, brds) {
5151
if (!s.sql) add(at, 'no `sql`');
5252
if (s.name && seen.has(s.name)) add(at, `duplicate seed name "${s.name}"`);
5353
if (s.sql && !/\bAS\b/i.test(s.sql))
54-
add(at, 'SQL has no column alias — oqlScalar reads the first cell and mxcli rejects an unaliased SELECT');
54+
add(at, 'SQL has no column alias — oqlScalar reads the first SELECT alias and mxcli rejects an unaliased SELECT');
5555
for (const k of [...String(s.sql || '').matchAll(/\{\{(\w+)\}\}/g)].map(m => m[1]))
5656
if (!seen.has(k)) add(at, `references {{${k}}}, which is not an EARLIER seed — seeds resolve in array order`);
5757
if (s.name) seen.add(s.name);
@@ -107,13 +107,29 @@ function validate(j, brds) {
107107
if (typeof t !== 'string')
108108
add(`${at}.ui.${key}`, 'contains a non-string — comment objects inside these arrays become assertions');
109109

110+
// A negative-only claim (`mustNotFire` with no `ordered`) is evaluated since #150: the
111+
// runner judges it once tracing is shown live — the step's `liveness` span, or any span from
112+
// the service since the journey began. `liveness` is read only for such a claim.
110113
const sp = st.spans;
111-
if (sp) {
114+
if (Array.isArray(sp)) {
115+
add(`${at}.spans`, 'is an array — the old shape [{ mf, status }], which the runner never read (it records it INVALID); use { "ordered": [...], "mustNotFire": [...] }');
116+
} else if (sp) {
112117
const ordered = sp.ordered || [];
113-
if (!Array.isArray(ordered) || !ordered.length) {
114-
if ((sp.mustNotFire || []).length)
115-
add(`${at}.spans`, 'declares `mustNotFire` with no non-empty `ordered` — the runner evaluates the whole trace block only when `ordered` is non-empty, so this claim is NEVER checked');
116-
else add(`${at}.spans`, 'is present but declares nothing the runner reads');
118+
const mustNot = sp.mustNotFire || [];
119+
if (!Array.isArray(ordered)) add(`${at}.spans.ordered`, 'must be an array');
120+
if (!Array.isArray(mustNot)) add(`${at}.spans.mustNotFire`, 'must be an array');
121+
const hasOrdered = Array.isArray(ordered) && ordered.length > 0;
122+
const hasMustNot = Array.isArray(mustNot) && mustNot.length > 0;
123+
if (!hasOrdered && !hasMustNot)
124+
add(`${at}.spans`, 'is present but declares nothing the runner reads (neither `ordered` nor `mustNotFire`) — the runner records it INVALID');
125+
for (const n of hasMustNot ? mustNot : []) {
126+
try { new RegExp(n); } catch (e) { add(`${at}.spans.mustNotFire`, `"${n}" is not a valid regex — the runner compiles it with new RegExp()`); }
127+
}
128+
if (sp.liveness !== undefined) {
129+
if (typeof sp.liveness !== 'string' || !sp.liveness.trim())
130+
add(`${at}.spans.liveness`, 'must be a non-empty string: a microflow name or span operation the step is known to produce');
131+
else if (hasOrdered)
132+
add(`${at}.spans.liveness`, 'is only read for a negative-only claim (`mustNotFire` with no `ordered`) — with `ordered`, the claimed microflows are the liveness evidence');
117133
}
118134
for (const n of ordered)
119135
if (!String(n).includes('.'))
@@ -206,7 +222,12 @@ function selftest() {
206222
['outcome with no sql', j => { delete j.outcome.sql; return j; }, 'reads `outcome.sql`'],
207223
['ready given a bare widget name', j => { j.steps[0].ui.ready = 'dvThing'; return j; }, 'is not a CSS selector'],
208224
['checks given a selector', j => { j.steps[0].ui.checks = ['.mx-name-btnGo']; return j; }, 'looks like a selector'],
209-
['mustNotFire with no ordered', j => { j.steps[0].spans = { mustNotFire: ['ACT_Del'] }; return j; }, 'NEVER checked'],
225+
['negative-only spans claim is accepted', j => { j.steps[0].spans = { mustNotFire: ['ACT_Del'] }; return j; }, null],
226+
['negative-only claim with liveness is accepted', j => { j.steps[0].spans = { ordered: [], mustNotFire: ['ACT_Del'], liveness: 'M.DS_Page' }; return j; }, null],
227+
['spans block declaring nothing', j => { j.steps[0].spans = { ordered: [] }; return j; }, 'declares nothing the runner reads'],
228+
['liveness not a string', j => { j.steps[0].spans = { mustNotFire: ['ACT_Del'], liveness: 5 }; return j; }, 'must be a non-empty string'],
229+
['liveness beside ordered', j => { j.steps[0].spans.liveness = 'M.DS_Page'; return j; }, 'only read for a negative-only claim'],
230+
['mustNotFire not a regex', j => { j.steps[0].spans.mustNotFire = ['ACT_(Del']; return j; }, 'not a valid regex'],
210231
['unqualified ordered span', j => { j.steps[0].spans.ordered = ['ACT_Go']; return j; }, 'is unqualified'],
211232
['unknown action', j => { j.steps[0].actions[0].do = 'swipe'; return j; }, 'unknown action'],
212233
['wait using settleMs', j => { j.steps[0].actions = [{ do: 'wait', settleMs: 500 }]; return j; }, 'reads `ms`'],
@@ -224,7 +245,7 @@ function selftest() {
224245
console.log(`${ok ? 'ok ' : 'FAIL'} ${name}${ok ? '' : `\n got: ${problems.join('\n ') || '(no problems reported)'}`}`);
225246
}
226247
console.log(bad ? `\n${bad} rule(s) could not be shown failing — do not trust a green from this tool.`
227-
: `\n${cases.length - 1} rules each watched going red; the clean case stays green.`);
248+
: `\n${cases.filter(c => c[2] !== null).length} rules each watched going red; ${cases.filter(c => c[2] === null).length} clean case(s) stay green.`);
228249
return bad ? 1 : 0;
229250
}
230251

0 commit comments

Comments
 (0)