You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a5235d6
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: CHANGELOG.md
+3Lines changed: 3 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -32,6 +32,9 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are.
32
32
- fix(token-burn): **usage per `message.id` is each counter's max over its records (in practice the last), not the first record's — subagent output was undercounted; a late empty record cannot erase real numbers.** The header claimed every streamed record of a message carries the same usage; in subagent transcripts the early records carry a partial `output_tokens`. On the captured fixture the subagent model's output went 60 → 1,559 (headline 59,858 → 61,357); on a 2026-10 multi-agent session 37,791 → 84,915 subagent output tokens; a field measurement on harness 2.1.296 saw up to 7×. Main-thread records were identical first to last, so main numbers do not move. **Every existing subagent-heavy token count rises after sync**; fixture expectations re-derived independently (`tests/wave2/fixtures/token-burn/CAPTURE.md`) — found while promoting [#171](https://github.com/MendixMau/mxcli-project-toolkit/issues/171) (measured-claims §7), MendixMau
33
33
- new(extraction-report): **the Stage 1 surface now reads the validator's `extraction-quality*.json` — one row per dimension the file declares, a self-graded pass shown as a fault** ([#151](https://github.com/MendixMau/mxcli-project-toolkit/issues/151)). `extractor-quality-loop.md` makes that file the proof of the 95% gate, yet nothing read it: a field build scored its process-model extractor at 100% on 7 dimensions and the report showed "2 pass, 18 skipped" with no quality section. Found at `analysis/<src>/`, `analysis/<src>/extracted/` and the same two inside the KB; dimension keys come from the file. Faults only where the skill's rules show in the file: `selfGraded` + `passed`, `passed` under its own `threshold`, a 0/0 dimension absent from `confirmedZeros`, unparseable JSON. A code-extracted or mixed KB with no score file shows a `manual` section ("not produced yet — no shipped pipeline writes a validator"), not a fault: nothing mandatory produces the file yet; documents KBs get `skipped`; the gate (`gate-check.sh`, presence of the page) and the exit code are unchanged. A score file inside `<kb>/extracted/` is no longer misread as an unparseable item file. No captured validator output exists in this repo (no pipeline ships a validator), so the fixture reads the skill's documented example at run time; `tests/wave2/test-extraction-quality-report.sh` — MendixMau
34
34
- new(token-burn): **`--rates`, `--windows`, `--effort`: cost at the project's own per-Mtok rates, agent working time, human prompt count, per-window attribution — aggregates only** ([#296](https://github.com/MendixMau/mxcli-project-toolkit/issues/296)). A field team had to answer "money and time per round and per feature" with ad-hoc transcript scripts. No price is built in (rates file, model globs, first match wins; an unmatched model is UNPRICED and the total marked incomplete, never 0); agent time = main-thread gaps <= `--gap` minutes (default 5) not ending at a human prompt; human prompts = `origin.kind == "human"` user records (task notifications, peer messages, tool results, meta and compact summaries excluded), once per uuid, text never printed. Without the new flags the new code adds nothing to screen, `--json` or `--brief` output (byte-identical to the previous script before the last-record fix below; fixture T9 asserts no new section or key). Field run on a real multi-day transcript: human prompts matched an independent count (89 = 89). The human-prompt fixture records are hand-built from a profile of real record shapes — committing a scrubbed real capture was refused; `tests/wave2/test-token-burn.sh` T9-T14 — jopterhorst
35
+
- fix(journeys): **the #302 preflight no longer refuses a password the project set in `project.config.js`; only the template placeholder is refused.** As shipped, #302 refused every run whose `TEST_PASS`/`APP_PASS` was not in the environment unless `ALLOW_CONFIG_CREDENTIALS=1` was set. But `project.config.template.js` tells every project to put its real demo password in `DEFAULT_PASS`, nothing in the toolkit loads an env file before `verify-module.sh` runs the journeys, and `sync-project.sh` refreshes unedited runner/config copies. So after the next sync every journeys rung would have faulted with rc 2. Now `config.js` reports `passSource` (`env` / `project.config.js` / `template placeholder`) and only the template's placeholder (`Demo2025!`) is refused. The env still wins, and `ALLOW_CONFIG_CREDENTIALS=1` still overrides. **Verdict change to know about from #302:** a refused admin credential (HTTP 401/403) used to leave every data rung INVALID with exit 1; the suite now aborts before the first journey with exit 2. A SIGINT/SIGTERM now writes the findings slot as `aborted: "signal SIGINT"` instead of leaving the previous run's file there. `report-normalize.js` keeps `rung: 'setup'` rows as kind `setup` instead of folding them into `ui` (`report-schema.md` enum updated). It also honours `aborted` in a findings file: the report gets a top-level `aborted`, every kept journey row is flagged `notAResult`, and the journey-runner instrument is a fault, so an aborted run never reads as results. It also reads the per-BRD conformance reports `report-<date>-<BRD>.tsv` that conformance-check now writes ([#307](https://github.com/MendixMau/mxcli-project-toolkit/pull/307)). It used to keep only `report-<date>.tsv`, so per-BRD modules never reached report.json and a stale base file could show as measured. It now takes the newest date among `report-<date>[-<any suffix>].tsv` and aggregates every file of that date, including BRD ids that contain a space. A per-BRD pointer `<BRD>:`/x`` keeps its requirement `/x`. Single-file projects are unchanged. Golden input: conformance-check.sh's real per-BRD output on #307's own fixture layout (`tests/wave2/fixtures/conformance-per-brd/`), plus `--selftest` cases and case 3 of `test-report-normalize-two-tree.sh`. Fixtures: `test-journey-preflight.sh` 74 cases (9 new, all failing on the #302 code: project-set password runs, placeholder refuses, env wins, SIGINT writes aborted findings end to end) and `report-normalize.js --selftest` cases (setup kind, aborted run). Refs [#286](https://github.com/MendixMau/mxcli-project-toolkit/issues/286) — MendixMau
36
+
- fix(e2e): **`helpers.js` reads OQL rows by SELECT alias, never by position.** `mxcli oql --direct --json` prints keys alphabetically, leaves out a column that is null in every row, and over a view entity adds `ID` beside the alias. `Object.values(r)` therefore shifted columns, and `oqlScalar` returned a view row's ID instead of its count; a string value holding `]` or `}` truncated the brace-counting slice, so a real answer read as "no measurement" ([#154](https://github.com/MendixMau/mxcli-project-toolkit/issues/154), card-disbursement build). Rows are now keyed by the parsed SELECT aliases in SELECT order with explicit nulls, `oqlScalar` reads the first alias (null rather than a guess when the list has no alias), new `oqlRows`, string-aware JSON slicing; the admin preflight uses the same parser. **Behaviour change to check in your project:** `oqlScalar` now returns the **first SELECT alias**, where it used to return the alphabetically-first key. A multi-column seed, scope capture or baseline query whose value flows into `{{vars}}` may now resolve to a different column. Put the column you mean first, or make the query single-column. Golden: the released mxcli v0.26.0 binary's verbatim stdout/stderr for four queries against a stub admin API (`tests/wave2/fixtures/oql-json/`); fixture `tests/wave2/test-oql-rows.sh` (15 cases, 0/15 on the old helper). No live-app field run yet — MendixMau
37
+
- fix(e2e): **a negative-only span claim is judged against a liveness reference, and an app that dies mid-suite reads APP-DOWN, not FAIL.** A `mustNotFire`-only step (Cancel on a reset confirmation) captures zero spans, which also happens when Jaeger is down, so it was INVALID on every run and its control mutant was unsupported ([#150](https://github.com/MendixMau/mxcli-project-toolkit/issues/150)). Rung 2 now waits the full window and requires a declared `spans.liveness` span or any service span since the journey began (`otel.js liveness`; the window starts at the journey's own start with a 2 s clock-skew margin); no evidence stays INVALID. An `otel.js` older than the runner, i.e. a hand-edited copy kept by sync, reads INVALID naming the stale file instead of throwing a TypeError. The `trace-negative` mutant injects the forbidden name after that check. A `spans` block declaring neither `ordered` nor `mustNotFire` is now a loud INVALID instead of a silent skip; the array shape `[{mf,status}]` and `{}` are named as such. `examples/validate-journeys.js` accepts a negative-only claim, which it used to reject as "NEVER checked" (contradicting the runner). It also checks the new `spans.liveness` field (only read for a negative-only claim) and a `mustNotFire` that is not a valid regex. `skills/journey-examples.md` documents the liveness rule. Fixture `tests/wave2/test-validate-journeys-spans.sh` (12 cases, 8 fail on the old validator). The shipped `project-tests/e2e/example.journey.json` is rewritten into the shape the runner reads. Its `name`/`role`, `value` seed, `page`/`expectPage`/`expectText`/`click` steps, array-shaped `spans` and SQL-style `<module>$<entity>` queries had drifted from it: the validator reported 3 problems on it, now 0, and all 7 control mutants are supported on it. Between journeys the runner probes `baseUrl` (3 tries); down before a journey → `APP-DOWN` INVALID, not walked; down after one → its FAIL rows are flagged ([#285](https://github.com/MendixMau/mxcli-project-toolkit/issues/285) harness half; a watch rebuild took a runtime down mid-suite on 2026-10-12). `APP_PROBE=0` disables it. Fixture `tests/wave2/test-journey-liveness.sh` (36 cases over the captured Jaeger trace and a real local HTTP server); main-loop wiring not fixtured (needs a browser), no field run yet — MendixMau
35
38
- learn(testing-shape, test-result-audit): **three test-strength rules, each with a denominator: refusals probed on the server, every undeclared 4xx a failure, one clean-database final run** ([#172](https://github.com/MendixMau/mxcli-project-toolkit/issues/172)). A field comparison of two builds against one frozen spec found the toolkit build checked permission refusals only in the client (a hidden menu item), logged unexpected 400/422 responses as information, and reached 20/20 by stitching three runs. `testing-shape.md` §7 gains three report lines (`N of N restricted pairs probed server-side`, `undeclared 4xx/5xx: 0 of M requests — each fails its step`, `one clean-DB run <id>: N of N scenarios incl. timers, not stitched`); `test-result-audit.md` gains check 2e, which carries the reasons and the field evidence and asks for those numbers. Both baseline skills stay inside the 80,000-word baseline budget (79,997): the reasons live in the on-demand audit skill, and two redundant sentences in `testing-shape.md` §4's verdict paragraphs were folded (the empty-set corollary is stated in the Trace row, `harness-architecture.md` and `report-schema.md`). The backing inbox note is trimmed to its unpromoted "Keep" items. — MendixMau
36
39
- learn(learned-detection-gaps, module-review): **two UI defects every model gate passes: a lazy-loaded combo box filter hides its preset, and a theme that pins row heights by position lets the next row cover an inserted widget** ([#299](https://github.com/MendixMau/mxcli-project-toolkit/issues/299)). Both passed check, mxbuild, lint and the scripted checks on a 127-module production app and were found only in the running app. Two register rows with the first oracle and the pre-build check (pluggable properties that change *when* data loads; grep the theme for fixed `height` on positional row classes before an insert), and two LOOK lines in `module-review.md` §4c with denominators (`N of N filters show their preset on first load`, `N of N widgets inserted into an existing container fully visible`). — jopterhorst
37
40
- fix(app-restart, test-stack-up): **a port held by another process before `run --local` is named and reported as "instrument blocked" (exit 4), never killed** ([#262](https://github.com/MendixMau/mxcli-project-toolkit/issues/262)). Studio Pro had started its own runtime on the app/admin ports a restart needed; `mxcli run --local` then failed with "Authentication failed" (the admin API answering was Studio Pro's), the next journey ran against Studio Pro's runtime, and four more runs were NOT RUN. `app-restart.sh` now checks the app port and the admin port (record's `adminPort`, else app + 10) before stopping anything — a holder outside the recorded process tree exits 4 with its name, pid and parent — and again after the stop, where anything still holding a port is by definition not ours. `test-stack-up.sh` runs the same check wherever it would tell you to start `run --local` (`--no-docker`, no Docker daemon, `--check` with a leftover record), on the record's ports, `RUN_LOCAL_APP_PORT`/`RUN_LOCAL_ADMIN_PORT`, or mxcli's defaults 8080/8090. Holder lookup: lsof, then ss, then netstat + tasklist on Windows; unnamed holders still block when nothing of ours is running. Fixtures: `test-app-restart.sh` +2 cases (14 assertions, all fail on the old script), `test-stack-up-local.sh` T7 (4 of 6 fail on the old script). — AlvinnYeo
if(!seen.has(k))add(at,`references {{${k}}}, which is not an EARLIER seed — seeds resolve in array order`);
57
57
if(s.name)seen.add(s.name);
@@ -107,13 +107,29 @@ function validate(j, brds) {
107
107
if(typeoft!=='string')
108
108
add(`${at}.ui.${key}`,'contains a non-string — comment objects inside these arrays become assertions');
109
109
110
+
// A negative-only claim (`mustNotFire` with no `ordered`) is evaluated since #150: the
111
+
// runner judges it once tracing is shown live — the step's `liveness` span, or any span from
112
+
// the service since the journey began. `liveness` is read only for such a claim.
110
113
constsp=st.spans;
111
-
if(sp){
114
+
if(Array.isArray(sp)){
115
+
add(`${at}.spans`,'is an array — the old shape [{ mf, status }], which the runner never read (it records it INVALID); use { "ordered": [...], "mustNotFire": [...] }');
116
+
}elseif(sp){
112
117
constordered=sp.ordered||[];
113
-
if(!Array.isArray(ordered)||!ordered.length){
114
-
if((sp.mustNotFire||[]).length)
115
-
add(`${at}.spans`,'declares `mustNotFire` with no non-empty `ordered` — the runner evaluates the whole trace block only when `ordered` is non-empty, so this claim is NEVER checked');
116
-
elseadd(`${at}.spans`,'is present but declares nothing the runner reads');
118
+
constmustNot=sp.mustNotFire||[];
119
+
if(!Array.isArray(ordered))add(`${at}.spans.ordered`,'must be an array');
120
+
if(!Array.isArray(mustNot))add(`${at}.spans.mustNotFire`,'must be an array');
add(`${at}.spans.liveness`,'must be a non-empty string: a microflow name or span operation the step is known to produce');
131
+
elseif(hasOrdered)
132
+
add(`${at}.spans.liveness`,'is only read for a negative-only claim (`mustNotFire` with no `ordered`) — with `ordered`, the claimed microflows are the liveness evidence');
117
133
}
118
134
for(constnofordered)
119
135
if(!String(n).includes('.'))
@@ -206,7 +222,12 @@ function selftest() {
206
222
['outcome with no sql',j=>{deletej.outcome.sql;returnj;},'reads `outcome.sql`'],
207
223
['ready given a bare widget name',j=>{j.steps[0].ui.ready='dvThing';returnj;},'is not a CSS selector'],
208
224
['checks given a selector',j=>{j.steps[0].ui.checks=['.mx-name-btnGo'];returnj;},'looks like a selector'],
209
-
['mustNotFire with no ordered',j=>{j.steps[0].spans={mustNotFire: ['ACT_Del']};returnj;},'NEVER checked'],
225
+
['negative-only spans claim is accepted',j=>{j.steps[0].spans={mustNotFire: ['ACT_Del']};returnj;},null],
226
+
['negative-only claim with liveness is accepted',j=>{j.steps[0].spans={ordered: [],mustNotFire: ['ACT_Del'],liveness: 'M.DS_Page'};returnj;},null],
227
+
['spans block declaring nothing',j=>{j.steps[0].spans={ordered: []};returnj;},'declares nothing the runner reads'],
228
+
['liveness not a string',j=>{j.steps[0].spans={mustNotFire: ['ACT_Del'],liveness: 5};returnj;},'must be a non-empty string'],
229
+
['liveness beside ordered',j=>{j.steps[0].spans.liveness='M.DS_Page';returnj;},'only read for a negative-only claim'],
230
+
['mustNotFire not a regex',j=>{j.steps[0].spans.mustNotFire=['ACT_(Del'];returnj;},'not a valid regex'],
0 commit comments