Skip to content

Commit ba461e5

Browse files
committed
report: honour aborted findings; setup kind in the schema; drop dead credentials branch
- report-normalize.js: a findings file with `aborted` (preflight refusal or a signal) yields a report with top-level `aborted`, every kept journey row flagged notAResult/abortedRun, and the journey-runner instrument a fault — an aborted run never reads as results. --selftest cases; and test-journey-preflight.sh runs the real runner's aborted file through the real normalizer (fails on master's normalizer). - skills/report-schema.md: kind enum gains `setup`; `aborted` documented. - config.js: drop the `credentials().source === 'template default'` branch no template emits; the Demo2025! placeholder rule stays. Refs #286 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
1 parent cc56d9b commit ba461e5

5 files changed

Lines changed: 43 additions & 6 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are.
1717

1818
## Unreleased
1919

20-
- fix(journeys): **the #302 preflight no longer refuses a password the project set in `project.config.js`; only the template placeholder is refused.** As shipped, #302 refused every run whose `TEST_PASS`/`APP_PASS` was not in the environment unless `ALLOW_CONFIG_CREDENTIALS=1` was set. But `project.config.template.js` tells every project to put its real demo password in `DEFAULT_PASS`, nothing in the toolkit loads an env file before `verify-module.sh` runs the journeys, and `sync-project.sh` refreshes unedited runner/config copies. So after the next sync every journeys rung would have faulted with rc 2. Now `config.js` reports `passSource` (`env` / `project.config.js` / `template placeholder`) and only the placeholder (`Demo2025!`, or a `credentials()` reporting `source: 'template default'`) is refused. The env still wins, and `ALLOW_CONFIG_CREDENTIALS=1` still overrides. **Verdict change to know about from #302:** a refused admin credential (HTTP 401/403) used to leave every data rung INVALID with exit 1; the suite now aborts before the first journey with exit 2. A SIGINT/SIGTERM now writes the findings slot as `aborted: "signal SIGINT"` instead of leaving the previous run's file there. `report-normalize.js` keeps `rung: 'setup'` rows as kind `setup` instead of folding them into `ui`. Fixtures: `test-journey-preflight.sh` 74 cases (9 new, all failing on the #302 code: project-set password runs, placeholder refuses, env wins, SIGINT writes aborted findings end to end) and a `report-normalize.js --selftest` case. Refs [#286](https://github.com/MendixMau/mxcli-project-toolkit/issues/286) — MendixMau
20+
- fix(journeys): **the #302 preflight no longer refuses a password the project set in `project.config.js`; only the template placeholder is refused.** As shipped, #302 refused every run whose `TEST_PASS`/`APP_PASS` was not in the environment unless `ALLOW_CONFIG_CREDENTIALS=1` was set. But `project.config.template.js` tells every project to put its real demo password in `DEFAULT_PASS`, nothing in the toolkit loads an env file before `verify-module.sh` runs the journeys, and `sync-project.sh` refreshes unedited runner/config copies. So after the next sync every journeys rung would have faulted with rc 2. Now `config.js` reports `passSource` (`env` / `project.config.js` / `template placeholder`) and only the template's placeholder (`Demo2025!`) is refused. The env still wins, and `ALLOW_CONFIG_CREDENTIALS=1` still overrides. **Verdict change to know about from #302:** a refused admin credential (HTTP 401/403) used to leave every data rung INVALID with exit 1; the suite now aborts before the first journey with exit 2. A SIGINT/SIGTERM now writes the findings slot as `aborted: "signal SIGINT"` instead of leaving the previous run's file there. `report-normalize.js` keeps `rung: 'setup'` rows as kind `setup` instead of folding them into `ui` (`report-schema.md` enum updated). It also honours `aborted` in a findings file: the report gets a top-level `aborted`, every kept journey row is flagged `notAResult`, and the journey-runner instrument is a fault, so an aborted run never reads as results. Fixtures: `test-journey-preflight.sh` 74 cases (9 new, all failing on the #302 code: project-set password runs, placeholder refuses, env wins, SIGINT writes aborted findings end to end) and `report-normalize.js --selftest` cases (setup kind, aborted run). Refs [#286](https://github.com/MendixMau/mxcli-project-toolkit/issues/286) — MendixMau
2121
- fix(e2e): **`helpers.js` reads OQL rows by SELECT alias, never by position.** `mxcli oql --direct --json` prints keys alphabetically, leaves out a column that is null in every row, and over a view entity adds `ID` beside the alias. `Object.values(r)` therefore shifted columns, and `oqlScalar` returned a view row's ID instead of its count; a string value holding `]` or `}` truncated the brace-counting slice, so a real answer read as "no measurement" ([#154](https://github.com/MendixMau/mxcli-project-toolkit/issues/154), card-disbursement build). Rows are now keyed by the parsed SELECT aliases in SELECT order with explicit nulls, `oqlScalar` reads the first alias (null rather than a guess when the list has no alias), new `oqlRows`, string-aware JSON slicing; the admin preflight uses the same parser. **Behaviour change to check in your project:** `oqlScalar` now returns the **first SELECT alias**, where it used to return the alphabetically-first key. A multi-column seed, scope capture or baseline query whose value flows into `{{vars}}` may now resolve to a different column. Put the column you mean first, or make the query single-column. Golden: the released mxcli v0.26.0 binary's verbatim stdout/stderr for four queries against a stub admin API (`tests/wave2/fixtures/oql-json/`); fixture `tests/wave2/test-oql-rows.sh` (15 cases, 0/15 on the old helper). No live-app field run yet — MendixMau
2222
- fix(e2e): **a negative-only span claim is judged against a liveness reference, and an app that dies mid-suite reads APP-DOWN, not FAIL.** A `mustNotFire`-only step (Cancel on a reset confirmation) captures zero spans, which also happens when Jaeger is down, so it was INVALID on every run and its control mutant was unsupported ([#150](https://github.com/MendixMau/mxcli-project-toolkit/issues/150)). Rung 2 now waits the full window and requires a declared `spans.liveness` span or any service span since the journey began (`otel.js liveness`; the window starts at the journey's own start with a 2 s clock-skew margin); no evidence stays INVALID. An `otel.js` older than the runner, i.e. a hand-edited copy kept by sync, reads INVALID naming the stale file instead of throwing a TypeError. The `trace-negative` mutant injects the forbidden name after that check. A `spans` block declaring neither `ordered` nor `mustNotFire` is now a loud INVALID instead of a silent skip; the array shape `[{mf,status}]` and `{}` are named as such. `examples/validate-journeys.js` accepts a negative-only claim, which it used to reject as "NEVER checked" (contradicting the runner). It also checks the new `spans.liveness` field (only read for a negative-only claim) and a `mustNotFire` that is not a valid regex. `skills/journey-examples.md` documents the liveness rule. Fixture `tests/wave2/test-validate-journeys-spans.sh` (12 cases, 8 fail on the old validator). The shipped `project-tests/e2e/example.journey.json` is rewritten into the shape the runner reads. Its `name`/`role`, `value` seed, `page`/`expectPage`/`expectText`/`click` steps, array-shaped `spans` and SQL-style `<module>$<entity>` queries had drifted from it: the validator reported 3 problems on it, now 0, and all 7 control mutants are supported on it. Between journeys the runner probes `baseUrl` (3 tries); down before a journey → `APP-DOWN` INVALID, not walked; down after one → its FAIL rows are flagged ([#285](https://github.com/MendixMau/mxcli-project-toolkit/issues/285) harness half; a watch rebuild took a runtime down mid-suite on 2026-10-12). `APP_PROBE=0` disables it. Fixture `tests/wave2/test-journey-liveness.sh` (36 cases over the captured Jaeger trace and a real local HTTP server); main-loop wiring not fixtured (needs a browser), no field run yet — MendixMau
2323
- learn(testing-shape, test-result-audit): **three test-strength rules, each with a denominator: refusals probed on the server, every undeclared 4xx a failure, one clean-database final run** ([#172](https://github.com/MendixMau/mxcli-project-toolkit/issues/172)). A field comparison of two builds against one frozen spec found the toolkit build checked permission refusals only in the client (a hidden menu item), logged unexpected 400/422 responses as information, and reached 20/20 by stitching three runs. `testing-shape.md` §7 gains three report lines (`N of N restricted pairs probed server-side`, `undeclared 4xx/5xx: 0 of M requests — each fails its step`, `one clean-DB run <id>: N of N scenarios incl. timers, not stitched`); `test-result-audit.md` gains check 2e, which carries the reasons and the field evidence and asks for those numbers. Both baseline skills stay inside the 80,000-word baseline budget (79,997): the reasons live in the on-demand audit skill, and two redundant sentences in `testing-shape.md` §4's verdict paragraphs were folded (the empty-set corollary is stated in the Trace row, `harness-architecture.md` and `report-schema.md`). The backing inbox note is trimmed to its unpromoted "Keep" items. — MendixMau

‎project-tests/e2e/config.js‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -89,11 +89,10 @@ const CREDS = PROJ.credentials();
8989
// Only the last is refused by journey-runner's preflight. Refusing every non-env password
9090
// (as #302 first did) broke every project whose real password lives in project.config.js,
9191
// because nothing in the toolkit loads an env file before verify-module runs the journeys.
92-
// A credentials() that reports `source: 'template default'` is honoured as well.
9392
const TEMPLATE_PASSWORDS = ['Demo2025!']; // project.config.template.js DEFAULT_PASS
9493
function passwordSource({ env = process.env, creds = CREDS } = {}) {
9594
if (env.TEST_PASS || env.APP_PASS) return 'env';
96-
if (creds.source === 'template default' || TEMPLATE_PASSWORDS.includes(creds.pass)) return 'template placeholder';
95+
if (TEMPLATE_PASSWORDS.includes(creds.pass)) return 'template placeholder';
9796
return 'project.config.js';
9897
}
9998

‎project-tests/e2e/report-normalize.js‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -549,6 +549,16 @@ function collectJourney(input, contracts, ctx) {
549549
note: 'PASS/FAIL/INVALID are distinct at the source; INVALID → fault',
550550
checksEmitted: checks.length,
551551
});
552+
// An ABORTED run (journey-runner wrote `aborted: "preflight"` or `"signal SIGINT"`) is not
553+
// a result: the suite stopped before or while walking. Its rows are kept so the reader sees
554+
// why, but each is flagged notAResult and the instrument is a fault — never a green or a
555+
// finding about the app.
556+
if (d.aborted) {
557+
for (const c of checks) { c.notAResult = true; c.abortedRun = String(d.aborted); }
558+
Object.assign(instruments[instruments.length - 1], { verdict: 'fault', aborted: String(d.aborted),
559+
reason: `${input.path} is an ABORTED run (${d.aborted}) — the suite stopped before or while walking; ` +
560+
'its rows are kept and flagged notAResult, none of them is a result. Fix the cause and re-run.' });
561+
}
552562
return { checks, instruments, walks: collectWalks(d, checks) };
553563
}
554564

@@ -2272,6 +2282,9 @@ function normalize(inputs) {
22722282

22732283
return {
22742284
schemaVersion: SCHEMA_VERSION,
2285+
// The journey run was aborted (preflight refusal or a signal): its rows are kept and
2286+
// flagged notAResult, and nothing in this report is a journey result.
2287+
aborted: (jf && jf.aborted) ? String(jf.aborted) : null,
22752288
project: {
22762289
id: m.projectId || null,
22772290
displayName: m.displayName || null,
@@ -2609,6 +2622,16 @@ function selftest() {
26092622
t('a setup row is kind "setup", not "ui"',
26102623
jSetup.checks.filter(c => c.id === 'journey/_/preflight-admin').map(c => c.kind), ['setup']);
26112624

2625+
// An aborted run (preflight / signal) is kept but is never a result.
2626+
const jAb = normalize({ ...base, journeyFindings: { status: 'ok', path: 'journey-findings.json',
2627+
data: { capturedAt: '2026-01-01T00:00:00Z', positiveControl: false, aborted: 'preflight',
2628+
results: [{ rung: 'setup', name: 'preflight: admin', verdict: 'INVALID', detail: 'x', requirement: '' }] } } });
2629+
t('an aborted run marks the report aborted', jAb.aborted, 'preflight');
2630+
t('and faults the journey-runner instrument', inst(jAb, 'journey-runner').verdict, 'fault');
2631+
t('and flags every kept row notAResult',
2632+
jAb.checks.filter(c => c.instrument === 'journey-runner').every(c => c.notAResult === true && c.abortedRun === 'preflight'), true);
2633+
t('a completed run is not aborted', jSetup.aborted, null);
2634+
26122635
// ---- 3b. the control instrument is per-rung, and a copied real run is not one ----
26132636
const ctlRow = (rung, verdict) => ({
26142637
rung: 'control', verdict, detail: `d-${rung}`,

‎skills/report-schema.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,9 @@ any real project. The **Deriving the values** section says where each one actual
5757
```jsonc
5858
{
5959
"schemaVersion": "1.2", // renderer refuses on major mismatch, warns on newer minor
60+
"aborted": null, // "preflight" | "signal SIGINT" | … when journey-findings.json is an
61+
// aborted run: its checks carry notAResult: true + abortedRun, and the
62+
// journey-runner instrument is a fault. None of them is a result.
6063

6164
"project": {
6265
"id": "acme-procurement", // stable key; a fleet index groups on this
@@ -97,8 +100,9 @@ any real project. The **Deriving the values** section says where each one actual
97100
{
98101
"id": "journey/J-PR-01/step5/landing", // stable, sortable, unique
99102
"instrument": "journey-runner", // WHO measured
100-
"kind": "ui", // ui | trace | data | outcome | control
103+
"kind": "ui", // ui | trace | data | outcome | control | setup
101104
// | build | lint | coverage | conformance | stack
105+
// setup = journey-runner preflight / precondition / APP-DOWN rows
102106
"module": "Procurement", // null for cross-cutting
103107
"journey": "J-PR-01", // null when not a journey check
104108
"stepIndex": 5, // ordinal position on the path

‎tests/wave2/test-journey-preflight.sh‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,6 @@ out('PS', [
125125
ps({ env: { APP_PASS: 'x' }, creds: { pass: 'Real-Proj-Pass' } }),
126126
ps({ env: {}, creds: { pass: 'Real-Proj-Pass' } }),
127127
ps({ env: {}, creds: { pass: 'Demo2025!' } }),
128-
ps({ env: {}, creds: { pass: 'anything', source: 'template default' } }),
129128
]);
130129
// the shipped template's DEFAULT_PASS is the placeholder this list names
131130
out('TPL_LISTED', (C.TEMPLATE_PASSWORDS || []).includes(E.TPL_PASS));
@@ -266,7 +265,7 @@ case "$(line CRED)" in *'"abort":2,"calls":0'*'SETUP credentials: TEST_PASS is
266265
echo "== T2b': only the template placeholder is refused (#302 follow-up) =="
267266
[ "$(line CRED_PROJ)" = '{"abort":null,"calls":1,"cred":false}' ] && ok "password set in project.config.js → no refusal, probe runs" || bad "project-set password refused (breaks every project after sync)" "$(line CRED_PROJ)"
268267
case "$(line CRED_TPL)" in *'"abort":2'*'placeholder password'*DEFAULT_PASS*) ok "template placeholder → refused, names DEFAULT_PASS as the fix" ;; *) bad "placeholder not refused" "$(line CRED_TPL)" ;; esac
269-
[ "$(line PS)" = '["env","env","project.config.js","template placeholder","template placeholder"]' ] && ok "passwordSource: env wins; project value is the project's; template value / 'template default' is the placeholder" || bad "passwordSource wrong" "$(line PS)"
268+
[ "$(line PS)" = '["env","env","project.config.js","template placeholder"]' ] && ok "passwordSource: env wins; project value is the project's; the template value is the placeholder" || bad "passwordSource wrong" "$(line PS)"
270269
[ -n "$TPL_PASS" ] && [ "$(line TPL_LISTED)" = true ] && ok "the shipped template's DEFAULT_PASS is the listed placeholder" || bad "template DEFAULT_PASS not in TEMPLATE_PASSWORDS (got '$TPL_PASS')" "$(line TPL_LISTED)"
271270

272271
echo "== T2c: resetsData / --protect-state (#279) =="
@@ -357,6 +356,18 @@ FJ="$WORK/tests/e2e/artifacts/journey-findings.json"
357356
node -e 'const f=require(process.argv[1]); process.exit(f.aborted==="preflight" && f.results.length===1 && f.results[0].verdict==="INVALID" ? 0 : 1)' "$FJ" 2>/dev/null \
358357
&& ok "findings slot overwritten with the aborted run (no stale file reads as this run)" || bad "findings file not written as aborted"
359358
grep -qF "$MARK" "$FJ" 2>/dev/null && bad "credential written to the findings file" || ok "credential not in the findings file"
359+
# The aborted findings file through the real normalizer: a report, marked aborted, no results.
360+
if [ -f "$E2E/report-normalize.js" ]; then
361+
cp "$E2E/report-normalize.js" "$WORK/tests/e2e/"
362+
( cd "$WORK/tests/e2e" && APP_PORT=1 node report-normalize.js --out docs/report.json ) > "$WORK/rn.out" 2>&1
363+
node -e 'const r=require(process.argv[1]); const j=r.checks.filter(c=>c.instrument==="journey-runner");
364+
const i=r.instruments.find(x=>x.name==="journey-runner");
365+
process.exit(r.aborted==="preflight" && j.length>0 && j.every(c=>c.notAResult===true) && i.verdict==="fault" ? 0 : 1)' "$WORK/docs/report.json" 2>/dev/null \
366+
&& ok "report-normalize marks the aborted run: report.aborted, rows flagged notAResult, instrument fault" \
367+
|| bad "aborted run normalised as results" "$(tail -3 "$WORK/rn.out"; node -e 'const r=require(process.argv[1]);console.log(r.aborted)' "$WORK/docs/report.json" 2>&1)"
368+
else
369+
echo " skip — report-normalize.js not beside $SUT"
370+
fi
360371

361372
echo "== T10: real runner — missing env var, every journey affected → one SETUP line, no probe =="
362373
rm -f "$WORK/argv"

0 commit comments

Comments
 (0)