Skip to content

Commit 57bf81b

Browse files
committed
report: read per-BRD conformance reports (report-<date>-<BRD>.tsv)
conformance-check now writes one report per BRD (#307), but report-normalize.js kept only report-<date>.tsv, so per-BRD modules never reached report.json and a stale base file could read as measured. It now takes the newest date among report-<date>[-<BRD>].tsv and concatenates every file of that date; single-file projects are unchanged. pickConformanceFiles is exported; --selftest cases and case 3 of test-report-normalize-two-tree.sh (fails on master: reads the stale file). Also: the new fixtures use here-strings instead of `printf | grep -q`, which under pipefail can SIGPIPE and flip a check (seen once under load). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
1 parent ba461e5 commit 57bf81b

6 files changed

Lines changed: 75 additions & 27 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are.
1717

1818
## Unreleased
1919

20-
- fix(journeys): **the #302 preflight no longer refuses a password the project set in `project.config.js`; only the template placeholder is refused.** As shipped, #302 refused every run whose `TEST_PASS`/`APP_PASS` was not in the environment unless `ALLOW_CONFIG_CREDENTIALS=1` was set. But `project.config.template.js` tells every project to put its real demo password in `DEFAULT_PASS`, nothing in the toolkit loads an env file before `verify-module.sh` runs the journeys, and `sync-project.sh` refreshes unedited runner/config copies. So after the next sync every journeys rung would have faulted with rc 2. Now `config.js` reports `passSource` (`env` / `project.config.js` / `template placeholder`) and only the template's placeholder (`Demo2025!`) is refused. The env still wins, and `ALLOW_CONFIG_CREDENTIALS=1` still overrides. **Verdict change to know about from #302:** a refused admin credential (HTTP 401/403) used to leave every data rung INVALID with exit 1; the suite now aborts before the first journey with exit 2. A SIGINT/SIGTERM now writes the findings slot as `aborted: "signal SIGINT"` instead of leaving the previous run's file there. `report-normalize.js` keeps `rung: 'setup'` rows as kind `setup` instead of folding them into `ui` (`report-schema.md` enum updated). It also honours `aborted` in a findings file: the report gets a top-level `aborted`, every kept journey row is flagged `notAResult`, and the journey-runner instrument is a fault, so an aborted run never reads as results. Fixtures: `test-journey-preflight.sh` 74 cases (9 new, all failing on the #302 code: project-set password runs, placeholder refuses, env wins, SIGINT writes aborted findings end to end) and `report-normalize.js --selftest` cases (setup kind, aborted run). Refs [#286](https://github.com/MendixMau/mxcli-project-toolkit/issues/286) — MendixMau
20+
- fix(journeys): **the #302 preflight no longer refuses a password the project set in `project.config.js`; only the template placeholder is refused.** As shipped, #302 refused every run whose `TEST_PASS`/`APP_PASS` was not in the environment unless `ALLOW_CONFIG_CREDENTIALS=1` was set. But `project.config.template.js` tells every project to put its real demo password in `DEFAULT_PASS`, nothing in the toolkit loads an env file before `verify-module.sh` runs the journeys, and `sync-project.sh` refreshes unedited runner/config copies. So after the next sync every journeys rung would have faulted with rc 2. Now `config.js` reports `passSource` (`env` / `project.config.js` / `template placeholder`) and only the template's placeholder (`Demo2025!`) is refused. The env still wins, and `ALLOW_CONFIG_CREDENTIALS=1` still overrides. **Verdict change to know about from #302:** a refused admin credential (HTTP 401/403) used to leave every data rung INVALID with exit 1; the suite now aborts before the first journey with exit 2. A SIGINT/SIGTERM now writes the findings slot as `aborted: "signal SIGINT"` instead of leaving the previous run's file there. `report-normalize.js` keeps `rung: 'setup'` rows as kind `setup` instead of folding them into `ui` (`report-schema.md` enum updated). It also honours `aborted` in a findings file: the report gets a top-level `aborted`, every kept journey row is flagged `notAResult`, and the journey-runner instrument is a fault, so an aborted run never reads as results. It also reads the per-BRD conformance reports `report-<date>-<BRD>.tsv` that conformance-check now writes ([#307](https://github.com/MendixMau/mxcli-project-toolkit/pull/307)). It used to keep only `report-<date>.tsv`, so per-BRD modules never reached report.json and a stale base file could show as measured. It now takes the newest date and aggregates every file of that date; single-file projects are unchanged (`--selftest` cases plus case 3 of `test-report-normalize-two-tree.sh`). Fixtures: `test-journey-preflight.sh` 74 cases (9 new, all failing on the #302 code: project-set password runs, placeholder refuses, env wins, SIGINT writes aborted findings end to end) and `report-normalize.js --selftest` cases (setup kind, aborted run). Refs [#286](https://github.com/MendixMau/mxcli-project-toolkit/issues/286) — MendixMau
2121
- fix(e2e): **`helpers.js` reads OQL rows by SELECT alias, never by position.** `mxcli oql --direct --json` prints keys alphabetically, leaves out a column that is null in every row, and over a view entity adds `ID` beside the alias. `Object.values(r)` therefore shifted columns, and `oqlScalar` returned a view row's ID instead of its count; a string value holding `]` or `}` truncated the brace-counting slice, so a real answer read as "no measurement" ([#154](https://github.com/MendixMau/mxcli-project-toolkit/issues/154), card-disbursement build). Rows are now keyed by the parsed SELECT aliases in SELECT order with explicit nulls, `oqlScalar` reads the first alias (null rather than a guess when the list has no alias), new `oqlRows`, string-aware JSON slicing; the admin preflight uses the same parser. **Behaviour change to check in your project:** `oqlScalar` now returns the **first SELECT alias**, where it used to return the alphabetically-first key. A multi-column seed, scope capture or baseline query whose value flows into `{{vars}}` may now resolve to a different column. Put the column you mean first, or make the query single-column. Golden: the released mxcli v0.26.0 binary's verbatim stdout/stderr for four queries against a stub admin API (`tests/wave2/fixtures/oql-json/`); fixture `tests/wave2/test-oql-rows.sh` (15 cases, 0/15 on the old helper). No live-app field run yet — MendixMau
2222
- fix(e2e): **a negative-only span claim is judged against a liveness reference, and an app that dies mid-suite reads APP-DOWN, not FAIL.** A `mustNotFire`-only step (Cancel on a reset confirmation) captures zero spans, which also happens when Jaeger is down, so it was INVALID on every run and its control mutant was unsupported ([#150](https://github.com/MendixMau/mxcli-project-toolkit/issues/150)). Rung 2 now waits the full window and requires a declared `spans.liveness` span or any service span since the journey began (`otel.js liveness`; the window starts at the journey's own start with a 2 s clock-skew margin); no evidence stays INVALID. An `otel.js` older than the runner, i.e. a hand-edited copy kept by sync, reads INVALID naming the stale file instead of throwing a TypeError. The `trace-negative` mutant injects the forbidden name after that check. A `spans` block declaring neither `ordered` nor `mustNotFire` is now a loud INVALID instead of a silent skip; the array shape `[{mf,status}]` and `{}` are named as such. `examples/validate-journeys.js` accepts a negative-only claim, which it used to reject as "NEVER checked" (contradicting the runner). It also checks the new `spans.liveness` field (only read for a negative-only claim) and a `mustNotFire` that is not a valid regex. `skills/journey-examples.md` documents the liveness rule. Fixture `tests/wave2/test-validate-journeys-spans.sh` (12 cases, 8 fail on the old validator). The shipped `project-tests/e2e/example.journey.json` is rewritten into the shape the runner reads. Its `name`/`role`, `value` seed, `page`/`expectPage`/`expectText`/`click` steps, array-shaped `spans` and SQL-style `<module>$<entity>` queries had drifted from it: the validator reported 3 problems on it, now 0, and all 7 control mutants are supported on it. Between journeys the runner probes `baseUrl` (3 tries); down before a journey → `APP-DOWN` INVALID, not walked; down after one → its FAIL rows are flagged ([#285](https://github.com/MendixMau/mxcli-project-toolkit/issues/285) harness half; a watch rebuild took a runtime down mid-suite on 2026-10-12). `APP_PROBE=0` disables it. Fixture `tests/wave2/test-journey-liveness.sh` (36 cases over the captured Jaeger trace and a real local HTTP server); main-loop wiring not fixtured (needs a browser), no field run yet — MendixMau
2323
- learn(testing-shape, test-result-audit): **three test-strength rules, each with a denominator: refusals probed on the server, every undeclared 4xx a failure, one clean-database final run** ([#172](https://github.com/MendixMau/mxcli-project-toolkit/issues/172)). A field comparison of two builds against one frozen spec found the toolkit build checked permission refusals only in the client (a hidden menu item), logged unexpected 400/422 responses as information, and reached 20/20 by stitching three runs. `testing-shape.md` §7 gains three report lines (`N of N restricted pairs probed server-side`, `undeclared 4xx/5xx: 0 of M requests — each fails its step`, `one clean-DB run <id>: N of N scenarios incl. timers, not stitched`); `test-result-audit.md` gains check 2e, which carries the reasons and the field evidence and asks for those numbers. Both baseline skills stay inside the 80,000-word baseline budget (79,997): the reasons live in the on-demand audit skill, and two redundant sentences in `testing-shape.md` §4's verdict paragraphs were folded (the empty-set corollary is stated in the Trace row, `harness-architecture.md` and `report-schema.md`). The backing inbox note is trimmed to its unpromoted "Keep" items. — MendixMau

‎project-tests/e2e/report-normalize.js‎

Lines changed: 44 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1366,7 +1366,7 @@ function collectLoop(summaries) {
13661366
}
13671367

13681368
// ============================================================================
1369-
// SOURCE 4 — docs/conformance/report-<date>.tsv (newest by filename)
1369+
// SOURCE 4 — docs/conformance/report-<date>[-<BRD>].tsv (newest date by filename, all BRDs)
13701370
// ============================================================================
13711371
function collectConformance(input, ctx) {
13721372
const checks = [];
@@ -1430,6 +1430,17 @@ function collectConformance(input, ctx) {
14301430
};
14311431
}
14321432

1433+
// Which conformance files make up the newest run: the newest date among report-<date>.tsv
1434+
// and report-<date>-<BRD>.tsv, and every file of that date (sorted). A per-BRD run and an
1435+
// older single-file run never mix: only the newest date is read.
1436+
function pickConformanceFiles(names) {
1437+
const re = /^report-(\d{4}-\d{2}-\d{2})(?:-[A-Za-z0-9_.-]+)?\.tsv$/;
1438+
const hits = names.map(n => [n, re.exec(n)]).filter(([, m]) => m);
1439+
if (!hits.length) return { date: null, files: [] };
1440+
const date = hits.map(([, m]) => m[1]).sort().pop();
1441+
return { date, files: hits.filter(([, m]) => m[1] === date).map(([n]) => n).sort() };
1442+
}
1443+
14331444
// ============================================================================
14341445
// SOURCE 5 — architecture/modules/*/coverage-ledger.md
14351446
// ============================================================================
@@ -2440,21 +2451,24 @@ function loadInputs(outFile) {
24402451
return { module: mod, file: readFileInput(rel), mtime };
24412452
});
24422453

2443-
// Newest conformance report BY FILENAME — the names are ISO dates, so lexical sort
2444-
// is chronological, and mtime would reorder on a git checkout.
2445-
let confRel = null, confDate = null;
2446-
try {
2447-
const files = fs.readdirSync(abs(INPUTS.conformanceDir))
2448-
.filter(f => /^report-\d{4}-\d{2}-\d{2}\.tsv$/.test(f)).sort();
2449-
if (files.length) {
2450-
confRel = `${INPUTS.conformanceDir}/${files[files.length - 1]}`;
2451-
confDate = files[files.length - 1].slice(7, 17);
2452-
}
2453-
} catch { /* handled below */ }
2454-
const conformance = confRel
2455-
? readFileInput(confRel)
2456-
: { status: 'missing', path: `${INPUTS.conformanceDir}/report-<date>.tsv`,
2457-
error: 'no report-<date>.tsv found' };
2454+
// Newest conformance report date BY FILENAME — the names are ISO dates, so lexical sort
2455+
// is chronological, and mtime would reorder on a git checkout. Every file of that date is
2456+
// read: the single report-<date>.tsv and/or the per-BRD report-<date>-<BRD>.tsv files
2457+
// conformance-check writes (one per BRD); their rows are concatenated.
2458+
let confPick = { date: null, files: [] };
2459+
try { confPick = pickConformanceFiles(fs.readdirSync(abs(INPUTS.conformanceDir))); }
2460+
catch { /* handled below */ }
2461+
const confDate = confPick.date;
2462+
let conformance;
2463+
if (!confPick.files.length) {
2464+
conformance = { status: 'missing', path: `${INPUTS.conformanceDir}/report-<date>.tsv`,
2465+
error: 'no report-<date>.tsv or report-<date>-<BRD>.tsv found' };
2466+
} else {
2467+
const parts = confPick.files.map(n => readFileInput(`${INPUTS.conformanceDir}/${n}`));
2468+
const broken = parts.find(p => p.status !== 'ok');
2469+
conformance = broken || (parts.length === 1 ? parts[0]
2470+
: { status: 'ok', path: parts.map(p => p.path).join(', '), text: parts.map(p => p.text).join('\n') });
2471+
}
24582472

24592473
const ledgers = listDirs(INPUTS.modulesDir)
24602474
.map(mod => ({ module: mod, file: readFileInput(`${INPUTS.modulesDir}/${mod}/coverage-ledger.md`) }))
@@ -2622,6 +2636,19 @@ function selftest() {
26222636
t('a setup row is kind "setup", not "ui"',
26232637
jSetup.checks.filter(c => c.id === 'journey/_/preflight-admin').map(c => c.kind), ['setup']);
26242638

2639+
// Conformance: newest date wins; per-BRD files of that date are all read; single file unchanged.
2640+
t('one base file → that file', pickConformanceFiles(['report-2026-10-01.tsv', 'notes.md']),
2641+
{ date: '2026-10-01', files: ['report-2026-10-01.tsv'] });
2642+
t('per-BRD files of the newest date, all of them; a stale older base file is ignored',
2643+
pickConformanceFiles(['report-2026-09-30.tsv', 'report-2026-10-12-F002.tsv', 'report-2026-10-12-F001.tsv', 'report-2026-10-01-F001.tsv']),
2644+
{ date: '2026-10-12', files: ['report-2026-10-12-F001.tsv', 'report-2026-10-12-F002.tsv'] });
2645+
t('no report files → none', pickConformanceFiles(['readme.txt']), { date: null, files: [] });
2646+
const confMulti = normalize({ ...base, conformance: { status: 'ok', path: 'a, b',
2647+
text: 'module\tpointer\tstored\tobserved\tverdict\nOrders\t/x\t1\t1\tOK\n' +
2648+
'\nmodule\tpointer\tstored\tobserved\tverdict\nBilling\t/y\t2\t3\tSTALE\n' } });
2649+
t('concatenated per-BRD reports yield both modules',
2650+
confMulti.checks.filter(c => c.instrument === 'conformance' && c.kind === 'conformance').map(c => c.module).sort(), ['Billing', 'Orders']);
2651+
26252652
// An aborted run (preflight / signal) is kept but is never a result.
26262653
const jAb = normalize({ ...base, journeyFindings: { status: 'ok', path: 'journey-findings.json',
26272654
data: { capturedAt: '2026-01-01T00:00:00Z', positiveControl: false, aborted: 'preflight',
@@ -3122,5 +3149,5 @@ function main() {
31223149

31233150
if (require.main === module) main();
31243151

3125-
module.exports = { normalize, normReq, declaredChecks, parseLedger,
3152+
module.exports = { normalize, normReq, declaredChecks, parseLedger, pickConformanceFiles,
31263153
V_JOURNEY, V_LOOP, V_WALK, V_CONF, selftest };

‎skills/report-schema.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,9 @@ any real project. The **Deriving the values** section says where each one actual
143143
{ "name": "positive-control", "verdict": "fault", "reason": "not requested (--positive-control absent)" },
144144
{ "name": "monkey", "verdict": "skipped", "reason": "--skip-monkey" },
145145
{ "name": "conformance", "verdict": "pass", "log": "docs/conformance/report-<date>.tsv" }
146+
// conformance reads the NEWEST date among report-<date>.tsv and the per-BRD
147+
// report-<date>-<BRD>.tsv files, and every file of that date (rows concatenated;
148+
// "log" then lists them comma-separated). An older date is never mixed in.
146149
],
147150

148151
// Rollups have denominators, so they are not checks.

‎tests/wave2/test-journey-preflight.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -396,7 +396,7 @@ PROJ_CFG="$WORK/tests/e2e/project.config.js"; cp "$PROJ_CFG" "$WORK/cfg.bak"
396396
sed -e "s/^const DEFAULT_PASS = '.*';/const DEFAULT_PASS = 'fixture-project-pass';/" "$WORK/cfg.bak" > "$PROJ_CFG"
397397
E2EOUT="$(cd "$WORK/tests/e2e" && env -u TEST_PASS -u APP_PASS -u ALLOW_CONFIG_CREDENTIALS -u M2EE_ADMIN_PORT APP_PORT=1 M2EE_ADMIN_PASS="$MARK" \
398398
MXCLI="$WORK/stub/mxcli" STUB_ARGV="$WORK/argv" node journey-runner.js "$J/J-one.journey.json" 2>&1)"; RC=$?
399-
printf '%s\n' "$E2EOUT" | grep -q 'SETUP credentials' && bad "project-set password refused" "$E2EOUT" || ok "no credentials refusal"
399+
grep -q 'SETUP credentials' <<< "$E2EOUT" && bad "project-set password refused" "$E2EOUT" || ok "no credentials refusal"
400400
[ -f "$WORK/argv" ] && ok "went on to the admin probe (the stub answers 403 → exit $RC)" || bad "probe never ran" "$E2EOUT"
401401
cp "$WORK/cfg.bak" "$PROJ_CFG"
402402

‎tests/wave2/test-report-normalize-two-tree.sh‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,8 @@
99
# step then showed 8 of 9 instruments FAULT ("design-audit.json does not exist", "no
1010
# coverage-ledger.md under architecture/modules/*/") over artefacts that were on disk, and wrote
1111
# the report into a stray app/docs/report.json — so the render step found nothing to render.
12-
# Case 1 is that shape in miniature; case 2 is the single-tree regression guard.
12+
# Case 1 is that shape in miniature; case 2 is the single-tree regression guard; case 3 reads
13+
# per-BRD conformance reports (report-<date>-<BRD>.tsv, PR #307).
1314
#
1415
# The artefacts here are SYNTHETIC and minimal. That is deliberate and allowed: the field-proof
1516
# rule's "golden input is captured" governs parsers of tool output, and nothing below exercises a
@@ -91,6 +92,23 @@ else
9192
bad "single tree: no docs/report.json" "$(tr '\n' ' ' < "$WORK/out2" | cut -c1-300)"
9293
fi
9394

95+
# --- 3. per-BRD conformance reports (conformance-check writes report-<date>-<BRD>.tsv) ---------
96+
# The newest date wins and every BRD file of that date is read; an older single-file report is
97+
# not mixed in (it used to be the ONLY file read, shown as measured although stale).
98+
P="$WORK/perbrd"; lay_out "$P" "$P"
99+
rm -f "$P/docs/conformance/"report-*.tsv
100+
printf 'module\tclaim\tstored\tobserved\tverdict\nStale\t/old\t1\t1\tOK\n' > "$P/docs/conformance/report-2026-09-26.tsv"
101+
printf 'module\tclaim\tstored\tobserved\tverdict\nOrders\t/useCases\t3\t3\tOK\n' > "$P/docs/conformance/report-2026-10-12-F001.tsv"
102+
printf 'module\tclaim\tstored\tobserved\tverdict\nBilling\t/pages\t2\t4\tUNDERSTATED\n' > "$P/docs/conformance/report-2026-10-12-F002.tsv"
103+
( cd "$P/tests/e2e" && node report-normalize.js --out docs/report.json ) >"$WORK/out3" 2>&1
104+
if [ -s "$P/docs/report.json" ]; then
105+
mods="$(node -e 'const r=require(process.argv[1]); console.log(r.checks.filter(c=>c.instrument==="conformance"&&c.kind==="conformance").map(c=>c.module).sort().join(","))' "$P/docs/report.json")"
106+
[ "$mods" = "Billing,Orders" ] && ok "per-BRD reports of the newest date are all read (Billing, Orders); the older single file is not" \
107+
|| bad "per-BRD conformance reports not aggregated" "$mods"
108+
else
109+
bad "per-BRD: no docs/report.json" "$(tr '\n' ' ' < "$WORK/out3" | cut -c1-300)"
110+
fi
111+
94112
echo ""
95113
echo "PASS=$PASS FAIL=$FAIL"
96114
[ "$FAIL" -eq 0 ]

0 commit comments

Comments
 (0)