Skip to content

Commit 4019726

Browse files
refactor(mutation): keep the summary script to the shape the review asked for
Review threads on #2149: - normalizeReportPath rejects only a leading .. component (relative === ".." or a "../" prefix), so a file legitimately named "..foo.ts" is no longer rejected. - Append with fs.appendFileSync inside the existing try, the way appendSummary() in scripts/stryker-diff.mjs does: the runner gives this step its own summary file, so the temp file, rename and stale sweep were solving a problem this step does not have. - Drop escapeMarkdownCell. A string reaches the table only after REPORT_NAME or SAFE_RELATIVE_PATH matched, so no Markdown table or link character can be rendered from a report; the rejection path is now tested with pipe, backtick, bracket/link, space, absolute and both outside-path keys instead of testing an unreachable escaper. - Cap the table at the 40 worst rows (GitHub caps a step's job summary at 1 MiB and fails the upload above it) and count the remainder in a pointer to the artifact. - Run the script as a subprocess in the tests: writing GITHUB_STEP_SUMMARY appends and prints nothing, and an unwritable GITHUB_STEP_SUMMARY emits a ::warning and still exits 0. A broken entry guard would have written nothing while every in-process test passed. - Assert the whole row with distinct Survived and NoCoverage counts, so a swapped column pair cannot pass. - AGENTS.md: the summary lists every file with counted mutants, not every mutated file. Negative controls: with the old startsWith("..") guard the path test fails; with the row cap removed the cap test fails; with the entry guard disabled both subprocess tests fail. node --test scripts/stryker-diff.test.mjs scripts/mutation-summary.test.mjs: 58/58 green.
1 parent 00071a9 commit 4019726

4 files changed

Lines changed: 164 additions & 190 deletions

File tree

‎.github/workflows/mutation-testing.yml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -57,8 +57,9 @@ jobs:
5757
- name: Summarize per-file mutation results
5858
if: always() && github.event_name == 'pull_request'
5959
# The summary logic lives in scripts/mutation-summary.mjs so it is unit-tested;
60-
# the script validates the Stryker report shape, rejects unsafe path keys and
61-
# escapes Markdown before writing the table.
60+
# the script reads counts from the Stryker JSON report (StrykerJS has no Markdown
61+
# reporter), rejects report paths that are not repo-relative source names, and
62+
# renders at most the 40 worst rows so the step summary stays under its size cap.
6263
run: node scripts/mutation-summary.mjs
6364

6465
- name: Upload mutation reports

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ Prefer the narrowest test layer that proves the behavior. This follows standard
6262

6363
The `mutation-diff` job publishes the changed-code mutation report in two places:
6464

65-
- The job summary (`mutation-diff` → Summary tab) lists every mutated file with its `Survived` and `NoCoverage` counts, worst file first. Use the `changed-code-mutation-report` artifact to inspect each surviving or uncovered mutant's line, mutator, and replacement.
65+
- The job summary (`mutation-diff` → Summary tab) lists every file with counted mutants (a file whose mutants are all `Ignored` or `CompileError` gets no row), worst file first, capped at the 40 worst rows with the rest counted in a pointer to the artifact. Use the `changed-code-mutation-report` artifact to inspect each surviving or uncovered mutant's line, mutator, and replacement.
6666
- The report is uploaded as the `changed-code-mutation-report` artifact (`reports/mutation/<package>/mutation.json` and `mutation.html`, retained 7 days) for a browsable view.
6767

6868
Close a listed mutant by adding or strengthening a focused test that fails under the mutation. If a suppression is genuinely unavoidable, follow the existing `Stryker disable` reason rule under Test Placement Guidelines above.

‎scripts/mutation-summary.mjs‎

Lines changed: 56 additions & 88 deletions
Original file line numberDiff line numberDiff line change
@@ -6,82 +6,72 @@ import { pathToFileURL } from "node:url"
66
* Build the per-file mutation summary that the mutation-testing workflow appends to the job
77
* summary.
88
*
9-
* The Stryker JSON reports are untrusted input for this script: report.files keys are file
10-
* paths and they are rendered into a Markdown table. Every path is therefore validated
11-
* (normalised repo-relative form, safe characters) and Markdown-escaped before insertion, and
12-
* the report shape is checked rather than assumed.
9+
* StrykerJS has no Markdown reporter (its reporters are progress, dots, clear-text, dashboard
10+
* and json, plus html as a plugin), so the counts are read from the mutation.json report this
11+
* job already writes and the table is rendered here.
12+
*
13+
* Report contents are data, not text to display: a string reaches the summary only if it
14+
* passed REPORT_NAME or SAFE_RELATIVE_PATH, so no Markdown table or link character can be
15+
* rendered from a report, and a path that resolves outside the working directory is rejected
16+
* rather than shown.
1317
*/
1418

1519
const REPORT_DIRECTORY = "reports/mutation"
1620
const VALID_STATUSES = new Set(["Killed", "Survived", "NoCoverage", "Timeout"])
17-
// Repo-relative paths only: letters, digits and the separators/characters a source path
18-
// normally uses. Anything else (backticks, pipes, control characters, drive letters, ...)
19-
// is rejected instead of rendered into the summary.
21+
// Repo-relative source paths only: letters, digits and the separators a source path uses.
22+
// A key carrying anything else (pipe, backtick, brackets, control characters, drive letters,
23+
// leading ".." component) is rejected instead of rendered.
2024
const SAFE_RELATIVE_PATH = /^[A-Za-z0-9._@+/-]+$/
21-
const BACKSLASH = String.fromCharCode(92)
22-
const BACKTICK = String.fromCharCode(96)
23-
24-
/** Escape the Markdown table metacharacters and line breaks a file path could carry. */
25-
export function escapeMarkdownCell(value) {
26-
return String(value)
27-
.split(BACKSLASH)
28-
.join(BACKSLASH + BACKSLASH)
29-
.split("|")
30-
.join(BACKSLASH + "|")
31-
.split(BACKTICK)
32-
.join(BACKSLASH + BACKTICK)
33-
.split("[")
34-
.join(BACKSLASH + "[")
35-
.split("]")
36-
.join(BACKSLASH + "]")
37-
.split("(")
38-
.join(BACKSLASH + "(")
39-
.split(")")
40-
.join(BACKSLASH + ")")
41-
.split("\r")
42-
.join(BACKSLASH + "r")
43-
.split("\n")
44-
.join(BACKSLASH + "n")
45-
}
25+
// Package directory names in reports/mutation/<package> are simple slugs.
26+
const REPORT_NAME = /^[A-Za-z0-9._@+-]+$/
27+
// GitHub caps a step's job summary at 1 MiB and fails the summary upload above it, so the
28+
// table shows the worst rows and counts the rest instead of risking the whole summary.
29+
const MAX_SUMMARY_ROWS = 40
30+
const BACKTICK = "`"
4631

47-
/**
48-
* Render a report directory name for diagnostics. Package directory names are simple slugs; a
49-
* name that does not match that shape is not rendered verbatim.
50-
*/
32+
/** Return a report directory name for a diagnostic, or a fixed label when it is not slug-shaped. */
5133
function renderReportName(name) {
52-
return /^[A-Za-z0-9._@+-]+$/.test(name) ? name : "report"
34+
return REPORT_NAME.test(name) ? name : "report"
5335
}
5436

5537
/**
5638
* Return the normalised repo-relative path for one report entry, or undefined when the entry
57-
* cannot be trusted for display.
39+
* cannot be shown: non-string, empty, absolute, escaping the working directory through a ".."
40+
* component, or containing a character a source path would not have.
5841
*/
5942
export function normalizeReportPath(filePath, projectRoot, cwd) {
6043
if (typeof filePath !== "string" || filePath.length === 0) return undefined
6144
if (projectRoot !== undefined && typeof projectRoot !== "string") return undefined
6245
if (filePath.includes("\u0000")) return undefined
6346
const absolute = path.resolve(projectRoot ?? cwd, filePath)
6447
const relative = path.relative(cwd, absolute).replaceAll("\\", "/")
65-
if (relative.length === 0 || path.isAbsolute(relative) || relative.startsWith("..")) return undefined
48+
// A leading ".." component only: a file legitimately named "..foo.ts" stays a file.
49+
if (relative.length === 0 || relative === ".." || relative.startsWith("../") || path.isAbsolute(relative)) {
50+
return undefined
51+
}
6652
if (!SAFE_RELATIVE_PATH.test(relative)) return undefined
6753
return relative
6854
}
6955

7056
/**
71-
* Collect per-file mutation counts from every report under reportRoot.
57+
* Count per-file mutants from every reports/mutation/<package>/mutation.json under reportRoot.
7258
*
73-
* Returns the Markdown lines for the summary (including the header when any row exists) and
74-
* the list of unreadable reports that were skipped.
59+
* Returns the Markdown lines for the summary (header included when any row is rendered), the
60+
* sorted rows, and the fixed diagnostics of reports that were skipped.
7561
*/
76-
export function summarizeMutationReports({ reportRoot = REPORT_DIRECTORY, cwd = process.cwd() } = {}) {
62+
export function summarizeMutationReports({
63+
reportRoot = REPORT_DIRECTORY,
64+
cwd = process.cwd(),
65+
maxRows = MAX_SUMMARY_ROWS,
66+
} = {}) {
7767
const rows = []
7868
const unreadable = []
7969
let directories = []
8070

8171
try {
8272
directories = fs.existsSync(reportRoot) ? fs.readdirSync(reportRoot, { withFileTypes: true }) : []
8373
} catch {
84-
// Raw filesystem error text is untrusted input for the summary; keep a fixed diagnostic.
74+
// Raw filesystem error text is not summary content; keep a fixed diagnostic.
8575
unreadable.push("reports/mutation: directory could not be read")
8676
}
8777

@@ -119,6 +109,7 @@ export function summarizeMutationReports({ reportRoot = REPORT_DIRECTORY, cwd =
119109
if (mutant.status === "NoCoverage") counts.noCoverage += 1
120110
if (mutant.status === "Timeout") counts.timeout += 1
121111
}
112+
// Files with no counted mutants (Ignored, CompileError, net-only) have no row.
122113
if (counts.mutants === 0) continue
123114
const relative = normalizeReportPath(filePath, report.projectRoot, cwd)
124115
if (relative === undefined) {
@@ -152,14 +143,14 @@ export function summarizeMutationReports({ reportRoot = REPORT_DIRECTORY, cwd =
152143
"| File | Mutants | Killed | Survived | NoCoverage | Score |",
153144
"| --- | ---: | ---: | ---: | ---: | ---: |",
154145
)
155-
for (const row of rows) {
146+
for (const row of rows.slice(0, maxRows)) {
156147
// Stryker counts a Timeout mutant as detected, so the mutation score numerator is
157148
// Killed + Timeout (the table's Killed column stays Killed-only).
158149
const score = (((row.killed + row.timeout) / row.mutants) * 100).toFixed(1)
159150
lines.push(
160151
"| " +
161152
BACKTICK +
162-
escapeMarkdownCell(row.file) +
153+
row.file +
163154
BACKTICK +
164155
" | " +
165156
row.mutants +
@@ -174,57 +165,31 @@ export function summarizeMutationReports({ reportRoot = REPORT_DIRECTORY, cwd =
174165
"% |",
175166
)
176167
}
168+
if (rows.length > maxRows) {
169+
lines.push(
170+
"",
171+
"and " +
172+
(rows.length - maxRows) +
173+
" more file" +
174+
(rows.length - maxRows === 1 ? "" : "s") +
175+
"; see the " +
176+
BACKTICK +
177+
"changed-code-mutation-report" +
178+
BACKTICK +
179+
" artifact.",
180+
)
181+
}
177182
lines.push(
178183
"",
179184
"Close a surviving or uncovered mutant with a focused test that fails under the mutation; the artifact does not need to be downloaded to read this table.",
180185
)
181186
}
182187

183-
if (unreadable.length > 0) lines.push("", "Unreadable report(s): " + unreadable.map(escapeMarkdownCell).join("; "))
188+
if (unreadable.length > 0) lines.push("", "Unreadable report(s): " + unreadable.join("; "))
184189

185190
return { lines, rows, unreadable }
186191
}
187192

188-
/**
189-
* Append through a crash-safe path: read the existing summary, write the combined content to a
190-
* temporary file in the same directory, then rename it over the target. A partial append would
191-
* leave the job summary truncated; the rename is atomic within one filesystem.
192-
*/
193-
function appendSummaryAtomically(outputFile, summary) {
194-
const existing = fs.existsSync(outputFile) ? fs.readFileSync(outputFile, "utf8") : ""
195-
const directory = path.dirname(outputFile)
196-
const baseName = path.basename(outputFile)
197-
const temporary = path.join(directory, "." + baseName + "." + process.pid + "." + Date.now() + ".tmp")
198-
let renamed = false
199-
try {
200-
fs.writeFileSync(temporary, existing + summary)
201-
fs.renameSync(temporary, outputFile)
202-
renamed = true
203-
} finally {
204-
if (!renamed) {
205-
try {
206-
fs.unlinkSync(temporary)
207-
} catch {
208-
// Best effort: the caller is already failing the write.
209-
}
210-
}
211-
}
212-
// A cancelled earlier run can leave its temp file beside the summary; clean those up.
213-
try {
214-
for (const entry of fs.readdirSync(directory)) {
215-
if (entry.startsWith("." + baseName + ".") && entry.endsWith(".tmp")) {
216-
try {
217-
fs.unlinkSync(path.join(directory, entry))
218-
} catch {
219-
// Another writer may own it; never fail the summary for the sweep.
220-
}
221-
}
222-
}
223-
} catch {
224-
// The directory sweep is best effort.
225-
}
226-
}
227-
228193
/** Entry point used by the workflow: append the summary to GITHUB_STEP_SUMMARY (or stdout). */
229194
export function writeMutationSummary({
230195
reportRoot = REPORT_DIRECTORY,
@@ -234,7 +199,10 @@ export function writeMutationSummary({
234199
const { lines } = summarizeMutationReports({ reportRoot, cwd })
235200
const summary = lines.join("\n") + "\n"
236201
try {
237-
if (outputFile) appendSummaryAtomically(outputFile, summary)
202+
// The runner gives each step its own summary file and reads it after the step, so a
203+
// plain append is the whole contract here - same shape as appendSummary() in
204+
// scripts/stryker-diff.mjs.
205+
if (outputFile) fs.appendFileSync(outputFile, summary)
238206
else process.stdout.write(summary)
239207
} catch (error) {
240208
console.log(

0 commit comments

Comments
 (0)