Skip to content

fix(ci): keep JVM warnings out of codeql test JSON reports (stacked on #240) - #242

Merged
felickz merged 6 commits into
mainfrom
felickz/ci-test-report-json-robust
Oct 27, 2026
Merged

felickz merged 6 commits into
mainfrom
felickz/ci-test-report-json-robust

Conversation

@felickz

@felickz felickz commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Stacked on #240 (which is stacked on #238). Base is felickz-hotspots-oom-codeql-2-27-3, so the diff here is only the ci.yml change.

Problem

On the #240 CI run, all 13 compile-and-test jobs passed and every reported test passed, but "Validate test results" failed with jq: parse error: Invalid numeric literal at line 1, column 8.

The bad file was javascript-test-results/test_report_slice_1_of_4.json. Its first line was a JVM warning, not JSON:

[0.001s][warning][perf,memops] Cannot use file /tmp/hsperfdata_runner/2405 because it is locked by another process (errno = 11)

ci.yml starts four codeql test run --format=json processes at once and redirects each one's stdout directly into its report file. The JVM writes its own warnings to stdout too, so anything it prints ahead of the JSON corrupts the report. I scanned the test reports from 6 recent CI runs (about 150 files) and this was the only polluted one, so it is rare, but nothing in the workflow tolerates it.

Changes

  1. Add -J=-XX:-UsePerfData to the codeql test run command. This stops the JVM from creating the per-PID /tmp/hsperfdata_<user>/<pid> file whose lock collided. The CLI accepts -J=<opt> (confirmed locally).
  2. In "Validate test results", strip everything before the opening [ of each report (a lone [, or [] for an empty report) before running jq. A ::warning:: is emitted when something was dropped, and the step still fails if the remainder is not valid JSON or contains a failing test.

Testing

I ran the exact validate step from the workflow file under bash against sample reports:

Case Expected Result
clean report pass pass
warning line before JSON, plus a second clean slice pass with warning pass with warning
empty [] report pass pass
warning line before a report with a failing test fail fail, failing test printed
non-JSON garbage fail fail, "is not valid JSON"
no reports pass pass

The -J flag was only checked for CLI acceptance. Its effect on codeql test run is exercised by this PR's own CI run.

Notes

felickz and others added 6 commits October 21, 2026 16:54
Co-authored-by: felickz <1760475+felickz@users.noreply.github.com>
Co-authored-by: felickz <1760475+felickz@users.noreply.github.com>
…trings

getImportStringFrom compared the config file's dotted path against
Import.getImportString() with matches("%" + importString). That non-equality
string join gives the optimizer nothing to bind on, and with CodeQL 2.27.3
(github/codeql codeql-cli/v2.27.3) it planned a cartesian product of ~33k
import tuples with the 15M-row locations_default relation, filling the runner
disk.

Materialize the config's dotted-path suffixes in a pragma[nomagic] helper and
join on string equality instead.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…anches

hotspots-ci.yml only triggers for PRs targeting main, so a PR stacked on
chore/update-codeql-cli-* would get no hotspots signal. Drop this commit before
merging the fix into the bump branch.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
`ci.yml` redirects each `codeql test run --format=json` process's stdout straight into
`test_report_slice_N_of_4.json`. The JVM prints its own warnings to stdout too, so a stray
line such as

  [0.001s][warning][perf,memops] Cannot use file /tmp/hsperfdata_runner/2405 because it is
  locked by another process (errno = 11)

ahead of the JSON made `jq` fail in "Validate test results" even though every test passed.
Four JVMs start at the same instant per language job, which makes the per-PID hsperfdata
file collision possible.

1. Pass `-J=-XX:-UsePerfData` to `codeql test run` so the JVM never creates those files.
2. In "Validate test results", drop anything before the opening `[` of each report before
   running `jq`, emit a warning when something was dropped, and still fail if the remainder
   is not valid JSON.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The stacked PR has no reported end-to-end CI checks exercising the modified CodeQL invocation and report validation.

No open issues 🎉

What changed in this PR

This PR hardens CodeQL CI report handling against JVM warnings that can corrupt JSON output and fail validation despite passing tests.

Changes:

  • Disables JVM performance-data files for codeql test run to prevent the observed lock warning.
  • Removes leading output before the report’s array opener, emits a warning when output is removed, and retains JSON and test-failure validation.
File Description
.github/​workflows/​ci.yml Updates the CodeQL JVM option and sanitizes reports before validation.

Review effort: Balanced

@felickz
felickz added this pull request to stack #241 October 26, 2026 17:43
Base automatically changed from felickz-hotspots-oom-codeql-2-27-3 to main October 27, 2026 19:20
@felickz
felickz merged commit 5930377 into main Oct 27, 2026
24 checks passed
@felickz
felickz deleted the felickz/ci-test-report-json-robust branch October 27, 2026 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants