Skip to content

fix: the 2026-10-12 dry-run reds (GCC 11 sha256, fuzz link, cypher leak, memwaste budget) - #2603

Merged
DeusData merged 5 commits into
mainfrom
fix/dry-run-reds-2026-10-12
Oct 12, 2026
Merged

DeusData merged 5 commits into
mainfrom
fix/dry-run-reds-2026-10-12

Conversation

@DeusData

Copy link
Copy Markdown
Owner

The 2026-10-12 dry run (run 38186491924, main 475db1e) was red on four causes. This PR fixes three of them, plus a Cypher parser leak the fuzz job found as soon as it could build. The fourth cause, a doc_mentions_go hang on windows-11-arm, did not reproduce locally (0 hangs in 32 runs on a Windows arm64 VM) and is not touched here; the dry run on this branch is its second CI sample.

dry-run job cause fix
test-unix ubuntu-22.04-arm, all 3 shards src/foundation/sha256.c (#2441) did not compile with GCC 11: its ARMv8 path was marked target("+sha2"), but GCC 11's arm_neon.h declares the SHA2 intrinsics under +nothing+crypto, so they could not be inlined. PR CI builds only with GCC 13 (ubuntu-24.04-arm); ubuntu-22.04-arm runs only in the dry run and the release. GCC gets target("+crypto"), which implies +sha2 on every GCC. clang keeps sha2.
fuzz the fuzz binary links through the C driver, which on Linux leaves out UBSan's C++ runtime; the C++ objects' -fsanitize=vptr checks need it (__ubsan_vptr_type_cache). The job has never built in CI since it was added on 09-18. -fsanitize-link-c++-runtime on the link line.
fuzz, once it links the cypher target leaks on a checked-in seed, MATCH p = (a:Class)-[:INHERITS*1..3]->(b:Class) … (288 bytes, 2 allocations). parse_match_pattern returned on a parse error without freeing the pattern it had started, and callers count a pattern only after it parsed. Every named path (p = (...), not supported) leaked, as did the failed element's own allocations and parse_props' already-parsed entries. free and zero the partial pattern on every failure (counting the failed element, which parse_node / parse_rel zero first); parse_props frees its parsed entries through one helper. New test cypher_parse_error_frees_partial_pattern covers five failure points.
memwaste / event-macos scripts/memwaste.sh set no budget, so the daemon derived one from the runner's RAM: 1792 MB on the 7 GB macos-14 runner, about 4 GB on the 16 GB Linux and Windows runners. The event layer itself holds 0.5-1 GB, and the index stopped with over_memory_budget before measuring anything (10-04 and 10-12). memwaste.sh passes one explicit CBM_MEM_BUDGET_MB=4096, what the larger runners already derived, unless the caller sets one.

Checks

  • sha256, arm64 containers:
    • sha256.c with CI's ASan/UBSan flags compiles on GCC 11, 12, 13 and 14; on GCC 11 the old +sha2 fails with the CI errors.
    • A known-answer harness ("abc", "", a 1 MiB pattern) matches Python's hashlib on the arm-sha2 backend and on the forced portable backend, with GCC 11 and 14 at -O0 and -O2.
  • fuzz, ubuntu:24.04 container with CI's packages (clang 18, llvm, libclang-rt-dev):
    • minimal reproduction (a C++ virtual call under -fsanitize=undefined, linked through clang): without the flag, CI's two undefined symbols; with it, it links and runs. macOS Homebrew clang accepts the flag under -Werror.
    • the real make fuzz links. One deviation: lsp_all.o was compiled with -g0 added, because the 12.5 GB container ran out of memory compiling it with full debug info (CI's runner gets past it).
    • the CI smoke, scripts/fuzz.sh all --runs 20000 --seed 1: first the cypher leak above. With the Cypher fix, extract, cypher and config each complete 20000 runs.
  • cypher leak, the fuzz binary on five failing queries: before the fix the first four leak (288-348 bytes, LSan rc 77); after it all five are clean. macOS: the cypher suite passes, 217 tests, including the new one.
  • Lint on the changed C files: Homebrew clang-format and CI's cppcheck 2.20 are clean.
  • memwaste, this Mac with CBM_WORKERS=3 (the macos-14 runner's core count):
    • budget 1792: over_memory_budget in parallel_extract (RSS 3904 MB), the CI failure;
    • budget 4096: the index completes and the report is written;
    • the edited script with no budget in the environment applies 4096 and exits 0.

The target gate is the dry run dispatched on this branch.

GCC 11's arm_neon.h declares the SHA2 intrinsics under
`#pragma GCC target ("+nothing+crypto")`. sha256_blocks_arm was marked
`target("+sha2")`, a narrower target, so GCC 11 refused to inline
vsha256hq_u32 and friends ("target specific option mismatch") and the
build stopped. GCC 13 accepts "+sha2", which is why PR CI (ubuntu-24.04-arm)
stayed green; the dry run's ubuntu-22.04-arm leg (GCC 11) failed on all
three shards (run 38186491924).

GCC now gets "+crypto", which implies "+sha2" on every GCC. The function
still uses only SHA2 instructions, and the runtime HWCAP check gates it.
clang keeps "sha2".

Checked in arm64 containers:
- sha256.c with CI's ASan/UBSan flags compiles on GCC 11, 12, 13 and 14;
  GCC 11 with "+sha2" fails with the CI errors;
- a known-answer harness ("abc", "", a 1 MiB pattern) matches hashlib on
  the arm-sha2 backend and the forced portable backend, GCC 11 and 14,
  -O0 and -O2.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
scripts/memwaste.sh set no memory budget, so the daemon derived one from
the machine's RAM: 1792 MB on the 7 GB macos-14 runner, about 4 GB on the
16 GB Linux and Windows runners. The event layer itself holds 0.5-1 GB, and
on macOS the index stopped with over_memory_budget before it measured
anything, in both full-mode runs since the lane was added (dry runs
37212102899 and 38186491924, peaks 3449 and 3111 MB at the stop).

The script now passes CBM_MEM_BUDGET_MB=4096, what the larger runners
already derived, unless the caller sets one.

Checked on macOS arm64 with CBM_WORKERS=3, the macos-14 runner's core
count:
- budget 1792: over_memory_budget in parallel_extract (rss 3904 MB), the
  CI failure;
- budget 4096: the index completes and the report is written;
- the edited script with no budget in the environment: 4096 is applied,
  rc 0.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The cbm-fuzz rule links through $(CC), the C driver. On Linux, clang's C
driver links UBSan's C runtime but leaves out its C++ part
(ubsan_standalone_cxx), while the C++ objects (preprocessor.cpp and the
vendored simplecpp) carry -fsanitize=vptr checks. The link failed with
undefined references to __ubsan_vptr_type_cache and
__ubsan_handle_dynamic_type_cache_miss, so the fuzz job has never built in
CI since it was added (dry runs 37212102899 and 38186491924).

-fsanitize-link-c++-runtime on the link line pulls that runtime in.

Checked in an ubuntu:24.04 arm64 container with CI's packages (clang 18,
llvm, libclang-rt-dev):
- a minimal reproduction (a C++ virtual call under -fsanitize=undefined,
  linked through clang) fails with exactly those two symbols without the
  flag, and links and runs with it;
- `make fuzz` links cbm-fuzz (lsp_all.o was compiled with -g0 added: the
  12.5 GB container ran out of memory on it with full debug info, which
  CI's runner does not);
- macOS Homebrew clang accepts the flag under -Werror.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
parse_match_pattern allocated the pattern's node and relationship arrays
and returned on a parse error without freeing them. Its callers count a
pattern only after it parsed, so cbm_query_free never saw it: every query
whose pattern failed partway leaked, including every named path
(`MATCH p = (...)`, not supported, fails at the first node). The element
whose parse failed leaked too (its variable, label or properties), and
parse_props freed only the array, not the entries it had already parsed.

The fuzz job found it on its first run, from a checked-in seed:
`MATCH p = (a:Class)-[:INHERITS*1..3]->(b:Class) RETURN a.name, length(p)`
(LeakSanitizer: 288 bytes in 2 allocations).

parse_match_pattern now frees and zeroes the pattern on every failure,
counting the failed element (parse_node and parse_rel zero it first, so
only what they allocated is freed). parse_props frees its parsed entries
on every failure through one helper.

New test cypher_parse_error_frees_partial_pattern drives five failure
points: a named path, a property list after one entry, a cut-off
relationship, a missing label after a relationship, and a second MATCH
clause's properties.

Checked with the fuzz binary (clang 18 ASan/LSan, arm64 container) on the
five queries: before this change the first four leak (288, 306, 339 and
348 bytes); after it all five are clean, and scripts/fuzz.sh all --runs
20000 --seed 1 passes for extract, cypher and config. The fifth goes
through the same parse_props failure as the second. macOS: the cypher
suite passes, 217 tests.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Smoke's binary string audit blocked on every platform in the dry run on
70543ba (run 38204190338, all 12 smoke jobs): "Credential patterns
found in binary". #2068 rebuilt the graph-ui bundle. The two reviewed
Monaco 0.56.0 quick-input lines are still there but minified to different
bytes, so their SHA-256 no longer matched the pair recorded on 09-09.

Re-reviewed: the CI-built bundle has exactly two flagged lines (15,644 and
72,472 bytes). Every match is the input box's password-mode flag
(_password=!1, password=!!e.password, inputBox.password=this.password,
inputBox.password=!1), not a credential. All 12 smoke logs show the same
two lines. The new pair is computed as filter_reviewed_lines does
(printf '%s' line | sha256) on CI's binaries: GNU strings on the Linux
amd64 artifact and Apple strings on the darwin arm64 artifact give the same
two hashes, and both are exempted. The comment now says how to redo this
when the bundle changes again.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData
DeusData merged commit 37a3d32 into main Oct 12, 2026
24 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant