Skip to content

Commit 124ce0e

Browse files
committed
Merge main into Ruby namespaced constructor fix
Refresh the contributor branch onto current main while preserving the reviewed constructor extraction change and regression coverage. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2 parents fb679e0 + 402cc81 commit 124ce0e

100 files changed

Lines changed: 13057 additions & 1524 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/codeql/codeql-config.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# CodeQL scope: the repository's first-party source.
2+
#
3+
# Read by .github/workflows/codeql.yml (every scan, full and PR) and by
4+
# scripts/ci/select-lanes.sh (which changed paths select which language's PR
5+
# scan). Keep the lists plain: one path prefix per line, no globs.
6+
#
7+
# paths / paths-ignore narrow what the INTERPRETED languages extract
8+
# (javascript-typescript, python, actions). For c-cpp they have no effect:
9+
# CodeQL extracts exactly what the traced build compiles. A PR's c-cpp scan
10+
# keeps vendored C out by compiling it before tracing starts (except vendored
11+
# files the PR changes); the full scan on main, the dry run and the release
12+
# traces everything, vendored C included.
13+
name: cbm first-party source
14+
paths:
15+
- src
16+
- internal
17+
- graph-ui
18+
- pkg
19+
- scripts
20+
- tests
21+
- .github
22+
paths-ignore:
23+
- internal/cbm/vendored
24+
- graph-ui/vendor
25+
- graph-ui/fixtures
26+
- graph-ui/verification
27+
- tests/fixtures

‎.github/workflows/_lint.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# Reusable: lint only (cppcheck + clang-format).
2-
# Security-static and CodeQL gate are separate — see _security.yml.
2+
# Security-static is separate — see _security.yml; CodeQL — see codeql.yml.
33
name: Lint & Security
44

55
on:

‎.github/workflows/_security.yml‎

Lines changed: 3 additions & 94 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
1-
# Reusable: security-static + CodeQL gate.
1+
# Reusable: security-static + license gate.
2+
# CodeQL is not here: PRs do not run it. It runs on push to main, weekly, and
3+
# inside dry-run.yml / release.yml, which call codeql.yml directly.
24
# Runs independently from lint/test/build — does not block the main pipeline.
35
# Affects overall workflow success status.
46
name: Security Gate
@@ -52,96 +54,3 @@ jobs:
5254
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
5355
run: scripts/audit-license-provenance.py
5456

55-
codeql-gate:
56-
if: ${{ inputs.lanes == 'all' || contains(fromJSON(inputs.lanes), 'codeql-gate') }}
57-
runs-on: ubuntu-latest
58-
timeout-minutes: 240
59-
# Without security-events:read this job inherits the workflow's
60-
# contents:read, the code-scanning alert API answers 403, and the gate
61-
# below reported "0 alerts" on every run it has ever made. actions:read is
62-
# what the CodeQL-run poll above needs for the same reason.
63-
permissions:
64-
contents: read
65-
security-events: read
66-
actions: read
67-
steps:
68-
- name: Wait for CodeQL on current commit (max 150 min)
69-
env:
70-
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
71-
run: |
72-
# On pull_request events github.sha is the synthetic merge commit;
73-
# CodeQL runs are recorded against the PR head SHA.
74-
CURRENT_SHA="${{ github.event.pull_request.head.sha || github.sha }}"
75-
echo "Waiting for CodeQL to complete on $CURRENT_SHA..."
76-
# 300 attempts x 30s = 150 min. The previous budget was 90 x 30s = 45 min,
77-
# which is shorter than CodeQL actually takes on this repository: a measured
78-
# run on PR #1426 (head 7b72652a) completed with conclusion=success after
79-
# 124 min, 7 minutes AFTER this gate had already given up. That marked at
80-
# least five contributor PRs red for a scan that passed. The job's own
81-
# timeout-minutes is 240, so 150 still leaves headroom.
82-
for attempt in $(seq 1 300); do
83-
LATEST=$(gh api "repos/${{ github.repository }}/actions/workflows/codeql.yml/runs?head_sha=$CURRENT_SHA&per_page=1" \
84-
--jq '.workflow_runs[] | "\(.conclusion) \(.status)"' 2>/dev/null | head -1 || echo "")
85-
if [ -z "$LATEST" ]; then
86-
echo " $attempt/300: no run yet..."; sleep 30; continue
87-
fi
88-
CONCLUSION=$(echo "$LATEST" | cut -d' ' -f1)
89-
STATUS=$(echo "$LATEST" | cut -d' ' -f2)
90-
if [ "$STATUS" = "completed" ] && [ "$CONCLUSION" = "success" ]; then
91-
echo "=== CodeQL passed ==="; exit 0
92-
elif [ "$STATUS" = "completed" ]; then
93-
echo "BLOCKED: CodeQL $CONCLUSION"; exit 1
94-
fi
95-
echo " $attempt/300: $STATUS..."; sleep 30
96-
done
97-
echo "BLOCKED: CodeQL timeout"; exit 1
98-
99-
- name: Check for open code scanning alerts
100-
env:
101-
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
102-
run: |
103-
# FAIL CLOSED. The previous form was `gh api ... 2>/dev/null || echo "0"`,
104-
# which looks defensive and is the opposite. gh writes the API error
105-
# BODY to stdout, so a 403 made ALERTS the string
106-
# {"message":"Resource not accessible by integration",...,"status":"403"}0
107-
# every `[` comparison then failed as non-integer, the `if` took its
108-
# false branch, and the step printed "CodeQL gate passed (0 alerts)"
109-
# and exited 0. That is exactly how a security gate should NEVER fail:
110-
# silently, in the direction of "everything is fine". An unreadable
111-
# alert list is now an error, not a zero.
112-
# NOTE: this runs in a command substitution, so it must `return`, not
113-
# `exit` -- an `exit` here would only end the subshell and let the
114-
# caller continue with an empty count, recreating the very bug this
115-
# replaces. The callers below check the status explicitly.
116-
count_open_alerts() {
117-
local body status
118-
body=$(gh api 'repos/${{ github.repository }}/code-scanning/alerts?state=open' \
119-
--jq 'length' 2>/dev/null)
120-
status=$?
121-
if [ $status -ne 0 ]; then
122-
echo "BLOCKED: cannot read code scanning alerts (gh exit $status)." >&2
123-
echo " Refusing to treat an unreadable alert list as zero." >&2
124-
echo " Most likely cause: the job lacks security-events:read." >&2
125-
return 1
126-
fi
127-
case "$body" in
128-
''|*[!0-9]*)
129-
echo "BLOCKED: alert count was not a number: ${body:-<empty>}" >&2
130-
return 1
131-
;;
132-
esac
133-
printf '%s' "$body"
134-
}
135-
echo "Waiting 60s for alert API to settle..."
136-
sleep 60
137-
ALERTS=$(count_open_alerts) || exit 1
138-
sleep 15
139-
ALERTS2=$(count_open_alerts) || exit 1
140-
[ "$ALERTS" -lt "$ALERTS2" ] && ALERTS=$ALERTS2
141-
if [ "$ALERTS" -gt 0 ]; then
142-
echo "BLOCKED: $ALERTS open alert(s)"
143-
gh api 'repos/${{ github.repository }}/code-scanning/alerts?state=open' \
144-
--jq '.[] | " #\(.number) [\(.rule.security_severity_level // .rule.severity)] \(.rule.id) — \(.most_recent_instance.location.path):\(.most_recent_instance.location.start_line)"' 2>/dev/null || true
145-
exit 1
146-
fi
147-
echo "=== CodeQL gate passed (0 alerts) ==="

‎.github/workflows/codeql.yml‎

Lines changed: 163 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,47 +1,194 @@
11
name: CodeQL SAST
22

3+
# One analysis per language: c-cpp, javascript-typescript (graph-ui, pkg/npm),
4+
# python, actions. Scope: .github/codeql/codeql-config.yml (first-party source).
5+
#
6+
# FULL scan -- push to main, weekly, dry run, release: every language; c-cpp
7+
# traces the whole production build, vendored C included.
8+
# PR scan -- pr.yml's `codeql` job (light: true), needed by ci-ok. One leg per
9+
# language whose scope the PR changes (codeql-* lanes of
10+
# scripts/ci/select-lanes.sh; an undeterminable diff or a change to the
11+
# workflows, the build or the CodeQL machinery selects every language).
12+
# Each leg analyses its language's WHOLE first-party scope, so cross-file
13+
# data flow is kept; c-cpp compiles the vendored objects before tracing
14+
# starts, except vendored files the PR changes. Diff-informed analysis then
15+
# restricts the REPORTED alerts to the changed lines; what is extracted and
16+
# analysed is never narrowed by the diff.
17+
# There is no pull_request trigger and no job anywhere that polls for a CodeQL
18+
# run: the former PR gate waited up to 150 min per PR for a full analysis and,
19+
# with ~30 PRs pushed at once, starved PR CI for hours.
320
on:
421
push:
522
branches: [main]
6-
pull_request:
7-
branches: [main]
23+
schedule:
24+
- cron: "23 5 * * 1"
25+
workflow_dispatch:
26+
inputs:
27+
light:
28+
description: "PR-style c-cpp build (vendored objects compiled untraced) -- to time it"
29+
type: boolean
30+
default: false
31+
workflow_call:
32+
inputs:
33+
light:
34+
description: "PR scan: vendored C compiled untraced except changed files; diff-informed results"
35+
type: boolean
36+
default: false
37+
matrix:
38+
description: 'The legs to run, {"include":[...]} (pr.yml: the selected codeql-* lanes); empty = every language'
39+
type: string
40+
default: ""
841

942
permissions:
1043
contents: read
1144

12-
# One analysis per pull request and one per pushed branch. A newer push to main
13-
# cancels the analysis of an older main commit; only the newest commit's
14-
# results matter for main, and a merge-heavy day otherwise queues one 40-100
15-
# minute analysis per merge ahead of every pull request run.
16-
concurrency:
17-
group: codeql-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}
18-
cancel-in-progress: true
19-
2045
jobs:
2146
analyze:
2247
runs-on: ubuntu-latest
23-
# CodeQL needs security-events: write to upload results; scoped to this job so
24-
# the workflow's top-level token stays read-only (Scorecard TokenPermissions).
48+
strategy:
49+
fail-fast: false
50+
# The same legs (and keys) everywhere, so a PR's analysis and main's
51+
# share one code-scanning configuration per language.
52+
matrix: ${{ fromJSON(inputs.matrix || '{"include":[{"language":"c-cpp","build-mode":"manual","lane":"codeql-cpp"},{"language":"javascript-typescript","build-mode":"none","lane":"codeql-js"},{"language":"python","build-mode":"none","lane":"codeql-py"},{"language":"actions","build-mode":"none","lane":"codeql-actions"}]}') }}
53+
# Superseded runs are always cancelled: a newer push to main cancels the
54+
# analysis of an older main commit (batched merges -> only the newest main
55+
# commit is scanned), a newer push to a PR cancels that PR's scan. Every
56+
# other trigger gets a group of its own and is never cancelled: a dry run
57+
# or a release tests ONE commit, and two dispatched together must not
58+
# cancel each other's gate. Per language, so legs never cancel each other.
59+
# In a called workflow `github` is the CALLER's context: event_name is
60+
# pull_request for pr.yml, github.workflow names the dry run or release.
61+
concurrency:
62+
group: codeql-${{ matrix.language }}-${{ github.event_name == 'push' && github.ref || github.event_name == 'pull_request' && format('pr-{0}', github.event.pull_request.number) || format('{0}-{1}', github.workflow, github.run_id) }}
63+
cancel-in-progress: ${{ github.event_name == 'push' || github.event_name == 'pull_request' }}
64+
# CodeQL needs security-events: write to upload results (and read for the
65+
# alert check); scoped to this job so the workflow's top-level token stays
66+
# read-only (Scorecard TokenPermissions).
2567
permissions:
2668
security-events: write
2769
actions: read
2870
contents: read
71+
env:
72+
# One code-scanning configuration for every trigger. The action derives
73+
# the key from the RUN's workflow file, which for a called workflow is
74+
# the caller: pr.yml / dry-run.yml / release.yml would each open a
75+
# configuration of their own, and on main an alert fixed by a later push
76+
# would stay open in a stale one. The value is the key the push-to-main
77+
# analysis has always used (the action documents this variable as
78+
# settable for non-standard setups).
79+
CODEQL_ACTION_ANALYSIS_KEY: .github/workflows/codeql.yml:analyze
80+
# Diff-informed analysis: codeql-action v4.38.1 (CLI 2.27.0) reports a
81+
# pull request's alerts only on its changed lines when the
82+
# diff_informed_queries feature is on (default on; this variable forces
83+
# it, src/feature-flags.ts). It applies only when the event payload is a
84+
# pull request -- pr.yml's call -- and is inert on main, dry run, release.
85+
CODEQL_ACTION_DIFF_INFORMED_QUERIES: ${{ inputs.light && 'true' || '' }}
86+
# No compiler cache: a cache hit skips the compiler, and CodeQL extracts
87+
# only what a traced compiler invocation actually compiles.
88+
CBM_NO_CCACHE: "1"
2989
steps:
3090
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
91+
with:
92+
# The PR scan reads the merge commit's first parent (the base) to
93+
# find the vendored files the PR changes.
94+
fetch-depth: ${{ inputs.light && 2 || 1 }}
3195

3296
- name: Install build dependencies
97+
if: ${{ matrix.language == 'c-cpp' }}
3398
run: sudo apt-get update && sudo apt-get install -y zlib1g-dev
3499

100+
# PR c-cpp: compile the third-party objects (tree-sitter grammars,
101+
# sqlite, mimalloc, lz4, zstd, the tree-sitter runtime: ~1.3 GB of
102+
# vendored C against ~45 MB first-party) BEFORE tracing starts, so the
103+
# traced build below compiles first-party code only ...
104+
- name: Build third-party objects (PR scan, untraced)
105+
if: ${{ matrix.language == 'c-cpp' && inputs.light }}
106+
run: scripts/build.sh --vendored-only
107+
108+
# ... except the vendored files this PR changes: those are rebuilt
109+
# under tracing (an undeterminable diff leaves the list empty; the scan
110+
# still covers all first-party C).
111+
- name: Vendored files this PR changes (traced too)
112+
if: ${{ matrix.language == 'c-cpp' && inputs.light }}
113+
run: scripts/ci/codeql-retrace-vendored.sh "$RUNNER_TEMP/codeql-retrace.txt"
114+
35115
- name: Initialize CodeQL
36116
uses: github/codeql-action/init@1c5b675653bb5c22dbe9b12b556ec555138e09fd # v4.38.1
37117
with:
38-
languages: c-cpp
39-
build-mode: manual
118+
languages: ${{ matrix.language }}
119+
build-mode: ${{ matrix.build-mode }}
120+
config-file: ./.github/codeql/codeql-config.yml
40121

41-
- name: Build for CodeQL analysis
122+
- name: Build for CodeQL analysis (full)
123+
if: ${{ matrix.language == 'c-cpp' && !inputs.light }}
42124
run: scripts/build.sh
43125

126+
- name: Build for CodeQL analysis (PR scan, first-party + changed vendored)
127+
if: ${{ matrix.language == 'c-cpp' && inputs.light }}
128+
run: scripts/build.sh --reuse-vendored --retrace-from "$RUNNER_TEMP/codeql-retrace.txt"
129+
44130
- name: Perform CodeQL Analysis
45131
uses: github/codeql-action/analyze@1c5b675653bb5c22dbe9b12b556ec555138e09fd # v4.38.1
46132
with:
47-
category: "/language:c-cpp"
133+
category: "/language:${{ matrix.language }}"
134+
135+
# Alert policy: the default branch carries ZERO open code-scanning
136+
# alerts. Any open alert on the analysed ref fails this leg: on main the
137+
# commit turns red and GitHub notifies whoever triggered the run; the
138+
# dry run and the release cannot pass; a PR (refs/pull/<n>/merge: what
139+
# its changed lines introduce) fails ci-ok. With a zero baseline "any
140+
# open alert" is exactly "a new alert"; a dismissal (with a reason, in
141+
# the Security tab) holds on every ref. Every leg checks after its own
142+
# upload, so each language's alerts are seen by at least its own leg.
143+
# Moved from _security.yml's former PR-CI codeql-gate job.
144+
- name: Check for open code scanning alerts
145+
env:
146+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
147+
ALERT_REF: ${{ github.ref }}
148+
run: |
149+
# FAIL CLOSED. The previous form was `gh api ... 2>/dev/null || echo "0"`,
150+
# which looks defensive and is the opposite. gh writes the API error
151+
# BODY to stdout, so a 403 made ALERTS the string
152+
# {"message":"Resource not accessible by integration",...,"status":"403"}0
153+
# every `[` comparison then failed as non-integer, the `if` took its
154+
# false branch, and the step printed "CodeQL gate passed (0 alerts)"
155+
# and exited 0. That is exactly how a security gate should NEVER fail:
156+
# silently, in the direction of "everything is fine". An unreadable
157+
# alert list is now an error, not a zero.
158+
# NOTE: this runs in a command substitution, so it must `return`, not
159+
# `exit` -- an `exit` here would only end the subshell and let the
160+
# caller continue with an empty count, recreating the very bug this
161+
# replaces. The callers below check the status explicitly.
162+
count_open_alerts() {
163+
local body status
164+
body=$(gh api "repos/${{ github.repository }}/code-scanning/alerts?state=open&ref=$ALERT_REF" \
165+
--jq 'length' 2>/dev/null)
166+
status=$?
167+
if [ $status -ne 0 ]; then
168+
echo "BLOCKED: cannot read code scanning alerts (gh exit $status)." >&2
169+
echo " Refusing to treat an unreadable alert list as zero." >&2
170+
echo " Most likely cause: the job lacks security-events:read." >&2
171+
return 1
172+
fi
173+
case "$body" in
174+
''|*[!0-9]*)
175+
echo "BLOCKED: alert count was not a number: ${body:-<empty>}" >&2
176+
return 1
177+
;;
178+
esac
179+
printf '%s' "$body"
180+
}
181+
echo "Waiting 60s for alert API to settle..."
182+
sleep 60
183+
ALERTS=$(count_open_alerts) || exit 1
184+
sleep 15
185+
ALERTS2=$(count_open_alerts) || exit 1
186+
[ "$ALERTS" -lt "$ALERTS2" ] && ALERTS=$ALERTS2
187+
if [ "$ALERTS" -gt 0 ]; then
188+
echo "::error title=Open code scanning alerts::$ALERTS open alert(s) on $ALERT_REF -- fix, or dismiss with a reason in the Security tab"
189+
echo "BLOCKED: $ALERTS open alert(s)"
190+
gh api "repos/${{ github.repository }}/code-scanning/alerts?state=open&ref=$ALERT_REF" \
191+
--jq '.[] | " #\(.number) [\(.rule.security_severity_level // .rule.severity)] \(.rule.id) — \(.most_recent_instance.location.path):\(.most_recent_instance.location.start_line)"' 2>/dev/null || true
192+
exit 1
193+
fi
194+
echo "=== CodeQL gate passed (0 alerts) ==="

‎.github/workflows/dry-run.yml‎

Lines changed: 17 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,10 @@
22
# Each step can be skipped for faster iteration.
33
#
44
# Pipeline: lint → test → build + candidate scan/select → smoke/soak
5-
# Security: security-static + codeql-gate (independent island)
5+
# Security: security-static + license-gate (independent island)
6+
# CodeQL: codeql.yml full scan of this commit + alert check (independent island)
67
#
7-
# Security does NOT block lint/test/build/smoke/soak. All jobs must pass
8+
# Security and CodeQL do NOT block lint/test/build/smoke/soak. All jobs must pass
89
# for the overall workflow to be green.
910
#
1011
# Candidate VirusTotal scanning and tuple-local selection are part of the
@@ -62,15 +63,24 @@ jobs:
6263
# ── Security (independent island — does not block main pipeline) ──
6364
security:
6465
uses: ./.github/workflows/_security.yml
65-
# The called job requests security-events:read to count code-scanning
66-
# alerts; a reusable workflow cannot request more than its caller grants,
67-
# so withholding this here fails the whole run at startup.
66+
# No job in _security.yml reads code scanning or the Actions API any more
67+
# (the CodeQL gate moved to codeql.yml), so the token stays read-only.
6868
permissions:
6969
contents: read
70-
security-events: read
71-
actions: read
7270
secrets: inherit
7371

72+
# ── CodeQL: full scan of this exact commit + the open-alert check ──
73+
# Every language, vendored C included (PRs run the narrower PR scan).
74+
# codeql.yml uploads results and reads the alerts: a called workflow
75+
# cannot request more than its caller grants, so withholding either
76+
# permission fails the whole run at startup.
77+
codeql:
78+
uses: ./.github/workflows/codeql.yml
79+
permissions:
80+
contents: read
81+
security-events: write
82+
actions: read
83+
7484
# ── Lint (cppcheck + clang-format) ────────────────────────────
7585
lint:
7686
if: ${{ inputs.skip_lint != true }}

0 commit comments

Comments
 (0)