Skip to content

Commit 18ddebd

Browse files
coverage: adoption fixes from first consumers (#436)
Three issues surfaced by early adopters of the coverage module (persistency #380, inc_someip_gateway #256): - generate_coverage_html: --yaml is now OPTIONAL. Without it, justification processing is skipped and the COVERAGE_THRESHOLD gate applies to the raw line coverage computed from the LCOV data (NOT llvm-cov's text TOTAL, which omits baseline-only files and would let untested files escape the gate). Previously the script hard-errored before extracting any HTML, which read as 'no report is generated' for repos without a justification YAML. - generate_coverage_html: new --archive-dir <dir> assembles the artifacts tree without zipping it. Preferred for CI uploads: actions/upload-artifact zips its input itself, so uploading the --archive zip nested it in a second zip. - README: document both changes, plus the requirement that the coverage bazelrc import must precede a user.bazelrc try-import (bazelrc is last-wins; the local-override file must stay last). Integration test extended: no-yaml mode (HTML produced, raw gate enforced at high/low thresholds) and --archive-dir content checks.
1 parent 617ce5e commit 18ddebd

3 files changed

Lines changed: 152 additions & 64 deletions

File tree

‎coverage/README.md‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -137,7 +137,11 @@ Everything in scope but untested shows up at 0%; everything outside the scope
137137

138138
Copy the `coverage:llvm_cov` block from
139139
[integration_tests/.bazelrc](integration_tests/.bazelrc) into your
140-
repository's bazelrc (directly or via `import`). The two labels to adapt:
140+
repository's bazelrc (directly or via `import`). If your `.bazelrc` ends with
141+
a `try-import %workspace%/user.bazelrc` (or similar local-override file),
142+
place the coverage import BEFORE it — bazelrc conflicts resolve last-wins,
143+
and the local override file must stay last to keep working. The two labels
144+
to adapt:
141145

142146
```
143147
coverage:llvm_cov --coverage_output_generator=@score_tooling//coverage:merger
@@ -189,12 +193,20 @@ bazel coverage --config=llvm_cov //... --build_tests_only
189193
bazel run @score_tooling//coverage:generate_coverage_html -- \
190194
--yaml tools/coverage/coverage_justifications.yaml
191195

192-
# CI variant: archive HTML + LCOV + JUnit XMLs, gate at 95%:
196+
# CI variant: assemble HTML + LCOV + JUnit XMLs for artifact upload, gate at 95%:
193197
COVERAGE_THRESHOLD=95 bazel run @score_tooling//coverage:generate_coverage_html -- \
194198
--yaml tools/coverage/coverage_justifications.yaml \
195-
--archive coverage_artifacts
199+
--archive-dir coverage_artifacts
200+
# then: actions/upload-artifact with path: coverage_artifacts
201+
# (upload-artifact zips its input itself — use --archive <name> only when you
202+
# want a local <name>.zip; uploading that zip would nest it in a second zip)
196203
```
197204

205+
`--yaml` is optional: without it, justification processing is skipped and the
206+
`COVERAGE_THRESHOLD` gate applies to the **raw** line coverage. Start without
207+
a YAML; add one (`version: 1` + `justifications: []`) when you introduce your
208+
first `COV_JUSTIFIED` marker.
209+
198210
`--build_tests_only` matters: without it, coverage builds (not runs) every
199211
target matched by the pattern, including e.g. `manual`-tagged or
200212
platform-incompatible test binaries.
@@ -203,7 +215,7 @@ platform-incompatible test binaries.
203215

204216
| Need | Knob |
205217
|---|---|
206-
| Different gate | `COVERAGE_THRESHOLD=<pct>` env var (default 100; exit 1 below) |
218+
| Different gate | `COVERAGE_THRESHOLD=<pct>` env var (default 100; exit 1 below; gates effective coverage with `--yaml`, raw coverage without) |
207219
| Output directory | positional `output-dir` argument (default `coverage_<platform>`) |
208220
| Platform-specific justifications | `--platform linux\|qnx` (default linux) |
209221
| JUnit XMLs subtree in the archive | `--testlogs-subdir <dir>` (default: whole `bazel-testlogs`) |

‎coverage/generate_coverage_html.sh‎

Lines changed: 111 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -18,15 +18,21 @@
1818
# Usage (from the CONSUMER workspace):
1919
# bazel coverage --config=llvm_cov //... --build_tests_only
2020
# bazel run @score_tooling//coverage:generate_coverage_html -- \
21-
# --yaml <path/to/coverage_justifications.yaml> \
22-
# [--archive <archive-name>] [--platform <platform>] \
23-
# [--testlogs-subdir <subdir>] [output-dir]
21+
# [--yaml <path/to/coverage_justifications.yaml>] \
22+
# [--archive <archive-name>] [--archive-dir <dir>] \
23+
# [--platform <platform>] [--testlogs-subdir <subdir>] [output-dir]
2424
#
2525
# Arguments:
2626
# --yaml <path> Justification YAML, relative to the workspace
27-
# root (required).
28-
# --archive <archive-name> Also create a zip archive named <archive-name>.zip
27+
# root. When omitted, justification processing and
28+
# the effective-coverage metric are skipped and the
29+
# threshold gate applies to the RAW line coverage.
30+
# --archive <archive-name> Create a zip archive named <archive-name>.zip
2931
# containing the HTML report, raw LCOV data and JUnit XMLs.
32+
# --archive-dir <dir> Assemble the same content as --archive into <dir>
33+
# WITHOUT zipping — preferred for CI artifact
34+
# uploads (actions/upload-artifact zips its input
35+
# itself; a pre-zipped file would be zipped twice).
3036
# --platform <platform> Target platform for justification filtering
3137
# (default: linux). Also affects the default output
3238
# directory (coverage_<platform>).
@@ -37,13 +43,15 @@
3743
# (default: coverage_<platform>)
3844
#
3945
# Environment:
40-
# COVERAGE_THRESHOLD Minimum effective line coverage percentage
41-
# (default: 100). The script exits non-zero when
42-
# effective coverage is below this threshold.
46+
# COVERAGE_THRESHOLD Minimum line coverage percentage (default: 100).
47+
# The script exits non-zero when the gated metric
48+
# (effective coverage with --yaml, raw coverage
49+
# without) is below this threshold.
4350

4451
set -euo pipefail
4552

4653
ARCHIVE_NAME=""
54+
ARCHIVE_DIR=""
4755
PLATFORM="linux"
4856
OUTPUT_DIR=""
4957
JUSTIFICATION_YAML_REL=""
@@ -59,6 +67,10 @@ while [[ $# -gt 0 ]]; do
5967
ARCHIVE_NAME="${2:?--archive requires a name argument}"
6068
shift 2
6169
;;
70+
--archive-dir)
71+
ARCHIVE_DIR="${2:?--archive-dir requires a directory argument}"
72+
shift 2
73+
;;
6274
--platform)
6375
PLATFORM="${2:?--platform requires a platform argument (e.g. linux or qnx)}"
6476
shift 2
@@ -74,10 +86,9 @@ while [[ $# -gt 0 ]]; do
7486
esac
7587
done
7688

77-
if [[ -z "${JUSTIFICATION_YAML_REL}" ]]; then
78-
echo "ERROR: --yaml <path/to/coverage_justifications.yaml> is required." >&2
79-
exit 1
80-
fi
89+
# --yaml is optional: without it, justification processing and the effective
90+
# coverage metric are skipped and the threshold gate applies to the RAW line
91+
# coverage from llvm-cov's summary instead.
8192

8293
# Set default output directory based on platform if not explicitly provided.
8394
if [[ -z "${OUTPUT_DIR}" ]]; then
@@ -126,80 +137,120 @@ fi
126137
echo "Coverage report written to: ${OUTPUT_DIR}"
127138

128139
# ---------------------------------------------------------------------------
129-
# Run coverage justification processing.
140+
# Run coverage justification processing (only when --yaml was given) and
141+
# enforce the coverage threshold.
130142
# ---------------------------------------------------------------------------
131-
JUSTIFICATION_YAML="${BUILD_WORKSPACE_DIRECTORY}/${JUSTIFICATION_YAML_REL}"
143+
THRESHOLD="${COVERAGE_THRESHOLD:-100}"
144+
JUSTIFICATION_DIR=""
132145

133-
if [[ ! -f "${JUSTIFICATION_YAML}" ]]; then
134-
echo "ERROR: ${JUSTIFICATION_YAML} not found." >&2
135-
exit 1
136-
fi
146+
if [[ -n "${JUSTIFICATION_YAML_REL}" ]]; then
147+
JUSTIFICATION_YAML="${BUILD_WORKSPACE_DIRECTORY}/${JUSTIFICATION_YAML_REL}"
137148

138-
echo ""
139-
echo "Running coverage justification processing..."
140-
141-
JUSTIFICATION_DIR="${TMPDIR_EXTRACT}/justification_report"
142-
mkdir -p "${JUSTIFICATION_DIR}"
143-
144-
# Run justify.py / effective_coverage.py via nested bazel invocations from the
145-
# consumer workspace. This deliberately avoids runfiles resolution across
146-
# module boundaries (canonical repo names vary between Bazel versions).
147-
bazel run @score_tooling//coverage:justify -- \
148-
--yaml "${JUSTIFICATION_YAML}" \
149-
--source-root "${BUILD_WORKSPACE_DIRECTORY}" \
150-
--platform "${PLATFORM}" \
151-
--output "${JUSTIFICATION_DIR}/manifest.json"
152-
153-
bazel run @score_tooling//coverage:effective_coverage -- \
154-
--html-dir "${OUTPUT_DIR}" \
155-
--manifest "${JUSTIFICATION_DIR}/manifest.json" \
156-
--output "${JUSTIFICATION_DIR}/report.json"
157-
158-
# Display effective coverage summary and enforce the threshold.
159-
if [[ ! -f "${JUSTIFICATION_DIR}/summary.txt" ]]; then
160-
echo "ERROR: Effective coverage summary was not produced." >&2
161-
exit 1
162-
fi
149+
if [[ ! -f "${JUSTIFICATION_YAML}" ]]; then
150+
echo "ERROR: ${JUSTIFICATION_YAML} not found." >&2
151+
exit 1
152+
fi
163153

164-
echo ""
165-
cat "${JUSTIFICATION_DIR}/summary.txt"
154+
echo ""
155+
echo "Running coverage justification processing..."
156+
157+
JUSTIFICATION_DIR="${TMPDIR_EXTRACT}/justification_report"
158+
mkdir -p "${JUSTIFICATION_DIR}"
159+
160+
# Run justify.py / effective_coverage.py via nested bazel invocations from the
161+
# consumer workspace. This deliberately avoids runfiles resolution across
162+
# module boundaries (canonical repo names vary between Bazel versions).
163+
bazel run @score_tooling//coverage:justify -- \
164+
--yaml "${JUSTIFICATION_YAML}" \
165+
--source-root "${BUILD_WORKSPACE_DIRECTORY}" \
166+
--platform "${PLATFORM}" \
167+
--output "${JUSTIFICATION_DIR}/manifest.json"
168+
169+
bazel run @score_tooling//coverage:effective_coverage -- \
170+
--html-dir "${OUTPUT_DIR}" \
171+
--manifest "${JUSTIFICATION_DIR}/manifest.json" \
172+
--output "${JUSTIFICATION_DIR}/report.json"
173+
174+
# Display effective coverage summary and enforce the threshold.
175+
if [[ ! -f "${JUSTIFICATION_DIR}/summary.txt" ]]; then
176+
echo "ERROR: Effective coverage summary was not produced." >&2
177+
exit 1
178+
fi
166179

167-
# Extract effective coverage percentage for threshold check.
168-
EFFECTIVE_PCT=$(grep -oP 'Effective line coverage:\s+\K[0-9.]+' \
169-
"${JUSTIFICATION_DIR}/summary.txt" 2>/dev/null || echo "0")
180+
echo ""
181+
cat "${JUSTIFICATION_DIR}/summary.txt"
182+
183+
# Extract effective coverage percentage for threshold check.
184+
GATE_PCT=$(grep -oP 'Effective line coverage:\s+\K[0-9.]+' \
185+
"${JUSTIFICATION_DIR}/summary.txt" 2>/dev/null || echo "0")
186+
GATE_KIND="Effective"
187+
else
188+
# No justification YAML: gate on the raw line coverage computed from the
189+
# LCOV data. Deliberately NOT llvm-cov's text summary TOTAL — that summary
190+
# omits baseline-only files (in-scope files no test links against), which
191+
# would let untested files escape the gate. The LCOV includes them.
192+
echo ""
193+
echo "INFO: no --yaml given; justification processing skipped, gating on raw line coverage."
194+
if [[ ! -f "${TMPDIR_EXTRACT}/lcov_report/lcov.dat" ]]; then
195+
echo "ERROR: lcov_report/lcov.dat not found in ${COVERAGE_REPORT}" >&2
196+
exit 1
197+
fi
198+
GATE_PCT=$(awk -F: '/^LF:/ {lf += $2} /^LH:/ {lh += $2}
199+
END { if (lf > 0) printf "%.2f", lh * 100 / lf; }' \
200+
"${TMPDIR_EXTRACT}/lcov_report/lcov.dat")
201+
if [[ -z "${GATE_PCT}" ]]; then
202+
echo "ERROR: could not compute raw line coverage from lcov_report/lcov.dat" >&2
203+
exit 1
204+
fi
205+
echo "Raw line coverage: ${GATE_PCT}%"
206+
GATE_KIND="Raw"
207+
fi
170208

171209
# Threshold check (default: 100%). Fails the run when below.
172-
THRESHOLD="${COVERAGE_THRESHOLD:-100}"
173-
if ! awk "BEGIN {exit (${EFFECTIVE_PCT} >= ${THRESHOLD}) ? 0 : 1}"; then
174-
echo "ERROR: Effective coverage ${EFFECTIVE_PCT}% is below threshold ${THRESHOLD}%" >&2
210+
if ! awk "BEGIN {exit (${GATE_PCT} >= ${THRESHOLD}) ? 0 : 1}"; then
211+
echo "ERROR: ${GATE_KIND} coverage ${GATE_PCT}% is below threshold ${THRESHOLD}%" >&2
175212
RC=1
176213
else
177214
RC=0
178215
fi
179216

180217
# ---------------------------------------------------------------------------
181-
# Optional: create a zip archive with the HTML report, raw LCOV data and
182-
# JUnit XML test results.
218+
# Optional: assemble the HTML report, raw LCOV data, justification report and
219+
# JUnit XML test results into an artifacts tree.
220+
# --archive <name> zip the tree into <name>.zip (and remove the tree)
221+
# --archive-dir <dir> keep the tree at <dir> — preferred for CI artifact
222+
# uploads, since actions/upload-artifact zips its input
223+
# anyway (a pre-zipped file would be zipped twice)
183224
# ---------------------------------------------------------------------------
184-
if [[ -n "${ARCHIVE_NAME}" ]]; then
185-
mkdir -p artifacts
225+
assemble_artifacts() {
226+
local dest="$1"
227+
mkdir -p "${dest}"
186228

187229
# Copy JUnit XML test results preserving directory structure.
188-
find "bazel-testlogs/${TESTLOGS_SUBDIR}" -name 'test.xml' -exec cp --parents {} artifacts/ \;
230+
find "bazel-testlogs/${TESTLOGS_SUBDIR}" -name 'test.xml' -exec cp --parents {} "${dest}/" \;
189231

190232
# Copy the HTML coverage report
191-
cp -r "${OUTPUT_DIR}" artifacts/
233+
cp -r "${OUTPUT_DIR}" "${dest}/"
192234

193235
# Include the LCOV .dat file from the reporter zip.
194236
if [[ -f "${TMPDIR_EXTRACT}/lcov_report/lcov.dat" ]]; then
195-
cp "${TMPDIR_EXTRACT}/lcov_report/lcov.dat" artifacts/coverage_report.dat
237+
cp "${TMPDIR_EXTRACT}/lcov_report/lcov.dat" "${dest}/coverage_report.dat"
196238
fi
197239

198240
# Include the justification report (manifest + effective coverage json).
199-
if [[ -d "${JUSTIFICATION_DIR}" ]]; then
200-
cp -r "${JUSTIFICATION_DIR}" artifacts/
241+
if [[ -n "${JUSTIFICATION_DIR}" && -d "${JUSTIFICATION_DIR}" ]]; then
242+
cp -r "${JUSTIFICATION_DIR}" "${dest}/"
201243
fi
244+
}
202245

246+
if [[ -n "${ARCHIVE_DIR}" ]]; then
247+
rm -rf "${ARCHIVE_DIR}"
248+
assemble_artifacts "${ARCHIVE_DIR}"
249+
echo "Coverage artifacts written to: ${ARCHIVE_DIR}/"
250+
fi
251+
252+
if [[ -n "${ARCHIVE_NAME}" ]]; then
253+
assemble_artifacts artifacts
203254
zip -r "${ARCHIVE_NAME}.zip" artifacts/
204255
rm -rf artifacts/
205256
echo "Coverage archive written to: ${ARCHIVE_NAME}.zip"

‎coverage/integration_tests/run_integration_test.sh‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,31 @@ COVERAGE_THRESHOLD=10 bazel run @score_tooling//coverage:generate_coverage_html
4040
--yaml "${YAML}"
4141
echo "OK: gate passed as expected"
4242

43+
echo "=== Without --yaml: HTML still produced, gate applies to RAW coverage ==="
44+
if COVERAGE_THRESHOLD=100 bazel run @score_tooling//coverage:generate_coverage_html; then
45+
echo "ERROR: raw-coverage gate passed at threshold 100" >&2
46+
exit 1
47+
fi
48+
COVERAGE_THRESHOLD=10 bazel run @score_tooling//coverage:generate_coverage_html
49+
if [[ ! -f coverage_linux/index.html ]]; then
50+
echo "ERROR: HTML report missing after no-yaml run" >&2
51+
exit 1
52+
fi
53+
echo "OK: no-yaml mode works (HTML produced, raw gate enforced)"
54+
55+
echo "=== --archive-dir must produce an unzipped artifacts tree ==="
56+
COVERAGE_THRESHOLD=10 bazel run @score_tooling//coverage:generate_coverage_html -- \
57+
--yaml "${YAML}" --archive-dir artifacts_dir
58+
for f in artifacts_dir/coverage_linux/index.html artifacts_dir/coverage_report.dat \
59+
artifacts_dir/justification_report/summary.txt; do
60+
if [[ ! -f "$f" ]]; then
61+
echo "ERROR: ${f} missing from --archive-dir output" >&2
62+
exit 1
63+
fi
64+
done
65+
rm -rf artifacts_dir
66+
echo "OK: --archive-dir works"
67+
4368
echo "=== Untested files must appear at exact 0% in the LCOV ==="
4469
unzip -p coverage_artifacts.zip artifacts/coverage_report.dat > lcov.dat
4570

0 commit comments

Comments
 (0)