Skip to content

Commit f0ce432

Browse files
shenxianpengclaude
andauthored
feat: render a skipped check as skipped, not as a green tick (#260)
* feat: render a skipped check as skipped, not as a green tick PR #258 in this repository reported "All 5 checks passed" over five green ticks while validating nothing: its author is dependabot[bot], which the org config lists in ignore_authors, so every rule was bypassed. The report had no way to say so -- ScopeResult knew only pass and fail, and a skip arrived looking exactly like a pass. The bare labels in that summary were the only hint anything was different, and only because a skipped check reports no value. That was incidental, not a signal. commit-check 2.13.4 reports "status": "skip", so this consumes it: * ScopeResult.status gains "skip", set when every check in the scope skipped. One real verdict outranks the skips. * Skipped scopes render "⊘ <label> (skipped)" -- deliberately not a ✔, and with no value, because nothing was examined. * The headline distinguishes the three cases. All skipped reads "⊘ All N checks skipped — nothing was validated"; a partial skip reads "✅ 3 of 5 checks passed, 2 skipped" rather than claiming all passed. * The step log line follows the same rule. Back-compatible by construction: against an engine that never emits "skip" no branch here is reachable, and a test pins that the old rendering is unchanged. Adds a golden test for the fully skipped report so the layout stays exact, alongside tests for the partial-skip headline, failure precedence, and the one-real-verdict rule. The output specification comment and the README gain the skipped case, the README's rendered from the real renderer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U9zFxq8V4qxG4aMzJhGBFn * fix: stop reporting a skipped run as a failure Review found the same defect this PR set out to fix, one layer down. run_commit_check has always failed only on "fail", but add_job_summary, add_pr_comments (two call sites) and set_result_output each asked `all(scope.status == "pass")` instead. Those were equivalent only while pass and fail were the only statuses. With skip added, a skipped-only run rendered "⊘ All N checks skipped" and then exited 1 and emitted result status "fail" -- a bypassed policy turned into a broken build. Four copies of a two-state assumption is the same shape as the four copies of the reduce-to-overall rule in commit-check#537, so it gets the same remedy: overall_status() and exit_code_for(), defined once and used by every path. Three smaller findings, all real: * _markdown_table skipped only `pass` scopes, so a run with both a failure and a skip added a row with no value and no rule links -- a blank accusation under a "Failed checks" heading. It now takes failures only. * The all-skipped verdict fired on an empty result set, where `skipped == total` is trivially true, reporting "All 0 checks skipped". It now requires a scope, matching the guard the step log already had. * The step log said "all checks passed (2 skipped)" for a partial skip, contradicting its own report headline. It now reads "2 of 3 checks passed, 1 skipped". Each fix is pinned by a test that fails when the fix is reverted, checked one at a time. Not changed: the requirements.txt pin. The README documents 2.13.4 as the minimum for ⊘ to appear, and 2.13.4 does not exist yet -- commit-check#537 is still open. Pinning it now would install a version that cannot be resolved. The rendering is back-compatible by construction, so the bump belongs with the release, not here. The subject avoids "treat", which reads as imperative but is absent from the 396-verb list in commit-check 2.13.1 -- the version this action still pins. It was added by commit-check#527 and shipped in 2.13.2, so the whitelist on the runner is three releases behind the project's own. The pin bump in #258 fixes that; this only sidesteps it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U9zFxq8V4qxG4aMzJhGBFn --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 parent e6270e1 commit f0ce432

3 files changed

Lines changed: 302 additions & 13 deletions

File tree

‎README.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -303,6 +303,40 @@ A scope is one thing that was checked — a commit message, the branch, the auth
303303
— not one rule evaluation, so the total matches the ✔/✖ lines you can count and
304304
does not grow with the number of rules in your config.
305305

306+
### Skipped Job Summary
307+
308+
Some runs validate nothing at all — most commonly when the commit author is
309+
listed in `ignore_authors`, which is how Dependabot and other bots are usually
310+
exempted. Those runs report `⊘`, never `✔`:
311+
312+
> <img src="https://raw.githubusercontent.com/commit-check/commit-check-action/main/assets/logo.png" width="20" align="top" alt=""> **Commit Check**
313+
>
314+
> ⊘ **All 3 checks skipped** — nothing was validated
315+
>
316+
> <details>
317+
> <summary>Show all 3 checks</summary>
318+
>
319+
> ```text
320+
> Commit message
321+
> ⊘ PR title (skipped)
322+
> ⊘ Commit 1/1 (skipped)
323+
> Branch
324+
> ⊘ Branch (skipped)
325+
> ```
326+
>
327+
> </details>
328+
>
329+
> _commit-check &lt;version&gt; · [Rules reference](https://commit-check.com/rules/)_
330+
331+
A skipped scope carries no checked value, because nothing was examined. When
332+
only some scopes skip, the verdict counts them separately —
333+
`✅ **3 of 5 checks passed**, 2 skipped` — so the headline never claims a pass
334+
that did not happen. Failures still take precedence over skips.
335+
336+
This needs commit-check 2.13.4 or newer, which reports `"status": "skip"` in
337+
its JSON. Against an older engine every check is `pass` or `fail` as before,
338+
and the report is unchanged.
339+
306340
## GitHub Pull Request Comments
307341

308342
With `pr-comments: true` the same report is posted as a pull request comment.

‎main.py‎

Lines changed: 109 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -101,17 +101,57 @@ class ScopeResult:
101101

102102
@property
103103
def status(self) -> str:
104-
"""Overall status: ``pass`` when every check passed."""
104+
"""Overall status: ``pass``, ``fail``, or ``skip``.
105+
106+
``skip`` means every rule in this scope declined to run — the author
107+
is on an ``ignore_authors`` list, or there was nothing to check. It
108+
is reported separately from ``pass`` because a skipped scope
109+
validated nothing, and rendering the two identically let a bypassed
110+
policy read as an enforced one.
111+
112+
A single real verdict outranks the skips: a scope is ``skip`` only
113+
when *all* of its checks skipped.
114+
"""
105115
if self.raw_text and not self.checks:
106116
return "fail"
107-
return "fail" if any(c["status"] == "fail" for c in self.checks) else "pass"
117+
if any(c["status"] == "fail" for c in self.checks):
118+
return "fail"
119+
if self.checks and all(c["status"] == "skip" for c in self.checks):
120+
return "skip"
121+
return "pass"
108122

109123
@property
110124
def failures(self) -> list[dict[str, str]]:
111125
"""The checks that failed in this scope."""
112126
return [c for c in self.checks if c["status"] == "fail"]
113127

114128

129+
def overall_status(results: list[ScopeResult]) -> str:
130+
"""Reduce scope statuses to one of ``pass``/``fail``/``skip``.
131+
132+
One function, used by every completion path, because the alternative
133+
is what this replaced: four separate ``all(... == "pass")`` tests, each
134+
correct only while exactly two statuses existed. The moment ``skip``
135+
appeared they all silently reclassified a skipped run as a failure.
136+
137+
``skip`` requires at least one scope and all of them skipped.
138+
"""
139+
if any(scope.status == "fail" for scope in results):
140+
return "fail"
141+
if results and all(scope.status == "skip" for scope in results):
142+
return "skip"
143+
return "pass"
144+
145+
146+
def exit_code_for(results: list[ScopeResult]) -> int:
147+
"""Only a failure is an error.
148+
149+
A skipped run validated nothing, but it violated no policy either, so
150+
it must not fail the workflow.
151+
"""
152+
return 1 if overall_status(results) == "fail" else 0
153+
154+
115155
def log_env_vars():
116156
"""Logs the environment variables for debugging purposes.
117157
@@ -326,7 +366,7 @@ def run_commit_check() -> tuple[int, list[ScopeResult]]:
326366
args = [a for a in args if a != "--message"]
327367
results.extend(run_other_checks(args))
328368

329-
exit_code = 1 if any(scope.status == "fail" for scope in results) else 0
369+
exit_code = exit_code_for(results)
330370
return exit_code, results
331371

332372

@@ -384,6 +424,11 @@ def _render_scopes(scopes: list[ScopeResult], include_docs: bool) -> list[str]:
384424
"""
385425
lines: list[str] = []
386426
for scope in scopes:
427+
if scope.status == "skip":
428+
# Deliberately not a ✔. Nothing was validated here, and a tick
429+
# claiming otherwise is what made a bypassed policy look enforced.
430+
lines.append(f" ⊘ {scope.label} (skipped)")
431+
continue
387432
if scope.status == "pass":
388433
value = _scope_value(scope)
389434
lines.append(f" ✔ {scope.label}{f' ({value})' if value else ''}")
@@ -467,7 +512,16 @@ def render_step_log(results: list[ScopeResult]) -> None:
467512
)
468513

469514
if not annotations:
470-
print("\u2714 commit-check: all checks passed")
515+
skipped, total = _skip_count(results), len(results)
516+
if total and skipped == total:
517+
print("\u2298 commit-check: all checks skipped, nothing was validated")
518+
elif skipped:
519+
print(
520+
f"\u2714 commit-check: {total - skipped} of {total} checks passed, "
521+
f"{skipped} skipped"
522+
)
523+
else:
524+
print("\u2714 commit-check: all checks passed")
471525

472526

473527
def _check_counts(results: list[ScopeResult]) -> tuple[int, int]:
@@ -495,6 +549,15 @@ def _failure_count(results: list[ScopeResult]) -> int:
495549
return _check_counts(results)[0]
496550

497551

552+
def _skip_count(results: list[ScopeResult]) -> int:
553+
"""Number of scopes that never ran.
554+
555+
Reported separately from the pass count so the headline cannot claim
556+
that checks passed when they were skipped.
557+
"""
558+
return sum(1 for scope in results if scope.status == "skip")
559+
560+
498561
def _markdown_table(results: list[ScopeResult]) -> str:
499562
"""Render the failure table shared by summary and PR comment.
500563
@@ -507,7 +570,10 @@ def _markdown_table(results: list[ScopeResult]) -> str:
507570
"|---|---|---|",
508571
]
509572
for scope in results:
510-
if scope.status == "pass":
573+
# Only failures belong in this table. A skipped scope has no failed
574+
# checks and no checked value, so it contributed an entirely blank
575+
# row \u2014 an empty accusation in a table headed "Failed checks".
576+
if scope.status != "fail":
511577
continue
512578
value = _scope_value(scope)
513579
value_display = f"`{value}`" if value else "\u2014"
@@ -592,6 +658,25 @@ def _scope_value(scope: ScopeResult, max_len: int = 60) -> str:
592658
#
593659
# _commit-check 2.13.1 · [Rules reference](https://commit-check.com/rules/)_
594660
#
661+
# Skipped (every rule declined to run — e.g. the author is in ignore_authors):
662+
#
663+
# ⊘ **All 5 checks skipped** — nothing was validated
664+
#
665+
# ```text
666+
# Commit message
667+
# ⊘ PR title (skipped)
668+
# ⊘ Commit 1/1 (skipped)
669+
# Branch
670+
# ⊘ Branch (skipped)
671+
# Author
672+
# ⊘ Author name (skipped)
673+
# ⊘ Author email (skipped)
674+
# ```
675+
#
676+
# A skipped scope deliberately carries no ✔ and no checked value: nothing was
677+
# examined, so there is no value to report and no pass to claim. When only some
678+
# scopes skip, the verdict reads "✅ **3 of 5 checks passed**, 2 skipped".
679+
#
595680
# Failure:
596681
#
597682
# <!-- commit-check-action -->
@@ -666,15 +751,26 @@ def render_report(results: list[ScopeResult]) -> str:
666751
failure table (failures only) and the collapsible per-scope details.
667752
"""
668753
failed, total = _check_counts(results)
754+
skipped = _skip_count(results)
669755
unit = "check" if total == 1 else "checks"
670756

671757
lines = [COMMENT_MARKER, REPORT_TITLE, ""]
672-
if failed == 0:
673-
lines.append(f"✅ **All {total} {unit} passed**")
674-
lines.append("")
675-
else:
758+
if failed:
676759
lines.append(f"❌ **{failed} of {total} {unit} failed**")
677760
lines.extend(["", _markdown_table(results), ""])
761+
elif total and skipped == total:
762+
# Nothing ran, so there is no success to announce. Saying "all
763+
# checks passed" here is the defect this branch exists to prevent.
764+
lines.append(f"⊘ **All {total} {unit} skipped** — nothing was validated")
765+
lines.append("")
766+
elif skipped:
767+
lines.append(
768+
f"✅ **{total - skipped} of {total} {unit} passed**, {skipped} skipped"
769+
)
770+
lines.append("")
771+
else:
772+
lines.append(f"✅ **All {total} {unit} passed**")
773+
lines.append("")
678774
lines.extend([_markdown_details(results), "", _report_footer()])
679775
return "\n".join(lines)
680776

@@ -702,7 +798,7 @@ def add_job_summary(results: list[ScopeResult]) -> int:
702798
with open(GITHUB_STEP_SUMMARY, "a", encoding="utf-8") as summary_file:
703799
summary_file.write(render_job_summary(results))
704800

705-
return 0 if all(scope.status == "pass" for scope in results) else 1
801+
return exit_code_for(results)
706802

707803

708804
def set_result_output(results: list[ScopeResult]) -> None:
@@ -714,7 +810,7 @@ def set_result_output(results: list[ScopeResult]) -> None:
714810
if not output_path:
715811
return
716812
payload = {
717-
"status": "pass" if all(s.status == "pass" for s in results) else "fail",
813+
"status": overall_status(results),
718814
"scopes": [
719815
{"label": scope.label, "status": scope.status, "checks": scope.checks}
720816
for scope in results
@@ -872,7 +968,7 @@ def add_pr_comments(results: list[ScopeResult]) -> int:
872968
if target is not None:
873969
if target.body == pr_comment_body:
874970
print(f"PR comment already up-to-date for PR #{pr_number}.")
875-
return 0 if all(scope.status == "pass" for scope in results) else 1
971+
return exit_code_for(results)
876972
print(f"Updating the last comment on PR #{pr_number}.")
877973
target.edit(pr_comment_body)
878974
for comment in stale:
@@ -882,7 +978,7 @@ def add_pr_comments(results: list[ScopeResult]) -> int:
882978
print(f"Creating a new comment on PR #{pr_number}.")
883979
pull_request.create_comment(body=pr_comment_body)
884980

885-
return 0 if all(scope.status == "pass" for scope in results) else 1
981+
return exit_code_for(results)
886982
except GithubException as e:
887983
if e.status == 403:
888984
# GithubException.data is whatever the response decoded to, which

0 commit comments

Comments
 (0)