Skip to content

Commit 3bbddff

Browse files
committed
ci: post Unit Tests push runs to Discussions (#2030)
- coverage-comment.yml: a `discussion` job takes Unit Tests runs whose event is `push` and posts them as `--mode push` to `unit-tests: <branch> push runs`. Its report comes from the run's check suite in one GraphQL call: run link, commit, conclusion, the jobs that did not pass and the annotations of those that did. A green run with a failed non-blocking job or a warning annotation reports `warning`. - `EXPECTED` lists the annotations main's KTLO legs write by design (the 3.7 and pypy3.10 install failures), by job, level and title prefix; they neither count nor appear. Any other annotation still counts. - It alone holds `discussions: write`, beside `checks: read` and `contents: read` for a sparse checkout of the default branch's script. Top-level permissions are now `{}`; `comment` keeps its own scopes. - A push relay's concurrency group is its own run's, so a newer relay cannot cancel a pending one; a pull request's group is unchanged. - tests/project/test_unit_tests_push_reports_2030.py: the job layout, scopes, events and checkout, the conclusion-to-status mapping, the by-design list and run 39718608950's annotations, the report against a stand-in gh, and both steps end to end. - workflows.rst: CI Reports covers the relay.
1 parent 9c8a110 commit 3bbddff

3 files changed

Lines changed: 711 additions & 6 deletions

File tree

‎.github/workflows/coverage-comment.yml‎

Lines changed: 195 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -15,27 +15,45 @@ name: Coverage Comment
1515
# head sha), not from the artifact, and accepts it only if it is all digits;
1616
# * renders its own Markdown from the artifact's numbers, accepting only the
1717
# shapes the `test` job writes, rather than posting text from the run.
18+
#
19+
# A push run has no pull request to comment on, so the `discussion` job posts it
20+
# to its branch's thread in the Discussions category REPORT_CATEGORY names
21+
# (#2030). It reads the run's jobs through the API, not the run's artifacts,
22+
# runs util/discussion_report.py from the default branch's checkout, never the
23+
# run's head, and alone holds `discussions: write`.
1824
on:
1925
workflow_run:
2026
workflows: ["Unit Tests"]
2127
types: [completed]
2228

23-
permissions:
24-
actions: read
25-
pull-requests: write
29+
# Each job holds its own scopes (#2030).
30+
permissions: {}
2631

2732
# One comment per PR, so serialise per head branch. A run that finishes after a
28-
# newer push skips itself below, since its sha is no longer the PR's head.
33+
# newer push skips itself below, since its sha is no longer the PR's head. A
34+
# push run's group is its own (#2030): a group cancels a pending run when a
35+
# newer one joins it, which would drop that run's report.
2936
concurrency:
30-
group: coverage-comment-${{ github.event.workflow_run.head_repository.full_name }}-${{ github.event.workflow_run.head_branch }}
37+
group: >-
38+
coverage-comment-${{ github.event.workflow_run.event == 'pull_request' &&
39+
format('{0}-{1}', github.event.workflow_run.head_repository.full_name, github.event.workflow_run.head_branch) ||
40+
github.event.workflow_run.id }}
3141
cancel-in-progress: false
3242

43+
# #2030: one Discussions category holds every report with no pull request to
44+
# comment on, one thread per job.
45+
env:
46+
REPORT_CATEGORY: CI reports
47+
3348
jobs:
3449
comment:
3550
name: Coverage comment
3651
if: ${{ github.event.workflow_run.event == 'pull_request' && github.event.workflow_run.conclusion != 'cancelled' }}
3752
runs-on: ubuntu-latest
3853
timeout-minutes: 5
54+
permissions:
55+
actions: read
56+
pull-requests: write
3957

4058
steps:
4159
- name: Post or update the coverage comment
@@ -112,3 +130,175 @@ jobs:
112130
else
113131
gh api -X POST "repos/$REPO/issues/$pr/comments" -F body=@"$RUNNER_TEMP/body.md" > /dev/null
114132
fi
133+
134+
# #2030: a Unit Tests run on a push, posted as a `push` report to the thread
135+
# `unit-tests: <branch> push runs`: a run that failed or warned posts, then one
136+
# green run after it. The report is built from the run's check suite, read in
137+
# one GraphQL call rather than from an artifact, and quotes each job name and
138+
# annotation as a code span, so none renders or mentions. A green run reports
139+
# `warning` where a job failed without failing it, or a job that passed holds
140+
# a warning or failure annotation, as a failing KTLO leg does (#1672), other
141+
# than those `EXPECTED` lists as written on every run by design. The job and
142+
# its steps are `continue-on-error`; util/discussion_report.py warns instead.
143+
discussion:
144+
name: Post the push run report
145+
if: ${{ github.event.workflow_run.event == 'push' }}
146+
runs-on: ubuntu-latest
147+
timeout-minutes: 10
148+
continue-on-error: true
149+
permissions:
150+
checks: read
151+
contents: read
152+
discussions: write
153+
154+
steps:
155+
# The default branch, which is what `workflow_run` checks out, and only
156+
# the script; nothing from the run's head.
157+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
158+
with:
159+
sparse-checkout: |
160+
util/discussion_report.py
161+
sparse-checkout-cone-mode: false
162+
persist-credentials: false
163+
164+
- name: Write the run report
165+
id: report
166+
continue-on-error: true
167+
env:
168+
GH_TOKEN: ${{ github.token }}
169+
SUITE: ${{ github.event.workflow_run.check_suite_node_id }}
170+
RUN_URL: ${{ github.event.workflow_run.html_url }}
171+
RUN_CONCLUSION: ${{ github.event.workflow_run.conclusion }}
172+
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
173+
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
174+
run: |
175+
set -u
176+
# The run's conclusion as a --status; anything else, `timed_out` and
177+
# `startup_failure` among them, is a failure.
178+
case "$RUN_CONCLUSION" in
179+
success) status=success ;;
180+
cancelled|stale) status=cancelled ;;
181+
skipped|neutral) status=skipped ;;
182+
*) status=failure ;;
183+
esac
184+
branch=$(printf '%s' "$HEAD_BRANCH" | tr -c 'A-Za-z0-9._/-' '-')
185+
echo "title=unit-tests: ${branch:-unknown} push runs" >> "$GITHUB_OUTPUT"
186+
187+
# shellcheck disable=SC2016 # `$id` is a GraphQL variable
188+
query='query($id: ID!) { node(id: $id) { ... on CheckSuite { checkRuns(first: 100) {
189+
pageInfo { hasNextPage }
190+
nodes { name conclusion annotations(first: 20) { nodes { annotationLevel title message } } } } } } }'
191+
if ! gh api graphql -f query="$query" -f id="$SUITE" > "$RUNNER_TEMP/jobs.json"; then
192+
echo "::warning::The run's jobs could not be read, so the report names none of them."
193+
: > "$RUNNER_TEMP/jobs.json"
194+
fi
195+
196+
mkdir -p report
197+
if ! STATUS="$status" python3 - "$RUNNER_TEMP/jobs.json" report/unit-tests-report.md <<'PY'
198+
import json, os, re, sys
199+
status, conclusion = os.environ['STATUS'], os.environ['RUN_CONCLUSION']
200+
sha, url = os.environ['HEAD_SHA'], os.environ['RUN_URL']
201+
202+
def code(text):
203+
"""One line of ``text``, at most 200 characters, as a code span, so it renders and mentions nothing."""
204+
text = ' '.join(str(text).replace('`', "'").split())
205+
return '`{}`'.format(text if len(text) <= 200 else text[:197] + '...')
206+
207+
# The annotations a job writes on every run by design, which neither
208+
# count nor are listed: per job, `(level, title prefix)`, where an
209+
# empty prefix matches only an empty title, the runner's own line for
210+
# the step that failed. Matched on the title, not the message, which
211+
# #2041 rewords. Anything else in these jobs, a failed tests step
212+
# among it, still makes a green run `warning`.
213+
EXPECTED = {
214+
# #1633, #1672: no pytest>=8 for 3.7, so its KTLO legs stop at install.
215+
'Python 3.7 (KTLO)': (
216+
('WARNING', 'install failed on Python 3.7 (KTLO, non-blocking)'),
217+
('FAILURE', ''),
218+
('WARNING', 'pcapkit did not install on Python 3.7 (KTLO, non-blocking)'),
219+
),
220+
'Integration Python 3.7 (KTLO)': (
221+
('WARNING', 'install failed on Python 3.7 (KTLO, non-blocking)'),
222+
('FAILURE', ''),
223+
('WARNING', 'pcapkit did not install on Python 3.7 (KTLO, non-blocking)'),
224+
),
225+
# #1633, #1672: pypy3.10's KTLO leg stops at install too, until #2040.
226+
'Python pypy3.10 (KTLO)': (
227+
('WARNING', 'install failed on Python pypy3.10 (KTLO, non-blocking)'),
228+
('FAILURE', ''),
229+
('WARNING', 'pcapkit did not install on Python pypy3.10 (KTLO, non-blocking)'),
230+
),
231+
}
232+
233+
def expected(job, note):
234+
"""Whether ``note``, an annotation of ``job``, is one :data:`EXPECTED` lists."""
235+
title = note.get('title') or ''
236+
return any(note.get('annotationLevel') == level and (title.startswith(prefix) if prefix else not title)
237+
for level, prefix in EXPECTED.get(job, ()))
238+
239+
try:
240+
with open(sys.argv[1], encoding='utf-8') as file:
241+
runs = json.load(file)['data']['node']['checkRuns']
242+
nodes, more = [node for node in runs['nodes'] if node], runs['pageInfo']['hasNextPage']
243+
except (OSError, ValueError, LookupError, TypeError):
244+
nodes, more = None, False
245+
failed, warned, by_design = [], [], False
246+
for node in nodes or []:
247+
name, result = node.get('name') or '?', (node.get('conclusion') or 'unknown').lower()
248+
if result not in ('success', 'skipped', 'neutral'):
249+
failed.append(f'{code(name)} ({result})')
250+
elif result == 'success':
251+
flagged = [note for note in (node.get('annotations') or {}).get('nodes') or []
252+
if note and note.get('annotationLevel') in ('WARNING', 'FAILURE')]
253+
kept = [note for note in flagged if not expected(name, note)]
254+
by_design = by_design or len(kept) < len(flagged)
255+
notes = [note.get('title') or note.get('message') or '' for note in kept]
256+
notes = list(dict.fromkeys(note for note in notes if note.strip()))[:5]
257+
if notes:
258+
warned.append(f'{code(name)}: ' + '; '.join(map(code, notes)))
259+
if status == 'success' and (failed or warned):
260+
status = 'warning'
261+
262+
run = 'The Unit Tests run'
263+
if re.fullmatch(r'https://[\w.-]+/[\w.-]+/[\w.-]+/actions/runs/\d+', url):
264+
run = f'[Unit Tests run]({url})'
265+
commit = f' for `{sha[:9]}`' if re.fullmatch(r'[0-9a-f]{40}', sha) else ''
266+
lines = ['### Unit Tests push run', '', f'{run}{commit} concluded {code(conclusion or "unknown")}.']
267+
if failed:
268+
heading = 'Failed without failing the run' if conclusion == 'success' else 'Did not pass'
269+
lines += ['', f'**{heading}:**', '', *(f'- {item}' for item in failed)]
270+
if warned:
271+
lines += ['', '**Warned, in jobs that passed:**', '', *(f'- {item}' for item in warned)]
272+
if nodes is None:
273+
lines += ['', "The run's jobs could not be read, so none is named here."]
274+
elif more:
275+
lines += ['', 'Only the first 100 jobs were read.']
276+
elif not failed and not warned:
277+
lines += ['', 'Every job passed, with no warning but the by-design ones.' if by_design
278+
else 'Every job passed, with no warning.']
279+
with open(sys.argv[2], 'w', encoding='utf-8') as file:
280+
print('\n'.join(lines), file=file)
281+
with open(os.environ['GITHUB_OUTPUT'], 'a', encoding='utf-8') as file:
282+
print(f'status={status}', file=file)
283+
PY
284+
then
285+
echo "::warning::The report could not be written, so it holds the run's conclusion alone."
286+
echo "### Unit Tests push run, which concluded \`$status\`" > report/unit-tests-report.md
287+
echo "status=$status" >> "$GITHUB_OUTPUT"
288+
fi
289+
cat report/unit-tests-report.md
290+
291+
# The script labels the run it links with GITHUB_WORKFLOW, which names
292+
# this workflow, so the command sets the relayed run's for that process.
293+
- name: Post the report to Discussions
294+
continue-on-error: true
295+
env:
296+
GH_TOKEN: ${{ github.token }}
297+
TITLE: ${{ steps.report.outputs.title }}
298+
STATUS: ${{ steps.report.outputs.status }}
299+
RUN_URL: ${{ github.event.workflow_run.html_url }}
300+
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
301+
run: |
302+
GITHUB_WORKFLOW='Unit Tests' python3 util/discussion_report.py --category "$REPORT_CATEGORY" \
303+
--title "$TITLE" --report report/unit-tests-report.md \
304+
--mode push --status "$STATUS" --run-url "$RUN_URL" --sha "$HEAD_SHA"

‎docs/source/contributing/workflows.rst‎

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -528,7 +528,8 @@ treats the artifact as untrusted: it never checks out pull-request code. It
528528
takes the PR number from the API, as the open PR whose head is the run's head
529529
sha, and renders its own Markdown from validated numbers. It edits one comment,
530530
found by a hidden marker, on each push. The first line says when the tests
531-
failed. Neither workflow's coverage step is a required check.
531+
failed. Neither workflow's coverage step is a required check. A push run has no
532+
pull request, so its report goes to Discussions instead (see `CI Reports`_).
532533

533534
CI Timing Review (`#1538 <https://github.com/JarryShaw/PyPCAPKit/issues/1538>`__)
534535
---------------------------------------------------------------------------------
@@ -1274,6 +1275,36 @@ failed API call is one warning, and the job is ``continue-on-error``, since a
12741275
failed Vendor Update run is one Create Release reads as not updated. A CI timing
12751276
run failed by a HARD threshold is posted all the same, on ``always()``.
12761277

1278+
Unit Tests push runs post as ``push`` to ``unit-tests: <branch> push runs``, so
1279+
``main``'s thread is ``unit-tests: main push runs``. ``unit-tests.yml`` is a
1280+
reusable workflow whose callers grant ``contents: read``, so the post comes from
1281+
``coverage-comment.yml``'s ``workflow_run`` relay instead: its ``discussion`` job
1282+
takes a run whose event is ``push``, and ``comment`` keeps a pull request's. Each
1283+
job holds its own scopes, ``discussion`` alone ``discussions: write``, with
1284+
``contents: read`` for a checkout of the default branch's script, never the run's
1285+
head, and ``checks: read`` for the run's check suite. Its ``Write the run report``
1286+
step reads that suite in a single GraphQL call rather than any artifact, and
1287+
renders every name and annotation it quotes as a code span. The report links the
1288+
run, names its commit and conclusion, the jobs that did not pass, and the warning
1289+
and failure annotations of the jobs that did. The conclusion becomes the
1290+
``--status``: ``success``, ``cancelled`` (and ``stale``), ``skipped`` (and
1291+
``neutral``), and ``failure`` for anything else, ``timed_out`` and
1292+
``startup_failure`` among them. A green run reports ``warning`` where a job failed
1293+
without failing it, or a job that passed holds a warning or failure annotation, as
1294+
a KTLO leg whose tests fail does (`#1672
1295+
<https://github.com/JarryShaw/PyPCAPKit/issues/1672>`__). The exception is the
1296+
step's ``EXPECTED`` list: the annotations a job writes on every run by design,
1297+
today the install failures of the KTLO legs that cannot install, which neither
1298+
count nor appear in the report. Each entry is a job, a level and a title prefix,
1299+
an empty prefix matching only an empty title, so a reworded message still
1300+
matches, and anything else those jobs write still counts.
1301+
:file:`tests/project/test_unit_tests_push_reports_2030.py` pins the list, so a
1302+
leg that starts to install, or a new by-design annotation, changes it and the
1303+
pin together. The script labels the
1304+
run it links with ``GITHUB_WORKFLOW``, so the post step sets ``Unit Tests`` for
1305+
that one command. A push relay's concurrency group is its own run's, since a
1306+
group cancels a pending run when a newer one joins it.
1307+
12771308
.. _Action Pins:
12781309

12791310
Action Pins (`#1968 <https://github.com/JarryShaw/PyPCAPKit/issues/1968>`__)

0 commit comments

Comments
 (0)