Skip to content

Commit 0fcb2ee

Browse files
authored
devops(bots): publish the agent's GitHub writes as typed calls (#3310)
1 parent 637cb1b commit 0fcb2ee

3 files changed

Lines changed: 93 additions & 98 deletions

File tree

‎.claude/skills/playwright-triage/SKILL.md‎

Lines changed: 26 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -69,18 +69,32 @@ file, and delete the file when you are done, so that the checkout is clean.
6969
language). The gap is in the docs, so it is an upstream change too. Do not edit a generated file,
7070
and do not add to `scripts/codegen/python_rules.js` to go around the docs.
7171

72-
Create `$OUTPUT_DIR/upstream/` with:
73-
74-
- `report.md`: the report for an engineer of `microsoft/playwright` who did not read the Python
75-
issue: what fails, expected versus actual, the versions (the Python package, the commit in
76-
`DRIVER_VERSION`, the browser, the platform), the root cause when you found it, and the condensed
77-
Node test as the repro.
78-
- `report.json`: `{"title": "..."}`, at most 256 characters, for example
79-
`route.fulfill() drops the charset of Content-Type`.
80-
81-
The comment job appends the title and the body to your comment, for a maintainer to file in
82-
`microsoft/playwright`. The bot never files it. Write `report.json` last, when `report.md` and the
83-
comment are done. Like `proposal.json`, it marks the report as ready.
72+
The report is for an engineer of `microsoft/playwright` who did not read the Python issue: what
73+
fails, expected versus actual, the versions (the Python package, the commit in `DRIVER_VERSION`, the
74+
browser, the platform), the root cause when you found it, and the condensed Node test as the repro.
75+
The title has at most 256 characters, for example `route.fulfill() drops the charset of
76+
Content-Type`.
77+
78+
The bot never files it in `microsoft/playwright`: this run allows no `issue` call (the calls are
79+
in `$AGENT_GUIDES_DIR/calls.d.ts`). Put the report at the end of your comment, for a maintainer to
80+
file, in this form:
81+
82+
`````markdown
83+
Upstream bug report, for a maintainer to file in microsoft/playwright:
84+
85+
**Title:** <title>
86+
87+
<details>
88+
<summary>Body</summary>
89+
90+
````markdown
91+
<the report>
92+
````
93+
94+
</details>
95+
`````
96+
97+
Write the rest of the comment as the note: the verdict, the evidence, and that the bug is upstream.
8498

8599
## PRs from third parties
86100

‎.github/workflows/revision.yml‎

Lines changed: 14 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
# Copilot addresses the maintainers' review comments on a draft PR that the triage workflow opened,
2-
# without write credentials. A fresh job pushes the new commit to the PR's branch using the app, so
3-
# CI runs on it. The steps that are the same for every Playwright repo are actions in
4-
# microsoft/playwright, and the shared guide they carry says how to revise; the prompt below has
5-
# what is specific to this repo.
2+
# without write credentials. The agent leaves the calls to make on GitHub in calls.json. A fresh job
3+
# checks them and makes them using the app: it pushes the new commit to the PR's branch, so CI runs
4+
# on it, and replies in the review threads. The steps that are the same for every Playwright repo
5+
# are actions in microsoft/playwright, and the shared guide they carry says how to revise; the
6+
# prompt below has what is specific to this repo.
67
name: PR revision
78

89
on:
@@ -24,6 +25,7 @@ jobs:
2425
github.event.pull_request.head.repo.full_name == github.repository
2526
runs-on: ubuntu-latest
2627
permissions:
28+
issues: read
2729
pull-requests: write
2830
env:
2931
PLATFORM: ${{ fromJSON('{"needs-revision":"linux","needs-revision-mac":"mac","needs-revision-win":"windows"}')[github.event.label.name] }}
@@ -97,14 +99,14 @@ jobs:
9799
98100
Pull request: https://github.com/microsoft/playwright-python/pull/${{ github.event.pull_request.number }}
99101
output-dir: ${{ env.OUTPUT_DIR }}
100-
result-file: commit/commit.json
102+
result-file: calls.json
101103
artifact-name: revision
102104
exec: ${{ env.CI_ENV_EXEC }}
103105

104106
push:
105107
name: Push the revision
106108
needs: [prepare, revise]
107-
# commit.json is written last, so a finished revision is pushed even when the job failed later.
109+
# calls.json is written when the revision is done, so it is pushed even if the job failed later.
108110
if: ${{ !cancelled() && needs.revise.outputs.has_revision == 'true' }}
109111
runs-on: ubuntu-latest
110112
steps:
@@ -113,6 +115,8 @@ jobs:
113115
with:
114116
app-id: ${{ vars.PLAYWRIGHT_APP_ID }}
115117
private-key: ${{ secrets.PLAYWRIGHT_PRIVATE_KEY }}
118+
permission-contents: write
119+
permission-pull-requests: write
116120
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
117121
with:
118122
ref: ${{ needs.prepare.outputs.head }}
@@ -121,12 +125,9 @@ jobs:
121125
with:
122126
name: revision
123127
path: ${{ runner.temp }}/revision
124-
# Never execute code or scripts from the proposed changes in the app job.
125-
- name: Validate and push
126-
uses: microsoft/playwright/.github/actions/publish-patch@main
128+
# Never execute code or scripts from the artifact in the app job. The calls are data.
129+
- name: Check and make the calls
130+
uses: microsoft/playwright/.github/actions/publish-calls@main
127131
with:
128132
token: ${{ steps.app-token.outputs.token }}
129-
dir: ${{ runner.temp }}/revision/commit
130-
patch: revision.patch
131-
metadata: commit.json
132-
branch: ${{ needs.prepare.outputs.branch }}
133+
dir: ${{ runner.temp }}/revision

‎.github/workflows/triage.yml‎

Lines changed: 53 additions & 73 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
# Copilot triages and fixes an issue, or works on a task given by hand, without write credentials.
2-
# Fresh jobs post the run link on the issue, open the completed fix as a draft PR using the app, so
3-
# CI runs on it, and comment on the issue. The steps that are the same for every Playwright repo are
4-
# actions in microsoft/playwright. The skill in .claude/skills/playwright-triage/ has what is
5-
# specific to this repo.
2+
# The agent leaves the calls to make on GitHub in calls.json. A fresh job posts the run link on the
3+
# issue first. After the agent, a fresh job checks the calls and makes them using the app: it opens
4+
# the fix as a draft PR, so CI runs on it, and comments on the issue. The steps that are the same
5+
# for every Playwright repo are actions in microsoft/playwright. The skill in
6+
# .claude/skills/playwright-triage/ has what is specific to this repo.
67
name: Issue triage
78

89
on:
@@ -86,7 +87,6 @@ jobs:
8687
base: ${{ steps.base.outputs.sha }}
8788
setup: ${{ steps.setup.outcome }}
8889
agent: ${{ steps.agent.outputs.outcome }}
89-
has_fix: ${{ steps.agent.outputs.has-result }}
9090
steps:
9191
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
9292
with:
@@ -130,98 +130,78 @@ jobs:
130130
with:
131131
prompt: ${{ steps.prompt.outputs.prompt }}
132132
output-dir: ${{ env.OUTPUT_DIR }}
133-
result-file: fix/proposal.json
133+
result-file: calls.json
134134
artifact-name: triage
135135
github-tools: true
136136
exec: ${{ env.CI_ENV_EXEC }}
137-
- name: Check the upstream bug report
138-
run: |
139-
if [[ -f "$OUTPUT_DIR/upstream/report.json" ]]; then
140-
title=$(node -p 'String(JSON.parse(require("fs").readFileSync(process.argv[1], "utf8")).title ?? "")' "$OUTPUT_DIR/upstream/report.json" || true)
141-
if [[ -z "$title" ]]; then
142-
echo "::warning::upstream/report.json has no title, so the report is not appended to the comment"
143-
else
144-
echo "Upstream bug report: $title" >> "$GITHUB_STEP_SUMMARY"
145-
fi
146-
fi
147137

148138
publish:
149-
name: Open draft pull request
150-
needs: triage
151-
# proposal.json is written last, so a finished proposal is published even when the job failed later.
152-
if: ${{ !cancelled() && needs.triage.outputs.has_fix == 'true' }}
139+
name: Publish the results
140+
needs: [start, triage]
141+
# Runs after every triage run, also a failed one: without a comment from the agent, the comment
142+
# says why the bot could not finish.
143+
if: ${{ !cancelled() && needs.triage.result != 'skipped' }}
153144
runs-on: ubuntu-latest
154-
outputs:
155-
pr: ${{ steps.publish.outputs.pull-request }}
156145
steps:
157146
- uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0
158147
id: app-token
159148
with:
160149
app-id: ${{ vars.PLAYWRIGHT_APP_ID }}
161150
private-key: ${{ secrets.PLAYWRIGHT_PRIVATE_KEY }}
151+
permission-contents: write
152+
permission-issues: write
153+
permission-pull-requests: write
162154
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
163155
with:
164156
ref: ${{ needs.triage.outputs.base }}
165157
token: ${{ steps.app-token.outputs.token }}
166-
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
167-
with:
168-
name: triage
169-
path: ${{ runner.temp }}/triage
170-
# Never execute code or scripts from the proposed changes in the app job.
171-
- name: Validate, push and open draft
172-
id: publish
173-
uses: microsoft/playwright/.github/actions/publish-patch@main
174-
with:
175-
token: ${{ steps.app-token.outputs.token }}
176-
dir: ${{ runner.temp }}/triage/fix
177-
patch: fix.patch
178-
metadata: proposal.json
179-
body-file: pr-body.md
180-
branch: triage/${{ github.event.issue.number || inputs.issue || 'task' }}/${{ github.run_id }}-${{ github.run_attempt }}
181-
commit-body: "${{ (github.event.issue.number || inputs.issue) && format('Fixes: https://github.com/{0}/issues/{1}', github.repository, github.event.issue.number || inputs.issue) || '' }}"
182-
pull-request: true
183-
184-
comment:
185-
name: Comment on the issue
186-
needs: [start, triage, publish]
187-
# Runs after every triage run that has an issue, including failed ones, and whatever publish did.
188-
if: ${{ !cancelled() && needs.triage.result != 'skipped' && (github.event_name == 'issues' || inputs.issue != '') }}
189-
runs-on: ubuntu-latest
190-
permissions:
191-
issues: write
192-
steps:
193158
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
194159
# A failed triage run may not have uploaded anything.
195160
continue-on-error: true
196161
with:
197162
name: triage
198163
path: ${{ runner.temp }}/triage
199-
# The bot never files upstream; the comment hands the report to a maintainer.
200-
- name: Describe the upstream bug report
201-
id: upstream
164+
# Without a comment from the agent on the issue, for example after a failed run, the issue gets
165+
# one that says that the bot could not finish.
166+
- name: Add a comment when the agent left none
167+
if: github.event.issue.number || inputs.issue
202168
env:
203-
OUTPUT_DIR: ${{ runner.temp }}/triage
169+
CALLS: ${{ runner.temp }}/triage/calls.json
170+
ISSUE: ${{ github.event.issue.number || inputs.issue }}
171+
SETUP: ${{ needs.triage.outputs.setup }}
172+
AGENT: ${{ needs.triage.outputs.agent }}
173+
FOOTER: Triaged on ${{ fromJSON('{"linux":"Linux","mac":"macOS","windows":"Windows","needs-triage":"Linux","needs-triage-mac":"macOS","needs-triage-win":"Windows"}')[inputs.platform || github.event.label.name] }} by the Playwright bot
174+
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
204175
run: |
205-
extra=""
206-
if [[ -s "$OUTPUT_DIR/upstream/report.md" && -f "$OUTPUT_DIR/upstream/report.json" ]]; then
207-
title=$(jq -r '.title // empty' "$OUTPUT_DIR/upstream/report.json")
208-
if [[ -n "$title" ]]; then
209-
extra=$(printf 'Upstream bug report, for a maintainer to file in microsoft/playwright:\n\n**Title:** %s\n\n<details>\n<summary>Body</summary>\n\n````markdown\n%s\n````\n\n</details>' "$title" "$(cat "$OUTPUT_DIR/upstream/report.md")")
210-
fi
176+
mkdir -p "$(dirname "$CALLS")"
177+
if jq -e --argjson issue "$ISSUE" 'any(.[]; .call == "comment" and .issue == $issue)' "$CALLS" > /dev/null 2>&1; then
178+
exit 0
211179
fi
212-
delimiter="EXTRA_$RANDOM$RANDOM"
213-
{ echo "extra<<$delimiter"; echo "$extra"; echo "$delimiter"; } >> "$GITHUB_OUTPUT"
214-
- name: Post the triage comment
215-
uses: microsoft/playwright/.github/actions/triage-comment@main
180+
if [[ "$SETUP" == "failure" ]]; then
181+
reason="setting up the runner failed"
182+
elif [[ "$AGENT" == "failure" ]]; then
183+
reason="the Copilot session failed or ran out of budget"
184+
else
185+
reason="the session ended without a comment"
186+
fi
187+
body=$(printf "Hi, I'm the Playwright bot. I couldn't finish triaging this issue: %s. The agent run has the details.\n\n<sub>%s - [agent run](%s)</sub>\n<!-- playwright-ai-triage -->" "$reason" "$FOOTER" "$RUN_URL")
188+
comment=$(jq -n --arg repo "$GITHUB_REPOSITORY" --argjson issue "$ISSUE" --arg body "$body" '{call: "comment", repo: $repo, issue: $issue, body: $body}')
189+
# A calls.json that is not an array is replaced: its calls would be rejected anyway.
190+
jq --argjson comment "$comment" '. + [$comment]' "$CALLS" > "$CALLS.tmp" 2> /dev/null \
191+
|| jq -n --argjson comment "$comment" '[$comment]' > "$CALLS.tmp"
192+
mv "$CALLS.tmp" "$CALLS"
193+
# Never execute code or scripts from the artifact in the app job. The calls are data.
194+
- name: Check and make the calls
195+
id: publish
196+
uses: microsoft/playwright/.github/actions/publish-calls@main
216197
with:
217-
issue: ${{ github.event.issue.number || inputs.issue }}
198+
token: ${{ steps.app-token.outputs.token }}
218199
dir: ${{ runner.temp }}/triage
219-
footer: Triaged on ${{ fromJSON('{"linux":"Linux","mac":"macOS","windows":"Windows","needs-triage":"Linux","needs-triage-mac":"macOS","needs-triage-win":"Windows"}')[inputs.platform || github.event.label.name] }} by the Playwright bot
220-
job-result: ${{ needs.triage.result }}
221-
setup-outcome: ${{ needs.triage.outputs.setup }}
222-
agent-outcome: ${{ needs.triage.outputs.agent }}
223-
setup-failure-reason: setting up the runner, building the project or downloading the browsers failed
224-
pull-request: ${{ needs.publish.outputs.pr }}
225-
publish-result: ${{ needs.publish.result }}
226-
extra: ${{ steps.upstream.outputs.extra }}
227-
start-comment: ${{ needs.start.outputs.comment }}
200+
- name: Hide the comment with the run link
201+
if: ${{ !cancelled() && steps.publish.outputs.comment == 'true' && needs.start.outputs.comment != '' }}
202+
env:
203+
GH_TOKEN: ${{ steps.app-token.outputs.token }}
204+
ID: ${{ needs.start.outputs.comment }}
205+
run: |
206+
gh api graphql -f query='mutation($id: ID!) { minimizeComment(input: { subjectId: $id, classifier: OUTDATED }) { clientMutationId } }' -f id="$ID" \
207+
|| echo "::warning::Could not hide the comment with the run link"

0 commit comments

Comments
 (0)