Skip to content

Commit 7aa95b5

Browse files
pavelfeldmanpavelfeldman
authored
devops(triage): add the issue triage and revision bots (#3298)
1 parent 6f6823d commit 7aa95b5

5 files changed

Lines changed: 516 additions & 0 deletions

File tree

‎.claude/skills/playwright-triage/SKILL.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
---
2+
name: playwright-triage
3+
description: Triage and fix one microsoft/playwright-python issue, or do a maintainer's task, unattended inside the Issue triage GitHub Actions workflow (.github/workflows/triage.yml), which a needs-triage, needs-triage-mac or needs-triage-win label starts, or a maintainer starts by hand with an issue, a task or both. Reproduces the issue on the runner, then leaves a verified fix proposal for the workflow to open as a draft PR, an investigation note for it to post on the issue, or an upstream bug report for a maintainer to file in microsoft/playwright. Only for that workflow.
4+
---
5+
6+
# Playwright Python Issue Triage
7+
8+
You run unattended in the [Issue triage](../../../.github/workflows/triage.yml) workflow of
9+
`microsoft/playwright-python`. First read the shared guide, `$AGENT_GUIDES_DIR/triage.md`, a copy of
10+
[microsoft/playwright/.github/actions/run-agent/guides/triage.md](https://github.com/microsoft/playwright/blob/main/.github/actions/run-agent/guides/triage.md),
11+
and then [AGENTS.md](../../../AGENTS.md), both in full. The guide says how to triage an issue in any
12+
Playwright repo. AGENTS.md says how to build, test and lint this one, and where a regression test
13+
goes. This file covers what neither has: the checkouts, how to run a published version, and the
14+
outcome that this repo adds, an upstream bug report.
15+
16+
## The checkouts
17+
18+
`$REPO_DIR` is the checkout of `microsoft/playwright-python` that the guide describes. Its requirements
19+
are installed (`pip install -r local-requirements.txt`, `pip install -r requirements.txt`,
20+
`pip install -e .`), the wheel is built (`python -m build --wheel`, which builds the driver into
21+
`playwright/driver/`), and Chromium, Firefox and WebKit are installed with
22+
`python -m playwright install --with-deps`. It is the working directory, and the checkout that the
23+
guides refer to. The clone is shallow: run `git fetch --unshallow --tags origin` in it before you
24+
bisect or use `git tag --contains`.
25+
26+
`$PLAYWRIGHT_DIR` is `$REPO_DIR/.upstream/`, a full clone of `microsoft/playwright` at the commit
27+
in `DRIVER_VERSION`, created by `python scripts/upstream.py checkout`. The driver was built from it.
28+
Read upstream code there. Do not edit, commit or switch branches in it: the wheel build and
29+
`./scripts/lint.sh` stop when it is not at that commit or has local changes. Before you run upstream
30+
tests in it, it needs `npm ci`, `npm run build` and `npx playwright install`. The wheel build
31+
(`scripts/build_driver.py`) ran the first two there; run them again when `node_modules/` is missing.
32+
33+
## Published versions
34+
35+
To run a published version, for example to bisect, create `$SCRATCH_DIR/playwright-python-<n>/<version>/`
36+
and run there:
37+
38+
```bash
39+
python -m venv env && source env/bin/activate # env\Scripts\activate on Windows
40+
pip install playwright==<version>
41+
python -m playwright install
42+
```
43+
44+
The fixtures of the repo are not available there. Write the condensed repro as a plain script with
45+
`sync_playwright()` or `async_playwright()` in that directory, with `page.set_content()` in place of
46+
the test server, and run it from there. In `$REPO_DIR`, the source tree shadows the installed package.
47+
48+
The driver of a release is in `DRIVER_VERSION` at its tag: `git show v<version>:DRIVER_VERSION`
49+
prints the upstream commit it was built from, or, for releases up to 1.64, the `playwright-core`
50+
version of the driver; `npm view playwright-core@<core version> gitHead` prints its commit. Match a
51+
regression in the driver to commits with `git -C $PLAYWRIGHT_DIR log <good>..<bad> -- <path>`.
52+
53+
## Upstream bugs
54+
55+
The client talks to the Node driver that is built from `$PLAYWRIGHT_DIR`, and `microsoft/playwright`
56+
and this repo are one team. Reproduce the bug in Python first. Then write the same calls as an
57+
upstream test and run it in `$PLAYWRIGHT_DIR`. The triage skill of `microsoft/playwright`, in
58+
`$PLAYWRIGHT_DIR/.claude/skills/playwright-triage/`, says where such a test goes, which fixtures it
59+
uses and how to run it. The rest of that skill is not for this run. Put the test in a new spec
60+
file, and delete the file when you are done, so that the checkout is clean.
61+
62+
- The Node test passes: the bug is in the Python client. Fix it here.
63+
- The Node test fails too: the root cause is upstream, and the fix lands in `microsoft/playwright`.
64+
Do not work around it in `playwright/_impl/`. Run `git -C $PLAYWRIGHT_DIR fetch origin main`.
65+
When `git -C $PLAYWRIGHT_DIR log HEAD..origin/main -- <path>` shows a fix that already landed,
66+
the note says that it reaches this repo with the next roll. Otherwise write the report.
67+
- The Python API lacks a member or an option: the generated API has everything that the upstream
68+
docs in `$PLAYWRIGHT_DIR/docs/src/api` give Python (`* langs:` decides; no `langs` means every
69+
language). The gap is in the docs, so it is an upstream change too. Do not edit a generated file,
70+
and do not add to `scripts/codegen/python_rules.js` to go around the docs.
71+
72+
Create `$OUTPUT_DIR/upstream/` with:
73+
74+
- `report.md`: the report for an engineer of `microsoft/playwright` who did not read the Python
75+
issue: what fails, expected versus actual, the versions (the Python package, the commit in
76+
`DRIVER_VERSION`, the browser, the platform), the root cause when you found it, and the condensed
77+
Node test as the repro.
78+
- `report.json`: `{"title": "..."}`, at most 256 characters, for example
79+
`route.fulfill() drops the charset of Content-Type`.
80+
81+
The comment job appends the title and the body to your comment, for a maintainer to file in
82+
`microsoft/playwright`. The bot never files it. Write `report.json` last, when `report.md` and the
83+
comment are done. Like `proposal.json`, it marks the report as ready.
84+
85+
## PRs from third parties
86+
87+
Do not run a PR that changes `setup.py`, `pyproject.toml`, `setup.cfg`, `local-requirements.txt`,
88+
`requirements.txt`, `DRIVER_VERSION`, `NODE_VERSION`, `.pre-commit-config.yaml`, `test.sh`,
89+
`scripts/`, `utils/`, `.github/`, `.azure-pipelines/`, `.claude/`, or a generated file. A PR that
90+
changes a ported test under `tests/{async,sync}/{page,library}`, or adds a mark to one, has faults:
91+
the ports have zero deviations.

‎.github/actions/setup-triage-runner/action.yml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
name: 'Set up the triage runner'
2+
description: >
3+
Prepares the environment that the triage and revision agents work in: the checkout built as CI
4+
builds it, with the browsers and the Copilot CLI installed, and the upstream checkout in .upstream/
5+
that the driver is built from. Exports PLAYWRIGHT_DIR, the path of that checkout, and CI_ENV_EXEC,
6+
the wrapper from microsoft/playwright that runs a command under xvfb on Linux and at normal I/O
7+
priority on Windows.
8+
runs:
9+
using: composite
10+
steps:
11+
- uses: microsoft/playwright/.github/actions/ci-env@main
12+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
13+
with:
14+
python-version: '3.12'
15+
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
16+
with:
17+
node-version: '24'
18+
- name: Install the requirements
19+
shell: bash
20+
run: |
21+
python -m pip install --upgrade pip
22+
pip install -r local-requirements.txt
23+
pip install -r requirements.txt
24+
pip install -e .
25+
- name: Check out upstream and build the wheel
26+
shell: bash
27+
run: |
28+
python scripts/upstream.py checkout
29+
python -m build --wheel
30+
echo "PLAYWRIGHT_DIR=${GITHUB_WORKSPACE//\\//}/.upstream" >> "$GITHUB_ENV"
31+
- name: Install the browsers
32+
shell: bash
33+
run: python -m playwright install --with-deps chromium firefox webkit
34+
- name: Install the Copilot CLI
35+
shell: bash
36+
run: npm install -g @github/copilot

‎.github/workflows/revision.yml‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
# Copilot addresses the maintainers' review comments on a draft PR that the triage workflow opened,
2+
# without write credentials. A fresh job pushes the new commit to the PR's branch using the app, so
3+
# CI runs on it. The steps that are the same for every Playwright repo are actions in
4+
# microsoft/playwright, and the shared guide they carry says how to revise; the prompt below has
5+
# what is specific to this repo.
6+
name: PR revision
7+
8+
on:
9+
# Apply needs-revision, needs-revision-mac or needs-revision-win; re-apply it to trigger another pass.
10+
# pull_request_target always runs this file from main; the jobs only accept the triage app's own PRs.
11+
pull_request_target:
12+
types: [labeled]
13+
14+
permissions: {}
15+
16+
jobs:
17+
prepare:
18+
name: Collect the maintainers' comments
19+
if: >-
20+
github.repository == 'microsoft/playwright-python' &&
21+
contains(fromJSON('["needs-revision","needs-revision-mac","needs-revision-win"]'), github.event.label.name) &&
22+
github.event.pull_request.state == 'open' &&
23+
github.event.pull_request.user.login == 'microsoft-playwright-automation[bot]' &&
24+
github.event.pull_request.head.repo.full_name == github.repository
25+
runs-on: ubuntu-latest
26+
permissions:
27+
pull-requests: write
28+
env:
29+
PLATFORM: ${{ fromJSON('{"needs-revision":"linux","needs-revision-mac":"mac","needs-revision-win":"windows"}')[github.event.label.name] }}
30+
outputs:
31+
head: ${{ steps.collect.outputs.head }}
32+
base: ${{ steps.collect.outputs.base }}
33+
branch: ${{ steps.collect.outputs.branch }}
34+
platform: ${{ env.PLATFORM }}
35+
steps:
36+
- name: Collect the PR and the maintainers' comments
37+
id: collect
38+
uses: microsoft/playwright/.github/actions/collect-review@main
39+
with:
40+
pull-request: ${{ github.event.pull_request.number }}
41+
- name: Post the run link
42+
env:
43+
GH_TOKEN: ${{ github.token }}
44+
PR: ${{ github.event.pull_request.number }}
45+
PLATFORM_NAME: ${{ fromJSON('{"linux":"Linux","mac":"macOS","windows":"Windows"}')[env.PLATFORM] }}
46+
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
47+
run: |
48+
body=$(printf "Hi, I'm the Playwright bot. I started to address the review comments on %s: [agent run](%s)\n<!-- playwright-revision-started -->" "$PLATFORM_NAME" "$RUN_URL")
49+
gh pr comment "$PR" --body "$body" || echo "::warning::Could not post the run link"
50+
51+
revise:
52+
name: Revise PR ${{ github.event.pull_request.number }} (${{ needs.prepare.outputs.platform }})
53+
needs: prepare
54+
runs-on: ${{ fromJSON('{"linux":"playwright-x64-ubuntu24-64-core","mac":"macos-15-xlarge","windows":"playwright-x64-server2025-64-core"}')[needs.prepare.outputs.platform] }}
55+
timeout-minutes: 240
56+
concurrency:
57+
group: revision-${{ github.event.pull_request.number }}
58+
cancel-in-progress: false
59+
defaults:
60+
run:
61+
shell: bash
62+
env:
63+
PR: ${{ github.event.pull_request.number }}
64+
HEAD_SHA: ${{ needs.prepare.outputs.head }}
65+
BASE_SHA: ${{ needs.prepare.outputs.base }}
66+
TRIAGE_PLATFORM: ${{ needs.prepare.outputs.platform }}
67+
permissions:
68+
copilot-requests: write
69+
outputs:
70+
has_revision: ${{ steps.agent.outputs.has-result }}
71+
steps:
72+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
73+
with:
74+
ref: ${{ needs.prepare.outputs.head }}
75+
persist-credentials: false
76+
- name: Fetch the base and record the directories
77+
run: |
78+
git fetch --no-tags --depth=1 origin "$BASE_SHA"
79+
echo "OUTPUT_DIR=${RUNNER_TEMP//\\//}/revision" >> "$GITHUB_ENV"
80+
echo "REVIEW_DIR=${RUNNER_TEMP//\\//}/review" >> "$GITHUB_ENV"
81+
echo "SCRATCH_DIR=${RUNNER_TEMP//\\//}/scratch" >> "$GITHUB_ENV"
82+
echo "REPO_DIR=${GITHUB_WORKSPACE//\\//}" >> "$GITHUB_ENV"
83+
mkdir -p "${RUNNER_TEMP//\\//}/revision" "${RUNNER_TEMP//\\//}/scratch"
84+
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
85+
with:
86+
name: review
87+
path: ${{ runner.temp }}/review
88+
- name: Set up the runner
89+
uses: ./.github/actions/setup-triage-runner
90+
- name: Revise with Copilot CLI
91+
id: agent
92+
# No GitHub tools and no URL access: the agent reads the PR only from the review the prepare job filtered.
93+
uses: microsoft/playwright/.github/actions/run-agent@main
94+
with:
95+
prompt: |
96+
Follow the shared guide revision.md and this repo's AGENTS.md. Read both in full before starting. $PLAYWRIGHT_DIR is the .upstream/ checkout of microsoft/playwright at the commit in DRIVER_VERSION; do not edit, commit or switch branches in it.
97+
98+
Pull request: https://github.com/microsoft/playwright-python/pull/${{ github.event.pull_request.number }}
99+
output-dir: ${{ env.OUTPUT_DIR }}
100+
result-file: commit/commit.json
101+
artifact-name: revision
102+
exec: ${{ env.CI_ENV_EXEC }}
103+
104+
push:
105+
name: Push the revision
106+
needs: [prepare, revise]
107+
# commit.json is written last, so a finished revision is pushed even when the job failed later.
108+
if: ${{ !cancelled() && needs.revise.outputs.has_revision == 'true' }}
109+
runs-on: ubuntu-latest
110+
steps:
111+
- uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0
112+
id: app-token
113+
with:
114+
app-id: ${{ vars.PLAYWRIGHT_APP_ID }}
115+
private-key: ${{ secrets.PLAYWRIGHT_PRIVATE_KEY }}
116+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
117+
with:
118+
ref: ${{ needs.prepare.outputs.head }}
119+
token: ${{ steps.app-token.outputs.token }}
120+
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
121+
with:
122+
name: revision
123+
path: ${{ runner.temp }}/revision
124+
# Never execute code or scripts from the proposed changes in the app job.
125+
- name: Validate and push
126+
uses: microsoft/playwright/.github/actions/publish-patch@main
127+
with:
128+
token: ${{ steps.app-token.outputs.token }}
129+
dir: ${{ runner.temp }}/revision/commit
130+
patch: revision.patch
131+
metadata: commit.json
132+
branch: ${{ needs.prepare.outputs.branch }}

0 commit comments

Comments
 (0)