Skip to content

Commit f4ddee3

Browse files
committed
test: fix the ports and the helpers that the full matrix showed
- PR-007 and CK-011 follow the upstream tests as changed on 2026-10-19 (#43516, #43530, #43480): branded Chromium fails the navigation of the proxy auth test with ERR_INVALID_AUTH_CREDENTIALS, and the clock test has no real-time bound. They failed on every Chrome and Edge channel and on Windows WebKit. - The sync port of NE-027 starts the click in the background and checks the frame as soon as the request arrives, as upstream does. Checking after the click let the popup page appear first on the macOS channels. - The test server registers the waiter of expect_request() at once, as server.waitForRequest() does upstream. A coroutine registered it one turn of the event loop later, after a fast request had reached the server (FE-034 and FE-059 on macOS). - The test server offers http/1.1 over ALPN, as twisted.web.server.Site does. Twisted 26.4 answers a client that offers ALPN, as every browser does, with a fatal alert when the factory offers no protocol; Twisted 25.5 let the connection go on. Every HTTPS test failed since the bump to 26.4. - Screenshots are compared with upstream's ssim-cie94 comparator, ported from packages/utils/image_tools, instead of pixelmatch, and the goldens are upstream's files: three were stale renderings, two per browser were copies under -upstream names, five per browser were unused. - A test that hits its timeout writes the stacks of its process to the standard error before the worker ends: the threads, the greenlets that wait for the event loop, and the pending tasks. pytest-timeout wrote them to the terminal of the test, which an xdist worker does not have. - CI runs the suites with scripts/pytest_dot_reporter.py: one character per test, as Playwright's dot reporter, and every failure in full as soon as it happens, so that the log of a run that is cut short still has it. test.sh uses it when the output is not a terminal.
1 parent 1818e98 commit f4ddee3

49 files changed

Lines changed: 837 additions & 125 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci.yml‎

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,9 @@ jobs:
4040
build:
4141
name: Build
4242
timeout-minutes: 90
43+
env:
44+
# The dot reporter of the test steps (scripts/pytest_dot_reporter.py).
45+
PYTHONPATH: scripts
4346
strategy:
4447
fail-fast: false
4548
matrix:
@@ -117,21 +120,24 @@ jobs:
117120
run: pytest tests/test_installation.py --browser=${{ matrix.browser }}
118121
- name: Test Sync API
119122
if: matrix.os != 'ubuntu-22.04'
120-
run: pytest tests/sync --browser=${{ matrix.browser }} -n auto
123+
run: pytest tests/sync --browser=${{ matrix.browser }} -n auto -p pytest_dot_reporter -rfE --color=yes
121124
- name: Test Sync API
122125
if: matrix.os == 'ubuntu-22.04'
123-
run: xvfb-run pytest tests/sync --browser=${{ matrix.browser }} -n auto
126+
run: xvfb-run pytest tests/sync --browser=${{ matrix.browser }} -n auto -p pytest_dot_reporter -rfE --color=yes
124127
- name: Test Async API
125128
# Also after a failed sync step: the two suites report on their own.
126129
if: ${{ !cancelled() && steps.install.outcome == 'success' && matrix.os != 'ubuntu-22.04' }}
127-
run: pytest tests/async --browser=${{ matrix.browser }} -n auto
130+
run: pytest tests/async --browser=${{ matrix.browser }} -n auto -p pytest_dot_reporter -rfE --color=yes
128131
- name: Test Async API
129132
if: ${{ !cancelled() && steps.install.outcome == 'success' && matrix.os == 'ubuntu-22.04' }}
130-
run: xvfb-run pytest tests/async --browser=${{ matrix.browser }} -n auto
133+
run: xvfb-run pytest tests/async --browser=${{ matrix.browser }} -n auto -p pytest_dot_reporter -rfE --color=yes
131134

132135
test-stable:
133136
name: Stable
134137
timeout-minutes: 90
138+
env:
139+
# The dot reporter of the test steps (scripts/pytest_dot_reporter.py).
140+
PYTHONPATH: scripts
135141
strategy:
136142
fail-fast: false
137143
matrix:
@@ -162,17 +168,17 @@ jobs:
162168
run: pytest tests/common --browser=chromium --browser-channel=${{ matrix.browser-channel }}
163169
- name: Test Sync API
164170
if: matrix.os != 'ubuntu-22.04'
165-
run: pytest tests/sync --browser=chromium --browser-channel=${{ matrix.browser-channel }} -n auto
171+
run: pytest tests/sync --browser=chromium --browser-channel=${{ matrix.browser-channel }} -n auto -p pytest_dot_reporter -rfE --color=yes
166172
- name: Test Sync API
167173
if: matrix.os == 'ubuntu-22.04'
168-
run: xvfb-run pytest tests/sync --browser=chromium --browser-channel=${{ matrix.browser-channel }} -n auto
174+
run: xvfb-run pytest tests/sync --browser=chromium --browser-channel=${{ matrix.browser-channel }} -n auto -p pytest_dot_reporter -rfE --color=yes
169175
- name: Test Async API
170176
# Also after a failed sync step: the two suites report on their own.
171177
if: ${{ !cancelled() && steps.install.outcome == 'success' && matrix.os != 'ubuntu-22.04' }}
172-
run: pytest tests/async --browser=chromium --browser-channel=${{ matrix.browser-channel }} -n auto
178+
run: pytest tests/async --browser=chromium --browser-channel=${{ matrix.browser-channel }} -n auto -p pytest_dot_reporter -rfE --color=yes
173179
- name: Test Async API
174180
if: ${{ !cancelled() && steps.install.outcome == 'success' && matrix.os == 'ubuntu-22.04' }}
175-
run: xvfb-run pytest tests/async --browser=chromium --browser-channel=${{ matrix.browser-channel }} -n auto
181+
run: xvfb-run pytest tests/async --browser=chromium --browser-channel=${{ matrix.browser-channel }} -n auto -p pytest_dot_reporter -rfE --color=yes
176182

177183
test_examples:
178184
name: Examples

‎local-requirements.txt‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,6 @@ build==1.6.1
44
mypy==2.4.0
55
objgraph==3.6.2
66
Pillow==12.3.0
7-
pixelmatch==0.3.0
87
pre-commit==3.5.0
98
pyOpenSSL==26.0.0
109
pytest==9.0.3

‎pyproject.toml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -66,8 +66,9 @@ version_file = "playwright/_repo_version.py"
6666
addopts = "-Wall -rsx -vv -s"
6767
# The timeout of a test is 30 seconds, as upstream; a test marked slow gets three
6868
# times that, as test.slow() gives upstream (tests/conftest.py). The fixtures count,
69-
# as upstream counts them. The thread method ends the process of a test that hangs;
70-
# xdist replaces the worker. The signal method can not be used: it raises inside the
69+
# as upstream counts them. The thread method ends the process of a test that hangs,
70+
# after tests/conftest.py has written its stacks to the standard error; xdist
71+
# replaces the worker. The signal method can not be used: it raises inside the
7172
# running code, and asyncio's Handle._run() catches and logs every BaseException but
7273
# SystemExit and KeyboardInterrupt, so a timeout that fires while the event loop runs
7374
# a callback is lost and the test hangs on.

‎scripts/pytest_dot_reporter.py‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
# Copyright (c) Microsoft Corporation.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""A pytest reporter that prints one character per test, like Playwright's dot reporter.
16+
17+
Load it with `-p pytest_dot_reporter` (scripts/ on PYTHONPATH); CI runs the suites
18+
with it. Each test is one character: `·` passed, `F` failed, `E` failed in a fixture,
19+
`°` skipped, `x` failed as expected, `X` passed unexpectedly, `×` failed and to be run
20+
again. Each failure prints in full as soon as it happens, between the dots, so that
21+
the log of a run that is cut short still has it; the end summary lists the failures
22+
again, one line each. Unlike the line reporter, it does not need a terminal.
23+
"""
24+
25+
from typing import Dict, Optional, Tuple
26+
27+
import pytest
28+
from _pytest.terminal import TerminalReporter
29+
30+
Location = Tuple[str, Optional[int], str]
31+
32+
_LINE_WIDTH = 80
33+
34+
# The character of each test outcome, and its colors.
35+
_MARKS: Dict[str, Tuple[str, Dict[str, bool]]] = {
36+
"passed": ("·", {"green": True}),
37+
"failed": ("F", {"red": True, "bold": True}),
38+
"error": ("E", {"red": True, "bold": True}),
39+
"skipped": ("°", {"yellow": True}),
40+
"xfailed": ("x", {"yellow": True}),
41+
"xpassed": ("X", {"yellow": True}),
42+
"rerun": ("×", {"light": True}),
43+
}
44+
45+
46+
class DotStatus:
47+
"""Takes over the per-test output of the terminal reporter."""
48+
49+
def __init__(self, reporter: TerminalReporter) -> None:
50+
self._reporter = reporter
51+
self._column = 0
52+
53+
def pytest_runtest_logstart(self, nodeid: str, location: Location) -> None:
54+
pass
55+
56+
def pytest_runtest_logreport(self, report: pytest.TestReport) -> None:
57+
reporter = self._reporter
58+
reporter._tests_ran = True
59+
category, letter, _ = reporter.config.hook.pytest_report_teststatus(
60+
report=report, config=reporter.config
61+
)
62+
reporter._add_stats(category, [report])
63+
if report.failed or category == "rerun":
64+
self._print_failure(report, category)
65+
if not letter:
66+
# A fixture that ran: the test gets its character from its call.
67+
return
68+
if self._column == _LINE_WIDTH:
69+
reporter._tw.line()
70+
self._column = 0
71+
mark, markup = _MARKS.get(category, (letter[:1], {}))
72+
reporter._tw.write(mark, **markup)
73+
reporter._tw.flush()
74+
self._column += 1
75+
# Start the next output of the reporter on a new line.
76+
reporter.currentfspath = -2
77+
78+
def pytest_runtest_logfinish(self, nodeid: str, location: Location) -> None:
79+
pass
80+
81+
def summary_failures(self) -> None:
82+
# The failures were printed as they happened.
83+
pass
84+
85+
def summary_errors(self) -> None:
86+
pass
87+
88+
def _print_failure(self, report: pytest.TestReport, category: str) -> None:
89+
# As summary_errors() and summary_failures() print it, on a line of its own.
90+
reporter = self._reporter
91+
headline = reporter._getfailureheadline(report)
92+
if category == "rerun":
93+
headline = f"RERUN {headline}"
94+
elif report.when != "call":
95+
headline = f"ERROR at {report.when} of {headline}"
96+
reporter.write_sep("_", headline, red=True, bold=True)
97+
reporter._outrep_summary(report)
98+
reporter._tw.flush()
99+
self._column = 0
100+
101+
102+
@pytest.hookimpl(trylast=True)
103+
def pytest_configure(config: pytest.Config) -> None:
104+
reporter = config.pluginmanager.getplugin("terminalreporter")
105+
if not isinstance(reporter, TerminalReporter) or hasattr(config, "workerinput"):
106+
return
107+
# Keep the reporter object, which xdist already holds. Replace its per-test hooks
108+
# and its failure sections with DotStatus, then register it again so that pluggy
109+
# calls the replacements.
110+
status = DotStatus(reporter)
111+
config.pluginmanager.unregister(reporter)
112+
for name in (
113+
"pytest_runtest_logstart",
114+
"pytest_runtest_logreport",
115+
"pytest_runtest_logfinish",
116+
"summary_failures",
117+
"summary_errors",
118+
):
119+
setattr(reporter, name, getattr(status, name))
120+
config.pluginmanager.register(reporter, "terminalreporter")

‎test.sh‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
#!/usr/bin/env bash
22
# Runs tests/sync and then tests/async under xvfb, with one worker for every two
3-
# cores. On a terminal, the progress is one status line (scripts/pytest_line_reporter.py),
4-
# and the summary lists only failures and errors, without the warnings.
3+
# cores. On a terminal, the progress is one status line (scripts/pytest_line_reporter.py);
4+
# elsewhere it is one character per test, as on CI (scripts/pytest_dot_reporter.py).
5+
# The summary lists only failures and errors, without the warnings.
56
# Usage: ./test.sh [chromium|firefox|webkit] [extra pytest args]
67
set -u
78
cd "$(dirname "$0")"
@@ -14,7 +15,9 @@ python=python
1415
export PYTHONPATH="scripts${PYTHONPATH:+:$PYTHONPATH}"
1516
# --maxschedchunk=1: a worker gets one test at a time, as soon as it is free, so
1617
# that the slow tests at the end of the collection spread over all workers.
17-
args=(--browser "$browser" -n "$workers" --maxschedchunk=1 -p pytest_line_reporter -rfE --disable-warnings)
18+
reporter=pytest_dot_reporter
19+
[ -t 1 ] && reporter=pytest_line_reporter
20+
args=(--browser "$browser" -n "$workers" --maxschedchunk=1 -p "$reporter" -rfE --disable-warnings)
1821
status=0
1922
xvfb-run -a "$python" -m pytest tests/sync "${args[@]}" "$@" || status=$?
2023
xvfb-run -a "$python" -m pytest tests/async "${args[@]}" "$@" || status=$?

‎tests/async/library/test_headful.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -477,7 +477,7 @@ async def test_HE_015_headless_and_headful_should_use_same_default_fonts(
477477
page.screenshot(), headless_page.screenshot()
478478
)
479479
# Port of utils.compare(image1, image2), which counts the different pixels.
480-
count = count_different_pixels(image1, image2, threshold=0.1)
480+
count = count_different_pixels(image1, image2)
481481
assert count == 0
482482
await headless_browser.close()
483483

‎tests/async/library/test_page_clock.py‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -219,11 +219,8 @@ async def test_CK_011_run_for_passes_2_hours_34_minutes_and_10_seconds(
219219
setInterval(window.stub, 10000);
220220
}"""
221221
)
222-
start = time.monotonic()
223222
await page.clock.run_for("02:34:10")
224223
assert len(calls) == 925
225-
# Nested setTimeout(fn, 0) is clamped to 4ms, so firing 925 timers used to take over 4 seconds.
226-
assert (time.monotonic() - start) * 1000 < 2000
227224

228225

229226
@pytest.mark.usefixtures("install_and_pause")

‎tests/async/library/test_proxy.py‎

Lines changed: 16 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -207,7 +207,11 @@ async def test_PR_006_should_allow_bypassing_requests(
207207

208208

209209
async def test_PR_007_should_authenticate(
210-
browser_type: BrowserType, launch_arguments: Dict, server: Server
210+
browser_type: BrowserType,
211+
launch_arguments: Dict,
212+
server: Server,
213+
browser_name: str,
214+
is_headless_shell: bool,
211215
) -> None:
212216
def handle_target(request: TestServerRequest) -> None:
213217
auth = request.getHeader("proxy-authorization")
@@ -244,10 +248,17 @@ def handle_protected(request: TestServerRequest) -> None:
244248
await page.goto("http://non-existent.test/target.html")
245249
assert await page.title() == "Basic " + base64.b64encode(b"user:secret").decode()
246250
# Proxy credentials must not be sent to the website's own auth challenge.
247-
response = await page.goto("http://non-existent.com/protected.html")
248-
assert response
249-
assert response.status == 401
250-
assert await page.locator("body").text_content() == "no credentials"
251+
response_or_error: Any
252+
try:
253+
response_or_error = await page.goto("http://non-existent.test/protected.html")
254+
except Error as error:
255+
response_or_error = error
256+
if browser_name == "chromium" and not is_headless_shell:
257+
# Branded Chromium shows a login dialog for the 401 and fails the navigation.
258+
assert "net::ERR_INVALID_AUTH_CREDENTIALS" in response_or_error.message
259+
else:
260+
assert response_or_error.status == 401
261+
assert await page.locator("body").text_content() == "no credentials"
251262
await browser.close()
252263

253264

‎tests/async/library/test_screenshot.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ async def new_page() -> Page:
8181
for i in range(n):
8282
assert_to_be_golden(
8383
screenshots[i],
84-
"grid-cell-0-upstream.png" if i % 2 == 0 else "grid-cell-1.png",
84+
"grid-cell-0.png" if i % 2 == 0 else "grid-cell-1.png",
8585
)
8686
await asyncio.gather(*[page.close() for page in pages])
8787

‎tests/async/page/test_page_screenshot.py‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -117,7 +117,7 @@ async def test_SS_001_page_screenshot_should_work_smoke(
117117
await page.set_viewport_size({"width": 500, "height": 500})
118118
await page.goto(server.PREFIX + "/grid.html")
119119
screenshot = await page.screenshot()
120-
assert_to_be_golden(screenshot, "screenshot-sanity-upstream.png")
120+
assert_to_be_golden(screenshot, "screenshot-sanity.png")
121121

122122

123123
@pytest.mark.usefixtures("skip_firefox_headed")
@@ -548,7 +548,7 @@ async def test_SS_029_page_screenshot_path_option_should_work(
548548
await page.goto(server.PREFIX + "/grid.html")
549549
output_path = tmp_path / "screenshot.png"
550550
await page.screenshot(path=output_path)
551-
assert_to_be_golden(output_path.read_bytes(), "screenshot-sanity-upstream.png")
551+
assert_to_be_golden(output_path.read_bytes(), "screenshot-sanity.png")
552552

553553

554554
@pytest.mark.usefixtures("skip_firefox_headed")
@@ -559,7 +559,7 @@ async def test_SS_030_page_screenshot_path_option_should_create_subdirectories(
559559
await page.goto(server.PREFIX + "/grid.html")
560560
output_path = tmp_path / "these" / "are" / "directories" / "screenshot.png"
561561
await page.screenshot(path=output_path)
562-
assert_to_be_golden(output_path.read_bytes(), "screenshot-sanity-upstream.png")
562+
assert_to_be_golden(output_path.read_bytes(), "screenshot-sanity.png")
563563

564564

565565
@pytest.mark.usefixtures("skip_firefox_headed")
@@ -915,7 +915,7 @@ async def test_SS_056_page_screenshot_animations_should_resume_infinite_animatio
915915
buffer1 = await page.screenshot()
916916
await utils.rafraf(page)
917917
buffer2 = await page.screenshot()
918-
assert not compare_images(buffer1, buffer2, max_diff_pixels=50)
918+
assert compare_images(buffer1, buffer2, max_diff_pixels=50) is not None
919919

920920

921921
async def test_SS_057_page_screenshot_animations_should_not_capture_infinite_web_animations(
@@ -932,7 +932,7 @@ async def test_SS_057_page_screenshot_animations_should_not_capture_infinite_web
932932
buffer1 = await page.screenshot()
933933
await utils.rafraf(page)
934934
buffer2 = await page.screenshot()
935-
assert not compare_images(buffer1, buffer2, max_diff_pixels=50)
935+
assert compare_images(buffer1, buffer2, max_diff_pixels=50) is not None
936936

937937

938938
async def test_SS_058_page_screenshot_animations_should_fire_transitionend_for_finite_transitions(

0 commit comments

Comments
 (0)