feat(v2): add the sync API on top of playwright/_impl_v2 (#3277)

Author: pavelfeldman

.github/workflows/ci.yml

@@ -127,8 +127,8 @@ jobs:
 
   build-v2:
     # playwright/_impl_v2, the line-by-line port of the JS client, behind PW_IMPL=v2.
-    # It has only the async API, and runs all of tests/async: the older tests, the
-    # ports of the upstream tests and the Python-only tests of v2.
+    # It runs all of tests/sync and tests/async: the older tests, the ports of the
+    # upstream tests and the Python-only tests of v2.
     name: Build (v2)
     timeout-minutes: 45
     strategy:
@@ -152,6 +152,8 @@ jobs:
         pip install -e .
         python -m build --wheel
         python -m playwright install --with-deps ${{ matrix.browser }}
+    - name: Test Sync API
+      run: xvfb-run pytest tests/sync --browser=${{ matrix.browser }} --timeout 90 --timeout-method thread -n auto
     - name: Test Async API
       run: xvfb-run pytest tests/async --browser=${{ matrix.browser }} --timeout 90 --timeout-method thread -n auto
 

AGENTS.md

@@ -9,7 +9,7 @@ Python bindings for [Playwright](https://playwright.dev). The Python client talk
 ## Layout
 
 - `playwright/_impl/` — hand-written client implementation (one module per object: `_browser.py`, `_page.py`, `_locator.py`, `_network.py`, etc.). Edit these to add or change behavior. Internal types are in `_internal_structures.py`.
-- `playwright/_impl_v2/` — the second implementation, a line-by-line port of upstream's JS client (`packages/playwright-core/src/client/*.ts`), one module per client file in snake_case (`browserContext.ts` is `browser_context.py`), members in the JS order. It has only the async API; `PW_IMPL=v2` switches the async API to it (`playwright/_impl/_api_classes.py`). Each difference from the JS code has a `# Python:` comment; see its `__init__.py` for the rules. `isomorphic/`, `protocol/` and `utils/` hold its own ports of the parts of `packages/isomorphic`, `packages/protocol` and `packages/utils` that it uses; from `playwright/_impl` it uses only the public error classes, `FormData`, the generated API types, the driver location and the protocol check. `protocol_channels.py`, `protocol_types.py` and `protocol/validator.py` are generated by `scripts/codegen/generate_protocol.py`.
+- `playwright/_impl_v2/` — the second implementation, a line-by-line port of upstream's JS client (`packages/playwright-core/src/client/*.ts`), one module per client file in snake_case (`browserContext.ts` is `browser_context.py`), members in the JS order. `PW_IMPL=v2` switches the async and the sync API to it (`playwright/_impl/_api_classes.py`, `playwright/{async,sync}_api/_context_manager.py`); the sync API is the greenlet mechanism of v1, shared (`playwright/_impl/_sync_base.py`, `playwright/_impl/_greenlets.py`). Each difference from the JS code has a `# Python:` comment; see its `__init__.py` for the rules. `isomorphic/`, `protocol/` and `utils/` hold its own ports of the parts of `packages/isomorphic`, `packages/protocol` and `packages/utils` that it uses; from `playwright/_impl` it uses only the public error classes, `FormData`, the generated API types, the driver location, the protocol check and the greenlets of the sync API. `protocol_channels.py`, `protocol_types.py` and `protocol/validator.py` are generated by `scripts/codegen/generate_protocol.py`.
 - `playwright/async_api/_generated.py`, `playwright/sync_api/_generated.py`, and the public TypedDicts in `playwright/_impl/_api_structures.py` — **auto-generated** from the upstream API docs. Never edit by hand; rerun `./scripts/update_api.sh` after a roll or after changing `scripts/codegen/`.
 - `scripts/codegen/generate_api.js` — the generator. A Node.js script that loads the upstream doc parser (`utils/doclint/api_parser.js`) from `$PW_SRC_DIR`, filters the docs for Python, and writes the whole public API: signatures, types, docstrings, event overloads, the calls into `_impl/`, and the TypedDicts. It does not read `_impl/`.
 - `scripts/codegen/python_rules.js` — the Python facts that the docs do not contain (positional options, struct `total=`, hidden members and options). Policy: the shipped Python API wins; each entry is a candidate for an upstream docs fix, and the generator fails on entries that no longer match anything. Every entry needs a one-line rationale comment above it.
@@ -18,7 +18,7 @@ Python bindings for [Playwright](https://playwright.dev). The Python client talk
   - `_protocol_types.py` has the TypedDicts for initializers, results, events and nested types, named as upstream's `utils/generate_channels.js` names them (`PageInitializer`, `FrameGotoResult`, `PageRouteEvent`, …).
   - `_protocol.py` has the parameters of every command. With `PW_CHECK_PROTOCOL=1`, `playwright/_impl/_protocol_check.py` checks each message that the client sends against it. The test suite sets this, and fails a test that sends an unknown command, an unknown or missing parameter, or a value of the wrong type.
 - `scripts/codegen/compare_api.py` — compares two generated files (signatures, annotations, call bodies, docstrings) and lists every difference. Use it to check that a generator change does not change the API.
-- `tests/async/`, `tests/sync/` — pytest suites, ported test for test from the upstream tests, which `tests/upstream_registry.md` lists by identifier. See [Tests](#tests). `playwright/_impl` (v1, the default) runs the older tests in `tests/{async,sync}/legacy/` and the other tests outside of `tests/async` and `tests/sync`; `PW_IMPL=v2` runs `playwright/_impl_v2` on all of `tests/async` (`tests/conftest.py`). CI runs v2 on Ubuntu 22.04 with Chromium, Firefox and WebKit. An older test that checks a behavior where v2 follows upstream and v1 does not is marked `@only_v1("<tests of v2>")` (`tests/utils.py`), naming the upstream port that covers v2, or the Python-only test of v2 in `tests/async/python/`, which v1 does not run.
+- `tests/async/`, `tests/sync/` — pytest suites, ported test for test from the upstream tests, which `tests/upstream_registry.md` lists by identifier. See [Tests](#tests). `playwright/_impl` (v1, the default) runs the older tests in `tests/{async,sync}/legacy/` and the other tests outside of `tests/async` and `tests/sync`; `PW_IMPL=v2` runs `playwright/_impl_v2` on all of `tests/async` and `tests/sync` (`tests/conftest.py`). CI runs v2 on Ubuntu 22.04 with Chromium, Firefox and WebKit. An older test that checks a behavior where v2 follows upstream and v1 does not is marked `@only_v1("<tests of v2>")` (`tests/utils.py`), naming the upstream port that covers v2, or the Python-only test of v2 in `tests/async/python/`, which v1 does not run.
 - `DRIVER_VERSION` — the single source of truth for which Playwright build the driver is assembled from. Two lines: `npm=<version>`, the `playwright-core` npm version (e.g. `npm=1.61.0`, no `v` prefix), and `rev=<sha>`, the `microsoft/playwright` commit that the package is built from (`npm view playwright-core@<version> gitHead`). Read by `setup.py`, `scripts/build_driver.py`, `scripts/update_api.sh`, and CI (through `setup.py`). The wheel build downloads `playwright-core` at this version from npm plus the matching Node.js binary and assembles the per-platform bundles — no source build. The version is baked into the staged bundle filenames (`driver/playwright-<version>-<suffix>.zip`), so it doubles as the build cache key.
 - `NODE_VERSION` — the Node.js version bundled with the driver (one line, e.g. `24.16.0`). Maintained at roll time by `scripts/update_node_version.py` (latest LTS, mirroring upstream's `utils/build/update-playwright-node.mjs`).
 - `scripts/build_driver.py` — assembles the per-platform driver bundles into `driver/` by downloading the `playwright-core` npm package (`DRIVER_VERSION`) and the official Node.js binaries (`NODE_VERSION`). Fetches `playwright-core` with `npm pack` (needs Node.js/npm on PATH; honours a root `.npmrc`) and the Node.js binaries over plain HTTP; invoked from `setup.py`'s `bdist_wheel` with the target platform's suffix (no arg builds all six).
@@ -35,17 +35,16 @@ pip install --upgrade pip
 pip install -r local-requirements.txt
 pip install -e .
 python -m build --wheel        # downloads playwright-core @ DRIVER_VERSION + Node.js and assembles the driver
-pre-commit install
 ```
 
 If the system lacks `python3-venv`, `uv venv env` is an acceptable substitute (then `uv pip install --python env/bin/python --upgrade pip`).
 
 ## Common commands
 
 - Regenerate `_generated.py`: `./scripts/update_api.sh` (runs codegen + pre-commit on the generated files).
-- Lint everything: `pre-commit run --all-files`.
+- Lint everything: `pre-commit run --all-files` (what the Lint job of CI runs; the checks are not a git hook, do not `pre-commit install`).
 - Type-check: `mypy playwright`.
-- Run tests: `pytest --browser chromium [-k name]` (v1), `PW_IMPL=v2 pytest tests/async --browser chromium` (v2), or `./test.sh [browser]` for both. Browsers are installed via `playwright install chromium` (do **not** use `--with-deps`, which requires sudo).
+- Run tests: `pytest --browser chromium [-k name]` (v1), `PW_IMPL=v2 pytest tests/sync tests/async --browser chromium` (v2; `./test-v2.sh [browser]` runs both suites), or `./test.sh [browser]` for both implementations. Browsers are installed via `playwright install chromium` (do **not** use `--with-deps`, which requires sudo).
 
 The public API comes from the upstream docs. After `./scripts/update_api.sh`, run `mypy playwright`: the generated classes type their `_impl_obj`, so every new member or option that `_impl/` does not implement yet is a mypy error in `_generated.py`. Fix it in `_impl/`, or record a justified difference in `scripts/codegen/python_rules.js` — not by hand-editing `_generated.py`.
 
@@ -91,7 +90,7 @@ It documents the full process: the upstream commit-range diff over `docs/src/api
 
 ## Commit Convention
 
-Before committing, run `mypy playwright` and fix errors.
+Before committing, run `pre-commit run --all-files` (the Lint job of CI) and fix errors.
 
 Semantic commit messages: `label(scope): description`
 

CONTRIBUTING.md

@@ -64,10 +64,9 @@ Checking for typing errors
 mypy playwright
 ```
 
-Format the code
+Lint and format the code (what the Lint job of CI runs; the checks are not a git hook)
 
 ```sh
-pre-commit install
 pre-commit run --all-files
 ```
 

ROLLING.md

@@ -8,7 +8,6 @@
 ```
 python -m pip install --upgrade pip
 pip install -r local-requirements.txt
-pre-commit install
 pip install -e .
 ```
 * change the driver pin in `DRIVER_VERSION`: `npm=` the `playwright-core` npm version (e.g. `1.61.0`) and `rev=` the commit it is built from (`npm view playwright-core@<version> gitHead`); refresh `NODE_VERSION`: `python scripts/update_node_version.py`

playwright/_impl/_greenlets.py

@@ -13,7 +13,7 @@
 # limitations under the License.
 import asyncio
 import os
-from typing import Tuple
+from typing import Any, Callable, Tuple, Type
 
 import greenlet
 
@@ -52,10 +52,41 @@ def __str__(self) -> str:
         return "<EventGreenlet>"
 
 
+class BindingGreenlet(greenlet.greenlet):
+    def __str__(self) -> str:
+        return "<BindingGreenlet>"
+
+
+class WebSocketRouteGreenlet(greenlet.greenlet):
+    def __str__(self) -> str:
+        return "<WebSocketRouteGreenlet>"
+
+
 def connection_closed_error() -> TargetClosedError:
     return TargetClosedError("Playwright connection closed")
 
 
+def run_in_greenlet(
+    greenlet_class: Type[greenlet.greenlet], func: Callable[[], Any]
+) -> "asyncio.Future[Any]":
+    """Runs a callback of the sync API in its own greenlet.
+
+    The callback may block in a sync call. In its own greenlet it switches to the
+    dispatcher fiber then, and the event loop goes on. The future settles with the
+    result of the callback when it returns, or with its error.
+    """
+    future: "asyncio.Future[Any]" = asyncio.get_running_loop().create_future()
+
+    def body() -> None:
+        try:
+            future.set_result(func())
+        except Exception as e:
+            future.set_exception(e)
+
+    greenlet_class(body).switch()
+    return future
+
+
 def wait_for_future(
     loop: asyncio.AbstractEventLoop,
     dispatcher_fiber: greenlet.greenlet,

playwright/_impl/_sync_base.py

@@ -13,6 +13,8 @@
 # limitations under the License.
 
 import asyncio
+import os
+import sys
 import traceback
 from contextlib import AbstractContextManager
 from types import TracebackType
@@ -36,6 +38,9 @@
 
 mapping = ImplToApiMapping()
 
+# PW_IMPL=v2: playwright/_impl_v2 reads the stack of a sync call from its frames.
+_IS_V2 = os.environ.get("PW_IMPL") == "v2"
+
 
 T = TypeVar("T")
 Self = TypeVar("Self", bound="SyncContextManager")
@@ -104,8 +109,14 @@ def _sync(
             raise connection_closed_error()
 
         task: asyncio.tasks.Task[Any] = self._loop.create_task(coro)
-        setattr(task, "__pw_stack__", _capture_stack_trace())
-        setattr(task, "__pw_stack_trace__", traceback.extract_stack(limit=10))
+        # The task runs in the dispatcher fiber, where the stack of this call is not
+        # visible. The implementation reads it from the task: v2 walks the frames of
+        # the caller (capture_raw_stack()), v1 takes the parsed stack.
+        if _IS_V2:
+            setattr(task, "__pw_caller_frame__", sys._getframe(1))
+        else:
+            setattr(task, "__pw_stack__", _capture_stack_trace())
+            setattr(task, "__pw_stack_trace__", traceback.extract_stack(limit=10))
 
         wait_for_future(self._loop, self._dispatcher_fiber, task)
         return task.result()

playwright/_impl_v2/__init__.py

@@ -37,5 +37,9 @@
   runs the prefix and returns the awaitable of the rest (the wait_for_*() methods).
 - Python has no AbortSignal: the `signal` options are not ported. Cancelling the task
   that waits for a call aborts the call, as an aborted signal does.
-- There is no sync code here: no greenlet, no sync waits.
+- The sync API is the one of playwright/_impl (playwright/sync_api/_context_manager.py,
+  playwright/_impl/_sync_base.py, playwright/_impl/_greenlets.py): a greenlet runs the
+  event loop, and a sync call switches to it. Each object carries `_loop` and
+  `_dispatcher_fiber` for the generated API; a user callback that may block runs in a
+  greenlet of its own (transform.md, section 13).
 """

playwright/_impl_v2/api_classes.py

@@ -15,7 +15,7 @@
 """Python only: the v2 class of each class in the generated API.
 
 playwright/_impl/_api_classes.py takes these when PW_IMPL=v2 is set. The expect()
-matchers are not part of the client; they stay in playwright/_impl/_assertions.py.
+matchers are not part of the JS client; the Python ones are in assertions.py.
 """
 
 from playwright._impl_v2.assertions import (

playwright/_impl_v2/assertions.py

@@ -62,9 +62,10 @@ def __init__(
         is_soft: bool = False,
     ) -> None:
         self._actual_locator = locator
+        # Python: the generated API reads the event loop and the dispatcher fiber of
+        # each object.
         self._loop = locator._loop
-        # The sync API reads it; playwright/_impl_v2 (async only) has none.
-        self._dispatcher_fiber = getattr(locator, "_dispatcher_fiber", None)
+        self._dispatcher_fiber = locator._dispatcher_fiber
         self._timeout = timeout
         self._is_not = is_not
         self._custom_message = message
@@ -1057,9 +1058,10 @@ def __init__(
         message: Optional[str] = None,
         is_soft: bool = False,
     ) -> None:
+        # Python: the generated API reads the event loop and the dispatcher fiber of
+        # each object.
         self._loop = response._loop
-        # The sync API reads it; playwright/_impl_v2 (async only) has none.
-        self._dispatcher_fiber = getattr(response, "_dispatcher_fiber", None)
+        self._dispatcher_fiber = response._dispatcher_fiber
         self._timeout = timeout
         self._is_not = is_not
         self._actual = response

playwright/_impl_v2/browser_context.py

@@ -187,7 +187,7 @@ def on_service_worker(
         def on_console(event: "channels.BrowserContextConsoleEvent") -> None:
             worker = Worker.from_nullable(event["worker"])
             page = Page.from_nullable(event["page"])
-            console_message = ConsoleMessage(event, page, worker)
+            console_message = ConsoleMessage(event, page, worker, self)
             if worker:
                 worker.emit(Events.Worker.Console, console_message)
             if page:
@@ -208,7 +208,7 @@ def on_page_error(event: "channels.BrowserContextPageErrorEvent") -> None:
             parsed_error = parse_error(event["error"])
             self.emit(
                 Events.BrowserContext.WebError,
-                WebError(page_object, parsed_error, event["location"]),
+                WebError(page_object, parsed_error, event["location"], self),
             )
             if page_object:
                 page_object.emit(Events.Page.PageError, parsed_error)
@@ -590,7 +590,7 @@ async def route(
             ),
         )
         await self._update_interception_patterns()
-        return DisposableStub(lambda: self.unroute(url=url, handler=handler))
+        return DisposableStub(lambda: self.unroute(url=url, handler=handler), self)
 
     @api_call(title="Route WebSockets")
     async def route_web_socket(