Skip to content

Commit 1d4007e

Browse files
authored
test: Rebalance parallel test scheduling and trim fixture overhead (#2199)
The unit suite spends most of its wall-clock waiting. On a reference CI job (`ubuntu-latest / 3.14`, run 33047203526) the main pass does 578 worker-seconds of work in 131.8s across 8 workers and burns 476 of them idle. `--dist load` seeds each worker with a fixed chunk of consecutive tests and never reassigns, so `gw4` was still running `_playwright/test_utils.py` at 125s with the other seven idle for about a minute. Perfect balance would finish at 72.3s. Same failure mode as apify/apify-client-python#1037. ## Changes 1. **`--dist worksteal` in `addopts`** so an idle worker takes over tests still queued on a busy one. Nothing in the repo uses `xdist_group`, and `--dist` is inert without `-n`. 2. **`proxy` and `disabled_proxy` are session-scoped and sync.** Each teardown blocks ~1s on proxy.py's hardcoded `selector.select(timeout=1)`; with 17 servers built and destroyed that is ~17 worker-seconds of pure waiting. The server is a stateless forwarder, so one per session is enough. Sync because a session-scoped async fixture is incompatible with `asyncio_default_fixture_loop_scope = "function"`, and neither ever awaited. 3. **Dropped `--numprocesses=1` from the `run_alone` pass** - it paid the full xdist worker bootstrap for a 15-test serial suite. 4. **`COVERAGE_CORE=sysmon` on `unit-tests-cov`.** Measured on a 1397-test subset: default core +14%, sysmon +0%. Coverage falls back with a warning where `sys.monitoring` is unavailable, so 3.10 and 3.11 are safe. 5. **`test_infinite_scroll_on_dynamic_page` waits on the page instead of the clock** - four `wait_for_timeout(1000)` sleeps for a `loadMore()` that awaits 100ms, on the critical-path worker. ## Measured Full suite, both passes, same machine, two reps each: **~114s -> ~74s (-35%)**. These runs are without coverage, so change 4 is not reflected. 2456 tests pass on both sides; `ruff format`, `ruff check` and `ty` are clean. ## Not in this PR - `from crawlee.crawlers import HttpCrawler` costs 3.51s, of which 1.28s is scikit-learn and scipy, imported eagerly by `_adaptive_playwright._rendering_type_predictor` (behind the optional `adaptive-crawler` extra). Every worker pays it, as does every `crawlee[all]` cold start. Not a one-liner - `RenderingTypePredictorState` needs `LogisticRegression` at class-creation time. Worth its own issue. - 14 of the 15 matrix jobs compute coverage that is never uploaded; skipping it needs a change in `apify/workflows`. - `tests_concurrency: "8"` may be 2x oversubscription on 4-vCPU runners; worth one run with `nproc` echoed and an `-n4` comparison. - `test_http_2` still hits the real internet - it needs an HTTP/2 endpoint the local uvicorn server does not speak. *✍️ Drafted by Claude Code*
1 parent 24a4bc5 commit 1d4007e

3 files changed

Lines changed: 35 additions & 20 deletions

File tree

‎pyproject.toml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -263,7 +263,7 @@ builtins-ignorelist = ["id"]
263263
known-first-party = ["crawlee"]
264264

265265
[tool.pytest.ini_options]
266-
addopts = "-r a --verbose"
266+
addopts = "-r a --verbose --dist worksteal"
267267
asyncio_default_fixture_loop_scope = "function"
268268
asyncio_mode = "auto"
269269
timeout = 1800
@@ -339,7 +339,6 @@ shell = "uv run ruff check --fix && uv run ruff format"
339339
[tool.poe.tasks.unit-tests]
340340
shell = """
341341
uv run pytest \
342-
--numprocesses=1 \
343342
-m "run_alone" \
344343
tests/unit && \
345344
uv run pytest \
@@ -349,9 +348,10 @@ uv run pytest \
349348
"""
350349

351350
[tool.poe.tasks.unit-tests-cov]
351+
# Falls back to the default core with a warning on Python < 3.12.
352+
env = { COVERAGE_CORE = "sysmon" }
352353
shell = """
353354
uv run pytest \
354-
--numprocesses=1 \
355355
-m "run_alone" \
356356
--cov=src/crawlee \
357357
--cov-report=xml:coverage-unit.xml \

‎tests/unit/conftest.py‎

Lines changed: 26 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -103,23 +103,37 @@ def _set_crawler_log_level(pytestconfig: pytest.Config, monkeypatch: pytest.Monk
103103
monkeypatch.setattr(_log_config, 'get_configured_log_level', lambda: getattr(logging, loglevel.upper()))
104104

105105

106-
@pytest.fixture
107-
async def proxy_info(unused_tcp_port: int) -> ProxyInfo:
106+
def _proxy_info(port: int) -> ProxyInfo:
107+
"""Describe a local proxy listening on `port`, authenticated with fixed throwaway credentials."""
108108
username = 'user'
109109
password = 'pass'
110110

111111
return ProxyInfo(
112-
url=f'http://{username}:{password}@127.0.0.1:{unused_tcp_port}',
112+
url=f'http://{username}:{password}@127.0.0.1:{port}',
113113
scheme='http',
114114
hostname='127.0.0.1',
115-
port=unused_tcp_port,
115+
port=port,
116116
username=username,
117117
password=password,
118118
)
119119

120120

121-
@pytest.fixture
122-
async def proxy(proxy_info: ProxyInfo) -> AsyncGenerator[ProxyInfo, None]:
121+
# Session-scoped because a `Proxy` teardown blocks for up to a second on its acceptor's
122+
# `selector.select(timeout=1)`. Sync because a session-scoped async fixture is incompatible with
123+
# `asyncio_default_fixture_loop_scope = "function"`.
124+
@pytest.fixture(scope='session')
125+
def proxy_info(unused_tcp_port_factory: Callable[[], int]) -> ProxyInfo:
126+
return _proxy_info(unused_tcp_port_factory())
127+
128+
129+
@pytest.fixture(scope='session')
130+
def disabled_proxy_info(unused_tcp_port_factory: Callable[[], int]) -> ProxyInfo:
131+
"""Describe the disabled proxy, which needs a port of its own now that both servers outlive a test."""
132+
return _proxy_info(unused_tcp_port_factory())
133+
134+
135+
@pytest.fixture(scope='session')
136+
def proxy(proxy_info: ProxyInfo) -> Iterator[ProxyInfo]:
123137
with Proxy(
124138
[
125139
'--hostname',
@@ -139,24 +153,24 @@ async def proxy(proxy_info: ProxyInfo) -> AsyncGenerator[ProxyInfo, None]:
139153
yield proxy_info
140154

141155

142-
@pytest.fixture
143-
async def disabled_proxy(proxy_info: ProxyInfo) -> AsyncGenerator[ProxyInfo, None]:
156+
@pytest.fixture(scope='session')
157+
def disabled_proxy(disabled_proxy_info: ProxyInfo) -> Iterator[ProxyInfo]:
144158
with Proxy(
145159
[
146160
'--hostname',
147-
proxy_info.hostname,
161+
disabled_proxy_info.hostname,
148162
'--port',
149-
str(proxy_info.port),
163+
str(disabled_proxy_info.port),
150164
'--basic-auth',
151-
f'{proxy_info.username}:{proxy_info.password}',
165+
f'{disabled_proxy_info.username}:{disabled_proxy_info.password}',
152166
'--disable-http-proxy',
153167
'--num-workers',
154168
'1',
155169
'--num-acceptors',
156170
'1',
157171
]
158172
):
159-
yield proxy_info
173+
yield disabled_proxy_info
160174

161175

162176
@pytest.fixture(scope='session')

‎tests/unit/crawlers/_playwright/test_utils.py‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,13 @@ async def test_infinite_scroll_on_dynamic_page(server_url: URL) -> None:
2020
# Get data with manual scrolling
2121
await page.goto(target_url)
2222

23-
manual_items = []
24-
for _ in range(4):
25-
items = await page.query_selector_all('.item')
26-
manual_items = items
23+
# The page appends one batch of items per scroll, for three batches. Wait for each batch to land
24+
# instead of sleeping, so the baseline costs as long as the page takes rather than a fixed 4 seconds.
25+
manual_items = await page.query_selector_all('.item')
26+
for _ in range(3):
2727
await page.evaluate('window.scrollTo(0, document.body.scrollHeight)')
28-
await page.wait_for_timeout(1000)
28+
await page.wait_for_function(f'document.querySelectorAll(".item").length > {len(manual_items)}')
29+
manual_items = await page.query_selector_all('.item')
2930

3031
# Reset page
3132
await page.close()

0 commit comments

Comments
 (0)