Skip to content

Commit f15e8f7

Browse files
authored
test: use spawn context for ProcessPoolExecutor in basic crawler tests (#2276)
The state persistence tests in `test_basic_crawler.py` start `ProcessPoolExecutor` with the default fork start method. On Linux with Python 3.12 and 3.13, forking a multi-threaded process emits `DeprecationWarning: This process is multi-threaded, use of fork() may lead to deadlocks in the child.` The executors now pass `mp_context=get_context('spawn')`. Each spawned worker imports `crawlee.crawlers` again, which adds a few seconds to each of the two tests. Windows (spawn) and Python 3.14 (forkserver) already pay that cost. Closes: #2273 *✍️ Drafted by Claude Code*
1 parent d22ef78 commit f15e8f7

1 file changed

Lines changed: 5 additions & 4 deletions

File tree

‎tests/unit/crawlers/_basic/test_basic_crawler.py‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414
from dataclasses import dataclass
1515
from datetime import datetime, timedelta, timezone
1616
from itertools import product
17+
from multiprocessing import get_context
1718
from typing import TYPE_CHECKING, Any, Literal, cast
1819
from unittest.mock import ANY, AsyncMock, Mock, call, patch
1920

@@ -2347,7 +2348,7 @@ async def test_crawler_state_persistence(tmp_path: Path) -> None:
23472348
storage_client=FileSystemStorageClient(), configuration=Configuration(storage_dir=str(tmp_path))
23482349
)
23492350

2350-
with ProcessPoolExecutor() as executor:
2351+
with ProcessPoolExecutor(mp_context=get_context('spawn')) as executor:
23512352
# Crawl 2 requests in the first run and automatically persist the state.
23522353
first_run_state = executor.submit(
23532354
_process_run_crawlers,
@@ -2360,7 +2361,7 @@ async def test_crawler_state_persistence(tmp_path: Path) -> None:
23602361
assert state.get('urls') == ['https://a.placeholder.com', 'https://b.placeholder.com']
23612362

23622363
# Do not reuse the executor to simulate a fresh process to avoid modified class attributes.
2363-
with ProcessPoolExecutor() as executor:
2364+
with ProcessPoolExecutor(mp_context=get_context('spawn')) as executor:
23642365
# Crawl 1 additional requests in the second run, but use previously automatically persisted state.
23652366
second_run_state = executor.submit(
23662367
_process_run_crawlers,
@@ -2396,7 +2397,7 @@ async def test_crawler_state_persistence_2_crawlers_with_migration(tmp_path: Pat
23962397
storage_client=FileSystemStorageClient(), configuration=Configuration(storage_dir=str(tmp_path))
23972398
)
23982399

2399-
with ProcessPoolExecutor() as executor:
2400+
with ProcessPoolExecutor(mp_context=get_context('spawn')) as executor:
24002401
# Run 2 crawler, each crawl 1 request in and automatically persist the state.
24012402
first_run_states = executor.submit(
24022403
_process_run_crawlers,
@@ -2414,7 +2415,7 @@ async def test_crawler_state_persistence_2_crawlers_with_migration(tmp_path: Pat
24142415
state_1 = await state_kvs.get_value(f'{BasicCrawler._CRAWLEE_STATE_KEY}_1')
24152416
assert state_1.get('urls') == ['https://c.placeholder.com']
24162417

2417-
with ProcessPoolExecutor() as executor:
2418+
with ProcessPoolExecutor(mp_context=get_context('spawn')) as executor:
24182419
# Run 2 crawler, each crawl 1 request in and automatically persist the state.
24192420
second_run_states = executor.submit(
24202421
_process_run_crawlers,

0 commit comments

Comments
 (0)