Skip to content

test: use spawn context for ProcessPoolExecutor in basic crawler tests - #2276

Merged
vdusek merged 1 commit into
masterfrom
test/spawn-process-pool
Oct 5, 2026
Merged

vdusek merged 1 commit into
masterfrom
test/spawn-process-pool

Conversation

@vdusek

@vdusek vdusek commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator

The state persistence tests in test_basic_crawler.py start ProcessPoolExecutor with the default fork start method. On Linux with Python 3.12 and 3.13, forking a multi-threaded process emits DeprecationWarning: This process is multi-threaded, use of fork() may lead to deadlocks in the child. The executors now pass mp_context=get_context('spawn').

Each spawned worker imports crawlee.crawlers again, which adds a few seconds to each of the two tests. Windows (spawn) and Python 3.14 (forkserver) already pay that cost.

Closes #2273

✍️ Drafted by Claude Code

@vdusek vdusek added the t-tooling Issues with this label are in the ownership of the tooling team. label Oct 5, 2026
@vdusek vdusek self-assigned this Oct 5, 2026
@vdusek vdusek added the t-tooling Issues with this label are in the ownership of the tooling team. label Oct 5, 2026
@codecov

codecov Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.91%. Comparing base (87561ee) to head (edf5b7d).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #2276   +/-   ##
=======================================
  Coverage   93.90%   93.91%           
=======================================
  Files         182      182           
  Lines       13144    13144           
=======================================
+ Hits        12343    12344    +1     
+ Misses        801      800    -1     
Flag Coverage Δ
unit 93.91% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vdusek
vdusek requested a review from Mantisus October 5, 2026 16:58
@vdusek
vdusek marked this pull request as ready for review October 5, 2026 16:58
@apify-service-account apify-service-account added the tested Temporary label used only programatically for some analytics. label Oct 5, 2026

@Mantisus Mantisus left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@apify-approval-mirror apify-approval-mirror Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mirroring approval from @Mantisus.

@vdusek
vdusek merged commit f15e8f7 into master Oct 5, 2026
37 of 39 checks passed
@vdusek
vdusek deleted the test/spawn-process-pool branch October 5, 2026 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use spawn context for ProcessPoolExecutor in basic crawler tests

3 participants