Repository navigation
Commit 1d4007e
authored
test: Rebalance parallel test scheduling and trim fixture overhead (#2199)
The unit suite spends most of its wall-clock waiting. On a reference CI
job (`ubuntu-latest / 3.14`, run 33047203526) the main pass does 578
worker-seconds of work in 131.8s across 8 workers and burns 476 of them
idle. `--dist load` seeds each worker with a fixed chunk of consecutive
tests and never reassigns, so `gw4` was still running
`_playwright/test_utils.py` at 125s with the other seven idle for about
a minute. Perfect balance would finish at 72.3s.
Same failure mode as apify/apify-client-python#1037.
## Changes
1. **`--dist worksteal` in `addopts`** so an idle worker takes over
tests still queued on a busy one. Nothing in the repo uses
`xdist_group`, and `--dist` is inert without `-n`.
2. **`proxy` and `disabled_proxy` are session-scoped and sync.** Each
teardown blocks ~1s on proxy.py's hardcoded
`selector.select(timeout=1)`; with 17 servers built and destroyed that
is ~17 worker-seconds of pure waiting. The server is a stateless
forwarder, so one per session is enough. Sync because a session-scoped
async fixture is incompatible with `asyncio_default_fixture_loop_scope =
"function"`, and neither ever awaited.
3. **Dropped `--numprocesses=1` from the `run_alone` pass** - it paid
the full xdist worker bootstrap for a 15-test serial suite.
4. **`COVERAGE_CORE=sysmon` on `unit-tests-cov`.** Measured on a
1397-test subset: default core +14%, sysmon +0%. Coverage falls back
with a warning where `sys.monitoring` is unavailable, so 3.10 and 3.11
are safe.
5. **`test_infinite_scroll_on_dynamic_page` waits on the page instead of
the clock** - four `wait_for_timeout(1000)` sleeps for a `loadMore()`
that awaits 100ms, on the critical-path worker.
## Measured
Full suite, both passes, same machine, two reps each: **~114s -> ~74s
(-35%)**. These runs are without coverage, so change 4 is not reflected.
2456 tests pass on both sides; `ruff format`, `ruff check` and `ty` are
clean.
## Not in this PR
- `from crawlee.crawlers import HttpCrawler` costs 3.51s, of which 1.28s
is scikit-learn and scipy, imported eagerly by
`_adaptive_playwright._rendering_type_predictor` (behind the optional
`adaptive-crawler` extra). Every worker pays it, as does every
`crawlee[all]` cold start. Not a one-liner -
`RenderingTypePredictorState` needs `LogisticRegression` at
class-creation time. Worth its own issue.
- 14 of the 15 matrix jobs compute coverage that is never uploaded;
skipping it needs a change in `apify/workflows`.
- `tests_concurrency: "8"` may be 2x oversubscription on 4-vCPU runners;
worth one run with `nproc` echoed and an `-n4` comparison.
- `test_http_2` still hits the real internet - it needs an HTTP/2
endpoint the local uvicorn server does not speak.
*✍️ Drafted by Claude Code*1 parent 24a4bc5 commit 1d4007e
3 files changed
Lines changed: 35 additions & 20 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
263 | 263 | | |
264 | 264 | | |
265 | 265 | | |
266 | | - | |
| 266 | + | |
267 | 267 | | |
268 | 268 | | |
269 | 269 | | |
| |||
339 | 339 | | |
340 | 340 | | |
341 | 341 | | |
342 | | - | |
343 | 342 | | |
344 | 343 | | |
345 | 344 | | |
| |||
349 | 348 | | |
350 | 349 | | |
351 | 350 | | |
| 351 | + | |
| 352 | + | |
352 | 353 | | |
353 | 354 | | |
354 | | - | |
355 | 355 | | |
356 | 356 | | |
357 | 357 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
103 | 103 | | |
104 | 104 | | |
105 | 105 | | |
106 | | - | |
107 | | - | |
| 106 | + | |
| 107 | + | |
108 | 108 | | |
109 | 109 | | |
110 | 110 | | |
111 | 111 | | |
112 | | - | |
| 112 | + | |
113 | 113 | | |
114 | 114 | | |
115 | | - | |
| 115 | + | |
116 | 116 | | |
117 | 117 | | |
118 | 118 | | |
119 | 119 | | |
120 | 120 | | |
121 | | - | |
122 | | - | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
123 | 137 | | |
124 | 138 | | |
125 | 139 | | |
| |||
139 | 153 | | |
140 | 154 | | |
141 | 155 | | |
142 | | - | |
143 | | - | |
| 156 | + | |
| 157 | + | |
144 | 158 | | |
145 | 159 | | |
146 | 160 | | |
147 | | - | |
| 161 | + | |
148 | 162 | | |
149 | | - | |
| 163 | + | |
150 | 164 | | |
151 | | - | |
| 165 | + | |
152 | 166 | | |
153 | 167 | | |
154 | 168 | | |
155 | 169 | | |
156 | 170 | | |
157 | 171 | | |
158 | 172 | | |
159 | | - | |
| 173 | + | |
160 | 174 | | |
161 | 175 | | |
162 | 176 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
23 | | - | |
24 | | - | |
25 | | - | |
26 | | - | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
27 | 27 | | |
28 | | - | |
| 28 | + | |
| 29 | + | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| |||
0 commit comments