Skip to content

Commit 96d15b2

Browse files
rustyconoverclaude
andcommitted
tests: start every process with one OpenBLAS thread
pyarrow imports numpy, and numpy's OpenBLAS starts one thread per core at import and busy-waits them: 2.7 CPU-seconds per process on a 48-core machine, for a suite that does no linear algebra. The conftest now sets OPENBLAS_NUM_THREADS=1 (unless already set) before xdist starts its workers, so every process of the session inherits it. On 48 cores the full suite's CPU time fell from 363 s to 208 s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 847dce3 commit 96d15b2

1 file changed

Lines changed: 12 additions & 4 deletions

File tree

‎tests/conftest.py‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,18 @@
1111
import uuid
1212
from typing import Any
1313

14-
import pyarrow as pa
15-
import pytest
16-
17-
from vgi import schema
14+
# pyarrow imports numpy, and numpy's OpenBLAS starts one thread per core at
15+
# import and busy-waits them: about 2.7 CPU-seconds per process on a 48-core
16+
# machine. No test does linear algebra. This module is loaded before xdist
17+
# starts its workers, so every process of the session inherits the setting:
18+
# xdist workers, fixture and example workers, HTTP servers, CLI runs. On
19+
# 48 cores it cut the suite's CPU time from 363 s to 208 s.
20+
os.environ.setdefault("OPENBLAS_NUM_THREADS", "1")
21+
22+
import pyarrow as pa # noqa: E402
23+
import pytest # noqa: E402
24+
25+
from vgi import schema # noqa: E402
1826

1927
# =============================================================================
2028
# Fixture-worker transport

0 commit comments

Comments
 (0)