Skip to content

Update OpenBLAS and raise NUM_THREADS on Windows - #287

Open
ndabas wants to merge 3 commits into
MacPython:mainfrom
ndabas:ndabas-openblas-overflow-fix-and-threads
Open

ndabas wants to merge 3 commits into
MacPython:mainfrom
ndabas:ndabas-openblas-overflow-fix-and-threads

Conversation

@ndabas

@ndabas ndabas commented Oct 28, 2026

Copy link
Copy Markdown
  • I updated the package version in pyproject.toml and made sure the first 3 numbers match ./openblas_commit.txt. If I did not update ./openblas_commit.txt, I incremented the wheel build number (i.e. 0.3.29.0.0 to 0.3.29.0.1)

Note: update ./openblas_commit.txt with cd OpenBLAS; git describe --tags --abbrev=8 > ../openblas_commit.txt

Summary

Update OpenBLAS to v0.3.34-497-g791b02a1

This picks up OpenMathLib/OpenBLAS#6105 ("Default allocator speedup"), which fixes a memory-corruption crash. If more buffers are in use at once than the main table holds (2 * NUM_THREADS * NUM_PARALLEL), blas_memory_alloc() falls back to an overflow area. Each overflow allocation then mapped a new buffer instead of reusing the slot's buffer. It also called blas_release_register() with no bound check, so after NUM_BUFFERS + NEW_BUFFERS mappings it wrote past the end of new_release_info. This was seen on Windows ARM64 with many concurrent BLAS callers. The same PR also fixes two overflow-area races.

I pinned the #6105 merge commit rather than develop HEAD to keep the bump focused. PRs merged after that are mostly Reference-LAPACK syncs. Let me know if you would prefer to move to HEAD instead.

Raise NUM_THREADS from 24 to 64 on Windows

This matches the Linux/macOS builds (tools/build_steps.sh already uses 64). With 24, the main buffer table has only 48 slots, so applications with more concurrent callers reach the overflow path much sooner. That is the path that was crashing. I know a higher NUM_THREADS was declined before over memory use (#97). Linux and macOS have used 64 for a while, and the main buffers are only mapped on first use.

Related: #283 (win_amd64 crash with concurrent dgemm + dgemv).

Note: #248 removes build_steps_win_arm64.bat and moves NUM_THREADS=24 into build_steps_windows.sh. Whichever PR merges second will need to keep 64.

@mattip

mattip commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

How is this related to #283, which was solved by #284?

@ndabas

ndabas commented Oct 28, 2026

Copy link
Copy Markdown
Author

Perhaps 'related' isn't the correct term here -- they're both Windows concurrency bugs, I was looking through previous issues to see if there was already a fix for this one and found #283. They're not actually related in terms of the code or fixes.

@ndabas
ndabas force-pushed the ndabas-openblas-overflow-fix-and-threads branch from ddda0e1 to 20bd2c8 Compare November 3, 2026 09:51
@mattip

mattip commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

I pinned the #6105 merge commit rather than develop HEAD to keep the bump focused. PRs merged after that are mostly Reference-LAPACK syncs. Let me know if you would prefer to move to HEAD instead.

Hmm. I think I would prefer to move to HEAD of develop, it might expose some issues before upstream releases the next version. I think it means deleting a windows patch, but CI will tell us ...

@ndabas

ndabas commented Nov 3, 2026

Copy link
Copy Markdown
Author

Okay, I have updated to develop HEAD -- there is a minor issue (will be fixed by OpenMathLib/OpenBLAS#6222) which needed an additional patch for Windows.

@ndabas

ndabas commented Nov 3, 2026

Copy link
Copy Markdown
Author

I had a look at the CI failures:

  • posix / manylinux_i686, 32 -- re-running should fix it, seems to have been a transient DNS issue.
  • posix / android_x86_64, 32/64 -- Gradle apparently can't handle the newer Java 25 on Ubuntu 26 images (which are being phased in as the new ubuntu-latest images.) We would either need to use an ubuntu-24.04 runner or install Java 21 explicitly.

@martin-frbg

Copy link
Copy Markdown

Sorry for moving OpenBLAS HEAD out from under you, but I suspect it would make sense to update to a HEAD that has both your fix and the one that exposed the xerbla error handler to callers again ? The pending PR for sbgemm is probably irrelevant and I hope to move on to the pending "AI-assisted" AppleM4 and RISVC PRs which shouldn't be relevant for Windows either

@ndabas
ndabas force-pushed the ndabas-openblas-overflow-fix-and-threads branch from 6d0b342 to 9c280fb Compare November 4, 2026 09:48
@ndabas

ndabas commented Nov 4, 2026

Copy link
Copy Markdown
Author

No worries, I have moved to 02f58544. This includes OpenMathLib/OpenBLAS#6222 (the gensymbol.pl comma), OpenMathLib/OpenBLAS#6220 (xerbla_ exported again) and OpenMathLib/OpenBLAS#6223 (SBGEMM performance).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants