Skip to content

fix: exclude OpenBLAS 0.3.34 from the development environment - #7

Merged
SMI-Lab-Inha merged 1 commit into
devfrom
fix/modal-openblas-crash
Oct 14, 2026
Merged

SMI-Lab-Inha merged 1 commit into
devfrom
fix/modal-openblas-crash

Conversation

@SMI-Lab-Inha

Copy link
Copy Markdown
Owner

Summary

Resolves the 0.1.1 known issue: the access violation of the modal and stack_modal tests in openblas.dll.

The fault is in OpenBLAS 0.3.34, not in CableDyn. On an AMD Zen 4 workstation (Threadripper PRO 7965WX, Windows, conda-forge libopenblas 0.3.34 pthreads_h952cde7_2) the tests fail on every run. A standalone C program that calls only OpenBLAS reproduces it:

Call (OPENBLAS_NUM_THREADS=1) 0.3.34 0.3.33
dgemm n = 200, k = 128 ok ok
dgemm n = 200, k = 160 access violation ok
dgemm n = k = 1000 access violation ok
dsygv n = 297 (size of the test's taut-string reference) access violation ok

The fault offsets map to dgemm_incopy_ZEN / dgemm_oncopy_ZEN. The cause is OpenMathLib/OpenBLAS#6013 and #6021, fixed upstream after 0.3.34:

  • 0.3.34 raises the GEMM blocking to 512 on AMD Zen 4/5 CPUs that expose AVX-512.
  • The Zen and Haswell DGEMM kernels keep a stack buffer sized for an inner dimension of 128, so they overrun it.
  • This is reached in builds without the AVX-512 kernels (the conda-forge win-64 package) and, in any build, with a forced OPENBLAS_CORETYPE.

On CI the failure was intermittent because only some hosted runners expose AVX-512.

Scope of the fault:

  • The solver is not affected. Its LAPACK calls are banded (DGBTRF/DGBTRS, DPBTRF/DPBTRS, DSBGVX, DGBSVX) plus a 4 × 4 DSYEV. None of them passes DGEMM an inner dimension above 128: blocked DGBTRF uses the LAPACK block size, at most 64.
  • The test is. Only the dense DSYGV reference of test_modal reached the bug.
  • The release executables use reference LAPACK. Results are unchanged.

Changes:

  • environment.yml excludes openblas/libopenblas 0.3.34. It does not pin an exact version, so the next OpenBLAS release is admitted. CI builds its environment from this file. A dry-run solve (with the glibc/macOS virtual packages set for the foreign platforms) resolves to 0.3.33 on win-64, linux-64 and osx-arm64.
  • CHANGELOG.md: a Fixed entry under [Unreleased] that gives the cause.
  • DEVELOPMENT.md: a troubleshooting row. After conda env update -f environment.yml, configure a fresh build directory. Otherwise the existing build keeps its staged openblas.dll, whose copy is newer than the package file.

Tests

  • No new test. The existing modal and stack_modal tests are the regression tests: they failed with 0.3.34 and pass with 0.3.33, on the host-selected Zen kernel.
  • Full suite on Windows (gfortran 16.2, Release, OpenBLAS 0.3.33, AMD Zen 4): ctest --test-dir build --output-on-failure with OMP_NUM_THREADS=1, OPENBLAS_NUM_THREADS=1 and --parallel 16 gives 245/245, slow tier included.

Checklist

  • pre-commit run --all-files passes
  • ctest --test-dir build --output-on-failure passes
  • python -m pytest tests python/tests passes (if Python or validation records changed) — not applicable, no Python or validation records changed
  • VALIDATION.md updated if a validation result changed — not applicable, no result changed
  • CHANGELOG.md updated under [Unreleased] (heading added if missing) for user-visible changes
  • Documentation in doc/ updated for changed inputs, options, or outputs — not applicable, no inputs, options or outputs changed
  • Conventional Commit title (feat:, fix:, docs:, ...)
  • Commits are signed off (git commit -s) under the
    Developer Certificate of Origin

The modal and stack_modal tests stopped with an access violation inside
openblas.dll on some hosted CI runners, and on every run on an AMD Zen 4
workstation. The fault is in OpenBLAS 0.3.34, not in CableDyn: a plain
DGEMM with an inner dimension above 128 reproduces it with no CableDyn
code loaded, and 0.3.33 runs the same calls cleanly.

OpenBLAS 0.3.34 enlarges the GEMM blocking on AMD Zen 4/5 CPUs that
expose AVX-512, and its Zen and Haswell DGEMM kernels then overrun their
stack buffer. It is reached in builds without the AVX-512 kernels, such
as the conda-forge Windows package, and with a forced OPENBLAS_CORETYPE
(OpenMathLib/OpenBLAS#6013, #6021, fixed upstream after 0.3.34). The
dense DSYGV reference of test_modal reached it; the solver's banded
LAPACK calls never pass DGEMM an inner dimension above 128, and the
release executables use reference LAPACK, so results are unchanged. The
CI failures were intermittent because only some runners expose AVX-512.

Exclude 0.3.34 in environment.yml, which CI also uses, so that the next
OpenBLAS release is admitted, and record the cause in the changelog and
the troubleshooting table of DEVELOPMENT.md.

Signed-off-by: Jae Hoon Seo <jaehoon.seo@inha.ac.kr>
@SMI-Lab-Inha

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-14T07:09:49.243533Z 6a7bcd0 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@read-the-docs-community

Copy link
Copy Markdown

Documentation build overview

📚 CableDyn | 🛠️ Build #35135370 | 📁 Comparing 6a7bcd0 against latest (f27c5e3)

  🔍 Preview build  

2 files changed
± changelog.html
± development.html

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 6a7bcd0a7e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@SMI-Lab-Inha
SMI-Lab-Inha merged commit b5c647f into dev Oct 14, 2026
15 checks passed
@SMI-Lab-Inha
SMI-Lab-Inha deleted the fix/modal-openblas-crash branch October 14, 2026 08:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant