setup.py: pass Python_EXECUTABLE to CMake from the build interpreter

[thebreadishard]

What

pyscf/lib/CMakeLists.txt locates the installed pyscf, when PYSCF_SOURCE_DIR is not given, by running the first python3 on PATH:

execute_process(COMMAND python3 -c "import pyscf; print(pyscf.__path__[0], end='')" OUTPUT_VARIABLE PYSCF_SOURCE_DIR)

That interpreter is not necessarily the one the build runs under. Two ways this goes wrong in practice:

• pip is called by its absolute path from an environment that is not activated (or the build runs on a system whose python3 has no pyscf). The system python3, started in the checkout directory, then imports the checkout's own pyscf/ namespace directory, and the configure step fails with

  -- Include pyscf source dir: /path/to/pyscf-forge/pyscf
  CMake Error: File /path/to/pyscf-forge/pyscf/lib/config.h.in does not exist.
  

• if that python3 has another pyscf installed, the extension is silently built against the wrong headers.

This PR makes setup.py resolve the pyscf location with sys.executable (from a neutral working directory) and pass -DPYSCF_SOURCE_DIR=... to CMake, unless CMAKE_CONFIGURE_ARGS already sets it. No change when the environment is activated; no change to the CMake files.

Test plan

All four in the same conda environment (Python 3.12, pyscf 2.14.0), pip install --no-build-isolation --no-deps ., BUILD_OCCRI=OFF, with PATH restricted so that python3 is the system interpreter without pyscf (cmake and gcc still reachable):

case	branch	PATH	result
A	upstream master	restricted	fails: Include pyscf source dir: /root/prs_cmake/pyscf, config.h.in does not exist
B	this branch	restricted	builds and installs; Include pyscf source dir: <site-packages>/pyscf
C	this branch	environment first	builds and installs, same directory as B
D	this branch	restricted, CMAKE_CONFIGURE_ARGS=-DPYSCF_SOURCE_DIR=... set explicitly	builds; the option appears once (no duplicate)

import pyscf.lno.lnoccsd works after B–D.

🤖 Generated with Claude Code

[jeanwsr]

I think we don't suppose the build works with wrong python env/wrong pip.

[thebreadishard]

Agreed that a wrong environment is not supported. The case here is narrower: the right pip (the environment's own, called by full path, or via a tool such as pipx/uv that does not activate the environment) building into the right environment, while a different python3 happens to be first on PATH. CMake then asks that other interpreter where pyscf lives, and from the checkout directory it answers with the checkout's own pyscf/ namespace directory, so the build stops with "config.h.in does not exist" although pyscf is installed correctly in the target environment. Passing -DPYSCF_SOURCE_DIR from sys.executable (the interpreter pip is running under, queried from a temporary directory) makes the build independent of PATH order; when CMAKE_CONFIGURE_ARGS already sets the variable nothing changes, and an activated environment behaves exactly as before (checked in the four-case QC in the description). Happy to narrow or drop it if you prefer documenting "activate the environment first" instead.

[MatthewRHermes]

I'm not a fan of this approach because having a slightly different version of python3 -c "import pyscf; print(pyscf.__path__[0], end='')" from CMakeLists.txt in setup.py creates more surface area for the same fundamental operation (finding PySCF) to go wrong in a variety of different ways for users who install PySCF-Forge in different ways, especially if in the future anything about that command has to change for any reason. It seems the actual root cause is python3 in CMakeLists.txt not matching sys.executable in setup.py. Is there no elegant way to fix it there?

[MatthewRHermes]

Comment composed by Claude Opus 5 following a conversation with me locally:

Following up on my earlier comment — I do think there is an elegant fix at the root, and it belongs in CMakeLists.txt rather than setup.py.

First, a data point on the root cause: the probe at pyscf/lib/CMakeLists.txt:99 is CWD-sensitive, not only PATH-sensitive. The forge checkout has no pyscf/__init__.py, so it is a namespace-package portion, and python3 -c puts the working directory at sys.path[0]. From the checkout root, an interpreter without pyscf therefore "finds" the checkout itself:

$ cd pyscf-forge
$ /usr/bin/python3 -c "import pyscf; print(pyscf.__path__[0], end='')"
/path/to/pyscf-forge/pyscf          # exit status 0, no error

$ cd /tmp && /usr/bin/python3 -c "import pyscf; print(pyscf.__path__[0], end='')"
ModuleNotFoundError: No module named 'pyscf'

Second, on why I would rather not duplicate the lookup in setup.py: the two spellings already disagree, on exactly the input that triggers this bug.

CMakeLists (pyscf.__path__[0])         -> /path/to/pyscf-forge/pyscf    wrong, silent
this PR    (os.path.dirname(__file__)) -> TypeError: ... not NoneType   namespace pkg: __file__ is None

In the PR as written that TypeError is swallowed — subprocess.run has no check and stderr is not surfaced — so pyscf_dir ends up empty, no -D is appended, and the build falls back to the original broken path. That is precisely the failure mode I had in mind when I said two versions of the same operation adds surface area.

Since the root cause is the hardcoded python3, I would suggest passing the interpreter rather than the answer:

find_package(Python COMPONENTS Interpreter REQUIRED)
if(NOT PYSCF_SOURCE_DIR)
  execute_process(
    COMMAND ${Python_EXECUTABLE} -c "import pyscf; print(pyscf.__path__[0], end='')"
    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
    OUTPUT_VARIABLE PYSCF_SOURCE_DIR
    RESULT_VARIABLE _pyscf_rc)
  if(NOT _pyscf_rc EQUAL 0 OR NOT EXISTS "${PYSCF_SOURCE_DIR}/lib/config.h.in")
    message(FATAL_ERROR
      "Could not locate an installed pyscf using ${Python_EXECUTABLE}. "
      "Pass -DPYSCF_SOURCE_DIR=<.../site-packages/pyscf>.")
  endif()
endif()

plus a single line in setup.py, next to the other configure args:

cmd.append(f'-DPython_EXECUTABLE={sys.executable}')

This keeps the "find pyscf" command in exactly one place and setup.py never learns how to locate pyscf, which addresses my original concern. Python_EXECUTABLE is a standard CMake variable (scikit-build and pybind11 key off the same one), so it helps other consumers as well. WORKING_DIRECTORY removes the CWD shadowing shown above, and the config.h.in existence check converts your second failure mode — a different pyscf first on PATH, silently building against the wrong headers — into an actionable error rather than a mislinked binary. That case matters more than the error message, because configure_file at lines 105-107 writes config.h into ${PYSCF_SOURCE_DIR}/lib/, i.e. into the installed pyscf.

One unrelated nit in the same block: option() at line 96 is the wrong command for a path. It declares a BOOL cache entry, defaulting to OFF. It should be:

set(PYSCF_SOURCE_DIR "" CACHE PATH "Pyscf source directory e.g. /usr/local/lib/python3.7/site-packages/pyscf")

Would you be willing to rework the PR along these lines?

[MatthewRHermes]

Please rework this so that it doesn't rely on copy-paste-modify of a line from CMakeLists.txt

[thebreadishard]

Reworked along your suggestion, thank you — it is the better fix. pyscf/lib/CMakeLists.txt now locates pyscf with the build interpreter (find_package(Python COMPONENTS Interpreter REQUIRED); setup.py passes -DPython_EXECUTABLE={sys.executable} next to the other configure arguments), runs the lookup from ${CMAKE_CURRENT_BINARY_DIR} instead of the checkout, and stops with a clear message when pyscf is not found or has no lib/config.h.in. PYSCF_SOURCE_DIR is a CACHE PATH instead of an option(). cmake_minimum_required goes from 3.5 to 3.12, which find_package(Python) needs; say so if that is a problem and I will guard it instead.

Configure step checked on Linux (CMake 4.2), from the checkout with /usr/bin/python3 (no pyscf) first on PATH: before, Include pyscf source dir: <checkout>/pyscf followed by the config.h.in does not exist error; now, Could not locate an installed pyscf using /usr/bin/python3. With -DPython_EXECUTABLE=<env>/bin/python: Include pyscf source dir: <env>/site-packages/pyscf, configuring done. The PR is a single commit on current master.

[MatthewRHermes]

Thanks, this is a better shape, but one last thing: please increase the cmake_minimum_required (VERSION to 3.15). See comment above for explanation.

[thebreadishard]

Thanks, done in 84230da: cmake_minimum_required (VERSION 3.15); a configure run picks the interpreter passed by setup.py and finds pyscf through it. The mode change on setup.py was not intended (a Windows checkout); the executable bit is restored.