Skip to content

Commit 4ce2ec2

Browse files
authored
Use dynamic CPU count for -j in vendor/example scripts and docs (pytorch#21455)
### Summary Follow-up to pytorch#20436, which replaced the hardcoded `cmake --build -j` parallelism in the general build docs and `test/` scripts. This PR finishes the same job for the vendor backend scripts, example scripts, and their documentation — 65 sites across 37 files, all mechanical: -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) `nproc` on Linux, `sysctl -n hw.ncpu` on macOS (the mps and coreml scripts are Apple-only), and the arithmetic degrades to `-j1` if neither tool exists. "Core count + 1" is the guidance already in `docs/source/using-executorch-building-from-source.md`. The pinned values being removed ranged from `-j4` to `-j100`, including `-j64` in the Vulkan test scripts and `-j100` in `tools/cmake/preset/README.md`. Two Python sites differ. `extension/llm/export/quantizer_lib.py` holds a shell command inside a user-facing error string, so it takes the same shell expression. `backends/mlx/test/test_utils.py` builds an argv list handed to `subprocess.run` with no shell, where a shell expression would reach `cmake` as a literal string, so it uses `f"-j{(os.cpu_count() or 1) + 1}"` instead. Deliberately out of scope: `.ci/**` and `.github/workflows/**`, where the runners are fixed-size and the parallelism is a resource-tuning decision rather than a portability problem (`cuda.yml` pins `-j4`, likely to bound peak memory); the `-j4` in `backends/mlx`'s READMEs and `run_all_tests.py`, which is a test-worker count and not a build flag; and `docs/source/archive/`. Happy to take the CI files in a separate PR if you'd like them changed. Review order: the three groups are independent — vendor backend scripts under `backends/`, example scripts and docs under `examples/`, then the two Python files, which are the only sites that are not a pure token swap. Partial fix for pytorch#10887. ### Test plan ExecuTorch does not build on my Windows host, so verification is static and per-site: - `bash -n` passes on all 20 modified shell scripts. - Every edited command line was re-run with `cmake --build`/`make` swapped for `echo`, confirming all 65 sites expand to a single valid integer flag (`-j17` on this 16-core machine) with zero expansion failures. This covers the markdown sites too, including the two lines that begin with `&& ` and the one with a `$ ` prompt prefix. - `git diff` normalised on the `-j` token shows every removed line has a matching added line, so nothing outside the flag changed. Every changed markdown line contains a `-j` token. - Both Python files parse; the argv site renders `-j17`; the instruction string was extracted via `ast` and shell-expanded to confirm a user pasting it gets `-j17`. - `black --check` reports both Python files unchanged. Line endings are unchanged (still LF) and `git diff --check` reports no whitespace errors. - `lintrunner` was **not** run locally — it is not installed on this Windows host and is unavailable in my WSL environment, so CI lint is the gate for that. E501 is in the repo's flake8 ignore list, so the one long instruction string in `quantizer_lib.py` (already 165 chars before this change) is not a new violation. This PR was authored with AI assistance (Claude Code); the diff and every verification step above were reviewed by me. cc @GregoryComer @digantdesai @cbilgin @JakeStevens @larryliu0820
1 parent 5cdcc02 commit 4ce2ec2

34 files changed

Lines changed: 66 additions & 59 deletions

File tree

‎backends/apple/coreml/scripts/build_tests.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ cmake "$EXECUTORCH_ROOT_PATH" -B"$CMAKE_EXECUTORCH_BUILD_DIR_PATH" \
3636
-DEXECUTORCH_BUILD_EXECUTOR_RUNNER=OFF \
3737
-DEXECUTORCH_BUILD_XNNPACK=OFF
3838

39-
cmake --build "$CMAKE_EXECUTORCH_BUILD_DIR_PATH" -j9 -t executorch
39+
cmake --build "$CMAKE_EXECUTORCH_BUILD_DIR_PATH" -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) -t executorch
4040

4141
# Build protobuf
4242
echo "ExecuTorch: Building libprotobuf-lite"
@@ -53,7 +53,7 @@ cmake "$PROTOBUF_DIR_PATH/cmake" -B"$CMAKE_PROTOBUF_BUILD_DIR_PATH" \
5353
-DCMAKE_MACOSX_BUNDLE=OFF \
5454
-DCMAKE_CXX_STANDARD=17
5555

56-
cmake --build "$CMAKE_PROTOBUF_BUILD_DIR_PATH" -j9 -t libprotobuf-lite
56+
cmake --build "$CMAKE_PROTOBUF_BUILD_DIR_PATH" -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) -t libprotobuf-lite
5757

5858
# Copy required libraries
5959
echo "ExecuTorch: Copying libraries"

‎backends/cadence/build_cadence_fusionG3.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ if $STEPWISE_BUILD; then
5555
-DFLATCC_ALLOW_WERROR=OFF \
5656
-Bcmake-out/backends/cadence \
5757
backends/cadence
58-
cmake --build cmake-out/backends/cadence -j8
58+
cmake --build cmake-out/backends/cadence -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 ))
5959
else
6060
echo "Building Cadence toolchain with ExecuTorch packages"
6161
cmake_prefix_path="${PWD}/cmake-out/lib/cmake/ExecuTorch;${PWD}/cmake-out/third-party/gflags"
@@ -80,7 +80,7 @@ else
8080
-DHAVE_FNMATCH_H=OFF \
8181
-DFLATCC_ALLOW_WERROR=OFF \
8282
-Bcmake-out
83-
cmake --build cmake-out --target install --config Release -j8
83+
cmake --build cmake-out --target install --config Release -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 ))
8484
fi
8585

8686
echo "Run simple model to verify cmake build"

‎backends/cadence/build_cadence_hifi4.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ if $STEPWISE_BUILD; then
5454
-DFLATCC_ALLOW_WERROR=OFF \
5555
-Bcmake-out/backends/cadence \
5656
backends/cadence
57-
cmake --build cmake-out/backends/cadence -j8
57+
cmake --build cmake-out/backends/cadence -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 ))
5858
else
5959
echo "Building Cadence toolchain with ExecuTorch packages"
6060
cmake_prefix_path="${PWD}/cmake-out/lib/cmake/ExecuTorch;${PWD}/cmake-out/third-party/gflags"
@@ -78,7 +78,7 @@ else
7878
-DHAVE_FNMATCH_H=OFF \
7979
-DFLATCC_ALLOW_WERROR=OFF \
8080
-Bcmake-out
81-
cmake --build cmake-out --target install --config Release -j8
81+
cmake --build cmake-out --target install --config Release -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 ))
8282
fi
8383

8484
echo "Run simple model to verify cmake build"

‎backends/cadence/build_cadence_runner.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ main() {
2727
-DEXECUTORCH_ENABLE_EVENT_TRACER=ON \
2828
-DEXECUTORCH_ENABLE_LOGGING=ON \
2929
-Bcmake-out .
30-
cmake --build cmake-out --target install --config Release -j16
30+
cmake --build cmake-out --target install --config Release -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 ))
3131

3232
local example_dir=backends/cadence
3333
local build_dir="cmake-out/${example_dir}"
@@ -46,7 +46,7 @@ main() {
4646
-DPYTHON_EXECUTABLE="$(which python3)" \
4747
-B"${build_dir}" \
4848
"${example_dir}"
49-
cmake --build "${build_dir}" --config Release -j16
49+
cmake --build "${build_dir}" --config Release -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 ))
5050

5151
local runner="${PWD}/${build_dir}/cadence_runner"
5252
if [[ ! -f "${runner}" ]]; then

‎backends/cadence/build_cadence_vision.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ if $STEPWISE_BUILD; then
5454
-DFLATCC_ALLOW_WERROR=OFF \
5555
-Bcmake-out/backends/cadence \
5656
backends/cadence
57-
cmake --build cmake-out/backends/cadence -j8
57+
cmake --build cmake-out/backends/cadence -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 ))
5858
else
5959
echo "Building Cadence toolchain with ExecuTorch packages"
6060
cmake_prefix_path="${PWD}/cmake-out/lib/cmake/ExecuTorch;${PWD}/cmake-out/third-party/gflags"
@@ -78,7 +78,7 @@ else
7878
-DHAVE_FNMATCH_H=OFF \
7979
-DFLATCC_ALLOW_WERROR=OFF \
8080
-Bcmake-out
81-
cmake --build cmake-out --target install --config Release -j8
81+
cmake --build cmake-out --target install --config Release -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 ))
8282
fi
8383

8484
echo "Run simple model to verify cmake build"

‎backends/cadence/runtime/executor_main.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ cmake_install_executorch_devtools_lib() {
2424
-DEXECUTORCH_ENABLE_EVENT_TRACER=ON \
2525
-DPYTHON_EXECUTABLE="$PYTHON_EXECUTABLE" \
2626
-Bcmake-out .
27-
cmake --build cmake-out -j9 --target install --config Release
27+
cmake --build cmake-out -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --target install --config Release
2828
}
2929

3030
test_cmake_devtools_example_runner() {
@@ -40,7 +40,7 @@ test_cmake_devtools_example_runner() {
4040
${example_dir}
4141

4242
echo "Building ${example_dir}"
43-
cmake --build ${build_dir} -j9 --config Release
43+
cmake --build ${build_dir} -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --config Release
4444

4545
echo 'Running devtools/example_runner'
4646
${build_dir}/example_runner --bundled_program_path="./CadenceDemoModel.bpte"

‎backends/mediatek/scripts/mtk_build.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ cmake -DCMAKE_INSTALL_PREFIX="${build_dir}" \
3535
-B"${build_dir}"
3636

3737
# Build the project
38-
cmake --build "${build_dir}" --target install --config Release -j5
38+
cmake --build "${build_dir}" --target install --config Release -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 ))
3939

4040
# Switch back to the original directory
4141
cd - > /dev/null

‎backends/mlx/test/test_utils.py‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -587,7 +587,14 @@ def rebuild_op_test_runner(verbose: bool = False) -> bool:
587587

588588
print(f"Rebuilding op_test_runner in {build_dir}...")
589589

590-
cmd = ["cmake", "--build", str(build_dir), "--target", "op_test_runner", "-j8"]
590+
cmd = [
591+
"cmake",
592+
"--build",
593+
str(build_dir),
594+
"--target",
595+
"op_test_runner",
596+
f"-j{(os.cpu_count() or 1) + 1}",
597+
]
591598

592599
if verbose:
593600
print(f"Running: {' '.join(cmd)}")

‎backends/samsung/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ cmake extension/android \
6464
-DCMAKE_INSTALL_PREFIX=cmake-android-out \
6565
-Bcmake-android-out/extension/android
6666

67-
cmake --build cmake-android-out/extension/android -j8
67+
cmake --build cmake-android-out/extension/android -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 ))
6868
```
6969

7070
## Examples

‎backends/vulkan/test/custom_ops/build_and_run.sh‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -30,13 +30,13 @@ configure_and_build_main() {
3030
-B$CMAKE_OUT_DIR
3131
fi
3232

33-
cmake --build $CMAKE_OUT_DIR -j16 --target install
33+
cmake --build $CMAKE_OUT_DIR -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --target install
3434
# -DCMAKE_CXX_FLAGS="-DVULKAN_DEBUG" \
3535
}
3636

3737
# Function to build main project only
3838
build_main() {
39-
cmake --build $CMAKE_OUT_DIR -j16 --target install
39+
cmake --build $CMAKE_OUT_DIR -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --target install
4040
}
4141

4242
# Function to configure and build tests
@@ -65,12 +65,12 @@ configure_and_build_tests() {
6565
-B$CMAKE_OUT_DIR/backends/vulkan/test/custom_ops
6666
fi
6767

68-
cmake --build $CMAKE_OUT_DIR/backends/vulkan/test/custom_ops -j16 --target all
68+
cmake --build $CMAKE_OUT_DIR/backends/vulkan/test/custom_ops -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --target all
6969

7070
}
7171

7272
build_tests() {
73-
cmake --build $CMAKE_OUT_DIR/backends/vulkan/test/custom_ops -j16 --target all
73+
cmake --build $CMAKE_OUT_DIR/backends/vulkan/test/custom_ops -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --target all
7474
}
7575

7676
# Function to rebuild both main and tests

0 commit comments

Comments
 (0)