Repository navigation
Commit 4ce2ec2
authored
Use dynamic CPU count for -j in vendor/example scripts and docs (pytorch#21455)
### Summary
Follow-up to pytorch#20436, which replaced the hardcoded `cmake --build -j`
parallelism in the general
build docs and `test/` scripts. This PR finishes the same job for the
vendor backend scripts,
example scripts, and their documentation — 65 sites across 37 files, all
mechanical:
-j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 ))
`nproc` on Linux, `sysctl -n hw.ncpu` on macOS (the mps and coreml
scripts are Apple-only), and the
arithmetic degrades to `-j1` if neither tool exists. "Core count + 1" is
the guidance already in
`docs/source/using-executorch-building-from-source.md`. The pinned
values being removed ranged from
`-j4` to `-j100`, including `-j64` in the Vulkan test scripts and
`-j100` in
`tools/cmake/preset/README.md`.
Two Python sites differ. `extension/llm/export/quantizer_lib.py` holds a
shell command inside a
user-facing error string, so it takes the same shell expression.
`backends/mlx/test/test_utils.py`
builds an argv list handed to `subprocess.run` with no shell, where a
shell expression would reach
`cmake` as a literal string, so it uses `f"-j{(os.cpu_count() or 1) +
1}"` instead.
Deliberately out of scope: `.ci/**` and `.github/workflows/**`, where
the runners are fixed-size and
the parallelism is a resource-tuning decision rather than a portability
problem (`cuda.yml` pins
`-j4`, likely to bound peak memory); the `-j4` in `backends/mlx`'s
READMEs and `run_all_tests.py`,
which is a test-worker count and not a build flag; and
`docs/source/archive/`. Happy to take the CI
files in a separate PR if you'd like them changed.
Review order: the three groups are independent — vendor backend scripts
under `backends/`, example
scripts and docs under `examples/`, then the two Python files, which are
the only sites that are not
a pure token swap.
Partial fix for pytorch#10887.
### Test plan
ExecuTorch does not build on my Windows host, so verification is static
and per-site:
- `bash -n` passes on all 20 modified shell scripts.
- Every edited command line was re-run with `cmake --build`/`make`
swapped for `echo`, confirming all
65 sites expand to a single valid integer flag (`-j17` on this 16-core
machine) with zero
expansion failures. This covers the markdown sites too, including the
two lines that begin with
`&& ` and the one with a `$ ` prompt prefix.
- `git diff` normalised on the `-j` token shows every removed line has a
matching added line, so
nothing outside the flag changed. Every changed markdown line contains a
`-j` token.
- Both Python files parse; the argv site renders `-j17`; the instruction
string was extracted via
`ast` and shell-expanded to confirm a user pasting it gets `-j17`.
- `black --check` reports both Python files unchanged. Line endings are
unchanged (still LF) and
`git diff --check` reports no whitespace errors.
- `lintrunner` was **not** run locally — it is not installed on this
Windows host and is unavailable
in my WSL environment, so CI lint is the gate for that. E501 is in the
repo's flake8 ignore list,
so the one long instruction string in `quantizer_lib.py` (already 165
chars before this change) is
not a new violation.
This PR was authored with AI assistance (Claude Code); the diff and
every verification step above
were reviewed by me.
cc @GregoryComer @digantdesai @cbilgin @JakeStevens @larryliu08201 parent 5cdcc02 commit 4ce2ec2
34 files changed
Lines changed: 66 additions & 59 deletions
File tree
- backends
- apple/coreml/scripts
- cadence
- runtime
- mediatek/scripts
- mlx/test
- samsung
- vulkan/test
- custom_ops
- scripts
- docs/source
- backends/vulkan/tutorials
- examples
- mediatek
- models
- llama
- phi-3-mini-lora
- phi-3-mini
- nxp
- portable
- custom_ops
- scripts
- qualcomm
- selective_build
- wasm
- xnnpack/quantization
- extension
- llm/export
- wasm
- tokenizers
- tools/cmake/preset
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
39 | | - | |
| 39 | + | |
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| |||
53 | 53 | | |
54 | 54 | | |
55 | 55 | | |
56 | | - | |
| 56 | + | |
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
58 | | - | |
| 58 | + | |
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
| |||
80 | 80 | | |
81 | 81 | | |
82 | 82 | | |
83 | | - | |
| 83 | + | |
84 | 84 | | |
85 | 85 | | |
86 | 86 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
54 | 54 | | |
55 | 55 | | |
56 | 56 | | |
57 | | - | |
| 57 | + | |
58 | 58 | | |
59 | 59 | | |
60 | 60 | | |
| |||
78 | 78 | | |
79 | 79 | | |
80 | 80 | | |
81 | | - | |
| 81 | + | |
82 | 82 | | |
83 | 83 | | |
84 | 84 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | | - | |
| 30 | + | |
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
49 | | - | |
| 49 | + | |
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
54 | 54 | | |
55 | 55 | | |
56 | 56 | | |
57 | | - | |
| 57 | + | |
58 | 58 | | |
59 | 59 | | |
60 | 60 | | |
| |||
78 | 78 | | |
79 | 79 | | |
80 | 80 | | |
81 | | - | |
| 81 | + | |
82 | 82 | | |
83 | 83 | | |
84 | 84 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
| 27 | + | |
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
43 | | - | |
| 43 | + | |
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
38 | | - | |
| 38 | + | |
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
587 | 587 | | |
588 | 588 | | |
589 | 589 | | |
590 | | - | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
591 | 598 | | |
592 | 599 | | |
593 | 600 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
67 | | - | |
| 67 | + | |
68 | 68 | | |
69 | 69 | | |
70 | 70 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
33 | | - | |
| 33 | + | |
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
39 | | - | |
| 39 | + | |
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| |||
65 | 65 | | |
66 | 66 | | |
67 | 67 | | |
68 | | - | |
| 68 | + | |
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
72 | 72 | | |
73 | | - | |
| 73 | + | |
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
| |||
0 commit comments