Skip to content

Commit 7ea8e71

Browse files
committed
performance jobSize tests
1 parent e34a66e commit 7ea8e71

7 files changed

Lines changed: 126 additions & 43 deletions

File tree

‎PARALLEL_LPT_POC_NOTES.md‎

Lines changed: 117 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
Branch: `poc/parallel-lpt-buckets` (local only, **not committed** — no tests yet)
44

5-
Opt-in flag: `--experimental-runner`. Without it nothing changes.
5+
Opt-in flag: `--lpt`. Without it nothing changes.
66

77
Source: https://github.com/rectorphp/rector-src/issues/8494#issuecomment-5664864017 (@samsonasik)
88

@@ -23,8 +23,8 @@ Faithful to 1 + 3, with one deliberate deviation on 2.
2323
| | StructArmed | this PoC |
2424
| --- | --- | --- |
2525
| bucketing | LPT by filesize | LPT by filesize (same greedy) |
26-
| assignment | static, 1 bucket per worker | static, 1 bucket per worker |
27-
| dispatch | whole bucket in one shot | bucket sub-chunked into 16-file jobs, **only to its own worker** |
26+
| assignment | static, 1 bucket per worker | static, **plus stealing once a worker runs dry** |
27+
| dispatch | whole bucket in one shot | bucket sub-chunked into `jobSize` jobs |
2828
| worker lifetime | one process per bucket, never respawned | one process per bucket, never respawned |
2929
| transport | `proc_open` + stdout polling | unchanged (ReactPHP + NDJSON over TCP) |
3030

@@ -44,8 +44,8 @@ Sub-chunking inside a statically-assigned bucket keeps the heartbeat while prese
4444
win: **the worker process is never recycled**, so its warm in-memory caches (PHPStan reflection,
4545
parsed AST, node scope resolver) survive across the whole bucket.
4646

47-
Round-trip cost of sub-chunking: ~820 extra NDJSON round-trips on a 13 k file project spread over
48-
16 workers — sub-second, negligible next to a container boot.
47+
The round-trip cost of sub-chunking turned out to be far cheaper than expected — small chunks win
48+
outright, see the `jobSize` section below.
4949

5050
### The other half of the change: no forced respawn
5151

@@ -67,8 +67,8 @@ New, all isolated under `src/Parallel/Experimental/`:
6767

6868
Existing files, minimal edits only:
6969

70-
- `src/Configuration/Option.php` — `EXPERIMENTAL_RUNNER` constant
71-
- `src/Console/ProcessConfigureDecorator.php` — registers `--experimental-runner`
70+
- `src/Configuration/Option.php` — `LPT` constant
71+
- `src/Console/ProcessConfigureDecorator.php` — registers `--lpt`
7272
- `src/Application/ApplicationFileProcessor.php` — two constructor deps + one branch in `runParallel()`
7373

7474
Nothing on the default path changed, so a run without the flag is byte-identical to `main`.
@@ -94,8 +94,11 @@ same 2 errors. The scheduler does not change the outcome.
9494

9595
"busy cores" = `(user + sys) / real`, i.e. how many of the 14 were actually kept busy.
9696

97-
Headline: best legacy 152.2 s → best experimental 113.2 s, **-26 %**. At equal worker count (both on
98-
the default 14) it is 152.2 s → 122.1 s, **-20 %**.
97+
Every row above is at `jobSize: 16`, which was the hardcoded chunk at the time. `jobSize` turned out
98+
to matter more than anything in this table — see its own section below.
99+
100+
Headline: best legacy **152.2 s** → best `--lpt` **90.7 s** (14 workers, `jobSize: 2`), i.e. **-40 %**.
101+
At the same worker count and the same `jobSize: 16`, it is 152.2 s → 113.4 s, **-25 %**.
99102

100103
### Work stealing is the load-bearing part, not LPT
101104

@@ -113,16 +116,75 @@ This also sharpens item 1 in "what is left": a timing-based cost model would mak
113116
buckets right, but stealing is what makes the schedule robust when the model is wrong. Both, not
114117
either.
115118

116-
### Worker count interacts with the scheduler — do not tune it standalone
119+
### Worker count: a red herring once the chunk size is right
120+
121+
Measured at `jobSize: 16`, dropping from 14 to 10 workers looked like a win (122.1 → 113.2), and
122+
legacy got worse at 10 (152.2 → 164.3). That suggested tuning the worker count, and that
123+
`CpuCoreCountProvider` counting logical cores was a problem on big.LITTLE hardware.
124+
125+
**That was an artefact of the coarse chunk.** At `jobSize: 4` the ordering flips: 14 workers give
126+
99.2 s, 12 workers give 107.6 s. With fine-grained stealing an efficiency core simply takes fewer
127+
chunks and contributes instead of straggling, so there is nothing left to compensate for by starving
128+
the pool.
129+
130+
Conclusion: fix the granularity and the worker-count question mostly disappears. `hw.ncpu` is a fine
131+
default. Do not tune the worker count before tuning `jobSize` — you will tune against a symptom.
132+
133+
### jobSize: was inert, now it is the most important knob
134+
135+
Originally this PoC hardcoded the chunk at 16 files and let `jobSize` feed only the worker-count
136+
formula `min(ceil(files / jobSize), cpuCores, maxProcesses)`. That makes it inert on any real
137+
project: once `ceil(files / jobSize) >= cpuCores` — from ~224 files at 14 cores and `jobSize: 16` —
138+
it stops changing anything. On a 13 k file project it did nothing below `jobSize` ~936.
139+
140+
`jobSize` is now the chunk a worker is handed per request (it still implies at least one whole job
141+
per worker). It therefore controls three things at once: **the granularity at which an idle worker
142+
can steal work**, the heartbeat that advances the progress bar, and the window the per-job timeout
143+
measures.
144+
145+
14 workers, LPT + stealing, `vendor/`. Run-to-run spread across batches is ~5 %, so the two figures
146+
for `jobSize: 4` are the same measurement taken twice.
117147

118-
Dropping from 14 to 10 workers **helps** the bucket scheduler (122.1 → 113.2) and **hurts** legacy
119-
(152.2 → 164.3). There is no worker count that is right independent of how work is handed out, so
120-
`CpuCoreCountProvider` returning logical cores is not a standalone bug to fix — it only looks like
121-
one under the bucket scheduler.
148+
| jobSize | wall | diffs | |
149+
| --: | --: | --: | --- |
150+
| 1 | 98.8 s | 3 338 | round-trip overhead takes over |
151+
| 2 | **90.7 / 91.4 s** | 3 338 | optimum |
152+
| 4 | 96.7 / 99.2 s | 3 338 | |
153+
| 8 | 104.6 s | 3 338 | |
154+
| 16 | 113.4 s | 3 338 | today's default |
155+
| 50 | 118.1 s | 3 338 | |
156+
| 150 | 127.5 s | **3 237** | **timed out, results truncated** |
122157

123-
`user` time also drops as workers drop (1089 → 931 under stealing). Careful with that number: work
124-
done on an efficiency core accumulates more CPU-seconds for the same result, so part of the "extra
125-
CPU" at 14 workers is just E-cores being slow, not waste.
158+
The curve is U-shaped with a clear floor at **`jobSize: 2`** — reproduced twice at 90.7 s and 91.4 s,
159+
so the ~6 % gap to `jobSize: 4` is outside run-to-run noise. Below that, at `jobSize: 1`, the round
160+
trips finally cost more than the balance they buy (and `user` time peaks at 1 173 s, the highest of
161+
any run).
162+
163+
This **inverts the recommendation in #8494**. There, raising `jobSize` was the only lever that
164+
reduced respawn churn, which is why 150 and 300 looked good. With respawns gone, a large `jobSize`
165+
buys nothing and only coarsens stealing — and stealing is what carries the win.
166+
167+
Caveat before generalising: the optimum is a balance between round-trip cost and stealing
168+
granularity, and both scale with the corpus. A heavier rule set makes each file cost more, which
169+
shifts the floor upward. Treat "2" as the answer for this corpus on this machine, and re-measure
170+
the 2-8 range elsewhere rather than hardcoding it.
171+
172+
### A large jobSize both fails the build and truncates the result
173+
174+
`jobSize: 150` hit `Child process timed out after 120 seconds`. That calls `handleErrorCallable` →
175+
`quitAll()`, and the run finished **reporting 3 237 diffs instead of 3 338** — 101 missing.
176+
177+
The build does go red: `ProcessCommand` returns `ExitCode::FAILURE` whenever there are system errors
178+
(`src/Console/Command/ProcessCommand.php:232`), which matches the `exit 1` reported in #8494. The
179+
extra hazard is that the *output* is silently short by 101 diffs on top of that, so anyone who
180+
tolerates or filters system errors in CI loses findings without a signal.
181+
182+
Worth reporting upstream separately: the message printed in that case is
183+
`Reached system errors count limit of 50, exiting...` even though only 4 errors occurred, because
184+
`ParallelFileProcessor::$handleErrorCallable` sets `$reachedSystemErrorsCountLimit = true`
185+
unconditionally on *any* error (`src/Parallel/Application/ParallelFileProcessor.php:134`). This is a
186+
pre-existing bug on `main`, faithfully copied into the experimental processor. It hides the real
187+
cause behind a wrong one — exactly the kind of thing that makes a timeout hard to diagnose.
126188

127189
### Memory is the consistent cost
128190

@@ -136,7 +198,7 @@ noise. At that size a worker never reaches the `MAX_CHUNKS_PER_WORKER × jobSize
136198
threshold, so both schedulers do exactly the same thing. **The gain appears precisely where the
137199
respawns do**, which supports reading #8494's table as respawn churn rather than round-trip overhead.
138200

139-
sellero at ~13 k files sits far past that threshold.
201+
Any project past ~2 000 files sits beyond that threshold on a 14-core machine.
140202

141203
### Why work stealing matters here
142204

@@ -147,8 +209,8 @@ bucketing throws that correction away. Stealing puts it back while keeping the n
147209
hence it beats both.
148210

149211
Consequence for benchmarking: **the split between static and stealing depends on the hardware.** On a
150-
homogeneous CI runner static buckets should lose much less. Worth measuring on sellero's CI too, not
151-
only on a laptop. `RECTOR_EXPERIMENTAL_NO_STEAL=1` turns stealing off for that A/B (temporary knob).
212+
homogeneous CI runner static buckets should lose much less. Worth measuring on the target CI too, not
213+
only on a laptop. `RECTOR_LPT_NO_STEAL=1` turns stealing off for that A/B (temporary knob).
152214

153215
### What is left on the table
154216

@@ -174,7 +236,7 @@ only on a laptop. `RECTOR_EXPERIMENTAL_NO_STEAL=1` turns stealing off for that A
174236
and full-JSON runs differ little, so the main loop is not saturated.
175237
6. **Worker boot.** 14 container boots, concurrent, ~1 s out of 113. Not worth chasing.
176238

177-
## How to A/B benchmark on sellero
239+
## How to A/B benchmark on a real project
178240

179241
The flag is opt-in, so baseline and experiment differ by one argument — no branch switching:
180242

@@ -183,22 +245,28 @@ vendor/bin/rector process --dry-run --clear-cache --no-progress-bar
183245
```
184246

185247
```bash
186-
vendor/bin/rector process --dry-run --clear-cache --no-progress-bar --experimental-runner
248+
vendor/bin/rector process --dry-run --clear-cache --no-progress-bar --lpt
187249
```
188250

189251
Run them sequentially, never in parallel, and keep `--clear-cache` on both (or off on both).
190252

191253
Suggested matrix:
192254

193-
| run | flag | `jobSize` |
194-
| --- | --- | --- |
195-
| baseline | — | current |
196-
| baseline tuned | — | 150 |
197-
| experimental | `--experimental-runner` | 16 |
198-
| experimental, fewer workers | `--experimental-runner` | 100 |
255+
| run | flag | `jobSize` | why |
256+
| --- | --- | --: | --- |
257+
| baseline | — | 16 | today's default |
258+
| baseline tuned | — | 150 | what #8494 recommends raising it to |
259+
| lpt | `--lpt` | 16 | same knob as the default, scheduler is the only variable |
260+
| lpt, fine chunks | `--lpt` | 4 | fastest here; expected best |
261+
| lpt, coarse chunks | `--lpt` | 150 | expected worst, and watch for the timeout |
262+
263+
**`jobSize` means something different under `--lpt`.** On the default path it is the chunk any worker
264+
may pull from a shared pool. Under `--lpt` it is the chunk handed to a worker from its own bucket,
265+
which makes it the granularity at which an idle worker can steal from a busy one — the setting that
266+
turned out to matter most. It still implies at least one whole job per worker, so it also caps the
267+
worker count on small file sets, exactly as before.
199268

200-
`jobSize` keeps its old meaning under the flag — it is the number of files that makes a worker worth
201-
starting, so it still controls the worker count; it no longer controls how work is split.
269+
Do not carry a tuned `jobSize` across the two paths: the optimum moves in opposite directions.
202270

203271
Watch, besides wall time:
204272

@@ -254,8 +322,23 @@ Nothing below is covered by a test yet — the user asked for a PoC first.
254322

255323
### Cleanup before this could become a PR
256324

257-
- [ ] drop the `RECTOR_PARALLEL_SCHEDULER` env var, or promote it to a real `Option`
258-
- [ ] `MAX_CHUNKS_PER_WORKER` becomes dead in bucket mode — decide its fate
259-
- [ ] revisit the `jobSize` default (#8494 argues 16 is too low); with warm workers the optimum
260-
likely moves again — re-measure before changing it
325+
- [ ] **Give the bucket path its own chunk-size default, separate from `PARALLEL_JOB_SIZE`.**
326+
Right now `--lpt` reuses the shared `jobSize`, whose default is 16
327+
(`src/Config/RectorConfig.php:137` and `src/Configuration/RectorConfigBuilder.php:106`),
328+
untouched by this PoC. Two consequences:
329+
- `--lpt` alone gives 113.4 s, not the 90.7 s headline. Reaching the best number needs an
330+
explicit `withParallel(jobSize: 2)`.
331+
- The optimum moves in **opposite directions** on the two paths — down to ~2 under `--lpt`,
332+
up towards 100-300 on the default path per #8494 — so one shared default cannot serve both,
333+
and lowering the shared one would hurt every run without the flag.
334+
335+
A dedicated default for the bucket path lets each side sit at its own optimum. Pick it by
336+
measuring on more than one corpus first: "2" comes from a single repo on a single machine, and
337+
a heavier rule set raises the per-file cost and shifts the floor upward.
338+
- [ ] drop the `--lpt` flag and `RECTOR_LPT_NO_STEAL` env var, or promote them to real options
339+
- [ ] `MAX_CHUNKS_PER_WORKER` is dead in bucket mode — decide its fate (see the memory-driven
340+
recycle item above, which would give it a real job again)
341+
- [ ] fold `ExperimentalParallelFileProcessor` back into `ParallelFileProcessor` — it is a near-copy,
342+
kept separate only so the default path stays provably untouched, and it will trip the
343+
duplicate-code gate
261344
- [ ] `composer check-cs`, `composer phpstan`, `vendor/bin/phpunit`

‎src/Application/ApplicationFileProcessor.php‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,7 @@ public function __construct(
5353
private readonly ArrayParametersMerger $arrayParametersMerger,
5454
private readonly MissConfigurationReporter $missConfigurationReporter,
5555
private readonly UsedSkipCollector $usedSkipCollector,
56-
// @experimental, see --experimental-runner
56+
// @experimental, see --lpt
5757
private readonly LptScheduleFactory $lptScheduleFactory,
5858
private readonly ExperimentalParallelFileProcessor $experimentalParallelFileProcessor,
5959
) {
@@ -271,8 +271,8 @@ private function runParallel(
271271
Option::PARALLEL_MAX_NUMBER_OF_PROCESSES
272272
);
273273

274-
// @experimental opt-in, see --experimental-runner
275-
if ($input->hasOption(Option::EXPERIMENTAL_RUNNER) && (bool) $input->getOption(Option::EXPERIMENTAL_RUNNER)) {
274+
// @experimental opt-in, see --lpt
275+
if ($input->hasOption(Option::LPT) && (bool) $input->getOption(Option::LPT)) {
276276
$bucketSchedule = $this->lptScheduleFactory->create(
277277
$cpuCores,
278278
$jobSize,

‎src/Configuration/Option.php‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -189,7 +189,7 @@ final class Option
189189
* @see \Rector\Parallel\Experimental\LptScheduleFactory
190190
* @var string
191191
*/
192-
public const string EXPERIMENTAL_RUNNER = 'experimental-runner';
192+
public const string LPT = 'lpt';
193193

194194
public const string MEMORY_LIMIT = 'memory-limit';
195195

‎src/Console/ProcessConfigureDecorator.php‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,7 @@ public static function decorate(Command $command): void
100100
$command->addOption(Option::PARALLEL_IDENTIFIER, null, InputOption::VALUE_REQUIRED);
101101

102102
$command->addOption(
103-
Option::EXPERIMENTAL_RUNNER,
103+
Option::LPT,
104104
null,
105105
InputOption::VALUE_NONE,
106106
'[EXPERIMENTAL] Balance files across parallel workers by size up front and keep every worker alive for its whole share.'

‎src/Parallel/Experimental/ExperimentalParallelFileProcessor.php‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@
3131

3232
/**
3333
* @experimental Alternative to @see \Rector\Parallel\Application\ParallelFileProcessor, used by
34-
* "--experimental-runner".
34+
* "--lpt".
3535
*
3636
* Two differences, both on purpose:
3737
*
@@ -311,7 +311,7 @@ private function popJobChunk(
311311
return array_pop($jobsPerWorker[$bucketKey]);
312312
}
313313

314-
if (getenv('RECTOR_EXPERIMENTAL_NO_STEAL') !== false) {
314+
if (getenv('RECTOR_LPT_NO_STEAL') !== false) {
315315
return null;
316316
}
317317

‎src/Parallel/Experimental/LptScheduleFactory.php‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
use Webmozart\Assert\Assert;
99

1010
/**
11-
* @experimental Alternative to @see \Rector\Parallel\ScheduleFactory, used by "--experimental-runner".
11+
* @experimental Alternative to @see \Rector\Parallel\ScheduleFactory, used by "--lpt".
1212
*
1313
* The default scheduler cuts the file list into jobSize chunks and lets any worker pull any chunk. This
1414
* one splits the files into one fixed bucket per worker, balanced by total byte size, so a worker is

‎src/Parallel/Experimental/ValueObject/BucketSchedule.php‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66

77
/**
88
* @experimental Filled by @see \Rector\Parallel\Experimental\LptScheduleFactory, consumed only by the
9-
* "--experimental-runner" parallel run.
9+
* "--lpt" parallel run.
1010
*
1111
* Unlike @see \Rector\Parallel\ValueObject\Schedule, jobs are not a shared pool: every worker owns its
1212
* own queue, decided up front.

0 commit comments

Comments
 (0)