Skip to content

Commit e34a66e

Browse files
committed
lpt experimental runner
1 parent 71a969f commit e34a66e

7 files changed

Lines changed: 774 additions & 7 deletions

File tree

‎PARALLEL_LPT_POC_NOTES.md‎

Lines changed: 261 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,261 @@
1+
# PoC: LPT bucket scheduling for parallel run
2+
3+
Branch: `poc/parallel-lpt-buckets` (local only, **not committed** — no tests yet)
4+
5+
Opt-in flag: `--experimental-runner`. Without it nothing changes.
6+
7+
Source: https://github.com/rectorphp/rector-src/issues/8494#issuecomment-5664864017 (@samsonasik)
8+
9+
## What samsonasik proposed
10+
11+
1. LPT (Longest Processing Time first): sort files largest-first by byte size, assign each file
12+
to the worker with the smallest total byte load so far.
13+
2. One worker per bucket, started once.
14+
3. Poll workers until no pending data. No timeout tweak, no job size tweak, no repeated
15+
close-open of workers.
16+
17+
Reference: `boundwize/structarmed` `src/Analyser/Parallel/ParallelAnalysisNodeExtractor.php`.
18+
19+
## What was actually implemented here
20+
21+
Faithful to 1 + 3, with one deliberate deviation on 2.
22+
23+
| | StructArmed | this PoC |
24+
| --- | --- | --- |
25+
| bucketing | LPT by filesize | LPT by filesize (same greedy) |
26+
| assignment | static, 1 bucket per worker | static, 1 bucket per worker |
27+
| dispatch | whole bucket in one shot | bucket sub-chunked into 16-file jobs, **only to its own worker** |
28+
| worker lifetime | one process per bucket, never respawned | one process per bucket, never respawned |
29+
| transport | `proc_open` + stdout polling | unchanged (ReactPHP + NDJSON over TCP) |
30+
31+
### Why the bucket is still sub-chunked
32+
33+
Rector's worker reports back once per job, and that single message does two things:
34+
35+
- advances the progress bar (`Bridge::FILES_COUNT`),
36+
- cancels/re-arms the per-job timeout (`ParallelProcess::request()`).
37+
38+
A true one-shot bucket would mean: progress bar frozen at 0 % until a worker finishes its whole
39+
bucket, and `parallelJobTimeout` (default 120 s) applied to the *entire* bucket — i.e. guaranteed
40+
`Child process timed out after 120 seconds` on any non-trivial project. That is precisely the
41+
failure mode reported in #8494.
42+
43+
Sub-chunking inside a statically-assigned bucket keeps the heartbeat while preserving the actual
44+
win: **the worker process is never recycled**, so its warm in-memory caches (PHPStan reflection,
45+
parsed AST, node scope resolver) survive across the whole bucket.
46+
47+
Round-trip cost of sub-chunking: ~820 extra NDJSON round-trips on a 13 k file project spread over
48+
16 workers — sub-second, negligible next to a container boot.
49+
50+
### The other half of the change: no forced respawn
51+
52+
`ParallelFileProcessor::MAX_CHUNKS_PER_WORKER = 8` kills a worker after 8 chunks and spawns a fresh
53+
one. At `jobSize: 16` that is a full container boot + cold caches every 128 files. On 13 111 files /
54+
16 workers that is ~100 respawns. Strong suspicion this — not the chunk round-trips — is what makes
55+
`jobSize: 16` (218 s) three times slower than `jobSize: 300` (75 s) in the issue's table.
56+
57+
In bucket mode the respawn is disabled entirely. **This is the main thing to benchmark and the main
58+
risk** (see below).
59+
60+
## Files touched
61+
62+
New, all isolated under `src/Parallel/Experimental/`:
63+
64+
- `LptScheduleFactory.php` — LPT bucketing, heartbeat sub-chunking, worker-count math
65+
- `ExperimentalParallelFileProcessor.php` — static bucket→worker binding, no respawn
66+
- `ValueObject/BucketSchedule.php`
67+
68+
Existing files, minimal edits only:
69+
70+
- `src/Configuration/Option.php` — `EXPERIMENTAL_RUNNER` constant
71+
- `src/Console/ProcessConfigureDecorator.php` — registers `--experimental-runner`
72+
- `src/Application/ApplicationFileProcessor.php` — two constructor deps + one branch in `runParallel()`
73+
74+
Nothing on the default path changed, so a run without the flag is byte-identical to `main`.
75+
76+
## Measurements
77+
78+
Machine: **Apple M3 Max, 10 performance + 4 efficiency cores** (`hw.ncpu` = 14, which is what Rector
79+
uses as the worker count). Corpus `vendor/`, 5 746 files.
80+
`--dry-run --clear-cache --no-progress-bar --output-format=json`, runs strictly sequential.
81+
82+
| scheduler | workers | wall | user | sys | peak RSS | busy cores |
83+
| --- | --: | --: | --: | --: | --: | --: |
84+
| legacy | 14 | 152.2 s | 1043.8 s | 52.7 s | 751 MB | 7.2 |
85+
| legacy | 10 | 164.3 s | 1026.9 s | 37.1 s | 691 MB | 6.5 |
86+
| LPT, static buckets | 14 | 127.8 s | 1014.3 s | 39.6 s | 938 MB | 8.2 |
87+
| LPT, static buckets | 10 | 153.8 s | 961.6 s | 31.9 s | 1256 MB | 6.5 |
88+
| LPT + work stealing | 14 | 122.1 s | 1089.2 s | 52.3 s | 902 MB | 9.3 |
89+
| LPT + work stealing | 12 | 113.6 s | 999.1 s | 33.3 s | 1075 MB | 9.1 |
90+
| LPT + work stealing | 10 | **113.2 s** | 930.6 s | 31.5 s | 1156 MB | 8.5 |
91+
92+
All seven runs produce **3 338 identical diffs** (same sha over the file-sorted diff list) and the
93+
same 2 errors. The scheduler does not change the outcome.
94+
95+
"busy cores" = `(user + sys) / real`, i.e. how many of the 14 were actually kept busy.
96+
97+
Headline: best legacy 152.2 s → best experimental 113.2 s, **-26 %**. At equal worker count (both on
98+
the default 14) it is 152.2 s → 122.1 s, **-20 %**.
99+
100+
### Work stealing is the load-bearing part, not LPT
101+
102+
At 10 workers — all of them on performance cores, so the hardware is homogeneous — static buckets
103+
give 153.8 s and stealing gives 113.2 s. That is **-26 % from stealing alone**, on hardware where
104+
static assignment had no excuse.
105+
106+
So the E-core theory is at best a secondary effect. The primary one is that **byte size is a bad
107+
predictor of processing cost**: a bucket balanced perfectly by bytes still finishes at wildly
108+
different times. LPT bucketing on its own (samsonasik's proposal as written) lands at 153.8 s vs
109+
legacy's 164.3 s at the same worker count — a real but modest gain. Nearly all of the win comes from
110+
keeping the workers alive and letting idle ones take over work.
111+
112+
This also sharpens item 1 in "what is left": a timing-based cost model would make the initial
113+
buckets right, but stealing is what makes the schedule robust when the model is wrong. Both, not
114+
either.
115+
116+
### Worker count interacts with the scheduler — do not tune it standalone
117+
118+
Dropping from 14 to 10 workers **helps** the bucket scheduler (122.1 → 113.2) and **hurts** legacy
119+
(152.2 → 164.3). There is no worker count that is right independent of how work is handed out, so
120+
`CpuCoreCountProvider` returning logical cores is not a standalone bug to fix — it only looks like
121+
one under the bucket scheduler.
122+
123+
`user` time also drops as workers drop (1089 → 931 under stealing). Careful with that number: work
124+
done on an efficiency core accumulates more CPU-seconds for the same result, so part of the "extra
125+
CPU" at 14 workers is just E-cores being slow, not waste.
126+
127+
### Memory is the consistent cost
128+
129+
Peak RSS is higher in every experimental run: 691-751 MB legacy vs 902-1256 MB experimental, worst
130+
case **+82 %** (1256 MB vs 691 MB at 10 workers). It gets worse as workers get fewer, because each
131+
worker then holds more files and nothing ever releases. This is the direct price of removing the
132+
respawn and the main reason a memory-driven recycle is worth building (see below).
133+
134+
Control run on `rules/` (802 files, ~57 per worker): 6.17 s legacy vs 6.67 s experimental — pure
135+
noise. At that size a worker never reaches the `MAX_CHUNKS_PER_WORKER × jobSize = 128` respawn
136+
threshold, so both schedulers do exactly the same thing. **The gain appears precisely where the
137+
respawns do**, which supports reading #8494's table as respawn churn rather than round-trip overhead.
138+
139+
sellero at ~13 k files sits far past that threshold.
140+
141+
### Why work stealing matters here
142+
143+
Static LPT assumes every worker is equally fast. On a big.LITTLE CPU it is not: 4 of the 14 workers
144+
land on efficiency cores that are several times slower, so their bucket finishes long after the rest
145+
and the run waits for them. Legacy's shared job pool corrected for that automatically; static
146+
bucketing throws that correction away. Stealing puts it back while keeping the no-respawn win —
147+
hence it beats both.
148+
149+
Consequence for benchmarking: **the split between static and stealing depends on the hardware.** On a
150+
homogeneous CI runner static buckets should lose much less. Worth measuring on sellero's CI too, not
151+
only on a laptop. `RECTOR_EXPERIMENTAL_NO_STEAL=1` turns stealing off for that A/B (temporary knob).
152+
153+
### What is left on the table
154+
155+
~9.3 of 14 cores busy means roughly a third of the machine is still idle. Ranked by expected value:
156+
157+
1. **Memory-driven worker recycle.** Not a speed item — the one that decides whether this is shippable
158+
at all. Report `memory_get_peak_usage()` in the result payload and recycle a worker above a
159+
threshold, handing its remaining chunks to the replacement. Cheap with sub-chunked buckets,
160+
impossible with true one-shot buckets.
161+
2. **Cost model.** Byte size mispredicts badly (see above). Rector already writes a per-file cache
162+
entry; persisting the measured processing time there and using it as the LPT weight on the next
163+
run would make the first assignment right, with byte size as the cold-start fallback. Stealing
164+
already absorbs most of the error, so measure before assuming this is worth much on top.
165+
3. **Steal policy.** Currently: take the largest remaining chunk from whoever has the most left.
166+
Untuned. Stealing the *smallest* chunk near the end of a run avoids creating a fresh straggler;
167+
a smaller chunk size for the tail would do the same. Given stealing is worth 26 %, its policy is
168+
worth an experiment.
169+
4. **Cache write contention.** `sys` is 32-53 s, up to ~0.4 cores of kernel time; Rector writes one
170+
cache file per processed file, from every worker into one directory. Batching those writes per
171+
chunk is outside the scheduler and may be worth more than further scheduling work.
172+
5. **Round-trip idle.** A worker sits idle between finishing a chunk and being handed the next one by
173+
the single-threaded main loop. Cheap fix: a one-chunk prefetch buffer. Bounded upside — `--no-diffs`
174+
and full-JSON runs differ little, so the main loop is not saturated.
175+
6. **Worker boot.** 14 container boots, concurrent, ~1 s out of 113. Not worth chasing.
176+
177+
## How to A/B benchmark on sellero
178+
179+
The flag is opt-in, so baseline and experiment differ by one argument — no branch switching:
180+
181+
```bash
182+
vendor/bin/rector process --dry-run --clear-cache --no-progress-bar
183+
```
184+
185+
```bash
186+
vendor/bin/rector process --dry-run --clear-cache --no-progress-bar --experimental-runner
187+
```
188+
189+
Run them sequentially, never in parallel, and keep `--clear-cache` on both (or off on both).
190+
191+
Suggested matrix:
192+
193+
| run | flag | `jobSize` |
194+
| --- | --- | --- |
195+
| baseline | — | current |
196+
| baseline tuned | — | 150 |
197+
| experimental | `--experimental-runner` | 16 |
198+
| experimental, fewer workers | `--experimental-runner` | 100 |
199+
200+
`jobSize` keeps its old meaning under the flag — it is the number of files that makes a worker worth
201+
starting, so it still controls the worker count; it no longer controls how work is split.
202+
203+
Watch, besides wall time:
204+
205+
- peak RSS (`/usr/bin/time -l` on macOS reports the largest single child) — removing the respawn is a
206+
memory trade
207+
- any `Child process timed out after N seconds`
208+
- CPU % — a higher number at equal user time means better balance
209+
- diffs identical across runs (`--output-format=json`, sort by file, compare) — the correctness gate
210+
211+
## Open risks / what is worth testing later
212+
213+
Nothing below is covered by a test yet — the user asked for a PoC first.
214+
215+
### Correctness
216+
217+
- [ ] `ScheduleFactory`: every input file lands in exactly one bucket, exactly once (no loss, no dupes).
218+
Easy to get wrong with the greedy loop.
219+
- [ ] `ScheduleFactory`: duplicate file paths in the input list must not collapse (that is why the
220+
implementation sorts a list of tuples rather than using paths as array keys).
221+
- [ ] `ScheduleFactory`: numeric-looking paths (e.g. `/tmp/123`) must not be cast to int array keys.
222+
- [ ] `ScheduleFactory`: `$files === []` and `count($files) < $numberOfProcesses` → no empty buckets
223+
handed to `ParallelFileProcessor` (an empty job would trip `Assert::notEmpty()` in `WorkerCommand`).
224+
- [ ] `ScheduleFactory`: non-existent / unreadable file → size 0, no warning, no crash.
225+
- [ ] worker count math is unchanged vs. legacy: `min(ceil(count/jobSize), cpuCores, maxProcesses)`.
226+
- [ ] `ParallelFileProcessor`: a worker only ever receives files from its own bucket.
227+
- [ ] `ParallelFileProcessor`: all buckets drain; the run does not hang when one bucket is empty.
228+
- [ ] end-to-end: file diffs produced in bucket mode == legacy mode == non-parallel mode.
229+
230+
### Failure handling
231+
232+
- [ ] a worker that dies mid-bucket: its remaining sub-chunks are currently **lost** (legacy mode had
233+
the same hole, but respawn masked it). Decide: redistribute, or fail loudly.
234+
- [ ] `SYSTEM_ERROR_LIMIT` / `quitAll()` path still terminates cleanly with static buckets.
235+
- [ ] timeout still fires for a genuinely hung worker (heartbeat must not accidentally mask it).
236+
237+
### Memory (the big one)
238+
239+
- [ ] a worker now processes its whole bucket without recycling. On a 13 k file project that is
240+
~800 files in one process vs. 128 today. Measure peak RSS; consider a memory-driven respawn
241+
(`memory_get_peak_usage()` reported in the result payload → recycle above a threshold and hand
242+
the remaining sub-chunks to the fresh worker). This is *easy* with sub-chunked buckets and
243+
*impossible* with true one-shot buckets — a point in favour of the deviation above.
244+
245+
### Scheduling quality
246+
247+
- [ ] byte size is a proxy for work, not work itself. A 2 kB file hitting an expensive rule can cost
248+
more than a 50 kB one. Worth measuring per-worker finish times: if the spread is wide, LPT on
249+
bytes is not enough and work-stealing (idle worker pulls from the fattest remaining bucket)
250+
should be added. Ordering largest-first already helps, since the tail is fine-grained.
251+
- [ ] `filesize()` is called once per file in the main process (13 k stats). Measure; the file list
252+
may already be able to carry sizes.
253+
- [ ] does LPT interact badly with `--max-changes` (early break inside `processFiles`)?
254+
255+
### Cleanup before this could become a PR
256+
257+
- [ ] drop the `RECTOR_PARALLEL_SCHEDULER` env var, or promote it to a real `Option`
258+
- [ ] `MAX_CHUNKS_PER_WORKER` becomes dead in bucket mode — decide its fate
259+
- [ ] revisit the `jobSize` default (#8494 argues 16 is too low); with warm workers the optimum
260+
likely moves again — re-measure before changing it
261+
- [ ] `composer check-cs`, `composer phpstan`, `vendor/bin/phpunit`

‎src/Application/ApplicationFileProcessor.php‎

Lines changed: 30 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,8 @@
1414
use Rector\Parallel\Application\ParallelFileProcessor;
1515
use Rector\Parallel\CpuCoreCountProvider;
1616
use Rector\Parallel\Exception\ParallelShouldNotHappenException;
17+
use Rector\Parallel\Experimental\ExperimentalParallelFileProcessor;
18+
use Rector\Parallel\Experimental\LptScheduleFactory;
1719
use Rector\Parallel\ScheduleFactory;
1820
use Rector\PhpParser\Parser\ParserErrors;
1921
use Rector\Reporting\MissConfigurationReporter;
@@ -51,6 +53,9 @@ public function __construct(
5153
private readonly ArrayParametersMerger $arrayParametersMerger,
5254
private readonly MissConfigurationReporter $missConfigurationReporter,
5355
private readonly UsedSkipCollector $usedSkipCollector,
56+
// @experimental, see --experimental-runner
57+
private readonly LptScheduleFactory $lptScheduleFactory,
58+
private readonly ExperimentalParallelFileProcessor $experimentalParallelFileProcessor,
5459
) {
5560
}
5661

@@ -255,18 +260,36 @@ private function runParallel(
255260
InputInterface $input,
256261
callable $postFileCallback,
257262
): ProcessResult {
258-
$schedule = $this->scheduleFactory->create(
259-
$this->cpuCoreCountProvider->provide(),
260-
SimpleParameterProvider::provideIntParameter(Option::PARALLEL_JOB_SIZE),
261-
SimpleParameterProvider::provideIntParameter(Option::PARALLEL_MAX_NUMBER_OF_PROCESSES),
262-
$filePaths
263-
);
264-
265263
$mainScript = $this->resolveCalledRectorBinary();
266264
if ($mainScript === null) {
267265
throw new ParallelShouldNotHappenException('[parallel] Main script was not found');
268266
}
269267

268+
$cpuCores = $this->cpuCoreCountProvider->provide();
269+
$jobSize = SimpleParameterProvider::provideIntParameter(Option::PARALLEL_JOB_SIZE);
270+
$maxNumberOfProcesses = SimpleParameterProvider::provideIntParameter(
271+
Option::PARALLEL_MAX_NUMBER_OF_PROCESSES
272+
);
273+
274+
// @experimental opt-in, see --experimental-runner
275+
if ($input->hasOption(Option::EXPERIMENTAL_RUNNER) && (bool) $input->getOption(Option::EXPERIMENTAL_RUNNER)) {
276+
$bucketSchedule = $this->lptScheduleFactory->create(
277+
$cpuCores,
278+
$jobSize,
279+
$maxNumberOfProcesses,
280+
$filePaths
281+
);
282+
283+
return $this->experimentalParallelFileProcessor->process(
284+
$bucketSchedule,
285+
$mainScript,
286+
$postFileCallback,
287+
$input
288+
);
289+
}
290+
291+
$schedule = $this->scheduleFactory->create($cpuCores, $jobSize, $maxNumberOfProcesses, $filePaths);
292+
270293
// mimics see https://github.com/phpstan/phpstan-src/commit/9124c66dcc55a222e21b1717ba5f60771f7dda92#diff-387b8f04e0db7a06678eb52ce0c0d0aff73e0d7d8fc5df834d0a5fbec198e5daR139
271294
return $this->parallelFileProcessor->process($schedule, $mainScript, $postFileCallback, $input);
272295
}

‎src/Configuration/Option.php‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -184,6 +184,13 @@ final class Option
184184
*/
185185
public const string PARALLEL_JOB_TIMEOUT_IN_SECONDS = 'parallel-job-timeout-in-seconds';
186186

187+
/**
188+
* @experimental Opt-in to the LPT bucket scheduler
189+
* @see \Rector\Parallel\Experimental\LptScheduleFactory
190+
* @var string
191+
*/
192+
public const string EXPERIMENTAL_RUNNER = 'experimental-runner';
193+
187194
public const string MEMORY_LIMIT = 'memory-limit';
188195

189196
/**

‎src/Console/ProcessConfigureDecorator.php‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,13 @@ public static function decorate(Command $command): void
9999
$command->addOption(Option::PARALLEL_PORT, null, InputOption::VALUE_REQUIRED);
100100
$command->addOption(Option::PARALLEL_IDENTIFIER, null, InputOption::VALUE_REQUIRED);
101101

102+
$command->addOption(
103+
Option::EXPERIMENTAL_RUNNER,
104+
null,
105+
InputOption::VALUE_NONE,
106+
'[EXPERIMENTAL] Balance files across parallel workers by size up front and keep every worker alive for its whole share.'
107+
);
108+
102109
$command->addOption(Option::XDEBUG, null, InputOption::VALUE_NONE, 'Display xdebug output.');
103110

104111
$command->addOption(

0 commit comments

Comments
 (0)