|
| 1 | +# PoC: LPT bucket scheduling for parallel run |
| 2 | + |
| 3 | +Branch: `poc/parallel-lpt-buckets` (local only, **not committed** — no tests yet) |
| 4 | + |
| 5 | +Opt-in flag: `--experimental-runner`. Without it nothing changes. |
| 6 | + |
| 7 | +Source: https://github.com/rectorphp/rector-src/issues/8494#issuecomment-5664864017 (@samsonasik) |
| 8 | + |
| 9 | +## What samsonasik proposed |
| 10 | + |
| 11 | +1. LPT (Longest Processing Time first): sort files largest-first by byte size, assign each file |
| 12 | + to the worker with the smallest total byte load so far. |
| 13 | +2. One worker per bucket, started once. |
| 14 | +3. Poll workers until no pending data. No timeout tweak, no job size tweak, no repeated |
| 15 | + close-open of workers. |
| 16 | + |
| 17 | +Reference: `boundwize/structarmed` `src/Analyser/Parallel/ParallelAnalysisNodeExtractor.php`. |
| 18 | + |
| 19 | +## What was actually implemented here |
| 20 | + |
| 21 | +Faithful to 1 + 3, with one deliberate deviation on 2. |
| 22 | + |
| 23 | +| | StructArmed | this PoC | |
| 24 | +| --- | --- | --- | |
| 25 | +| bucketing | LPT by filesize | LPT by filesize (same greedy) | |
| 26 | +| assignment | static, 1 bucket per worker | static, 1 bucket per worker | |
| 27 | +| dispatch | whole bucket in one shot | bucket sub-chunked into 16-file jobs, **only to its own worker** | |
| 28 | +| worker lifetime | one process per bucket, never respawned | one process per bucket, never respawned | |
| 29 | +| transport | `proc_open` + stdout polling | unchanged (ReactPHP + NDJSON over TCP) | |
| 30 | + |
| 31 | +### Why the bucket is still sub-chunked |
| 32 | + |
| 33 | +Rector's worker reports back once per job, and that single message does two things: |
| 34 | + |
| 35 | +- advances the progress bar (`Bridge::FILES_COUNT`), |
| 36 | +- cancels/re-arms the per-job timeout (`ParallelProcess::request()`). |
| 37 | + |
| 38 | +A true one-shot bucket would mean: progress bar frozen at 0 % until a worker finishes its whole |
| 39 | +bucket, and `parallelJobTimeout` (default 120 s) applied to the *entire* bucket — i.e. guaranteed |
| 40 | +`Child process timed out after 120 seconds` on any non-trivial project. That is precisely the |
| 41 | +failure mode reported in #8494. |
| 42 | + |
| 43 | +Sub-chunking inside a statically-assigned bucket keeps the heartbeat while preserving the actual |
| 44 | +win: **the worker process is never recycled**, so its warm in-memory caches (PHPStan reflection, |
| 45 | +parsed AST, node scope resolver) survive across the whole bucket. |
| 46 | + |
| 47 | +Round-trip cost of sub-chunking: ~820 extra NDJSON round-trips on a 13 k file project spread over |
| 48 | +16 workers — sub-second, negligible next to a container boot. |
| 49 | + |
| 50 | +### The other half of the change: no forced respawn |
| 51 | + |
| 52 | +`ParallelFileProcessor::MAX_CHUNKS_PER_WORKER = 8` kills a worker after 8 chunks and spawns a fresh |
| 53 | +one. At `jobSize: 16` that is a full container boot + cold caches every 128 files. On 13 111 files / |
| 54 | +16 workers that is ~100 respawns. Strong suspicion this — not the chunk round-trips — is what makes |
| 55 | +`jobSize: 16` (218 s) three times slower than `jobSize: 300` (75 s) in the issue's table. |
| 56 | + |
| 57 | +In bucket mode the respawn is disabled entirely. **This is the main thing to benchmark and the main |
| 58 | +risk** (see below). |
| 59 | + |
| 60 | +## Files touched |
| 61 | + |
| 62 | +New, all isolated under `src/Parallel/Experimental/`: |
| 63 | + |
| 64 | +- `LptScheduleFactory.php` — LPT bucketing, heartbeat sub-chunking, worker-count math |
| 65 | +- `ExperimentalParallelFileProcessor.php` — static bucket→worker binding, no respawn |
| 66 | +- `ValueObject/BucketSchedule.php` |
| 67 | + |
| 68 | +Existing files, minimal edits only: |
| 69 | + |
| 70 | +- `src/Configuration/Option.php` — `EXPERIMENTAL_RUNNER` constant |
| 71 | +- `src/Console/ProcessConfigureDecorator.php` — registers `--experimental-runner` |
| 72 | +- `src/Application/ApplicationFileProcessor.php` — two constructor deps + one branch in `runParallel()` |
| 73 | + |
| 74 | +Nothing on the default path changed, so a run without the flag is byte-identical to `main`. |
| 75 | + |
| 76 | +## Measurements |
| 77 | + |
| 78 | +Machine: **Apple M3 Max, 10 performance + 4 efficiency cores** (`hw.ncpu` = 14, which is what Rector |
| 79 | +uses as the worker count). Corpus `vendor/`, 5 746 files. |
| 80 | +`--dry-run --clear-cache --no-progress-bar --output-format=json`, runs strictly sequential. |
| 81 | + |
| 82 | +| scheduler | workers | wall | user | sys | peak RSS | busy cores | |
| 83 | +| --- | --: | --: | --: | --: | --: | --: | |
| 84 | +| legacy | 14 | 152.2 s | 1043.8 s | 52.7 s | 751 MB | 7.2 | |
| 85 | +| legacy | 10 | 164.3 s | 1026.9 s | 37.1 s | 691 MB | 6.5 | |
| 86 | +| LPT, static buckets | 14 | 127.8 s | 1014.3 s | 39.6 s | 938 MB | 8.2 | |
| 87 | +| LPT, static buckets | 10 | 153.8 s | 961.6 s | 31.9 s | 1256 MB | 6.5 | |
| 88 | +| LPT + work stealing | 14 | 122.1 s | 1089.2 s | 52.3 s | 902 MB | 9.3 | |
| 89 | +| LPT + work stealing | 12 | 113.6 s | 999.1 s | 33.3 s | 1075 MB | 9.1 | |
| 90 | +| LPT + work stealing | 10 | **113.2 s** | 930.6 s | 31.5 s | 1156 MB | 8.5 | |
| 91 | + |
| 92 | +All seven runs produce **3 338 identical diffs** (same sha over the file-sorted diff list) and the |
| 93 | +same 2 errors. The scheduler does not change the outcome. |
| 94 | + |
| 95 | +"busy cores" = `(user + sys) / real`, i.e. how many of the 14 were actually kept busy. |
| 96 | + |
| 97 | +Headline: best legacy 152.2 s → best experimental 113.2 s, **-26 %**. At equal worker count (both on |
| 98 | +the default 14) it is 152.2 s → 122.1 s, **-20 %**. |
| 99 | + |
| 100 | +### Work stealing is the load-bearing part, not LPT |
| 101 | + |
| 102 | +At 10 workers — all of them on performance cores, so the hardware is homogeneous — static buckets |
| 103 | +give 153.8 s and stealing gives 113.2 s. That is **-26 % from stealing alone**, on hardware where |
| 104 | +static assignment had no excuse. |
| 105 | + |
| 106 | +So the E-core theory is at best a secondary effect. The primary one is that **byte size is a bad |
| 107 | +predictor of processing cost**: a bucket balanced perfectly by bytes still finishes at wildly |
| 108 | +different times. LPT bucketing on its own (samsonasik's proposal as written) lands at 153.8 s vs |
| 109 | +legacy's 164.3 s at the same worker count — a real but modest gain. Nearly all of the win comes from |
| 110 | +keeping the workers alive and letting idle ones take over work. |
| 111 | + |
| 112 | +This also sharpens item 1 in "what is left": a timing-based cost model would make the initial |
| 113 | +buckets right, but stealing is what makes the schedule robust when the model is wrong. Both, not |
| 114 | +either. |
| 115 | + |
| 116 | +### Worker count interacts with the scheduler — do not tune it standalone |
| 117 | + |
| 118 | +Dropping from 14 to 10 workers **helps** the bucket scheduler (122.1 → 113.2) and **hurts** legacy |
| 119 | +(152.2 → 164.3). There is no worker count that is right independent of how work is handed out, so |
| 120 | +`CpuCoreCountProvider` returning logical cores is not a standalone bug to fix — it only looks like |
| 121 | +one under the bucket scheduler. |
| 122 | + |
| 123 | +`user` time also drops as workers drop (1089 → 931 under stealing). Careful with that number: work |
| 124 | +done on an efficiency core accumulates more CPU-seconds for the same result, so part of the "extra |
| 125 | +CPU" at 14 workers is just E-cores being slow, not waste. |
| 126 | + |
| 127 | +### Memory is the consistent cost |
| 128 | + |
| 129 | +Peak RSS is higher in every experimental run: 691-751 MB legacy vs 902-1256 MB experimental, worst |
| 130 | +case **+82 %** (1256 MB vs 691 MB at 10 workers). It gets worse as workers get fewer, because each |
| 131 | +worker then holds more files and nothing ever releases. This is the direct price of removing the |
| 132 | +respawn and the main reason a memory-driven recycle is worth building (see below). |
| 133 | + |
| 134 | +Control run on `rules/` (802 files, ~57 per worker): 6.17 s legacy vs 6.67 s experimental — pure |
| 135 | +noise. At that size a worker never reaches the `MAX_CHUNKS_PER_WORKER × jobSize = 128` respawn |
| 136 | +threshold, so both schedulers do exactly the same thing. **The gain appears precisely where the |
| 137 | +respawns do**, which supports reading #8494's table as respawn churn rather than round-trip overhead. |
| 138 | + |
| 139 | +sellero at ~13 k files sits far past that threshold. |
| 140 | + |
| 141 | +### Why work stealing matters here |
| 142 | + |
| 143 | +Static LPT assumes every worker is equally fast. On a big.LITTLE CPU it is not: 4 of the 14 workers |
| 144 | +land on efficiency cores that are several times slower, so their bucket finishes long after the rest |
| 145 | +and the run waits for them. Legacy's shared job pool corrected for that automatically; static |
| 146 | +bucketing throws that correction away. Stealing puts it back while keeping the no-respawn win — |
| 147 | +hence it beats both. |
| 148 | + |
| 149 | +Consequence for benchmarking: **the split between static and stealing depends on the hardware.** On a |
| 150 | +homogeneous CI runner static buckets should lose much less. Worth measuring on sellero's CI too, not |
| 151 | +only on a laptop. `RECTOR_EXPERIMENTAL_NO_STEAL=1` turns stealing off for that A/B (temporary knob). |
| 152 | + |
| 153 | +### What is left on the table |
| 154 | + |
| 155 | +~9.3 of 14 cores busy means roughly a third of the machine is still idle. Ranked by expected value: |
| 156 | + |
| 157 | +1. **Memory-driven worker recycle.** Not a speed item — the one that decides whether this is shippable |
| 158 | + at all. Report `memory_get_peak_usage()` in the result payload and recycle a worker above a |
| 159 | + threshold, handing its remaining chunks to the replacement. Cheap with sub-chunked buckets, |
| 160 | + impossible with true one-shot buckets. |
| 161 | +2. **Cost model.** Byte size mispredicts badly (see above). Rector already writes a per-file cache |
| 162 | + entry; persisting the measured processing time there and using it as the LPT weight on the next |
| 163 | + run would make the first assignment right, with byte size as the cold-start fallback. Stealing |
| 164 | + already absorbs most of the error, so measure before assuming this is worth much on top. |
| 165 | +3. **Steal policy.** Currently: take the largest remaining chunk from whoever has the most left. |
| 166 | + Untuned. Stealing the *smallest* chunk near the end of a run avoids creating a fresh straggler; |
| 167 | + a smaller chunk size for the tail would do the same. Given stealing is worth 26 %, its policy is |
| 168 | + worth an experiment. |
| 169 | +4. **Cache write contention.** `sys` is 32-53 s, up to ~0.4 cores of kernel time; Rector writes one |
| 170 | + cache file per processed file, from every worker into one directory. Batching those writes per |
| 171 | + chunk is outside the scheduler and may be worth more than further scheduling work. |
| 172 | +5. **Round-trip idle.** A worker sits idle between finishing a chunk and being handed the next one by |
| 173 | + the single-threaded main loop. Cheap fix: a one-chunk prefetch buffer. Bounded upside — `--no-diffs` |
| 174 | + and full-JSON runs differ little, so the main loop is not saturated. |
| 175 | +6. **Worker boot.** 14 container boots, concurrent, ~1 s out of 113. Not worth chasing. |
| 176 | + |
| 177 | +## How to A/B benchmark on sellero |
| 178 | + |
| 179 | +The flag is opt-in, so baseline and experiment differ by one argument — no branch switching: |
| 180 | + |
| 181 | +```bash |
| 182 | +vendor/bin/rector process --dry-run --clear-cache --no-progress-bar |
| 183 | +``` |
| 184 | + |
| 185 | +```bash |
| 186 | +vendor/bin/rector process --dry-run --clear-cache --no-progress-bar --experimental-runner |
| 187 | +``` |
| 188 | + |
| 189 | +Run them sequentially, never in parallel, and keep `--clear-cache` on both (or off on both). |
| 190 | + |
| 191 | +Suggested matrix: |
| 192 | + |
| 193 | +| run | flag | `jobSize` | |
| 194 | +| --- | --- | --- | |
| 195 | +| baseline | — | current | |
| 196 | +| baseline tuned | — | 150 | |
| 197 | +| experimental | `--experimental-runner` | 16 | |
| 198 | +| experimental, fewer workers | `--experimental-runner` | 100 | |
| 199 | + |
| 200 | +`jobSize` keeps its old meaning under the flag — it is the number of files that makes a worker worth |
| 201 | +starting, so it still controls the worker count; it no longer controls how work is split. |
| 202 | + |
| 203 | +Watch, besides wall time: |
| 204 | + |
| 205 | +- peak RSS (`/usr/bin/time -l` on macOS reports the largest single child) — removing the respawn is a |
| 206 | + memory trade |
| 207 | +- any `Child process timed out after N seconds` |
| 208 | +- CPU % — a higher number at equal user time means better balance |
| 209 | +- diffs identical across runs (`--output-format=json`, sort by file, compare) — the correctness gate |
| 210 | + |
| 211 | +## Open risks / what is worth testing later |
| 212 | + |
| 213 | +Nothing below is covered by a test yet — the user asked for a PoC first. |
| 214 | + |
| 215 | +### Correctness |
| 216 | + |
| 217 | +- [ ] `ScheduleFactory`: every input file lands in exactly one bucket, exactly once (no loss, no dupes). |
| 218 | + Easy to get wrong with the greedy loop. |
| 219 | +- [ ] `ScheduleFactory`: duplicate file paths in the input list must not collapse (that is why the |
| 220 | + implementation sorts a list of tuples rather than using paths as array keys). |
| 221 | +- [ ] `ScheduleFactory`: numeric-looking paths (e.g. `/tmp/123`) must not be cast to int array keys. |
| 222 | +- [ ] `ScheduleFactory`: `$files === []` and `count($files) < $numberOfProcesses` → no empty buckets |
| 223 | + handed to `ParallelFileProcessor` (an empty job would trip `Assert::notEmpty()` in `WorkerCommand`). |
| 224 | +- [ ] `ScheduleFactory`: non-existent / unreadable file → size 0, no warning, no crash. |
| 225 | +- [ ] worker count math is unchanged vs. legacy: `min(ceil(count/jobSize), cpuCores, maxProcesses)`. |
| 226 | +- [ ] `ParallelFileProcessor`: a worker only ever receives files from its own bucket. |
| 227 | +- [ ] `ParallelFileProcessor`: all buckets drain; the run does not hang when one bucket is empty. |
| 228 | +- [ ] end-to-end: file diffs produced in bucket mode == legacy mode == non-parallel mode. |
| 229 | + |
| 230 | +### Failure handling |
| 231 | + |
| 232 | +- [ ] a worker that dies mid-bucket: its remaining sub-chunks are currently **lost** (legacy mode had |
| 233 | + the same hole, but respawn masked it). Decide: redistribute, or fail loudly. |
| 234 | +- [ ] `SYSTEM_ERROR_LIMIT` / `quitAll()` path still terminates cleanly with static buckets. |
| 235 | +- [ ] timeout still fires for a genuinely hung worker (heartbeat must not accidentally mask it). |
| 236 | + |
| 237 | +### Memory (the big one) |
| 238 | + |
| 239 | +- [ ] a worker now processes its whole bucket without recycling. On a 13 k file project that is |
| 240 | + ~800 files in one process vs. 128 today. Measure peak RSS; consider a memory-driven respawn |
| 241 | + (`memory_get_peak_usage()` reported in the result payload → recycle above a threshold and hand |
| 242 | + the remaining sub-chunks to the fresh worker). This is *easy* with sub-chunked buckets and |
| 243 | + *impossible* with true one-shot buckets — a point in favour of the deviation above. |
| 244 | + |
| 245 | +### Scheduling quality |
| 246 | + |
| 247 | +- [ ] byte size is a proxy for work, not work itself. A 2 kB file hitting an expensive rule can cost |
| 248 | + more than a 50 kB one. Worth measuring per-worker finish times: if the spread is wide, LPT on |
| 249 | + bytes is not enough and work-stealing (idle worker pulls from the fattest remaining bucket) |
| 250 | + should be added. Ordering largest-first already helps, since the tail is fine-grained. |
| 251 | +- [ ] `filesize()` is called once per file in the main process (13 k stats). Measure; the file list |
| 252 | + may already be able to carry sizes. |
| 253 | +- [ ] does LPT interact badly with `--max-changes` (early break inside `processFiles`)? |
| 254 | + |
| 255 | +### Cleanup before this could become a PR |
| 256 | + |
| 257 | +- [ ] drop the `RECTOR_PARALLEL_SCHEDULER` env var, or promote it to a real `Option` |
| 258 | +- [ ] `MAX_CHUNKS_PER_WORKER` becomes dead in bucket mode — decide its fate |
| 259 | +- [ ] revisit the `jobSize` default (#8494 argues 16 is too low); with warm workers the optimum |
| 260 | + likely moves again — re-measure before changing it |
| 261 | +- [ ] `composer check-cs`, `composer phpstan`, `vendor/bin/phpunit` |
0 commit comments