PILOT is not stealing bandwidth — it is fair-sharing a pie that sometimes cannot grow

[terrizoaguimor]

docs/tuning.md says of PILOT: "On disk-saturated hosts hint-only PILOT can be net negative
— measure on yours." I wanted to know what makes a host one of those, so I measured it instead
of reasoning about it. The answer came out to be a single device property, which is why I am
raising it here rather than sending a patch: if it holds up, the engine could decide this for
itself and the knob would not need an operator.

The thing I assumed, which was wrong

I started from the theory that speculative reads hurt demand reads disproportionately — that
prefetches starve the critical path and PILOT therefore needs admission control. That is not
what happens. One demand reader plus k speculative readers on the same NVMe, and the demand
stream gets almost exactly its fair 1/(1+k) share of whatever the device delivers at that
queue depth:

	demand GB/s	aggregate, same thread count	fair 1/N	ratio
1 demand + 1 spec	7.90	15.32	7.66	103.1%
1 demand + 2 spec	5.99	17.33	5.78	103.7%
1 demand + 4 spec	3.81	18.43	3.69	103.4%
1 demand + 8 spec	2.04	17.88	1.99	102.7%

No starvation and no queue pathology. A speculative read is just an ordinary reader.

What does decide it

One property: does the device return more total bytes when you ask for more concurrency?

Same benchmark, same block size, twice — once on the bare NVMe, once with the device capped to
300 MB/s through a cgroup v2 IOReadBandwidthMax, which is a crude stand-in for a saturated or
shared disk.

	bare NVMe	capped to 300 MB/s
aggregate, 1 → max threads	9.72 → 20.84 GB/s (+114%)	0.32 → 0.30 GB/s (flat)
demand throughput with 4 spec	9.33 → 3.81 (−59%)	0.32 → 0.06 (−81%)
demand p50 with 4 spec	2.0 → 4.9 ms	94.7 → 304 ms
extra bytes moved by the system	+90%	zero

That is the whole mechanism. On the scaling device the speculation costs the demand path 59%
of its throughput but the system moves 90% more bytes overall — and with the 71.6% router
recall your own docs report, most of those extra bytes are ones you were going to want. On the
capped device the demand path pays 81% and the system moves nothing extra, because there
was nothing extra to move. Same fair sharing, opposite verdict.

The idea

Not admission control, which is what I set out to build and no longer think is needed. A
startup probe: the engine already reads gigabytes of dense weights at load time. Read one
part of that at queue depth 1 and another at depth 4, compare the two rates, and let the ratio
pick PILOT's default. Devices that scale get the prefetch; devices that do not, do not. It
turns "measure on yours" into something nobody has to do by hand, and it costs nothing extra
because the reads happen anyway.

Caveats, including one that bothers me

Run-to-run variance is larger than I would like. Two passes of the same single-class
measurement, median-of-3 each, disagreed by 36% at 2 threads (11.24 vs 15.32 GB/s). The likely
cause is that the first pass ran immediately after dd-ing the 24 GB test file, and the drive
was still doing garbage collection — which would mean that pass is depressed, so "the device
scales" is if anything understated. But I have not proven that, and it means any comparison
across passes is soft. The numbers above are all within-pass comparisons for that reason, and
the fair-share ratio is consistent across four points spanning 2 to 9 threads, which is the
main reason I believe it.

Other limits: one device class (DO premium Intel, virtio — which incidentally reports
rotational=1); synthetic 19 MB O_DIRECT reads, not the engine itself; and a cgroup cap is not
the same thing as a genuinely slow disk. I did not run colibri end to end because I have no
GLM-scale model on hand.

Everything is reproducible: iobench.c from the tree for the single-class arms, a two-class
variant I wrote for the rest, fixed seeds, page cache dropped between arms, environment and
commit recorded. Happy to attach the scripts and raw output.

What I would like your opinion on

1. Does the two-regime picture match the hosts where you actually saw PILOT go net negative,
   or was that a different failure — cache thrash from prefetched experts evicting warm ones,
   say, rather than bandwidth?
2. Is a self-calibrating default something you want at all, or do you prefer knobs to stay
   explicit and operator-set?
3. If the probe is worth having, is there a natural hook in the dense-weight load, or would it
   need its own pass?