Edge placement algorithms usually pick a node with an analytical latency model:
latency = propagation_delay + data_size / bandwidth
The model knows about distance and bandwidth, but nothing about how loaded the server is. Edge nodes are small. Once a node gets crowded, the queueing delay inside the server can exceed the network delay that edge placement was supposed to save. And since these algorithms are almost always evaluated in simulators built on the same model, the error never shows up.
Server load is not ignored in practice — it is just wired to a different job than placement latency. Autoscalers and load balancers read it as a threshold (scale out, or route away, when CPU crosses a line). Cost models read it as a roughly linear billing term. Queueing-aware placement does put a delay term in the objective, but from an analytical M/M/1-style model, not from measurement. In every one of these the load→latency relationship is either reacted to after the placement decision or assumed in closed form — never learned from a real workload and fed back into the placement ranking itself. This project targets exactly that seam, for tail-sensitive interactive workloads (gaming), where the p99 the analytical model omits is the number that actually decides whether the placement was any good.
This repo measures that error on a real (emulated) testbed: an edge-cloud topology built with Containerlab, running an actual Minecraft (Paper) server under strict CPU limits, with load generated by protocol-level bot clients (mineflayer). The end goal is to show the model mispredicts badly enough to make placement algorithms pick the wrong node, then replace it with a latency predictor trained on measured data. Current state is under Results: each node's capacity is now measured, and the load→latency curve has the flat-then-vertical shape the formula cannot express.
The same network path is measured at four depths, concurrently, under the same load — so the delay can be attributed to a layer instead of guessed at:
| Probe | Path it exercises |
|---|---|
| ICMP ping | kernel to kernel — network only. This is what the analytical model predicts. |
| TCP handshake | network + the server host's accept path |
| server status ping | network + the server process answering a request |
| in-game RTT | network + the server's single main game thread (chat echo, via mineflayer) |
The gap between the first and the last is model error. If ICMP stays flat while in-game RTT climbs, the extra delay is server-side queueing — the term the formula has no place for — and it is not the emulated link or the load generator. Alongside these, the server's own counters (MSPT, TPS, container CPU, player count) are sampled throughout, so the cause is recorded next to the effect. Details in docs/04-metrics.md.
The pieces already exist — separately. Server load is measured everywhere; it is just never assembled into a measured load→latency term inside the placement decision:
| System | Reads load as | Acts | Predicts latency? |
|---|---|---|---|
| Autoscaler | threshold (CPU > X) | after placement | no |
| Load balancer | relative (least-load) | per request, after placement | no |
| Cost / billing model | ~linear utilization term | accounting | no |
| Queueing-aware placement | analytical M/M/1 delay | at placement | modeled, not measured |
Two reasons the join is missing. First, the load→latency curve is nonlinear and
tail-heavy — delay grows like 1/(1 − utilization), flat until the node is
nearly saturated, then vertical — so it does not drop cleanly into a closed-form
placement objective, and the simulators used to evaluate placement inherit the
same load-blind model. Second, cloud nodes rarely saturate (you autoscale first),
so the coupling stayed invisible; edge nodes are small and capped and cannot
scale out, so queueing delay resurfaces exactly where placement is supposed to
help.
Novelty, stated narrowly (and defensibly): learn the load→latency curve empirically from a real interactive workload, then use it as the latency term that ranks candidate nodes — rather than reacting to load after placement, or assuming its shape in closed form. The claim is not "nobody uses load"; it is "nobody measures the load→latency relationship for a tail-sensitive interactive workload and puts it into the placement ranking."
The measurements below are the first evidence that the omitted term is not a correction but the dominant one: on a 0.5-core node, tick time is flat from 8 to 40 players (6.9 → 16.5 ms) and then quadruples by 60 (69.1 ms), with CPU pinned at its quota exactly where it turns. A closed form fitted to the flat part predicts nothing about the cliff.
bots/ bot.js load generator, probe.js latency probes
topology/ containerlab edge-cloud testbed + bring-up/verify scripts
controller/ collect.sh — the metric collection driver
results/ experiment notes and raw measurement streams
docs/ full documentation, written for a beginner
data/ measured dataset (planned)
client ──eth1── 10.0.1.0/30 ──── edge1 0.5 core 5.10 ms RTT
bots ──eth2── 10.0.2.0/30 ──── edge2 1 core 15.29 ms
probes ──eth3── 10.0.3.0/30 ──── edge3 2 cores 30.27 ms
──eth4── 10.0.4.0/30 ──── cloud 4 cores 40.29 ms
Four candidates, so the question is a ranking, not an A/B. Which IP the bots dial is the placement decision.
Capacity is structural (it needs a redeploy, so it lives in the topology);
distance is a runtime knob — ./netem.sh ladder gives the realistic tradeoff
where the closest node is the weakest, ./netem.sh flat puts every edge at the
same distance so capacity can be isolated from distance, with no redeploy.
Every number above is measured with ping, 0% loss, mdev < 0.1 ms — never the
configured value.
cd topology && ./prepare.sh && ./up.sh ladder # deploy, impair, verify
cd ../controller && ./warmup.sh # generate terrain, discard
&& ./capacity.sh # find each node's N_max
&& ./collect.sh --tag run2 # the measurement gridwarmup.sh is not optional. A freshly copied world makes a healthy node look
saturated, because the server is generating terrain rather than simulating
it — and it biases the weakest node hardest. Cold vs warm, same node and load:
p95 474 ms vs 60 ms.
up.sh refuses to hand over a testbed that does not verify — CPU caps read
from /sys/fs/cgroup/cpu.max, netem qdiscs present, and the RTT actually
measured with ping. Full procedure in docs/05-runbook.md;
every command explained in docs/03-commands.md.
Players a node holds before its tick time breaks the 50 ms budget that the
game's 20 Hz design imposes. This is the denominator that turns a raw player
count into utilization, the unit 1/(1 − utilization) is written in and the
only one comparable across nodes of different size.
| node | CPU | measured RTT | N_max | MSPT at N_max | first breach |
|---|---|---|---|---|---|
| edge1 | 0.5 core | 5.10 ms | 50 | 44.5 ms | 60 bots → 69.1 ms |
| edge2 | 1 core | 15.29 ms | 100 | 44.7 ms | none ≤ 100 |
| edge3 | 2 cores | 30.27 ms | > 100 | 11.1 ms | none ≤ 100 |
| cloud | 4 cores | 40.29 ms | > 100 | 9.3 ms | none ≤ 100 |
Capacity scales at roughly 100 players per core, and every node's tick time turns vertical exactly where its CPU reaches its own quota — flat, then a knee, then a cliff. edge1: 8 → 40 players moves MSPT only 6.9 → 16.5 ms; 40 → 60 takes it to 69.1 ms. ICMP RTT never moved across any of it.
edge3 and cloud never saturate: the 100-player ceiling is the testbed's
(max-players, and the load generator), not theirs.
Superseded. The run below is confounded by cold-world chunk generation — see the correction in results/sweep.md. The inversion is still expected, but its magnitude and the player count at which it appears are being re-measured on warm worlds. Treat the table as historical.
p95 in-game RTT (ms), edge at 5.1 ms network distance and 0.5 core, cloud at 40.1 ms and 4 cores. Every cell held its full player count with zero kicks.
| bots | 1 | 5 | 10 | 20 | 30 | 40 |
|---|---|---|---|---|---|---|
| edge p95 | 11 | 18 | 71 | 3753 | 3942 | 6248 |
| cloud p95 | 43 | 43 | 44 | 82 | 138 | 310 |
| model's pick | edge | edge | edge | edge | edge | edge |
| measured winner | edge | edge | cloud | cloud | cloud | cloud |
The analytical model ranks by distance alone, so it picks edge at every load, by a fixed 35 ms. From 10 players on it is picking the worse node — by 45× at 20 players. Meanwhile ICMP RTT on the same links never moved: 5.07–5.10 ms and 40.10 ms in every cell, saturated or idle. The cause sits next to the effect in the same data: edge CPU pins at its 0.5-core cap from 30 bots, and MSPT crosses the 50 ms tick budget between 30 and 40 — exactly where latency becomes seconds.
One run, one host, with capacities chosen rather than discovered. It is first evidence, not the analysis. Full table and caveats: docs/04-metrics.md.
Full load sweep, single server on localhost, only the bot count changed (so network distance is fixed at zero):
| ms | 1 bot | 5 | 10 | 20 | 30 | 40 |
|---|---|---|---|---|---|---|
| p50 | 2 | 3 | 3 | 4 | 8 | 40 |
| p95 | 15 | 51 | 54 | 63 | 95 | 224 |
| p99 | 81 | 75 | 929* | 274 | 398 | 583 |
| max | 81 | 133 | 1306* | 603 | 704 | 1241 |
The analytical model predicts an identical latency for every column, because distance and bandwidth never move, so the growth is all server-side.
But the magnitudes here are confounded and should not be quoted: every run started from a freshly copied world and the grid ramped upward, so each cell pushed bots into terrain the previous ones never reached. The server was generating chunks as well as simulating players, and the amount of generation rose together with the player count. On a warm world the same 0.5-core node holds 50 players before breaking the tick budget — it was not close to saturated at 40. Full correction in results/sweep.md; the artefact and its fix in docs/06-troubleshooting.md §6.2b. Earlier de-risk: results/derisk.md.
| server | Paper 1.21.4 (itzg/minecraft-server) |
| mineflayer | 4.37.1 |
| node | v22.22.2 |
| host | Fedora, i7-14700HX, 16 GB RAM |
- Single-node localhost baseline (bot RTT under load, fixed zero network)
- Concurrent network probes, to measure the network vs server split per run
- Containerlab edge-cloud topology with two capped nodes at different emulated distances
- Automated sweep across bot counts and placement targets
- Four-candidate testbed with a capacity ladder (0.5 / 1 / 2 / 4 cores)
- Per-node capacity measured, so load can be expressed as utilization
- Warm re-run of the placement grid — where does edge stop winning?
- Labelled dataset
- Train a latency predictor on measured data and compare against the analytical model's node ranking
Two generations of data live under results/:
results/raw/*.log— the single-node sweeps, bot RTT only, one file per run. Written up in sweep.md and derisk.md.results/raw/<tag>/<target>-<N>bots/— the two-node testbed. Four raw streams per cell (bot.csv,probe.csv,icmp.txt,server.csv) plus ameta.jsonrecording the CPU cap, the netem delay, the player count actually held, and the exact millisecond boundaries of the idle and steady windows.
Nothing is aggregated at collection time — see
docs/04-metrics.md for why, and for how to read a run
with nothing but grep and sort.
docs/ covers the project from zero: the concepts (01), the testbed (02), every shell command (03), the measurement design (04), the runbook (05), and every failure hit so far with its fix (06).
MIT — see LICENSE.