Consensus-Coordinated Data-Locality Engine

A miniature hyperconverged storage cluster: metadata is strongly consistent via Raft, data replication is primary-based, and VM migration triggers re-replication so data locality follows the workload.

Modelled on Nutanix's architecture (Stargate + Cassandra/Zeus + Curator), built in Python with FastAPI, etcd, and Docker Compose.

Read latency stays at ~2ms while the VM's data is local. At t=10s the VM migrates to a node holding no copy — reads fall back to the network and latency jumps to ~68ms. Re-replication completes at ~12.2s and locality is restored.

---

The core idea

A distributed storage cluster has to answer one question constantly: where does this data physically live, and is that near the workload reading it?

Two planes, deliberately separated:

Plane	What it carries	Consistency model	Why
Metadata	vdisk → {owner, replicas, locations}	Strongly consistent (etcd / Raft)	Tiny, rarely written, everyone must agree
Data	The actual chunks	Primary-based replication	Large, constantly written — consensus per write would kill throughput

This separation is the whole design. If every data block had to be committed through consensus, throughput would collapse. So Raft holds only the map; chunks move node-to-node directly, owner → replicas, ack on write.

Architecture

   node1            node2            node3          <- Stargate-lite (data plane)
  +--------+      +--------+      +--------+          local chunk store
  | chunks |<---->| chunks |<---->| chunks |          primary replication
  +---+----+      +---+----+      +---+----+          local vs remote read path
      |               |               |
      +---------------+---------------+
                      |
            +---------v---------+
            |   etcd (Raft)     |                   <- metadata plane
            | vdisk -> {owner,  |                      strongly consistent map
            |  replicas, locs}  |
            +---------+---------+
                      |
              +-------v--------+
              |  coordinator   |                    <- control plane
              |  - migration   |                       Curator-lite
              |  - re-replicate|
              |  - scheduler   |
              +-------+--------+
                      |
                 +----v----+
                 | vm_sim  |                        <- workload
                 +---------+

Component	Role	Nutanix analogue
stargate/main.py	Per-node data service. Stores chunks, serves reads, picks local vs remote path, replicates to peers.	Stargate
common.py + etcd	The replicated location map everyone trusts.	Cassandra / Zeus
coordinator/coordinator.py	Executes migrations, drives re-replication.	Curator
coordinator/scheduler.py	Locality-aware placement policy.	ADS (Acropolis Dynamic Scheduling)
vm_sim/vm.py	Workload: owns a vdisk, reads in a loop, logs latency.	the VM

Results

1. Locality lost and recovered

Migrating a VM to a node holding no copy of its data breaks locality. Reads fall back to the network until re-replication catches up.

Phase	Latency	Path
Before migration	~2 ms	local disk
During (t=10–12.2s)	~60–68 ms	remote fetch over network
After re-replication	~2 ms	local again

The first remote read costs ~68ms vs ~60ms for subsequent ones — it pays an extra metadata lookup and a cold HTTP connection.

2. The control case: locality preserved for free

Migrating the same VM to a node that already held a replica produces a completely flat line. No spike, no re-replication, nothing to fix.

latency_replica_hit.csv   ->  flat 2ms throughout
latency.csv               ->  spike and recover

Same system, same migration, opposite outcome — the only variable was where the spare copies happened to sit.

This is the argument for why replica placement and VM scheduling cannot be solved independently, and it motivates the scheduler below.

3. Locality-aware placement

The scheduler prices every candidate node:

cost(node) = load(node) * LOAD_WEIGHT  +  (0 if node holds a copy else MIGRATION_PENALTY)

The current owner is priced too — if staying put is cheapest, nothing moves. Live decision from a running cluster:

{
  "action": "migrate",
  "from": "node3",
  "to": "node1",
  "reason": "node3 is hot (80.0); node1 costs 0.0 (already holds a replica -> locality-free); gain 80.0",
  "costs": { "node1": 0.0, "node2": 0.0, "node3": 80.0 }
}

Every decision explains itself. Set MIGRATION_PENALTY = 0 and the scheduler stops caring about locality entirely — a direct way to show the term is doing real work.

Hysteresis: a candidate must beat the incumbent by more than IMPROVEMENT_THRESHOLD before anything moves. Without it the scheduler thrashes — migrating shifts load, which triggers migrating back, forever.

Running it

Requires Docker Desktop.

mkdir out
docker compose up --build

Watch the vm_sim logs: latency low -> migration at t=10s -> spike -> recover. Runs ~40s, then:

docker compose down
pip install matplotlib
python plot/graph.py        # -> locality_demo.png

Test the placement policy without Docker

choose_placement() is a pure function — data in, decision out, no network. The policy's judgement is testable in milliseconds:

python -m tests.test_scheduler

PASS quiet cluster -> hold          | owner node1 not hot (5.0 ops/s < 20.0 threshold)
PASS hot owner -> replica holder    | node1 is hot (60.0); node2 costs 5.0 (already holds a replica)
PASS cold node worth the penalty    | node1 is hot (80.0); node3 costs 25.0 (cold -> remote-read window)
PASS marginal gain -> hold          | best alternative saves only 5.0 -- not worth thrashing
PASS locality beats raw idleness    | node1 is hot (70.0); node2 costs 10.0 (already holds a replica)

Poke it by hand

# write a vdisk, replicated to node1 + node2 (node3 stays cold)
curl -X POST localhost:8001/write/vdisk-B -H "content-type: application/json" \
     -d '{"data":"hi","replicas":["node1","node2"]}'

curl localhost:8001/read/vdisk-B                    # local -> ~2ms
curl -X POST localhost:8080/migrate/vdisk-B/node3   # migrate to a cold node
curl localhost:8003/read/vdisk-B                    # remote -> ~60ms, then local

curl -X POST "localhost:8080/schedule/vdisk-B?dry_run=true"   # ask the scheduler
curl localhost:8080/stats                                     # cluster load

Design decisions worth defending

Consensus is for the map, not for voting on placement. The coordinator decides placement and commits it to the replicated log. The cluster doesn't hold a vote. Paxos/Raft exists to keep the location map strongly consistent, not to run a democracy.

Data never passes through consensus. Chunks flow owner -> replicas directly. Routing them through etcd would destroy throughput and stop this resembling a real HCI system.

Re-replication here is eager; real systems are lazy. This demo copies the chunk immediately so the recovery is a clean cliff. Nutanix pulls hot extents local on read and lets Curator rebalance in the background — the curve would decay back to baseline instead of dropping off a ledge. Eager: fast, predictable recovery, bandwidth-heavy. Lazy: bandwidth-cheap, locality recovers slowly.

The shape is real, the constants are dialled. LOCAL_DELAY = 2ms and REMOTE_DELAY = 50ms are simulated — containers on one host talk in under a millisecond, too fast to see. The mechanism and the shape of the curve are genuine; the magnitudes are chosen for legibility.

Known limitations

• Heat is measured on the owner only. A node slammed serving remote fetches for vdisks it doesn't own looks idle to the scheduler.
• Ties are broken by dict iteration order, not deliberately. When two candidates cost the same, min() returns whichever comes first — so under steady load every vdisk drifts to node1. Real fix: random or capacity-based tie-break.
• Re-replication is eager, not lazy pull-on-read + background scan.
• One chunk per vdisk. No extent granularity, no RF enforcement, no stale-copy garbage collection.
• Single-node etcd. Real Raft, but no leader-election demo. A 3-node etcd cluster would show that.
• State is ephemeral — chunks live inside containers and are wiped on docker compose down.

Where this pattern shows up

The same problem, under different names: HDFS/YARN (move compute to data, not data to compute), Kubernetes local persistent volumes (pod reschedules, volume doesn't follow), Ceph and vSAN, CDN cache misses (a remote read is a cache miss; pull-on-read is lazy re-replication), and replica-aware query routing in Cassandra and CockroachDB.

Next

• Lazy re-replication — pull-on-read instead of coordinator push; compare curve shapes
• Real network delay via tc netem instead of time.sleep()
• Node failure mid-run: docker stop node1 and watch the map resolve to a surviving replica
• 3-node etcd cluster to demonstrate leader election
• Total node load instead of owner-only heat

Stack

Python · FastAPI · etcd (Raft) · Docker Compose · matplotlib