Skip to content

README.md

Latest commit

 

History

History
272 lines (184 loc) · 4.19 KB

README.md

File metadata and controls

272 lines (184 loc) · 4.19 KB
Raw

Raft-Based Distributed Key-Value Store

A distributed key-value store implemented in Node.js using a Raft-inspired consensus algorithm.

This project was built from scratch to explore the internals of distributed systems, including leader election, log replication, majority consensus, node recovery, and persistence.

Features

Consensus

  • Leader election
  • Heartbeats
  • Term-based voting
  • Majority quorum consensus
  • Automatic leader failover

Replication

  • Leader-based writes
  • Log replication to followers
  • Commit after majority acknowledgment
  • Consistent state across nodes

Recovery

  • Node restart recovery
  • Log synchronization from leader
  • Divergence detection
  • Log repair and catch-up
  • Automatic cache reconstruction from logs

Persistence

  • Persistent log storage
  • Persistent term and vote information
  • Recovery after process restart

Key-Value Operations

  • Set key-value pairs
  • Delete keys
  • Retrieve data from replicated state

Architecture

The cluster consists of multiple servers.

Client
   |
   v
Follower ----> Leader
                  |
                  v
            Replicated Log
                  |
         -----------------
         |               |
         v               v
     Follower       Follower

Only the leader accepts writes.

If a follower receives a write request, it forwards the request to the current leader.

The leader:

  1. Appends the operation to its log.
  2. Replicates the log entry to followers.
  3. Waits for a majority of acknowledgments.
  4. Commits the entry.
  5. Applies the operation to its cache.
  6. Notifies followers of the commit.

Endpoints

Write Operations

Set

POST /set

Body:

{
    "key":"goal",
    "value":"peace"
}

Delete

DELETE /delete/:key

Internal Replication Routes

Write

Used when followers forward client requests to the leader.

POST /write

Change

Used for log replication.

POST /change

Commit

Used to commit replicated entries.

POST /commit

Heartbeat

Used by the leader to maintain authority.

POST /heartbeat

Request Vote

Used during elections.

POST /requestVote

Sync

Used by recovering nodes to request missing log entries.

GET /sync/:lastIndex

Debug Routes

View Log

GET /log

View Cache

GET /cache

Log Replication

Each operation is recorded as a log entry:

{
    "term":10,
    "operation":"set",
    "key":"goal",
    "value":"peace"
}

Followers verify:

  • Previous log index
  • Previous log term
  • Current term

before accepting entries.

If divergence is detected, the leader rewinds and repairs the follower's log before continuing replication.

Recovery

When a node rejoins the cluster:

  1. It discovers the current leader through heartbeats.
  2. It requests missing entries using /sync.
  3. It appends the missing log entries.
  4. It replays unapplied operations.
  5. Its cache becomes consistent with the cluster.

Running Multiple Nodes

Example:

node server1.js
node server2.js
node server3.js

Each server should have:

  • Unique ID
  • Unique port
  • Appropriate peer list

Example:

const myId = "server1";
const myAddress = "http://localhost:3001";

const servers = [
    "http://localhost:3002",
    "http://localhost:3003"
];

Learning Goals

This project was built to understand:

  • Distributed systems
  • Consensus algorithms
  • Replication
  • Leader election
  • Failure recovery
  • Eventual consistency
  • State machine replication

Rather than relying on an existing implementation, the goal was to build the mechanisms manually and understand how they interact under failure and recovery scenarios.

Future Improvements

  • Snapshotting and log compaction
  • Dynamic cluster membership
  • Read consistency guarantees
  • Authentication and authorization
  • Better persistence layer
  • Network partition testing
  • Automated integration tests

Disclaimer

This is an educational implementation inspired by Raft and distributed database design principles. It is intended for learning and experimentation rather than production use.