Quick Start Guide

This guide will help you get started with Secure Model-Distributed LLM Inference in under 5 minutes!

Prerequisites

Python 3.8 or higher
4GB+ RAM recommended
Internet connection (for downloading models)

Installation

Option 1: Automated Setup (Recommended)

chmod +x setup.sh
./setup.sh

Option 2: Manual Setup

# Create virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

Running the Demos

1. Simple Demo (Recommended for First Time)

The simple demo shows the security features in a single process:

cd examples
python run_demo_simple.py

Expected output:

Simple Secure MDI-LLM Demo
==================================================

[1] Loading model...
[2] Setting up security...
    Weight commitment: 8f3a4b9c2d1e5f6a...
[3] Generating text with verification...
    Prompt: The future of artificial intelligence
    Token 0: Verification ✓
    Token 5: Verification ✓
    Token 10: Verification ✓
    Token 15: Verification ✓
[4] Generated text:
    The future of artificial intelligence is likely to...

✓ Demo complete!

Runtime: ~1-2 minutes

2. Full Distributed Demo

The full demo runs actual distributed inference with 3 nodes.

Use the thread-based version (recommended):

cd examples
python demo_simple_distributed.py

Or try the process-based version (experimental):

cd examples
python demo.py

What this demonstrates:

Real multi-process distributed execution
Secure encrypted communication between nodes
Pipeline parallelism for generation
Cryptographic verification

Expected output:

============================================================
Secure Model-Distributed LLM Inference Demo
============================================================

Configuration:
  Model: gpt2
  Nodes: 3
  Device: cpu

[1/5] Loading tokenizer...
[2/5] Partitioning model...
  Created 3 chunks:
    Node 0 (starter): 41,158,656 parameters
    Node 1 (secondary): 42,467,328 parameters
    Node 2 (secondary): 40,894,464 parameters

[3/5] Starting secondary nodes...
[4/5] Creating starter node...
[5/5] Generating text with secure distributed inference...
...

Runtime: ~2-5 minutes

Troubleshooting

Issue: Port already in use

If you see "Address already in use" errors:

Wait a few seconds and try again (previous demo may still be shutting down)
Or change the ports in examples/demo.py:

nodes_config = [
    {'host': 'localhost', 'port': 6000, 'next_port': 6001},
    {'host': 'localhost', 'port': 6001, 'next_port': 6002},
    {'host': 'localhost', 'port': 6002, 'next_port': 6000},
]

Issue: Out of memory

If you run out of memory:

Close other applications
Or use a smaller batch size/fewer tokens in demo.py:

results = starter_node.generate(
    prompts=["Once upon a time"],  # Reduce number of prompts
    max_new_tokens=10,              # Reduce tokens to generate
    temperature=0.8
)

Issue: Model download fails

If the model download fails:

Check your internet connection
Try downloading manually:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

Understanding the Output

Security Features in Action

Weight Commitment:

Weight commitment: 8f3a4b9c2d1e5f6a...

This is a cryptographic hash of all model weights, ensuring the model hasn't been tampered with.

Verification Checks:

Token 0: Verification ✓

Shows when intermediate computations are being verified using cryptographic proofs.

Node Communication:

[node_1] Accepted connection from ('127.0.0.1', xxxxx)
[node_2] Connected to next node at localhost:5002

Indicates secure encrypted channels have been established between nodes.

Performance Metrics

Generation complete: 36.2s, 2.48 tokens/sec

Total time for generation
Tokens generated per second (includes verification overhead)

Verification Stats:

Verification stats: {
  'total_processed': 90,
  'verified': 9,
  'failed': 0,
  'success_rate': 1.0
}

Shows 10% of operations were verified (probabilistic checking)
All verified operations passed (success_rate: 1.0)

Next Steps

1. Experiment with Configuration

Try different settings in examples/demo.py:

# Try different models
MODEL_NAME = "gpt2"        # 124M parameters
MODEL_NAME = "gpt2-medium" # 355M parameters

# Change number of nodes
NUM_NODES = 3  # More nodes = more distribution

# Adjust generation
max_new_tokens=50          # Generate more text
temperature=0.8            # Control randomness (0.0 = deterministic)

2. Adjust Security Settings

In src/verification.py:

# Change verification rate
VerificationManager(verification_rate=0.2)  # Verify 20% instead of 10%

3. Deploy on Multiple Machines

To run nodes on different machines:

Update nodes_config with actual IP addresses:

nodes_config = [
    {'host': '192.168.1.10', 'port': 5000, ...},
    {'host': '192.168.1.11', 'port': 5000, ...},
    ...
]

Ensure firewall allows connections on the specified ports
Run each node separately on its respective machine

Getting Help

Read the full README.md for detailed documentation
Check the source code in src/ for implementation details
Review the example scripts in examples/

What's Happening Under the Hood?

Model Partitioning: The GPT-2 model is split across 3 nodes
Secure Channels: Each connection uses AES-256-GCM encryption
Pipeline Execution: Multiple samples flow through the pipeline simultaneously
Verification: Random spot-checks verify computation integrity
Token Generation: Starter node samples tokens from final logits

This achieves secure distributed inference without any single node having the complete model!

Ready to dive deeper? Check out the README.md for advanced usage and architecture details.

FilesExpand file tree

QUICKSTART.md

Latest commit

History