Skip to content

STARTUP.md

Latest commit

 

History

History
302 lines (223 loc) · 7.79 KB

STARTUP.md

File metadata and controls

302 lines (223 loc) · 7.79 KB
Raw

SHATTERED Startup Guide

Quick reference for starting the SHATTERED development environment.

Prerequisites

Required Services

  • PostgreSQL 15+ with pgvector extension - All data storage (documents, vectors, job queue)

Development Tools

  • Python 3.10+
  • Node.js 18+
  • npm

Quick Start (Docker - Recommended)

The easiest way to run SHATTERED is with Docker Compose:

# Start all services (PostgreSQL + App)
docker compose up -d

# View logs
docker compose logs -f app

# Stop services
docker compose down

# Stop and delete all data
docker compose down -v

Access Points:

Manual Setup (Development)

1. Start PostgreSQL with pgvector

Start PostgreSQL with the pgvector extension:

# Using Docker for PostgreSQL only
docker compose up -d postgres

Or use a local PostgreSQL instance with pgvector installed:

export DATABASE_URL=postgresql://arkham:arkhampass@localhost:5432/arkhamdb

The migration script (migrations/001_consolidation.sql) creates required extensions and schemas automatically.

2. Install Python Packages

# Install the Frame (required)
cd packages/arkham-frame && pip install -e .

# Install all shards
for dir in packages/arkham-shard-*/; do pip install -e "$dir"; done

# Or install specific shards
pip install -e packages/arkham-shard-dashboard
pip install -e packages/arkham-shard-ach
pip install -e packages/arkham-shard-ingest
# ... etc

3. Install spaCy Model (for entity extraction)

python -m spacy download en_core_web_sm

4. Install Node Dependencies

cd packages/arkham-shard-shell && npm install

5. Start Backend

python -m uvicorn arkham_frame.main:app --host 127.0.0.1 --port 8100

With auto-reload for development:

python -m uvicorn arkham_frame.main:app --host 127.0.0.1 --port 8100 --reload

6. Start UI Shell

In a new terminal:

cd packages/arkham-shard-shell
npm run dev

Access Points

Service URL Description
UI Shell http://localhost:5173 Development UI (Vite)
Backend API http://localhost:8100 FastAPI backend
Health Check http://localhost:8100/health Service status
API Docs http://localhost:8100/docs Swagger UI
OpenAPI Spec http://localhost:8100/openapi.json API specification

When using Docker Compose, both UI and API are served from port 8100.

Architecture

SHATTERED uses a PostgreSQL-only architecture:

Component Technology Purpose
Database PostgreSQL 15+ Relational data, schema per shard
Vectors pgvector extension Semantic search, embeddings
Job Queue PostgreSQL (SKIP LOCKED) Background task processing
Events PostgreSQL LISTEN/NOTIFY Real-time event distribution

This consolidation (replacing Redis + Qdrant) provides:

  • Single dependency for deployment
  • Air-gap compatible operation
  • ACID transactions across vectors and data
  • Simplified operations

Available Shards (25)

All shards auto-register when installed:

Category Shards
System dashboard, projects, settings
Data ingest, documents, parse, embed, ocr, entities
Search search
Analysis ach, claims, credibility, contradictions, anomalies, patterns, provenance, summary
Visualize graph, timeline
Export export, reports, letters, packets, templates

Configuration

Copy .env.example to .env and customize:

cp .env.example .env

Required Settings

Variable Default Description
DATABASE_URL postgresql://arkham:arkhampass@localhost:5432/arkhamdb PostgreSQL connection
AUTH_SECRET_KEY (generate for prod) JWT signing key - use openssl rand -hex 32

Optional LLM Settings

Variable Default Description
LLM_ENDPOINT http://localhost:1234/v1 LLM API endpoint (LM Studio, OpenAI, etc.)
LM_STUDIO_URL - Convenience variable for LM Studio
VLM_ENDPOINT - Vision model for OCR (Qwen-VL, GPT-4o)
EMBED_MODEL all-MiniLM-L6-v2 Embedding model name

API Keys (environment only, never in config files)

  • LLM_API_KEY - Generic LLM API key
  • OPENAI_API_KEY - OpenAI
  • OPENROUTER_API_KEY - OpenRouter
  • ANTHROPIC_API_KEY - Anthropic
  • TOGETHER_API_KEY - Together AI
  • GROQ_API_KEY - Groq

Other Settings

Variable Default Description
CORS_ORIGINS localhost:5173,localhost:8100 Allowed CORS origins
ARKHAM_OFFLINE_MODE false Air-gap mode (no model downloads)
ARKHAM_SERVE_SHELL false Serve static UI from backend

Worker Pools

Workers auto-spawn when jobs are submitted. Available pools:

Pool Type Workers Purpose
io-file IO 20 File operations
io-db IO 10 Database operations
cpu-light CPU 50 Light processing
cpu-heavy CPU 6 Heavy computations
cpu-ner CPU 8 Named Entity Recognition
cpu-extract CPU 4 Data extraction
cpu-image CPU 4 Image processing
cpu-archive CPU 2 Archive processing
gpu-paddle GPU 1 OCR (2GB VRAM)
gpu-qwen GPU 1 Vision tasks (8GB VRAM)
gpu-whisper GPU 1 Audio transcription (4GB VRAM)
gpu-embed GPU 1 Embedding generation (2GB VRAM)
llm-enrich LLM 4 Data enrichment
llm-analysis LLM 2 Analysis tasks

Workers are managed via the Dashboard shard's Workers tab.

Troubleshooting

Backend won't start

  1. Check Python version: python --version (need 3.10+)
  2. Verify packages installed: pip list | grep arkham
  3. Check port availability: netstat -an | grep 8100
  4. Verify PostgreSQL is running and accessible

UI won't start

  1. Check Node version: node --version (need 18+)
  2. Delete node_modules and reinstall: rm -rf node_modules && npm install
  3. Check for port conflicts on 5173

Database connection errors

  1. Verify PostgreSQL is running
  2. Check DATABASE_URL in .env
  3. Ensure database exists with pgvector extension

Vector search not working

  1. Check pgvector extension is installed: SELECT * FROM pg_extension WHERE extname = 'vector';
  2. Verify embedding model is downloaded (check logs for download progress)
  3. For air-gap mode, ensure models are pre-cached in ARKHAM_MODEL_CACHE

Shards not discovered

  1. Verify shard package installed: pip show arkham-shard-ach
  2. Check entry_points in shard's pyproject.toml
  3. Look for import errors in backend logs
  4. Restart the backend after installing new shards

LLM features not working

  1. LLM is optional - app works without it
  2. Check LLM_ENDPOINT in .env
  3. Verify LLM service is running (LM Studio, Ollama, etc.)
  4. Check API key is set in environment (not config file)

Development Tips

Adding a new shard

  1. Create package in packages/arkham-shard-{name}/
  2. Use arkham-shard-ach as reference implementation
  3. Add pyproject.toml with entry_points
  4. Create shard.yaml manifest (v5 format)
  5. Install with pip install -e .
  6. Restart backend to discover

Hot reload

  • Backend: Use --reload flag with uvicorn
  • Frontend: Vite HMR is automatic

Checking shard registration

curl http://localhost:8100/api/shards | jq

Checking service health

curl http://localhost:8100/health | jq

Stop Services

# Docker
docker compose down

# Manual
# Ctrl+C in each terminal, or:

# Windows
netstat -ano | findstr :8100
taskkill /PID <pid> /F

# Linux/Mac
lsof -i :8100
kill -9 <pid>