Reduce AI API costs by 60-80% with intelligent request routing
A production-ready AI orchestration system that intelligently routes requests between local models (CPU-based) and cloud APIs to optimize costs. Run lightweight models locally for simple tasks, use cloud APIs only for complex tasks.
This project follows the MASTER DIRECTORY STANDARDS.
See .directory-standards.md for details.
All documentation is stored in 01-Docs/ using the NNN-abv-description.ext format.
- Smart Request Routing - Automatically estimates prompt complexity and routes to the optimal model
- 60-80% Cost Reduction - Local models handle 60-80% of requests for free
- Full Docker Stack - n8n, Ollama, API Gateway, Prometheus, Grafana, Redis
- Cloud-Ready - Deploy to AWS or GCP with Terraform
- Complete Observability - Prometheus metrics + Grafana dashboards
- Workflow Automation - n8n workflows for orchestration and monitoring
- Taskwarrior Integration - Track all deployments and routing decisions
- Production-Grade - Proper error handling, health checks, logging
- Ubuntu 22.04+ (or compatible Linux)
- 4GB+ RAM (15GB recommended)
- Sudo access
- Docker installed
# Clone the repository
git clone https://github.com/jeremylongshore/Hybrid-ai-stack-intent-solutions.git
cd hybrid-ai-stack
# Run one-command installation
./install.sh
# Edit .env and add your API key
nano .env
# Set: ANTHROPIC_API_KEY=sk-ant-your-key-here
# Deploy everything
./deploy-all.sh dockerThat's it! Your system is now running.
# Simple question (routes to local TinyLlama)
curl -X POST http://localhost:8080/api/v1/chat \
-H "Content-Type: application/json" \
-d '{"prompt": "What is Python?"}'
# Complex task (routes to Claude)
curl -X POST http://localhost:8080/api/v1/chat \
-H "Content-Type: application/json" \
-d '{"prompt": "Write a Python function to implement a binary search tree with insertion, deletion, and balancing."}'
# Check routing statistics
curl http://localhost:8080/api/v1/stats User Request
|
v
API Gateway (:8080)
Smart Router
/ | \
/ | \
Complexity < 0.3 0.3-0.6 > 0.6
| | |
v v v
TinyLlama Phi-2 Claude Sonnet
(Local CPU) (Local CPU) (Cloud API)
$0.00/req $0.00/req $0.003-0.015
How It Works:
- Complexity Estimation - Analyzes prompt length, keywords, code presence
- Intelligent Routing - Routes to cheapest model that can handle the task
- Cost Tracking - Monitors actual vs. estimated costs
- Continuous Learning - Taskwarrior tracks routing decisions for optimization
| Tier | RAM/CPU | Monthly Cost | Local Models | Strategy | Use Case |
|---|---|---|---|---|---|
| 1 Budget | 2GB, 1 CPU | $26 | TinyLlama only | 30% Local | Learning/Testing |
| 2 Standard | 4GB, 2 CPU | $52 | TinyLlama + Phi-2 | 70% Local | Production (Recommended) |
| 3 Performance | 8GB, 4 CPU | $120 | + Mistral-7B | 85% Local | High Volume |
| 4 GPU | 16GB + GPU | $310 | All + GPU Accel | 95% Local | Maximum Power |
Tier 2 (Standard) is recommended - Best balance of cost and performance.
10,000 requests/month × $0.015 average = $150/month
6,000 simple requests → Local (TinyLlama) = $0
2,000 medium requests → Local (Phi-2) = $0
2,000 complex requests → Cloud (Claude) = $30
Total: $30/month + $52 VPS = $82/month
Savings: $68/month (45% reduction)
High-volume scenario:
Cloud Only: 50,000 requests × $0.01 avg = $500/month
Hybrid Stack:
- 30,000 simple → Local = $0
- 15,000 medium → Local = $0
- 5,000 complex → Cloud = $50
Total: $50 + $52 VPS = $102/month
Savings: $398/month (80% reduction)
The stack includes these services (all configured and ready):
| Service | Port | Purpose |
|---|---|---|
| API Gateway | 8080 | Main entry point, smart routing |
| n8n | 5678 | Workflow automation |
| Ollama | 11434 | Local LLM server |
| Prometheus | 9090 | Metrics collection |
| Grafana | 3000 | Metrics visualization |
| Redis | 6379 | Caching |
| Qdrant | 6333 | Vector database |
| PostgreSQL | 5432 | n8n database |
# n8n (workflow automation)
open http://localhost:5678
# Grafana (metrics dashboards)
open http://localhost:3000
# Login: admin / admin
# Prometheus (raw metrics)
open http://localhost:9090
# API Health Check
curl http://localhost:8080/health# Set AWS credentials
export AWS_ACCESS_KEY_ID=your-key
export AWS_SECRET_ACCESS_KEY=your-secret
# Deploy Tier 2 (Standard)
./deploy-all.sh aws 2# Authenticate with gcloud
gcloud auth login
# Deploy Tier 3 (Performance)
./deploy-all.sh gcp 3hybrid-ai-stack/
├── scripts/
│ ├── smart_router.py # Core routing logic
│ ├── moderation_pipeline.py # Content moderation example
│ ├── support_router.py # Customer support example
│ └── tw-helper.sh # Taskwarrior helpers
├── gateway/
│ ├── app.py # Flask API Gateway
│ └── Dockerfile # Gateway container
├── terraform/
│ ├── aws/ # AWS infrastructure
│ └── gcp/ # GCP infrastructure
├── workflows/ # n8n workflow definitions
├── docs/ # Complete documentation
├── configs/ # Service configurations
├── docker-compose.yml # Full stack definition
├── install.sh # One-command installation
└── deploy-all.sh # One-command deployment
# API Keys
ANTHROPIC_API_KEY=sk-ant-your-key-here
OPENAI_API_KEY=sk-your-key-here # Optional
# Local Model Settings
OLLAMA_URL=http://localhost:11434
USE_LOCAL_FOR_SIMPLE=true
COMPLEXITY_THRESHOLD=0.5
# Deployment
DEPLOYMENT_TIER=2
CLOUD_PROVIDER=awsEdit scripts/smart_router.py:
# Adjust complexity thresholds
if complexity < 0.3:
return 'tinyllama' # Very simple
elif complexity < 0.6:
return 'phi2' # Medium
else:
return 'claude-sonnet' # Complex# View metrics
curl http://localhost:8080/metrics
# Key metrics:
# - api_gateway_requests_total{model, backend, status}
# - api_gateway_request_duration_seconds{model, backend}
# - api_gateway_cost_total{model}- Open http://localhost:3000 (admin/admin)
- Navigate to Dashboards
- View pre-configured AI Stack dashboard
# Source helper functions
source scripts/tw-helper.sh
# View routing statistics
tw_routing_stats
# View cost report
tw_cost_report
# View all routing tasks
task project:vps_ai.router list# Activate virtual environment
source venv/bin/activate
# Run tests
pytest tests/
# Run with coverage
pytest --cov=scripts tests/# Start only essential services
docker-compose up -d ollama-cpu redis postgres
# Run API Gateway locally
source venv/bin/activate
python gateway/app.py# View logs
docker-compose logs -f api-gateway
# Restart specific service
docker-compose restart ollama-cpu
# Rebuild and restart
docker-compose up -d --build api-gateway# Check if Ollama is running
docker exec ollama ollama list
# Pull models manually
docker exec ollama ollama pull tinyllama
docker exec ollama ollama pull phi# Check if .env has ANTHROPIC_API_KEY
cat .env | grep ANTHROPIC_API_KEY
# View detailed logs
docker-compose logs -f api-gateway- Quick Start Guide
- VPS Tier Selection
- Deployment Guide
- Architecture Deep-Dive
- Smart Router Logic
- Cost Optimization
- Monitoring Guide
- Troubleshooting
- Use Case Examples
- n8n Workflows
- Taskwarrior Integration
This system is perfect for:
- Content Moderation - 95% cost savings (see
scripts/moderation_pipeline.py) - Customer Support - 70% savings routing FAQs locally
- Code Review - Smart routing for PR analysis
- Data Extraction - Batch processing optimization
- Translation Services - Quality-based routing
- Product Descriptions - Scale content generation
Contributions welcome! Please:
- Fork the repository
- Create a feature branch
- Make your changes
- Submit a pull request
See CONTRIBUTING.md for details.
MIT License - see LICENSE file for details.
- Ollama - Local LLM inference
- Anthropic - Claude API
- n8n - Workflow automation
- Taskwarrior - Task management
- Documentation: Full Docs
- Issues: GitHub Issues
- Discussions: GitHub Discussions
Made for cost-conscious AI developers
Star this repo if you find it useful!