Pull-Based Worker System over Tor

A minimal, technical demo showing how to deploy a pull-based worker system using Docker and Tor hidden services.

Overview

• Coordinator exposes a hidden service (.onion) over Tor.
• Workers pull jobs from the coordinator via Tor SOCKS5h proxy.
• Jobs are stored in Redis (shared queue).
• Optional debugging via a local Tor proxy (localhost:9050) using Postman or curl.
• Minimal security: API key authentication for coordinator endpoints.

Architecture

    ┌───────────────┐
    │   Browser /   │
    │   Postman     │
    └──────┬────────┘
           │ HTTPS / HTTP via Tor
           │
    ┌──────▼────────┐
    │   Tor Proxy   │  <-- Hidden service exposes coordinator
    └──────┬────────┘
           │
    ┌──────▼────────┐
    │ Coordinator   │
    │  FastAPI app  │
    └──────┬────────┘
           │
     ┌─────▼─────┐
     │   Redis   │
     └─────┬─────┘
           │
   ┌───────▼─────────┐
   │  Workers        │
   │  Pull jobs via  │
   │  Tor SOCKS5h    │
   └─────────────────┘

Project structure

.
├── coordinator
│   ├── app.py
│   └── Dockerfile
├── docker-compose.yml
├── README.MD
├── tor
├── torproxy 
└── worker
    ├── Dockerfile
    └── worker.py

coordinator/app.py

from fastapi import FastAPI, Request, HTTPException
import os, redis, json

app = FastAPI()
API_KEY = os.getenv("API_KEY")
REDIS_HOST = os.getenv("REDIS_HOST", "redis")
r = redis.Redis(host=REDIS_HOST, port=6379, db=0)

@app.get("/health")
async def health():
    return {"status": "ok"}

@app.post("/submit-job")
async def submit_job(request: Request):
    key = request.headers.get("x-api-key")
    if key != API_KEY:
        raise HTTPException(status_code=401, detail="Invalid API key")
    data = await request.json()
    r.rpush("job_queue", json.dumps(data))
    return {"status": "submitted"}

@app.get("/get-job")
async def get_job(request: Request):
    key = request.headers.get("x-api-key")
    if key != API_KEY:
        raise HTTPException(status_code=401, detail="Invalid API key")
    job = r.lpop("job_queue")
    if job:
        return json.loads(job)
    return {"status": "no-job"}

@app.get("/get-redis-job")
async def get_redis_job(request: Request):
    job_raw = r.lpop("job_queue")
    if job_raw:
        job_data = json.loads(job_raw)
        print("Job from Redis:", job_data)
        return job_data
    else:
        print("No jobs in queue")
        return {"status": "no-job"}

coordinator/Dockerfile

FROM python:3.12-alpine
WORKDIR /app
COPY app.py .
RUN pip install fastapi uvicorn redis
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

worker/worker.py

import os, requests, time, json, redis

TOR_SOCKS = os.getenv("TOR_SOCKS", "socks5h://tor:9050")
ONION_ADDRESS = os.getenv("ONION_ADDRESS")
API_KEY = os.getenv("API_KEY")
REDIS_HOST = os.getenv("REDIS_HOST", "redis")

session = requests.Session()
session.proxies = {"http": TOR_SOCKS, "https": TOR_SOCKS}
headers = {"x-api-key": API_KEY}

r = redis.Redis(host=REDIS_HOST, port=6379, db=0)

print("Starting worker...")

while True:
    try:
        # Pull job from coordinator
        resp = session.get(f"http://{ONION_ADDRESS}:80/get-job", headers=headers, timeout=30)
        if resp.status_code == 200:
            job = resp.json()
            if "status" in job and job["status"] == "no-job":
                print("No job available, retrying...")
                time.sleep(5)
                continue
            # Execute job
            print(f"Executing job: {job}")
            # Example: just echo to Redis
            r.rpush("results", json.dumps({"job": job, "result": "done"}))
        time.sleep(1)
    except Exception as e:
        print("Worker error:", e)
        time.sleep(5)

worker/Dockerfile

FROM python:3.12-alpine
WORKDIR /app
COPY worker.py .
RUN apk add --no-cache curl torsocks\
    && pip install requests pysocks redis
CMD ["python", "-u", "worker.py"]

docker-compose.yml

services:

  # Tor Onion Service for coordinators
  tor:
    image: dperson/torproxy
    container_name: tor
    volumes:
      - ./tor:/var/lib/tor
    command: >
      -s "80;coordinator:8000"
      -p "StrongPassword"
    networks:
      - tor-net

  # Redis shared queue
  redis:
    image: redis:alpine
    container_name: redis
    ports:
      - "6379:6379"
    networks:
      - tor-net

  # Coordinator replicas (stateless)
  coordinator:
    build: ./coordinator
    container_name: coordinator
    environment:
      - REDIS_HOST=redis
      - API_KEY=supersecret123
    networks:
      - tor-net
    expose:
      - "8000"

  # Worker containers (pull-based)
  worker1:
    build: ./worker
    container_name: worker1
    depends_on:
      - tor
      - redis
    environment:
      - TOR_SOCKS=socks5h://tor:9050
      - ONION_ADDRESS=<your-address>.onion
      - API_KEY=supersecret123
      - REDIS_HOST=redis
    networks:
      - tor-net

  worker2:
    build: ./worker
    container_name: worker2
    depends_on:
      - tor
      - redis
    environment:
      - TOR_SOCKS=socks5h://tor:9050
      - ONION_ADDRESS=<your-address>.onion
      - API_KEY=supersecret123
      - REDIS_HOST=redis      
    networks:
      - tor-net


# Tor proxy for Onion Service access - Allow debugging with Postman via socks5h://localhost:9050

  torproxy:
    image: dperson/torproxy
    container_name: torproxy
    volumes:
      - ./torproxy:/var/lib/tor

    networks:
      - tor-net
    ports:
      - "9050:9050"

networks:
  tor-net:
    driver: bridge

---

Coordinator

• Python FastAPI application
• Endpoints:
  • GET /health → checks liveness
  • POST /submit-job → submit a job (requires x-api-key)
  • GET /get-job → pull next job (requires x-api-key)
  • GET /get-redis-job → debug jobs directly from Redis
• Uses Redis for job queue storage

Dockerfile installs fastapi, uvicorn, redis.

---

Worker

• Pulls jobs continuously from the coordinator hidden service via Tor SOCKS5h proxy
• Executes the job (example: writes result to Redis)
• Prints logs directly to Docker stdout (use -u or PYTHONUNBUFFERED=1)

---

Tor Proxy

• dperson/torproxy container exposes hidden service for coordinator
• Optional local Tor proxy for debugging: localhost:9050
• Workers and Postman can use SOCKS5h proxy to access .onion address

---

Running the System

# Build and start all containers
docker compose up --build

Test

torsocks curl -X POST \
  -H "x-api-key: supersecret123" \
  -H "Content-Type: application/json" \
  -d '{"task":"process_data","payload":{"value":42}}' \
  http://<your-onion>.onion/submit-job

Monitor worker logs

docker logs -f worker1
docker logs -f worker2

Security Notes

Coordinator is internal only, not exposed to the public network .onion service encrypts traffic end-to-end API key authentication prevents unauthorized job submission Workers communicate over Docker network and Tor; Redis is internal

Job Flow

Submit a job → coordinator → stored in Redis queue Workers pull jobs via Tor (/get-job) Worker executes job → writes results back to Redis (job_results) Coordinator or debug tools can monitor jobs

Utils

Get the onion address

cat tor/hidden_service/hostname

docker compose up -d
docker compose down -v
--
docker compose build worker1
docker compose build worker2
docker compose build coordinator

docker logs worker1 -f
docker logs worker2 -f
docker logs coordinator -f
docker logs tor
--
docker exec -it tor sh
docker exec -it worker1 sh 
docker exec -it worker2 sh 

docker exec -it redis redis-cli LRANGE job_results 0 -1

Enjoy!