Zero Router - Remaining Tasks & Final Checklist

Status: Phase 3 - Database & Runtime Infrastructure (Started) Archive Location: C:\Users\Asus\Desktop\zero-router-checkpoint.zip (18.1 MB) Current Date: 2026-07-26

📋 Task Summary

All remaining work to ship a production-ready router. Estimated effort: 40-60 hours for one engineer working continuously.

Phase 3: Database & Runtime Persistence (IN PROGRESS)

✅ PgStore Implementation (90% done)

File: packages/store/src/pg/store.ts Status: Core store class + all 6 sub-stores completed What's Done:

PgOrgStore — org lifecycle and policy management
PgApiKeyStore — key issuance, revocation, usage tracking
PgCredentialStore — BYOK management with envelope encryption, status tracking, atomic rotation
PgUsageStore — per-request billing and month-to-date budget checks
PgAuditStore — immutable audit trail
PgIdempotencyStore — atomic claim, complete, release, purge for request deduplication

Tests Needed:

Unit tests for each store (OrgStore CRUD, ApiKeyStore revocation idempotency, CredentialStore concurrent updates)
Tenant isolation verification (org_id in WHERE clause on every read)
Concurrency tests (2+ goroutines race to update same credential, verify no lost counts)
Transaction tests (verify rollback on error, no partial writes)
Encryption tests (credentials always stored as ciphertext, never plaintext in DB)

File: packages/store/src/index.ts Status: Needs implementation What to Do:

// Export the store interfaces and implementations
export * from "./types.js";
export { PgStore, PgStoreTx } from "./pg/store.js";
export { PgRedisStore } from "./redis/store.js"; // Will create below

🔴 Redis Runtime Store (0% done)

File: packages/store/src/redis/store.ts (create new) Purpose: Ephemeral state that survives a single instance failure but not cluster reboot What to Implement:

RateLimiterStore

interface RateLimiterStore {
  // Check and record a token usage
  check(key: string, limit: number, window_seconds: number): Promise<{
    allowed: boolean;
    current: number;
    reset_at: Date;
  }>;
  // Reset on successful billing
  reset(key: string): Promise<void>;
}

Key Pattern: rl:${org_id}:${endpoint}:${date} (hourly buckets) Behavior: Every request increments a counter, returns current count and TTL

CircuitBreakerStore

interface CircuitBreakerStore {
  // Check if circuit is open
  isOpen(provider: string, model: string): Promise<boolean>;
  // Mark a failure (increments consecutive failures)
  recordFailure(provider: string, model: string): Promise<number>;
  // Reset to closed after cooldown
  recover(provider: string, model: string): Promise<void>;
}

Key Pattern: cb:${provider}:${model} → JSON { failures: N, opened_at: timestamp } Logic:

After 3 failures: set to open, TTL 5 min (exponential backoff with multiplier)
On each request: if TTL expired, reset to closed

HealthTrackerStore

interface HealthTrackerStore {
  // Get snapshot of health (fail count, last failure time)
  snapshot(provider: string, model: string): Promise<HealthState>;
  // Record a success
  success(provider: string, model: string): Promise<void>;
}

Key Pattern: health:${provider}:${model} Behavior: Track last 10 failures in a sorted set, delete oldest when new ones arrive

CredentialCooldownStore

interface CredentialCooldownStore {
  // Check if credential is available
  isAvailable(credential_id: string): Promise<boolean>;
  // Mark as in-cooldown until timestamp
  setCooldown(credential_id: string, available_at: Date): Promise<void>;
}

Key Pattern: cooldown:${credential_id} → timestamp Behavior: TTL is the cooldown duration; key disappears on expiry

DistributedLockStore

interface DistributedLockStore {
  // Acquire a lock with timeout
  acquire(key: string, ttl_seconds: number): Promise<string | null>; // null if taken
  // Release (only by owner)
  release(key: string, token: string): Promise<boolean>;
}

Key Pattern: lock:${resource_id} → UUID token Behavior: SET NX with EX, value is a UUID to prevent ABA

Tests Needed:

Rate limiter increments correctly, resets after charge
Circuit breaker opens after N failures, recovers after cooldown
Health tracker records successes/failures and ages out old data
Distributed locks prevent concurrent updates
TTL expiry cleans up stale entries

Phase 4: Gateway Middleware (0% done)

File: apps/gateway/src/middleware/ (create directory) Status: All skeletal, implement in this order

rate-limit.ts

export interface RateLimitMiddleware {
  // Called after auth, before routing
  check(key: string, org_id: string, limit?: number): Promise<void>;
}

Behavior:

Check RPM (requests per minute) from ApiKey
Check TPM (tokens per minute) after response
Return 429 with Retry-After if exceeded
Never fail the entire request on rate limit error (log, continue)

idempotency.ts

export interface IdempotencyMiddleware {
  // Before running request
  before(org_id: string, key: string, request_hash: string): Promise<{
    found: boolean;
    response?: string; // Cached response body
  }>;
  // After running request
  after(org_id: string, key: string, status: number, body: string): Promise<void>;
}

Behavior:

Hash the request (canonical form: model, messages, params)
If Idempotency-Key header present, check store
If found and not expired: return cached response
If claim fails (already running): return 409
After response: store result with TTL=24h

cancellation.ts

export interface CancellationMiddleware {
  makeSignal(request_id: string): AbortSignal;
}

Behavior:

Watch for HTTP client disconnect or explicit cancel request
Propagate abort signal through engine

audit.ts

export interface AuditWriter {
  log(event: AuditEvent): Promise<void>;
}

Events to Log:

API key issued, revoked, used
Credential added, rotated, disabled
Budget exceeded
Failed authentication
Org policy changes

error-normalization.ts

export function normalizeError(e: unknown): CanonicalError {
  // Convert all error types to canonical form
  // Never leak internal error messages to client
  // Log full stack internally
}

usage-metering.ts

export async function recordUsage(
  response: CanonicalResponse,
  auth: RequestAuth,
  trace: RoutingTrace
): Promise<void> {
  // Already partially done in pipeline.ts
  // Ensure called for ALL paths (success, error, timeout, abort)
}

Tests Needed:

Rate limit enforced before request runs
Idempotency key prevents duplicate charges
Audit trail records all sensitive actions
Error messages never leak secrets
Usage recorded even on failure

Phase 5: API Endpoints (0% done)

File: apps/gateway/src/routes/ (create directory)

GET /v1/models

export async function listModels(c: HonoContext): Promise<Response> {
  const registry = c.get("app").registry;
  // Return OpenAI-compatible format
  return c.json({
    object: "list",
    data: registry.allModels.map(m => ({
      id: m.id,
      object: "model",
      created: Date.now() / 1000,
      owned_by: m.provider,
      permission: [{ allow: "all", deny: [] }],
      root: m.id,
      parent: null,
    })),
  });
}

POST /v1/chat/completions

export async function chatCompletions(c: HonoContext): Promise<Response> {
  const auth = await authenticate(c);
  const ingressNotes: TranslationNote[] = [];

  // Parse request (OpenAI format)
  const ingressResult = await ingestOpenAIChat(c.req.json(), ingressNotes);

  if (!ingressResult.ok) {
    return c.json(ingressResult.error, { status: 400 });
  }

  const canonical = ingressResult.value;

  // Handle streaming vs non-streaming
  if (canonical.stream) {
    const { events } = await startStream(deps, {
      request: canonical,
      auth,
      ingressNotes,
      signal: c.req.signal,
    });

    return streamSSE(c, events, "openai");
  } else {
    const { response } = await runRequest(deps, {
      request: canonical,
      auth,
      ingressNotes,
      signal: c.req.signal,
    });

    return c.json(emitOpenAIChat(response, "openai"));
  }
}

POST /v1/messages

export async function messages(c: HonoContext): Promise<Response> {
  // Same flow but with ingestAnthropicMessages + emitAnthropicMessage
}

POST /v1/responses

export async function responses(c: HonoContext): Promise<Response> {
  // Optional: response-specific operations (e.g., retry, re-rank)
  // For MVP: 501 Not Implemented is fine
}

Tests Needed:

/v1/models returns list of all registered models
POST /v1/chat/completions non-streaming returns valid response
POST /v1/chat/completions streaming returns valid SSE events
POST /v1/messages works identically for Anthropic format
OpenAI Python SDK (python -m openai api chat_completions...) works
Anthropic Python SDK works
Node SDK (both) works
Tool calls are parsed and passed through correctly
Streaming works end-to-end (headers sent immediately, chunks arrive)

Phase 6: Production Verification (0% done)

Test Suite Expansion

Integration Tests

File: tests/integration.test.ts

describe("Integration: Full request path", () => {
  // Real request → ingress → canonical → router → engine → egress → response

  it("OpenAI chat completions non-streaming", async () => {
    // POST to /v1/chat/completions
    // Verify response structure
    // Verify usage billing
  });

  it("Streaming events arrive in order", async () => {
    // POST with stream=true
    // Collect all SSE events
    // Verify no gaps in block indices
  });

  it("Request cancelled mid-stream", async () => {
    // Start request
    // Abort after 500ms
    // Verify graceful cleanup
    // Verify partial usage recorded
  });
});

Failure Path Tests

File: tests/failure-paths.test.ts

describe("Failure paths", () => {
  it("Provider timeout triggers failover", async () => {
    // Mock provider timeout
    // Verify fallback target is used
    // Verify original timeout is recorded in trace
  });

  it("Provider 429 triggers circuit breaker", async () => {
    // Enqueue 5 x 429 responses
    // Verify circuit opens after 3
    // Verify next request skips that provider
    // Verify cooldown timer works
  });

  it("Budget exceeded blocks request", async () => {
    // Set monthly budget to $1
    // Make request that costs more
    // Verify 402 Payment Required
    // Verify usage NOT recorded (charge skipped)
  });

  it("All fusion branches fail", async () => {
    // Set up 3 branch models, all error
    // Verify 502 Bad Gateway
    // Verify all 3 errors listed in causes
  });

  it("Judge synthesis returns unparseable output", async () => {
    // Fusion best_of_n with judge returning garbage
    // Verify fallback to first branch
    // Verify trace.fusion.judge.source === "heuristic"
  });
});

Concurrency Tests

File: tests/concurrency.test.ts

describe("Concurrency", () => {
  it("Credential rotation racing updates", async () => {
    // 2 instances both rewrap same credential
    // Verify only one commit wins
    // Verify old ciphertext not retained
  });

  it("Circuit breaker under concurrent load", async () => {
    // 10 concurrent requests to a failing provider
    // Verify circuit opens exactly once
    // Verify no double-count in consecutive_failures
  });

  it("Idempotency key blocks duplicate charge", async () => {
    // Send same request twice simultaneously
    // Verify second request doesn't run
    // Verify usage recorded once
  });
});

Security Tests

File: tests/security.test.ts

describe("Security", () => {
  it("Tenant isolation: org_id in every query", async () => {
    // For each Store method
    // Verify WHERE clause includes org_id or credential.org_id
    // Verify no cross-tenant data leaks
  });

  it("Credentials never plaintext in database", async () => {
    // Insert credential via store.credentials.create()
    // Query DB directly (raw connection)
    // Verify secret column is JSON ciphertext, not plaintext API key
  });

  it("Error messages never leak secrets", async () => {
    // Trigger various errors
    // Verify no API keys in error messages
    // Verify no ciphertext in logs
  });

  it("Rate limiter cannot be bypassed", async () => {
    // Hit rate limit
    // Try to bypass via header tampering
    // Verify 429 still returned
  });
});

Mark Pricing & Quality Unverified

File: packages/provider-sdk/src/pricing.ts

// Add comment to every pricing entry:
export const PRICING: PricingRegistry = {
  "openai/gpt-4o": {
    // ⚠️ UNVERIFIED: These are list prices as of 2026-01-01
    // Actual rates may differ. Always check provider's current pricing page.
    input_per_mtok: 2.5e-6,
    output_per_mtok: 10e-6,
  },
  // ... all other models
};

File: config/models/*.yaml

# At top of each file:
# ⚠️ QUALITY SCORES UNVERIFIED
# These are estimates based on public benchmarks and limited testing.
# Scores will be updated as real user traffic provides feedback.

- id: openai/gpt-4o
  quality:
    general: 92        # Estimated (unverified)
    reasoning: 90      # Estimated (unverified)
    coding: 89         # Estimated (unverified)
    vision: 88         # Estimated (unverified)

Production Readiness Checklist

🐛 FINAL DEBUG & VERIFICATION

Pre-Ship Checklist

# 1. Clean build from scratch
cd zero-router
npm run clean
npm install
npm run typecheck  # Must pass, no @ts-expect-error

# 2. Run full test suite
npm test 2>&1 | tail -50  # Should show all tests green

# 3. Start the router
npm run dev
# Should output:
#   Gateway listening on http://localhost:3000
#   Loaded 13 providers, 50+ models
#   Postgres migrations applied
#   Redis connected
#   Ready for requests

# 4. Test a real request (in another terminal)
curl -X POST http://localhost:3000/v1/chat/completions \
  -H "Authorization: Bearer sk_test_..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zero/auto",
    "messages": [{"role": "user", "content": "Hello"}]
  }'
# Should return:
#   {
#     "id": "chatcmpl-...",
#     "object": "chat.completion",
#     "model": "anthropic/claude-opus",
#     "choices": [{ "message": { "role": "assistant", "content": "..." } }]
#   }

# 5. Test streaming
curl -X POST http://localhost:3000/v1/chat/completions \
  -H "Authorization: Bearer sk_test_..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zero/auto",
    "stream": true,
    "messages": [{"role": "user", "content": "Count to 3"}]
  }'
# Should stream:
#   data: {"choices":[{"index":0,"delta":{"role":"assistant"}}]}
#   data: {"choices":[{"index":0,"delta":{"content":"One"}}]}
#   ...

# 6. Verify database encryption
psql -U postgres zero_router -c \
  "SELECT id, (secret->>'salt')::text as has_salt FROM credentials LIMIT 1;"
# Should show ciphertext, not plaintext key

# 7. Verify idempotency
REQUEST_ID="1234567890"
curl -X POST http://localhost:3000/v1/chat/completions \
  -H "Authorization: Bearer sk_test_..." \
  -H "Idempotency-Key: $REQUEST_ID" \
  -H "Content-Type: application/json" \
  -d '{"model":"zero/auto","messages":[{"role":"user","content":"hi"}]}'
# Make same request again
# Should return identical response, no charge
curl -X POST http://localhost:3000/v1/chat/completions \
  -H "Authorization: Bearer sk_test_..." \
  -H "Idempotency-Key: $REQUEST_ID" \
  -H "Content-Type: application/json" \
  -d '{"model":"zero/auto","messages":[{"role":"user","content":"hi"}]}'

Common Gotchas to Catch

Issue	How to Detect	Fix
Credentials stored plaintext	Query DB, see API key directly	Ensure envelope encryption wraps secret before INSERT
Cross-tenant data leak	Org B queries Org A's creds	Add org_id check to every CredentialStore method
Rate limit counts lost	2 concurrent requests, count < 2	Use INCR (atomic) not GET + SET
Streaming stutters mid-token	Collect all SSE events, find gaps	Verify block_start, delta, block_stop sequence
Fusion judge failure crashes	Pass garbage to judge, observe 500	Catch judge errors, fall back to first branch
Budget not enforced	Burn $1000 with $10 budget	Ensure `assertWithinBudget` runs before engine.execute
Idempotency doesn't dedupe	Send same request 2x, see 2x charge	Verify store.idempotency.claim() returns existing record

📦 Deployment Readiness

Before shipping to production:

Secrets Management
- All API keys in .env.production (not git)
- Database credentials in AWS Secrets Manager (not env file)
- KEK for credential encryption in AWS KMS
- Test backup/restore of encrypted credentials (verify DEK→new KEK works)
Monitoring
- Log all request errors to CloudWatch
- Alert on circuit breaker open (any provider)
- Alert on rate limit exhaustion (any key)
- Track token usage per org (for auto-scaling advice)
- Monitor Postgres connection pool utilization
Database
- Apply migrations in production (with advisory lock)
- Backup before any migration
- Test rollback procedure (apply migration, verify rollback works, re-apply)
- Index on (org_id, created_at) for audit queries
Load Testing
- Simulate 100 concurrent requests
- Verify circuit breaker behavior under sustained 429s
- Measure p50, p95, p99 latency
- Verify rate limiter doesn't exceed Redis throughput

📝 Documentation Needed

File: README.md (create/expand)

# Zero Router

An independent, production-ready AI Router core for multi-model inference.

## Quick Start

### Prerequisites
- Node.js 20+
- PostgreSQL 14+
- Redis 7+

### Setup

\`\`\`bash
git clone ...
cd zero-router

# Install dependencies
npm install

# Configure environment
cp .env.example .env
# Edit .env with your settings

# Run migrations
npm run migrate

# Start the router
npm run dev
\`\`\`

### API Examples

**Chat Completions (OpenAI-compatible)**
\`\`\`bash
curl -X POST http://localhost:3000/v1/chat/completions \
  -H "Authorization: Bearer sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zero/auto",
    "messages": [{"role": "user", "content": "What is 2+2?"}]
  }'
\`\`\`

**Messages (Anthropic-compatible)**
\`\`\`bash
curl -X POST http://localhost:3000/v1/messages \
  -H "x-api-key: sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zero/balanced",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Hello"}]
  }'
\`\`\`

## Testing

\`\`\`bash
npm test                 # Run all tests
npm run test:watch      # Watch mode
npm run typecheck       # Type checking only
\`\`\`

## Architecture

See `docs/architecture.md` for:
- Canonical schema
- Protocol translation layer
- Smart routing & fusion
- Failure handling & circuit breakers
- Multi-tenant isolation

## Configuration

See `config/providers.yaml` and `config/models/*.yaml` for provider/model setup.
See `.env.example` for environment variables.

## License

AGPL-3.0

🎯 Success Criteria

Router is production-ready when:

✅ All 150+ tests pass (existing + new integration/failure/concurrency)
✅ Can handle 100 concurrent requests without dropping any
✅ Credentials stored encrypted, never plaintext
✅ Tenant isolation verified (no cross-org leaks)
✅ Idempotency prevents duplicate charges
✅ Circuit breaker opens/closes correctly
✅ Streaming works end-to-end (events in order, no gaps)
✅ Both OpenAI and Anthropic SDKs compatible
✅ All pricing & quality scores marked unverified
✅ Documentation complete (README, architecture, setup)

⏱️ Estimated Timeline

Phase	Tasks	Hours	Person
3	PgStore tests + Redis store	12-16	1 eng
4	Middleware (rate limit, idempotency, audit)	8-12	1 eng
5	API endpoints (/v1/chat/completions, /v1/messages)	10-14	1 eng
6	Tests (integration, failure, concurrency, security)	12-16	1 eng
QA	Verification, documentation, edge cases	8-12	1 eng
Total		50-70	1 eng

Next Step: Implement PgStore tests, then Redis store, then middleware. Current Blocker: None. Ready to proceed.

FilesExpand file tree

TASKS_REMAINING.md

Latest commit

History

TASKS_REMAINING.md

File metadata and controls

Zero Router - Remaining Tasks & Final Checklist

📋 Task Summary

Phase 3: Database & Runtime Persistence (IN PROGRESS)

✅ PgStore Implementation (90% done)

🔴 Redis Runtime Store (0% done)

RateLimiterStore

CircuitBreakerStore

HealthTrackerStore

CredentialCooldownStore

DistributedLockStore

Phase 4: Gateway Middleware (0% done)

rate-limit.ts

idempotency.ts

cancellation.ts

audit.ts

error-normalization.ts

usage-metering.ts

Phase 5: API Endpoints (0% done)

GET /v1/models

POST /v1/chat/completions

POST /v1/messages

POST /v1/responses

Phase 6: Production Verification (0% done)

Test Suite Expansion

Integration Tests

Failure Path Tests

Concurrency Tests

Security Tests

Mark Pricing & Quality Unverified

Production Readiness Checklist

🐛 FINAL DEBUG & VERIFICATION

Pre-Ship Checklist

Common Gotchas to Catch

📦 Deployment Readiness

📝 Documentation Needed

🎯 Success Criteria

⏱️ Estimated Timeline