Skip to content

deployment.md

Latest commit

 

History

History
269 lines (188 loc) · 7.28 KB

deployment.md

File metadata and controls

269 lines (188 loc) · 7.28 KB
Raw

Deployment Guide

How to deploy the pdbfe stack from scratch. For upgrading an existing deployment, see Upgrading.

Prerequisites

  • Node.js 22+ and npx
  • Wrangler CLI (npm install -g wrangler)
  • A Cloudflare account (Workers Paid plan recommended for D1 row limits)
  • A PeeringDB account with an API key (https://www.peeringdb.com/apidocs/)
  • Python 3.12+ with venv: 
    python3 -m venv scripts/.venv
scripts/.venv/bin/pip install -r scripts/requirements.txt
  • sqlite3 on PATH

1. Cloudflare Resource Setup

Provision once per account. Record every ID and name printed — you'll need them for .env.deploy.

# D1 — PeeringDB mirror data
wrangler d1 create peeringdb

# D1 — user profiles, preferences, API keys
wrangler d1 create pdbfe-users

# KV — session token store
wrangler kv namespace create SESSIONS

# Vectorize — graph-structural embedding index (1024-dim node2vec, cosine)
wrangler vectorize create pdbfe-vectors --dimensions=1024 --metric=cosine

# Enable metadata filtering on entity type (required for graph search)
wrangler vectorize create-metadata-index pdbfe-vectors \
    --property-name=entity --type=string

# Queue — async task bus between sync and async workers
wrangler queues create pdbfe-tasks

# R2 — logo mirror bucket
wrangler r2 bucket create pdbfe-logos

2. PeeringDB OAuth Application

Register an OAuth application at https://auth.peeringdb.com:

  1. Log in to PeeringDB
  2. Navigate to your profile → OAuth Applications
  3. Create a new application:
    • Redirect URI: https://auth.<your-domain>/auth/callback
    • Grant type: Authorization Code
  4. Note the Client ID and Client Secret

3. Configure Credentials

.env.deploy — Cloudflare infrastructure IDs

Copy .env.deploy.example to .env.deploy (gitignored) and fill in:

CLOUDFLARE_ACCOUNT_ID=<from wrangler whoami>
D1_DATABASE_ID=<peeringdb database_id>
D1_USERDB_ID=<pdbfe-users database_id>
KV_SESSIONS_ID=<SESSIONS namespace_id>
API_DOMAIN=<your custom domain, e.g. pdbfe.dev>
VECTORIZE_INDEX_NAME=pdbfe-vectors
PDBFE_VERSION=<current version>

.env — Runtime secrets

Copy .env.example to .env (gitignored) and set:

PEERINGDB_API_KEY=<your api key>
OAUTH_CLIENT_ID=<from step 2>
OAUTH_CLIENT_SECRET=<from step 2>

4. Generate Wrangler Configs

./scripts/deploy.sh --generate-configs

Reads .env.deploy and expands all wrangler-*.toml.example files into their corresponding wrangler-*.toml counterparts. Verify the output confirms the correct domain before proceeding.

5. Worker Secrets

Set runtime secrets for each worker. These are stored in Cloudflare and never committed.

# Sync worker — upstream API access for delta fetching
wrangler secret put PEERINGDB_API_KEY --config workers/wrangler-sync.toml

# Auth worker — OAuth credentials + WAF bypass
wrangler secret put OAUTH_CLIENT_ID     --config workers/wrangler-auth.toml
wrangler secret put OAUTH_CLIENT_SECRET --config workers/wrangler-auth.toml
wrangler secret put PEERINGDB_API_KEY   --config workers/wrangler-auth.toml

The PEERINGDB_API_KEY on the auth worker is required to bypass PeeringDB's WAF during the OAuth token exchange — Cloudflare Workers IPs are blocked without it.

6. Bootstrap D1

Downloads public JSON dumps from public.peeringdb.com, builds a clean SQLite database locally, verifies referential integrity, and imports to D1.

./scripts/migrate-to-d1.sh --fetch --remote

schema.sql (generated by the pipeline) already includes all current columns, so no separate migration step is required after a fresh bootstrap.

Bootstrap users database

npx wrangler d1 execute pdbfe-users \
    --file=database/users/schema.sql --remote

npx wrangler d1 execute pdbfe-users \
    --file=database/users/0002_preference_options.sql --remote

Backfill private POC records

Public JSON dumps only contain visible=Public POC records. Backfill the rest:

source .env
python3 scripts/backfill_poc.py | \
    npx wrangler d1 execute peeringdb --remote --yes --file /dev/stdin

7. Deploy Workers

./scripts/deploy.sh --remote

Deploys all 8 workers in dependency order (async consumer before sync producer, search before API). The sync worker starts publishing to the pdbfe-tasks queue on its next cron tick (every 15 min). The async worker consumes those tasks.

8. Compute Graph Embeddings

Embeds all entities using the node2vec graph pipeline. Run without --incremental on a cold start to cover the full dataset.

source .env
python3 scripts/compute-graph-embeddings.py

This may take 30–60 minutes. After completion, the async worker handles ongoing re-embedding via the task queue automatically.

9. Frontend Deployment

The frontend is a static SPA deployed to Cloudflare Pages.

cd frontend
cp js/config.js.example js/config.js
cp _headers.example _headers
cp _redirects.example _redirects

Edit each file and replace the domain placeholders with your actual hostnames. Then deploy:

npx wrangler pages project create <your-pages-project>
npx wrangler pages deploy frontend/ --project-name <your-pages-project>

The _redirects file proxies /api/* requests to the API worker.

10. Verification

# Health check
curl https://api.<your-domain>/health

# First API call
curl https://api.<your-domain>/api/net?limit=5

# Sync status
curl https://pdbfe-sync.<subdomain>.workers.dev/sync/status

# GraphQL introspection
curl -X POST -H 'Content-Type: application/json' \
    -d '{"query":"{ __schema { types { name } } }"}' \
    https://graphql.<your-domain>/

# OpenAPI spec
curl https://rest.<your-domain>/openapi.json

Upgrading

To apply pending database/migrations/*.sql to a live deployment without a full re-bootstrap:

./scripts/deploy.sh --remote --apply-migrations

The deploy script tracks applied migrations in a _migrations D1 table and skips any already applied.

Ongoing Operations

Task Command
Upgrade DB schema ./scripts/deploy.sh --remote --apply-migrations
Re-seed D1 from dumps ./scripts/migrate-to-d1.sh --fetch --remote
Re-backfill private POCs python3 scripts/backfill_poc.py | wrangler d1 execute ...
Purge contaminated vectors python3 scripts/cleanup-junk-vectors.py
Re-embed incrementally python3 scripts/compute-graph-embeddings.py --incremental

Credential Summary

Credential Where it lives Used by
PEERINGDB_API_KEY .env (local), wrangler secret (prod) Sync worker, auth worker (WAF bypass), POC backfill
CLOUDFLARE_API_TOKEN .env (local only) Wrangler CLI
OAUTH_CLIENT_ID wrangler secret Auth worker
OAUTH_CLIENT_SECRET wrangler secret Auth worker
D1 database ID (peeringdb) .env.deploy → generated toml files API, sync, async, search, graphql, rest
D1 database ID (pdbfe-users) .env.deploy → generated toml files API, auth, graphql, rest
KV namespace ID (SESSIONS) .env.deploy → generated toml files API, auth, search, graphql, rest
Vectorize index name .env.deploy → generated toml files Async, search