Skip to content

runbook.md

Latest commit

 

History

History
100 lines (69 loc) · 5.61 KB

runbook.md

File metadata and controls

100 lines (69 loc) · 5.61 KB
Raw

Operational Runbook

(Startup, health checks, shutdown, and common operations)

Scope

This runbook covers:

  • Startup procedures (local development and production)
  • Health checks and verification
  • Shutdown and cleanup
  • Common operations (migrations, backups)
  • Emergency procedures and troubleshooting pointers

It does not cover:

  • Feature-level behavior (see subsystem docs)
  • Deployment pipeline (see Deployment)
  • Detailed troubleshooting (see Troubleshooting)

Startup Procedures

Local development

  1. Prerequisites: Node.js v18/v20, npm 9+.
  2. Environment: Copy or create .env (see Getting started). For local storage, no env vars are required (defaults: local backend, ./data, ./data/neotoma.db, ./data/sources).
  3. Install and migrate: 
    npm install
npm run migrate
  4. Start services:
    • MCP server (stdio): npm run dev
    • API only (MCP at /mcp): npm run dev:server
    • Full stack (API + UI): npm run dev:full
    • WebSocket MCP bridge: npm run dev:ws
  5. Verify: Run npm run doctor (see Health check).

Production

  1. Build: npm run build:server.
  2. Set production env (data paths, HTTPS, OAuth config).
  3. Run migrations: npm run migrate or ensure local DB path is correct.
  4. Start: node dist/actions.js (API) or node dist/index.js (stdio MCP). Use a process manager (e.g. systemd, PM2) and HTTPS in front.
  5. Verify health: npm run doctor and smoke-test critical endpoints.

For a headless systemd-supervised deployment of the npm-installed CLI (including a reference unit file, lockdown flags, and smoke-test commands), see install.md § Production deployment (headless / systemd).

Health Checks

Two distinct tools share the word "doctor": npm run doctor (the dev/ops health-check script, scripts/doctor.ts — environment, database, RLS, storage, migrations, security) and neotoma status (the CLI command formerly named neotoma doctor — CLI path, data directory, local API, MCP entries, permissions). This runbook uses npm run doctor for operator health checks and neotoma status --json only where CLI/data-dir state is needed.

  • Primary: Run npm run doctor for environment, database, RLS, storage, migrations, and security checks. See Health check.
  • Manual: Run tests (npm test, npm run test:integration), type-check (npm run type-check), and lint (npm run lint).
  • Runtime: Hit API health/readiness endpoints if configured; confirm MCP server starts (e.g. stdio mode).

Shutdown Procedures

  • Graceful: Stop the Node process (SIGTERM). Allow in-flight requests to complete; process managers typically send SIGTERM then SIGKILL after a timeout.
  • Cleanup: No mandatory cleanup for normal shutdown. For local dev, ensure no stray tsx watch or dev servers if switching branches (ports may be in use).

Common Operations

  • Migrations: Apply with npm run migrate. Dry-run: npm run migrate:dry-run. See Troubleshooting for "relation does not exist" and RLS issues.
  • Backups: For local SQLite, copy the DB (default ./data/neotoma.db in dev, ./data/neotoma.prod.db in prod, under NEOTOMA_DATA_DIR), NEOTOMA_RAW_STORAGE_DIR (default ./data/sources in dev, ./data/sources_prod in prod), and NEOTOMA_LOGS_DIR (default ./data/logs in dev, ./data/logs_prod in prod; includes events.log).
  • User and access: Auth and user management are in Auth.

Branch Protection

  • main must require the GitHub Actions check security_gates.
  • Keep this rule enabled even if /release is run manually; the required check is the PR-time guard against manifest drift, auth-matrix regressions, and security-lint failures.

Emergency Procedures

  • Rollback: Revert deployment to previous version; re-run migrations only if a migration rollback is defined (see migration docs). Restore DB/storage from backup if data corruption is suspected.
  • SQLite corruption: If Neotoma reports database disk image is malformed, btreeInitPage, or a failed PRAGMA integrity_check, run neotoma storage recover-db first. If it reports corruption, stop Neotoma and run neotoma storage recover-db --recover to write a recovered copy. Review and swap files manually; do not auto-replace the live DB. If corruption recurs on macOS, move NEOTOMA_DATA_DIR out of iCloud-synced Documents, Desktop, or iCloud Drive folders with neotoma storage set-data-dir "~/Library/Application Support/neotoma/data" --move-db-files, then rerun neotoma status --json and confirm data.risks is empty.
  • Incident response: Triage using Troubleshooting. Check logs and npm run doctor output. For security issues, see SECURITY.md in the source root.

Troubleshooting Quick Reference

Symptom See
Missing env / connection errors Troubleshooting – Setup
Database / RLS errors Troubleshooting – Runtime
Port in use Troubleshooting – Port
Health check failures Health check