MAINTENANCE.md

Maintenance Runbook

1) Health checks

Use:

• GET /health

Expected:

• status: healthy
• checks.database.status: ok
• checks.redis.status: ok
• checks.queue.status: ok

If unhealthy:

1. Read the failing check message.
2. Verify dependency availability.
3. Confirm queue activity and backlog.

2) Metrics to watch

Core metrics:

• documents_generated_total
• batch_processing_duration_seconds
• queue_size

Recommended alerts:

1. queue_size{state="pending"} high for more than N minutes.
2. Failure ratio rising:
   • rate(documents_generated_total{status="failed"}[5m])
3. batch_processing_duration_seconds p95 above SLA.

3) Log correlation

Structured logs include correlation fields when available:

• batchId
• documentId
• userId
• jobId

Use these fields to trace a single document from enqueue to completion/failure.

4) Extreme case handling

MongoDB unavailable

• API endpoints return 503 for document operations.
• Worker retries Mongo operations with exponential backoff.

Redis unavailable

• enqueue falls back to an in-memory queue.
• a background drain attempts to push fallback jobs when Redis recovers.
• monitor fallback size from /health checks.queue.fallback.

PDF timeout

• PDF generation has a strict timeout (PDF_TIMEOUT_MS, default 5000ms).
• timed-out tasks are failed and retried by queue policy.

5) Graceful shutdown

Signals handled:

• SIGTERM
• SIGINT

Shutdown order:

1. stop accepting new HTTP connections
2. close worker gracefully
3. close queue client
4. close MongoDB
5. close Redis cache client

6) Common operations

Rebuild:

• pnpm run build

Run in watch mode:

• pnpm run watch

Run benchmark:

• pnpm run benchmark

7) Recovery checklist

1. Confirm /health is healthy.
2. Confirm queue_size pending is decreasing.
3. Verify recent logs for repeated errors.
4. Validate one end-to-end batch manually.