Skip to content

CLAUDE.md

Latest commit

 

History

History
214 lines (164 loc) · 13.8 KB

CLAUDE.md

File metadata and controls

214 lines (164 loc) · 13.8 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Overview

Databuddy is a comprehensive analytics platform — a Turborepo monorepo using Bun as the package manager and runtime. It consists of multiple apps (dashboard, API, data collectors) and shared packages, backed by PostgreSQL, ClickHouse, and Redis.

Development Commands

# Start dashboard + API only (most common)
bun run dev:dashboard

# Start all apps
bun run dev

# Lint
bun run lint

# Format
bun run format

# Type check
bun run check-types

# Run tests
bun run test
bun run test:watch

# Database
bun run db:push          # Apply schema changes (no migration files)
bun run db:migrate       # Run migration files
bun run db:studio        # Open Drizzle Studio GUI
bun run workspace [--reset] [--anomaly] [--events N] [--website <ID>]  # Local login, website and seeded analytics

# SDK (must build before dev if SDK changed)
bun run sdk:build

# Build everything
bun run build

Note: All commands use dotenv -- prefix internally to load .env — just run them from root.

Running a single test

# From root
cd apps/api && bun test path/to/test.ts

# Or with filter
cd apps/api && bun test --test-name-pattern "test name"

Initial Setup

bun install
cp .env.example .env
docker-compose up -d          # PostgreSQL, ClickHouse, Redis
bun run db:push
bun run clickhouse:init       # from packages/db
bun run sdk:build
bun run dev:dashboard

Architecture

Monorepo Structure

apps/
  dashboard/   # Next.js 16 frontend (React 19, TailwindCSS 4)
  api/         # Elysia.js backend (Bun, port 3001)
  basket/      # Analytics event ingestion service
  uptime/      # Uptime monitoring
  cron/        # Scheduled jobs
  links/       # Short link service
  docs/        # Documentation site
  status/      # Status page app
  slack/       # Slack integration

packages/
  db/          # Drizzle ORM schemas + clients (PostgreSQL + ClickHouse)
  rpc/         # ORPC router — type-safe API layer between dashboard and api
  auth/        # Better-Auth integration + permission system
  sdk/         # Public analytics SDK (React, Vue, Node.js)
  sdk-swift/   # Swift analytics SDK
  nuxt/        # Nuxt module for the SDK
  ui/          # Design system components (@databuddy/ui)
  ai/          # AI agent, query builders, MCP tools
  redis/       # Redis client, cacheable() caching, pub/sub, BullMQ queues, rate limiting
  shared/      # Shared types, utilities, constants
  validation/  # Zod schemas
  services/    # Business logic services
  email/       # Email via Resend
  notifications/ # Notification system
  tracker/     # Lightweight client-side tracking scripts
  mapper/      # Data transformation utilities
  env/         # Environment configuration (type-safe env vars)
  devtools/    # Browser devtools extension
  encryption/  # Encryption utilities
  api-keys/    # API key management and scopes
  test/        # Shared integration test infra (factories, contexts, assertions)
  migrate/     # Migrates competitor tracker attributes (Umami, Pirsch, Rybbit) to Databuddy

Data Flow

Browser (SDK/tracker) → basket (ingestion) → ClickHouse (analytics warehouse)
                                           → PostgreSQL (relational data)

Dashboard (Next.js) ←→ ORPC (rpc package) ←→ API (Elysia) → PostgreSQL + ClickHouse + Redis

Key Patterns

RPC Layer (packages/rpc): The central type-safe API contract between dashboard and api. Dashboard uses ORPC client with TanStack Query; API implements the procedures. Adding a new endpoint means defining it in rpc, implementing it in api, and calling it from dashboard.

Database Layer (packages/db): Single source of truth for all schemas. Uses Drizzle ORM for PostgreSQL (relational data: users, websites, settings) and a ClickHouse client for analytics data (events, sessions, pageviews). Schema changes use db:push for development; db:migrate for production migrations.

Caching (packages/redis): cacheable() wraps repeated lookups with positive + negative caching, single-flight dedup, stale-while-revalidate, and Redis fallback. Pass reviveDates: false when the cached value carries ISO-string timestamps validated by z.string() output schemas.

Auth (packages/auth): Better-Auth handles sessions. The package also contains the permission system used across all apps.

State management in Dashboard: Jotai for local UI state, TanStack Query for server state.

Tech Stack

  • Runtime: Bun 1.3.14+
  • Frontend: Next.js 16, React 19, TailwindCSS 4, Radix UI, Recharts
  • Backend: Elysia.js (Bun-native HTTP framework)
  • API layer: ORPC (type-safe RPC with OpenAPI generation)
  • Auth: Better-Auth
  • ORM: Drizzle ORM
  • Databases: PostgreSQL 17, ClickHouse 25.5, Redis 7
  • Validation: Zod 4
  • Linting/Formatting: Biome via Ultracite
  • Build: Turborepo + Bun

Code Style

  • Linter/Formatter: Ultracite (Biome-based). Run bun run lint / bun run format.
  • TypeScript: Strict mode. Always use proper types — avoid any.
  • Marketing copy: Never write staccato sentences on landing pages, docs, or any marketing copy. No runs of clipped fragments like "Fast. Private. Simple." or "No cookies. No banners. No guesswork." Write full sentences that connect the idea to what the reader gets.
  • Commit format: <type>(<scope>): <description> (e.g., feat(dashboard): add export button, fix(api): handle null session)
  • Commit slicing rule: Prefer one commit per coherent product or technical slice, not one giant snapshot and not ultra-fragmented file-by-file commits.
    • Split commits by intent: feature, bug fix, refactor, style/copy pass, or migration slice.
    • Use the dominant surface as scope: dashboard, api, rpc, basket, docs, db, sdk, tracker, deps, ci.
    • Group closely related UI files into one commit when they ship one visible change.
    • Keep unrelated surfaces in separate commits even if they were edited in the same session.
    • For broad migrations, follow the repo’s existing pattern: one commit per meaningful area, e.g. feat(dashboard): migrate home, events, insights, and links pages to DS primitives.
    • Before committing, check git diff --stat and git status --short; if the diff mixes unrelated intents, split it.
    • Only make a single snapshot commit for the whole worktree when the user explicitly asks to include everything as-is.
  • PRs: Open against staging branch (not main).

Tests

Runners: apps/api and apps/basket use vitest; every other package uses bun test. Import helpers from the runner the package actually runs (bun:test vs vitest); bun run lint:policies rejects the wrong one. Biome and most tsconfig.json files exclude test files, so format tests by hand (tabs, no blank line before a closing });) and keep imports honest.

Every test file must be reachable. A package that contains *.test.ts(x) needs a test script, and any file path named in a test* script must exist; lint:policies enforces both, and only counts a file as run when CI reaches it through the package's test or test:integration script. Env-gated integration files go in the package's test:integration script, which sets its own gate flags; CI runs every package's test:integration with one turbo run test:integration step, so never wire a suite as its own ci.yml step. A gated file nobody runs is dead, not "for later".

Integration tests use the shared services only. Postgres databuddy_test on 5432, Redis on 6379, ClickHouse on 8123, via import "@databuddy/test/env" as the first import. Never pin a test to a scratch container port; lint:policies rejects five-digit loopback ports in test files. A test that needs isolation uses its own organization, website, or table names and cleans them up.

Do not write these tests. They were removed in bulk once and will be removed again:

  • Constant snapshots: asserting a constant or config equals a literal copy of itself. Drift guards that compare two sources of truth are the exception.
  • Mock echo: mocking a dependency and asserting it was called with the arguments the test passed in.
  • Library behaviour: zod defaults, drizzle column config, ai-sdk retries, bun or node semantics.
  • Duplicates: add a case to the existing <module>.test.ts instead of creating <module>-extra.test.ts, <module>-boundary.test.ts, or a "fuzz" file that reruns fixed inputs.
  • .skip or .todo without an issue link, or a describe.skip that never turns on.
# Run every integration suite (requires Docker: postgres, redis, clickhouse)
bun run test:integration

# One package
cd apps/api && bun run test:integration

# One-time setup for test DB
cd packages/db && DATABASE_URL="postgres://databuddy:databuddy_dev_password@localhost:5432/databuddy_test" bunx drizzle-kit push

Key helpers from @databuddy/test:

  • signUp() — creates a real user via better-auth with session cookie
  • addToOrganization(userId, orgId, role) — inserts member row
  • userContext(user, orgId) / apiKeyContext(orgId, scopes) — builds RPC Context
  • insertOrganization() / insertWebsite() / insertApiKey() — DB factories
  • expectCode(promise, "FORBIDDEN") — asserts ORPCError code
  • reset() / cleanup() — truncate tables / close connections
  • import "@databuddy/test/env" — sets test env vars (must be first import)

Rules for integration tests:

  • Always import "@databuddy/test/env" as the first line
  • Use const iit = hasTestDb ? it : it.skip for graceful skip when Docker is down
  • Use userContext / apiKeyContext / expectCode from the test package — don't redefine locally
  • Type API key objects against Context["apiKey"] to catch schema drift
  • beforeEach(() => reset()) and afterAll(() => cleanup()) in every file
  • Business context lives in organization_business_contexts, never organization.metadata; fixtures that write metadata test nothing

Drift Prevention

  • Time series bucket in the site's timezone. Query builders bucket with toDate(toTimeZone(time, {timezone:String})) (or toStartOfHour), passing ctx.timezone || "UTC"; a bare toDate(time) buckets by UTC day and shifts evening traffic to the next day. Date-only ranges are bound in the same timezone: SimpleQueryBuilder turns toDateTime({startDate:String}) and toDateTime(concat({endDate:String}, ' 23:59:59')) into timezone-aware bounds for non-UTC requests, so write those forms and never convert dates by hand; totals and daily buckets then cover the same local days.
  • AI agent analytics has one source per list. The agent registry (ids, names, products, purposes) is packages/shared/src/utils/bot-detection/ai-agents.ts, generated from the vendored well-known-bots.json; the lists the dashboard needs (FEATURED_AI_PRODUCTS, CONTENT_FORMATS, AgentPurpose, ROBOTS_ACCESS) live in the lightweight bot-detection/types.ts, because importing ai-agents.ts into the dashboard pulls the 437 KB bot JSON into the bundle. Name crawlers and products from the registry by agent_id, never from the stored bot_name (rows written before the October 2026 detector fix can hold browser engines like "WebKit" or "unknown"). The logo list (AI_ICON_COLORS, resolved with aiProductIcon) also lives in bot-detection/types.ts and must match apps/dashboard/public/ai/*.svg and the email logos in apps/dashboard/public/ai/email/*.png; icon.test.ts enforces both and that every featured product has a logo.
  • Type test objects against their source type. Fake API keys must be typed as Context["apiKey"], fake users as User, etc. Tests are not type-checked, so this does not fail the build; it makes a partial fixture visible in review and in the editor instead of silently passing.
  • Never hand-write dependency versions. Use bun add <pkg> to add dependencies. Hand-written version ranges drift from lockfile reality and cause phantom resolution bugs.
  • Shared test helpers over local copies. expectCode, userContext, apiKeyContext, env setup — these live in @databuddy/test. If you're about to define a helper that already exists there, import it instead.
  • Scope maps must match. If RESOURCE_SCOPE_OVERRIDES changes in packages/api-keys/src/scopes.ts, the integration tests in link-handlers.test.ts and with-workspace.test.ts must be updated to match. The link resource is mapped there (read:links for read, write:links for create/update/delete), as is the flag resource (manage:flags for create/update/delete). Both are enforced solely by withWorkspace; there are no separate pre-check layers. New resources also need role grants in packages/auth/src/permissions.ts (statement plus each role).
  • Identity columns are governed by packages/db/src/clickhouse/identity.ts. PROFILE_ID_TABLES lists every ClickHouse table carrying profile_id; identity.test.ts enforces that each entry has a CREATE column, an idempotent migration, and profile_id + anonymous_id in the agent SQL allowlist. Adding profile_id to a table means adding it there and following the failing test. Query-side identity stitching must use EVENTS_VISITOR_KEY / CUSTOM_EVENTS_VISITOR_KEY / visitorMatch() from that module — never inline the expression. Analytics columns labelled visitors use uniq(anonymous_id) so every page reports the same number as the overview; uniq(session_id) counts sessions and must be labelled that way. Profile write semantics (trait splitting, upserts) live in @databuddy/services/identity; profile_id values are customer-supplied and are never salted, unlike anonymous_id.

AI Policy Note

The project has a formal AI usage policy (AI_POLICY.md). For contributions: all AI usage must be disclosed, PRs must reference an accepted issue, and all AI-generated code must be fully human-verified. Maintainers are exempt and may use AI at their discretion.