Merge pull request #10 from crydensync/feat/tier6-sqlite-backend

Feat/tier6 sqlite backend

Author: raymondproguy

.env.example

@@ -1,4 +1,9 @@
+# Which backend. Set exactly one — starting with both or neither is a
+# startup error. DATABASE_URL runs everything; SQLITE_PATH runs core auth
+# only, with the whole admin console answering 501 (its tables are
+# Postgres-only). See README's "The two backends".
 DATABASE_URL=postgresql://postgres.xxxxxxxx:your-password@aws-1-eu-west-3.pooler.supabase.com:5432/postgres?sslmode=require
+# SQLITE_PATH=/var/lib/cryden/api.db
 JWT_SECRET=
 CORS_ORIGINS=http://localhost:5173,https://yourapp.com
 PORT=8080

README.md

@@ -7,7 +7,7 @@ Every consumer talks to this over plain HTTP — no Go required. This is what a
 ## Prerequisites
 
 - Go 1.22+ (check `go.mod` for exact version)
-- A running Postgres instance (local, Docker, or hosted — e.g. Supabase, Neon, RDS)
+- A running Postgres instance (local, Docker, or hosted — e.g. Supabase, Neon, RDS) — or nothing but a writable path, if you run on SQLite (see [The two backends](#the-two-backends))
 
 ## Getting started
 
@@ -20,6 +20,8 @@ go run .
 
 Run the migrations in `migrations/` against your database first, in order (copies of CrydenSync's own migrations, kept here so this repo is self-contained for local dev and CI — same as `typebook` keeps its own copy). `002_oauth_identities` is required even if you don't use OAuth yet — `NewOAuthStore` is wired into the engine config unconditionally. `004` through `008` are the TOTP, WebAuthn, recovery-code, login-attempt and API-key tables; run them even if you leave `ENCRYPTION_KEY` unset, since `007` is what the engine's credential-stuffing detection reads once Tier 2 wires it up and `008` is what the API-key work will use.
 
+That paragraph is the Postgres path only. On SQLite there is nothing to run by hand — `main.go` calls cryden's own `sqlite.Migrate` at startup. See [The two backends](#the-two-backends).
+
 OAuth is optional. To enable a provider, set its client ID/secret plus `BASE_URL` (used to build the callback URL registered in that provider's console):
 
 ```
@@ -57,6 +59,35 @@ APPLE_PRIVATE_KEY=-----BEGIN PRIVATE KEY-----\nMIG...\n-----END PRIVATE KEY-----
 
 All four `APPLE_*` values are required; a partially configured Apple is simply unavailable, like any other unconfigured provider.
 
+## The two backends
+
+This API runs on Postgres or on SQLite, chosen by exactly one environment variable:
+
+```
+DATABASE_URL=postgres://...      # everything
+SQLITE_PATH=/var/lib/cryden/api.db   # core auth only
+```
+
+Setting both is a startup error, and so is setting neither. That is deliberate rather than a convenience: `DATABASE_URL` is what the admin console needs and `SQLITE_PATH` is what the store wiring reads, so quietly preferring one would run a deployment on a backend its own configuration does not describe.
+
+**SQLite runs core auth and nothing under `/v1/admin`.** Signup, login, refresh, sessions, password change, email change, OAuth, TOTP, passkeys, recovery codes, magic links and API keys all work. Every admin route — the whole console: the user surface, metadata, webhook and log history, the digest, support diagnosis, config tuning, flagged-event review, AI settings — answers:
+
+```json
+{"error": {"code": "not_implemented_on_sqlite", "message": "the admin console requires a Postgres backend; this deployment runs on SQLite"}}
+```
+
+with `501`, before the token is looked at. This is one decision applied in one place (`httpapi.AdminOnly`), not a list of routes to maintain: the admin console's tables are this repo's own and Postgres-only, and `RequireAdmin` itself depends on the `operators` table, so there is no partial console to offer and no way for a SQLite deployment to have an operator at all. A `403 not_operator` would have been the easy answer and the wrong one — it tells a legitimate operator they personally lack access when the truth is that this backend has no console.
+
+The one AI surface that is *not* under `/v1/admin` is `POST /v1/ask-ai`, which serves the widget to a signed-in end user. It is still unavailable on SQLite, because the provider behind it lives in the Postgres-only `settings` table: it answers `404 not_configured`, the same shape it gives on a Postgres deployment that has not configured a provider. One rule, stated once: **the AI-assisted surface needs `DATABASE_URL`.**
+
+These variables are accepted but inert on SQLite, and the server says so at startup rather than letting an operator wonder — `SETTINGS_ENCRYPTION_KEY`, `WEBHOOK_URL`, `CLOUD_LOGGING`, `DIGEST_INTERVAL_HOURS`. `ENCRYPTION_KEY` is *not* in that list: TOTP and passkeys are cryden's own tables and work on both backends.
+
+No migration step exists on SQLite. `main.go` calls cryden's own `sqlite.Migrate` at startup, which embeds its migrations and records what it applied, so a second boot is a no-op. The copy of those files under `migrations/sqlite/` is reference material, not what runs — see the README in that directory.
+
+The connection is opened with three pragmas, all of them load-bearing: `foreign_keys(1)` (off by default, so the schema's `ON DELETE` clauses would silently not run), `busy_timeout(5000)` (zero by default, so a concurrent writer gets an immediate `SQLITE_BUSY` instead of waiting), and `journal_mode(WAL)`. The server verifies the first two on every boot with cryden's own `CheckPragmas` and refuses to start if the DSN and the driver have drifted apart.
+
+**Backing up a SQLite deployment means copying `api.db`, `api.db-wal` and `api.db-shm` together**, or checkpointing first. With WAL, recent writes — including, on a fresh deployment, the entire schema — live in the `-wal` file until a checkpoint folds them into the main file, and there is no graceful shutdown here yet to force one on exit. Copying `api.db` alone can silently produce an empty database.
+
 ## Second factors
 
 TOTP and passkeys are optional and all-or-nothing on `ENCRYPTION_KEY`: cryden refuses to construct an engine with a TOTP or WebAuthn store set and no encryption key (a TOTP secret must be recoverable in plaintext to check a code, so it is encrypted rather than hashed). With the key unset, both methods answer `404 totp_not_configured` / `404 passkeys_not_configured` per request, the same shape an unconfigured OAuth provider uses, rather than the server refusing to start.
@@ -209,6 +240,8 @@ Authenticated endpoints expect `Authorization: Bearer <access_token>`.
 
 Everything under `/v1/admin` requires an **operator** token: a valid access token whose `role` claim is `admin`. Operator status is this repo's own concept, not cryden's — it lives in its own `operators` table (`migrations/003_operators.*.sql`), and the claim is attached to the token at issue time by the `AccessTokenClaims` provider in `main.go`. An ordinary user's token carries no `role` claim at all, so "revoked operator", "never was one" and "no such user" are indistinguishable to a caller, deliberately.
 
+On a SQLite deployment this entire section is unavailable and every route in it answers `501 not_implemented_on_sqlite` — the console's tables are Postgres-only. See [The two backends](#the-two-backends).
+
 The first operator is created with `cmd/grant-operator`, from a machine with direct database access — deliberately not an HTTP bootstrap route, which would be needless attack surface reachable over the network:
 
 ```

config/config.go

@@ -14,7 +14,10 @@ import (
 )
 
 type Config struct {
+	// DatabaseURL and SQLitePath select the backend and are mutually
+	// exclusive — Load refuses both or neither. See UsesSQLite.
 	DatabaseURL         string
+	SQLitePath          string
 	JWTSecret           string
 	Port                string
 	CORSOrigins         []string
@@ -294,6 +297,19 @@ const (
 	CloudLogRedactionHash = "hash"
 )
 
+// UsesSQLite reports which backend this deployment runs on.
+//
+// It is a method rather than a field so there is exactly one expression
+// of the rule, and every place that needs to branch — main.go's store
+// wiring, RequireAdmin's 501 — asks the same question rather than
+// re-deriving it from SQLitePath and drifting.
+//
+// A SQLite deployment serves core auth only. The admin console's tables
+// are this repo's own and Postgres-only by decision (see NEXT.md Tier
+// 6), and RequireAdmin itself depends on the operators table, so the
+// whole of /v1/admin is unavailable rather than parts of it.
+func (c Config) UsesSQLite() bool { return c.SQLitePath != "" }
+
 // Load reads .env (if present, filling only gaps — real env vars
 // always win) then reads the actual environment. No external
 // dependency for .env parsing — same minimal-loader approach as csax.
@@ -302,6 +318,7 @@ func Load() (Config, error) {
 
 	cfg := Config{
 		DatabaseURL: os.Getenv("DATABASE_URL"),
+		SQLitePath:  os.Getenv("SQLITE_PATH"),
 		JWTSecret:   os.Getenv("JWT_SECRET"),
 		Port:        os.Getenv("PORT"),
 	}
@@ -328,8 +345,11 @@ func Load() (Config, error) {
 		cfg.AccessTokenTTL = time.Duration(n) * time.Minute
 	}
 
-	if cfg.DatabaseURL == "" {
-		return cfg, fmt.Errorf("DATABASE_URL is required")
+	if cfg.DatabaseURL == "" && cfg.SQLitePath == "" {
+		return cfg, fmt.Errorf("one of DATABASE_URL or SQLITE_PATH is required — see README's note on the two backends")
+	}
+	if cfg.DatabaseURL != "" && cfg.SQLitePath != "" {
+		return cfg, fmt.Errorf("DATABASE_URL and SQLITE_PATH are mutually exclusive — set one; the admin console's tables are Postgres-only, so a SQLite deployment serves core auth and nothing under /v1/admin")
 	}
 	if cfg.JWTSecret == "" {
 		return cfg, fmt.Errorf("JWT_SECRET is required")

config/config_test.go

@@ -52,6 +52,13 @@ var tieredEnvVars = []string{
 func loadForTest(t *testing.T, env map[string]string) (Config, error) {
 	t.Helper()
 	t.Setenv("DATABASE_URL", "postgres://user:pw@localhost/db")
+	// Cleared as well as set, so this helper describes exactly one thing:
+	// a Postgres deployment. Without it a SQLite_PATH set by an earlier
+	// call in the same test would survive into the next one and turn it
+	// into the mutually-exclusive case by accident. A caller wanting the
+	// other backend passes DATABASE_URL:"" and a SQLITE_PATH, which the
+	// env map below applies last.
+	t.Setenv("SQLITE_PATH", "")
 	t.Setenv("JWT_SECRET", "test-secret")
 	t.Setenv("CORS_ORIGINS", "http://localhost:5173")
 	for _, name := range tieredEnvVars {
@@ -404,6 +411,85 @@ func TestTier3WebhookMaxAttemptsIsBounded(t *testing.T) {
 	}
 }
 
+// Tier 6: the backend is chosen by exactly one variable, and Load refuses
+// every other combination at startup rather than letting a deployment come
+// up pointed at neither or at both.
+//
+// Both refusals matter for different reasons. Neither set is a deployment
+// that would otherwise reach a nil *sql.DB somewhere deep in startup. Both
+// set is the dangerous one: DATABASE_URL is what the admin console needs
+// and SQLITE_PATH is what the store wiring would read, so silently
+// preferring either would run a deployment on the wrong backend while its
+// configuration said otherwise.
+func TestTier6TheBackendIsSelectedByExactlyOneVariable(t *testing.T) {
+	cfg, err := loadForTest(t, map[string]string{
+		"DATABASE_URL": "",
+		"SQLITE_PATH":  "/var/lib/cryden/api.db",
+	})
+	if err != nil {
+		t.Fatalf("SQLITE_PATH on its own was rejected: %v", err)
+	}
+	if !cfg.UsesSQLite() {
+		t.Error("UsesSQLite() = false with SQLITE_PATH set")
+	}
+	if cfg.SQLitePath != "/var/lib/cryden/api.db" {
+		t.Errorf("SQLitePath = %q, want the value that was set", cfg.SQLitePath)
+	}
+
+	// loadForTest sets DATABASE_URL and no SQLITE_PATH: the Postgres
+	// deployment every other test in this file already describes.
+	cfg, err = loadForTest(t, nil)
+	if err != nil {
+		t.Fatalf("Load() failed with DATABASE_URL and no SQLITE_PATH: %v", err)
+	}
+	if cfg.UsesSQLite() {
+		t.Error("UsesSQLite() = true on a deployment with no SQLITE_PATH")
+	}
+
+	for name, tc := range map[string]struct {
+		env  map[string]string
+		want string
+	}{
+		"neither backend": {
+			map[string]string{"DATABASE_URL": ""},
+			"one of DATABASE_URL or SQLITE_PATH is required",
+		},
+		"both backends": {
+			map[string]string{"SQLITE_PATH": "/var/lib/cryden/api.db"},
+			"mutually exclusive",
+		},
+	} {
+		t.Run(name, func(t *testing.T) {
+			_, err := loadForTest(t, tc.env)
+			if err == nil {
+				t.Fatalf("%v was accepted, want a startup failure", tc.env)
+			}
+			if !strings.Contains(err.Error(), tc.want) {
+				t.Errorf("error = %q, want it to contain %q", err, tc.want)
+			}
+		})
+	}
+}
+
+// UsesSQLite is the single expression of the rule, so it is pinned
+// directly: an empty string is "unset", which is what the mutual-exclusion
+// check above treats it as, and a Config built by hand rather than by Load
+// has to answer the same way.
+func TestTier6UsesSQLiteIsTheEmptyCheck(t *testing.T) {
+	if (Config{}).UsesSQLite() {
+		t.Error("a zero Config reports SQLite")
+	}
+	if !(Config{SQLitePath: "api.db"}).UsesSQLite() {
+		t.Error("a Config with SQLitePath set reports Postgres")
+	}
+	// A Postgres URL with no path set is the Postgres backend — the case
+	// that would break if this ever became "SQLitePath == '' means
+	// Postgres OR ..." rather than a plain emptiness check.
+	if (Config{DatabaseURL: "postgres://localhost/db"}).UsesSQLite() {
+		t.Error("a Config with only DatabaseURL set reports SQLite")
+	}
+}
+
 // The Tier 4 defaults: cryden's own lockout numbers restated, because the
 // engine takes them straight off its config with no defaulting of its own
 // and both zero values are wrong in the same direction — a zero threshold

docs/development/CURRENT-STATE.md

@@ -32,6 +32,14 @@ suggestion by itself. The widget's serving endpoint is not an admin
 endpoint at all, and is the one AI-assisted surface here that answers an
 end user rather than an operator.
 
+Tier 6 made the database a choice: this API runs on Postgres or on
+SQLite, picked by exactly one of `DATABASE_URL` and `SQLITE_PATH`. A
+SQLite deployment serves core auth and nothing else — the whole admin
+console answers `501 not_implemented_on_sqlite`, because every table it
+reads is Postgres-only. See that section below; the one-sentence version
+is that a small deployment can now skip Postgres entirely, and skipping
+it costs the console.
+
 Tier 1 also added the second-factor surface: TOTP enroll/confirm/
 disable, passkey registration/list/delete, magic-link request/complete,
 recovery-code generation, and the three public completion endpoints a
@@ -659,3 +667,97 @@ live path from a question to a real model is exercised only through the
 `Providers` seam with doubles; the wire shape is covered by
 `aiprovider`'s own tests against a local fake.
 
+## Tier 6 — SQLite backend, core auth only
+
+Built on `feat/tier6-sqlite-backend`. This repo runs on Postgres or on
+SQLite, chosen by exactly one variable. The scope decision was made in
+advance rather than here — `NEXT.md`'s Tier 6 section carries it — and
+this is the resulting state.
+
+**The switch.** `DATABASE_URL` or `SQLITE_PATH`, mutually exclusive,
+with `config.Load` refusing both or neither as a startup error.
+`Config.UsesSQLite()` is the single expression of the rule; `main.go`'s
+`openStores` returns one `stores` struct built from cryden's
+`store/postgres` or `store/sqlite` constructors, and nothing downstream
+of it knows which ran. The ten engine stores (`Users`, `Sessions`,
+`Audit`, `Verifications`, `OAuth`, `TOTP`, `WebAuthn`, `RecoveryCodes`,
+`APIKeys`, `Anomalies`) exist in both cryden packages, so this is wiring
+rather than engine work — the one asymmetry is this repo's own three
+Postgres-only stores, which are nil on SQLite.
+
+**The whole admin console answers `501 not_implemented_on_sqlite`**, and
+that is one decision in one place rather than a list of routes:
+`httpapi.AdminOnly` is `RequireAdmin` on Postgres and a flat 501 on
+SQLite, and all 25 admin registrations go through the value it returns.
+The reason it is a 501 and not the 403 the existing gate would have
+produced is the part worth keeping: `RequireAdmin` depends on the
+`operators` table, so a SQLite deployment has no operators, so *every*
+caller — including a legitimate operator — would have been told
+`403 not_operator`. That reads as "you personally lack access" when the
+truth is "this backend has no console". 501 is a statement about the
+deployment, which is what this is.
+
+- The console's tables (`operators`, `user_metadata`,
+  `webhook_deliveries`, `shipped_log_events`, `digest_runs`, `settings`,
+  `reviewed_anomalies`) remain Postgres-only, by the scope decision.
+  Nothing was built twice.
+- **`POST /v1/ask-ai` is not under `/v1/admin` and is still unavailable
+  on SQLite** — the provider it reads lives in the `settings` table. It
+  answers `404 not_configured`, the same shape it gives on a Postgres
+  deployment with no provider stored, so a client never has to know
+  which backend it is talking to. One rule: the AI-assisted surface
+  needs `DATABASE_URL`.
+- **The `AccessTokenClaims` provider is skipped entirely on SQLite.**
+  This is not an optimisation. `usermeta.ClaimsProvider` dereferences
+  its metadata store on every login, and a nil `*usermeta.PostgresStore`
+  passed through the interface is a non-nil interface holding a nil
+  pointer — it would pass its own nil check and panic on the first
+  query, on every login and every refresh. `claimsProvider` returns nil
+  for the SQLite case, which is the correct answer rather than a
+  workaround: cryden treats a nil provider as "this host attaches no
+  extra claims", and on SQLite there is nothing to attach. The
+  consequence is the same fact as the 501 seen from the other end — a
+  SQLite deployment issues tokens no admin route would accept anyway.
+- **Nothing is inert silently.** `SETTINGS_ENCRYPTION_KEY`,
+  `WEBHOOK_URL`, `CLOUD_LOGGING` and `DIGEST_INTERVAL_HOURS` are named
+  in a startup warning when set on SQLite. `ENCRYPTION_KEY` gets its own
+  positive log line precisely so it cannot be misread as part of that
+  list: second factors are cryden's own tables and work on both
+  backends.
+
+**Migrations on SQLite are cryden's, not this repo's.** This is the
+discovery that shaped the tier: cryden ships `sqlite.Migrate`, which
+embeds its own `migrations/*.sql` and records what it applied in
+`cryden_schema_migrations`, so `main.go` calls that at startup and there
+is no migrate step for an operator on this backend. The copy under
+`migrations/sqlite/` — cryden's `0001`–`0007`, verbatim, cryden's
+filenames kept rather than renumbered into this repo's `001`–`014`
+Postgres sequence — is therefore **reference material, not what runs**.
+The mirror image of Postgres, where this repo's copies are exactly what
+an operator pipes through `psql`. `migrations/sqlite/README.md` says so
+at the point of use. Three DSN pragmas are load-bearing:
+`foreign_keys(1)`, `busy_timeout(5000)` and `journal_mode(WAL)`; the
+server checks the first two on every boot with cryden's own
+`CheckPragmas` and refuses to start if the DSN and the driver have
+drifted apart.
+
+**What is still owed, said plainly.** The Postgres path is **not**
+newly verified: `openStores`'s Postgres arm is asserted to construct
+every store, but no Postgres was reachable in this environment, so
+migrations `001`–`014` still have never been applied to a real database
+and `anomalyreview.PostgresStore` still has never run — the same
+constraint Tiers 4 and 5 recorded. The SQLite path, by contrast, is
+verified end to end: cryden's own `store/sqlite` suite passes (run as
+the spec asked, as reference for the pragmas and type mappings), and
+this repo's server was started on a real SQLite file and passed the full
+`internal/smoketest` run — health, signup, duplicate rejection, login,
+wrong password, verify, session list, missing-header rejection, refresh
+rotation, reuse detection, family revocation, and both OAuth refusals.
+`-race` was still not run. Graceful shutdown is still unbuilt, and on
+SQLite it now has a second reason to exist: with no `Close()` there is
+no checkpoint on exit, so a fresh deployment's entire schema can sit in
+the `-wal` file — durable, but a backup that copies `api.db` alone can
+silently produce an empty database. `README.md` warns about that where
+an operator will see it. Per-user rate limiting on `POST /v1/ask-ai` is
+unchanged.
+