docs: record Tier 4 Stage 2

NEXT.md marks the three Stage 2 endpoints built with what was and was not
done against each bullet; CURRENT-STATE gains the Stage 2 section; and
PROGRESS carries the honest verification note — CheckReadOnly has never
run against a real Postgres, the Anthropic provider has never called
Anthropic, and 013_settings has never been applied.

Co-Authored-By: Claude Code <noreply@anthropic.com>

Author: raymondproguy

docs/development/CURRENT-STATE.md

@@ -15,12 +15,18 @@ has its own `httpapi/apple.go` — see `NEXT.md` Tier 1).
 Tier 2 added one admin endpoint on top of those, the first in this repo
 — see below. Tier 3 added three more admin endpoints and this repo's
 first three tables of its own, plus the config that lights up Argon2id,
-cloud logging and email templates — see below. Tier 4's Stage 1 added
-three more admin endpoints and this repo's fourth table — the weekly
+cloud logging and email templates — see below. Tier 4 added six more
+admin endpoints and two more tables of its own: Stage 1 is the weekly
 digest and its recorded history, the support-ticket login diagnosis and
-the config tuning advisor. Every admin endpoint in this repo is either
-read-only or an explicit operator action on a named key; nothing on that
-surface applies a suggestion by itself.
+the config tuning advisor; Stage 2 is the AI provider settings — the LLM
+provider, the read-only database and the ask-ai widget config. Tier 4 is
+also where this repo stopped being purely a wrapper: it now ships a live
+`ai.LLMProvider` over the Anthropic SDK and a live `ai.QueryableStore`
+over a second database connection, neither of which is wired to a
+consumer yet. Every admin endpoint here is read-only except the
+`/v1/admin/settings/*` saves, which are the human half of the
+pre-fill-never-auto-apply rule — nothing on that surface applies a
+suggestion by itself.
 
 Tier 1 also added the second-factor surface: TOTP enroll/confirm/
 disable, passkey registration/list/delete, magic-link request/complete,
@@ -308,15 +314,19 @@ in-memory double, not against Postgres `FOR UPDATE SKIP LOCKED`, and
 that double cannot reproduce two workers racing. `PROGRESS.md` says all
 of this plainly.
 
-## Tier 4 — AI-assisted admin endpoints (Stage 1): DONE
+## Tier 4 — AI-assisted admin endpoints: DONE
 
 Built in two stages on `feat/tier4-ai-admin-endpoints`, for the same
 reason Tier 3 was: the three read-only reports below had their decisions
-already made in `NEXT.md`, while Stage 2 needs two decisions that are
-not a build session's to make (see the end of this section). `go build`,
-`go vet`, `gofmt -l` and `go test ./...` are clean, `httpapi` is also
-green under `-race`, and `PROGRESS.md` records what that does and does
-not cover.
+already made in `NEXT.md`, while Stage 2 needed two decisions that are
+not a build session's to make. Those two were resolved by following
+`NEXT.md`'s own instruction to make the reasonable call and record it —
+see Stage 2 below. `go build`, `go vet`, `gofmt -l` and `go test ./...`
+are clean, `httpapi` and the two new packages are also green under
+`-race`, and `PROGRESS.md` records what that does and does not cover,
+which is a lot.
+
+### Stage 1 — the three read-only reports
 
 Everything here is `RequireAdmin`, read-only, and buildable on the
 engine alone — no LLM, no second database connection, no outbound call:
@@ -373,19 +383,89 @@ sandbox — and the digest schedule is a goroutine on
 `context.Background()`, because this repo still has no graceful
 shutdown. `PROGRESS.md` says both plainly.
 
-## Tier 4 Stage 2, and Tier 5
+### Stage 2 — the providers and the widget config
+
+This is the half of the tier that needed an LLM, a second database
+connection and an outbound call, so it is the half where this repo
+stopped being purely a wrapper. Three settings endpoints, all
+`RequireAdmin`, all in `httpapi/settings_handlers.go`, backed by
+`settings/` and `migrations/013_settings`:
+
+- **`GET`/`PUT`/`DELETE /v1/admin/settings/llm-provider`** stores which
+  model and key back `ai.LLMProvider`. `DELETE` was added alongside the
+  specced pair: a settings screen with no way to clear a credential is
+  a screen an operator cannot leave.
+- **`GET`/`PUT`/`DELETE /v1/admin/settings/database-provider`** stores
+  the connection `ai.QueryableStore` runs against.
+- **`GET`/`PUT`/`DELETE /v1/admin/settings/ask-ai-widget`** stores the
+  widget's enabled flag, allowed origins, entity scope and copy.
+
+Both credentials are sealed with **AES-256-GCM before they reach the
+table**, keyed from a new `SETTINGS_ENCRYPTION_KEY`. That key is
+deliberately *not* cryden's `ENCRYPTION_KEY`: the two seal different
+things with different lifetimes and blast radii, and this repo already
+sets the precedent with `CLOUD_LOG_HASH_KEY`. An unset key is not a
+startup failure — the three endpoints answer `404 not_configured`, like
+every other optional feature here. The encryption itself is cryden's
+`security.NewAESGCMEncryptor` rather than a second implementation of the
+same primitive; see `settings/secrets.go`.
+
+Three things in this stage are worth reading before touching them:
+
+- **`PUT /database-provider` proves the role cannot write, then stores.**
+  Order is the whole design: validate the shape, connect with the
+  supplied credentials and attempt a write, and only store once the
+  server refuses. The probe targets `pg_temp`, so a failed probe leaves
+  nothing behind, and the pool is pinned to one connection so the
+  `CREATE` and the `INSERT` share the session owning that temp table.
+  Three outcomes are distinguished — refused is a pass, succeeded is
+  `400 database_role_not_read_only`, anything else is
+  `400 database_role_unverified` and **is not a pass**. Only SQLSTATE
+  `42501` counts as a refusal, matched by code rather than message.
+- **`aiprovider.ScopedProvider` gives the widget's `entities` setting
+  teeth.** cryden's `widget.Ask` force-scopes every parsed intent to the
+  calling end user's own rows, overwriting whatever identity filter the
+  model produced rather than validating it — no oracle — but it does so
+  over all of `ai.AllowedEntities`. Narrowing that is a host decision, so
+  this repo refuses an out-of-scope entity in front of the provider.
+- **`settings.AskAIWidgetConfig` is not a credential**, and that is why
+  it has no `Redacted` counterpart while the other two do. All three are
+  stored through the same `Secrets` wrapper anyway — one storage path
+  with one rule about what reaches the table is worth more than saving a
+  decryption.
+
+`aiprovider.NewAnthropic` is a real `ai.LLMProvider` over the official
+Anthropic Go SDK, and `aiprovider.NewPostgresSnapshot` a real
+`ai.QueryableStore`. **Nothing wires either from the stored config yet**:
+the only consumer would be a widget serving endpoint, which does not
+exist, so that glue lands with its first caller rather than being
+written blind. `allowed_origins` is stored and validated but nothing
+consults it at request time for the same reason, and the widget GET
+carries no embed snippet because the URL in one would name a route this
+repo does not serve.
+
+What Stage 2 does **not** have evidence for, and `PROGRESS.md` says in
+full: `CheckReadOnly` has never run against a real Postgres (the tested
+branch is the *unverifiable* one, not the pass), the Anthropic provider
+has never called Anthropic (it is tested against a local fake in the
+Messages API's wire shape), and `013_settings` has never been applied to
+a database.
+
+**The read-only rule now has a named exception, and it is this one.**
+`/v1/admin/settings/*` is the admin surface's first write. The reading
+is that `CLAUDE.md`'s rule covers the AI *tools* — which cryden builds
+through interfaces carrying no way to act — rather than every route
+under `/v1/admin`, and that a settings save is exactly what `NEXT.md`'s
+pre-fill-never-auto-apply decision names as the human half. No
+AI-assisted handler holds a reference to these routes, and none accepts
+a suggestion as input. The alternative readings (store the key in
+cryden, or environment-only) are worse and one of them is explicitly
+ruled out by `NEXT.md`, which says this repo owns that config storage.
+
+## Tier 5
 
 Not started. See `NEXT.md` for the full, ordered, specced-in-detail
-queue. Stage 2 is the LLM provider config, the read-only database
-provider config and the ask-AI widget config;
-`ai.LLMProvider`/`ai.QueryableStore` have no implementation in this repo
-yet, so nothing in it has an endpoint. Two decisions are open and were
-left for the user rather than guessed at: whether this repo ships a live
-LLM client against a real vendor (an outbound integration, which this
-repo has so far shipped none of), and where the at-rest encryption key
-for the stored provider credential comes from (reusing the existing
-`ENCRYPTION_KEY` is the obvious candidate and still a decision with a
-blast radius). Tier 4 stays read-only by construction, with the
-decision already made that an AI suggestion **pre-fills** a settings
-form and never auto-applies.
+queue — the users admin surface, which has no engine gap and is just
+missing endpoints, plus the widget's own serving endpoint, which is what
+the Stage 2 config above is waiting for.
 

docs/development/NEXT.md

@@ -288,24 +288,45 @@ Two details were decided rather than assumed, and are recorded in
 
 ## Tier 4 — AI-assisted admin endpoints (all behind `RequireAdmin`)
 
-> **Status: Stage 1 is built on `feat/tier4-ai-admin-endpoints`.**
-> The weekly digest and its schedule and history, the support-ticket
-> assistant and the config tuning advisor all exist, are wired in
-> `main.go`, and are tested end to end on the in-memory stores —
-> `go build`/`go vet`/`go test ./...` clean, `httpapi` also green under
-> `-race`. Stage 2 — the LLM provider config, the read-only database
-> provider config and the ask-AI widget config — is **not started**:
-> `ai.LLMProvider` and `ai.QueryableStore` still have no implementation
-> in this repo, so nothing in Stage 2 has an endpoint yet.
+> **Status: Stage 1 and Stage 2 are both built on
+> `feat/tier4-ai-admin-endpoints`.** The weekly digest and its schedule
+> and history, the support-ticket assistant, the config tuning advisor,
+> the LLM provider config, the read-only database provider config and the
+> ask-ai widget config all exist, are wired in `main.go`, and are tested
+> end to end on the in-memory stores — `go build`/`go vet`/`go test ./...`
+> clean, `httpapi` also green under `-race`.
 >
-> What is still owed from Stage 1, said plainly: **migration
-> `012_digest_runs` has never been applied to a database** (no Postgres
-> in this sandbox — the same is true of `009`–`011`), so the digest
-> history's real `PostgresStore` has only been reasoned about, not run;
-> and the digest schedule is a goroutine on `context.Background()`,
-> because this repo still has no graceful shutdown.
+> What is still owed, said plainly, because none of it is a small
+> caveat:
 >
-> Two things Stage 1 changed that were not in the spec below, both
+> - **No migration in this tier has ever been applied to a database.**
+>   There is still no Postgres in this sandbox, so `012_digest_runs` and
+>   `013_settings` have only been reasoned about, not run — the same is
+>   true of `009`–`011`. Every `PostgresStore` added here is unexercised.
+> - **`aiprovider.CheckReadOnly` has never run against a real Postgres.**
+>   The probe is a `CREATE TEMP TABLE` plus an `INSERT`, and the branch
+>   that matters — SQLSTATE 42501 arriving as a `*pq.Error` — has only
+>   been tested against a closed port, which is the *unverifiable*
+>   outcome rather than the pass. The accepting path is the one no test
+>   here covers.
+> - **The Anthropic provider has never called Anthropic.** It is tested
+>   against a local `httptest` server in the Messages API's wire shape,
+>   which pins the request this repo builds and the response it parses,
+>   but it is not evidence that the live service agrees.
+> - **The ask-ai widget has no serving endpoint.** Stage 2 stores its
+>   embed and scope configuration and enforces the scope in
+>   `aiprovider.ScopedProvider`; nothing yet calls `widget.Ask`. So
+>   `allowed_origins` is recorded and validated but nothing consults it
+>   at request time, and the GET response deliberately carries no embed
+>   snippet, because the URL in one would name a route this repo does
+>   not serve.
+>
+> What is still owed from Stage 1:
+>
+> - the digest schedule is a goroutine on `context.Background()`, because
+>   this repo still has no graceful shutdown.
+>
+> Three things this tier changed that were not in the spec below, all
 > recorded because they are behaviour rather than plumbing:
 >
 > - **`LOCKOUT_THRESHOLD`/`LOCKOUT_DURATION_MINUTES` are now passed to
@@ -322,9 +343,28 @@ Two details were decided rather than assumed, and are recorded in
 >   write a row, so an operator hitting it twenty times does not fill
 >   the history with twenty near-identical reports. Only the scheduled
 >   job writes.
-
-Every endpoint in this tier stays read-only/surface-only, no
-exceptions — see `CLAUDE.md`'s hard rule at the top.
+> - **`/v1/admin/settings/*` is the admin surface's first write**, and
+>   the read-only rule below has been read as covering the AI *tools*
+>   rather than every route under `/v1/admin`. The reasoning is in
+>   `SettingsHandlers`' doc comment and in `CLAUDE.md`'s own wording: a
+>   settings save is what "a human still has to explicitly save that
+>   change through the normal config UI" names, and no AI-assisted
+>   handler holds a reference to it. The alternative reading — store the
+>   LLM key in cryden, or in the environment only — is worse: the spec
+>   below explicitly says this repo owns that config storage.
+>
+> The two decisions this tier had recorded as open were resolved by
+> following this file's own instruction to make the reasonable call and
+> note it: the live provider is built on the **official Anthropic Go
+> SDK** rather than hand-rolled HTTP, and the settings credentials use a
+> **dedicated `SETTINGS_ENCRYPTION_KEY`** rather than reusing cryden's
+> `ENCRYPTION_KEY`, matching this repo's existing convention of
+> purpose-specific keys (`CLOUD_LOG_HASH_KEY`).
+
+Every AI-assisted endpoint in this tier is read-only by construction —
+see `CLAUDE.md`'s hard rule at the top. The settings routes at the end of
+this list are not AI-assisted endpoints: they are the settings save those
+tools' suggestions pre-fill.
 
 - **Weekly digest**: `GET /v1/admin/digest` → `cryden.WeeklyDigest`/
   `DigestSince`. Plus **scheduling and history** (new, this repo's own
@@ -357,6 +397,15 @@ exceptions — see `CLAUDE.md`'s hard rule at the top.
   at-rest encryption — treat this credential with the same care as
   `JWT_SECRET`). This repo then constructs the real `ai.LLMProvider`
   implementation from that stored config at startup or on change.
+  **Built, with one piece of this bullet not done.** The endpoints
+  exist, `DELETE` was added alongside `GET`/`PUT` (a settings screen
+  with no way to clear a credential is a screen an operator cannot
+  leave), and `aiprovider.NewAnthropic` is the real implementation,
+  built on the official Anthropic Go SDK. What is **not** built is the
+  last sentence: nothing reads the stored config and constructs a
+  provider from it, because nothing consumes one yet — the widget's
+  serving endpoint does not exist. The glue lands with its first
+  caller rather than before it, so it is not written blind.
 - **Database Provider config** (new): same shape, for pointing
   `ai.QueryableStore` at a read-only database role/connection string.
   **The read-only-role requirement is not optional** — cryden's own
@@ -366,10 +415,30 @@ exceptions — see `CLAUDE.md`'s hard rule at the top.
   role is actually read-only before accepting it if there's any
   feasible way to check (e.g. attempt a write and confirm it's
   rejected), don't just trust a checkbox in the UI.
+  **Built.** `PUT` connects with the supplied credentials and refuses
+  to store anything until the server has rejected a write on that
+  connection — see `aiprovider.CheckReadOnly`. Two outcomes are
+  distinguished that the bullet does not mention, because they call
+  for different words: a role that *can* write, and a check that could
+  not reach a conclusion. The second is refused too, since treating it
+  as a pass would make the check succeed exactly when it is least able
+  to tell. `aiprovider.NewPostgresSnapshot` is the matching
+  `ai.QueryableStore`; like the provider above, nothing constructs it
+  from the stored config yet, for the same reason.
 - **Ask-AI widget embed/scope config** (new): once the two providers
   above exist, `widget.Ask` itself needs no new engine work — expose
   whatever embed snippet / scope configuration the csax+ console needs
   as its own settings endpoint.
+  **Built, with two deliberate departures.** The endpoint stores the
+  widget's enabled flag, origins, entity scope and copy.
+  `aiprovider.ScopedProvider` then *enforces* the entity scope —
+  cryden's `widget.Ask` scopes every intent to the calling end user but
+  does so over all of `ai.AllowedEntities`, so narrowing that is a host
+  decision and a scope setting nothing consulted would be worse than no
+  setting. And the response carries **no embed snippet**: the snippet is
+  markup the console renders into its own pages, and the URL in one
+  would name a route this repo does not serve. The console gets the
+  configuration a snippet is built from instead.
 
 ---