WS-AI-2: the AI cost governor bites (per-key rate limit + fail-loud budget) - #16
Merged
Arcoders merged 50 commits intoJul 5, 2026
Conversation
The AI cost governor's one genuinely new mechanism: a per-key request rate limit, the DIFFERENT rail from the token cost reserve. The reserve caps token spend; this caps requests-per-window on a provider key, so a tenant flooding a shared or BYOK provider account is throttled (denial of wallet). They compose without double-counting. - AiRateLimiter (new service) wraps the kernel's sliding-window consumeRateLimit behind an injected seam, so the gateway never value-imports the eager /services barrel. Bucket key ext:ai:<op>:<tenant>:<keyFingerprint>: the tenant is baked in so one tenant can never exhaust another's window, and the fingerprint scopes the limit to the specific provider key so a future per-tenant BYOK key gets its own budget with no code change. - Fail-closed by construction: over the window is a 429 (rate_limited); a limiter-backend outage is a 503 (rate_limit_unavailable), never a silent pass, because a cost surface must not run unmetered when the limiter is blind. Both fire pre-flight in the controller (after provider selection, before the reserve), so the AIException's status surfaces with headers unsent. A replay served from cache returns before this, so it never consumes the rate budget. - A policy denial rides the public IsthmusGuardTripped channel via a new guard.ai_rate_limited registry entry (severity warn: rate limits trip in normal operation, monitored by rate not per event); an outage does not (a dependency outage, classified like the reserve rail's). The emission matrix gains its recipe (a new guard without a behavioral test stays a compile error). - provider.keyFingerprint: a one-way SHA-256 of the active key (never the key), on HttpAiProvider, feeding the bucket key and, later, the audit attribution (#4 "provider + key fingerprint, never the key"). - config.ai.rateLimit { limit, windowSeconds } (absent = no per-key limit, the reserve stays the only cap), validated at boot. Redis is resolved through the app container's 'redis' binding, so the satellite adds no direct redis dependency and shares the host's single RedisManager. Tests: the limiter (key shape, 429+guard emit, 503 fail-closed outage, no-op without a policy); the controller (rate-limit before reserve -> 429 with the provider never called; a replay skips the rate limit). Gates: AI unit+arch 219, AI integration 10 (+3 skip) vs real PG/Redis, typecheck all workspaces, eslint.
Close the production footgun commit 09809a8 named: with no per-plan limits.aiTokens and no plans.operatorCeiling.aiTokens, the kernel reserve returns an inert handle and the AI endpoint runs unmetered (denial of wallet), silently. The cost governor now fails loud, not silent. - aiTokensBudgetPosture (pure, static-config read): healthy when an operator ceiling meters every tenant; an info nudge when plans budget aiTokens but no ceiling guards denial of wallet; an info when the quota is unbudgeted but acknowledged or a dynamic getPlan / tenant_plans could budget it at runtime; and a warn only for the genuinely unbudgeted, not-acknowledged, non-dynamic case. - ai_budget doctor check surfaces the live posture via tenant:doctor. A boot warning fires only for the warn posture. - No hard mount block on purpose: a host that budgets aiTokens through a dynamic getPlan is invisible to a static boot read, so refusing to mount would false-positive it. The doctor check is run-time-aware; the boot side stays advisory. config.ai.acknowledgeUnbudgetedAiTokens silences the warning while keeping the accepted risk on the doctor's radar. Tests: the posture matrix (healthy ceiling, per-plan-no-ceiling info, unbudgeted warn, acknowledged info, dynamic-plans info) and the check's issue mapping; the integration run registers and executes ai_budget. Gates: AI unit+arch 226, AI integration 11 (+3 skip) vs real PG/Redis, typecheck all workspaces, eslint.
Close the gap the Understand map named: every AI cost test used FakeQuota, so nothing proved the wired aiTokens path actually meters. Integration specs against real Redis drive the gateway spine with the real QuotaService: - a stream whose worst case exceeds a configured limits.aiTokens resolves failed_preflight over_budget with headers unsent and no bytes; - a within-budget stream completes and settles exactly the used tokens; - the operator ceiling bites even with no per-plan limit (denial of wallet), committing neither the tenant nor the operator hold (both-or-neither); - the BYOK per-key rate limit throttles at the window: the limit-plus-one hit is a 429 and the ext:ai:<op>:<tenant>:<keyFingerprint> bucket recorded it. The spine is built with the real quota and no breaker on purpose: a breaker preflight fast-fails a fake tenant with no schema, which would mask the quota rail these specs isolate. The kernel already proves reserve/settle/release, handle tamper and the ceiling flood under @ai-tagged fault specs. Gates: AI integration 15 (+3 skip) vs real PG/Redis, typecheck all workspaces, eslint.
Upgrade the ai.md cost section to the full WS-AI-2 surface: budgeting aiTokens with a per-plan limit and the operator ceiling (denial-of-wallet backstop, with a config example), the ai_budget doctor check plus the acknowledge flag for an unbudgeted quota, and a new "Rate-limit the provider keys" section for config.ai.rateLimit (the per-key request rate, a different rail from the token budget). The guard-events list gains the per-key rate limit. CHANGELOG folds the WS-AI-2 additions into [1.0.0].
Owner
Author
Adversarial review: cleanRan a 6-lens adversarial review over the WS-AI-2 diff (each finding would be refuted by 3 independent skeptics, confirmed only on majority). All six finders completed a genuine investigation and raised zero findings.
Not a false green: these same finder lenses surfaced a real bug (the fatal-refusal 503 downgrade) on the gateway PR, so an empty result here reflects a genuinely clean diff, not finders that failed to run. ~525k tokens of review across the six agents. No code change from the review. WS-AI-2 stands as pushed. |
…3, C1) The non-streaming AIEmbeddingProviderContract (input string[] -> number[][] with a bound model+dimension and provider-reported tokens), the deterministic MockEmbeddingProvider (stable-hash vectors, recorded calls, abort-aware), and the checkAIEmbeddingProviderConformance mirror of the chat conformance check. Exported from the root barrel (types) and ./testing (mock + conformance).
…, C2) A generic OpenACompatibleEmbeddingProvider (BYOK baseUrl/model, POST /embeddings with encoding_format:float, over the pinned safeFetch) plus a pure openai_embeddings parser that reorders vectors by index, rejects a mixed-dimension or non-finite body as provider_unavailable, and reads usage.total_tokens for the reserve settle. A SafeFetchError maps to byok_endpoint_blocked; a non-2xx to rate_limited/provider_unavailable. The G12 model gate is extracted into a shared model_allowlist.assertModelAllowed so the streaming and embedding families run one registered guard site (guard.ai_model_allowlist repointed there). A self-skipping real-API smoke exercises the live endpoint when AI_EMBEDDING_* is set.
…d (WS-AI-3, C3)
The ai_embeddings per-tenant migration (SEAM-2, I1): a runnable migration that
lands in whatever placement the active driver reports (bare table name, no
tenant_<id>, no withSchema('backoffice')). It creates the vector(N) column at
the config-driven dimension (validated 1..2000), an hnsw cosine index, a
(model,dim) read-filter index, and UNIQUE(source,content_hash) for idempotent
ingestion. Provenance columns (source, actor, created_at) make a poisoned batch
reversible (#3).
Build wiring (the AI package is the first satellite to ship perTenantMigrations):
tsconfig include += tenant_migrations, manifest perTenantMigrations pointing at
the tsc output. A new check-satellite-migrations guard (registered in check.mjs)
statically pins that the source dir, tsconfig include, and manifest output path
agree, so the load-bearing include line cannot silently drift and leave a tenant
with no table.
The AIEmbeddingConfig block + the embedding constants land here (the migration
reads dimension from config). The database-pg after:provision pgvector hook
rides with the provider wiring in C7.
Arcoders
added
the
qa: product-testing
…g (WS-AI-3, C4) The per-tenant vector store (I1). It resolves placement via tableLocation(tenant) (SEAM-1) and runs parameterized raw SQL on that connection with the bare table name, never a tenant_<id> literal. Two structural isolation guarantees: a satellite ContextSeal (raw SQL bypasses the kernel one) refuses a query whose tenant differs from the active tenancy scope, and rowscope-pg is refused outright (logical isolation is too weak for inversion-sensitive embeddings). insert is idempotent (ON CONFLICT (source, content_hash) DO NOTHING) and enforces the embeddingCount cap (#18) atomically under a per-tenant advisory lock; a wrong- length or non-finite vector is refused before the write (dimension binding). search scopes by (model, dim); deleteBySource is the #3 rollback + WS-AI-9 purge seam. Four new fatal error codes + four guard-registry entries (rowscope_refused, scope_mismatch [critical], dimension_mismatch, embedding_quota_exhausted), each with a behavioral trip recipe in the emission matrix (the matrix is typed Record<AiGuardId,Recipe>, so a guard cannot ship without one). Unit specs use a fake db; a real-pgvector two-tenant no-leak + idempotency + cap integration spec self-skips where pgvector is absent.
…/settle (WS-AI-3, C5) The ingestion orchestrator runs the fail-closed cost order: resolve texts (inline + an optional SSRF-pinned document fetch, #11), reserve worst-case aiTokens (a non-streaming embed still costs money), embed, store idempotently under the embeddingCount cap, settle the actual tokens, and ALWAYS release the hold in finally. A reserve backend outage maps to a 503 (fail-closed, mirroring the streaming spine), an over-budget reserve to 402, and a SafeFetchError on a document URL to a 400 doc_fetch_blocked before any reservation is taken. Integer metrics (ai_embeddings_ingested / ai_embedding_tokens_total / ai_embedding_errors) carry no content (G3). New codes doc_fetch_blocked(400) + ingestion_denied(403); the SSRF site is not a satellite guard (the kernel pin is the enforcer, as with byok_endpoint_blocked).
…-3, C6) AiEmbedController mirrors the chat choke point in JSON form: authorize FIRST (resolveRequestTenant + authorizeAIAccess), then the new authorizeIngestion write gate (guard.ai_ingestion_denied, distinct from the access gate), validate the body against the chunk/batch/metadata bounds (field-named errors, never echoing content, G3), rate-limit the provider key (op:'embed'), and hand a validated request to the ingestion service. A denied caller reserves nothing; a malformed body is a 400 before any cost; an AIException maps to its pinned status. The POST /embed route mounts in the same fail-closed group as /chat. A PARALLEL AiEmbeddingAuditEvent (actorHash/sourceHash one-way, dimension + embeddingsCount + tokens, no fragments) with its own frozen non-PII pin spec, since the chat audit field set is frozen. Idempotency is durable at the store UNIQUE(source, content_hash), so no Idempotency-Key header is needed.
…ioning (WS-AI-3, C7) AiProvider registers VectorStoreService (driver via getActiveDriver, the db via the 'lucid.db' container alias so no direct lucid dep, the scope seal via tenancy.currentId) and EmbeddingIngestionService (kernel QuotaService for reserve/settle/release/getLimit, the SSRF-pinned safeFetch, integer metrics, and the generic OpenAI-compatible embedding provider built from config). Both are container singletons, resolved never new-ed. assertEmbeddingConfig validates the embedding block at boot (apiKey + required https baseUrl, dimension 1..2000, positive-integer bounds, function hooks), all through the existing fail() -> guard.ai_config_invalid choke. When embeddings are configured the provider registers the pgvector doctor check and an after:provision hook that installs pgvector in a new database-pg tenant's DB before its migration runs — the deferred SEAM-5 ordering, done satellite-side with zero core change.
…rd (WS-AI-3, C8)
check-ai-invariant-1 (registered in scripts/check.mjs, so it runs in npm run
check): scans the AI package's SQL surface (src + the per-tenant migration) for
the two I1 placement violations the frozen doc forbids — a global
public.(ai_)embeddings table, and a tenant_${id} / 'tenant_' + id schema
interpolation that hardcodes placement instead of asking tableLocation(tenant).
An @architecture spec drives its pure auditor with positive + negative controls.
An AI-side no_unsafe_raw_sql arch spec mirrors core's: the vector store is the
first raw SQL in the package, so any ${...} interpolation into rawQuery/raw must
carry a // safe-sql: marker (the AI package interpolates only fixed constants and
binds every value as ?). Both guards are green on the real tree.
A Vector store section in the AI guide: the I1 structural-isolation framing (tableLocation, rowscope refused), the embedding config block, provisioning pgvector (the command + the automatic database-pg hook + the doctor check), and the /ai/embed ingest flow (authorize-first, aiTokens reserve, idempotent by source+content, embeddingCount cap, the SSRF-pinned sourceUrl). Folds the WS-AI-3 surface into the curated [1.0.0] CHANGELOG. minMergedCoverage stays at the graduation floor 60/60/60; ratchet off the first CI merged-coverage run (it cannot be measured locally without PG/Redis).
Formatting-only: the C1/C2 embedding specs were committed before the whole-package eslint sweep, so prettier had queued wraps on them.
A 6-lens adversarial review (each finding faced by a correctness skeptic and an exploitability skeptic) surfaced real defects on the optional document-ingestion seam. Isolation, the scope seal, the advisory-locked cap, guard discipline and audit non-PII all held; these are fixed at the root: - HIGH: the `sourceUrl` document body was buffered whole before the ingestionMaxBytes cap, so a public host past the SSRF pin could OOM the worker. The fetch is now streamed (`streaming: true`) and the transfer aborted the instant the running byte total crosses the cap, never buffered whole first. - LOW: the document fetch had no request timeout, so a hung upstream could pin an ingest worker. Added config.ai.embedding.ingestionTimeoutMs (default 10s). - The row dedup identity keyed on (source, sha256(content)) only, so a same-dimension model swap silently dropped the re-embed and left retrieval under the new model empty. content_hash now folds the model in (via a fixed-length prefix, collision-free), so a swap stores a fresh vector while a same-model re-ingest stays idempotent. Kills the trap before the WS-AI-5 RAG search lands. - The database-pg `after:provision` pgvector hook swallowed a per-database install failure silently; it now passes a logger and logs the failure (still fail-closed downstream: the embeddings migration hard-fails and the doctor check flags it). Regression tests cover the streamed byte-cap abort (asserts early cancel, no full buffer), the streamed + time-bounded fetch opts, and the model-folded dedup key. Local gate green: AI suite 299, check 24/24, build:all, typecheck all workspaces, eslint, knip.
Add the WS-AI-5 (context integrity / RAG) public contract surface and the
per-user document ACL gate that consumes it, ahead of the store filter,
retrieval service, route and RAG-into-chat wiring in later commits.
- define_config.ts: RetrievalScope (a discriminated { all | sources | metadata }
union), RetrievalFilter (G2 per-user doc ACL), AIRetrievalConfig
(retrievalFilter + defaultLimit/maxLimit/maxQueryChars/maxContextItems/
maxContextChars bounds), AiConfig.retrieval?, AiConfig.acknowledgeUnscopedRetrieval?.
- access_gate.ts: resolveRetrievalScope(ctx, tenant, retrieval). Absent hook =>
{ kind: 'all' } (whole tenant corpus, the documented honest limit; tenant
isolation still holds). A wired hook is authoritative and fail-closed: a throw
or an invalid return emits guard.ai_retrieval_denied and throws
retrieval_denied (403), never a silent fallback to the whole corpus.
- ai_exception.ts: retrieval_denied (403, FATAL) across all three tables.
- ai_guard_registry.ts: guard.ai_retrieval_denied (missing-authorization, warn).
- validate_config.ts: assertRetrievalConfig (filter must be a function; bounds
positive integers) + the acknowledgeUnscopedRetrieval boolean check.
- constants.ts: DEFAULT_RETRIEVAL_LIMIT/MAX_RETRIEVAL_LIMIT/DEFAULT_MAX_QUERY_CHARS
/DEFAULT_MAX_CONTEXT_ITEMS/DEFAULT_MAX_CONTEXT_CHARS.
- index.ts: export RetrievalScope/RetrievalFilter/AIRetrievalConfig.
- specs: the guard.ai_retrieval_denied emission-matrix recipe + a focused
security_retrieval_gate spec (absent => all; sources/metadata pass through;
throw/invalid => 403 + guard).
AI suite 307 green; check 24/24.
…l doctor (WS-AI-5, C2) Close the retrieval READ path onto the WS-AI-3 search() seam, metered and document-ACL-scoped, plus the operator-visible posture. - vector_store_service.ts: search() gains an optional filter?: RetrievalScope. The (model, dim) scope and the tenant placement (I1) stay mandatory; the filter only NARROWS. `sources` -> a parameterized `AND source IN (?, ?, ...)` (built like insert's id-lookup), `metadata` -> `AND metadata @> ?::jsonb`, `all` -> nothing. Every filter value is a ? bind (safe-sql). An empty `sources` list returns [] without a query (no invalid `IN ()`, and a zero-doc scope costs nothing). - retrieval_service.ts (new, container singleton): the RAG read orchestrator. Same fail-closed cost order as ingestion minus the write: reserve one query embed's worst-case aiTokens, embed the query with the SAME provider the corpus used, search under the resolved scope filtering on the provider's EFFECTIVE model (so a model naming drift can never silently return zero rows), settle the actual tokens, always release. A metered read (G5). - ai_retrieval_gate_check.ts (new): the ai_retrieval_gate doctor check + shared aiRetrievalGateRisk wording (one voice for the boot warning and the check): embeddings usable + no retrievalFilter => warn (every tenant user sees the whole corpus); acknowledged => info; filter wired or no embeddings => healthy. - ai_provider.ts: bind RetrievalService (mirrors the ingestion singleton), register the doctor check, and warn at boot on the genuinely-unscoped, not-acknowledged posture. - fake_vector_db.ts: a searchHits option + an order-by-embedding branch (before the id-lookup branch its SELECT prefix would otherwise shadow). - specs: the filter WHERE-clause + bindings + scope-seal, the service reserve/ embed/settle/model-consistency/fail-closed/empty-scope paths, and the doctor posture. AI suite 327 green; check 24/24.
…uards (WS-AI-5, C3)
Make retrieved content safe to fold into a prompt, and pin the two context-
integrity invariants the ARCHITECTURE declared but never had scripts for.
- context_builder.ts (new, pure): buildRetrievalContext(matches, {maxItems,
maxChars}) -> a single role-separated, fenced, bounded AIMessage. Retrieved
content is untrusted DATA, so it is (1) role-separated into a `user` turn,
never a trusted instruction turn (a hard structural property, not a heuristic);
(2) fenced in <retrieved_context> with the fence token neutralized inside each
doc so it cannot forge a closing tag and break out; (3) bounded to maxItems docs
and maxChars total (lowest-ranked dropped, last truncated) so the ASSEMBLED
prompt cannot overflow. Returns null when there is nothing to inject. The doc's
wording is NOT scrubbed: role separation is the defense, not a regex.
- check-ai-invariant-4.mjs (new, I4): the satellite must never AUTHOR a
system-role message (host system prompts pass through as data; retrieved data is
a user turn). Scans for a constructed { role: 'system' } (not the type union or
a parse-time allow-list).
- check-ai-invariant-8.mjs (new, I8): every streaming-spine invocation
(.stream.stream(...)) wires a validateFragment output bound. Registered both in
scripts/check.mjs; the DEPTH of the bound stays a behavioral spec.
- index.ts: export buildRetrievalContext (a pure, boot-safe public helper).
- specs: the builder role-separation/neutralization/bounds, and both pure
auditors (clean + violation + comment controls).
AI suite 345 green; check 26/26.
…AI-5, C4)
Expose the retrieval read path as a standalone JSON endpoint behind the same
fail-closed mount gate (G4) the group already enforces.
- ai_retrieve_controller.ts (new): the /ai/retrieve choke point, mirroring the
embed controller. Authorize FIRST (a denied caller spends nothing): membership
gate, then resolveRetrievalScope (the G2 document ACL) BEFORE any cost, then
body validation (query non-empty and <= maxQueryChars; limit a positive int
clamped to maxLimit), then the per-key rate limit, then RetrievalService. Errors
name the field, never echo the query (G3); an AIException maps to its pinned
httpStatus and { error: code }.
- audit_seam.ts: AiRetrievalAuditEvent (a THIRD parallel non-PII event:
tenantId/actorHash/model/matchCount/tokens/outcome/reason/occurredAt) + sink +
noop. Never the query text, a returned document, or a vector.
- routes.ts: POST /retrieve inside the guarded group (no mount-gate change; it
rides G4). Resolves RetrievalService via container.make.
- specs: authorize-first (a denied gate or a throwing retrievalFilter reaches
neither reserve nor embed), the pinned non-PII audit field set + no leak, and
the happy/clamp/400/AIException-status behavior.
AI suite 354 green; check 26/26.
Fold retrieved context into the chat choke point, on a cache miss, as untrusted
data, bounded, without touching the isolation or cost contracts.
- ai_chat_controller.ts: an opt-in `retrieve: { query, limit? }` chat body field.
On a cache MISS (a replay already returned, so RAG never runs for a cached
answer), #augmentMessages resolves the document ACL (resolveRetrievalScope,
G2), runs the metered query embed + scoped search under the request's liveness
signal (G11), and folds the fenced matches into the messages as a user-role
DATA block right before the question (#10). The block's char budget is what
remains of maxPromptChars after the existing messages, so the ASSEMBLED prompt
can never exceed maxPromptChars (#8). A retrieval preflight failure (denied ACL,
over budget, embeddings unconfigured) fails the request BEFORE the stream
commits, with the code's pinned status, and lands both a retrieval and a chat
failed_preflight audit. Two separate aiTokens reservations (query embed +
completion) keep cost unified per tenant (G5).
- routes.ts: the chat closure resolves RetrievalService lazily, only when
config.ai.embedding is present, so non-RAG chat is unaffected when embeddings
are off (an unconditional make would throw config_missing).
- specs: the fold/position/non-RAG-untouched/embeddings-off-400 flow, and the
context-integrity properties (a hostile doc lands as neutralized user data
never a system directive; the assembled prompt stays within maxPromptChars; a
retrievalFilter denial fails 403 before the provider).
AI suite 361 green; check 26/26.
…on proof (WS-AI-5, C6) Prove the retrievalFilter document ACL (G2) on real pgvector: a sources allow-list and a jsonb metadata scope each NARROW a search within a tenant and never widen it; kind:all returns the whole tenant corpus; an empty sources list returns nothing; and the filter composes with I1 (a scoped search as tenant A never surfaces tenant B rows, even for the source key 'kb-eng' that exists in both schemas, and the same filter yields disjoint rows per tenant). Drives the real <=>, IN (...) and @> operators through per-tenant schemas, with the pgvector self-skip so it runs in CI and skips on the local postgres:16-alpine image.
- docs/guides/satellites/ai.md: a new "## Retrieval (RAG)" section documenting the retrievalFilter document ACL (the RetrievalScope union, fail-closed semantics, the absent-hook whole-corpus posture + ai_retrieval_gate + acknowledgeUnscopedRetrieval), the POST /ai/retrieve route, RAG-into-chat via the chat `retrieve` field (#10 role separation + neutralization, #8 bounds), and the check-ai-invariant-4/-8 guards. Turned the "retrievalFilter reserved" note into a real cross-link and added the retrieval document ACL to the Guard events list. - CHANGELOG [1.0.0]: the WS-AI-5 bullet (retrievalFilter G2, RetrievalService + /retrieve, RAG-into-chat, guard.ai_retrieval_denied, invariant-4/-8). check 26/26.
…(C8) Prettier over every WS-AI-5 file, and rename the per-request `ai` in the /chat route closure to `aiConfig` to stop it shadowing the mount-time `ai` (no-shadow). Gate green: AI suite 361, check 26/26, build:all + typecheck (all workspaces) + eslint clean.
With embeddings configured but no retrievalFilter wired, retrieval was served
unscoped (the whole tenant corpus) with only a boot warning. This honors the
frozen ARCHITECTURE contract (RAG retrieval = fail-closed) and mirrors the G4
mount gate: resolveRetrievalScope now refuses (403 retrieval_denied plus a
guard.ai_retrieval_denied {reason: unscoped_unacknowledged} trip) unless the
host wires retrievalFilter or opts in with acknowledgeUnscopedRetrieval.
- access_gate: the unacknowledged branch, before any embed or search; the gate
now reads the parent AiConfig (both callers pass it).
- ai_retrieval_gate_check: aiRetrievalGateRisk becomes aiRetrievalGatePosture
(refused warn / acknowledged info); the boot warning and the doctor check read
the one posture object.
- define_config: acknowledgeUnscopedRetrieval now ENABLES tenant-wide retrieval,
not just silences a warning.
- docs, CHANGELOG, and ARCHITECTURE prose aligned with the fail-closed default.
- tests: a fail-closed proof at both entry points; gate + doctor specs updated;
happy-path fixtures acknowledge the unscoped default.
Reuses the existing guard id and error code (one new reason). Gate green: AI
364 tests, check 26 guards, build:all + typecheck --workspaces + eslint.
…S-AI-5 review) The corrected adversarial 6-lens review (0428426..HEAD, each finding double- verified) found one real defect (low): the chat controller consumed a per-key rate-limit hit BEFORE #augmentMessages resolved the retrieval scope, so a RAG-chat destined to be refused by the document ACL (the C9 unscoped_unacknowledged default, or a per-request hook_error / invalid_scope) still burned one ext:ai:chat window hit before its 403. That broke the "a refused caller spends nothing" invariant that /ai/retrieve upholds (it resolves the ACL before the limiter). Fix: split the chat flow. #retrievePreflight resolves the document ACL BEFORE the rate limiter (only the cheap authorization; a refusal now spends nothing), and #applyRetrieval runs the metered query embed + scoped search AFTER the limiter so the embed stays rate-gated. A shared #failChatPreflight maps either step's AIException to its pinned status plus a failed_preflight chat audit. No token reserve or embed ever happened before the refusal in either path (that cost lives inside RetrievalService.retrieve), so only the early rate hit is closed. Test: a refused RAG-chat consumes 0 rate-limit hits; an acknowledged one consumes 1 (the limiter runs only after the ACL passes). Gate green: AI 365 tests, check 26 guards, build:all + typecheck --workspaces + eslint.
…query proofs (WS-AI-7, C7)
…ass (WS-AI-4, C1)
…, purge seams, guard + doctor + DI (WS-AI-4, C2)
…persist, X-Ai-Session (WS-AI-4, C3)
…tion, concurrency, purge (WS-AI-4, C5)
…uard Compose the WS-AI-3/4/7 purge seams into GDPR-grade erasure, add per-tenant data residency, and pin the no-train log surface. Enterprise-hardened across a design review, a 10-concern review, and a 7-lens gap-hunt (E1..E28). Purge orchestrator (AiComplianceService): bumpEpoch runs first as a verifiably fail-closed gate (read-back or throw), then memory, then embeddings — best-effort continue with an honest per-step summary and a non-zero exit. Per-user erasure keys memory off the raw principal and embeddings off its one-way actor hash (E1). Vector work runs inside tenancy.run so the ContextSeal actively protects (E16); a per-tenant Redis lock stops concurrent double-counting (E15). Full-table and per-actor deletes are batched (ctid loop under the advisory lock) with an optional per-batch statement_timeout, never one DELETE under a wall-clock abort (E4). tenant:ai:purge (operator, --tenant/--principal/--source/--dry-run/--force/ --verify-chain/--actor); auto-purge on TenantDeleted/TenantAnonymized clears the Redis-resident data only (memory + cache epoch), non-throwing but emitting guard.ai_auto_purge_failed so a silent failed erasure is impossible (E6). A memory tombstone blocks in-flight re-population (E5); the SCAN is keyPrefix-correct and fail-closed (E2), UNLINK with a DEL fallback. Residency (#7/#15): config.ai.residency enforced pre-cost at chat provider selection AND embedding egress (embed/retrieve/RAG), fail-closed on a bad resolver (E7/E8), with guard.ai_residency_denied + residency_denied (403, fatal). Structural guard check-ai-no-prompt-logging-for-training keeps prompts/responses/documents/ memory out of logs. Also: ai_compliance doctor check (read-only), three tenant:compliance:report controls (residency, right-to-erasure, embeddings-survive-anonymize transparency), the audit chain survives purge by design (G1, non-PII), and idempotencyTtlMs is clamped to bound the post-purge cache residual. Gates: AI unit 489, npm run check 29 guards, build:all + typecheck all workspaces, eslint/prettier/knip, commands_documented — all green.
… hardenings The final test-and-wire workstream for the AI satellite. An adversarial gap-hunt (leak vectors, chaos/GDPR-resurrection, seam/e2e/matrix honesty) found no production leak; the committed WS-AI-1..9 isolation, purge and fail-closed guarantees all held. This adds the chaos/resilience/enterprise test tier, two small additive hardenings the hunt surfaced, and stands up the demo e2e suite. Tests (all pass locally against real PG + Redis + pgvector): - vector-store outage during retrieval fails closed, reservation released - rate limiter fails closed (503) under an injected Redis outage - concurrent audit writers keep a contiguous (tenant_id, seq) chain - APP_KEY rotation reads memory via the grace key then drops fail-safe - cross-principal idempotency and cross-tenant rate-bucket isolation - many-tenant interleaved embed/memory fuzz plus tombstone-under-purge - purge-completeness scan (PII stores empty, audit chain survives, G1) - no-content-leak sweep and uniform-error/no-existence-disclosure (#17) - a per-vector coverage matrix over all 18 threat vectors (on-disk existence check) Two additive production changes: - EmbeddingProviderRegistry: a host override for the embedding provider, mirroring AIProviderRegistry; default path byte-identical, resolved at make-time - memory emits ai_memory_undecryptable on an undecryptable turn, so a botched APP_KEY rotation is observable instead of only warn-logged Plus check-ai-no-provider-prompt-cache (anti-drift guard) and the demo e2e AI suite (two-tenant isolation, rate cap, harmless injection, poisoned-RAG scoping) wired through offline mock providers.
…ction Publish the AI satellite's consolidated threat model as a dedicated page (docs/guides/satellites/ai-security.md): the 18 threat vectors mapped to mitigation, invariant and a GitHub link to each covering spec (the internal coverage matrix, now public); the eight invariants and their check-ai-invariant guards; the fail-closed postures and the three acknowledge* opt-outs with the risk each accepts; the honest residual limits; the guard / doctor / metric observability surface; the retrieval-outage operations note (monitor ai_retrieval_errors); and a production hardening checklist. Wire it into the Satellites sidebar and the satellites index, correct the AI guide's stale lead (vector store, RAG, memory, audit and compliance all shipped, no longer "a later workstream"), cross-link it from the kernel security page, and record it under [1.0.0] in the AI CHANGELOG. Docs-only, no production-code change. Gates green: test:integrity (10), check-doc-paths, docs:build (zero dead links).
…10 hardening) Validated an external OWASP LLM 2025 report against source: its four proposed code changes are not real gaps (output-filter contradicts frozen I8; SSRF redirect is a false premise — safeFetch refuses redirects and pins the IP; incident metrics already exist; injection-audit contradicts I4/I5) and it misstated vector isolation as RLS (it is tableLocation + ContextSeal). The one genuinely useful seam is added the architecturally-correct way. - config.ai.redactOutput (ctx, tenant, chunk) => string | null: an optional host DLP / PII-redaction hook composed with the mandatory I8 output bound at the single fragment choke point. The bound applies first and last, so I8 holds even against a misbehaving hook; a throwing or non-string redactor fails closed. It is host-owned defense-in-depth, never the isolation control (I4/I8 remain the guarantee). Because it sits upstream of the recording tee, the redacted bytes are what the client, the idempotency cache, and conversation memory all store: a replay serves redacted bytes and the model never re-sees the raw output. New content-free ai_output_redacted metric. - Tests: Tier 1 pure-gate matrix (redact / abort / throw-fail-closed / non-string / oversized re-bound / order / cost-preserved), Tier 2 coherence flow through the real stream + recorder + idempotency + memory (replay-redacted, memory-persists-redacted, throwing-aborts, cost-unaffected), Tier 3 demo e2e (sentinel PII stripped over real HTTP; existing four e2e unaffected). - Docs: an OWASP LLM Top 10 (2025) coverage crosswalk on the security page, a Redact model output guide section, the SSRF honest-limit corrected (pin + no-redirect close the rebind/302 classes on the AI path), and the CHANGELOG. AI unit 525, check 30 guards, typecheck all workspaces, docs:build zero dead-links.
…the tenant search_path resolves the type The AI integration and e2e tiers had never run in CI (the branch was unpushed). The first run surfaced two pre-existing gaps, neither from the redactOutput work. pgvector "type vector does not exist": schema-pg tenant connections used a tenant-only search_path, so a bare vector(N) column and its operators (installed in a shared schema) could not resolve during the ai_embeddings migration or at query time, which broke every tenant's provisioning. Fix: provisionVectorExtension now installs the extension into a dedicated `extensions` schema, and schema-pg tenant connections append that schema to their search_path after the tenant's own schema (never `public`). The type resolves while physical tenant isolation (I1) holds, because the shared schema carries no data. Updated: the schema-pg driver, vector provisioning, the pgvector doctor check (now asserts the schema), the demo tenant model plus the e2e bootstrap, and the AI integration spec setups. AI audit real-pg specs wrote non-UUID tenant ids into a uuid column; they now use real UUIDs so the append no longer fails the uuid cast. Also fixes a prettier wrap in behavior_embedding_provider_registry.spec.ts.
…; refresh api report Follow-ups to the pgvector extensions-schema fix, surfaced by the second CI run (the first-ever run of these tiers on this branch). - tenant:migrate:rollback now folds the satellite per-tenant migration dirs like tenant:migrate, so a rollback can find a satellite migration's down() (e.g. the AI ai_embeddings migration recorded in the ledger) instead of choking on a missing source for the whole batch. - the vector-store cap integration spec truncates tenant A's shared table first, so earlier tests in the group can't pre-fill it and skew the 2/2 cap bite. - refresh core's api-extractor golden report for the new PGVECTOR_EXTENSION_SCHEMA export.
… fails The AI embed/retrieve demo e2e (poisoned_rag, two_tenant) and the rate-limit e2e return a controlled 503 whose aiCode lives only in the JSON body, which assertStatus() does not print. Assert the status with the body in the message so the next CI run reveals the exact failure code and it can be root-caused. No behaviour change on the success path.
…limit refusal
The AI demo e2e had never run in CI; run 3's body-surfacing diagnostic exposed the
two real bugs behind the /ai/embed 503 and the rate-limit e2e.
- The demo enabled config.ai.audit but never migrated backoffice.ai_audit_logs, so
every AI audit write failed. /ai/embed and /ai/retrieve fail CLOSED on the audit
write (it runs before the 200), so they returned 503 audit_write_failed; chat
swallowed it post-stream. Add the demo backoffice migration (0018), mirroring the
satellite stub, so the audit chain is provisioned.
- The chat per-key rate-limit check was the only pre-flight NOT wrapped in
#failChatPreflight, so a rate-limit refusal escaped to the framework's default
exception renderer instead of the pinned `{ error: 'rate_limited' }` 429 (and was
never audited). Wrap it like the reserve/retrieval refusals; update the unit test
to assert the handled 429 response rather than an uncaught throw.
Lasagna-020726/ws-ai-3-vector-store
Arcoders
merged commit Jul 5, 2026
2a87a5f
into
LASAGNA-020626/isolation-hardening-and-benchmarks
1 check passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WS-AI-2: the AI cost governor bites
Makes the AI satellite's cost governor real and biting. Grounded by a 5-reader Understand pass whose headline finding reshaped the scope: the kernel already does the heavy lifting (atomic reserve/settle/release with a per-tenant budget AND an operator ceiling in one Lua, the reaper is in-band TTL eviction, and
limits.aiTokens/operatorCeiling.aiTokenswork generically). So WS-AI-2 is no kernel change — it is the one genuinely new mechanism plus fail-loud budgeting plus the bite tests.What it adds
C1):config.ai.rateLimit { limit, windowSeconds }, a different rail from the token reserve. Each request consumes one hit againstext:ai:<op>:<tenant>:<keyFingerprint>(the fingerprint a one-way SHA-256 of the active provider key, never the key); over the window is a fail-closed 429, a limiter-backend outage a fail-closed 503, and a replay served from cache consumes nothing. Checked pre-flight in the controller (after provider selection, before the reserve), so the status surfaces with headers unsent. A denial rides the publicIsthmusGuardTrippedchannel as a newguard.ai_rate_limitedregistry entry (with its emission-matrix recipe). Redis is resolved through the app container's'redis'binding, so the satellite adds no direct redis dependency.C2): theai_budgetdoctor check and a boot warning surface an unbudgetedaiTokensquota (no per-plan limit and no operator ceiling → the reserve is inert → the endpoint runs unmetered, the footgun commit09809a8documented). No hard mount block on purpose: a dynamic per-tenant budget viaplans.getPlanis invisible to the static boot read, so the posture stays advisory;config.ai.acknowledgeUnbudgetedAiTokensaccepts the risk explicitly.C3): real-Redis integration specs drive the gateway spine with the real QuotaService — over-budget resolvesfailed_preflight over_budgetwith no bytes, a within-budget stream completes and settles exactly the used tokens, the operator ceiling bites both-or-neither with no per-plan limit, and the per-key window throttles to a 429. Closes the gap that every prior AI cost test usedFakeQuota.C4): the ai.md cost section (budgeting, operator ceiling, the doctor check + acknowledge, the per-key rate limit) and the CHANGELOG fold-in.Verification
Each commit individually green. Final:
check22/22 (check-isthmus still 100%, check-quota-key-tenant-scoped green), unit+arch 1382, AI unit 226 (coverage 95.1/91.0/98.0), AI integration 15 (+3 skip) vs real PG/Redis, typecheck all workspaces, eslint, knip, docs:build.Deferred by design: per-tenant BYOK key storage + residency (the
tenant_provider_selectionbody stays a stub), input/prompt-token metering (the reserve holds output worst-case only), and the examples/api wiring + e2e (WS-AI-8). No background reaper job (the in-band TTL eviction is the design; the ARCHITECTURE reaper note was reconciled).🤖 Generated with Claude Code