1.0: tenant isolation hardening, the satellite split, and a real benchmark suite - #10
Open
Arcoders wants to merge 514 commits into
Open
1.0: tenant isolation hardening, the satellite split, and a real benchmark suite#10Arcoders wants to merge 514 commits into
Arcoders wants to merge 514 commits into
Conversation
Arcoders
changed the title
Arcoders
added
bug
Owner
Author
Product DecisionsAfter discussing it carefully as a team, we made two decisions. PostgreSQL OnlyLasagna will only support PostgreSQL. PostgreSQL is at the heart of how Lasagna works. Features like schemas, Row-Level Security (RLS), and Supporting MySQL would mean making compromises in areas that we consider essential. We'd rather focus on doing one thing really well than support multiple databases with a reduced experience. AdonisJS OnlyWe won't be building adapters for Express or NestJS. While it's technically possible, it would require turning Lasagna into a framework-agnostic solution and moving away from the deep integration we have with AdonisJS. We'd rather invest that effort into creating the best possible experience for Adonis developers. WhyThese are not technical limitations. They are product decisions based on focus, quality, and long-term maintainability. We love the AdonisJS ecosystem, and we want to build something that feels truly native to it. |
Arcoders
added
qa: product-testing
Owner
Author
External audit findings, confirmed and addressed.
@UnreaIIab |
Arcoders
added
documentation
Owner
Author
|
I know this pull request contains a lot of commits, which is usually considered an anti-pattern. That was intentional. Instead of squashing everything into a handful of large commits or separate branches, I wanted the full journey to remain visible. If someone revisits this PR months or even years from now, I hope they'll be able to follow the reasoning behind each decision, see how the architecture evolved, and maybe even learn something from the process. It's exactly the kind of pull request I wish I had found when I first started tackling these kinds of problems. Ultimately, all of this work had a single purpose: hardening the tenancy kernel and preparing it for what comes next. The ambition isn't simply to add AI. It's to make AI a first-class citizen of a multi-tenant SaaS platform, where tenant isolation, security, observability, quotas, auditing, and operational tooling are part of the foundation not something developers have to build afterward. To make that possible, the AI satellite should be built on top of the capabilities that already exist in the kernel, rather than reinventing them. 1. Respect the physical and logical tenant isolation modelThe AI satellite should fully respect the kernel's physical and logical isolation model. This validates the 2. Reuse the quota systemAI workloads should integrate directly with the existing quota infrastructure. Rather than introducing a separate rate-limiting mechanism, the AI satellite should reuse 3. Reuse the auditing systemEvery AI interaction should flow through the existing audit infrastructure. This validates that the AI satellite builds on 4. Reuse the encryption systemSecrets, provider credentials, and any sensitive AI configuration should rely on the kernel's existing encryption layer. This validates reusing the existing 5. Reuse the resilience systemThe AI satellite should inherit the kernel's resilience mechanisms. That includes circuit breakers, execution timeouts, retries, and extension execution through 6. Define its own contract versionThe AI satellite should define its own public contract version. This validates introducing an This release lays the groundwork for that vision. The journey has just begun. 🍝🚀 |
Arcoders
added
enhancement
…6/PLD-3/6/AD-07/CFG-7)
TS-1: type the Lucid ConnectionManager seam. Tenant routing and the read-only
plugin firewall ran entirely under `(this.db as any).manager`; it now reads
`this.db.manager as ConnectionManagerContract | undefined` (Lucid's own exported
type), so a rename or a `manager.hass` typo on the cross-tenant-write path fails to
compile. buildReadOnlyConnectionConfig returns a typed PostgreConfig. No behavior
change.
CFG-4 + F5: assertConfigBounds(config, { inDevOrTest }) and setConfig(config,
{ inProduction }) — both options optional with env-derived fallbacks; the provider
passes app.inDev||app.inTest and app.inProduction, so the framework is the single
authority for the current environment. The resolution-safety gate now fails CLOSED
everywhere EXCEPT a recognized dev/test env (was: production only). BREAKING for a
host on a client-controlled or host strategy with no membership gate / host suffix
deployed under an unrecognized NODE_ENV (staging/qa/unset): it now hard-fails at
boot instead of only warning. Deliberate — an insecure resolution posture must not
ship on a warning alone. The fixture, demo, and test runners are unaffected (they
gate the posture or run dev/test). The unit runner defaults NODE_ENV=test.
CFG-6: deepFreeze recurses into plain objects + arrays only; a stateful custom
resolver instance in resolverChain/resolvers stays mutable (freezing it violated
its ownership and could break a resolver that caches).
PLD-3: the boot() security warnings (rowscope-no-RLS, resolution-risks) and the
queue/bootstrapper diagnostics defer to app.booted, where the logger is guaranteed,
removing four `container.make('logger').catch(() => undefined)` reads that could
silently drop a warning. A diagnostic can never fail boot (isolated).
PLD-6: a provider disposer registry captures wireResolutionCacheInvalidation's
teardown in ready() and runs disposers LIFO in shutdown(), so a ready()/shutdown()
cycle no longer leaks the eight lifecycle listeners. Each disposer is isolated so a
throwing one never skips closeOwnedHandles (the SIGTERM queue drain).
AD-07: delete the dead BackofficeAdapter (never instantiated; the unified
TenantAdapter routes backoffice by the `static isolation` marker). Barrel, root
export, api-extractor golden, stability doc, and stale satellite comments updated.
CFG-7: forbid `[]` array grammar on enforce:true secret paths (the dotted-path
bounds reader cannot navigate them, so enforcement would silently no-op).
Adversarially reviewed (multi-agent); eight findings folded in: exception-isolated
shutdown/boot diagnostics, and doc/comment drift for the removed adapter and the
broadened fail-closed gate (security.md, changelog, release-notes). Gate green:
typecheck, unit 1651, integration 393, 37 guards, lint. Golden regenerated
(BackofficeAdapter removed, setConfig signature).
… the legacy switch (TRES-01) CRITICAL. Rate-limit buckets and the `backoffice.tenant_metrics` rows attributed the tenant through the public `resolveTenantId` — the v1 strategy switch, which reads `resolverStrategy` ALONE and never consults the resolver chain. On ANY `resolverChain` deployment that attributed to a DIFFERENT tenant than routing served: a silent cross-tenant rate-limit bypass (one tenant's requests counted against another's bucket, or collapsed into the shared per-IP 'global' bucket) and corrupted metering (a row forged under the wrong tenant). Fix: one internal, chain-aware `resolveTenantIdSync(request)` that walks the SAME boot-seeded registry chain the adapter routes by and `request.tenant()` resolves (honouring `legacyAdapterFallback`, degrading a domain hit to undefined). The rate-limit and metrics middlewares now use it instead of the chain-blind switch. The registry is seeded into the request module cache from the provider's boot() (the same pattern as TenantResolutionCache / TenantLogContext), so the sync path has a guaranteed hit from the first request and never falls back to the switch. The public `resolveTenantId` stays as the documented legacy fallback (TRES-02 folds it into the unified authority in a later wave). Wave 2, the resolution security patch (its critical item); AD-03/PLD-4/PLD-7/F2/F3 follow. Gate: unit 1654, integration 393, 37 guards, lint.
…LD-4/PLD-7/F3/F2) Second and final Wave 2 commit (TRES-01, the critical bug, landed separately in 8f13c95). All five remaining Wave 2 items, gated green (unit 1663, integration 395, typecheck, 37 guards, lint, knip, api-extractor golden regen). - AD-03: extract the cross-tenant cached-connection identity seal into a shared connection_identity_seal.ts (the single registered guardFile for seal.connection_identity, renamed from seal.connection_search_path since the event now covers both drivers). schema-pg keys on searchPath, database-pg now keys on connection.database — closing the missing seal on the strongest isolation. sqlite documents why it needs none. Regression specs for both PG drivers; this is the AD-05 verifyCachedConnection down-payment. - PLD-4: ImpersonationService resolves its optional AuditLogService lazily in #audit(), so the provider registers it with a plain sync factory — killing the only async factory among the provider's singletons. - PLD-7: replace boot()'s 4-way driver if/else with a data-driven BUILT_IN_ISOLATION_DRIVERS map; a sixth built-in is a map entry, not a new branch (OCP). Provider no longer imports the driver classes. - F3: decouple the breaker's fail-fast decision from the heavy opossum object via a lightweight per-tenant marker (openUntil), so a heavy OPEN breaker can be evicted under fleet-wide-outage memory pressure while isOpen()/run() keep failing fast synchronously, and self-heal once the window elapses (recreate on HALF_OPEN). Only HALF_OPEN breakers are now spared from eviction. - F2: give the ReadReplicaService LRU the same in-use-aware release the per-tenant drivers use (decline eviction while a query is in flight) and settlePending() in resolve(), so an in-flight analytics read is never severed under cap pressure.
…ard (CFG-2) Mirror the proven SECRET_CONFIG_FIELDS pattern for numeric tunables. The bounds were a hand-written list of atLeast()/inRange() calls in assertConfigBounds, so a new numeric tunable could ship unbounded just by no one adding a line. - NUMERIC_CONFIG_FIELDS: one declarative registry of every numeric config leaf, each either enforced (min/optional max) or exempt-with-reason. The enforced set is byte-identical to the pre-registry bounds, so this is behavior-preserving; currently-unbounded leaves are registered as exempt with a written reason (correctness-neutral tunables, operator-managed infra endpoints), candidates to promote to enforced later. - assertConfigBounds now loops the registry (keeping the impersonation maxDuration>=defaultDuration cross-field invariant, secrets, resolver chain, and the resolution-safety gate untouched). - numeric_fields_bounded.spec.ts: parses every `name: number` leaf of the config type (config.ts + config/isolation.ts) and fails unless it is registered (bounded or exempt), so a new numeric field cannot ship without a conscious bounds decision — the structural half of CFG-2 / F1's ceiling bound. First of the Wave 3 sub-commits; TS-2, TRES-02, CFG-1 follow.
BUILT_IN_ISOLATION_DRIVERS['database-pg'] is typed to return the IsolationDriver interface, which has no databaseName(); the test casts to reach the database-pg concrete method. Direct cast tripped TS2352 (insufficient overlap). Route it through unknown as the compiler suggests. Pre-existing since the wave-2 driver map.
…e authority (TRES-02)
Delete the legacy resolverStrategy switch (legacyResolveTenantId) and its
resolveTenantIdSync twin: a single chain-aware resolveTenantId now walks the
registry chain, and the config.resolver.legacyAdapterFallback opt-out is gone
(the adapter fallback and the provider primeTenancy are both unconditional now).
Resolution has ONE authority — the host-allowlist and UUID-border math live in
the resolvers alone. buildResolverRegistry(config) (pure, no Ignitor) is the sync
fallback and test builder; canonicalChainNames(config) single-sources the "which
resolvers run" question for the boot wiring and the host-trust audit.
F7 — UUID policy at the resolver border (builtins): header/subdomain/path mint an
id ONLY for a canonical UUID v4; a non-UUID value falls through the chain instead
of forging an id that only fails a downstream assert (and, on a chain deployment,
never reaches a rate-limit/metrics key). DomainOrSubdomain: a non-UUID label under
baseDomain falls through to its {domain} envelope.
F8 — ResolverHit.id() lowercases a UUID, so a mixed-case identifier collapses onto
one connection, one resolution-cache entry, one rate-limit bucket, one metrics row.
Adversarial review (5-lens workflow) surfaced three gaps, all folded in:
- the demo e2e resolution specs (resolution_strategies, forwarded_host_tenant_hop)
pinned the old non-UUID behaviour and are invisible to the unit tier — migrated
to UUID fixtures + the booted registry (verified 14/14 e2e green);
- the middleware attribution specs never pinned "attribution follows the chain,
not the header" AT THE SEAM — added a chain-divergence test to both;
- post-F7 a colon-injected header no longer reaches the middleware isSafeIdentifier
seam guard — added a custom-resolver-unsafe-id test to both, so the seam
defense-in-depth for custom resolvers stays pinned.
Breaking (pre-1.0, 0.3.0): removes resolver.legacyAdapterFallback; a host that
attributed tenants by opaque non-UUID ids must move to UUID v4 tenant ids.
Gate: unit 1675, integration 395/8-skip, e2e 14, typecheck clean, 37 guards,
lint, knip, test:integrity, docs:build, golden regen. Absorbs TRES-03/05, TS-4, F7/F8.
…guard (TS-2)
identifier.ts reached UP into isthmus/audit.js to emit guard.tenant_identifier
inside a throwing assertSafeIdentifier — a low-level policy leaf depending on the
guard/audit layer above it. And the non-throwing twin isSafeIdentifier, used at
the rate-limit and metrics attribution seams, dropped a forged custom-resolver id
with no audit trail at all.
TS-2 inverts the arrow and closes the gap:
- identifier.ts is now a pure zero-import leaf: the regexes and the predicates
(isUuidV4, isSafeIdentifier) only. No throw, no emit, no imports.
- new isthmus/guarded_identifier.ts (guard layer) owns the single
emitIsthmusEvent('guard.tenant_identifier'). It exports the throwing
assertSafeIdentifier (emit-then-throw, for every DDL / Redis-key slot) and a new
non-throwing guardedSafeIdentifier(value, kind) for attribution seams: it emits
on a PRESENT-but-unsafe id and stays silent on an ABSENT one (undefined/null/''),
so a forged attribution is audited without flooding on ordinary untenanted
traffic.
- rate_limit + track_metrics point their seam at guardedSafeIdentifier; the
degrade-to-'global' / skip control flow is byte-identical, the only new
observable is the audit emit on a present-but-unsafe id.
- 17 call sites + define_plugin repointed to the guard; the internal/services/sdk
barrels split (isUuidV4 from the leaf, assertSafeIdentifier from the guard);
registry guardFile for guard.tenant_identifier moved to the new file.
The degrade emit shares the high-severity dispatch window with the other high
guards; that residue is reachable only behind a lax custom resolver (built-ins
gate id hits on isUuidV4) and leaves the Prometheus counters accurate. It is
documented on the guard — a per-id sub-cap in the shared limiter is the eventual
structural fix.
Gate: unit 1679, typecheck (all workspaces), integration 395/8-skip, check (37
guards, Audit-coverage Index 100%), eslint, integrity 10, docs:build; api golden
regen (isUuidV4 doc-status flip only, signature unchanged).
…y-critical event (TS-2 review) The TS-2 adversarial review found one medium security-observability issue: the new guardedSafeIdentifier degrade emit dispatched an IsthmusGuardTripped event per present-but-unsafe id, sharing the one per-severity 'high' dispatch window with the security-critical high guards (SSRF, RLS, webhook). A forged-id flood — reachable behind a lax custom resolver, since the built-ins gate id hits on isUuidV4 — could crowd out those guards' EVENT dispatch (their counters stayed accurate). Fix by putting each refusal path on the right surface instead of throttling a shared budget: - guard_audit.ts: GuardEmitOptions gains an opt-in `dispatch?: boolean` (default true). When false, emit records the trip on the counters (and the metric bridge) but broadcasts no event, so it touches no dispatch window. Additive and backward-compatible — every existing caller omits it and is unchanged. - guarded_identifier.ts: the throwing assertSafeIdentifier (a near-miss DDL/key injection) still bumps the counter AND broadcasts the event; the non-throwing guardedSafeIdentifier degrade (a dropped attribution) bumps the counter with dispatch:false. A forged attribution stays visible on the Prometheus multitenancy_isthmus_* counters (the kernel's primary trip surface), while a high-volume degrade can never consume the high dispatch budget. Counter for the volume signal, event for the near-miss. Gate: core unit 1679, ai unit 526 (the only satellite that builds its own audit), typecheck (all workspaces), core integration 395/8-skip, check (37 guards, Audit-coverage 100%), eslint, integrity 10; api golden regen (additive optional GuardEmitOptions.dispatch only).
…AULTS (CFG-1) MultitenancyConfig stays the INPUT type an app authors; getConfig() now returns a new ResolvedMultitenancyConfig with the always-present tunables promoted to required. defineConfig and setConfig both run one pure, idempotent resolveConfig that merges a single CONFIG_DEFAULTS (src/config_defaults.ts). setConfig is the store choke point, so an untyped host config or a partial test config still yields a fully-resolved getConfig(). circuitBreaker / queue / isolation / resolver.cache are resolved, so their read sites drop the scattered `?? DEFAULT_*`: circuit_breaker_service, tenant_queue_service, schema_pg_driver, database_pg_driver, read_replica_service, extensions/request read the value directly. resolveConfig uses per-field `??`, so a valid 0 (circuitBreaker.volumeThreshold / isolation.evictionGracePeriodMs) survives. Genuine feature toggles stay optional and are never materialized — impersonation (secret boot check), plans (quota fallback), tenantReadReplicas (empty-hosts short-circuit) — so their presence gates keep working. Materializing isolation leaves its per-driver sub-fields undefined when unset. Deleted the now-scattered private consts DEFAULT_MAX_OPEN_QUEUES, DEFAULT_MAX_TRACKED_CIRCUITS, CIRCUIT_BREAKER_DEFAULTS. The publicly re-exported / benchmark-referenced defaults (DEFAULT_MAX_TENANT_CONNECTIONS, DEFAULT_EVICTION_GRACE_MS, DEFAULT_RESOLUTION_CACHE_*) are kept and sourced into CONFIG_DEFAULTS. New public exports: ResolvedMultitenancyConfig + sub-types + ResolverConfig/ResolverCacheConfig; api-extractor golden regenerated. Regression: behavior_config_resolution.spec.ts (defaults filled, 0 preserved, host wins, toggles not materialized, isolation sub-fields stay undefined, satellite blocks pass through, idempotent, getConfig resolved). Gate green: unit 1688, typecheck all workspaces, integration 395/8-skip, check 37 guards, eslint, integrity 10, docs:build, golden regen. Adversarial review (5 lenses + verify) -> 0 confirmed defects.
…ook (AD-06) enforce() was required of every driver but is a no-op in all four shipped ones: connection-isolated drivers (schema-pg/database-pg/sqlite-memory) have the connection as their boundary, and rowscope-pg scopes at query time via the withTenantScope mixin / RLS. The contract oversold it as "the" row-scoping point and forced a ceremonial no-op on every driver. Make it `enforce?(client, tenantId)`: an OPTIONAL synchronous per-query hook a custom driver may implement, called by the adapter as `driver.enforce?.()`. Drop the four no-op implementations. Rewrite the contract doc + architecture.md so the boundary story is the connection / mixin, not enforce. Update the driver-contract and adapter-enforce specs to the optional shape (the adapter still invokes enforce when a driver provides one). Behavior-preserving: every shipped enforce() was already a no-op. No public api-extractor surface change (IsolationDriver is not golden-tracked). Gate: unit 1688, 37 check guards, core typecheck, eslint on the touched files.
…try (TRES-04/06) Bumps RESOLVER_CONTRACT_VERSION 1->2 with two additions to the TenantResolver contract, both closing a real cross-tenant hazard. TRES-04 (+F4) — trust classification from one source. Adds `readonly trust: 'host'|'client'|'server'`. A new resolver_trust.ts (entryTrust/chainTrusts) classifies the resolution path from each resolver's declared trust, and the membership-gate (IDOR) and host-trust audits now derive from it instead of two hand-kept name sets (CLIENT_CONTROLLED, HOST_STRATEGIES, deleted). FAIL-SAFE: an undeclared/unknown resolver classifies as 'client', so a bespoke resolver can no longer silently disable the IDOR hard-fail — the host opts out with trust:'server' (or a wired gate). A built-in NAME wins the classification even when an inline instance reuses it, because wireResolverChain runs the built-in, so a host-based built-in can never escape the host-trust audit. TRES-06 (absorbs TS-3) — synchronous resolution is explicit. Adds optional `resolveSync(request)`. registry.resolveSync calls it directly (no `.then` sniffing); setChain FAILS CLOSED when a routing-chain resolver lacks it (an async-only resolver would split-brain: request.tenant() resolves a tenant the sync attribution path — model routing, rate-limit, metrics — skips). Built-ins extend a new SyncTenantResolver base (resolve() forwards to resolveSync(), carries contractVersion), also exported for host resolvers. Breaking (pre-1.0): a custom resolver in a routing chain must now expose resolveSync (extend SyncTenantResolver); an undeclared-trust resolver hard-fails the production resolution-safety gate unless trust:'server' / a gate / an ack. Gate: unit 1697, typecheck all workspaces, integration 395/8-skip, e2e resolution 14 + membership_gate 2, 37 check guards, eslint, knip, integrity 10, check-docs-code, golden unchanged (/services + resolver contract not api-extractor-tracked). Adversarial review (1 agent) -> 1 PLAUSIBLE finding folded (host-name-collision classification) + comment accuracy fix.
…(AD-05)
schema-pg and database-pg carried byte-for-byte copies of the per-tenant
connection pool: the same ConnectionLru wiring, connect/disconnect/markUsed/
migrate flow, hard-cap admission, settle/evict race handling, and the
"never re-adopt a draining pool" firewall. Two copies means markUsed and the
in-use-aware eviction were implemented (and could drift) twice.
Extract a shared abstract PooledPgDriver base that owns all of it. A subclass now
supplies only what genuinely differs, through two hooks plus its own DDL:
- buildTenantConfig(template, tenant): schema-pg sets searchPath; database-pg
overrides connection.database and drops searchPath.
- verifyCachedConnection(tenant, cachedConfig, name): the cached-connection
identity seal — schema-pg keys on searchPath, database-pg on
connection.database — both routed through the shared assertCachedConnectionIdentity.
schema_pg_driver / database_pg_driver drop to those hooks + provision/destroy/
reset/tableLocation. lucid() is one protected method; the template-not-found
error keeps each driver's label.
Behavior-preserving pure extraction. The searchpath-pin anti-regression lock is
updated for the new layout: it now asserts the base fast path delegates to
verifyCachedConnection AND schema-pg's hook keys the seal on searchPath, so
neither can be quietly dropped. This base is also where AD-01 will land the
read-only firewall clone so it counts against the same pool.
Gate: unit 1698, typecheck all workspaces, integration 395/8-skip, 37 check
guards (incl. no_adhoc_stateful_services — ConnectionLru is not a container
singleton), eslint, golden unchanged (PooledPgDriver internal; driver public
API identical).
…-01a)
The untrusted-plugin read-only firewall clones a tenant's connection into a
SELECT-only variant (`<primary>__plugin_ro`) so Postgres denies writes. The
ADAPTER built that clone inline via db.manager and NEVER released it: a
multi-tenant process running untrusted plugin code leaked one pool per tenant
(verified — there was no release site for `__plugin_ro` anywhere in src).
Move ownership to the driver. New optional contract method
`IsolationDriver.ensureReadOnlyClient(tenantId, db, role)`; PooledPgDriver
implements it synchronously (the adapter passes `this.db`): it builds/registers
the clone via buildReadOnlyConnectionConfig, tracks it in the SAME ConnectionLru
as the primary (so it counts against the cap and is touched on use), and
`disconnect` now releases BOTH the primary and the clone — closing the leak. The
adapter's read-only block collapses to `driver.ensureReadOnlyClient?.(...)` and
FAILS CLOSED when it is absent: a driver with no per-tenant PG pool (rowscope-pg,
sqlite-memory) offers no firewall, so untrusted code is denied rather than routed
to a shared/writable connection (this also closes a latent rowscope cross-tenant
read hole, where the "clone" was of the shared central connection).
Hardening from adversarial review, folded in:
- The clone is now identity-SEALED before it is built: ensureReadOnlyClient
runs verifyCachedConnection against the primary (emitting
seal.connection_identity) so a stale/collided primary can never be cloned
into a read-only window onto another tenant.
- ensureReadOnlyClient is synchronous and cannot await settlePending like
connect(), so it now refuses (fail-closed, transient) via a new
ConnectionLru.hasPendingRelease() check rather than re-adopting a draining
clone pool.
New spec isolation_read_only_clone_lifecycle (release-on-disconnect, register-
once, fail-closed-no-primary, primary-seal). pooled_pg_driver added to
NO_SILENT_GUARD_ALLOWLIST (RO fail-closed is a firewall SETUP failure, and the
primary it derives from is sealed by the emitting seal.connection_identity).
Gate: unit 1702, typecheck all workspaces, integration 395/8-skip, 37 check
guards (isthmus Audit-coverage Index 100%), eslint, golden unchanged
(IsolationDriver contract + adapter bodies not api-extractor-tracked).
maxTenantConnections is a SOFT target: with enforceConnectionCap off (the
availability-favouring default) a burst of concurrently-active tenants lets the
pool grow past it, trending toward the active-tenant count and, unchecked,
toward PostgreSQL max_connections. Add a second, absolute tier so an
availability-favouring deployment still has a hard backstop.
New optional isolation.maxTenantConnectionsHardCeiling (unset = no ceiling).
ConnectionLru.atHardCeiling() refuses admission at/above it ALWAYS (independent
of enforceConnectionCap); PooledPgDriver.connect() throws
TenantConnectionLimitException (503) on `atHardCeiling() || atHardLimit()` for
request-path connects (operational paths bypass, as with the soft cap). Bounded
+ registered like maxTenantConnections (numeric_config_fields, +docs so
test:integrity passes; golden gets the new field). A throttled runtime warning
fires when the ceiling bites, and the provider logs a boot warning when the
ceiling is set BELOW maxTenantConnections (a misconfig — the pool would be
refused before the LRU ever evicts an idle connection).
Adversarial review (3-lens workflow → 2 confirmed findings) folded, then
re-verified:
- LOCKUP (high): the ceiling must not refuse while idle connections sit unshed
(eviction only runs on a successful admit, so a pinned-at-ceiling pool would
refuse new tenants forever). atHardCeiling now yields to an evictable victim
via a shared #atCapacityWithNoVictim(threshold) — the same policy as the soft
cap — so it refuses only when the pool is full of ACTIVE connections.
- RO-CLONE BYPASS (medium): ensureReadOnlyClient added the __plugin_ro clone
(a second real connection) with no ceiling check, so untrusted-plugin traffic
could push the pool to ~2x the ceiling. The clone-creation branch now honors
the same two admission tiers, failing closed (503) rather than opening
connection N+1 for untrusted code.
Gate: unit 1707, typecheck all workspaces, integration 396/8-skip (incl. a new
ceiling-refuses-with-503-even-with-enforceConnectionCap-off spec), 37 check
guards, eslint, golden regenerated (IsolationConfig.maxTenantConnectionsHardCeiling).
The TRES-04/06 commit edited seedResolver in these two specs but they were not prettier-clean in unrelated regions; format them so `npm run lint` is green. No behavior change.
…uard (EXT-5) check-abi-boot-assertion now mirror-checks a third per-provider boot surface, nativeAddons, the same way it already pins satelliteApi and pluginApiVersion: - a definePlugin facade that declares `nativeAddons` must agree with package.json#lasagnaSatellite.nativeAddons (the manifest drives the install-time consent gate, doctor, and health-check); - when the manifest claims native addons, the provider MUST wire the boot-time sandbox fail-closed (assertNativeAddonsSandboxable) — a facade with nativeAddons:true runs it, or a raw boot() calls it. Otherwise a native addon loads in a --permission worker with no --allow-addons and the guard is silently absent. The backstop is conditional: a plugin with no native addons needs nothing. Self-test fixtures cover the mirror, the drift case, the missing-backstop case, the raw-provider-with-assert case, and the non-native no-op. No real provider declares nativeAddons today, so the guard is inert over the current fleet. Also documents the umbrella-vs-per-surface versioning rule in api_version.ts: there is deliberately no umbrella version; each axis (Satellite ABI, plugin facade contract, per-surface *_CONTRACT_VERSION, marketing version) bumps independently, and nativeAddons is a boolean flag mirrored the same way, not a version.
The v2 bump recorded an ADDITIVE change: the optional `sensitive?` trust flag. A provision that never sets it behaves exactly as before, and one that opts in gets the TRUSTED_SATELLITES gate for free — by the contract-version rule (bump only on a backward-incompatible change) that never earned a bump. The cost of the spurious bump is real: assertContractCompat WARNS every extension built against an older version, so bumping capability for a non-breaking change floods that channel with false alarms and desensitizes operators to the genuinely breaking bumps (isolation's v2 tableLocation requirement) that need the warning read. Reverting keeps the warning meaningful. The `sensitive?` field and its trust gate are unchanged; only the version integer moves 2 -> 1. Also fixes a latent Wave-4 doc staleness: the surfaces at 2 are ISOLATION and RESOLVER (RESOLVER bumped in TRES-04/06), not CAPABILITY. Golden regenerated; docs (contract-versions, plugins) updated.
…hape (EXT-1/EXT-2/EXT-3)
Nine single-source registries (resolver, isolation driver, authorizer, tenant
middleware, audit destination, webhook transformer, feature-flag strategy,
capability, bootstrapper) each hand-rolled the same registration gate — name
check, dup collision, contract-version compat — with six different collision
errors (four typed, six plain Error). Extract a shared ExtensionRegistry base
that owns the Map + has/unregister/clear/contractVersion and the ONE gate, via a
protected assertRegistrable(entry, opts). Each subclass keeps its own public
register()/list() and side effects (activation, LIFO order, chain, the trust
gate, a value transform) — only the plumbing is shared. HookRegistry stays out
(an append multimap, not a single-source registry).
EXT-2 (the prize): the base runs an abstract assertShape(entry) UNCONDITIONALLY
at registration, INDEPENDENT of the version comparison. assertContractCompat only
WARNS a plugin built against an older contract, so before this a v1 plugin
missing a v2-required member registered (with a warning) and crashed mid-request.
Now a missing member throws at register(). Isolation moves its tableLocation
presence check into assertShape; resolver gains one for its required resolve()
entry point (a resolver missing it used to register and crash on the async chain
walk). A single ExtensionCollisionException (E_EXTENSION_COLLISION) replaces the
six plain Errors; capability/authorizer/middleware keep their domain collision
exceptions via a collisionError override.
EXT-3 guard: check-extension-contracts now requires every surface that reaches a
breaking version (> 1) to declare a shapeGate — the registry file whose
assertShape override enforces the member, or null for a non-registry config hook.
An undeclared shapeGate at v>1 is an error, so a future bump (core OR satellite)
cannot silently reopen the warn-then-crash gap. A shared strip-js-comments lib
means a doc-comment MENTIONING assertShape no longer satisfies the gate.
No public surface change (the registry classes are internal); golden unchanged.
Gate: unit 1707, typecheck-all, integration 396/8, check 37 guards, api goldens
in sync, lint. Adversarial review (2 agents) folded: restored the { override:
true } remediation hint on the two override-supporting registries; hardened the
guard's shapeGate scoping and comment handling.
…XT-5) Adversarial review of the EXT-5 nativeAddons mirror found three latent defects, all now fixed: - the facade regexes read tokens from comments and strings (a "nativeAddons:" in a block comment inside the facade braces, a slash-slash in a URL string) — now the provider source is passed through the shared strip-js-comments lib first, so only real code is scanned. Hardens the sibling satelliteApi/pluginApiVersion reads against the same class of misread. - a truthy-non-boolean manifest value (nativeAddons: 1) was Boolean-true in the mirror but failed the strict "=== true" backstop, so the two disagreed — now normalized to a boolean up front, matching the runtime manifest parser which drops a non-boolean value. - the self-test never exercised an explicit "nativeAddons: false" facade or the normalization, so a regression in either branch would have passed — added fixtures for the false branch, the truthy-non-boolean case, and a comment-inside-braces misread. Guard-only; --self-test and the real run over 8 providers both green.
…ver (AD-02) The TenantAdapter is the tenant-routing chokepoint, but it had grown three private resolution methods, a per-request memo symbol, and an inline reach into `db.manager` — mixing tenant-context resolution and connection ownership into the one class that must stay obviously correct. Split those concerns out: - Extract TenantContextResolver, the single synchronous authority for "which tenant is active now?" (tenancy.run() scope first, then the request via the chain-aware resolver). It absorbs #resolveTenantId, #requestComparandId, #resolveIdFromRequest and the ContextSeal. The seal's HTTP-side comparand now derives from ONE cached value — the tenant request.tenant() resolved (memoizedTenantId) — falling back to the cheap synchronous chain, so the duplicate HTTP_COMPARAND per-request memo is gone. This also makes the seal strictly stricter: it can no longer read a stale cached comparand that hid a late context switch. - Fold the adapter's "connection not established" guard into an optional driver method assertConnected(tenantId, db), so the adapter no longer casts db.manager at all — the driver that owns the per-tenant pool owns the check and the single cast (the roadmap's confine-the-Lucid-cast theme). The typed, actionable IsolationConfigException is single-sourced in assertTenantConnectionEstablished and shared by the PG drivers and sqlite-memory (all three own a per-tenant connection); rowscope-pg's shared connection is always registered, so it needs no check. The adapter body is now resolve-id -> ask the driver for a client -> return. Behavior on the routing path is preserved (verified against every adapter spec, the context-seal fuzz + fail-closed integration specs, and an adversarial review): the fallback path still resolves sync-chain-only, the seal fires identically, and the read-only firewall is untouched. Gate: unit 1707, typecheck-all, integration 396/8, check 37 guards (isthmus Index 100%, adapter's read-only setup fail-closed newly allowlisted, seal.tenant_context guardFile repointed to the resolver), api goldens regenerated (adapter ctor param), lint.
…schema (AD-08) Backoffice models (audit logs, webhooks, SSO configs, billing rows, the host Tenant) read through the adapter's query() override, which pins them to backoffice.<table> with .withSchema regardless of search_path. But writes — insert/update/delete — and refresh bypass query() and resolved their schema purely through the backoffice connection's search_path, which the package never sets (only the host's database config does). A host that mis- or un-set it would silently write backoffice rows to the wrong schema (public by default) while reads still targeted backoffice, so the divergence read as data loss. That read/write asymmetry is the leak AD-08 closes. The roadmap's namingStrategy approach is blocked (satellites set a literal `static table`, ~14 raw-SQL sites qualify the schema themselves off a bare Model.table and would double-qualify, and reading config in a naming strategy throws at @column import time). Instead, qualify writes the SAME way reads are qualified, at the adapter: extract a single protected `queryForInstance(instance, action)` seam in the base adapter (insert/update/delete/refresh route through it, behavior-identical) and override it in TenantAdapter to add `.withSchema(backofficeSchemaName)` for backoffice-marked models. Backoffice ORM writes now resolve to backoffice.<table> independent of search_path, symmetric with reads; tenant/central writes are byte-identical (no qualification). refresh, which was also unqualified, is pinned too. The shipped test-tenant factory writer is pinned the same way. The bare `Model.table` is untouched, so the raw-SQL satellite writers (which qualify the schema themselves) don't double-qualify. Adversarial review found no regression across the satellite write paths; the .withSchema mechanism is the proven read-path one applied to writes. Gate: unit 1709 (2 new write-qualification specs), typecheck-all, core integration 396/8, SSO integration 17/5 (backoffice ORM writes green under .withSchema), check 37 guards, golden regenerated (protected queryForInstance), lint. NOTE: billing (126/93) and reporting (48/4) integration are red both before and after this change — pre-existing local issues (billing inserts omit a NOT-NULL `provider` column; reporting has a date-sensitive metrics-bucket test), neither caused by AD-08 (a schema-only change cannot omit a column) nor Wave 5.
…e-robust getAggregate's BUCKET_SQL selected the period bucket as a raw `date` (`period`, `DATE_TRUNC(...)::date`) and the rollup path selected the `month` date column directly. node-postgres round-trips a `date` through a JS Date, so under a non-UTC process timezone the bucket shifts by a day — day/week/month buckets, and the live-vs-rollup equivalence, came back off by one on a machine in e.g. Europe/Paris while passing under UTC. It also disagreed with the OpenAPI contract, which documents `period` as a `YYYY-MM-DD` string. Cast each bucket to `::text` in SQL so PostgreSQL emits the ISO string directly, with no Date round-trip and no timezone sensitivity. Pure output-shape fix; the grouping/order/filter are unchanged (WHERE compares the date column to the `since`/`until` strings in-SQL, which was already tz-safe). Reporting integration now passes under both UTC and Europe/Paris (52/52); unit 105/105.
…enum)
The billing integration suite was 126/93 red — untested for a while because CI
runs the satellite UNIT suites (test:coverage), not the integration ones. Two
pre-existing defects, both unrelated to the core waves:
1. Every integration spec seeds billing models with `new BillingCustomer()` /
`new BillingProcessedEvent()` / `Subscription` / `UsageEvent` / `InvoiceSnapshot`
and `.save()` WITHOUT the required NOT-NULL `provider` column — production code
sets it (the webhook controller's ledger insert, BillingService), but the test
seeders never did, so they rotted when the multi-provider refactor made
`provider` required. Add `provider = 'stripe'` (the MockStripe driver the suites
inject) to all 63 seed sites.
2. The demo app's billing_processed_events migration declared
`status enum('pending','completed','failed')` — MISSING 'processing', which the
worker's claim step (process_billing_event_job SET status='processing') and the
published stub both require. The stale enum made the demo/fixture DB reject every
worker claim once the provider inserts started landing. Sync the fixture migration
to the stub + the code (add 'processing').
Billing integration now 219 passed / 12 skipped. No production code changed —
only test seeders and the demo migration. (Existing local DBs also need the
constraint widened; a fresh migrate now creates it correctly.)
…ubsystem installers (PLD-1) The capstone. MultitenancyProvider was a ~545-line God-object owning every subsystem's register/boot/start/ready/shutdown wiring inline. Split it into 11 per-subsystem installers under providers/installers/, each owning its slice of the lifecycle, behind a small ProviderInstaller interface + a shared InstallerContext (app + warnWhenBooted + addDisposer). The provider collapses to a thin orchestrator that iterates the ordered INSTALLERS array for register/boot/start/ready and keeps the proven 3-step shutdown. This dogfoods the same decomposition the package already ships as definePlugin for satellites, and mirrors native AdonisJS idiom (@adonisjs/core's AppServiceProvider runs an ordered sequence of per-concern register methods; @adonisjs/lucid delegates to internal src/bindings/ modules kept off the public surface). Absorbs PLD-6: the old #disposers registry generalizes to ctx.addDisposer, still torn down LIFO in shutdown(). Installers (array = boot order; the only hard within-boot edge is FoundationWiring's setConfig before any getConfig() consumer): foundation, isolation, resolution, tenancy-context, resilience, diagnostics, audit, plugin-platform, queue, scheduler, http-extensions. IsolationWiring owns the unified-adapter construction (its hard dependency is the driver set; the resolver registry is a lazily-read soft dependency), so no boot-order coupling to ResolutionWiring. Zero behavior change on any successful boot, verified adversarially against the pre-split monolith: all 24 singletons map 1:1, every condition and every warning/error string is byte-identical, assertConfigShape is byte-equivalent, the lazy-vs-eager import discipline is preserved, and shutdown() is untouched. The decomposition necessarily fixes the relative order of a few independent side-effects (installers own contiguous lifecycle slices, so the monolith's interleaved start()/ready() ordering cannot be bit-reproduced): the start-phase driver/plugin-limit asserts now run before the route-macro install, and ready() wires cache-invalidation before arming the scheduler. Both were verified behavior-neutral for every successful boot — observable only in which exception surfaces first on a multi-fault failed boot (which aborts either way) and in log-line ordering. Failing fast on a misconfigured driver before doing route autoload is, if anything, the better order. Guards: - Repoint the three grep-the-provider-file anti-regression locks (resolution_safety_single_source, membership_gate_boot_wired, rowscope_rls_probe_wired) at the installer files that now own that wiring. - Extend no_adhoc_stateful_services' NEW_ALLOWED to providers/installers/*_wiring.ts (the register() slices are the new single authority for the singleton list) and drop the provider from the allowlist — it no longer `new`s anything. - Close a real gap: rename no_app_booted_imports_in_sdk_and_testing -> no_app_booted_imports and extend its roots to src/providers/, so a static import of a booted-touching module in any installer (which would make the provider — and the three integration specs that construct it — unloadable outside an Ignitor) fails structurally instead of by hand-review. Gate: build + typecheck (all workspaces) + test 1709/0 + test:integration 396/8 + lint + check 37/37 + test:integrity + api-extractor goldens in sync (the provider is internal, not golden-tracked — no regen).
The architecture-roadmap waves (0-6) re-introduced AI-flavored punctuation in code comments: em-dash separators, prose arrows (->, →, ⇒), and the balanced "not X, but Y" / "X — so Y" cadence. This rewrites those comments into the plain, calm voice of an engineer leaving a note for a teammate, keeping every technical fact and every "why" (often a touch clearer, since the comments double as learning notes). Scope: 48 wave-touched core source files. Comments only. Verified by the AST comment oracle (the TypeScript parser's leaf-token stream is byte-identical to HEAD for every file, so no code, identifier, or string literal changed — the warning/error message strings with their own em-dashes were deliberately left untouched). Gate: build, unit 1709/0, check 37/37, lint. Integration is unaffected by a proven comments-only change. No em-dash, no prose arrow, no robotic antithesis before adding a comment: this is the standing house voice, not a one-off cleanup.
Lasagna 090706/soft
Arcoders
added
qa: product-testing
Owner
Author
|
“The Lasagna Tenant Doctor already knows what you’ve got, but still won’t prescribe you anything... 🩺 We’ve been testing the doctor and, like a good GP, it tells you “you have a fever” (yes, it detects symptoms), but when you ask “so what do I take?” it just stares at the ceiling. So we’ve decided enough is enough, diagnoses without a cure are over. We’re going to make it actually heal: |
Owner
Author
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v1.0: getting the core ready and breaking the rest into satellites
This branch started as a focused pass on tenant isolation and a benchmark suite,
but over the last few weeks it became the place where 1.0 actually came together.
The goal was easy to say and slow to finish: make the core solid enough to put in
front of real traffic, turn the optional pieces into things you install rather
than inherit, and write docs that don't promise more than what's here.
Making isolation safe by default
Most of the early work went here. A swapped tenant id, a custom domain that
doesn't match, a tenant database that dies in the middle of a request, those were
the kind of thing you used to discover in production. Now they fail closed and
there are tests that prove it, including a row-level-security check that breaks CI
the moment the guarantee slips. The performance and resilience work landed next to
that: when a tenant's database is unreachable the request returns a
503insteadof dragging the shared connection down with it, tenant resolution caches what it
safely can, and the connection pool lets in-flight requests finish instead of
cutting them off.
Splitting the optional pieces out
The bigger change was structural. Everything that isn't strictly the isolation
core (admin, SSO, billing, backup, and now WebSockets) moved out into its own
package, with a small SDK contract and a
configurecommand so you only pull inwhat you need. Billing in particular stopped being Stripe-only and now sits behind
a provider interface with Stripe, Paddle, Lemon Squeezy and a mock driver, so
nobody is locked to one vendor. The whole satellite set is versioned at
1.0.0.A word on compliance
People kept asking whether Lasagna helps with SOC2 and GDPR. The honest answer is
that it can't make you compliant, but it can hand you the controls and the
evidence, so the branch adds audit-log export, a GDPR anonymization command that
runs your own erasure logic, and a posture report, with a docs page that's careful
not to claim more than that.
The unglamorous part
This is the work that makes a release real: a benchmark suite with a captured
baseline, an end-to-end suite that hammers the failure modes under load, a shared
test kit so every satellite gets the same integration treatment, a dependency
cleanup with pinned actions and Renovate keeping an eye on them, and a full docs
rewrite that retired the old design system and is honest about where things stand
(MySQL, for instance, isn't in 1.0).
Where it stands
The core and the SDK are release candidates, which means the API is final and
green in CI against real Postgres and Redis, but nothing is being called
stableyet. That label is waiting on an outside security review and some real production
mileage. Until then a corrective change could still land in a 1.x minor, and it
would come with a loud changelog note if it did.
الحمد لله الذي بِنِعْمَتِهِ تتم الصالحات وبشكره تدوم النعم