feat: gate MCP killswitch rollout

.changeset/mcp-killswitch-rollout.md

@@ -0,0 +1,5 @@
+---
+"server": patch
+---
+
+Authenticated MCP tool-call Killswitch evaluation now has a default-off server-side shadow and enforcement rollout gate, with audited break-glass deactivation and operational readiness guidance.

docs/runbooks/mcp-killswitch-evaluator-incidents.md

@@ -0,0 +1,95 @@
+# MCP Killswitch Evaluator Incidents
+
+Use this runbook when authenticated MCP `tools/call` traffic is denied because
+the authoritative Killswitch checkpoint cannot evaluate PostgreSQL state, or
+when rollout telemetry indicates unsafe evaluator behavior. The registered M2
+policy fails closed in enforce mode.
+
+## Immediate response
+
+1. Declare an incident and assign an incident commander and database owner.
+2. Stop rollout cohort expansion. Do not create or edit prescriptions.
+3. Check `killswitch.evaluation.duration{outcome:evaluator_failure}`, p95/p99
+   latency, PostgreSQL health, pool saturation, lock waits, and deploy changes.
+4. Compare hosted and private proxy outcomes. A one-surface failure can indicate
+   a mixed or unhealthy serving fleet.
+5. Determine whether lifecycle APIs can still reach PostgreSQL. Do not assume
+   that management access works merely because it bypasses evaluation.
+
+Do not copy bearer tokens, external notes, user identifiers, organization
+identifiers, server identifiers, or customer request bodies into logs, tickets,
+dashboards, or comments.
+
+## Safe mitigation order
+
+When lifecycle access is healthy and PostgreSQL is not saturated:
+
+1. Enumerate active and scheduled prescriptions through the restricted platform
+   management path.
+2. Lift or deactivate them with unique operation IDs. Deactivation remains
+   available while activation and change are rollout-gated.
+3. Verify each operation is audited and no active or scheduled prescriptions
+   remain.
+4. Switch affected cohorts to rollout off and verify every serving instance
+   received the mode and calls continue without an evaluator query.
+5. Repair the evaluator or database, restore shadow, observe, then re-enter the
+   normal rollout gate.
+
+When PostgreSQL latency, pool saturation, or lock load is itself the incident,
+switch affected cohorts off first and prove the local flag converged across the
+fleet. After load stabilizes, enumerate and deactivate prescriptions in bounded,
+paginated batches. If flag convergence cannot be proved, prefer deactivation
+before any binary rollback.
+
+When lifecycle access is also unavailable, use the approved out-of-band
+configuration path to switch affected cohorts off from the locally cached
+feature state. Confirm the new local state reached every serving instance. If
+that path cannot converge, perform an explicitly approved full-fleet rollback.
+A partial binary rollback is unsafe while prescriptions might remain active.
+
+## Break-glass rules
+
+- Break-glass credentials are restricted to approved incident responders.
+- Prefer deactivation. Activation or change during an evaluator incident needs
+  explicit incident-commander approval and an audited reason.
+- Use a new operation ID for each intended mutation; never retry with altered
+  input under an existing operation ID.
+- Confirm organization authorization and tenant binding. Never infer authority
+  from email, API-key ownership, creator fields, or cached attribution.
+- Record only prescription IDs and internal evidence in the restricted incident
+  system. Keep concrete customer identifiers out of repository artifacts.
+
+Break-glass lifecycle operations still depend on PostgreSQL. They do not solve a
+total database outage.
+
+## Recovery checks
+
+Before leaving off mode:
+
+1. PostgreSQL query latency, lock waits, CPU, and pool saturation are healthy.
+2. Every serving instance runs the approved checkpoint build.
+3. Hosted and private proxy coverage metrics are present.
+4. In a controlled environment, unmatched, matched, and evaluator-failure
+   outcomes are observable and bounded.
+5. Shadow traffic runs through a peak period without exceeding approved latency,
+   failure, coverage, or database-load thresholds.
+6. A controlled activation denies the next matching call, lift restores the
+   next call, and expiry follows database time.
+7. The incident commander and service owner approve cohort enforcement.
+
+## Rollback validation
+
+After mitigation or binary rollback:
+
+- no mixed serving version remains;
+- no active or scheduled prescription can be silently bypassed;
+- off mode performs no evaluator query;
+- ordinary MCP traffic is restored;
+- audit records exist for every break-glass mutation;
+- monitors have returned to baseline;
+- unsupported identities, resources, and methods remain outside the published
+  coverage boundary.
+
+Do not add a TTL negative cache as an incident fix. A future summary or cache
+must be transactionally updated with prescription activation and changes so the
+next call observes authoritative state.

docs/runbooks/mcp-killswitch-rollout.md

@@ -0,0 +1,146 @@
+# MCP Killswitch Rollout
+
+This runbook gates the authenticated MCP tool-call Killswitch rollout. Do not
+enable customer prescriptions until every serving instance has the approved
+hosted and private checkpoint build.
+
+## Coverage boundary
+
+This rollout covers `tools/call` for an authoritative, active organization user
+and a canonical organization-owned MCP server on these surfaces:
+
+- hosted MCP dispatch;
+- private remote or tunnel proxy forwarding.
+
+It does not cover anonymous sessions, API keys, assistants, chat-session end
+users, inactive users, unattributed requests, legacy toolset-only routes, meta
+or platform MCP, direct internal calls, or MCP methods other than `tools/call`.
+Do not describe this rollout as controlling chat, hooks, model inference,
+assistant work, or all AI activity.
+
+## Server-side modes
+
+Two locally evaluated PostHog flags form the backend gate. Target them by the
+organization distinct ID; do not put concrete organization identifiers in this
+repository or rollout tickets.
+
+| Mode    | `mcp-killswitch-shadow` | `mcp-killswitch-enforce` | Serving behavior                                | Management behavior                                 |
+| ------- | ----------------------- | ------------------------ | ----------------------------------------------- | --------------------------------------------------- |
+| Off     | off                     | off                      | Skip derivation and evaluation                  | Create and edit unavailable; lift remains available |
+| Shadow  | on                      | off                      | Evaluate and emit metrics, but always continue  | Create and edit unavailable; lift remains available |
+| Enforce | either                  | on                       | Apply matched denial and registered fail policy | Create and edit available                           |
+
+Enforce takes precedence if both flags are on. Missing, unavailable, or
+indeterminate local evaluation resolves to off. A successfully cached PostHog
+result remains in effect until the local poller refreshes it, so every mode
+change must be verified on every serving instance. Flag evaluation stays local
+to the process; the serving path does not perform a remote PostHog request.
+
+## Fleet-readiness gate
+
+Before enabling shadow for any production cohort:
+
+1. Identify the approved build containing both hosted and private checkpoints.
+2. Prove every server instance and tunnel/private forwarding instance runs that
+   build. Drain old instances; a mixed fleet is not ready.
+3. Confirm dashboards and monitors below are live and owned.
+4. Confirm no active production prescriptions exist before a mixed-version
+   deploy. If any exist, lift them before changing serving versions.
+5. Enable shadow for a controlled cohort. Then confirm both
+   `gram.mcp.killswitch.surface:hosted` and
+   `gram.mcp.killswitch.surface:private_proxy` appear in
+   `mcp.tool.call.killswitch_identity`. Off mode intentionally emits no coverage
+   observation.
+6. Confirm `killswitch.evaluation.duration` emits
+   `gram.outcome:matched`, `gram.outcome:unmatched`, and
+   `gram.outcome:evaluator_failure` in a non-production or controlled test. Do
+   not create synthetic customer prescriptions.
+
+Record the build, instance inventory, evidence links, reviewer, and timestamp in
+the internal rollout record. Repository tests cannot prove fleet uniformity.
+
+## Shadow observation
+
+Start with an internal or restricted cohort. Keep shadow active for at least one
+normal traffic cycle and one peak period. Select quantitative thresholds from a
+reviewed baseline; the values below are initial stop conditions, not permanent
+SLOs.
+
+Observe:
+
+- p95 and p99 of `killswitch.evaluation.duration`; p99 must remain comfortably
+  below the one-second private checkpoint timeout and two-second hosted timeout;
+- `outcome:evaluator_failure` ratio; stop on any sustained rate above 0.1% for
+  five minutes or any correlated serving incident;
+- matched and unmatched query volume; histogram count is the authoritative
+  evaluator-query count;
+- active-user plus canonical-server coverage by `surface`; investigate any
+  increase in `unavailable`, `invalid_owner`, or unsupported classes;
+- PostgreSQL query rate, latency, CPU, lock waits, and pool saturation compared
+  with the pre-shadow baseline. Stop on a sustained 10% load increase or pool
+  saturation unless the database owner approves a different bound.
+
+Shadow suppresses transport denial, including fail-closed evaluator outcomes.
+It does not suppress evaluator and coverage telemetry. Shadow evaluation is
+synchronous by design so it measures the serving-path cost that enforce mode
+will add; restrict cohort size and monitor end-to-end MCP request latency.
+
+## Datadog monitors
+
+Create these in Datadog; monitor configuration is not stored in this repository.
+Scope every monitor by environment. Evaluator duration has no surface attribute;
+only the identity-coverage counter can be split by surface. Tune only after a
+reviewed baseline.
+
+1. **Evaluator failure ratio**: count of
+   `killswitch.evaluation.duration{gram.outcome:evaluator_failure}` divided by
+   all outcomes over five minutes. Warn at 0.05%; alert at 0.1%.
+2. **Evaluator latency**: p95 and p99 of `killswitch.evaluation.duration`. Warn
+   at 250 ms; alert at 500 ms.
+3. **Coverage regression**: ratio of
+   `mcp.tool.call.killswitch_identity{gram.mcp.killswitch.identity_class:active_user,gram.mcp.killswitch.resource_class:canonical_server}`
+   to all observations, split by `gram.mcp.killswitch.surface`. Alert on a
+   reviewed baseline regression, not on an arbitrary global percentage.
+4. **Coverage unavailable**: any sustained
+   `gram.mcp.killswitch.identity_class:unavailable` or
+   `gram.mcp.killswitch.resource_class:unavailable`, plus
+   `gram.mcp.killswitch.resource_class:invalid_owner` above baseline.
+5. **PostgreSQL load**: query latency, pool saturation, lock waits, CPU, and
+   evaluator query volume. Correlate changes with shadow cohort expansion.
+
+Metric dimensions are bounded server classes. Never add organization IDs, user
+IDs, server IDs, notes, URLs, or error text as metric tags.
+
+## Enforce progression
+
+Enable `mcp-killswitch-enforce` only after the fleet gate and shadow criteria
+pass. Progress one restricted cohort at a time. For each cohort:
+
+1. Enable enforce while shadow remains on.
+2. Create a controlled prescription through the audited management path.
+3. Verify the next matching call is denied with the exact external note. No
+   restart, cache expiry, or propagation delay is expected.
+4. Verify a non-matching call continues.
+5. Lift the prescription and verify the next call continues.
+6. Exercise a bounded expiry and verify database time restores the next call.
+7. Review all monitors before expanding the cohort.
+
+There is no TTL allow/deny cache. Any future summary optimization must be
+updated transactionally with activation or change and visible on the next call.
+
+## Rollback criteria
+
+Stop expansion and roll back the cohort to off when any of these occurs:
+
+- a mixed serving fleet is detected;
+- evaluator failures exceed the approved threshold;
+- p99 approaches a checkpoint timeout;
+- authoritative identity or canonical-resource coverage regresses;
+- PostgreSQL load exceeds the approved bound;
+- denial text or JSON-RPC behavior differs from acceptance evidence.
+
+Before rolling binaries backward, stop cohort expansion, lift active
+prescriptions, verify no active or scheduled prescriptions remain, switch the
+cohort off, and then drain newer instances. Never leave active prescriptions
+while an older instance can bypass evaluation. Use the evaluator incident
+runbook for an active failure.

server/cmd/gram/start.go

@@ -1060,7 +1060,7 @@ func newStartCommand() *cli.Command {
 
 			toolDispositionCache := mcpservers.NewToolDispositionCache(logger, db, cache.NewRedisCacheAdapter(redisClient))
 			var platformSelectedUseRecorder toolcallobserver.SuccessRecorder = platformmcp.NewSelectedUseRecorder(db)
-			mcpToolExecutionCheckpoint, err := mcptoolexecution.NewCheckpoint(db, mcptoolexecution.DefaultEvaluationTimeout, meterProvider, logger)
+			mcpToolExecutionCheckpoint, err := mcptoolexecution.NewCheckpoint(db, mcptoolexecution.DefaultEvaluationTimeout, meterProvider, logger, featureFlags)
 			if err != nil {
 				return fmt.Errorf("initialize mcp tool-execution checkpoint: %w", err)
 			}
@@ -1526,13 +1526,13 @@ func newStartCommand() *cli.Command {
 			chatanalysis.Attach(mux, chatanalysis.NewService(logger, tracerProvider, db, sessionManager, authzEngine, auditLogger,
 				&background.TemporalChatAnalysisSignaler{TemporalEnv: temporalEnv, Logger: logger}))
 			openrouterkeys.Attach(mux, openrouterkeys.NewService(logger, tracerProvider, db, sessionManager, authzEngine, auditLogger, openRouter, encryptionClient))
-			// Platform break-glass remains separately unavailable; this composition is
-			// intentionally customer-only and enforces ordinary live admin sessions.
-			killswitches.AttachPlatformService(mux, killswitches.NewPlatformService(logger, tracerProvider, db, sessionManager, authzEngine, nil))
-			killswitchService, err := killswitchapi.NewService(logger, tracerProvider, db, sessionManager, authzEngine, auditLogger)
+			killswitchService, err := killswitchapi.NewService(logger, tracerProvider, db, sessionManager, authzEngine, auditLogger, featureFlags)
 			if err != nil {
 				return fmt.Errorf("build customer killswitch service: %w", err)
 			}
+			// Platform break-glass shares the validated lifecycle so list and
+			// deactivation remain available when rollout enforcement is off.
+			killswitches.AttachPlatformService(mux, killswitches.NewPlatformService(logger, tracerProvider, db, sessionManager, authzEngine, killswitchService.GenericService()))
 			killswitchapi.Attach(mux, killswitchService)
 			skillsService := skills.NewService(logger, tracerProvider, db, sessionManager, authzEngine, productFeatures, auditLogger,
 				&background.TemporalSkillSuggestionSignaler{TemporalEnv: temporalEnv, Logger: logger, StartDelay: 0}, siteURL)

server/internal/feature/flags.go

@@ -36,6 +36,13 @@ const (
 	FlagRiskFindingAnalytics Flag = "risk-finding-analytics"
 	FlagRiskAsyncScanShadow  Flag = "risk-async-scan-shadow"
 
+	// FlagMCPKillswitchShadow and FlagMCPKillswitchEnforce form the server-side
+	// rollout gate for authenticated MCP tools/call evaluation. Both are
+	// evaluated from PostHog's local cache on every call. Enforce takes
+	// precedence when both are enabled; absent or indeterminate flags mean off.
+	FlagMCPKillswitchShadow  Flag = "mcp-killswitch-shadow"
+	FlagMCPKillswitchEnforce Flag = "mcp-killswitch-enforce"
+
 	// FlagPlatformMCPRiskMutations is the exact-project kill switch for risk
 	// policy and exclusion writes exposed through Platform MCP. It is evaluated
 	// at invocation time and fails closed when absent, disabled, or indeterminate.

server/internal/killswitchapi/service.go

@@ -27,6 +27,7 @@ import (
 	"github.com/speakeasy-api/gram/server/internal/authz"
 	"github.com/speakeasy-api/gram/server/internal/constants"
 	"github.com/speakeasy-api/gram/server/internal/conv"
+	"github.com/speakeasy-api/gram/server/internal/feature"
 	"github.com/speakeasy-api/gram/server/internal/killswitches"
 	"github.com/speakeasy-api/gram/server/internal/killswitches/mcptoolexecution"
 	killswitchrepo "github.com/speakeasy-api/gram/server/internal/killswitches/repo"
@@ -49,19 +50,24 @@ type Service struct {
 	auth       *gramauth.Auth
 	db         *pgxpool.Pool
 	authorized *killswitches.AuthorizedService
+	generic    killswitches.GenericService
 	user       killswitches.PrincipalAdapter
 	server     killswitches.ResourceAdapter
+	features   feature.Provider
 }
 
 var _ gen.Service = (*Service)(nil)
 var _ gen.Auther = (*Service)(nil)
 
-func NewService(logger *slog.Logger, tracerProvider trace.TracerProvider, db *pgxpool.Pool, sessionManager *sessions.Manager, authzEngine *authz.Engine, auditLogger *audit.Logger) (*Service, error) {
+func NewService(logger *slog.Logger, tracerProvider trace.TracerProvider, db *pgxpool.Pool, sessionManager *sessions.Manager, authzEngine *authz.Engine, auditLogger *audit.Logger, features feature.Provider) (*Service, error) {
 	registry, err := mcptoolexecution.NewRegistry(db)
 	if err != nil {
 		return nil, fmt.Errorf("build MCP tool-call killswitch registry: %w", err)
 	}
-	lifecycle, err := killswitches.NewLifecycleService(db, registry, mcptoolexecution.NewCustomerLifecycleValidator(), killswitches.NewAuditBeforeCommitHook(auditLogger))
+	lifecycle, err := killswitches.NewLifecycleService(
+		db, registry, mcptoolexecution.NewCustomerLifecycleValidator(), killswitches.NewAuditBeforeCommitHook(auditLogger),
+		killswitches.WithBeforeApplyHook(rolloutBeforeApply(features)),
+	)
 	if err != nil {
 		return nil, fmt.Errorf("build killswitch lifecycle service: %w", err)
 	}
@@ -83,11 +89,31 @@ func NewService(logger *slog.Logger, tracerProvider trace.TracerProvider, db *pg
 	}
 	return &Service{
 		tracer: tracerProvider.Tracer("github.com/speakeasy-api/gram/server/internal/killswitchapi"), logger: logger,
-		auth: gramauth.New(logger, db, sessionManager, authzEngine), db: db, authorized: authorized,
-		user: user, server: server,
+		auth: gramauth.New(logger, db, sessionManager, authzEngine), db: db, authorized: authorized, generic: facade,
+		user: user, server: server, features: features,
 	}, nil
 }
 
+func (s *Service) GenericService() killswitches.GenericService {
+	return s.generic
+}
+
+func rolloutBeforeApply(features feature.Provider) killswitches.BeforeApplyHook {
+	return func(ctx context.Context, mutation killswitches.MutationContext, operation killswitches.MutationOperation) error {
+		if operation == killswitches.MutationOperationDeactivate {
+			return nil
+		}
+		mode, err := mcptoolexecution.ResolveRolloutMode(ctx, features, string(mutation.OrganizationID))
+		if err != nil {
+			return fmt.Errorf("%w: resolve MCP killswitch rollout mode: %w", killswitches.ErrOperationUnavailable, err)
+		}
+		if mode != mcptoolexecution.RolloutModeEnforce {
+			return fmt.Errorf("%w: killswitch activation is not enabled for this organization", killswitches.ErrOperationUnavailable)
+		}
+		return nil
+	}
+}
+
 func Attach(mux goahttp.Muxer, service *Service) {
 	endpoints := gen.NewEndpoints(service)
 	endpoints.Use(middleware.MapErrors())
@@ -737,8 +763,7 @@ func mapError(err error) error {
 	if err == nil {
 		return nil
 	}
-	var shareable *oops.ShareableError
-	if errors.As(err, &shareable) {
+	if _, ok := errors.AsType[*oops.ShareableError](err); ok {
 		return err
 	}
 	switch {