diff --git a/CHANGELOG.md b/CHANGELOG.md index be9fc56..1acf65b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,13 @@ All notable changes to this project are documented here. Format follows ## [Unreleased] +### Added +- Every policy reload is recorded: a `PolicyChanged` span (applied, ignored, + rejected or failed, with the versions and the SHA-256 of the submitted text) + in a `policy-change` run owned by that tenant, exported to the SIEM at + severity `medium`, and a `POLICY` line in the security log. Earlier versions + cannot replay these runs, since they do not know the event type. + ### Security - `versions.lock` moves past newly published advisories: certifi 2026.10.19 (PYSEC-2026-4320), and for the pre-commit tooling filelock 4.2.1 (three CVEs) @@ -16,6 +23,17 @@ All notable changes to this project are documented here. Format follows fresh install already picks up the fixed certifi. ### Fixed +- A policy replaced through `POST /tenants/{tenant}/policy` survives a restart. + It lived only in memory, so a restart silently put the configured file back, + and nothing recorded that the policy had changed. An applied policy is now + saved under `${SENTINEL_DATA_DIR}/policies/` before it takes effect (a failed + save changes nothing and answers 500). At startup a saved policy replaces the + configured one only when its `policy_version` is higher. Reloads are + serialized, so two at once cannot save one version and enforce another. + Files are named by a SHA-256 of the tenant, so any tenant name works and none + can escape the directory or collide with another on a case-insensitive file + system. A saved policy that cannot be read or no longer loads stops + `sentinel serve` and `sentinel check` with an error naming the file. - Retention could keep a trace forever. The SQLite store filled in span timestamps only while adding the `ts` column, but 0.2.2 can still write to a migrated database and leaves `ts` empty, so after a rollback and a second diff --git a/README.md b/README.md index 57c9f7e..4cf7ea1 100644 --- a/README.md +++ b/README.md @@ -168,7 +168,7 @@ Read this before deploying anything. Each limitation changes what SENTINEL prote | **Only guarded tools are guarded** | A tool without rules can carry tainted data out. Add rules for every outbound tool; there is no general notion of an outbound "sink" yet | | **Quarantine is recorded, not enforced** (default) | The trust score never recovers, so agents making only allowed calls cross the threshold within tens of calls. `SENTINEL_ENFORCE_QUARANTINE=1` enforces it | | **Downstream servers: streamable HTTP only** | No stdio servers, no per-server credentials or OAuth, one shared session per server, and no reconnect. Most published MCP servers are stdio | -| **State lives in one process** | Trust scores, MCP sessions, the event feed, the run index and API policy reloads are in memory. Two replicas share none of it | +| **State lives in one process** | Trust scores, MCP sessions, the event feed and the run index are in memory. Two replicas share none of it, and a policy replaced through the API is saved only on the replica that received it | | **Cosmos persistence is partial** | Needs `pip install "sentinel-prox[cosmos]"`; run ownership, the run index after a restart and retention purging exist only for SQLite | | **Some state is visible across tenants** | `/events` ids are one global counter; `/downstream` and `/capabilities` describe the whole deployment | | **Calls the SDK rejects are not recorded** | Arguments that fail a tool's input schema are refused by the MCP SDK before SENTINEL sees them | diff --git a/src/sentinel/authorization/registry.py b/src/sentinel/authorization/registry.py index 53d0333..739d47e 100644 --- a/src/sentinel/authorization/registry.py +++ b/src/sentinel/authorization/registry.py @@ -64,7 +64,10 @@ def reload_if_newer(self, tenant: str, text: str) -> ReloadResult: Raises ``PolicyLoadError`` (from :func:`load_policy`) on malformed input, BEFORE touching the active policy — a bad push cannot take a tenant down. """ - candidate = load_policy(text) + return self.apply_if_newer(tenant, load_policy(text)) + + def apply_if_newer(self, tenant: str, candidate: CompiledPolicy) -> ReloadResult: + """Swap in an already-compiled ``candidate`` for ``tenant`` if it is newer.""" current = self._policies.get(tenant) if current is None: self._policies[tenant] = candidate diff --git a/src/sentinel/control/app.py b/src/sentinel/control/app.py index bcf2ea8..87af3dd 100644 --- a/src/sentinel/control/app.py +++ b/src/sentinel/control/app.py @@ -635,11 +635,16 @@ async def list_tenants(request: Request) -> dict[str, Any]: @app.post("/tenants/{tenant}/policy", dependencies=[Depends(require_admin)]) async def reload_tenant_policy(tenant: str, body: PolicyReloadRequest) -> dict[str, Any]: - """Hot-reload a tenant's policy on version bump (rejects malformed).""" + """Replace a tenant's policy on version bump; saved, so it survives a restart.""" try: - result = mgr.reload_policy(tenant, body.policy) + result = await mgr.reload_policy(tenant, body.policy) except PolicyLoadError as exc: raise HTTPException(status_code=400, detail=str(exc)) from exc + except OSError as exc: + raise HTTPException( + status_code=500, + detail="the policy could not be saved, so it was not applied; see the server log", + ) from exc return { "tenant": result.tenant, "reloaded": result.reloaded, diff --git a/src/sentinel/control/manager.py b/src/sentinel/control/manager.py index a9ca31c..47e90d5 100644 --- a/src/sentinel/control/manager.py +++ b/src/sentinel/control/manager.py @@ -16,7 +16,9 @@ import asyncio import functools +import hashlib import importlib.util +import json import logging import os import time @@ -30,7 +32,12 @@ from sentinel.authn import tenant_credentials, tenant_policy_paths from sentinel.authorization.engine import AuthorizationEngine -from sentinel.authorization.policy import CompiledPolicy, load_default_policy, load_policy +from sentinel.authorization.policy import ( + CompiledPolicy, + PolicyLoadError, + load_default_policy, + load_policy, +) from sentinel.authorization.registry import DEFAULT_TENANT, PolicyRegistry, ReloadResult from sentinel.classifier import AttackClassifier from sentinel.classifier.attack_classifier import BlockedAttempt @@ -49,6 +56,8 @@ AgentQuarantined, AuthorizationDecided, InjectionScanned, + PolicyChanged, + PolicyOutcome, ToolBlocked, ) from sentinel.forensics.replay import TraceReplay, replay @@ -59,6 +68,7 @@ ) from sentinel.mcp_proxy.proxy import SentinelProxy from sentinel.mcp_proxy.router import DownstreamConnection +from sentinel.observability import log_policy_change from sentinel.shield import InputShield from sentinel.trust.config import load_default_trust_config from sentinel.trust.scorer import TrustScorer @@ -104,6 +114,101 @@ def _load_policy_file(path_text: str, *, what: str) -> CompiledPolicy: _LOG = logging.getLogger("sentinel.control.manager") +def _data_dir() -> Path: + return Path(os.environ.get("SENTINEL_DATA_DIR", "var")) + + +def saved_policy_dir() -> Path: + """Where policies applied through ``POST /tenants/{tenant}/policy`` are kept.""" + return _data_dir() / "policies" + + +def _saved_policy_path(tenant: str) -> Path: + # Named by a digest of the tenant rather than the tenant itself: any tenant + # name is valid, none can reach outside the directory, and "acme" and + # "ACME" cannot share a file on a case-insensitive file system. + digest = hashlib.sha256(tenant.encode("utf-8")).hexdigest() + return saved_policy_dir() / f"{digest}.json" + + +def _save_policy(tenant: str, text: str, version: int) -> None: + """Write ``tenant``'s policy so a restart keeps it: the old file or the new. + + Staged, flushed to disk, then renamed over the old copy, so a crash leaves + one complete file or the other, never half of one. + """ + path = _saved_policy_path(tenant) + path.parent.mkdir(parents=True, exist_ok=True) + staging = path.with_name(f".{path.name}.tmp") + record = { + "tenant": tenant, + "policy_version": version, + "saved_at": datetime.now(UTC).isoformat(), + "policy": text, + } + with staging.open("w", encoding="utf-8") as handle: + json.dump(record, handle, ensure_ascii=False) + handle.flush() + os.fsync(handle.fileno()) + os.replace(staging, path) + if hasattr(os, "O_DIRECTORY"): # POSIX: make the rename itself durable + fd = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY) + try: + os.fsync(fd) + finally: + os.close(fd) + + +def _overlay_saved_policies(registry: PolicyRegistry) -> None: + """Apply the policies saved by earlier API reloads, where newer than configured. + + A reload used to live only in memory, so a restart silently put the + configured file back. The rule that decides a reload decides here too: the + higher ``policy_version`` wins, so an operator who bumps the configured file + past a saved copy gets the file. + + A saved policy that cannot be read or no longer loads stops startup, as a + broken configured file does: it is the operator's latest policy, and + quietly enforcing an older one instead would be the worse surprise. + """ + directory = saved_policy_dir() + if not directory.is_dir(): + return + for path in sorted(directory.glob("*.json")): + try: + record = json.loads(path.read_text(encoding="utf-8")) + tenant, text = record["tenant"], record["policy"] + if not isinstance(tenant, str) or not isinstance(text, str): + raise TypeError("'tenant' and 'policy' must be strings") + except (OSError, ValueError, KeyError, TypeError) as exc: + raise ValueError( + f"saved policy {path} cannot be read ({exc}); repair or delete it" + ) from exc + if path.name != _saved_policy_path(tenant).name: + raise ValueError( + f"saved policy {path} is for tenant {tenant!r} but is not named for it; " + "repair or delete it" + ) + try: + saved = load_policy(text) + except PolicyLoadError as exc: + raise ValueError( + f"saved policy for tenant {tenant!r} ({path}) does not load: {exc}; " + "repair or delete it" + ) from exc + current = registry.get(tenant) + if current is None or saved.policy_version > current.policy_version: + registry.register(tenant, saved) + _LOG.warning( + "tenant %r: enforcing the policy saved by an API reload (v%d) %s", + tenant, + saved.policy_version, + "(no policy is configured for it)" + if current is None + else f"instead of the configured v{current.policy_version}", + ) + + def build_policy_registry(settings: Settings) -> PolicyRegistry: """Give every provisioned tenant a policy, and nobody else. @@ -116,6 +221,8 @@ def build_policy_registry(settings: Settings) -> PolicyRegistry: that authenticated callers and then could not execute a single action. * A tenant with neither a credential nor a file has no policy and is refused. That fail-closed behaviour is unchanged. + * A policy saved by an API reload replaces any of these that it is newer + than (see :func:`_overlay_saved_policies`). Failing loudly on a bad path is deliberate: silently running the example policy while the operator believes theirs is active would be a security @@ -137,6 +244,7 @@ def build_policy_registry(settings: Settings) -> PolicyRegistry: for tenant in tenant_credentials(): if tenant not in explicit and registry.get(tenant) is None: registry.register(tenant, deployment) + _overlay_saved_policies(registry) return registry @@ -243,6 +351,10 @@ def __init__( # Persistent history is a startup import, not an index refill on every # GET /runs. Re-adding every evicted trace defeated max_retained_runs. self._persisted_runs_restored = False + # One policy reload at a time. Two interleaved reloads could otherwise + # save one version and activate another, and a restart would then + # bring back the policy that lost. + self._policy_lock = asyncio.Lock() @property def bus(self) -> EventBus: @@ -684,9 +796,99 @@ def list_tenants(self) -> list[dict[str, Any]]: for t in self._registry.tenants() ] - def reload_policy(self, tenant: str, policy_text: str) -> ReloadResult: - """Hot-reload a tenant's policy on version bump (raises on malformed).""" - return self._registry.reload_if_newer(tenant, policy_text) + async def reload_policy(self, tenant: str, policy_text: str) -> ReloadResult: + """Replace ``tenant``'s policy if the submitted one is newer; save it; record it. + + A reload used to live only in memory, so a restart silently put the + configured file back, and nothing recorded that the policy had changed. + Now an applied policy is saved under the data directory BEFORE it takes + effect, so a failed save changes nothing, and every attempt becomes a + ``PolicyChanged`` record in the tenant's history: applied, ignored (not + newer), rejected (did not load) or failed (could not be saved). + + Raises :class:`PolicyLoadError` for a policy that does not load and + :class:`OSError` for one that could not be saved, after recording either. + """ + digest = hashlib.sha256(policy_text.encode("utf-8")).hexdigest() + async with self._policy_lock: + previous = self._registry.version(tenant) + try: + candidate = load_policy(policy_text) + except PolicyLoadError as exc: + await self._record_policy_change( + tenant, "rejected", None, previous, digest, str(exc) + ) + raise + if previous is None or candidate.policy_version > previous: + try: + await asyncio.to_thread( + _save_policy, tenant, policy_text, candidate.policy_version + ) + except OSError as exc: + _LOG.error("could not save the policy for tenant %r: %s", tenant, exc) + await self._record_policy_change( + tenant, "failed", candidate.policy_version, previous, digest, + f"could not be saved, so it was not applied: {exc}", + ) + raise + result = self._registry.apply_if_newer(tenant, candidate) + await self._record_policy_change( + tenant, + "applied" if result.reloaded else "ignored", + candidate.policy_version, + previous, + digest, + result.detail, + ) + return result + + async def _record_policy_change( + self, + tenant: str, + outcome: PolicyOutcome, + version: int | None, + previous: int | None, + digest: str, + detail: str, + ) -> None: + """One trace per attempt, in the history of the tenant whose policy it is. + + Best effort once the decision is made: an applied policy is already in + force, so failing the request because the record could not be written + would misreport what happened. The security log line is written either way. + """ + trace_id = self._emitter.new_trace_id() + record = RunRecord( + run_id=trace_id, + trace_id=trace_id, + agent_id="operator", + scenario="policy-change", + tenant=tenant, + status="completed", + ) + self._runs[trace_id] = record + self._persist_owner(record) + try: + await self._emitter.emit( + PolicyChanged( + tenant=tenant, + outcome=outcome, + version=version, + previous_version=previous, + policy_sha256=digest, + # A load error can quote the submitted text; keep it short. + detail=detail[:300], + ), + trace_id=trace_id, + ) + except Exception: # noqa: BLE001 - see the docstring; logged, not swallowed + _LOG.exception("could not record a policy change for tenant %r", tenant) + finally: + self._emitter.forget(trace_id) + self._prune_finished_runs() + log_policy_change( + tenant, outcome=outcome, version=version, previous=previous, trace_id=trace_id + ) # --- SOC audit (the classifier's ONE consumer) ----------------------------- @@ -716,6 +918,9 @@ async def audit(self, trace_id: str) -> list[dict[str, Any]]: severity = "high" elif isinstance(p, AgentQuarantined): severity = "critical" + elif isinstance(p, PolicyChanged): + # A change to what is enforced, or an attempt at one. + severity = "medium" alert: dict[str, Any] = { "timestamp": s.timestamp.isoformat(), "source": "SENTINEL", @@ -740,6 +945,14 @@ async def audit(self, trace_id: str) -> list[dict[str, Any]]: alert["attack_class"] = label.attack_class alert["attack_rationale"] = label.rationale alert["classifier"] = label.classifier + elif isinstance(p, PolicyChanged): + alert.update( + tenant=p.tenant, + outcome=p.outcome, + policy_version=p.version, + previous_version=p.previous_version, + policy_sha256=p.policy_sha256, + ) alerts.append(alert) return alerts diff --git a/src/sentinel/control/static/index.html b/src/sentinel/control/static/index.html index 2b30d58..8aab02f 100644 --- a/src/sentinel/control/static/index.html +++ b/src/sentinel/control/static/index.html @@ -521,6 +521,7 @@

SENTINEL_

else if(t==='TrustUpdated'){ lvl='TRUST'; cls='trust'; const d=p.delta>0?'+'+p.delta.toFixed(0):p.delta.toFixed(0); msg=esc(cleanReason(p.reason))+' score '+p.previous_score.toFixed(0)+'→'+p.new_score.toFixed(0)+' ('+d+')'; } else if(t==='AgentQuarantined'&&p.enforced===false){ lvl='TRUST'; msg='trust below threshold — score '+p.score_at_quarantine.toFixed(1)+'; quarantine not enforced, calls continue'; } else if(t==='AgentQuarantined'){ lvl='QUAR'; cls='block'; msg='agent quarantined — score '+p.score_at_quarantine.toFixed(1)+' < threshold, lane disabled'; } + else if(t==='PolicyChanged'){ lvl='POLICY'; cls=p.outcome==='applied'?'allow':(p.outcome==='ignored'?'dim':'deny'); msg='tenant '+esc(p.tenant)+' '+esc(p.outcome)+' v'+(p.previous_version===null?'-':p.previous_version)+' → v'+(p.version===null?'-':p.version)+' '+esc(trunc(p.detail,80)); } else { lvl=t.slice(0,6).toUpperCase(); msg=''; } return '
'+seq+''+lvl+''+msg+'
'; } diff --git a/src/sentinel/forensics/events.py b/src/sentinel/forensics/events.py index 701e651..96a6adc 100644 --- a/src/sentinel/forensics/events.py +++ b/src/sentinel/forensics/events.py @@ -195,6 +195,27 @@ class ToolBlocked(_FrozenPayload): matched_rule_id: str | None = None +PolicyOutcome = Literal["applied", "ignored", "rejected", "failed"] + + +class PolicyChanged(_FrozenPayload): + """The operator submitted a tenant's policy, and what became of it. + + Every attempt is recorded: ``applied`` (newer, saved, now active), + ``ignored`` (not newer than the active version), ``rejected`` (did not + load) and ``failed`` (loaded, but could not be saved, so not applied). The + policy is identified by the SHA-256 of the submitted text, not copied in. + """ + + event_type: Literal["PolicyChanged"] = "PolicyChanged" + tenant: str + outcome: PolicyOutcome + version: int | None # the submitted version; None when it did not load + previous_version: int | None + policy_sha256: str + detail: str + + # The closed discriminated union. Pydantic uses the `event_type` literal to pick # the right model on parse — so a stored span round-trips back to its exact # payload class, and an unknown event_type fails loudly instead of degrading to @@ -207,7 +228,8 @@ class ToolBlocked(_FrozenPayload): | TrustUpdated | AgentQuarantined | ToolExecuted - | ToolBlocked, + | ToolBlocked + | PolicyChanged, Field(discriminator="event_type"), ] @@ -222,5 +244,6 @@ class ToolBlocked(_FrozenPayload): "AgentQuarantined", "ToolExecuted", "ToolBlocked", + "PolicyChanged", } ) diff --git a/src/sentinel/observability.py b/src/sentinel/observability.py index 945bbf0..d240507 100644 --- a/src/sentinel/observability.py +++ b/src/sentinel/observability.py @@ -243,6 +243,31 @@ def log_quarantined( ) +def log_policy_change( + tenant: str, + *, + outcome: str, + version: int | None, + previous: int | None, + trace_id: str, +) -> None: + """The operator submitted a tenant's policy: applied, ignored, rejected or failed.""" + security_logger().warning( + "POLICY %s v%s -> v%s", + outcome.upper(), + previous, + version, + extra={ + "event": "policy_change", + "outcome": outcome, + "version": version, + "previous_version": previous, + "trace_id": _safe_log_value(trace_id), + **_tenant_field(tenant), + }, + ) + + def log_injection_flagged( target: str, *, trace_id: str, shield: str, tenant: str | None = None ) -> None: diff --git a/tests/test_authorization_config_integrity.py b/tests/test_authorization_config_integrity.py index 94eff94..3531795 100644 --- a/tests/test_authorization_config_integrity.py +++ b/tests/test_authorization_config_integrity.py @@ -170,10 +170,10 @@ def test_an_argument_cannot_supply_an_undeclared_comparand() -> None: assert "fail-closed" in decision.reason -def test_live_proxy_observes_a_higher_version_after_reload() -> None: +async def test_live_proxy_observes_a_higher_version_after_reload() -> None: """A long-lived MCP proxy should not retain an obsolete policy snapshot.""" manager = RunManager(store=InMemoryForensicStore()) - manager.reload_policy("acme", "policy_version: 1\ntools: {web_fetch: {rules: []}}\n") + await manager.reload_policy("acme", "policy_version: 1\ntools: {web_fetch: {rules: []}}\n") proxy = manager.new_live_proxy( downstream=object(), # type: ignore[arg-type] - no downstream call in this unit test cache=(), @@ -182,7 +182,7 @@ def test_live_proxy_observes_a_higher_version_after_reload() -> None: ) assert proxy._current_engine().authorize(ToolCall(name="web_fetch")).decision == "ALLOW" # noqa: SLF001 - manager.reload_policy( + await manager.reload_policy( "acme", """\ policy_version: 2 diff --git a/tests/test_forensics_events.py b/tests/test_forensics_events.py index 07c6c02..5ac0a3b 100644 --- a/tests/test_forensics_events.py +++ b/tests/test_forensics_events.py @@ -1,4 +1,4 @@ -"""Phase 1: the 8 typed event payloads and their discriminated union.""" +"""Phase 1: the typed event payloads and their discriminated union.""" from __future__ import annotations import pytest @@ -10,6 +10,7 @@ AuthorizationDecided, InjectionScanned, InputReceived, + PolicyChanged, RuleEvaluation, ToolBlocked, ToolCallProposed, @@ -52,11 +53,19 @@ blocked_by="authorization", matched_rule_id="r1", ), + PolicyChanged( + tenant="acme", + outcome="applied", + version=4, + previous_version=3, + policy_sha256="0" * 64, + detail="hot-reloaded v3 → v4", + ), ] -def test_there_are_exactly_eight_event_types() -> None: - assert len(EVENT_TYPES) == 8 +def test_there_are_exactly_nine_event_types() -> None: + assert len(EVENT_TYPES) == 9 assert {p.event_type for p in ALL_PAYLOADS} == EVENT_TYPES diff --git a/tests/test_policy_durability.py b/tests/test_policy_durability.py new file mode 100644 index 0000000..91574ba --- /dev/null +++ b/tests/test_policy_durability.py @@ -0,0 +1,253 @@ +"""A policy replaced through the API survives a restart, and every attempt is on record. + +``POST /tenants/{tenant}/policy`` used to swap the compiled policy in memory +only: a restart silently put the configured file back, and nothing recorded +that the policy had changed or that anyone had tried (review F21). +""" +from __future__ import annotations + +import asyncio +import json +import os +import time +from collections.abc import AsyncIterator, Iterator +from contextlib import asynccontextmanager +from pathlib import Path + +import httpx +import pytest + +import sentinel.control.manager as manager_module +from sentinel.authorization.engine import ToolCall +from sentinel.authorization.policy import PolicyLoadError +from sentinel.config import reset_settings_cache +from sentinel.control.app import create_app +from sentinel.control.manager import RunManager, saved_policy_dir +from sentinel.forensics.events import PolicyChanged +from sentinel.forensics.store import InMemoryForensicStore + +ACME = "acme-token" # noqa: S105 - fixed literals are the point of these tests +GLOBEX = "globex-token" # noqa: S105 +ADMIN = "operator-token" # noqa: S105 + + +def _policy(version: int, *, stop: bool = False) -> str: + """A policy for web_fetch: allowed, or (``stop``) always denied.""" + rules = "\n - id: emergency-stop\n deny_always: true" if stop else " []" + return f"policy_version: {version}\ntools:\n web_fetch:\n rules:{rules}\n" + + +def _decision(manager: RunManager, tenant: str) -> str: + engine = manager.registry.engine_for(tenant) + assert engine is not None + return engine.authorize(ToolCall(name="web_fetch")).decision + + +@pytest.fixture(autouse=True) +def _fresh_settings() -> Iterator[None]: + reset_settings_cache() + yield + reset_settings_cache() + + +def _env(monkeypatch: pytest.MonkeyPatch, **values: str) -> None: + for key, value in values.items(): + monkeypatch.setenv(key, value) + reset_settings_cache() + + +async def _policy_changes(manager: RunManager, tenant: str) -> list[PolicyChanged]: + """The PolicyChanged records in ``tenant``'s history, oldest first.""" + found: list[PolicyChanged] = [] + for record in manager.list_runs(): + if record.scenario != "policy-change" or record.tenant != tenant: + continue + for span in (await manager.replay(record.trace_id)).ordered: + assert isinstance(span.payload, PolicyChanged) + found.append(span.payload) + return found + + +async def test_an_applied_reload_survives_a_restart() -> None: + first = RunManager(store=InMemoryForensicStore()) + await first.reload_policy("acme", _policy(5, stop=True)) + assert _decision(first, "acme") == "DENY" + await first.aclose() + + restarted = RunManager(store=InMemoryForensicStore()) + assert restarted.registry.version("acme") == 5 + assert _decision(restarted, "acme") == "DENY" + await restarted.aclose() + + +async def test_a_configured_file_with_a_higher_version_beats_the_saved_copy( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + configured = tmp_path / "acme.yaml" + configured.write_text(_policy(3), encoding="utf-8") + _env(monkeypatch, SENTINEL_TENANT_POLICIES=json.dumps({"acme": str(configured)})) + + first = RunManager(store=InMemoryForensicStore()) + await first.reload_policy("acme", _policy(5, stop=True)) + await first.aclose() + assert RunManager(store=InMemoryForensicStore()).registry.version("acme") == 5 + + # The operator bumps the file past the saved copy: the file wins again. + configured.write_text(_policy(7), encoding="utf-8") + restarted = RunManager(store=InMemoryForensicStore()) + assert restarted.registry.version("acme") == 7 + assert _decision(restarted, "acme") == "ALLOW" + await restarted.aclose() + + +async def test_every_attempt_is_recorded() -> None: + manager = RunManager(store=InMemoryForensicStore()) + await manager.reload_policy("acme", _policy(2)) + await manager.reload_policy("acme", _policy(1)) # not newer + with pytest.raises(PolicyLoadError): + await manager.reload_policy("acme", "policy_version: 3\ntools: [not, a, mapping]\n") + + changes = await _policy_changes(manager, "acme") + assert [c.outcome for c in changes] == ["applied", "ignored", "rejected"] + assert [(c.previous_version, c.version) for c in changes] == [(None, 2), (2, 1), (2, None)] + assert all(len(c.policy_sha256) == 64 for c in changes) + assert manager.registry.version("acme") == 2 + await manager.aclose() + + +@asynccontextmanager +async def _client(monkeypatch: pytest.MonkeyPatch) -> AsyncIterator[httpx.AsyncClient]: + _env( + monkeypatch, + SENTINEL_API_TOKEN="", + SENTINEL_API_TOKENS=json.dumps({"acme": ACME, "globex": GLOBEX}), + SENTINEL_ADMIN_TOKEN=ADMIN, + SENTINEL_ALLOW_ANONYMOUS="0", + ) + manager = RunManager(store=InMemoryForensicStore()) + try: + async with httpx.AsyncClient( + transport=httpx.ASGITransport(app=create_app(manager)), + base_url="http://testserver", + ) as client: + yield client + finally: + await manager.aclose() + + +async def test_the_record_belongs_to_the_tenant_whose_policy_changed( + monkeypatch: pytest.MonkeyPatch, +) -> None: + async with _client(monkeypatch) as client: + admin = {"Authorization": f"Bearer {ADMIN}"} + # Newer than the bundled policy acme starts with (its credential gives + # it the deployment policy). + reply = await client.post( + "/tenants/acme/policy", json={"policy": _policy(10)}, headers=admin + ) + assert reply.status_code == 200 + assert reply.json()["reloaded"] is True + + acme = (await client.get("/runs", headers={"Authorization": f"Bearer {ACME}"})).json() + mine = [r for r in acme["runs"] if r["scenario"] == "policy-change"] + assert len(mine) == 1 + globex = (await client.get("/runs", headers={"Authorization": f"Bearer {GLOBEX}"})).json() + assert [r for r in globex["runs"] if r["scenario"] == "policy-change"] == [] + + audit = ( + await client.get( + f"/runs/{mine[0]['run_id']}/audit", + headers={"Authorization": f"Bearer {ACME}"}, + ) + ).json() + [alert] = audit["alerts"] + assert alert["alert_type"] == "PolicyChanged" + assert (alert["outcome"], alert["policy_version"], alert["severity"]) == ( + "applied", 10, "medium", + ) + + +async def test_any_tenant_name_is_saved_inside_the_directory_without_collisions() -> None: + manager = RunManager(store=InMemoryForensicStore()) + names = {"../escape": 2, "acme": 3, "ACME": 4, "Acme Corp/EU": 5} + for tenant, version in names.items(): + await manager.reload_policy(tenant, _policy(version)) + await manager.aclose() + + directory = saved_policy_dir().resolve() + saved = sorted(directory.iterdir()) + assert len(saved) == len(names) # one file each: no name reached another's + assert all(path.parent == directory for path in saved) + assert not (directory.parent / "escape.json").exists() + + restarted = RunManager(store=InMemoryForensicStore()) + assert {t: restarted.registry.version(t) for t in names} == names + await restarted.aclose() + + +async def test_a_failed_save_changes_nothing(monkeypatch: pytest.MonkeyPatch) -> None: + manager = RunManager(store=InMemoryForensicStore()) + await manager.reload_policy("acme", _policy(1)) + + def full_disk(tenant: str, text: str, version: int) -> None: + raise OSError(28, "No space left on device") + + monkeypatch.setattr(manager_module, "_save_policy", full_disk) + with pytest.raises(OSError, match="No space left"): + await manager.reload_policy("acme", _policy(2, stop=True)) + + assert manager.registry.version("acme") == 1 + assert _decision(manager, "acme") == "ALLOW" + assert [c.outcome for c in await _policy_changes(manager, "acme")] == ["applied", "failed"] + await manager.aclose() + + +async def test_the_api_reports_a_failed_save(monkeypatch: pytest.MonkeyPatch) -> None: + def full_disk(tenant: str, text: str, version: int) -> None: + raise OSError(28, "No space left on device") + + monkeypatch.setattr(manager_module, "_save_policy", full_disk) + async with _client(monkeypatch) as client: + reply = await client.post( + "/tenants/acme/policy", + json={"policy": _policy(9, stop=True)}, + headers={"Authorization": f"Bearer {ADMIN}"}, + ) + assert reply.status_code == 500 + assert "not applied" in reply.json()["detail"] + + +async def test_concurrent_reloads_save_the_policy_that_is_enforced( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Interleaved, a slow save of v2 could land after v3's and win the restart.""" + real_save = manager_module._save_policy # noqa: SLF001 + + def slow_for_v2(tenant: str, text: str, version: int) -> None: + if version == 2: + time.sleep(0.3) + real_save(tenant, text, version) + + monkeypatch.setattr(manager_module, "_save_policy", slow_for_v2) + manager = RunManager(store=InMemoryForensicStore()) + await asyncio.gather( + manager.reload_policy("acme", _policy(2)), + manager.reload_policy("acme", _policy(3, stop=True)), + ) + assert manager.registry.version("acme") == 3 + await manager.aclose() + + restarted = RunManager(store=InMemoryForensicStore()) + assert restarted.registry.version("acme") == 3 + assert _decision(restarted, "acme") == "DENY" + await restarted.aclose() + + +def test_a_saved_policy_that_cannot_be_read_stops_startup() -> None: + directory = saved_policy_dir() + directory.mkdir(parents=True) + broken = directory / ("0" * 64 + ".json") + broken.write_text("{not json", encoding="utf-8") + with pytest.raises(ValueError, match="repair or delete it") as raised: + RunManager(store=InMemoryForensicStore()) + assert os.fspath(broken) in str(raised.value) diff --git a/website/guide/deployment.md b/website/guide/deployment.md index 0ba127e..eb0b56c 100644 --- a/website/guide/deployment.md +++ b/website/guide/deployment.md @@ -73,10 +73,11 @@ work without changes. `sentinel serve --port` still overrides it. ## Run one replica -Trust scores, MCP sessions, the event feed, the run index and policies -replaced through the API live in process memory. Two replicas share none of -it, and each enforces its own rate limits. Run one instance, and enforce any -global limit at the load balancer. +Trust scores, MCP sessions, the event feed and the run index live in process +memory, and policies replaced through the API are saved in that instance's +`SENTINEL_DATA_DIR`. Two replicas share none of it, and each enforces its own +rate limits. Run one instance, and enforce any global limit at the load +balancer. ## Azure Container Apps (optional, untested) diff --git a/website/guide/forensics.md b/website/guide/forensics.md index dd6dc35..6cc7686 100644 --- a/website/guide/forensics.md +++ b/website/guide/forensics.md @@ -45,6 +45,10 @@ credential you can: Spans are immutable: re-emitting an identical span is a no-op, and a differing one is rejected. +Each policy reload through the API is a run too (scenario `policy-change`), +holding one `PolicyChanged` span: the outcome, the versions, and the SHA-256 +of the submitted policy. Its audit export is graded `medium`. + ## The forensic store The default store is SQLite at `${SENTINEL_DATA_DIR:-./var}/sentinel.db`. @@ -75,6 +79,9 @@ still be in `sentinel.db-wal`: sqlite3 sentinel.db ".backup copy.db" ``` +Back up `${SENTINEL_DATA_DIR}/policies/` with it: it holds the policies +replaced through the API, which a restart restores from there. + ## Cosmos DB (optional) Set `AZURE_COSMOS_ENDPOINT` to store spans in Azure Cosmos DB instead. It needs diff --git a/website/guide/limitations.md b/website/guide/limitations.md index b16e305..a5dcb30 100644 --- a/website/guide/limitations.md +++ b/website/guide/limitations.md @@ -64,10 +64,10 @@ server, and no reconnect. Most published MCP servers are stdio. ### State lives in one process -Trust scores, MCP sessions, the event feed, the run index and policies -replaced through `POST /tenants/{tenant}/policy` are in memory. A restart -forgets API policy reloads (put durable policy in `SENTINEL_POLICY_FILE` or -`SENTINEL_TENANT_POLICIES`), and two replicas share none of it. +Trust scores, MCP sessions, the event feed and the run index are in memory, +and two replicas share none of it. A policy replaced through +`POST /tenants/{tenant}/policy` survives a restart (it is saved under +`SENTINEL_DATA_DIR`), but only on the replica that received it. ### Cosmos persistence is partial diff --git a/website/guide/policy.md b/website/guide/policy.md index 411bfb0..112800f 100644 --- a/website/guide/policy.md +++ b/website/guide/policy.md @@ -168,6 +168,33 @@ for the guarantees. Each tenant can have its own policy through `SENTINEL_TENANT_POLICIES`, a JSON map of tenant to policy file. A tenant without an entry uses the deployment -policy. The operator can also replace a tenant's policy at runtime with -`POST /tenants/{tenant}/policy` and the admin token, but that replacement -lives in memory and is forgotten on restart. Keep durable policy in files. +policy. + +## Replacing a policy at runtime + +The operator can replace a tenant's policy without a restart: + +```bash +curl -X POST http://127.0.0.1:8765/tenants/acme/policy \ + -H "Authorization: Bearer $SENTINEL_ADMIN_TOKEN" \ + -H "content-type: application/json" \ + -d "$(jq -Rs '{policy: .}' policy.yaml)" +``` + +- **Only a newer `policy_version` replaces the active one.** A same or older + version is ignored, so a stale push cannot downgrade a tenant. A policy that + does not load is refused with `400`, and the active one stays. +- **It survives a restart.** An applied policy is saved under + `${SENTINEL_DATA_DIR}/policies/` before it takes effect. If it cannot be + saved, nothing changes and the request fails with `500`. At startup, a saved + policy replaces the configured one only if its version is higher, so to go + back to your file, give the file a higher `policy_version`. +- **Every reload is recorded.** Applied, ignored, rejected or failed, each + becomes a `PolicyChanged` record in that tenant's history (listed under + `GET /runs`), with the submitted version and the SHA-256 of the submitted + text, and a `POLICY` line in the security log. A request without the admin + token is refused with `403` before it gets that far. + +A saved policy that cannot be read, or no longer loads, stops `sentinel serve` +and `sentinel check` with an error naming the file. Repair the file or delete +it.