Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/Implementation_Status.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ Living checklist (detail also in [`.claude/PRPs/tasks/task.md`](../.claude/PRPs/
5. Postgres as supported production backend + multi-replica validation (#1194)
6. Helm/Docker packaging: broker (+ UI if split); ~~cage + llm-gateway~~ **Done**
7. ~~OIDC for console/admin~~ **Done (beta)** — self-service login/link flow (`src/src/oidc.rs`, `routes/oidc.rs`); **remaining:** SAML, per-SSO-user attribution/revocation, multi-IdP
8. Operator runbook gates: `JWT_REQUIRED`, admin key, `REPLAY_STORE=db`, TLS, backups
8. ~~Operator runbook gates~~ **Done (beta)** — `JWT_REQUIRED`+bind was already fail-closed (`assert_bind_security`); startup now also *warns* (not fail-closed — these have legitimate reasons to be absent, e.g. TLS terminated at a reverse proxy) on a non-loopback bind missing admin key / `REPLAY_STORE=db` / TLS (`warn_on_incomplete_production_hardening`, `src/src/main.rs`); **remaining:** backups has no runtime-observable "is it actually scheduled" signal, stays a doc-only runbook item (`docs/deployment-guide.md`)

### Wave C — P2 product surface / honesty

Expand Down
10 changes: 7 additions & 3 deletions docs/deployment-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -245,12 +245,16 @@ These only run on whichever gateway instance wins the SQLite advisory-lock leade

## 5. Production checklist

- [ ] **TLS**: set `AEGIS_TLS_CERT`/`AEGIS_TLS_KEY`, or terminate TLS at a reverse proxy / Kubernetes ingress in front of the gateway. Plain HTTP is fine for `127.0.0.1`-bound local dev only.
- [ ] **Secrets**: set `AEGIS_JWT_REQUIRED=true` with a real `AEGIS_JWT_SECRET` (not `default_secret`) before exposing the gateway beyond localhost. Set `AEGIS_POLICY_SIGNING_KEY` if you intend to use signed policy bundles. Never commit secret values — use your platform's secret store (Kubernetes `Secret`, systemd `EnvironmentFile`, etc.).
> On a non-loopback `AEGIS_BIND_ADDR`, the gateway checks three of these itself at startup and logs a `WARN` naming whichever are missing (`warn_on_incomplete_production_hardening`, `src/src/main.rs`) — it's a reminder, not a fail-closed gate, since each has legitimate reasons to be absent in a given deployment. `AEGIS_JWT_REQUIRED` is the one exception: an unauthenticated public bind fails closed at startup (`assert_bind_security`), it doesn't just warn.

- [ ] **TLS**: set `AEGIS_TLS_CERT`/`AEGIS_TLS_KEY`, or terminate TLS at a reverse proxy / Kubernetes ingress in front of the gateway. Plain HTTP is fine for `127.0.0.1`-bound local dev only. *(startup-warned)*
- [ ] **Secrets**: set `AEGIS_JWT_REQUIRED=true` with a real `AEGIS_JWT_SECRET` (not `default_secret`) before exposing the gateway beyond localhost. Set `AEGIS_POLICY_SIGNING_KEY` if you intend to use signed policy bundles. Never commit secret values — use your platform's secret store (Kubernetes `Secret`, systemd `EnvironmentFile`, etc.). *(startup fail-closed)*
Comment on lines +248 to +251

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Document the explicit demo-mode exception.

assert_bind_security permits a non-loopback unauthenticated bind when AEGIS_DEMO_MODE=true and emits a warning. The checklist currently says such binds fail closed, which can mislead operators. State that demo mode is an intentional insecure escape hatch and must not be used in production.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/deployment-guide.md` around lines 248 - 251, Update the deployment
checklist’s Secrets guidance around assert_bind_security to document that
AEGIS_DEMO_MODE=true intentionally permits a non-loopback unauthenticated bind
with a warning. Clearly label demo mode as an insecure escape hatch that must
not be used in production, while preserving the normal startup fail-closed
guidance.

- [ ] **Admin key**: set `AEGIS_ADMIN_API_KEY` before going beyond localhost — without it, admin/debug/metrics endpoints stay disabled by default (safe, but you lose access to them too). *(startup-warned)*
- [ ] **Replay store**: set `AEGIS_REPLAY_STORE=db` for any multi-replica deployment — the default per-process in-memory replay-nonce cache doesn't dedupe across instances. *(startup-warned)*
- [ ] **Database encryption at rest**: if required by your compliance posture, build with `--features sqlcipher` and set `AEGIS_DB_ENCRYPTION_KEY` — the gateway fails closed at startup if the key is set without the matching build feature.
- [ ] **Monitoring**: point `AEGIS_OTLP_ENDPOINT` at your collector for traces + metrics, or scrape `/metrics` (Prometheus text) directly — bound on the same listener, not separately exposed. Wire up the Helm chart's `ServiceMonitor` if you run prometheus-operator.
- [ ] **Health probes**: `/livez`, `/readyz`, `/startupz` are already wired into the Helm chart's Deployment; if deploying elsewhere, point your orchestrator's liveness/readiness checks at them directly rather than `/health` (which does a DB round-trip and is heavier).
- [ ] **Backups**: schedule `POST /v1/admin/backup`, which writes a consistent point-in-time copy via SQLite's `VACUUM INTO` (safe against a live database, no downtime) into `AEGIS_BACKUP_DIR`. See [Runbook: Backup and Restore](runbooks/backup-and-restore.md) for the restore procedure — there is no restore API, only a documented manual procedure.
- [ ] **Backups**: schedule `POST /v1/admin/backup`, which writes a consistent point-in-time copy via SQLite's `VACUUM INTO` (safe against a live database, no downtime) into `AEGIS_BACKUP_DIR`. See [Runbook: Backup and Restore](runbooks/backup-and-restore.md) for the restore procedure — there is no restore API, only a documented manual procedure. Not startup-checked: `AEGIS_BACKUP_DIR` always resolves to a value (defaults to `backups`), so its presence can't distinguish "backups are actually scheduled" from "nobody configured this."
- [ ] **CORS**: leave `AEGIS_CORS_ORIGINS` unset (no CORS headers) unless a browser-based dashboard or client genuinely needs cross-origin access; set it to an explicit allowlist, never a wildcard, if you do.
- [ ] **Rate limits & quotas**: the defaults (`AEGIS_RATE_LIMIT_CAPACITY=100`, refill `10`/s) are tuned for development. Size them per-agent based on your actual expected call volume before going live.

Expand Down
108 changes: 108 additions & 0 deletions src/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1221,6 +1221,77 @@ fn assert_bind_security(
))
}

/// Roadmap Wave B item 8 ("operator runbook gates"): on a non-loopback
/// bind, warn about any of `docs/deployment-guide.md`'s production
/// checklist items that don't have a runtime-observable signal configured.
///
/// Deliberately a *warning*, not a fail-closed check like
/// [`assert_bind_security`]: these are hardening recommendations with
/// legitimate reasons to be absent in a given deployment (e.g. TLS
/// terminated at a reverse proxy in front of this gateway, or a
/// single-replica deployment that doesn't need the shared replay store) —
/// unlike "publicly reachable with no authentication at all," none of these
/// three is unconditionally unsafe on its own.
///
/// `AEGIS_BACKUP_DIR` (the fifth runbook item) is deliberately not checked
/// here: it always resolves to a value (defaults to `"backups"`, see
/// `routes/tenant.rs`), so its mere presence can't distinguish "operator
/// configured backups" from "operator never thought about it" — checking
/// that for real would need new state (e.g. a last-backup timestamp), which
/// is out of scope for this warning. Backups stay a doc-only runbook item.
///
/// Pure function for unit testing; env reads happen in the caller.
/// Pure decision function behind [`warn_on_incomplete_production_hardening`]:
/// which runbook gates (if any) are missing for this configuration. Empty on
/// a loopback bind or when everything is configured.
fn missing_production_hardening_gates(
bind_addr: &str,
admin_key_configured: bool,
replay_store_db: bool,
tls_enabled: bool,
) -> Vec<&'static str> {
if host_is_loopback(bind_addr) {
return Vec::new();
}
let mut missing = Vec::new();
if !admin_key_configured {
missing
.push("AEGIS_ADMIN_API_KEY (admin/debug/metrics endpoints stay disabled without it)");
}
if !replay_store_db {
missing.push(
"AEGIS_REPLAY_STORE=db (replay-nonce dedup is per-process only, unsafe across multiple replicas)",
);
}
if !tls_enabled {
missing.push(
"AEGIS_TLS_CERT/AEGIS_TLS_KEY (or terminate TLS at a reverse proxy in front of this gateway)",
);
}
missing
}
Comment on lines +1247 to +1272

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Use the required Result<T, AegisError> return contract.

Both new production helpers return non-Result types, violating the repository rule. Wrap successful values in Ok(...), return Result<(), AegisError> from the wrapper, and propagate the result at Lines 2422-2427.

Also applies to: 1274-1292

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/src/main.rs` around lines 1247 - 1272, Update
missing_production_hardening_gates and the adjacent production-hardening helper
to follow the required Result<T, AegisError> contract, wrapping successful
values in Ok(...). Change the wrapper to return Result<(), AegisError> and
propagate its result at the call site around the production startup flow near
lines 2422-2427.

Source: Coding guidelines


fn warn_on_incomplete_production_hardening(
bind_addr: &str,
admin_key_configured: bool,
replay_store_db: bool,
tls_enabled: bool,
) {
let missing = missing_production_hardening_gates(
bind_addr,
admin_key_configured,
replay_store_db,
tls_enabled,
);
if !missing.is_empty() {
tracing::warn!(
"AEGIS_BIND_ADDR='{bind_addr}' is network-reachable but the following operator \
runbook gates (docs/deployment-guide.md) are not configured: {}",
missing.join("; ")
);
}
}

/// Whether a `host:port` bind address is a loopback (not network-reachable)
/// address — `127.0.0.0/8`, `::1`, or `localhost`. Shared by the public-bind
/// safety check and the admin-route guard.
Expand Down Expand Up @@ -2348,6 +2419,13 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
(None, None) => None,
};

warn_on_incomplete_production_hardening(
&bind_addr,
std::env::var("AEGIS_ADMIN_API_KEY").is_ok_and(|v| !v.trim().is_empty()),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

There is a discrepancy between how the startup warning and the runtime admin_decision check whether the admin API key is configured. The startup warning uses !v.trim().is_empty(), which treats a whitespace-only key as unconfigured. However, admin_decision uses !k.is_empty(), meaning a whitespace-only key would actually be accepted at runtime if provided by the client. To ensure consistency between the startup warning and runtime enforcement, align the startup check with the runtime check by using !v.is_empty().

        std::env::var("AEGIS_ADMIN_API_KEY").is_ok_and(|v| !v.is_empty()),

replay_store_db,
use_tls.is_some(),
);
Comment on lines +2422 to +2427

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Avoid duplicate warnings for partial TLS configuration.

When only one TLS environment variable is set, the earlier branch already logs a warning, then this call logs the aggregate missing-TLS warning as well. Consolidate these paths so one missing gate produces one startup warning.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/src/main.rs` around lines 2422 - 2427, Update the startup
hardening-warning flow around warn_on_incomplete_production_hardening so partial
TLS configuration is handled by only one warning path. Track or propagate
whether the earlier TLS branch already emitted its warning, and prevent the
aggregate missing-TLS warning from being logged for that same missing gate while
preserving warnings for other incomplete production-hardening requirements.


// Startup is complete: DB pool + migrations, policy engine, and background
// jobs are all initialized. /startupz now reports ready (#1208).
state
Expand Down Expand Up @@ -2712,6 +2790,36 @@ mod tests {
assert!(assert_bind_security("0.0.0.0:8080", false, true).is_ok());
}

#[test]
fn production_hardening_gates_are_all_inert_on_loopback() {
assert!(
missing_production_hardening_gates("127.0.0.1:8080", false, false, false).is_empty()
);
}

#[test]
fn production_hardening_gates_are_empty_when_everything_is_configured() {
assert!(missing_production_hardening_gates("0.0.0.0:8080", true, true, true).is_empty());
}

#[test]
fn production_hardening_gates_flags_each_missing_item_on_a_public_bind() {
let missing = missing_production_hardening_gates("0.0.0.0:8080", false, false, false);
assert_eq!(missing.len(), 3);
assert!(missing.iter().any(|m| m.contains("AEGIS_ADMIN_API_KEY")));
assert!(missing.iter().any(|m| m.contains("AEGIS_REPLAY_STORE")));
assert!(missing.iter().any(|m| m.contains("AEGIS_TLS_CERT")));
}

#[test]
fn production_hardening_gates_flags_only_the_specific_missing_items() {
// Admin key + replay store configured, TLS not -- only TLS should surface.
let missing = missing_production_hardening_gates("0.0.0.0:8080", true, true, false);
assert_eq!(missing, vec![
"AEGIS_TLS_CERT/AEGIS_TLS_KEY (or terminate TLS at a reverse proxy in front of this gateway)"
]);
}

#[test]
fn admin_decision_allows_anything_on_loopback_bind() {
// Local dev/ops: no admin key needed when not network-reachable.
Expand Down
Loading