Skip to content

feat(observability): supervisor decision and latency metrics #5364

Open

Description

@krishicks

User Story

I operate OpenShell gateways and sandboxes. I want alerts on supervisor latency, and I want to see which latency bucket is growing and whether request or denial rates rose with it. Today I'd have to reconstruct that from traces or OCSF records.

Problem Statement

The gateway serves Prometheus metrics, but supervisors export none. There is no rate of egress decisions by outcome, and no latency distribution for policy evaluation, request inspection, or upstream connect.

Impact / Why This Matters

Operators cannot alert on a denial storm or on supervisor-added latency. The workaround is keeping every trace or counting OCSF records in the log pipeline, which pushes rate questions onto traces and SIEMs and still gives no latency distribution.

Proposed Design

  • Counters for egress decisions, labeled by outcome, protocol, and policy.
  • Histograms for policy evaluation, supervisor-added time on inspected requests, and upstream connect.
  • Exemplars linking histogram buckets to sampled trace IDs, so an alert leads to a representative trace.
  • Bounded labels: no destination host labels.
  • Open question: how metrics leave a supervisor. Options include OTLP metrics push to the collector, aggregation through the gateway's Prometheus endpoint, or a per-supervisor scrape target.

Acceptance Criteria

  • Operators can query decision rates by outcome and policy, and latency histograms for the stages above.
  • Histogram buckets carry exemplars when a sampled span is active.
  • Label cardinality stays bounded under any workload behavior.
  • The docs list each metric alongside the gateway metrics page.

Alternatives Considered

  • Derive rates from OCSF records in the log pipeline. This works, but it pushes counting onto every SIEM and gives no latency data.
  • Derive rates and latency from unsampled traces. Traces are for latency and error investigation; exporting all of them to count requests is costly and depends on sampling being off.

Agent Investigation

Activity

  1. aegeiger commented on Nov 12, 2026

    @aegeiger

    @krishicks I would love to take this issue, as working on metrics in OpenShell is exactly what I want to contribute to this project. Could you confirm acceptance and assign to me @aegeiger ?

    I'll start by suggesting here metrics and their bounded labels.

  2. added
    area:runtimeSandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle.
    topic:observabilityLogs, metrics, traces, telemetry, dashboards, event correlation, and operational data export.
    on Nov 15, 2026
  3. aegeiger commented on Nov 16, 2026

    @aegeiger

    Proposed metrics:

    Metric Type Labels Measures
    openshell_supervisor_egress_decisions_total Counter scope, outcome, inspection-type Final egress admission decisions.
    openshell_supervisor_policy_evaluation_duration_seconds Histogram scope, result Policy evaluation time, including input construction and waiting for the policy-engine lock.
    openshell_supervisor_request_inspection_duration_seconds Histogram scope, inspection-type, result Supervisor processing time for an inspected request or message, including policy evaluation, middleware, and credential preparation.
    openshell_supervisor_upstream_connect_duration_seconds Histogram route, result Establishing a new upstream connection, including address fallback and proxy negotiation where applicable.
    openshell_supervisor_egress_preparation_duration_seconds Histogram scope, inspection-type, result Elapsed time from the first egress signal/request byte until connection availability and required pre-forward checks are complete—or the operation terminates unsuccessfully.
    Label Applies to Values and meaning
    scope Decisions, evaluation, inspection, preparation connection, request, message. Inspection uses only request and message.
    outcome Decisions allow, deny, audit_allow, fail_open. Records the final admission outcome; a subsequent denial during admission overrides an earlier allow.
    inspection-type Decisions, inspection, preparation tcp, http, rest, graphql, json_rpc, mcp, websocket, unknown. Uses the selected inspector where available; never inferred solely from the port.
    result Policy evaluation allow, deny, error. The evaluator’s verdict before audit mode changes enforcement.
    result Inspection completed, error, timeout, cancelled. A normal policy denial is a completed inspection.
    result Upstream connect success, error, timeout, cancelled.
    result Preparation ready_to_forward, deny, error, timeout, cancelled. Readiness includes requests admitted through audit or fail-open behavior.
    route Upstream connect direct, proxy.

    I considered adding a policy label to the decisions counter, because I think it'll be useful for users, but I don't see a way to make this label bounded and I'm not sure we want unbounded labels.

    As for the question of how to get metrics out of the supervisor, I think we should follow the gateway precedent, of exposing a /metrics endpoint and relying on other mechanism such as Prometheus to scrape it. Prometheus is the de-facto standard and is scrape-based by design. If a user wants to use other solutions which are push-based, they're probably already using otel-collector or a similar tool.

    Would love to have a maintainer validate my thinking here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:runtimeSandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle.topic:observabilityLogs, metrics, traces, telemetry, dashboards, event correlation, and operational data export.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions