feat: add Grafana dashboard template and Prometheus alerting rules for self-hosted workers

README.md

@@ -367,6 +367,16 @@ shows up as a distinct series.
 - `oz_worker_info{version,backend,worker_id}` (gauge, value `1`): build and
   runtime metadata, useful for joining other series by labels.
 
+### Grafana dashboard
+
+A ready-to-import Grafana dashboard and Prometheus alerting rules are available in the [`grafana/`](grafana/) directory:
+
+- [`grafana/oz-agent-worker-overview.json`](grafana/oz-agent-worker-overview.json) — importable Grafana 10+ dashboard covering all `oz_worker_*` metrics
+- [`grafana/alerts.yaml`](grafana/alerts.yaml) — Prometheus alerting rules for connectivity, saturation, and task failures
+- [`grafana/README.md`](grafana/README.md) — import instructions for Grafana UI, API, and Prometheus Operator
+
+Quick import via Grafana UI: **Dashboards → Import → Upload JSON file** → select `grafana/oz-agent-worker-overview.json` → choose your Prometheus datasource.
+
 ### Sample dashboards / alerts
 
 Direct mappings for the questions enterprise operators most commonly ask:

grafana/README.md

@@ -0,0 +1,127 @@
+# Oz Agent Worker — Grafana Resources
+
+This directory contains a ready-to-import Grafana dashboard and Prometheus alerting rules for self-hosted Oz worker deployments.
+
+## Files
+
+| File | Purpose |
+|------|---------|
+| `oz-agent-worker-overview.json` | Importable Grafana dashboard (Grafana 10+) |
+| `alerts.yaml` | Prometheus alerting rules (vanilla Prometheus or Prometheus Operator) |
+
+## Prerequisites
+
+Before importing, enable Prometheus metrics on the worker:
+
+```bash
+export OTEL_METRICS_EXPORTER=prometheus
+export OTEL_EXPORTER_PROMETHEUS_HOST=0.0.0.0
+export OTEL_EXPORTER_PROMETHEUS_PORT=9464
+oz-agent-worker --api-key "$WARP_API_KEY" --worker-id my-worker
+```
+
+Or in the Helm chart:
+
+```bash
+helm install oz-agent-worker ./charts/oz-agent-worker \
+  --namespace warp-oz \
+  --set worker.workerId=my-worker \
+  --set image.tag=VERSION \
+  --set metrics.enabled=true
+```
+
+Verify the endpoint is serving metrics before importing the dashboard:
+
+```bash
+curl -s localhost:9464/metrics | grep oz_worker_
+```
+
+## Importing the dashboard
+
+### Option 1 — Grafana UI
+
+1. In Grafana, go to **Dashboards** → **Import**.
+2. Click **Upload JSON file** and select `oz-agent-worker-overview.json`.
+3. When prompted, select your Prometheus datasource.
+4. Click **Import**.
+
+### Option 2 — Grafana API
+
+```bash
+curl -X POST \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer $GRAFANA_API_KEY" \
+  --data-binary @oz-agent-worker-overview.json \
+  http://your-grafana/api/dashboards/import
+```
+
+### Option 3 — Direct URL
+
+If the worker binary is available on GitHub, you can import from the raw URL directly in the Grafana UI (**Import** → **Import via grafana.com or URL** → paste the raw GitHub URL):
+
+```
+https://raw.githubusercontent.com/warpdotdev/oz-agent-worker/main/grafana/oz-agent-worker-overview.json
+```
+
+## Dashboard layout
+
+The dashboard has five sections:
+
+| Row | Panels |
+|-----|--------|
+| **Worker Health** | Connected workers, active workers, tasks active, fleet saturation |
+| **Task Throughput** | Task completion rate by result, 5m success rate, 5m rejection rate |
+| **Task Duration** | p50 / p95 / p99 latency |
+| **Failure Analysis** | Failure rate by reason, failure rate by phase |
+| **Connection Health** | WebSocket reconnect rate, worker fleet info table |
+
+### Instance filter
+
+The **Instance** variable at the top of the dashboard lets you filter to a specific worker pod or scrape target. Set it to **All** (the default) to see fleet-wide aggregations.
+
+## Using the alert rules
+
+### Vanilla Prometheus
+
+1. Copy `alerts.yaml` to your Prometheus rules directory (e.g., `/etc/prometheus/rules/`).
+2. Reference it in `prometheus.yml`:
+
+   ```yaml
+   rule_files:
+     - /etc/prometheus/rules/alerts.yaml
+   ```
+
+3. Reload Prometheus: `curl -X POST http://localhost:9090/-/reload`
+
+### Prometheus Operator (Kubernetes)
+
+Wrap the `groups` block in a `PrometheusRule` custom resource:
+
+```yaml
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: oz-agent-worker
+  namespace: warp-oz
+  labels:
+    # Match your Prometheus Operator's ruleSelector labels
+    prometheus: kube-prometheus
+    role: alert-rules
+spec:
+  groups:
+    # Paste the contents of alerts.yaml `groups:` block here
+```
+
+### Tuning thresholds
+
+All alert thresholds are conservative starting points. Tune them to match your fleet size, task volume, and SLOs:
+
+- `OzWorkerFleetSaturated`: raise the 0.9 threshold for fleets that run near capacity by design.
+- `OzWorkerHighFailureRate`: lower the 0.1 threshold (10%) to catch smaller failure spikes.
+- `OzWorkerReconnectStorm`: adjust the 0.1/s threshold based on your expected reconnect baseline.
+
+## Related documentation
+
+- [Self-hosted worker monitoring](https://docs.warp.dev/platform/self-hosting/monitoring/) — full metrics reference and PromQL examples
+- [Self-hosted worker overview](https://docs.warp.dev/platform/self-hosting/) — architecture and deployment patterns
+- [Helm chart configuration](https://docs.warp.dev/platform/self-hosting/managed-kubernetes/) — enabling metrics via Helm values

grafana/alerts.yaml

@@ -0,0 +1,153 @@
+# Prometheus alerting rules for self-hosted Oz agent workers.
+#
+# Usage:
+#   - Vanilla Prometheus: place this file in your Prometheus rules directory
+#     and reference it from prometheus.yml under `rule_files:`.
+#   - Prometheus Operator: wrap the `groups` block in a PrometheusRule CRD
+#     (apiVersion: monitoring.coreos.com/v1, kind: PrometheusRule).
+#
+# All thresholds are conservative starting points. Tune them for your fleet
+# size, task volume, and SLOs before enabling in production.
+
+groups:
+  - name: oz_agent_worker
+    interval: 60s
+    rules:
+
+      # ── Connectivity ──────────────────────────────────────────────────────
+
+      - alert: OzWorkerDisconnected
+        expr: |
+          oz_worker_connected == 0
+        for: 2m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Oz worker {{ $labels.instance }} has lost its connection to Oz"
+          description: >
+            Worker {{ $labels.instance }} has reported oz_worker_connected=0 for
+            more than 2 minutes. The worker is no longer accepting tasks.
+            Check network egress to oz.warp.dev and review worker logs.
+
+      - alert: OzWorkerReconnectStorm
+        expr: |
+          sum by (instance) (rate(oz_worker_websocket_reconnects_total[5m])) > 0.1
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Oz worker {{ $labels.instance }} is reconnecting frequently"
+          description: >
+            Worker {{ $labels.instance }} is reconnecting to warp-server at a rate
+            of {{ $value | humanize }}/s (threshold: 0.1/s). This may indicate
+            network instability, TLS issues, or server-side disruptions.
+
+      # ── Capacity ──────────────────────────────────────────────────────────
+
+      - alert: OzWorkerFleetSaturated
+        expr: |
+          (
+            sum(oz_worker_tasks_active) /
+            sum(oz_worker_tasks_max_concurrent != 0)
+          ) > 0.9
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Oz worker fleet is running at high capacity"
+          description: >
+            The fleet is using {{ $value | humanizePercentage }} of configured
+            concurrency capacity (threshold: 90%). Tasks may start experiencing
+            rejections. Consider increasing max_concurrent_tasks or deploying
+            additional worker instances.
+          runbook: |
+            1. Check oz_worker_tasks_rejected_total for recent rejections.
+            2. Review oz_worker_task_duration_seconds to see if tasks are running longer than expected.
+            3. Increase max_concurrent_tasks in worker config, or deploy more worker pods.
+
+      - alert: OzWorkerTasksRejected
+        expr: |
+          sum(rate(oz_worker_tasks_rejected_total[5m])) > 0
+        for: 1m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Oz workers are rejecting tasks"
+          description: >
+            Workers are rejecting tasks at {{ $value | humanize }}/s. Rejections
+            occur when a worker is at its concurrency limit (max_concurrent_tasks).
+            Increase capacity or the concurrency limit to stop dropping tasks.
+
+      # ── Task failures ─────────────────────────────────────────────────────
+
+      - alert: OzWorkerHighFailureRate
+        expr: |
+          (
+            sum(rate(oz_worker_tasks_completed_total{result="failed"}[5m])) /
+            sum(rate(oz_worker_tasks_completed_total[5m]))
+          ) > 0.1
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Oz worker task failure rate is elevated"
+          description: >
+            {{ $value | humanizePercentage }} of completed tasks have failed over
+            the last 5 minutes (threshold: 10%). Check
+            oz_worker_task_failures_total{phase,reason} for the root cause.
+
+      - alert: OzWorkerImagePullFailures
+        expr: |
+          sum(rate(oz_worker_task_failures_total{reason="image_pull"}[5m])) > 0
+        for: 2m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Oz worker is failing to pull task images"
+          description: >
+            Workers are failing to pull container images at
+            {{ $value | humanize }}/s. Check image registry access, pull secrets,
+            and image availability. For Kubernetes, confirm imagePullSecrets are
+            configured and the registry is reachable from the cluster.
+
+      - alert: OzWorkerContainerOOM
+        expr: |
+          sum(rate(oz_worker_task_failures_total{reason="container_oom"}[5m])) > 0
+        for: 2m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Oz worker task containers are being OOM-killed"
+          description: >
+            Task containers are being killed due to memory limits at
+            {{ $value | humanize }}/s. Review the memory limits set in
+            pod_template or the instance shape for the runs that are failing,
+            and increase them if tasks require more memory.
+
+      - alert: OzWorkerUnschedulablePods
+        expr: |
+          sum(rate(oz_worker_task_failures_total{reason="unschedulable"}[5m])) > 0
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Oz worker task pods are failing to schedule"
+          description: >
+            Kubernetes task pods are becoming unschedulable at
+            {{ $value | humanize }}/s. Check node capacity, taints, tolerations,
+            resource requests in pod_template, and cluster autoscaler status.
+
+      # ── No workers connected ──────────────────────────────────────────────
+
+      - alert: OzWorkerFleetEmpty
+        expr: |
+          sum(oz_worker_connected) == 0
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: "No Oz workers are connected"
+          description: >
+            No workers have an active connection to Oz. Cloud agent tasks cannot
+            be dispatched to self-hosted infrastructure. Check worker pod
+            readiness, API key validity, and network egress to oz.warp.dev.