|
| 1 | +# persys-meter |
| 2 | + |
| 3 | +Consumes per-workload resource usage events published by `persys-scheduler` |
| 4 | +onto a Redis Stream, stores them durably in ClickHouse, and exposes them |
| 5 | +three ways: live Prometheus metrics, a JSON query API, and raw historical |
| 6 | +rows in ClickHouse. It's the piece that makes historical/aggregated |
| 7 | +per-workload usage queryable at all - `compute-agent` and `persys-scheduler` |
| 8 | +only ever expose the *latest* sample. |
| 9 | + |
| 10 | +## Where this fits |
| 11 | + |
| 12 | +``` |
| 13 | +compute-agent --(heartbeat, real per-workload usage)--> persys-scheduler |
| 14 | + | |
| 15 | + XADD persys:usage:stream |
| 16 | + | |
| 17 | + v |
| 18 | + persys-meter |
| 19 | + (this service) |
| 20 | + / | \ |
| 21 | + Prometheus /metrics | JSON API |
| 22 | + (live, per-workload) | (:9092) |
| 23 | + v |
| 24 | + ClickHouse |
| 25 | + (durable history) |
| 26 | +``` |
| 27 | + |
| 28 | +`persys-meter` is a Redis Streams **consumer group**: run as many replicas |
| 29 | +as you want with the same `METER_CONSUMER_GROUP`, and Redis hands each one a |
| 30 | +different slice of the stream automatically. Nothing here talks directly to |
| 31 | +`compute-agent` or `persys-scheduler` beyond reading the stream they already |
| 32 | +publish to. |
| 33 | + |
| 34 | +## What this service is - and isn't |
| 35 | + |
| 36 | +**Is:** the accurate, queryable record of what every workload has actually |
| 37 | +consumed, live and historical. This is the necessary input to billing and |
| 38 | +quota enforcement. |
| 39 | + |
| 40 | +**Isn't:** a billing engine or a quota enforcer. It doesn't define price |
| 41 | +tiers, quota limits, or what happens when a limit is exceeded (deny new |
| 42 | +workloads? throttle? alert?) - those are product decisions that belong in |
| 43 | +whatever service makes admission/billing decisions (`persys-scheduler`, or a |
| 44 | +dedicated billing service), consuming this service's API. Baking a specific |
| 45 | +quota policy in here would mean guessing at a business model this service |
| 46 | +has no way to know. |
| 47 | + |
| 48 | +## Running it |
| 49 | + |
| 50 | +Needs a reachable Redis (the same one `persys-scheduler` publishes to) and a |
| 51 | +ClickHouse server/cluster. |
| 52 | + |
| 53 | +```bash |
| 54 | +export REDIS_ADDR=localhost:6379 |
| 55 | +export CLICKHOUSE_ADDR=localhost:9000 |
| 56 | +make run |
| 57 | +``` |
| 58 | + |
| 59 | +The `usage_events` table is created automatically on startup if it doesn't |
| 60 | +exist (see `internal/store/clickhouse.go`). |
| 61 | + |
| 62 | +### Docker |
| 63 | + |
| 64 | +```bash |
| 65 | +make docker # builds persys-dev/persys-meter:latest |
| 66 | +docker run --rm \ |
| 67 | + -e REDIS_ADDR=redis:6379 \ |
| 68 | + -e CLICKHOUSE_ADDR=clickhouse:9000 \ |
| 69 | + -p 9091:9091 -p 9092:9092 \ |
| 70 | + persys-dev/persys-meter:latest |
| 71 | +``` |
| 72 | + |
| 73 | +## Configuration |
| 74 | + |
| 75 | +Everything is an env var; every one has a default suitable for local dev |
| 76 | +against `localhost` services. |
| 77 | + |
| 78 | +| Variable | Default | Notes | |
| 79 | +|---|---|---| |
| 80 | +| `REDIS_ADDR` | `127.0.0.1:6379` | Must be the same Redis `persys-scheduler` publishes to | |
| 81 | +| `REDIS_PASSWORD` | _(empty)_ | | |
| 82 | +| `REDIS_DB` | `0` | | |
| 83 | +| `METER_REDIS_STREAM` | `persys:usage:stream` | **Must match** persys-scheduler's `METER_REDIS_STREAM` | |
| 84 | +| `METER_CONSUMER_GROUP` | `persys-meter` | Shared across all replicas | |
| 85 | +| `METER_CONSUMER_NAME` | `<hostname>-<pid>` | Should be unique per running instance | |
| 86 | +| `METER_READ_COUNT` | `200` | XREADGROUP COUNT (messages per read) | |
| 87 | +| `METER_READ_BLOCK` | `5s` | XREADGROUP BLOCK duration | |
| 88 | +| `METER_CLAIM_MIN_IDLE` | `30s` | XAUTOCLAIM: how long a message can sit pending before being reclaimed | |
| 89 | +| `METER_CLAIM_INTERVAL` | `15s` | How often the reclaim sweep runs | |
| 90 | +| `METER_WORKERS` | `4` | Concurrent XREADGROUP goroutines | |
| 91 | +| `METER_DEDUPE_TTL` | `24h` | How long an event_id is remembered to catch redeliveries | |
| 92 | +| `METER_BATCH_SIZE` | `500` | Max records per ClickHouse insert | |
| 93 | +| `METER_BATCH_FLUSH_INTERVAL` | `2s` | Max time a partial batch waits before flushing | |
| 94 | +| `CLICKHOUSE_ADDR` | `127.0.0.1:9000` | Native protocol port | |
| 95 | +| `CLICKHOUSE_DATABASE` | `persys` | | |
| 96 | +| `CLICKHOUSE_USERNAME` | `default` | | |
| 97 | +| `CLICKHOUSE_PASSWORD` | _(empty)_ | | |
| 98 | +| `CLICKHOUSE_TLS` | `false` | | |
| 99 | +| `METER_RETENTION_DAYS` | `90` | TTL on `usage_events` - **tune this to your actual billing/audit retention needs**, 90 is a placeholder | |
| 100 | +| `METER_HEALTH_ADDR` | `:9091` | Serves `/healthz`, `/readyz`, `/metrics` | |
| 101 | +| `METER_API_ADDR` | `:9092` | Serves the JSON query API (see below) | |
| 102 | +| `METER_API_TOKEN` | _(empty)_ | If set, required as `Authorization: Bearer <token>` on every API request. Empty = no auth | |
| 103 | +| `METER_CACHE_MAX_AGE` | `5m` | How stale a workload's cached sample can be and still be considered "live" (affects both the API and Prometheus metrics) | |
| 104 | +| `METER_CACHE_PRUNE_INTERVAL` | `2m` | How often stale cache entries are actually freed from memory | |
| 105 | + |
| 106 | +## Metrics (`GET :9091/metrics`) |
| 107 | + |
| 108 | +Two distinct sets, both under the `persys_meter_*` namespace: |
| 109 | + |
| 110 | +**Actual workload metrics** - what you almost certainly want for dashboards |
| 111 | +and alerting, one series per currently-live workload: |
| 112 | + |
| 113 | +| Metric | Type | Meaning | |
| 114 | +|---|---|---| |
| 115 | +| `persys_meter_workload_cpu_percent{workload_id,node_id,workload_type}` | gauge | Latest CPU utilization % | |
| 116 | +| `persys_meter_workload_memory_bytes{...}` | gauge | Latest resident memory usage | |
| 117 | +| `persys_meter_workload_disk_read_bytes_total{...}` | counter | Cumulative disk bytes read (resets if the workload restarts) | |
| 118 | +| `persys_meter_workload_disk_write_bytes_total{...}` | counter | Cumulative disk bytes written | |
| 119 | +| `persys_meter_workload_net_rx_bytes_total{...}` | counter | Cumulative network bytes received | |
| 120 | +| `persys_meter_workload_net_tx_bytes_total{...}` | counter | Cumulative network bytes transmitted | |
| 121 | +| `persys_meter_workload_sample_age_seconds{...}` | gauge | Seconds since the last sample - alert on this being persistently high to catch an agent that's stopped reporting | |
| 122 | + |
| 123 | +These are emitted by a custom Prometheus collector reading the shared |
| 124 | +in-memory cache (`internal/cache`), so a workload that stops reporting |
| 125 | +simply disappears from scrapes after `METER_CACHE_MAX_AGE` rather than |
| 126 | +leaving a stale series behind forever. |
| 127 | + |
| 128 | +**Pipeline health metrics** - about persys-meter itself, not workloads: |
| 129 | + |
| 130 | +| Metric | Meaning | |
| 131 | +|---|---| |
| 132 | +| `persys_meter_events_consumed_total{stream}` | Messages read via XREADGROUP | |
| 133 | +| `persys_meter_events_deduped_total{stream}` | Skipped as a redelivery | |
| 134 | +| `persys_meter_events_parse_failed_total{stream}` | Malformed payloads (logged at Error level too) | |
| 135 | +| `persys_meter_events_written_total{stream}` | Successfully written to ClickHouse | |
| 136 | +| `persys_meter_batch_write_failed_total{stream}` | Failed batch writes (left un-acked for retry) | |
| 137 | +| `persys_meter_messages_reclaimed_total{stream}` | Recovered via XAUTOCLAIM from a stalled consumer | |
| 138 | +| `persys_meter_batch_flush_duration_seconds{stream,result}` | ClickHouse write latency | |
| 139 | +| `persys_meter_batch_size{stream}` | Records per flushed batch | |
| 140 | +| `persys_meter_consumer_lag_seconds{stream}` | Time between an event being reported and processed | |
| 141 | + |
| 142 | +## Query API (`:9092`) |
| 143 | + |
| 144 | +JSON over HTTP. Set `METER_API_TOKEN` and send `Authorization: Bearer |
| 145 | +<token>` in production - unauthenticated by default for local dev only. |
| 146 | + |
| 147 | +| Endpoint | Description | |
| 148 | +|---|---| |
| 149 | +| `GET /v1/workloads` | Every live workload's latest sample. `?workload_type=container\|vm` to filter | |
| 150 | +| `GET /v1/workloads/{id}` | Latest sample for one workload. `404` if none within `METER_CACHE_MAX_AGE` | |
| 151 | +| `GET /v1/workloads/{id}/history?from=&to=&limit=` | Raw historical samples from ClickHouse, newest first. `from`/`to` are RFC3339, default to the last 7 days | |
| 152 | +| `GET /v1/workloads/{id}/summary?from=&to=` | Aggregated usage over a window - see below | |
| 153 | +| `GET /v1/nodes/{id}/workloads` | Live workloads on a given node | |
| 154 | + |
| 155 | +### Example: usage summary |
| 156 | + |
| 157 | +``` |
| 158 | +GET /v1/workloads/wl-abc123/summary?from=2026-07-01T00:00:00Z&to=2026-07-08T00:00:00Z |
| 159 | +``` |
| 160 | + |
| 161 | +```json |
| 162 | +{ |
| 163 | + "workload_id": "wl-abc123", |
| 164 | + "from": "2026-07-01T00:00:00Z", |
| 165 | + "to": "2026-07-08T00:00:00Z", |
| 166 | + "sample_count": 40320, |
| 167 | + "avg_cpu_percent": 12.4, |
| 168 | + "max_cpu_percent": 87.1, |
| 169 | + "avg_memory_bytes": 268435456, |
| 170 | + "max_memory_bytes": 402653184, |
| 171 | + "disk_read_bytes_delta": 1073741824, |
| 172 | + "disk_write_bytes_delta": 536870912, |
| 173 | + "net_rx_bytes_delta": 209715200, |
| 174 | + "net_tx_bytes_delta": 104857600, |
| 175 | + "first_sample": "2026-07-01T00:00:03Z", |
| 176 | + "last_sample": "2026-07-07T23:59:58Z", |
| 177 | + "estimated_cpu_core_seconds": 75277.44 |
| 178 | +} |
| 179 | +``` |
| 180 | + |
| 181 | +**Read the caveats before wiring this into anything that charges money:** |
| 182 | + |
| 183 | +- `*_bytes_delta` fields are `max(counter) - min(counter)` over the window. |
| 184 | + Since the underlying counters are cumulative *since the workload's runtime |
| 185 | + started* (see `compute-agent`), a restart during the window resets them to |
| 186 | + zero, which understates (or even negatives) the delta. This isn't |
| 187 | + silently patched over - it's a known limitation documented on |
| 188 | + `store.UsageSummary`. A more correct version would track restarts |
| 189 | + explicitly and sum deltas between them. |
| 190 | +- `estimated_cpu_core_seconds` is `avg(cpu_percent)/100 * window_seconds` - |
| 191 | + a simple approximation, not a rigorous integral over irregularly spaced |
| 192 | + samples. Fine as a first cut; revisit if billing needs tighter accuracy. |
| 193 | +- `sample_count: 0` means no data was reported in the window at all - it |
| 194 | + does **not** mean usage was zero. Every other field will also be |
| 195 | + zero-valued in that case; check `sample_count` first. |
| 196 | + |
| 197 | +## Known limitations / things to revisit |
| 198 | + |
| 199 | +- **Cross-repo schema coupling.** `internal/ingest/event.go` hand-mirrors |
| 200 | + the JSON shape `persys-scheduler` publishes (`internal/scheduler/usage_stream.go` |
| 201 | + and its `models.WorkloadUsage`). The two aren't shared via a common |
| 202 | + package (separate Go modules) - if the scheduler's shape changes, this |
| 203 | + file needs a matching manual update, and nothing will catch a drift at |
| 204 | + compile time. |
| 205 | +- **Dedup is TTL-bounded** (`METER_DEDUPE_TTL`, default 24h). A duplicate |
| 206 | + delivery arriving after the TTL expires would be double-counted. In |
| 207 | + practice redeliveries happen within seconds-to-minutes of |
| 208 | + `METER_CLAIM_MIN_IDLE`, not hours, so this is a deliberate, bounded |
| 209 | + tradeoff, not an oversight. |
| 210 | +- **Retention default (90 days) is a placeholder** - set |
| 211 | + `METER_RETENTION_DAYS` to match your actual billing/audit cycle. |
| 212 | +- **Not compiled in the environment that generated it.** Every non-trivial |
| 213 | + third-party API call here (`go-redis` streams commands, `clickhouse-go`'s |
| 214 | + `Query`/`QueryRow`/`PrepareBatch`, `client_golang`'s custom Collector) was |
| 215 | + checked against the real tagged source rather than written from memory, |
| 216 | + but this has not been run through `go build`/`go vet`. Run `make tidy && |
| 217 | + make build && make vet` before deploying. |
| 218 | + |
| 219 | +## Development |
| 220 | + |
| 221 | +```bash |
| 222 | +make help # list targets |
| 223 | +make tidy # resolve dependencies, generate go.sum (needed once, and after any go.mod edit) |
| 224 | +make build # compile to bin/persys-meter |
| 225 | +make test # go test ./... -race -cover |
| 226 | +make lint # gofmt + go vet |
| 227 | +``` |
0 commit comments