diff --git a/docs/concepts/performance.md b/docs/concepts/performance.md index 4b55721..775fd4c 100644 --- a/docs/concepts/performance.md +++ b/docs/concepts/performance.md @@ -88,6 +88,9 @@ pass `vacuum_cost_delay` / `vacuum_cost_limit` to let vacuum keep pace. A settings are applied. Full guidance, the eligibility-vs-throughput split, and the I/O caution on `vacuum_cost_delay=0` are in [Autovacuum tuning](../operations/alembic.md#autovacuum-tuning-recommended). +A table that only holds heartbeat timers is the exception: its live set stays +small, and the tuning made no measurable difference there (see +[below](#timer-extends-under-heartbeat-load)). ## Timer extends under heartbeat load @@ -101,14 +104,15 @@ runs it in about 0.02 ms. The cost is what comes with it: So the ceiling is set by commits and round trips, not by the statement. -One run of `just bench-extend`, on an Apple M2 with OrbStack, Postgres 17 at -default settings, one Python client process and a 20-connection pool. Each -identity pings every 2 seconds and has a 6-second timeout: +One 60-second run of `just bench-extend` per point, on an Apple M2 with +OrbStack, Postgres 17 at default settings, one Python client process and a +20-connection pool. Each identity pings every 2 seconds and has a 6-second +timeout: | Offered pings/s | Pings per transaction | Kept up | Ping p99 | Expiry lateness max | |---|---|---|---|---| | 1,000 | 1 | yes | 12 ms | 0.2 s | -| 3,000 | 1 | yes | 193 ms | 0.7 s | +| 3,000 | 1 | for 60 s only (see below) | 193 ms | 0.7 s | | 3,000 | 10 | yes | 14 ms | 1.2 s | | 5,000 | 1 | no | 17 s | 12.8 s | | 10,000 | 50 | yes | 618 ms | 3.2 s | @@ -119,10 +123,42 @@ saturated runs. Plan for a wide margin. Grouping pings into short transactions (sorted by `timer_id`, so concurrent transactions lock rows in the same order) raises the ceiling a lot, at the cost of latency. -Dead tuples pile up at the ping rate, about 150,000 in a minute at 10,000 -pings/s, and autovacuum only runs once per `autovacuum_naptime`, which -defaults to 60 seconds. The claim query slowed from about 0.2 to 1 ms over -such a minute. +3,000 pings/s with a commit per ping is this machine's limit, not a safe +rate. The 60-second run kept up, but three 15-minute runs at that rate fell +behind after 50 to 85 seconds, whatever the autovacuum settings. One +commit per ping held at 1,000 pings/s for 15 minutes, and 50 pings per +commit held at 10,000 pings/s. + +### Over 15 minutes + +Each identity is one live row, so the table stays small, and Postgres' +default autovacuum trigger (dead tuples above 20% of live rows) is crossed +within every minute. Autovacuum ran once per `autovacuum_naptime`, which +defaults to 60 seconds, in all of these runs, untuned ones included. Dead +tuples peak at about the ping rate times that interval. + +One 15-minute run per row, same machine. *untuned* has no autovacuum +reloptions, *tuned* applies `outbox_autovacuum_ddl()`, and *unthrottled* +adds `vacuum_cost_delay=0`: + +| Pings/s | Pings per transaction | Autovacuum | Table MB, start / end | Index MB, start / end | Dead tuples, peak | Claim mean, last minute | +|---|---|---|---|---|---|---| +| 1,000 | 1 | untuned | 0.5 / 2.0 | 0.3 / 3.6 | 53,622 | 0.38 ms | +| 1,000 | 1 | tuned | 0.5 / 1.9 | 0.3 / 3.5 | 53,771 | 0.42 ms | +| 1,000 | 1 | unthrottled | 0.5 / 1.9 | 0.3 / 6.1 | 54,571 | 0.34 ms | +| 10,000 | 50 | untuned | 5.0 / 25.5 | 2.4 / 46.8 | 590,515 | 3.27 ms | +| 10,000 | 50 | tuned | 5.0 / 25.4 | 2.4 / 40.2 | 608,410 | 3.10 ms | +| 10,000 | 50 | unthrottled | 5.0 / 25.4 | 2.4 / 60.4 | 523,700 | 2.79 ms | + +From the second minute on, table size held flat, dead tuples stayed in a band +(about 22,000 to 23,500 sampled at 1,000 pings/s, 270,000 to 320,000 at +10,000), and the claim query's per-minute mean stayed between about 0.3 and +0.5 ms at 1,000 pings/s and mostly between 2.7 and 3.9 ms at 10,000, with +one minute dipping to 1.3 to 1.9 ms in each 10,000 run. Index size rose +in a step or two early on and then held. The differences between the three +settings are within run-to-run noise: the largest index came from the +unthrottled runs. For a table that only holds heartbeat timers, autovacuum +tuning doesn't change the outcome; the commit rate does. ## Measuring it yourself @@ -134,7 +170,9 @@ can run it against a local Postgres: - `just bench-check` gates the deterministic counts against `benchmarks/baseline.json`; this is what CI runs. - `just bench-extend` runs the timer extend sweep above and prints its tables. - Nothing in it is gated: every number depends on timing. + Nothing in it is gated: every number depends on timing. The 15-minute + table came from `just bench-extend --only r1000-b1 r10000-b50 --seconds 900 + --vacuum untuned tuned unthrottled`; `--json` writes every sample. The statement-count reductions (`select_calls`, `delete_calls`, `insert_calls`, …) are machine-independent and exact. Wall-clock throughput is not: it scales diff --git a/docs/usage/timers.md b/docs/usage/timers.md index c52329b..189dbda 100644 --- a/docs/usage/timers.md +++ b/docs/usage/timers.md @@ -163,9 +163,25 @@ that matters, either: Every extend writes a new row version, even when the deadline does not move. `next_attempt_at` is indexed, so these updates are not HOT updates: -each one adds index entries and a dead tuple for autovacuum to clean up. -Size autovacuum for your heartbeat rate (see -[Alembic migrations](../operations/alembic.md), `outbox_autovacuum_ddl`). +each one leaves a dead tuple and dead index entries until autovacuum +reclaims them. + +A table of heartbeat timers holds about one live row per timer, so Postgres' +default autovacuum trigger, dead tuples above 20% of the live rows, is +crossed within a minute as long as each timer is pinged more often than +every five minutes. Autovacuum then runs once per `autovacuum_naptime`, a +cluster setting that defaults to 60 seconds, and dead tuples peak at about +the ping rate times that interval: around 600,000 at 10,000 pings/s. In +15-minute benchmark runs at 1,000 and 10,000 pings/s, table size, dead +tuples and claim time stopped growing within two minutes, and neither +`outbox_autovacuum_ddl` nor `vacuum_cost_delay=0` made a measurable +difference (see [Performance](../concepts/performance.md#timer-extends-under-heartbeat-load)). + +What limits a heartbeat is the commit rate. Group pings into short +transactions once you are past about a thousand pings per second. If the +same table also carries ordinary messages, the queue-table advice in +[Alembic migrations](../operations/alembic.md#autovacuum-tuning-recommended) +still applies. ## Cancellation