|
| 1 | +# faststream-outbox 0.11.0 — a performance pass: write-path, terminal-path, and autovacuum tooling |
| 2 | + |
| 3 | +**Minor release.** A throughput-focused release built from a performance review |
| 4 | +of the transactional-outbox transport: the producer stops emitting redundant |
| 5 | +`NOTIFY`s, the subscriber gains an opt-in batched terminal flush, and a new |
| 6 | +`outbox_autovacuum_ddl` helper (plus an opt-in `validate_schema` probe) makes the |
| 7 | +recommended table hygiene one function call. A reproducible benchmark harness |
| 8 | +underpins all of it. **Backward-compatible by default** — every new knob defaults |
| 9 | +to today's behavior, so an in-place upgrade changes nothing until you opt in. |
| 10 | + |
| 11 | +## Performance |
| 12 | + |
| 13 | +- **Producer `NOTIFY` dedup — one `pg_notify` per `(transaction, queue)`, not per |
| 14 | + publish.** `broker.publish` / `publish_batch` ran a `SELECT pg_notify(channel, |
| 15 | + queue)` on every call, so N publishes to one queue in one transaction emitted N |
| 16 | + `NOTIFY` statements — N−1 of them pure round-trip waste, since Postgres already |
| 17 | + coalesces identical `(channel, payload)` `NOTIFY`s per transaction at delivery. A |
| 18 | + transaction-scoped memo (a `WeakKeyDictionary` keyed on the innermost active |
| 19 | + transaction) now emits exactly one `pg_notify` per distinct queue per |
| 20 | + transaction. **Behavior-preserving** — the subscriber still gets the same |
| 21 | + "≥1 wake per queue per commit" it did before, fired inline at the first publish; |
| 22 | + delivery, ordering, and the transactional producer contract are unchanged. On a |
| 23 | + bulk publish (5000 rows, one queue, one transaction) the write path measured |
| 24 | + **~1.9× throughput / ~46% less wall-clock** on loopback Postgres, and the |
| 25 | + absolute saving grows with database round-trip latency. Single-publish-per- |
| 26 | + transaction workloads are unaffected (one row still emits one `NOTIFY`). Default-on, |
| 27 | + no knob. (#135) |
| 28 | + |
| 29 | +## New features |
| 30 | + |
| 31 | +- **Opt-in batched terminal flush — `terminal_flush_batch_size` (default `1`).** |
| 32 | + The subscriber deletes/DLQs each terminally-handled row individually by default |
| 33 | + (`terminal_flush_batch_size=1`, identical to prior behavior). Set it higher to |
| 34 | + batch terminal writes into one statement per N rows, cutting the terminal-path |
| 35 | + statement count under high throughput. This **widens the at-least-once |
| 36 | + redelivery window** — a crash between handler success and the batched flush |
| 37 | + redelivers up to `N−1` already-processed rows on recovery — so it is opt-in and |
| 38 | + off by default; only raise it when your handlers are idempotent. Available on the |
| 39 | + broker subscriber, `OutboxRouter`, and the FastAPI router. (#131, #132) |
| 40 | + |
| 41 | +- **`outbox_autovacuum_ddl(...)` — the recommended per-table autovacuum settings |
| 42 | + as one call.** A new top-level export that renders the |
| 43 | + `ALTER TABLE … SET (autovacuum_*)` statement for an outbox table — drop it into |
| 44 | + an Alembic migration (`op.execute(outbox_autovacuum_ddl("outbox"))`) or run it via |
| 45 | + psql. It sets `autovacuum_vacuum_scale_factor = 0` plus size-independent dead-tuple |
| 46 | + / insert thresholds (eligibility), breaking the stale-`reltuples` death-spiral that |
| 47 | + makes a high-churn queue table bloat. Signature: |
| 48 | + |
| 49 | + ```python |
| 50 | + outbox_autovacuum_ddl( |
| 51 | + table_name="outbox", *, schema=None, |
| 52 | + vacuum_threshold=1000, insert_threshold=1000, |
| 53 | + vacuum_cost_delay=None, vacuum_cost_limit=None, |
| 54 | + ) -> str |
| 55 | + ``` |
| 56 | + |
| 57 | + `vacuum_cost_delay` / `vacuum_cost_limit` are the vacuum **throughput** knobs |
| 58 | + (off by default → cluster default emitted). A churn measurement showed throughput |
| 59 | + — not eligibility — is the binding constraint under heavy sustained load, so these |
| 60 | + let vacuum keep pace; `vacuum_cost_delay=0` runs it unthrottled (fast, but |
| 61 | + I/O-heavy on a shared cluster — opt in deliberately). (#133, #134) |
| 62 | + |
| 63 | +- **`validate_schema(check_autovacuum=True)` — opt-in autovacuum probe.** The |
| 64 | + schema validator gains a `check_autovacuum` flag (default `False`) that asserts the |
| 65 | + live table carries the structural `scale_factor = 0` + threshold settings, so a CI |
| 66 | + gate can catch a table created without the recommended hygiene. The situational |
| 67 | + throughput knobs (`cost_delay` / `cost_limit`) are **not** enforced — they are |
| 68 | + tuning, not a structural requirement. Default `False` keeps `validate_schema()` |
| 69 | + identical to 0.10.x. (#133) |
| 70 | + |
| 71 | +## Bug fixes |
| 72 | + |
| 73 | +- **`validate_schema()` now reflects a table in a named schema correctly.** The |
| 74 | + validator's Alembic reflection omitted `include_schemas=True`, so a table created |
| 75 | + under a non-default schema (`make_outbox_table(..., schema="app")`) failed |
| 76 | + validation spuriously. Reflection now filters by the target schema, and an |
| 77 | + explicit-default-schema table (`MetaData(schema="public")`) is normalized against |
| 78 | + the connection's `default_schema_name` so it validates as unqualified. (#133) |
| 79 | + |
| 80 | +## Internals / tooling |
| 81 | + |
| 82 | +- **Benchmark harness for the outbox transport (`benchmarks/`, `just bench` / |
| 83 | + `just bench-check`).** A reproducible producer/consumer workload sweep that records |
| 84 | + per-message DB counters via `pg_stat_statements` (split by leading SQL keyword) and |
| 85 | + gates the deterministic, machine-independent counts against |
| 86 | + `benchmarks/baseline.json` in CI. This is contributor tooling — it drove and now |
| 87 | + locks the wins above (the producer `NOTIFY` dedup shows as `select_calls` 5000 → 1; |
| 88 | + the batched terminal flush as fewer `delete_calls`). Not part of the installed |
| 89 | + wheel. (#130) |
| 90 | +- **Honest autovacuum docs.** The operations and architecture docs were rewritten to |
| 91 | + scope the benefit precisely: the shipped `scale_factor=0` + threshold settings |
| 92 | + control vacuum *eligibility* (necessary, not sufficient); under heavy churn, |
| 93 | + *throughput* (`cost_delay`/`cost_limit`) is the binding lever. Framed as standard |
| 94 | + queue-table hygiene and insurance, not a measured silver bullet. (#134) |
| 95 | + |
| 96 | +## Breaking changes |
| 97 | + |
| 98 | +None. Every new parameter defaults to prior behavior: |
| 99 | +`terminal_flush_batch_size=1` (per-message flush), `check_autovacuum=False`, and the |
| 100 | +`outbox_autovacuum_ddl` cost knobs default to `None` (cluster default). The |
| 101 | +`NOTIFY` dedup is behavior-preserving. An in-place upgrade requires no code or |
| 102 | +schema change. |
| 103 | + |
| 104 | +## Migration |
| 105 | + |
| 106 | +- **No action required to upgrade.** Optional adoption: |
| 107 | + - Set `terminal_flush_batch_size=N` on your subscriber(s) **only if your handlers |
| 108 | + are idempotent** — it trades a wider redelivery window for terminal-path |
| 109 | + throughput. |
| 110 | + - Apply `outbox_autovacuum_ddl("<table>")` in an Alembic migration to adopt the |
| 111 | + recommended autovacuum settings; add `vacuum_cost_delay`/`vacuum_cost_limit` if |
| 112 | + you run a high-churn table and vacuum can't keep pace. |
| 113 | + - Add `check_autovacuum=True` to a `validate_schema()` CI gate to enforce the above. |
| 114 | + |
| 115 | +## Touched surface |
| 116 | + |
| 117 | +- `faststream_outbox/publisher/producer.py` — per-`(transaction, queue)` `NOTIFY` |
| 118 | + dedup memo. (#135) |
| 119 | +- `faststream_outbox/subscriber/` + `registrator.py`, `router.py`, |
| 120 | + `fastapi/router.py` — `terminal_flush_batch_size` plumbed through the subscriber |
| 121 | + config, factory, and all three registration entry points. (#131) |
| 122 | +- `faststream_outbox/autovacuum.py` (new) + `__init__.py` export — `outbox_autovacuum_ddl`. (#133, #134) |
| 123 | +- `faststream_outbox/client.py`, `broker.py`, `testing.py` — `validate_schema(check_autovacuum=...)` + named-schema reflection fix. (#133) |
| 124 | +- `benchmarks/` (new) + `Justfile` (`bench` / `bench-check`) + CI `bench` job. (#130) |
| 125 | +- `docs/operations/`, `architecture/` — batching, autovacuum (throughput-scoped), and producer-`NOTIFY` docs. (#132, #134, #135) |
| 126 | + |
| 127 | +## See also |
| 128 | + |
| 129 | +- [`planning/changes/2026-07-14.01-benchmark-harness.md`](../changes/2026-07-14.01-benchmark-harness.md) ([#130](https://github.com/modern-python/faststream-outbox/pull/130)) |
| 130 | +- [`planning/changes/2026-07-15.01-batched-terminal-flush.md`](../changes/2026-07-15.01-batched-terminal-flush.md) ([#131](https://github.com/modern-python/faststream-outbox/pull/131), [#132](https://github.com/modern-python/faststream-outbox/pull/132)) |
| 131 | +- [`planning/changes/2026-07-15.02-autovacuum-tuning.md`](../changes/2026-07-15.02-autovacuum-tuning.md) ([#133](https://github.com/modern-python/faststream-outbox/pull/133)) |
| 132 | +- [`planning/changes/2026-07-16.01-autovacuum-cost-knobs.md`](../changes/2026-07-16.01-autovacuum-cost-knobs.md) ([#134](https://github.com/modern-python/faststream-outbox/pull/134)) |
| 133 | +- [`planning/changes/2026-07-16.02-notify-dedup.md`](../changes/2026-07-16.02-notify-dedup.md) ([#135](https://github.com/modern-python/faststream-outbox/pull/135)) |
0 commit comments