Skip to content

Commit 306ad25

Browse files
lesnik512claude
andauthored
refactor(message): canonical Outcome sum type on OutboxInnerMessage (#145)
Replace the three loosely-coupled intent fields (terminal_failure_reason, pending_delay_seconds, state_set) with one canonical `outcome` holding a sum type Ack | Retry(delay) | Terminal(reason). Success becomes an explicit Ack() instead of the implicit "all None" state, and "reason and delay both set" is now unrepresentable. Option A (surgical): outcome is the single canonical field; the three old attributes become read-only @Property views of it, so the five other subscriber readers and every read-assertion in the tests are unchanged. dispatch_one matches on the disjoint variant -- which dissolves the terminal-before-retry ordering invariant (mutually exclusive variants have no ordering dependence). Only the 12 test sites that *wrote* the old fields change, to set `outcome`. DLQFailureReason (the public wire contract) is unchanged -- it is now Terminal.reason's type. dlq_table=None stays bit-for-bit identical. Truth-home promotion in architecture/dlq.md + the CLAUDE.md DLQ invariant line. Behavior-preserving. Full suite green (603 passed, coverage 100%), lint + ty + check-planning clean. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 7442058 commit 306ad25

6 files changed

Lines changed: 153 additions & 55 deletions

File tree

‎CLAUDE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ The package wires a FastStream `Broker`/`Registrator`/`Subscriber` trio whose tr
3030
- **Relay to foreign broker** — native FastStream publisher-chain (`@kafka_pub @broker_outbox.subscriber("q")`); bad chain composition raises `_OutboxConfigError` and retries via **lease expiry**, not the retry strategy. → [relay.md](architecture/relay.md)
3131
- **Timers** — `activate_in`/`activate_at` are mutually exclusive and gate eligibility; `timer_id` is "at most one *live* row per `(queue, timer_id)`", not a global idempotency key; `cancel_timer`'s `acquired_token IS NULL` guard is load-bearing. → [timers.md](architecture/timers.md)
3232
- **User-owned schema** — caller owns the table; the three partial indexes + the `<table>_lease_ck` CHECK are load-bearing; **no `state` column**; `validate_schema()` is opt-in. → [schema.md](architecture/schema.md)
33-
- **Opt-in DLQ** — with `dlq_table=None` every path is **bit-for-bit identical**; the `DLQFailureReason` `Literal` is the **public contract** (changes are API-breaking); branch on `terminal_failure_reason` **before** `last_exception`. → [dlq.md](architecture/dlq.md)
33+
- **Opt-in DLQ** — with `dlq_table=None` every path is **bit-for-bit identical**; the `DLQFailureReason` `Literal` is the **public contract** (changes are API-breaking); delivery records one disjoint `Outcome` (`Ack`/`Retry`/`Terminal`) that `dispatch_one` matches on (`terminal_failure_reason`/`pending_delay_seconds`/`state_set` are read-only views). → [dlq.md](architecture/dlq.md)
3434
- **Two-loop subscriber + lease-token invariant** — fetch loop + N worker loops; **every terminal write filters on `acquired_token`** (a stale write finds `rowcount==0` and is dropped) — any new fetch/terminal path must preserve this; `AckPolicy.ACK_FIRST` is rejected at registration; `lease_ttl_seconds` must exceed handler P99. → [subscriber.md](architecture/subscriber.md)
3535
- **Drain on stop** — custom `stop()` overrides + the `dispatch_one` shutdown-race guard prevent row leaks on rolling deploys; **re-check both overrides when touching shutdown**. → [drain.md](architecture/drain.md)
3636
- **Test broker** — `TestOutboxBroker` swaps in `FakeOutboxClient`; sync mode dispatches the handler before `publish` returns; `test_client_contract.py` pins real-vs-fake parity. → [test-broker.md](architecture/test-broker.md)

‎architecture/dlq.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -12,15 +12,15 @@ User-facing: `docs/usage/dlq.md`. Invariant summary: `CLAUDE.md` § Opt-in DLQ.
1212

1313
The CTE's `RETURNING` / `INSERT` / `SELECT` column lists are not hand-written — they derive from `_DLQ_PROJECTION` (the `(outbox_col, dlq_col)` pairs copied verbatim) plus `_DLQ_INJECTED_COLUMNS` (`failure_reason`, `last_exception`, supplied by the caller) in `schema.py`. The fake (`FakeOutboxClient.delete_with_lease`) builds its audit dict from the same constants, so the two substrates can't drift on which columns the archive carries — a DLQ column change is one edit in `schema.py`, verified across both adapters by `tests/test_client_contract.py`. `failed_at` is not in the projection; it rides the DLQ column's `server_default`.
1414

15-
## `terminal_failure_reason` routing
15+
## Terminal-outcome routing
1616

17-
`OutboxInnerMessage.terminal_failure_reason` is set on the three failure paths:
17+
`OutboxInnerMessage.outcome` records a single canonical `Outcome` (`message.py`): `Ack` (success), `Retry(delay_seconds)`, or `Terminal(reason)`. The three failure paths record a `Terminal`:
1818

19-
- `allow_delivery` False → `"max_deliveries"`
20-
- `_nack` strategy-exhausted → `"retry_terminal"`
21-
- `_reject` → `"rejected"`
19+
- `allow_delivery` False → `Terminal("max_deliveries")`
20+
- `_nack` strategy-exhausted → `Terminal("retry_terminal")`
21+
- `_reject` → `Terminal("rejected")`
2222

23-
`_flush_terminal` reads it to decide whether to build a DLQ payload; `dispatch_one` also reads it to pick the `nacked_terminal(reason=…)` tag value. **Branch on `terminal_failure_reason` BEFORE `last_exception`**, so manual `await msg.reject()` (no exception raised) routes correctly to `nacked_terminal(reason="rejected")` instead of the previously-incorrect `acked`. Success (`_ack`) leaves the field None; success rows never touch the DLQ.
23+
`terminal_failure_reason` is now a **read-only view** of `outcome` (`Terminal.reason`, else `None`), so existing readers are unchanged: `_flush_terminal` reads it to decide whether to build a DLQ payload; the `nacked_terminal(reason=…)` tag value comes from the same reason. `dispatch_one` **matches on the disjoint `Outcome` variant** — `Terminal → nacked_terminal`, `Retry → reschedule`, `Ack → delete`. Because the variants are mutually exclusive there is no ordering dependence between the arms: a manual `await msg.reject()` (no exception raised) records `Terminal("rejected")` directly, independent of `last_exception`. Success (`_ack` → `Ack`) leaves `terminal_failure_reason` None; success rows never touch the DLQ.
2424

2525
**The `DLQFailureReason` `Literal` type (`message.py`) is the public contract** for this string — operator queries and dashboard labels key off these values, so changing them is API-breaking.
2626

‎faststream_outbox/message.py‎

Lines changed: 64 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,32 @@
3636
DLQFailureReason = Literal["max_deliveries", "retry_terminal", "rejected"]
3737

3838

39+
# Canonical delivery outcome. ``ack``/``nack``/``reject`` record exactly one of these
40+
# on ``OutboxInnerMessage.outcome``; the worker loop matches on the variant to pick the
41+
# DB write. The legacy ``state_set``/``terminal_failure_reason``/``pending_delay_seconds``
42+
# attributes are read-only views derived from it.
43+
@dataclass(frozen=True, slots=True)
44+
class Ack:
45+
"""Handler succeeded; the row is deleted (no DLQ)."""
46+
47+
48+
@dataclass(frozen=True, slots=True)
49+
class Retry:
50+
"""Handler failed; the strategy scheduled another attempt after ``delay_seconds``."""
51+
52+
delay_seconds: float
53+
54+
55+
@dataclass(frozen=True, slots=True)
56+
class Terminal:
57+
"""Delivery failed terminally; the row is deleted (and DLQ'd when configured)."""
58+
59+
reason: DLQFailureReason
60+
61+
62+
Outcome = Ack | Retry | Terminal
63+
64+
3965
# Header keys the outbox envelope owns on a row's ``headers`` dict. ``_encode_payload``
4066
# (``envelope.py``) writes them, ``OutboxParser.parse_message`` (``parser.py``) reads
4167
# them back, and the relay header-propagation (``_maybe_propagate_inbound_headers``)
@@ -50,9 +76,11 @@
5076
class OutboxInnerMessage:
5177
"""In-memory copy of a claimed outbox row, plus ack/nack/reject intent helpers.
5278
53-
The ack/nack/reject methods set in-memory intent flags (``terminal_failure_reason``,
54-
``pending_delay_seconds``). The worker loop reads those flags and issues the
55-
actual DB write, scoped by ``acquired_token``.
79+
The ack/nack/reject methods record a single canonical ``Outcome``
80+
(``Ack``/``Retry``/``Terminal``) on ``outcome``. The worker loop matches on the
81+
variant and issues the actual DB write, scoped by ``acquired_token``.
82+
``state_set``/``terminal_failure_reason``/``pending_delay_seconds`` are read-only
83+
views of ``outcome`` kept for existing readers.
5684
"""
5785

5886
id: int
@@ -75,16 +103,32 @@ class OutboxInnerMessage:
75103
retry_strategy: "RetryStrategyProto | None" = None
76104
last_exception: BaseException | None = None
77105

78-
state_set: bool = field(default=False, init=False)
79-
# Set by ``_nack`` when the strategy schedules a retry; consumed by the
80-
# subscriber's ``_flush_retry`` to drive ``mark_pending_with_lease``.
81-
pending_delay_seconds: float | None = field(default=None, init=False)
82-
# Set on terminal-failure paths (``allow_delivery`` False, ``_nack`` exhausted,
83-
# ``_reject``). ``_flush_terminal`` reads it to decide whether to build a DLQ
84-
# payload; ``dispatch_one`` reads it to pick the ``nacked_terminal`` reason
85-
# tag. Stays ``None`` on the success (``_ack``) path so handler-success
86-
# never touches the DLQ.
87-
terminal_failure_reason: "DLQFailureReason | None" = field(default=None, init=False)
106+
# Canonical delivery outcome, set exactly once by ``_ack``/``_nack``/``_reject``/
107+
# ``allow_delivery``. ``None`` until intent is recorded. The three properties below
108+
# derive the legacy views the worker loop and metrics read.
109+
outcome: "Outcome | None" = field(default=None, init=False)
110+
111+
@property
112+
def state_set(self) -> bool:
113+
return self.outcome is not None
114+
115+
@property
116+
def terminal_failure_reason(self) -> "DLQFailureReason | None":
117+
"""Terminal reason (``allow_delivery`` False, ``_nack`` exhausted, ``_reject``), else ``None``.
118+
119+
``_flush_terminal`` reads it to decide whether to build a DLQ payload;
120+
``dispatch_one`` reads it to pick the ``nacked_terminal`` reason tag. ``None``
121+
on the ``Ack``/``Retry`` paths so handler-success never touches the DLQ.
122+
"""
123+
return self.outcome.reason if isinstance(self.outcome, Terminal) else None
124+
125+
@property
126+
def pending_delay_seconds(self) -> float | None:
127+
"""Retry delay set by ``_nack`` when the strategy schedules a retry, else ``None``.
128+
129+
Consumed by ``_flush_retry`` to drive ``mark_pending_with_lease``.
130+
"""
131+
return self.outcome.delay_seconds if isinstance(self.outcome, Retry) else None
88132

89133
# ``**_options`` are accepted-and-ignored: FastStream's AcknowledgementMiddleware
90134
# forwards ``message.nack(**extra_options)`` for native idioms like
@@ -103,13 +147,13 @@ async def reject(self, **_options: Any) -> None:
103147
await self._update_state_if_not_set(self._reject)
104148

105149
async def _update_state_if_not_set(self, fn: Callable[[], Awaitable[None]]) -> None:
106-
if self.state_set:
150+
if self.outcome is not None:
107151
return
108152
await fn()
109-
self.state_set = True
110153

111154
async def _ack(self) -> None:
112155
self._record_attempt()
156+
self.outcome = Ack()
113157

114158
async def _nack(self) -> None:
115159
self._record_attempt()
@@ -135,16 +179,16 @@ async def _nack(self) -> None:
135179
self.retry_strategy,
136180
self,
137181
)
138-
self.terminal_failure_reason = "retry_terminal"
182+
self.outcome = Terminal("retry_terminal")
139183
return
140184
if delay is None:
141-
self.terminal_failure_reason = "retry_terminal"
185+
self.outcome = Terminal("retry_terminal")
142186
else:
143-
self.pending_delay_seconds = delay
187+
self.outcome = Retry(delay)
144188

145189
async def _reject(self) -> None:
146190
self._record_attempt()
147-
self.terminal_failure_reason = "rejected"
191+
self.outcome = Terminal("rejected")
148192

149193
def _record_attempt(self) -> None:
150194
self.attempts_count += 1
@@ -156,8 +200,7 @@ def _record_attempt(self) -> None:
156200
def allow_delivery(self, *, max_deliveries: int | None, logger: "LoggerProto | None") -> bool:
157201
"""If ``max_deliveries`` is set and exceeded, mark for deletion without invoking the handler."""
158202
if max_deliveries is not None and self.deliveries_count > max_deliveries:
159-
self.state_set = True
160-
self.terminal_failure_reason = "max_deliveries"
203+
self.outcome = Terminal("max_deliveries")
161204
if logger is not None:
162205
logger.log(
163206
logging.ERROR,

‎faststream_outbox/subscriber/usecase.py‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@
3939
from faststream.specification.schema import Message, Operation, SubscriberSpec
4040
from typing_extensions import override
4141

42-
from faststream_outbox.message import ENVELOPE_MANAGED_HEADERS, OutboxInnerMessage
42+
from faststream_outbox.message import ENVELOPE_MANAGED_HEADERS, OutboxInnerMessage, Retry, Terminal
4343
from faststream_outbox.parser.parser import OutboxParser
4444
from faststream_outbox.publisher.fake import OutboxFakePublisher
4545
from faststream_outbox.response import OutboxResponse
@@ -701,23 +701,23 @@ async def dispatch_one( # linear pipeline: guard, consume, branch on outcome, f
701701
await row.assert_state_set(logger)
702702
duration_seconds = time.perf_counter() - start_perf
703703
common = {**base, "deliveries_count": row.deliveries_count, "duration_seconds": duration_seconds}
704-
# Branch on ``terminal_failure_reason`` (set by ``_nack``/``_reject``) before
705-
# ``last_exception`` so manual ``msg.reject()`` (no exception raised) still
706-
# surfaces as ``nacked_terminal(reason="rejected")``, and ack-policy-driven
707-
# rejects (REJECT_ON_ERROR) carry ``reason="rejected"`` rather than the
708-
# incorrect ``"retry_terminal"`` they got under the old ``last_exception``-first
709-
# ordering. Successful handlers leave ``terminal_failure_reason=None``.
710-
if row.terminal_failure_reason is not None:
711-
terminal_tags = self._with_exception_type({**common, "reason": row.terminal_failure_reason}, row)
712-
event, tags = "nacked_terminal", terminal_tags
713-
elif row.pending_delay_seconds is not None:
714-
retry_tags = self._with_exception_type({**common, "next_delay_seconds": row.pending_delay_seconds}, row)
715-
# Retry is a per-row UPDATE -- never batched. Flush inline, unchanged.
704+
# Match on the disjoint ``Outcome`` variant recorded by ack/nack/reject:
705+
# Terminal -> nacked_terminal (delete + DLQ), Retry -> reschedule, Ack -> delete.
706+
# The variants are mutually exclusive, so there is no ordering dependence between
707+
# the arms (e.g. a manual ``msg.reject()`` records ``Terminal("rejected")``
708+
# directly, independent of ``last_exception``).
709+
outcome = row.outcome
710+
if isinstance(outcome, Terminal):
711+
tags = self._with_exception_type({**common, "reason": outcome.reason}, row)
712+
event = "nacked_terminal"
713+
elif isinstance(outcome, Retry):
714+
retry_tags = self._with_exception_type({**common, "next_delay_seconds": outcome.delay_seconds}, row)
715+
# Retry is a per-row UPDATE -- never batched. Flush inline.
716716
# P17: emit only after the flush lands; a lease-lost update emits lease_lost instead.
717717
if await self._safe_flush(row, terminal=False, writer_conn=writer_conn):
718718
self._emit_metric("nacked_retried", retry_tags)
719719
return False
720-
else:
720+
else: # Ack (assert_state_set guaranteed outcome is set, so this is Ack, never None)
721721
event, tags = "acked", common
722722
# Terminal delete: batch it if batchable, else flush inline. P17 holds either way —
723723
# the metric fires only when the delete lands (inline here, or deferred in _flush_buffer).
Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
---
2+
summary: Replaced OutboxInnerMessage's terminal_failure_reason / pending_delay_seconds / state_set fields with one canonical Outcome sum type (Ack | Retry | Terminal); the old attributes become read-only views. Behavior-preserving; dissolves the terminal-before-retry ordering invariant.
3+
---
4+
5+
# Change: Canonical Outcome sum type on OutboxInnerMessage
6+
7+
**Lane:** lightweight. A behavior-preserving refactor of the load-bearing
8+
terminal/dispatch path that reshapes an internal representation and dissolves a
9+
documented ordering invariant — recorded here so the change is legible.
10+
11+
## Goal
12+
13+
Deepen `OutboxInnerMessage`: `ack`/`nack`/`reject`/`allow_delivery` recorded delivery
14+
intent across three loosely-coupled fields (`terminal_failure_reason`,
15+
`pending_delay_seconds`, `state_set`), where "success" was the implicit "all None"
16+
state and "reason and delay both set" was representable-but-never-used. Replace them
17+
with one canonical `Outcome` sum type so success is an explicit variant and the
18+
impossible combination is unrepresentable.
19+
20+
## Approach (Option A — canonical outcome + derived views)
21+
22+
- `message.py` defines `Ack` / `Retry(delay_seconds)` / `Terminal(reason)`
23+
(frozen, slotted) and `Outcome = Ack | Retry | Terminal`. `OutboxInnerMessage.outcome`
24+
is the single canonical field; the setters assign it (`_ack` → `Ack()`, `_nack` →
25+
`Retry`/`Terminal("retry_terminal")`, `_reject` → `Terminal("rejected")`,
26+
`allow_delivery` → `Terminal("max_deliveries")`).
27+
- `terminal_failure_reason` / `pending_delay_seconds` / `state_set` become **read-only
28+
`@property` views** of `outcome` (zero-drift derivations, no setters). So the five
29+
other subscriber readers (`_flush_terminal`, `_flush_retry`, `_terminal_has_dlq`, the
30+
shutdown-race `state_set` guard, metric tags) and every read-assertion in the tests
31+
are unchanged.
32+
- `dispatch_one` matches on the disjoint variant (`Terminal → nacked_terminal`,
33+
`Retry → reschedule`, `Ack → delete`). Because the variants are mutually exclusive,
34+
the **terminal-before-retry / before-`last_exception` ordering invariant is dissolved**
35+
— recorded in `architecture/dlq.md` and the `CLAUDE.md` DLQ invariant line.
36+
37+
The `DLQFailureReason` `Literal` (the public wire contract queried by operators) is
38+
unchanged — it is now `Terminal.reason`'s type. `dlq_table=None` stays bit-for-bit
39+
identical.
40+
41+
## Files
42+
43+
- `faststream_outbox/message.py` — `Outcome` types; `outcome` field + three property
44+
views; setters wired to `outcome`.
45+
- `faststream_outbox/subscriber/usecase.py` — `dispatch_one` matches on `outcome`
46+
(imports `Retry`, `Terminal`); no other reader changed.
47+
- `tests/test_unit.py` — the 12 sites that *wrote* the old fields now set `outcome`
48+
(`Retry(...)` / `Terminal(...)`); read-assertions unchanged (properties).
49+
- `architecture/dlq.md`, `CLAUDE.md` — truth-home promotion of the routing model.
50+
51+
## Verification
52+
53+
- [x] `just test` — full suite green: 603 passed, Postgres 17, coverage 100.00%.
54+
- [x] `just lint` — clean (ruff + ty).
55+
- [x] `just check-planning` — passes.

0 commit comments

Comments
 (0)