Skip to content

Commit 56ed38f

Browse files
authored
feat!: add InboxCleaner and drop the Inbox cleanup methods (#5)
InboxCleaner runs a whole cleanup run: one cutoff from the database clock, then batches in their own READ COMMITTED transactions until one deletes nothing. Inbox.cleanup_cutoff and Inbox.delete_expired_batch are removed; Inbox now only claims.
1 parent c140df8 commit 56ed38f

10 files changed

Lines changed: 421 additions & 285 deletions

File tree

‎AGENTS.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@ The things it does not say:
2222

2323
- `Inbox` never commits, rolls back, begins, or opens a session or connection. Transaction
2424
ownership stays with the caller. Any new method must keep that.
25+
- `InboxCleaner` is the only code that begins and commits, and only on connections it opens from
26+
the caller's engine. It never takes a caller's session and never disposes the engine.
2527
- `Inbox.claim` requires an open transaction on its session. That is what turns a wrongly wired
2628
second session into an error instead of a marker that never commits.
2729
- Concurrency claims are proven only by tests on separate connections, never one shared test

‎CONTEXT.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,3 +36,7 @@ _Avoid_: TTL (nothing expires on its own; cleanup deletes)
3636
**Cutoff**:
3737
The database timestamp, taken once per cleanup run, before which markers are deleted.
3838
_Avoid_: deadline, expiry time
39+
40+
**Cleanup run**:
41+
One pass that takes a cutoff and deletes the markers older than it, batch by batch.
42+
_Avoid_: vacuum (Postgres VACUUM is something else), purge, sweep, expiry

‎README.md‎

Lines changed: 29 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,9 @@
77
transaction as the consumer's own writes. A redelivered message finds its marker and is skipped. A
88
failed transaction rolls the marker back, so a retry processes the message again.
99

10-
It works with SQLAlchemy's asyncio extension on PostgreSQL. It stores no payload, runs no worker,
11-
and never begins, commits or rolls back a transaction for you.
10+
It works with SQLAlchemy's asyncio extension on PostgreSQL. It stores no payload and runs no worker.
11+
`Inbox` never begins, commits or rolls back a transaction for you. Only `InboxCleaner` does that, on
12+
its own connections.
1213

1314
```bash
1415
pip install sqlalchemy-inbox
@@ -92,31 +93,38 @@ or fall back to Kafka offsets.
9293

9394
## Cleaning up
9495

95-
Markers are kept until you delete them. Take one cutoff per cleanup run from the database clock,
96-
then delete in batches, one transaction per batch, until a batch deletes nothing:
96+
Markers are kept until you delete them. Run `InboxCleaner` from a cron job:
9797

9898
```python
99-
@dataclasses.dataclass(kw_only=True, frozen=True, slots=True)
100-
class CleanupInboxUseCase:
101-
transaction: Transaction
102-
inbox: Inbox
99+
import asyncio
100+
import datetime
103101

104-
async def __call__(self, *, retention: datetime.timedelta, batch_size: int) -> None:
105-
async with self.transaction:
106-
cutoff = await self.inbox.cleanup_cutoff(retention=retention)
107-
while await self._delete_batch(cutoff=cutoff, limit=batch_size):
108-
pass
102+
from sqlalchemy.ext.asyncio import create_async_engine
103+
from sqlalchemy_inbox import InboxCleaner
109104

110-
@postgres_retry
111-
async def _delete_batch(self, *, cutoff: datetime.datetime, limit: int) -> int:
112-
async with self.transaction:
113-
deleted = await self.inbox.delete_expired_batch(cutoff=cutoff, limit=limit)
114-
await self.transaction.commit()
115-
return deleted
105+
106+
async def main() -> None:
107+
engine = create_async_engine(DATABASE_DSN)
108+
try:
109+
cleaner = InboxCleaner(engine=engine, table=INBOX_TABLE, retention=datetime.timedelta(days=7))
110+
await cleaner.cleanup()
111+
finally:
112+
await engine.dispose()
113+
114+
115+
asyncio.run(main())
116116
```
117117

118-
Batches that committed stay deleted if a later batch fails, and the error reaches your scheduler.
119-
Cleanup skips markers that another transaction holds, so it never waits on a consumer.
118+
A cleanup run takes one cutoff from the database clock, `now()` minus `retention`, and deletes the
119+
markers older than it in batches of `batch_size` (1000 by default), one transaction per batch, until
120+
a batch deletes nothing. `cleanup()` returns a `CleanupResult` with the `cutoff` and the
121+
`deleted_count`. It logs each batch at DEBUG and the total at INFO to the `sqlalchemy_inbox` logger.
122+
123+
Batches that committed stay deleted if a later batch fails or the task is cancelled, and the error
124+
reaches your cron job. The cleaner doesn't retry: the next run deletes what is left. It skips markers
125+
that another transaction holds, so it never waits on a consumer, and two runs at once share the
126+
work. Each batch runs at READ COMMITTED whatever the engine's default. The engine stays yours to
127+
dispose.
120128

121129
## Guarantees
122130

‎docs/adr/0001-postgres-only.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# PostgreSQL only
2+
3+
The package supports PostgreSQL and no other database. Its value is the guarantees in the README, and
4+
those come from how Postgres behaves: `INSERT ... ON CONFLICT DO NOTHING` waits on an uncommitted
5+
duplicate under READ COMMITTED, and cleanup relies on `FOR UPDATE SKIP LOCKED` and `DELETE ...
6+
RETURNING`. Each extra dialect would need its own code paths, its own guarantees and its own CI matrix.
7+
8+
## Considered Options
9+
10+
- SQLite as well, for tests without a server. The insert syntax matches, but SQLite has one writer at a
11+
time and no row locks, so it would test a different concurrency model from the one in production.
12+
- MySQL. It has no `RETURNING`, rejects `LIMIT` in an `IN` subquery, and its `INSERT IGNORE` turns
13+
unrelated errors into warnings, so both claim and cleanup would need separate implementations.

‎sqlalchemy_inbox/__init__.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
1+
from sqlalchemy_inbox.cleaner import CleanupResult, InboxCleaner
12
from sqlalchemy_inbox.inbox import Inbox, SessionOrConnection
23
from sqlalchemy_inbox.schema import make_inbox_table
34

45

5-
__all__ = ["Inbox", "SessionOrConnection", "make_inbox_table"]
6+
__all__ = ["CleanupResult", "Inbox", "InboxCleaner", "SessionOrConnection", "make_inbox_table"]

‎sqlalchemy_inbox/cleaner.py‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
import contextlib
2+
import dataclasses
3+
import datetime as dt
4+
import logging
5+
from collections.abc import AsyncGenerator
6+
7+
from sqlalchemy import Table, delete, func, select, tuple_
8+
from sqlalchemy.ext.asyncio import AsyncConnection, AsyncEngine
9+
10+
from sqlalchemy_inbox.schema import validate_table_identifiers
11+
12+
13+
_LOGGER = logging.getLogger(__name__)
14+
15+
16+
@dataclasses.dataclass(kw_only=True, frozen=True, slots=True)
17+
class CleanupResult:
18+
cutoff: dt.datetime
19+
deleted_count: int
20+
21+
22+
class InboxCleaner:
23+
"""Deletes markers older than *retention*, one transaction per batch.
24+
25+
Opens its own connections from *engine* but never disposes it: the caller owns the engine.
26+
"""
27+
28+
def __init__(self, *, engine: AsyncEngine, table: Table, retention: dt.timedelta, batch_size: int = 1000) -> None:
29+
if not isinstance(engine, AsyncEngine):
30+
msg = "InboxCleaner requires an sqlalchemy.ext.asyncio.AsyncEngine"
31+
raise TypeError(msg)
32+
validate_table_identifiers(table.name)
33+
if retention <= dt.timedelta(0):
34+
msg = f"retention must be positive (got {retention})"
35+
raise ValueError(msg)
36+
if batch_size < 1:
37+
msg = f"batch_size must be positive (got {batch_size})"
38+
raise ValueError(msg)
39+
self._engine = engine
40+
self._table = table
41+
self._retention = retention
42+
self._batch_size = batch_size
43+
44+
async def cleanup(self) -> CleanupResult:
45+
"""Take one cutoff from the database clock, then delete the markers older than it until a batch deletes nothing.
46+
47+
Each batch commits on its own and skips markers other transactions hold. If a batch fails or the task
48+
is cancelled, that batch rolls back, earlier batches stay deleted, and the error propagates.
49+
"""
50+
async with self._transaction() as connection:
51+
cutoff = (await connection.execute(select(func.now() - self._retention))).scalar_one()
52+
deleted_count = 0
53+
while deleted := await self._delete_batch(cutoff):
54+
deleted_count += deleted
55+
_LOGGER.debug("Inbox cleanup batch deleted %d markers (%d so far)", deleted, deleted_count)
56+
_LOGGER.info("Inbox cleanup deleted %d markers older than %s", deleted_count, cutoff.isoformat())
57+
return CleanupResult(cutoff=cutoff, deleted_count=deleted_count)
58+
59+
async def _delete_batch(self, cutoff: dt.datetime) -> int:
60+
key = tuple_(self._table.c.scope, self._table.c.message_id)
61+
expired = (
62+
select(self._table.c.scope, self._table.c.message_id)
63+
.where(self._table.c.processed_at < cutoff)
64+
.limit(self._batch_size)
65+
.with_for_update(skip_locked=True)
66+
)
67+
async with self._transaction() as connection:
68+
result = await connection.execute(
69+
delete(self._table).where(key.in_(expired)).returning(self._table.c.scope)
70+
)
71+
return len(result.all())
72+
73+
@contextlib.asynccontextmanager
74+
async def _transaction(self) -> AsyncGenerator[AsyncConnection]:
75+
async with self._engine.connect() as connection:
76+
await connection.execution_options(isolation_level="READ COMMITTED")
77+
async with connection.begin():
78+
yield connection

‎sqlalchemy_inbox/inbox.py‎

Lines changed: 1 addition & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,4 @@
1-
import datetime as dt
2-
3-
from sqlalchemy import Table, delete, func, select, tuple_
1+
from sqlalchemy import Table
42
from sqlalchemy.dialects.postgresql import insert as pg_insert
53
from sqlalchemy.ext.asyncio import AsyncConnection, AsyncSession
64

@@ -52,34 +50,6 @@ async def claim(self, *, scope: str, message_id: str) -> bool:
5250
result = await self._session.execute(statement)
5351
return result.first() is not None
5452

55-
async def cleanup_cutoff(self, *, retention: dt.timedelta) -> dt.datetime:
56-
"""Return the database's ``now()`` minus *retention*: take it once per cleanup run."""
57-
if retention <= dt.timedelta(0):
58-
msg = f"retention must be positive (got {retention})"
59-
raise ValueError(msg)
60-
self._require_transaction("cleanup_cutoff")
61-
result = await self._session.execute(select(func.now() - retention))
62-
return result.scalar_one()
63-
64-
async def delete_expired_batch(self, *, cutoff: dt.datetime, limit: int) -> int:
65-
"""Delete up to *limit* markers older than *cutoff*, skipping rows other transactions hold.
66-
67-
Returns the number deleted; a run is done when a batch returns 0.
68-
"""
69-
if limit < 1:
70-
msg = f"limit must be positive (got {limit})"
71-
raise ValueError(msg)
72-
self._require_transaction("delete_expired_batch")
73-
key = tuple_(self._table.c.scope, self._table.c.message_id)
74-
expired = (
75-
select(self._table.c.scope, self._table.c.message_id)
76-
.where(self._table.c.processed_at < cutoff)
77-
.limit(limit)
78-
.with_for_update(skip_locked=True)
79-
)
80-
result = await self._session.execute(delete(self._table).where(key.in_(expired)).returning(self._table.c.scope))
81-
return len(result.all())
82-
8353
def _require_transaction(self, method: str) -> None:
8454
if not self._session.in_transaction():
8555
msg = f"Inbox.{method} requires an open transaction on its session"

0 commit comments

Comments
 (0)