Skip to content

Commit 6b507ce

Browse files
committed
feat(WP-3.1): final transport SPI and the transport conformance kit
- Pull-based SPI frozen (ADR-033): ReceivedMessage gains RenewAsync and DeadLetterAsync, ITransport gains CreateDeadLetterReceiverAsync; capabilities for ordered delivery, native DLQ, native filters, batch size, deduplication and lock duration; typed errors (TransportException with IsTransient/RetryAfter, per-message TransportSendException, DestinationNotFoundException, TopologyValidationException.Problems); topology with ordering, max deliveries and filters (SourceFlowRuntime.Topology()). - CloudEvents SQL subscription filters (CloudEventFilter): routing compiles Only<T>() and Where(...); native translation or client-side evaluation with identical results. - Core: lock renewal while a handler runs, client-side filtering, broker dead-letter ingestor, transport errors classified by transience. - In-memory broker as the reference transport: CloudEvents binary wire with a 10-attribute budget, ordered endpoints, locks, max deliveries and a DLQ, filtered fan-out, batches, provisioning diff, faults. - TCK-01..45 in SourceFlow.Testing with a generated end-to-end domain and the fixture contract (faults, defects); all pass on in-memory. - Tests: UT-CE-01..08, UT-RT-21..22, UT-RX-22..24, TCK-00..45.
1 parent 46ec142 commit 6b507ce

26 files changed

Lines changed: 3470 additions & 276 deletions

‎docs/v3.0.0/07-Cloud-Transports.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,12 @@ v2 put retries, idempotency, dead-lettering, encryption and telemetry *inside* e
88

99
## 2. Transport SPI
1010

11+
> **As built ([ADR-033](ADRs/ADR-033-final-transport-spi-pull-receivers-and-the-in-memory-reference-broker.md)).** The SPI is pull-based: `ITransport` has `SendAsync`, `PublishAsync`, `CreateReceiverAsync`, `CreateDeadLetterReceiverAsync` and `ProvisionAsync`. `ITransportReceiver.ReceiveAsync(max, wait)` feeds the core's receive loop, and `ReceivedMessage` settles with `CompleteAsync`, `AbandonAsync(delay)`, `RenewAsync` or `DeadLetterAsync(reason)`.
12+
>
13+
> `TransportCapabilities` adds `OrderedDelivery`, `NativeDeadLetter`, `SubscriptionFilters`, `MaxBatchSize`, `Deduplication` and `MaxLockDuration`. Errors are typed: `TransportException.IsTransient`, a per-message `TransportSendException`, and `TopologyValidationException.Problems`.
14+
>
15+
> The callback-based sketch below is the original design input.
16+
1117
```csharp
1218
public interface ITransport
1319
{
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# ADR-033: The final transport SPI: pull receivers, CESQL filters and the in-memory reference broker
2+
3+
| | |
4+
|---|---|
5+
| **Status** | Accepted |
6+
| **Date** | 2026-11-11 |
7+
| **Deciders** | v3 architecture review; WP-3.1 implementation |
8+
| **Amends** | [doc 07](../07-Cloud-Transports.md) §2, §7; [ADR-014](ADR-014-transport-spi-and-conformance-kit-parity-required.md) |
9+
10+
## Context
11+
12+
Doc 07 §2 sketched a callback-based SPI with a separate sender and provisioner. The core built in M1 uses a pull-based `ITransport`. Here the receive loop, per-key workers, bounded channels and settlement live in the core (`ReceiveEndpoint`, RX-R1..R6).
13+
14+
WP-3.1 has to freeze the SPI before three provider workstreams start in parallel. It also has to turn the 12-case kit into the planned TCK-01..45.
15+
16+
## Decision
17+
18+
1. **Pull, not callbacks.** `ITransportReceiver.ReceiveAsync(max, wait)` stays.
19+
- Every target broker has a native pull: SQS `ReceiveMessage`, Pub/Sub pull, Service Bus `ReceiveMessagesAsync`.
20+
- Keeping the loop in the core keeps backpressure, per-key ordering and shutdown identical on every broker.
21+
- The SPI stays at five types: `ITransport`, `ITransportReceiver`, `ReceivedMessage`, `TransportCapabilities` and `TransportTopology`.
22+
2. **Settlement is complete.** `ReceivedMessage` has `CompleteAsync`, `AbandonAsync(delay)` (native delay), `RenewAsync` (lock, visibility or lease extension) and `DeadLetterAsync(reason)` (the native backstop).
23+
- A late settlement after a lost lock is ignored.
24+
- The receive endpoint renews at half `ReceiveLockDuration` while a handler runs.
25+
- Disposing a receiver releases what it holds unsettled.
26+
3. **Capabilities drive the core.** `TransportCapabilities` keeps the size, attribute, retention and delay limits. It gains:
27+
- `OrderedDelivery`, `NativeDeadLetter`, `SubscriptionFilters`;
28+
- `MaxBatchSize`, `Deduplication` (with its window), `MaxLockDuration`.
29+
30+
The core uses them to decide:
31+
- client-side filtering when a broker has no native filters (UT-RX-22);
32+
- whether to run the dead-letter ingestor;
33+
- how the TCK judges deduplication.
34+
4. **Errors are typed.**
35+
- `TransportException.IsTransient` (and `RetryAfter` for throttling) drives the error classifier.
36+
- `TransportSendException` reports a partial batch per message: oversized messages are refused before anything is sent for them, and the rest are sent.
37+
- `DestinationNotFoundException` names a missing endpoint or topic.
38+
- `TopologyValidationException.Problems` is the full diff under `Validate`, typed by `TopologyProblemKind`.
39+
5. **The topology carries what brokers need to configure.** `TransportTopology` gains:
40+
- `OrderedEndpoint` and `OrderedTopics`;
41+
- `MaxDeliveries`, the dead-letter threshold: the retry budget (1 + delayed retries) plus two;
42+
- per-topic `Filters`.
43+
44+
The full topology model and naming conventions remain WP-3.5. `SourceFlowRuntime.Topology()` builds this topology for the host and the CLI.
45+
6. **Subscription filters are CloudEvents SQL.**
46+
- `CloudEventFilter` parses and evaluates CESQL 1.0 over the CloudEvents attributes: operators, `LIKE`, `IN`, `EXISTS` and the common functions. Errors never match.
47+
- Routing compiles `Only<T>()` to `type IN (…)` and ANDs it with `Where(…)`.
48+
- Transports translate `ExactTypes` and `TypePrefixes` natively. Anything else is evaluated client-side with the same evaluator, so results are identical (TCK-28).
49+
7. **The in-memory broker is the reference transport.** It sends messages through the real CloudEvents binary mapping with a 10-attribute budget, the strictest broker's, so the `sfext` overflow runs on every test. It also implements:
50+
- ordered endpoints (one in-flight message per key, head-of-line within a key);
51+
- locks with expiry and renewal;
52+
- a maximum delivery count and a dead-letter queue;
53+
- filtered fan-out, batches and partial failures;
54+
- a provisioning diff;
55+
- faults: outage, throttling, credential expiry, duplicates and dropped acks.
56+
57+
`StrictDestinations` gives real-broker semantics for unknown destinations. It is off by default, so tests can send to services that start later.
58+
8. **The TCK runs the real core.**
59+
- `SourceFlow.Testing` carries a small generated domain (`TckModule`). Cases 18, 24, 25 and 45 start real services on the transport under test.
60+
- The fixture contract adds `Faults` (null on real cloud; those cases are skipped, never weakened), `InjectDefectAsync` for the `Validate` cases, `QuietPeriod` and `DelayTolerance`.
61+
- Payload protection cases 41–43 cover the transport side: ciphertext and marker are carried byte for byte and never inspected. Decryption and `DecryptionFailed` parking arrive with WP-5.1.
62+
63+
## Consequences
64+
65+
- Provider packages (WP-3.2..3.4) implement one small SPI, and must pass the same 45 cases as the in-memory broker.
66+
- Every in-memory test now exercises the CloudEvents wire mapping, so envelope regressions surface everywhere, not only in provider suites.
67+
- Brokers that cannot filter natively still filter correctly, at the cost of receiving and acknowledging unwanted events.
68+
- The broker dead-letter queue is a backstop. The ingestor moves its messages to the store, where `dlq replay` re-sends them.

‎docs/v3.0.0/ADRs/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,3 +45,4 @@ Every ADR is **Proposed** until the v3 architecture review. "Traces" lists the [
4545
| [ADR-030](ADR-030-domain-queries-bind-at-startup.md) | Domain queries are checked in two stages: shape at build time, binding at startup | Accepted |
4646
| [ADR-031](ADR-031-cli-runs-store-commands-globally-and-model-commands-in-the-service.md) | The CLI runs store commands as a global tool and model commands inside the service | Accepted |
4747
| [ADR-032](ADR-032-blue-green-read-model-generations-behind-forward-only-aliases.md) | Blue/green read-model generations behind forward-only aliases | Accepted |
48+
| [ADR-033](ADR-033-final-transport-spi-pull-receivers-and-the-in-memory-reference-broker.md) | The final transport SPI: pull receivers, CESQL filters and the in-memory reference broker | Accepted |

‎docs/v3.0.0/implementation/05-Unit-Test-Plan.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -224,6 +224,10 @@ These are additional to the SCK, which the in-memory store also runs.
224224
| UT-RX-14..16 | Shutdown: stop pulling; in-flight complete within timeout; remainder released | RX-R6 | new |
225225
| UT-RX-17..19 | Trace: consumer span linked to the producer `traceparent`; consumed and process-duration metrics | ADR-019 | → `CloudTelemetryTests.ExtractTraceParent_*` |
226226
| UT-RX-20..22 | Tenant restored from the envelope; actor restored; CESQL subscription filter applied client-side when not native | ADR-022, doc 07 | → `SnsMessageFilteringIntegrationTests` (concept) |
227+
| UT-RX-23 | Transport errors classified by `IsTransient` (outage and throttling transient; missing destination and oversized body permanent) | ADR-033 | → `AwsServiceThrottlingAndFailureTests` (concept) |
228+
| UT-RX-24 | A handler longer than the lock renews it, so the message is not redelivered | doc 07 §4, ADR-033 | new |
229+
| UT-RT-21..22 | Subscription filter from routing: `Only<T>()` becomes `type IN (…)` and is ANDed with `Where(…)`; a malformed filter fails at configuration | doc 07 §7 | new |
230+
| UT-CE-01..08 | CloudEvents SQL filter: operators and precedence, `LIKE` with escapes, `IN`/`EXISTS`, errors never match, functions, parse errors with positions, native translation (`ExactTypes`, `TypePrefixes`), envelope attributes | doc 07 §7, ADR-033 | new |
227231
| UT-DLQ-01..10 | Dead-letter manager: list and filter (consumer, reason, type, date), show, replay keeps the message id (the inbox still protects), discard with an audited reason, purge older than a cutoff, metrics | GAP-11 | → `InMemoryDeadLetterStoreTests` (11), `AwsDeadLetterQueuePropertyTests` (replay concepts) |
228232

229233
### 3.7 Subscriptions, projections, process managers (UT-SUB, UT-PRJ, UT-PM, WP-1.9)

‎docs/v3.0.0/implementation/06-Integration-and-E2E-Test-Plan.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,14 @@ Isolation: every test uses a unique prefix (`t{testId}{random}`) for schemas, qu
6464

6565
## 3. Transport Conformance Kit (TCK)
6666

67-
`TransportConformanceTests` lives in `SourceFlow.Testing` and runs against in-memory, AWS (LocalStack, then real), Google Cloud (emulator, then real) and Azure (emulator, then real). Cases marked † may be `RealCloudOnly` with a written reason when an emulator diverges (as was already the case for two Azure session tests on `azure-cloud`).
67+
`TransportConformanceTests` lives in `SourceFlow.Testing` and runs against in-memory, AWS (LocalStack, then real), Google Cloud (emulator, then real) and Azure (emulator, then real).
68+
69+
As built ([ADR-033](../ADRs/ADR-033-final-transport-spi-pull-receivers-and-the-in-memory-reference-broker.md)):
70+
- TCK-18, 24, 25 and 45 start real services on the transport under test, using the kit's generated `TckModule`.
71+
- TCK-32..34 need fault injection (`ITransportFixture.Faults`) and are skipped on real cloud.
72+
- TCK-37..40 use `InjectDefectAsync`.
73+
- TCK-41..43 cover the transport side of payload protection; decryption and `DecryptionFailed` parking come with WP-5.1.
74+
- TCK-00 checks the case list. Cases marked † may be `RealCloudOnly` with a written reason when an emulator diverges (as was already the case for two Azure session tests on `azure-cloud`).
6875

6976
| ID | Case | v2 scenario sources |
7077
|----|------|---------------------|

‎docs/v3.0.0/implementation/STATUS.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,14 +4,14 @@ Updated in the PR that completes a work package (CONTRIBUTING.md). Status values
44

55
## Progress summary
66

7-
Updated 2026-11-11. **34 of 63 work packages are done (54%)**, 0 in progress and 29 not started. By estimated effort, 66 of 138.5 engineer-weeks are done (48%).
7+
Updated 2026-11-11. **35 of 63 work packages are done (56%)**, 0 in progress and 28 not started. By estimated effort, 69 of 138.5 engineer-weeks are done (50%).
88

99
| Milestone | Done | In progress | Todo | Effort done (ew) |
1010
|-----------|------|-------------|------|------------------|
1111
| M0 Foundations | 5 | 0 | 0 | 4 of 4 |
1212
| M1 Core | 15 | 0 | 0 | 37 of 37 |
1313
| M2 PostgreSQL | 11 | 0 | 0 | 19 of 19 |
14-
| M3 Cloud transports | 0 | 0 | 7 | 0 of 19 |
14+
| M3 Cloud transports | 1 | 0 | 6 | 3 of 19 |
1515
| M4 Tooling and operations | 2 | 0 | 5 | 5.5 of 16.5 |
1616
| M5 Security and migration | 1 | 0 | 4 | 0.5 of 11 |
1717
| Track S: Workflow DSL and Studio | 0 | 0 | 13 | 0 of 32 |
@@ -29,10 +29,11 @@ Completed in this round:
2929
- WP-5.3: redaction wiring (taxonomy, Erasing/HMAC redactors, `LogPayloads`).
3030
- WP-4.1: the `dotnet sourceflow` CLI (ADR-031).
3131
- WP-2.7: blue/green read-model generations behind forward-only aliases, including EF Core schema per generation (ADR-032, E2E-08).
32+
- WP-3.1: the final transport SPI and the 45-case transport conformance kit, passed by the in-memory reference broker (ADR-033).
3233

33-
Nothing is in progress. M2 (PostgreSQL) is complete.
34+
Nothing is in progress. M2 (PostgreSQL) is complete; M3 has started (WP-3.1 done, the provider packages are next).
3435

35-
Not started: the v2 cloud retirement (3.R), every cloud transport (M3), health and dashboards, SQL Server, event partitioning, chaos and load testing, payload protection, crypto-shredding (redaction is done), the v2 migration tool, and the whole Studio track.
36+
Not started: the cloud transport packages and provisioning (WP-3.2..3.6), the v2 cloud retirement (3.R), health and dashboards, SQL Server, event partitioning, chaos and load testing, payload protection, crypto-shredding (redaction is done), the v2 migration tool, and the whole Studio track.
3637

3738
| WP | Title | Estimate (ew) | Status | Notes |
3839
|----|-------|---------------|--------|-------|
@@ -67,7 +68,7 @@ Not started: the v2 cloud retirement (3.R), every cloud transport (M3), health a
6768
| WP-2.9 | Storage conformance kit complete | 2 | done | SCK-01..58 in SourceFlow.Testing, run on in-memory, PostgreSQL and PostgreSQL with RLS. New in this package: SCK-23 (in-flight writes invisible), SCK-37 (batched retention via IStoreRetention, plus the host retention loop), SCK-54 (CsCheck property: snapshot + tail = full fold), PostgreSQL SCK-28 (LISTEN loss and reconnect), and SCK-L01 CsCheck linearisability of appends and the outbox. SCK-36 is covered by SCK-34. SCK-20 (one round trip) moves to WP-2.10, and SCK-65..72 (partitioned events) to WP-4.5 |
6869
| WP-2.10 | Performance baseline | 1 | done | benchmarks/SourceFlow.Benchmarks (BenchmarkDotNet PERF-01..04, 06; measurement harness PERF-05, 08, 10); baselines in benchmarks/baselines and the report in implementation/perf-baseline.md. One-round-trip PostgreSQL commit (SCK-20), batch-aware document projections and adaptive subscription batches. Deviations logged with owners (PERF-01/02 allocations, PERF-10 borderline). PERF-07 waits for M3 and PERF-09 for WP-4.7 |
6970
| WP-2.R | Retire v2 EF stores | 0.5 | done | Every `SourceFlow.Stores.EntityFramework.Tests` row is satisfied (SCK on three adapters, UT-EF-01..10). `src/SourceFlow.Stores.EntityFramework` and its tests are deleted |
70-
| WP-3.1 | Transport SPI final + Transport Conformance Kit | 3 | todo | |
71+
| WP-3.1 | Transport SPI final + Transport Conformance Kit | 3 | done | Final pull-based SPI (ADR-033): `RenewAsync`, `DeadLetterAsync`, `CreateDeadLetterReceiverAsync`, capabilities (ordered delivery, native DLQ, native filters, batch size, deduplication, lock), typed errors (`TransportException.IsTransient`, per-message `TransportSendException`, `DestinationNotFoundException`, `TopologyValidationException.Problems`), topology with ordering, max deliveries and filters (`SourceFlowRuntime.Topology()`). CloudEvents SQL filters (`CloudEventFilter`; `Only<T>()` and `Where(…)` in routing; native translation or client-side evaluation). Core: lock renewal while handling, client-side filtering, broker dead-letter ingestor. In-memory broker as the reference: CloudEvents binary wire with a 10-attribute budget, ordered endpoints, locks, max deliveries and DLQ, filtered fan-out, batches, provisioning diff, faults. TCK-01..45 in `SourceFlow.Testing` with a generated end-to-end domain and the fixture contract (faults, defects); all 45 pass on in-memory. UT-CE-01..08, UT-RT-21..22, UT-RX-22..24 |
7172
| WP-3.2 | `SourceFlow.Cloud.Aws` | 4 | todo | |
7273
| WP-3.3 | `SourceFlow.Cloud.Gcp` | 3.5 | todo | |
7374
| WP-3.4 | `SourceFlow.Cloud.Azure` | 4 | todo | |

‎src/Core/SourceFlow.Net/Diagnostics/Log.cs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,9 @@ internal static partial class Log
4747
[LoggerMessage(1203, LogLevel.Error, "Receive loop for {Endpoint} failed; continuing")]
4848
public static partial void ReceiveLoopFailed(ILogger logger, Exception exception, string endpoint);
4949

50+
[LoggerMessage(1210, LogLevel.Warning, "Message {MessageId} was dead-lettered by the broker on {Endpoint} ({Reason}); parked for replay")]
51+
public static partial void BrokerDeadLetterIngested(ILogger logger, string messageId, string endpoint, string reason);
52+
5053
[LoggerMessage(1301, LogLevel.Error, "Subscription {Subscription} partition {Partition} failed on event {EventId}; {Policy}")]
5154
public static partial void SubscriptionFailed(ILogger logger, Exception exception, string subscription, int partition, string eventId, string policy);
5255

‎src/Core/SourceFlow.Net/Hosting/SourceFlowHost.cs‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ public sealed class SourceFlowHost : IHostedService, IHealthCheck, IAsyncDisposa
2727
private volatile bool _ready;
2828

2929
private readonly Query.QueryService? _queries;
30+
private DeadLetterIngestor? _ingestor;
3031

3132
internal SourceFlowHost(SourceFlowRuntime runtime, OutboxRelay relay, ReceiveEndpoint receiver, SubscriptionEngine subscriptions, HostSettings settings, Query.QueryService? queries = null)
3233
{
@@ -57,12 +58,7 @@ public ValueTask<bool> SubscriptionsCaughtUpAsync(CancellationToken cancellation
5758
/// <inheritdoc />
5859
public async Task StartAsync(CancellationToken cancellationToken)
5960
{
60-
var topology = new TransportTopology(
61-
_runtime.Routing.Endpoint,
62-
[.. _runtime.Routing.PublishedTopics],
63-
[.. _runtime.Routing.Subscriptions.Select(s => s.Topic)],
64-
[.. _runtime.Routing.Commands.Values.Select(c => c.Endpoint).Distinct(StringComparer.Ordinal)]);
65-
await _runtime.Transport.ProvisionAsync(topology, _settings.Provisioning, cancellationToken).ConfigureAwait(false);
61+
await _runtime.Transport.ProvisionAsync(_runtime.Topology(), _settings.Provisioning, cancellationToken).ConfigureAwait(false);
6662

6763
await RegisterSchemasAsync(cancellationToken).ConfigureAwait(false);
6864
CheckDomainQueries();
@@ -75,6 +71,11 @@ [.. _runtime.Routing.Subscriptions.Select(s => s.Topic)],
7571
{
7672
await _receiver.StartAsync(stopping).ConfigureAwait(false);
7773
_loops.Add(Task.Run(() => _receiver.RunAsync(stopping), CancellationToken.None));
74+
if (_runtime.Options.IngestBrokerDeadLetters && _runtime.Transport.Capabilities.NativeDeadLetter)
75+
{
76+
_ingestor = new DeadLetterIngestor(_runtime);
77+
_loops.Add(Task.Run(() => LoopAsync(ct => _ingestor.RunOnceAsync(ct), stopping), CancellationToken.None));
78+
}
7879
}
7980

8081
if (_settings.Roles.HasFlag(SourceFlowRoles.Relay))
@@ -159,6 +160,10 @@ public async ValueTask DisposeAsync()
159160
{
160161
_stopping?.Dispose();
161162
await _receiver.DisposeAsync().ConfigureAwait(false);
163+
if (_ingestor is not null)
164+
{
165+
await _ingestor.DisposeAsync().ConfigureAwait(false);
166+
}
162167
}
163168

164169
// Batched retention cleanup (doc 05 §9): cutoffs on the business clock, the interval on the system clock.

‎src/Core/SourceFlow.Net/Pipeline/Middleware.cs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -177,6 +177,10 @@ public ErrorClassifier()
177177
Treat<ArgumentException>(HandlingOutcome.Permanent);
178178
Treat<InvalidCastException>(HandlingOutcome.Permanent);
179179
Treat<PermanentException>(HandlingOutcome.Permanent);
180+
181+
// Broker errors say whether they are transient (outage, throttling) or not (missing destination, oversized body).
182+
_rules.Insert(0, (e => e is Transport.TransportException { IsTransient: false }, HandlingOutcome.Permanent));
183+
_rules.Insert(0, (e => e is Transport.TransportException { IsTransient: true }, HandlingOutcome.Transient));
180184
}
181185

182186
/// <summary>Classifies <typeparamref name="TException"/> (and subclasses); later rules win.</summary>

0 commit comments

Comments
 (0)