Skip to content

Commit 0a71adf

Browse files
committed
feat(WP-1.12): schema catalog, registration, compatibility and AsyncAPI export
- JsonSchemaGenerator: JSON Schema 2020-12 from the serializer's source-generated JsonTypeInfo (AOT-safe), with x-sourceflow-queryable, -personal-data, -renamed-from and -tag extensions; canonical text and SHA-256 fingerprints (ADR-029). - SchemaCatalog: every message type and version (incl. replies), read-model document and process state; lock file read/write. - ISchemaStore (optional port; in-memory and the PostgreSQL schemas table): startup registers the catalog and a changed published (type, version) fails startup (SchemaRegistration: Enforce/Warn/Off). - SchemaCompatibility rules SFC01-SFC10 and the AsyncAPI 3.0 exporter. - Tests UT-CAT-01..18 plus a PostgreSQL registry test; the PostgreSQL transfer E2E now waits for completion before checking the outbox.
1 parent 4c18c5e commit 0a71adf

21 files changed

Lines changed: 1533 additions & 6 deletions

File tree

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# ADR-029: The schema catalog is derived at startup from the serializer's source-generated metadata
2+
3+
| | |
4+
|---|---|
5+
| **Status** | Accepted |
6+
| **Date** | 2026-11-11 |
7+
| **Deciders** | v3 architecture review; WP-1.12 implementation |
8+
| **Amends** | [doc 14](../14-Query-Side-and-SFQL.md) §2.1 and §2.2; WBS WP-1.12 ("a generator extension that emits a JSON Schema") |
9+
10+
## Context
11+
12+
Doc 14 §2.1 says the source generator emits a JSON Schema 2020-12 document for every `[MessageType]`. The WBS repeats this for WP-1.12. Building the catalog showed three problems with emitting schemas from the SourceFlow generator:
13+
14+
- **The generator can't see the wire contract.** System.Text.Json decides member names, ignored members, converters, number handling and nullability. The `JsonSerializerContext` and its options control that, along with any custom converters. A Roslyn generator in `netstandard2.0` would have to re-implement those rules (ADR-004, DC-1), and every mismatch would produce a schema that doesn't describe what is actually stored or sent.
15+
- **One contract would have two sources.** The serializer already has source-generated, AOT-safe metadata for every message, document and state type. That metadata is `JsonTypeInfo`, the same one SFQL's query catalog uses (doc 14 §5). A second description written by the SourceFlow generator could drift from it.
16+
- **Decimals aren't strings.** Doc 14 §2.1 says decimals are serialised as strings with `format: decimal`. In fact System.Text.Json writes them as JSON numbers, and the PostgreSQL compiler reads them with `::numeric`, which is exact.
17+
18+
## Decision
19+
20+
1. **Schemas come from `JsonTypeInfo` at runtime.** `SourceFlow.Catalog.JsonSchemaGenerator` walks the type's `JsonTypeInfo` and those of its members. It covers object properties, enumerables, dictionaries, nullable value types and `JsonPropertyInfo.IsGetNullable`/`IsRequired` for `required`. It writes JSON Schema 2020-12. Member attributes become the `x-sourceflow-queryable`, `x-sourceflow-personal-data`, `x-sourceflow-renamed-from` and `x-sourceflow-tag` extensions.
21+
- It reads attributes through `JsonPropertyInfo.AttributeProvider`, which source generation populates, so no reflection over members is needed. The code stays AOT-compatible.
22+
2. **Value formats follow the serializer:**
23+
- decimals are `{"type": "number", "format": "decimal"}`;
24+
- strongly typed ids follow their `[StronglyTypedId]` backing (uuid, string or int64);
25+
- enums are `integer` or `string`, decided by serializing a value through the enum's own `JsonTypeInfo`, so a configured string converter is respected.
26+
3. **`SchemaCatalog.Build(RuntimeModel)` describes the whole service:**
27+
- every registered message type and version, including `CommandRejected<T>` replies;
28+
- every document projection's document (`readmodel`: projection name and version);
29+
- every process manager's state (`process-state`, version 1).
30+
4. **Fingerprints exclude the service.** The fingerprint is the SHA-256 of the canonical schema text (keys sorted ordinally, no whitespace). `x-sourceflow` holds the type, version, kind and `upcastsTo`, but not the service or the owning aggregate. Several services that share a contract (an integration event, or a command sent to a remote owner) therefore register the same fingerprint.
31+
5. **Registration and immutability follow doc 14 §2.2.** Host startup registers the catalog through the optional `ISchemaStore` port, implemented in memory and by PostgreSQL's `schemas` table. A different fingerprint for a registered `(type, version)` fails startup with `SchemaConflictException`. `SourceFlowOptions.SchemaRegistration` can set this to `Warn` or `Off`.
32+
6. **The SourceFlow generator emits no schemas.** The GEN-41..46 cases planned for the generator are covered by UT-CAT-01..03 against the runtime generator.
33+
34+
## Consequences
35+
36+
- The catalog describes exactly what the serializer writes, including custom naming policies and converters, and costs one walk over the metadata at startup.
37+
- Tools that need schemas without running the service get them through a host:
38+
- the CLI's `schema export` and `schema check`;
39+
- the lock file written by `SchemaCatalog.ToLockFile()`.
40+
- Doc 14 §2.1 is amended: decimals are numbers with `format: decimal`.

‎docs/v3.0.0/ADRs/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,3 +41,4 @@ Every ADR is **Proposed** until the v3 architecture review. "Traces" lists the [
4141
| [ADR-026](ADR-026-per-service-query-api-with-sql-syntax-and-injection-proof-compilation.md) | Per-service query API with a SQL syntax and injection-proof compilation | Proposed |
4242
| [ADR-027](ADR-027-single-core-package-with-extensibility-spis-and-categorised-adapters.md) | Single core package with extensibility SPIs, and categorised adapter packages | Accepted |
4343
| [ADR-028](ADR-028-postgresql-payload-columns-and-adapter-refinements.md) | PostgreSQL payload columns are `json`, plus adapter refinements | Accepted |
44+
| [ADR-029](ADR-029-schema-catalog-derived-from-serializer-metadata.md) | The schema catalog is derived at startup from the serializer's source-generated metadata | Accepted |

‎docs/v3.0.0/implementation/STATUS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ Updated in the PR that completes a work package (CONTRIBUTING.md). Status values
2020
| WP-1.9 | Subscriptions, projections, process managers, event handlers | 4 | done | SubscriptionEngine (fenced, partitioned, adaptive batches, park policy with per-event isolation), projections, process managers (incl. `CommandRejected<T>` replies: `sf-reply-to` header for local owners and `ReplyOnReject()` routes, reply committed with the rejection, ADR-018), event handlers; `INodeRegistry` heartbeat + fair-share rebalancer (one lease released per pass, graceful hand-back on stop). UT-SUB-01/05/06/08/09/10/12/17, UT-PRJ-01, UT-PM-01/10..13, E2E-09..15. Repartition and blue/green rebuild (UT-SUB-21..28) and the `projection skip` Stop policy come with the CLI (WP-4.1) |
2121
| WP-1.10 | Hosting, roles, tenancy context | 1.5 | done | AddSourceFlow, SourceFlowBuilder Use* slots, roles, SourceFlowHost lifecycle, health check |
2222
| WP-1.11 | `SourceFlow.Testing` | 2.5 | done | SourceFlowTestHost, AggregateSpec, ProcessManagerSpec, ProjectionSpec, approvals; UT-TST |
23-
| WP-1.12 | Schema catalog | 2 | todo | |
23+
| WP-1.12 | Schema catalog | 2 | done | JSON Schema 2020-12 per message type and version, read-model document and process state, derived from the serializer's JsonTypeInfo with `x-sourceflow-*` extensions (ADR-029); canonical text + SHA-256 fingerprints; `ISchemaStore` (in-memory, PostgreSQL `schemas`) registered at startup, a changed published `(type, version)` fails startup (`SchemaRegistration` = Enforce/Warn/Off); lock file; `SchemaCompatibility` rules SFC01–SFC10; AsyncAPI 3.0 exporter. UT-CAT-01..18 |
2424
| WP-1.13 | SFQL front end (pipeline + SQL syntax) + in-memory interpreter + domain queries | 5.5 | in progress | Done: request guard, lexer, both parsers into one AST (UT-QL-01..13, 61..65, 69..77); catalog from the runtime model (Q-B1), binder with per-version renames, typing, output naming, G-6 tie-break, G-10/G-11/G-12 (SFQ0201–0206; UT-QL-21..33, 79, 88); policy: permissions, PII keys, cost guard, join and offset rules, limits (SFQ0301, 0401, 0402, 0405, 0406, 0409, 0501; UT-QL-34..39, 80, 89); in-memory interpreter (SQL three-valued logic, hash joins, grouping, subqueries, aggregate as-of folds, schema rows) and `IQueryService` (typed parameters SFQ0202/0203, PII redaction, tenancy, keyset continuation tokens SFQ0601, `at_least` consistency waits SFQ0602, plan cache, `IQueryEngine` SPI for provider compilers; UT-QL-41..60). PostgreSQL compiler: WP-2.8. Next: the typed builder and `[DomainQuery]` |
2525
| WP-1.14 | Bank sample (ported v2 E2E domain) | 1.5 | done | Bank.Contracts/Accounts/Transfers/Notifications + E2E-01..07, 11, 13, 14, 17..20 green; Bank.Api host (commands, demo query endpoints), Studio demo console, docker-compose demo and ghcr.io images (`v3-images` workflow) |
2626
| WP-1.R | Retire v2 core | 0.5 | todo | |
Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
using System.Text;
2+
using System.Text.Json.Nodes;
3+
using SourceFlow.Messaging;
4+
using SourceFlow.Runtime;
5+
6+
namespace SourceFlow.Catalog;
7+
8+
/// <summary>
9+
/// Exports a service's messaging contract as an AsyncAPI 3.0 document (doc 14 §2.2): the service endpoint receives the
10+
/// commands it owns, each published topic carries its contracts, and every message has its JSON Schema payload and the
11+
/// CloudEvents headers. Tools that read AsyncAPI (for example EventCatalog) can publish it as a browsable catalog.
12+
/// </summary>
13+
public static class AsyncApiExporter
14+
{
15+
/// <summary>The AsyncAPI version written.</summary>
16+
public const string AsyncApiVersion = "3.0.0";
17+
18+
/// <summary>Writes the AsyncAPI document of a service.</summary>
19+
/// <param name="model">The runtime model.</param>
20+
/// <param name="routing">The routing table (endpoint and publications).</param>
21+
/// <param name="catalog">The schema catalog (payloads).</param>
22+
/// <param name="serviceName">The service name (document title).</param>
23+
/// <param name="version">The document version (for example the service's release).</param>
24+
public static string Export(RuntimeModel model, RoutingTable routing, SchemaCatalog catalog, string serviceName, string version = "1.0.0")
25+
{
26+
ArgumentNullException.ThrowIfNull(model);
27+
ArgumentNullException.ThrowIfNull(routing);
28+
ArgumentNullException.ThrowIfNull(catalog);
29+
ArgumentException.ThrowIfNullOrWhiteSpace(serviceName);
30+
31+
var messages = new JsonObject();
32+
var channels = new JsonObject();
33+
var operations = new JsonObject();
34+
35+
// The service endpoint: the commands this service owns (and its timeouts and replies).
36+
var endpointMessages = new JsonObject();
37+
foreach (MessageTypeInfo type in model.Registry.All.Where(t => t.Kind is MessageKind.Command && model.CommandOwners.ContainsKey(t.ClrType)))
38+
{
39+
endpointMessages[MessageId(type.Name, type.Version)] = Reference("#/components/messages/" + MessageId(type.Name, type.Version));
40+
AddMessage(messages, catalog, type.Name, type.Version);
41+
}
42+
43+
string endpointId = ChannelId(routing.Endpoint);
44+
channels[endpointId] = new JsonObject
45+
{
46+
["address"] = routing.Endpoint,
47+
["description"] = $"The {serviceName} service endpoint: commands it owns.",
48+
["messages"] = endpointMessages,
49+
};
50+
operations["receive-" + endpointId] = new JsonObject
51+
{
52+
["action"] = "receive",
53+
["channel"] = Reference("#/channels/" + endpointId),
54+
["messages"] = new JsonArray([.. endpointMessages.Select(m => (JsonNode)Reference($"#/channels/{endpointId}/messages/{m.Key}"))]),
55+
};
56+
57+
// Published topics: the contracts mapped from domain events (integration events, or events published as is).
58+
foreach (IGrouping<string, PublicationRoute> topic in routing.Publications.Values.SelectMany(r => r).GroupBy(r => r.Destination, StringComparer.Ordinal).OrderBy(g => g.Key, StringComparer.Ordinal))
59+
{
60+
var topicMessages = new JsonObject();
61+
foreach (Type contract in topic.Select(r => r.ContractType).Distinct())
62+
{
63+
if (model.Registry.TryGet(contract, out MessageTypeInfo? type))
64+
{
65+
topicMessages[MessageId(type.Name, type.Version)] = Reference("#/components/messages/" + MessageId(type.Name, type.Version));
66+
AddMessage(messages, catalog, type.Name, type.Version);
67+
}
68+
}
69+
70+
string id = ChannelId(topic.Key);
71+
channels[id] = new JsonObject { ["address"] = topic.Key, ["messages"] = topicMessages };
72+
operations["publish-" + id] = new JsonObject
73+
{
74+
["action"] = "send",
75+
["channel"] = Reference("#/channels/" + id),
76+
["messages"] = new JsonArray([.. topicMessages.Select(m => (JsonNode)Reference($"#/channels/{id}/messages/{m.Key}"))]),
77+
};
78+
}
79+
80+
var document = new JsonObject
81+
{
82+
["asyncapi"] = AsyncApiVersion,
83+
["info"] = new JsonObject { ["title"] = serviceName, ["version"] = version },
84+
["defaultContentType"] = MessageEnvelope.JsonContentType,
85+
["channels"] = channels,
86+
["operations"] = operations,
87+
["components"] = new JsonObject { ["messages"] = messages, ["schemas"] = new JsonObject { ["CloudEventsHeaders"] = CloudEventsHeaders() } },
88+
};
89+
return document.ToJsonString(new System.Text.Json.JsonSerializerOptions { WriteIndented = true, TypeInfoResolver = System.Text.Json.Serialization.Metadata.JsonTypeInfoResolver.Combine() });
90+
}
91+
92+
private static void AddMessage(JsonObject messages, SchemaCatalog catalog, string type, int version)
93+
{
94+
string id = MessageId(type, version);
95+
if (messages.ContainsKey(id) || !catalog.TryGet(type, version, out SchemaEntry? entry))
96+
{
97+
return;
98+
}
99+
100+
messages[id] = new JsonObject
101+
{
102+
["name"] = type,
103+
["title"] = $"{type} v{version}",
104+
["contentType"] = MessageEnvelope.JsonContentType,
105+
["headers"] = Reference("#/components/schemas/CloudEventsHeaders"),
106+
["payload"] = entry.Schema.DeepClone(),
107+
["x-sourceflow"] = new JsonObject { ["type"] = type, ["version"] = version, ["kind"] = SchemaEntry.KindName(entry.Kind) },
108+
};
109+
}
110+
111+
// CloudEvents attributes in binary mode (doc 04 §1).
112+
private static JsonObject CloudEventsHeaders()
113+
{
114+
static JsonObject Text(string description) => new() { ["type"] = "string", ["description"] = description };
115+
return new JsonObject
116+
{
117+
["type"] = "object",
118+
["properties"] = new JsonObject
119+
{
120+
["ce_specversion"] = Text("CloudEvents version (1.0)."),
121+
["ce_id"] = Text("Message id (UUIDv7)."),
122+
["ce_type"] = Text("Logical message type, for example bank.account.opened."),
123+
["ce_source"] = Text("Producing service."),
124+
["ce_time"] = Text("Creation time (RFC 3339)."),
125+
["ce_subject"] = Text("Stream id, when the message concerns one aggregate."),
126+
["ce_sftypeversion"] = Text("Message type version."),
127+
["ce_sfcorrelationid"] = Text("Correlation id."),
128+
["ce_sfcausationid"] = Text("Causation id."),
129+
["ce_sftenant"] = Text("Tenant id."),
130+
["traceparent"] = Text("W3C trace context."),
131+
},
132+
["required"] = new JsonArray("ce_specversion", "ce_id", "ce_type", "ce_source"),
133+
};
134+
}
135+
136+
private static JsonObject Reference(string pointer)
137+
{
138+
return new JsonObject { ["$ref"] = pointer };
139+
}
140+
141+
private static string MessageId(string type, int version)
142+
{
143+
return ChannelId(type) + "_v" + version.ToString(System.Globalization.CultureInfo.InvariantCulture);
144+
}
145+
146+
// AsyncAPI component keys: ^[a-zA-Z0-9\.\-_]+$.
147+
private static string ChannelId(string name)
148+
{
149+
var id = new StringBuilder(name.Length);
150+
foreach (char c in name)
151+
{
152+
id.Append(char.IsAsciiLetterOrDigit(c) || c is '.' or '-' or '_' ? c : '_');
153+
}
154+
155+
return id.ToString();
156+
}
157+
}

0 commit comments

Comments
 (0)