Skip to content

Commit 08504a6

Browse files
committed
feat(WP-2.8): PostgreSQL index sync for [Queryable] fields
- PostgreSqlIndexSync plans and applies one partial expression index per [Queryable] field: over events per event type and over documents per read-model collection, using the compiler's exact expression so the planner matches it; deterministic sfq_ names, stale ones dropped, CONCURRENTLY by default. - Bank sample: AccountSummary.Status and MoneyDeposited.AccountId are [Queryable]. - QRY-45: sync is idempotent and the planner uses the index.
1 parent ad6edba commit 08504a6

7 files changed

Lines changed: 191 additions & 11 deletions

File tree

‎docs/v3.0.0/implementation/STATUS.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,13 @@ Updated in the PR that completes a work package (CONTRIBUTING.md). Status values
44

55
## Progress summary
66

7-
Updated 2026-11-11. **28 of 63 work packages are done (44%)**, 2 in progress and 33 not started. By estimated effort, 56 of 138.5 engineer-weeks are done (40%).
7+
Updated 2026-11-11. **29 of 63 work packages are done (46%)**, 1 in progress and 33 not started. By estimated effort, 59.5 of 138.5 engineer-weeks are done (43%).
88

99
| Milestone | Done | In progress | Todo | Effort done (ew) |
1010
|-----------|------|-------------|------|------------------|
1111
| M0 Foundations | 5 | 0 | 0 | 4 of 4 |
1212
| M1 Core | 14 | 0 | 1 | 36.5 of 37 |
13-
| M2 PostgreSQL | 8 | 2 | 1 | 13 of 19 |
13+
| M2 PostgreSQL | 9 | 1 | 1 | 16.5 of 19 |
1414
| M3 Cloud transports | 0 | 0 | 7 | 0 of 19 |
1515
| M4 Tooling and operations | 1 | 0 | 6 | 2.5 of 16.5 |
1616
| M5 Security and migration | 0 | 0 | 5 | 0 of 11 |
@@ -24,11 +24,11 @@ Completed in this round:
2424
- WP-1.13: named domain queries (ADR-030), the typed builder, and analyzers SF0015, SF0016 and SF0018.
2525
- WP-1.14: the Bank.Api host, the Studio demo console, the docker-compose demo, and images on ghcr.io.
2626
- WP-4.2: the production query API and domain queries over HTTP.
27+
- WP-2.8: PostgreSQL index sync for `[Queryable]` fields.
2728

2829
Still in progress:
2930

3031
- WP-2.7 (`SourceFlow.Storage.EntityFrameworkCore` read models): rebuilding a projection into a new schema, which waits for the blue/green rebuild tooling.
31-
- WP-2.8 (SFQL on PostgreSQL): index sync (`db index sync`).
3232

3333
Not started: the v2 retirements (1.R, 2.R, 3.R), every cloud transport (M3), the CLI, health and dashboards, SQL Server, event partitioning, chaos and load testing, payload protection, crypto-shredding, the v2 migration tool, and the whole Studio track.
3434

@@ -61,7 +61,7 @@ Not started: the v2 retirements (1.R, 2.R, 3.R), every cloud transport (M3), the
6161
| WP-2.5 | Snapshots and aggregate cache | 1.5 | done | Background snapshot writer (bounded, drop-oldest, drained on shutdown), MemoryCache aggregate cache (ReadTail, Optimistic with rejection confirmation and no shortcut for If-Match), cache and snapshot metrics; UT-SNAP-01..10, SCK-51..53/55 on in-memory and PostgreSQL (with and without RLS), parallel deposits on PostgreSQL in both cache modes. PERF-04 moves to the WP-2.10 baseline |
6262
| WP-2.6 | Tenancy with row-level security | 1 | done | V002 tenant_isolation policies; TenantIsolation.RowLevelSecurity enables and forces RLS (migrator applies and verifies the state); session-scoped set_config; system (BYPASSRLS) data source for the global log; tenant-aware buckets. The whole SCK passes under RLS through a DML-only role; SCK-59/60/M07, Bank transfer E2E under RLS. Per-tenant metric tags (bounded top-N) move to WP-4.3 |
6363
| WP-2.7 | `SourceFlow.Storage.EntityFrameworkCore` read models | 2 | in progress | Core session-projection SPI (ISessionProjection, IProjectionUnitOfWork, IStorageSession.FlushAsync for ES-R1), generator support (GEN-41/42), IEfProjection, EfSessionContextFactory on the session connection and transaction, UseEntityFrameworkProjections, IdempotentUpsert; UT-EF-01..04, SCK-61..64 inline and async on PostgreSQL. Rebuild with schema-per-generation waits for the blue/green rebuild tooling (doc 06 §6) |
64-
| WP-2.8 | SFQL on PostgreSQL (incl. joins and grouping) + index sync + domain queries | 3.5 | in progress | PostgreSqlQueryEngine (`IQueryEngine`): one parameterised statement per page (every request value a typed `$n` parameter, I-3), keyset predicate and ORDER BY outside a derived table, `COLLATE "C"` text ordering, horizon predicate, per-version rename CASE, tenant predicates (G-9), `NULLIF` division, read-only transaction with `statement_timeout` (`QueryTimeout`, default 5 s) and the RLS tenant setting; aggregate and schema sources stay in the interpreter. Cross-provider equivalence with the interpreter on the same data (QRY-01..20, incl. joins, grouping/having, distinct, subqueries, keyset pages interchangeable between engines), golden SQL (QRY-42/43), injection corpus (QRY-44), no-literal CsCheck property (QRY-60). Catalog now types enums by their JSON (number or string). Domain queries run through the engine (ExecuteAsync and `/queries`). `/query/explain` shows the parameterised SQL. Next: index sync (`db index sync`) |
64+
| WP-2.8 | SFQL on PostgreSQL (incl. joins and grouping) + index sync + domain queries | 3.5 | done | PostgreSqlQueryEngine (`IQueryEngine`): one parameterised statement per page (every request value a typed `$n` parameter, I-3), keyset predicate and ORDER BY outside a derived table, `COLLATE "C"` text ordering, horizon predicate, per-version rename CASE, tenant predicates (G-9), `NULLIF` division, read-only transaction with `statement_timeout` (`QueryTimeout`, default 5 s) and the RLS tenant setting; aggregate and schema sources stay in the interpreter. Cross-provider equivalence with the interpreter on the same data (QRY-01..20, incl. joins, grouping/having, distinct, subqueries, keyset pages interchangeable between engines), golden SQL (QRY-42/43), injection corpus (QRY-44), no-literal CsCheck property (QRY-60). Catalog now types enums by their JSON (number or string). Domain queries run through the engine (ExecuteAsync and `/queries`). `/query/explain` shows the parameterised SQL. `PostgreSqlIndexSync` (plan/apply, CONCURRENTLY by default): one partial expression index per `[Queryable]` field over `events` (per type) and `documents` (per collection), using the compiler's exact expression, `sfq_` hash names, stale ones dropped; QRY-45 shows the planner using it. The CLI command (`db index sync`) is WP-4.1 |
6565
| WP-2.9 | Storage conformance kit complete | 2 | done | SCK-01..58 in SourceFlow.Testing, run on in-memory, PostgreSQL and PostgreSQL with RLS. New in this package: SCK-23 (in-flight writes invisible), SCK-37 (batched retention via IStoreRetention, plus the host retention loop), SCK-54 (CsCheck property: snapshot + tail = full fold), PostgreSQL SCK-28 (LISTEN loss and reconnect), and SCK-L01 CsCheck linearisability of appends and the outbox. SCK-36 is covered by SCK-34. SCK-20 (one round trip) moves to WP-2.10, and SCK-65..72 (partitioned events) to WP-4.5 |
6666
| WP-2.10 | Performance baseline | 1 | done | benchmarks/SourceFlow.Benchmarks (BenchmarkDotNet PERF-01..04, 06; measurement harness PERF-05, 08, 10); baselines in benchmarks/baselines and the report in implementation/perf-baseline.md. One-round-trip PostgreSQL commit (SCK-20), batch-aware document projections and adaptive subscription batches. Deviations logged with owners (PERF-01/02 allocations, PERF-10 borderline). PERF-07 waits for M3 and PERF-09 for WP-4.7 |
6767
| WP-2.R | Retire v2 EF stores | 0.5 | todo | |

‎samples/Bank/Bank.Accounts/AccountSummary.cs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
namespace Bank.Accounts;
66

77
/// <summary>The account summary read model (ported from the v2 AccountView).</summary>
8-
public sealed record AccountSummary(AccountId Id, string Holder, decimal Balance, AccountStatus Status, long Version);
8+
public sealed record AccountSummary(AccountId Id, string Holder, decimal Balance, [property: Queryable] AccountStatus Status, long Version);
99

1010
/// <summary>Maintains <see cref="AccountSummary"/> documents.</summary>
1111
[Projection("account-summary")]

‎samples/Bank/Bank.Accounts/BankAccount.cs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ public sealed record AccountOpened(AccountId AccountId, [property: PersonalData]
4141

4242
/// <summary>Money was deposited.</summary>
4343
[MessageType("bank.account.deposited", 1)]
44-
public sealed record MoneyDeposited(AccountId AccountId, decimal Amount, decimal BalanceAfter) : IEvent;
44+
public sealed record MoneyDeposited([property: Queryable] AccountId AccountId, decimal Amount, decimal BalanceAfter) : IEvent;
4545

4646
/// <summary>Money was withdrawn.</summary>
4747
[MessageType("bank.account.withdrawn", 1)]
Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
using System.Security.Cryptography;
2+
using System.Text;
3+
using Npgsql;
4+
using SourceFlow.Query;
5+
6+
namespace SourceFlow.Storage.PostgreSQL.Query;
7+
8+
/// <summary>One managed index (doc 14 §7.2).</summary>
9+
/// <param name="Name">The index name: <c>sfq_</c> + a hash of the definition, so a changed definition is a new index.</param>
10+
/// <param name="Table">The table (<c>events</c> or <c>documents</c>).</param>
11+
/// <param name="Source">What it serves, for example <c>bank.account.opened.customerId</c>.</param>
12+
/// <param name="Sql">The <c>CREATE INDEX</c> statement.</param>
13+
public sealed record QueryIndex(string Name, string Table, string Source, string Sql);
14+
15+
/// <summary>What an index sync did or would do.</summary>
16+
/// <param name="Create">Indexes to create (missing).</param>
17+
/// <param name="Drop">Managed indexes no longer in the catalog.</param>
18+
/// <param name="Unchanged">Indexes already present.</param>
19+
public sealed record QueryIndexPlan(IReadOnlyList<QueryIndex> Create, IReadOnlyList<string> Drop, IReadOnlyList<string> Unchanged);
20+
21+
/// <summary>
22+
/// Index sync for SFQL (doc 14 §7.2; <c>dotnet sourceflow db index sync</c>): one expression index per <c>[Queryable]</c>
23+
/// field, a partial index per event type over <c>events</c> and per read-model generation over <c>documents</c>. The
24+
/// expression is exactly the one the SFQL compiler emits, so PostgreSQL matches it; indexes follow the schema and are never
25+
/// written by hand. Timestamp fields are skipped: the text-to-timestamptz cast isn't immutable, so it can't be indexed.
26+
/// </summary>
27+
public static class PostgreSqlIndexSync
28+
{
29+
/// <summary>The prefix of managed index names; sync drops managed indexes the catalog no longer needs.</summary>
30+
public const string Prefix = "sfq_";
31+
32+
/// <summary>The indexes the catalog needs.</summary>
33+
public static IReadOnlyList<QueryIndex> Indexes(QueryCatalog catalog, string schema)
34+
{
35+
ArgumentNullException.ThrowIfNull(catalog);
36+
ArgumentException.ThrowIfNullOrWhiteSpace(schema);
37+
var indexes = new List<QueryIndex>();
38+
foreach (QueryEventType type in catalog.EventTypes)
39+
{
40+
foreach ((IReadOnlyList<object> path, QueryField field) in Queryable(type.Latest.Fields, []))
41+
{
42+
string expression = PostgreSqlQueryCompiler.Extract("events", path, field.Type).Replace("events.data", "data", StringComparison.Ordinal);
43+
indexes.Add(Create(schema, "events", $"{type.Name}.{string.Join('.', path)}", $"tenant_id, ({expression})", $"event_type = {Literal(type.Name)}"));
44+
}
45+
}
46+
47+
foreach (QueryReadModel model in catalog.ReadModels)
48+
{
49+
foreach ((IReadOnlyList<object> path, QueryField field) in Queryable(model.Fields, []))
50+
{
51+
string expression = PostgreSqlQueryCompiler.Extract("documents", path, field.Type).Replace("documents.data", "data", StringComparison.Ordinal);
52+
indexes.Add(Create(schema, "documents", $"{model.Collection}.{string.Join('.', path)}", $"tenant_id, ({expression})", $"collection = {Literal(model.Collection)}"));
53+
}
54+
}
55+
56+
return [.. indexes.OrderBy(i => i.Source, StringComparer.Ordinal)];
57+
}
58+
59+
/// <summary>Compares the catalog's indexes with the database's managed indexes.</summary>
60+
public static async ValueTask<QueryIndexPlan> PlanAsync(NpgsqlDataSource dataSource, QueryCatalog catalog, string schema, CancellationToken cancellationToken)
61+
{
62+
ArgumentNullException.ThrowIfNull(dataSource);
63+
IReadOnlyList<QueryIndex> wanted = Indexes(catalog, schema);
64+
HashSet<string> existing = await ExistingAsync(dataSource, schema, cancellationToken).ConfigureAwait(false);
65+
return new QueryIndexPlan(
66+
[.. wanted.Where(i => !existing.Contains(i.Name))],
67+
[.. existing.Where(name => wanted.All(i => i.Name != name)).Order(StringComparer.Ordinal)],
68+
[.. wanted.Where(i => existing.Contains(i.Name)).Select(i => i.Name)]);
69+
}
70+
71+
/// <summary>
72+
/// Creates missing indexes and drops stale managed ones. <paramref name="concurrently"/> (the default) builds without
73+
/// blocking writes; each statement then runs on its own, outside a transaction.
74+
/// </summary>
75+
public static async ValueTask<QueryIndexPlan> ApplyAsync(NpgsqlDataSource dataSource, QueryCatalog catalog, string schema, bool concurrently = true, CancellationToken cancellationToken = default)
76+
{
77+
QueryIndexPlan plan = await PlanAsync(dataSource, catalog, schema, cancellationToken).ConfigureAwait(false);
78+
await using NpgsqlConnection connection = await dataSource.OpenConnectionAsync(cancellationToken).ConfigureAwait(false);
79+
foreach (string name in plan.Drop)
80+
{
81+
await using var drop = new NpgsqlCommand($"DROP INDEX {(concurrently ? "CONCURRENTLY " : string.Empty)}IF EXISTS {schema}.{name}", connection);
82+
await drop.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
83+
}
84+
85+
foreach (QueryIndex index in plan.Create)
86+
{
87+
string sql = concurrently ? index.Sql.Replace("CREATE INDEX ", "CREATE INDEX CONCURRENTLY ", StringComparison.Ordinal) : index.Sql;
88+
await using var create = new NpgsqlCommand(sql, connection);
89+
await create.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
90+
}
91+
92+
return plan;
93+
}
94+
95+
private static QueryIndex Create(string schema, string table, string source, string columns, string predicate)
96+
{
97+
string definition = $"{table}|{columns}|{predicate}";
98+
string name = Prefix + Convert.ToHexStringLower(SHA256.HashData(Encoding.UTF8.GetBytes(definition)))[..20];
99+
return new QueryIndex(name, table, source, $"CREATE INDEX IF NOT EXISTS {name} ON {schema}.{table} ({columns}) WHERE {predicate}");
100+
}
101+
102+
private static IEnumerable<(IReadOnlyList<object> Path, QueryField Field)> Queryable(IReadOnlyList<QueryField> fields, IReadOnlyList<object> prefix)
103+
{
104+
foreach (QueryField field in fields)
105+
{
106+
IReadOnlyList<object> path = [.. prefix, field.Name];
107+
if (field.Queryable && !field.PersonalData && field.Type is not (QueryType.Timestamp or QueryType.Object or QueryType.Array or QueryType.Any))
108+
{
109+
yield return (path, field);
110+
}
111+
112+
foreach ((IReadOnlyList<object> Path, QueryField Field) nested in Queryable(field.Fields, path))
113+
{
114+
yield return nested;
115+
}
116+
}
117+
}
118+
119+
private static async ValueTask<HashSet<string>> ExistingAsync(NpgsqlDataSource dataSource, string schema, CancellationToken cancellationToken)
120+
{
121+
await using NpgsqlCommand command = dataSource.CreateCommand("SELECT indexname FROM pg_indexes WHERE schemaname = @schema AND starts_with(indexname, 'sfq_')");
122+
command.Parameters.Add(new NpgsqlParameter<string>("schema", schema));
123+
var names = new HashSet<string>(StringComparer.Ordinal);
124+
await using NpgsqlDataReader reader = await command.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false);
125+
while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false))
126+
{
127+
names.Add(reader.GetString(0));
128+
}
129+
130+
return names;
131+
}
132+
133+
// Catalog names (types, collections) in DDL: quoted SQL string constants; DDL takes no parameters.
134+
private static string Literal(string value)
135+
{
136+
return "'" + value.Replace("'", "''", StringComparison.Ordinal) + "'";
137+
}
138+
}

‎src/Storage/SourceFlow.Storage.PostgreSQL/Query/PostgreSqlQueryCompiler.cs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -709,7 +709,7 @@ private string Data(Resolved resolved, QueryType type)
709709
return sql.Append(" END").ToString();
710710
}
711711

712-
private static string Extract(string alias, IReadOnlyList<object> path, QueryType type)
712+
internal static string Extract(string alias, IReadOnlyList<object> path, QueryType type)
713713
{
714714
var sql = new StringBuilder($"{alias}.data");
715715
bool json = type is QueryType.Object or QueryType.Array or QueryType.Any;

‎tests/Core/SourceFlow.Core.Tests/Catalog/SchemaCatalogTests.cs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ public void Catalog_CoversEveryKind_ReadModelsAndProcessStates()
5151

5252
catalog.TryGet("account-summary", 1, out SchemaEntry? summary).ShouldBeTrue();
5353
summary!.Kind.ShouldBe(SchemaKind.ReadModel);
54-
summary.Schema["properties"]!["status"]!.ToJsonString().ShouldBe("{\"type\":\"integer\",\"x-sourceflow-enum\":\"AccountStatus\"}");
54+
summary.Schema["properties"]!["status"]!.ToJsonString().ShouldBe("{\"type\":\"integer\",\"x-sourceflow-enum\":\"AccountStatus\",\"x-sourceflow-queryable\":true}");
5555
catalog.TryGet("money-transfer", 1, out SchemaEntry? process).ShouldBeTrue();
5656
process!.Kind.ShouldBe(SchemaKind.ProcessState);
5757
catalog.TryGet("bank.account.withdraw.rejected", 1, out SchemaEntry? reply).ShouldBeTrue();

0 commit comments

Comments
 (0)