Skip to content

Commit a5531c8

Browse files
committed
feat(WP-2.8): SFQL compiler and query engine for PostgreSQL
- PostgreSqlQueryEngine implements IQueryEngine: one parameterised statement per page, every request value a typed $n parameter (I-3), keyset predicate and ORDER BY over a derived table, COLLATE "C" text ordering, horizon predicate, per-version rename CASE, tenant predicates, NULLIF division, read-only transaction with statement_timeout (QueryTimeout) and the RLS tenant setting. Aggregate and schema sources stay in the interpreter. - Catalog types enums by their JSON form (number or string); the interpreter returns whole event data and honours parameterised versions; Expr.ToString prints canonical text. - Tests: cross-provider equivalence with the interpreter on the same PostgreSQL data (QRY-01..20, 40, 41), golden SQL (QRY-42/43), injection corpus (QRY-44), no-literal property (QRY-60), UT-QL-66.
1 parent 3a48c1d commit a5531c8

13 files changed

Lines changed: 1385 additions & 6 deletions

File tree

‎docs/v3.0.0/implementation/STATUS.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ Updated in the PR that completes a work package (CONTRIBUTING.md). Status values
2121
| WP-1.10 | Hosting, roles, tenancy context | 1.5 | done | AddSourceFlow, SourceFlowBuilder Use* slots, roles, SourceFlowHost lifecycle, health check |
2222
| WP-1.11 | `SourceFlow.Testing` | 2.5 | done | SourceFlowTestHost, AggregateSpec, ProcessManagerSpec, ProjectionSpec, approvals; UT-TST |
2323
| WP-1.12 | Schema catalog | 2 | todo | |
24-
| WP-1.13 | SFQL front end (pipeline + SQL syntax) + in-memory interpreter + domain queries | 5.5 | in progress | Done: request guard, lexer, both parsers into one AST (UT-QL-01..13, 61..65, 69..77); catalog from the runtime model (Q-B1), binder with per-version renames, typing, output naming, G-6 tie-break, G-10/G-11/G-12 (SFQ0201–0206; UT-QL-21..33, 79, 88); policy: permissions, PII keys, cost guard, join and offset rules, limits (SFQ0301, 0401, 0402, 0405, 0406, 0409, 0501; UT-QL-34..39, 80, 89); in-memory interpreter (SQL three-valued logic, hash joins, grouping, subqueries, aggregate as-of folds, schema rows) and `IQueryService` (typed parameters SFQ0202/0203, PII redaction, tenancy, keyset continuation tokens SFQ0601, `at_least` consistency waits SFQ0602, plan cache, `IQueryEngine` SPI for provider compilers; UT-QL-41..60). Next: the PostgreSQL compiler (WP-2.8), then the typed builder and `[DomainQuery]` |
24+
| WP-1.13 | SFQL front end (pipeline + SQL syntax) + in-memory interpreter + domain queries | 5.5 | in progress | Done: request guard, lexer, both parsers into one AST (UT-QL-01..13, 61..65, 69..77); catalog from the runtime model (Q-B1), binder with per-version renames, typing, output naming, G-6 tie-break, G-10/G-11/G-12 (SFQ0201–0206; UT-QL-21..33, 79, 88); policy: permissions, PII keys, cost guard, join and offset rules, limits (SFQ0301, 0401, 0402, 0405, 0406, 0409, 0501; UT-QL-34..39, 80, 89); in-memory interpreter (SQL three-valued logic, hash joins, grouping, subqueries, aggregate as-of folds, schema rows) and `IQueryService` (typed parameters SFQ0202/0203, PII redaction, tenancy, keyset continuation tokens SFQ0601, `at_least` consistency waits SFQ0602, plan cache, `IQueryEngine` SPI for provider compilers; UT-QL-41..60). PostgreSQL compiler: WP-2.8. Next: the typed builder and `[DomainQuery]` |
2525
| WP-1.14 | Bank sample (ported v2 E2E domain) | 1.5 | in progress | Bank.Contracts/Accounts/Transfers/Notifications + E2E-01..07, 11, 13, 14, 17..20 green; Bank.Api waits for WP-4.2 |
2626
| WP-1.R | Retire v2 core | 0.5 | todo | |
2727
| WP-2.1 | PostgreSQL schema and migrator | 1.5 | done | V001 schema (ADR-028: json payload columns, tenant columns), PostgreSqlSchemaMigrator (advisory lock, checksummed schema_history, db script), UsePostgreSql with AutoMigrate (Development only by default); SCK-M01..M06. V002 (RLS) and V003 (partitioned events) come with WP-2.6 and WP-4.5 |
@@ -31,7 +31,7 @@ Updated in the PR that completes a work package (CONTRIBUTING.md). Status values
3131
| WP-2.5 | Snapshots and aggregate cache | 1.5 | done | Background snapshot writer (bounded, drop-oldest, drained on shutdown), MemoryCache aggregate cache (ReadTail, Optimistic with rejection confirmation and no shortcut for If-Match), cache and snapshot metrics; UT-SNAP-01..10, SCK-51..53/55 on in-memory and PostgreSQL (with and without RLS), parallel deposits on PostgreSQL in both cache modes. PERF-04 moves to the WP-2.10 baseline |
3232
| WP-2.6 | Tenancy with row-level security | 1 | done | V002 tenant_isolation policies; TenantIsolation.RowLevelSecurity enables and forces RLS (migrator applies and verifies the state); session-scoped set_config; system (BYPASSRLS) data source for the global log; tenant-aware buckets. The whole SCK passes under RLS through a DML-only role; SCK-59/60/M07, Bank transfer E2E under RLS. Per-tenant metric tags (bounded top-N) move to WP-4.3 |
3333
| WP-2.7 | `SourceFlow.Storage.EntityFrameworkCore` read models | 2 | in progress | Core session-projection SPI (ISessionProjection, IProjectionUnitOfWork, IStorageSession.FlushAsync for ES-R1), generator support (GEN-41/42), IEfProjection, EfSessionContextFactory on the session connection and transaction, UseEntityFrameworkProjections, IdempotentUpsert; UT-EF-01..04, SCK-61..64 inline and async on PostgreSQL. Rebuild with schema-per-generation waits for the blue/green rebuild tooling (doc 06 §6) |
34-
| WP-2.8 | SFQL on PostgreSQL (incl. joins and grouping) + index sync + domain queries | 3.5 | todo | |
34+
| WP-2.8 | SFQL on PostgreSQL (incl. joins and grouping) + index sync + domain queries | 3.5 | in progress | PostgreSqlQueryEngine (`IQueryEngine`): one parameterised statement per page (every request value a typed `$n` parameter, I-3), keyset predicate and ORDER BY outside a derived table, `COLLATE "C"` text ordering, horizon predicate, per-version rename CASE, tenant predicates (G-9), `NULLIF` division, read-only transaction with `statement_timeout` (`QueryTimeout`, default 5 s) and the RLS tenant setting; aggregate and schema sources stay in the interpreter. Cross-provider equivalence with the interpreter on the same data (QRY-01..20, incl. joins, grouping/having, distinct, subqueries, keyset pages interchangeable between engines), golden SQL (QRY-42/43), injection corpus (QRY-44), no-literal CsCheck property (QRY-60). Catalog now types enums by their JSON (number or string). Next: index sync (`db index sync`) and `[DomainQuery]` |
3535
| WP-2.9 | Storage conformance kit complete | 2 | done | SCK-01..58 in SourceFlow.Testing, run on in-memory, PostgreSQL and PostgreSQL with RLS. New in this package: SCK-23 (in-flight writes invisible), SCK-37 (batched retention via IStoreRetention, plus the host retention loop), SCK-54 (CsCheck property: snapshot + tail = full fold), PostgreSQL SCK-28 (LISTEN loss and reconnect), and SCK-L01 CsCheck linearisability of appends and the outbox. SCK-36 is covered by SCK-34. SCK-20 (one round trip) moves to WP-2.10, and SCK-65..72 (partitioned events) to WP-4.5 |
3636
| WP-2.10 | Performance baseline | 1 | done | benchmarks/SourceFlow.Benchmarks (BenchmarkDotNet PERF-01..04, 06; measurement harness PERF-05, 08, 10); baselines in benchmarks/baselines and the report in implementation/perf-baseline.md. One-round-trip PostgreSQL commit (SCK-20), batch-aware document projections and adaptive subscription batches. Deviations logged with owners (PERF-01/02 allocations, PERF-10 borderline). PERF-07 waits for M3 and PERF-09 for WP-4.7 |
3737
| WP-2.R | Retire v2 EF stores | 0.5 | todo | |

‎src/Core/SourceFlow.Net/Query/Catalog.cs‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -222,12 +222,15 @@ private static (QueryType Type, IReadOnlyList<QueryField> Fields, QueryField? El
222222
return (QueryType.Duration, [], null);
223223
}
224224

225+
JsonTypeInfo? info = TryGetTypeInfo(type, options);
225226
if (type.IsEnum)
226227
{
227-
return (QueryType.Any, [], null);
228+
// Numbers by default; strings with a string-enum converter. Serialize a value to see which.
229+
return (info is not null && JsonSerializer.SerializeToElement(Enum.ToObject(type, 0), info).ValueKind == JsonValueKind.String
230+
? QueryType.String
231+
: QueryType.Integer, [], null);
228232
}
229233

230-
JsonTypeInfo? info = TryGetTypeInfo(type, options);
231234
return info?.Kind switch
232235
{
233236
JsonTypeInfoKind.Object => (QueryType.Object, Fields(info, options, depth + 1), null),

‎src/Core/SourceFlow.Net/Query/Interpreter.cs‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -206,6 +206,7 @@ private async Task<List<SourceRow>> LoadSourceAsync(BoundSource source, Cancella
206206
case SourceKind.Events:
207207
{
208208
var names = new HashSet<string>(source.EventTypes.Select(t => t.Name), StringComparer.Ordinal);
209+
long? version = source.Version ?? (((EventsSource)source.Syntax).Version is { } v ? Evaluate(v, new Frame(null, null, null, null)) as long? : null);
209210
var rows = new List<SourceRow>();
210211
GlobalPosition after = GlobalPosition.Start;
211212
while (true)
@@ -220,7 +221,7 @@ private async Task<List<SourceRow>> LoadSourceAsync(BoundSource source, Cancella
220221
{
221222
if ((query.Options.AllTenants || e.TenantId == environment.Tenant)
222223
&& (names.Count == 0 || names.Contains(e.Type))
223-
&& (source.Version is null || e.TypeVersion == source.Version))
224+
&& (version is null || e.TypeVersion == version))
224225
{
225226
rows.Add(new EventRow(e));
226227
}
@@ -456,6 +457,7 @@ private List<SourceRow> SchemaRows(BoundSource source)
456457
"sfcausationid" => e.Metadata.CausationId?.ToString(),
457458
"tenant" => e.TenantId,
458459
"tags" => e.Tags.ToArray(),
460+
"data" => row.Data,
459461
_ => null,
460462
};
461463
}

‎src/Core/SourceFlow.Net/Query/Language/Ast.cs‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,14 @@ public enum QueryLanguage
2727
// ---------------------------------------------------------------- expressions
2828

2929
/// <summary>An SFQL expression (CESQL v1 extended, doc 14 §4.1).</summary>
30-
public abstract record Expr(TextSpan Span);
30+
public abstract record Expr(TextSpan Span)
31+
{
32+
/// <summary>The canonical text of the expression, independent of syntax and positions.</summary>
33+
public sealed override string ToString()
34+
{
35+
return AstPrinter.Expr(this);
36+
}
37+
}
3138

3239
/// <summary>The kind of a literal.</summary>
3340
public enum LiteralKind

‎src/Storage/SourceFlow.Storage.PostgreSQL/PostgreSqlBuilderExtensions.cs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,8 @@ public static SourceFlowBuilder UsePostgreSql(this SourceFlowBuilder builder, Ac
3434
builder.Services.TryAddSingleton(sp => new PostgreSqlStore(options, sp.GetService<TimeProvider>()));
3535
builder.Services.TryAddEnumerable(ServiceDescriptor.Singleton<IHostedService, PostgreSqlSchemaStartup>(sp =>
3636
new PostgreSqlSchemaStartup(sp.GetRequiredService<PostgreSqlStore>(), sp.GetService<IHostEnvironment>())));
37+
builder.Services.TryAddEnumerable(ServiceDescriptor.Singleton<SourceFlow.Query.IQueryEngine, Query.PostgreSqlQueryEngine>(sp =>
38+
new Query.PostgreSqlQueryEngine(sp.GetRequiredService<PostgreSqlStore>())));
3739
return builder.UseStorage(sp => sp.GetRequiredService<PostgreSqlStore>().ToPorts());
3840
}
3941
}

‎src/Storage/SourceFlow.Storage.PostgreSQL/PostgreSqlStorageOptions.cs‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,9 @@ public sealed partial class PostgreSqlStorageOptions
5353
/// <summary>An existing data source for cross-tenant readers; takes precedence over <see cref="SystemConnectionString"/>.</summary>
5454
public NpgsqlDataSource? SystemDataSource { get; set; }
5555

56+
/// <summary>The statement timeout for SFQL queries (doc 14 §9.4: default 5 s, at most 30 s).</summary>
57+
public TimeSpan QueryTimeout { get; set; } = TimeSpan.FromSeconds(5);
58+
5659
internal string Channel => NotificationChannel ?? Schema;
5760

5861
internal bool RowLevelSecurity => TenantIsolation == TenantIsolation.RowLevelSecurity;
@@ -79,6 +82,11 @@ internal void Validate()
7982
throw new InvalidOperationException("ListenReconnectDelay must be positive.");
8083
}
8184

85+
if (QueryTimeout <= TimeSpan.Zero || QueryTimeout > TimeSpan.FromSeconds(30))
86+
{
87+
throw new InvalidOperationException("QueryTimeout must be positive and at most 30 seconds.");
88+
}
89+
8290
if (RowLevelSecurity && SystemDataSource is null && string.IsNullOrWhiteSpace(SystemConnectionString))
8391
{
8492
throw new InvalidOperationException(

0 commit comments

Comments
 (0)