Skip to content

Commit dfd4332

Browse files
committed
Phase 21c: correlated subqueries — subquery story complete
- Binder: innermost-first outer-scope resolution; subplans carry their correlation set (bound outer identities); nested/grandparent correlation supported in binding and oracle. Structural dispatch on empty correlation set keeps all uncorrelated behavior byte-identical. - Oracle spec: correlated subqueries evaluate per outer row with substituted correlation values — scalar per-row cardinality rules (0=NULL/1=value/>1=loud error at the offending row, oracle order defining timing), IN/EXISTS per-row 3VL matrices. - Decorrelation (the payoff): CorrelatedExistsToSemiJoin / CorrelatedNotExistsToAntiJoin / CorrelatedInToSemiJoin fire when correlation appears only as top-level equality conjuncts in the subquery WHERE — correlation conjuncts become join equi-keys via a synthesized key projection. Proofs cover the 3VL corners (NULL correlation values never match == per-row UNKNOWN rejection). Blocked shapes each negative-tested: non-equality correlation, OR/AND-embedded, correlation in projections/aggregates/HAVING/ON, result-shaping nodes above WHERE, depth>1 rewrites, scalar, NOT IN. - Cost: materialized correlated evaluation = subplan cost x outer rows (documented) — extract_best genuinely chooses the semi join; EXPLAIN golden pins the decorrelated winner. - Vectorized: decorrelated plans run natively; residual correlated plans throw one recognizable guard, harness asserts it and verifies those queries oracle-side (399 guard paths per fuzzer run). - Fuzzer: 320 queries, 12348 execution paths, 5343 alternatives; 32 firings per correlated rule; prior uncorrelated firings unchanged. Implemented in one pass; reviewed, gate-verified, merged by lead.
1 parent ca1188f commit dfd4332

18 files changed

Lines changed: 1358 additions & 124 deletions

‎ARCHITECTURE.md‎

Lines changed: 18 additions & 3 deletions
Large diffs are not rendered by default.

‎ROADMAP.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,8 @@
2020
18. EXPLAIN and global HAVING polish. Delivered in Phase 18: `EXPLAIN <select-statement>` parses as an additive wrapper and returns a deterministic one-column `plan` string batch from a shared optimizer-side renderer used by both engines; the report includes the bound logical plan, memo group/rule summary, `extract_best` chosen plan with per-node row/cost estimates, and total cost. HAVING without GROUP BY is now legal over the single global group with the same aggregate-output and 3VL rules as grouped HAVING, including HAVING-only aggregates that are computed and dropped by final Project. Differential/fuzz alternative extraction caps were raised so the current string-widened corpus no longer hits the plan bound.
2121
19. Boolean-mask filter kernels. Delivered in Phase 19: vectorized Filter now evaluates WHERE/HAVING predicate trees as paired `is_true`/`is_known` byte masks using SQL Kleene 3VL algebra, with comparison leaves scanning pre-resolved int64 or string values column-at-a-time and `IS [NOT] NULL` leaves reading validity directly. Scan-adjacent identity selections use a dense full-batch domain, while filtered/non-identity inputs use dense selection-position masks that map back to the immutable input selection. Join residual predicates deliberately remain on the existing row-pair path. Release benchmark evidence in `docs/benchmarks.md` shows the three scan/filter vectorized medians improving with byte-identical correctness checks and no material non-target regression.
2222
20. LEFT/RIGHT OUTER JOIN arc. Delivered across Phase 20a/20b: the parser accepts `LEFT [OUTER] JOIN` and `RIGHT [OUTER] JOIN`; binding normalizes RIGHT to LEFT while stable identities preserve SQL-visible projection order; both interpreted and vectorized engines implement deterministic left-row-major NULL-extension, NULL-key non-matching, and residual-aware matchedness for hash and non-equi joins. `LeftJoinToInnerRule` additively simplifies a filtered LEFT join only under a proof-bearing conservative 3VL null-rejection test, unlocking the existing INNER commute/associate/pushdown exploration without weakening their guards. The deterministic fuzzer now mixes INNER/LEFT/RIGHT chains with actual NULL-bearing key data and verifies standalone rewrites, every memo alternative, and `extract_best` through both engines. The Phase 20a LEFT cardinality rule remains sufficient; no 20b cost-model change was required. The outer-join lowering guard and its verifier path are gone.
23-
21. Complete uncorrelated-subquery arc. Delivered across Phase 21a/21b: scalar-comparison, IN/NOT IN, and EXISTS/NOT EXISTS use fresh-scope immutable bound subplans and the pinned eager once-per-query 3VL/cardinality oracle. Explicit left SEMI/ANTI algebra has left-only output identity, deterministic interpreted and vectorized hash/NLJ execution, memo identity/printers/EXPLAIN/costing, and conservative transform guards. Top-level Filter EXISTS, NOT EXISTS, and IN decorrelate through proof-bearing memo rules; OR-contained leaves, NOT IN, and scalar forms stay materialized. Residual vectorized subqueries recursively execute once through the vectorized engine, scalar results compile as nullable constants, and IN sets use lookup-only typed hashes plus empty/NULL flags. Exact EXPLAIN goldens, a ten-mode 160-seed full-stack fuzzer, eager error equivalence, and a 100k-row selective-IN benchmark close the arc; the temporary lowering guard and oracle-only verifier path are gone.
24-
22. Correlated subqueries (future). Define an explicit outer-reference scope model and correlation algebra before adding binding or execution. Phase 21 deliberately rejects correlated names; no decorrelation rule may infer correlation from unresolved identifiers.
23+
21. Complete subquery arc. Delivered across Phase 21a/21b/21c: scalar-comparison, IN/NOT IN, and EXISTS/NOT EXISTS use immutable bound subplans with explicit lexical-depth correlation metadata. Empty correlation retains the eager once-per-query 3VL/cardinality model; correlated forms use the per-outer-row interpreted specification with nested/grandparent frames and row-ordered scalar errors. Explicit left SEMI/ANTI algebra has left-only output identity, deterministic interpreted/vectorized hash/NLJ execution, memo identity/printers/EXPLAIN/costing, and conservative guards. Uncorrelated and immediate equality-correlated top-level EXISTS, NOT EXISTS, and IN decorrelate through proof-bearing memo rules. Residual correlated forms remain oracle-defined and fail once at physical lowering; uncorrelated residuals still run vectorized recursively. Exact EXPLAIN goldens, eager/per-row error equivalence, and deterministic uncorrelated/correlated fuzz lanes close the story.
24+
22. Null-aware anti join (future). Extend algebra and execution before considering NOT IN decorrelation; ordinary Anti remains intentionally insufficient for a NULL-bearing right set, where unmatched NOT IN is UNKNOWN rather than TRUE.
2525

2626
## Phase 1 first tasks
2727

‎docs/rewrite-rule-template.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,5 +6,6 @@ For every optimizer rule, include:
66
- Replacement expression.
77
- Semantic equivalence argument.
88
- Preconditions, especially around NULLs, duplicates, side effects, and ordering.
9+
- For correlated rules, the explicit correlation-set/depth test, where every outer reference may occur, how substitution maps to join keys, and which residual/result-shaping locations block the rule.
910
- For predicate-tree rules, whether conjunct trees move as whole units, whether OR/AND splitting is forbidden, and what measure proves simplification or exploration terminates.
1011
- Golden query that proves interpreted equality before/after rewrite.

‎include/optimizer/rewrite.hpp‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -200,6 +200,53 @@ class InToSemiJoinRule final : public MemoRule {
200200
[[nodiscard]] bool apply(Memo& memo, GroupId group, const MemoExpression& expression) const override;
201201
};
202202

203+
// Pattern matched: a top-level Filter EXISTS whose correlated subplan is
204+
// exactly Project(Filter(Q)) and every outer reference occurs in a whole,
205+
// top-level `outer_col = inner_col` WHERE conjunct.
206+
// Replacement expression: remove those conjuncts from Q, project deterministic
207+
// hidden inner keys, and add SemiJoin(outer, decorrelated-Q, outer=hidden-inner).
208+
// Semantic equivalence: for each outer row, substituting its correlation values
209+
// accepts exactly inner rows whose equality keys are TRUE. Equi-SemiJoin tests
210+
// the same TRUE matches and emits the outer row once iff one exists. A NULL on
211+
// either key makes equality UNKNOWN, hence no Semi match in both forms.
212+
// Preconditions: correlations must be immediate-parent/local column equality;
213+
// non-equality, AND/OR-contained, projection, aggregate, HAVING, ON, nested,
214+
// DISTINCT/ORDER/LIMIT, and any residual correlation block the rule. Existing
215+
// projections remain evaluated and bag/order of the preserved side is unchanged.
216+
// Golden query: `SELECT a FROM t WHERE EXISTS (SELECT b FROM t1 WHERE t1.a=t.a)`.
217+
class CorrelatedExistsToSemiJoinRule final : public MemoRule {
218+
public:
219+
[[nodiscard]] std::string_view name() const override { return "CorrelatedExistsToSemiJoinRule"; }
220+
[[nodiscard]] bool apply(Memo& memo, GroupId group, const MemoExpression& expression) const override;
221+
};
222+
223+
// Pattern/replacement/preconditions are identical to
224+
// CorrelatedExistsToSemiJoinRule except NOT EXISTS becomes an equi AntiJoin.
225+
// Semantic equivalence: Anti emits an outer row exactly when no inner equality
226+
// is TRUE. FALSE and UNKNOWN are both non-matches, so a NULL correlation value
227+
// is retained, exactly as per-row NOT EXISTS over an UNKNOWN-rejected WHERE.
228+
// Golden query: `SELECT a FROM t WHERE NOT EXISTS (SELECT b FROM t1 WHERE t1.a=t.a)`.
229+
class CorrelatedNotExistsToAntiJoinRule final : public MemoRule {
230+
public:
231+
[[nodiscard]] std::string_view name() const override { return "CorrelatedNotExistsToAntiJoinRule"; }
232+
[[nodiscard]] bool apply(Memo& memo, GroupId group, const MemoExpression& expression) const override;
233+
};
234+
235+
// Pattern matched: the same equality-only correlated subplan owned by a whole
236+
// top-level `x IN (SELECT c ...)` Filter conjunct.
237+
// Replacement: an equi SemiJoin with the extracted correlation keys plus x=c.
238+
// Semantic equivalence: the filter survives only when some correlation-matched
239+
// row makes x=c TRUE; Semi has the same TRUE match set. NULL x, NULL c, and NULL
240+
// correlation keys cannot match and are rejected in both forms. NOT IN is
241+
// excluded because a NULL-bearing right set requires null-aware anti semantics.
242+
// All shape/precondition guards from the correlated EXISTS rule apply.
243+
// Golden query: `SELECT b FROM t WHERE b IN (SELECT b FROM t1 WHERE t1.a=t.a)`.
244+
class CorrelatedInToSemiJoinRule final : public MemoRule {
245+
public:
246+
[[nodiscard]] std::string_view name() const override { return "CorrelatedInToSemiJoinRule"; }
247+
[[nodiscard]] bool apply(Memo& memo, GroupId group, const MemoExpression& expression) const override;
248+
};
249+
203250
// Pattern matched: Filter(P, LeftJoin(L, R, J)) where at least one whole top-level conjunct in P
204251
// is provably null-rejecting for R's binding identities.
205252
// Replacement expression: Add Filter(P, InnerJoin(L, R, J)) as an equivalent expression in the

‎include/plan/logical_plan.hpp‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,9 @@ struct BoundColumnRef {
2525
std::string column;
2626
std::size_t position{0};
2727
catalog::ColumnType type{catalog::ColumnType::Int64};
28+
// Zero identifies the current query block. Positive values identify the
29+
// lexical distance to an enclosing query block (1 = immediate parent).
30+
std::size_t outer_depth{0};
2831
};
2932

3033
struct BoundScalarSubquery {
@@ -158,6 +161,9 @@ struct LogicalPlan {
158161
std::vector<AggregateExpression> aggregate_expressions;
159162
std::vector<SortKey> sort_keys;
160163
std::vector<BoundPredicate> predicates;
164+
// Set only on a bound subplan root. Empty is the structural Phase 21a
165+
// dispatch; non-empty means execution requires the listed outer values.
166+
std::vector<BoundColumnRef> correlation_columns;
161167
JoinKind join_kind{JoinKind::Inner};
162168
std::size_t limit_count{0};
163169
std::shared_ptr<LogicalPlan> input;

0 commit comments

Comments
 (0)