Merge phase21c-correlated: subquery story complete

Author: Aly700

ARCHITECTURE.md

@@ -20,8 +20,8 @@
 18. EXPLAIN and global HAVING polish. Delivered in Phase 18: `EXPLAIN <select-statement>` parses as an additive wrapper and returns a deterministic one-column `plan` string batch from a shared optimizer-side renderer used by both engines; the report includes the bound logical plan, memo group/rule summary, `extract_best` chosen plan with per-node row/cost estimates, and total cost. HAVING without GROUP BY is now legal over the single global group with the same aggregate-output and 3VL rules as grouped HAVING, including HAVING-only aggregates that are computed and dropped by final Project. Differential/fuzz alternative extraction caps were raised so the current string-widened corpus no longer hits the plan bound.
 19. Boolean-mask filter kernels. Delivered in Phase 19: vectorized Filter now evaluates WHERE/HAVING predicate trees as paired `is_true`/`is_known` byte masks using SQL Kleene 3VL algebra, with comparison leaves scanning pre-resolved int64 or string values column-at-a-time and `IS [NOT] NULL` leaves reading validity directly. Scan-adjacent identity selections use a dense full-batch domain, while filtered/non-identity inputs use dense selection-position masks that map back to the immutable input selection. Join residual predicates deliberately remain on the existing row-pair path. Release benchmark evidence in `docs/benchmarks.md` shows the three scan/filter vectorized medians improving with byte-identical correctness checks and no material non-target regression.
 20. LEFT/RIGHT OUTER JOIN arc. Delivered across Phase 20a/20b: the parser accepts `LEFT [OUTER] JOIN` and `RIGHT [OUTER] JOIN`; binding normalizes RIGHT to LEFT while stable identities preserve SQL-visible projection order; both interpreted and vectorized engines implement deterministic left-row-major NULL-extension, NULL-key non-matching, and residual-aware matchedness for hash and non-equi joins. `LeftJoinToInnerRule` additively simplifies a filtered LEFT join only under a proof-bearing conservative 3VL null-rejection test, unlocking the existing INNER commute/associate/pushdown exploration without weakening their guards. The deterministic fuzzer now mixes INNER/LEFT/RIGHT chains with actual NULL-bearing key data and verifies standalone rewrites, every memo alternative, and `extract_best` through both engines. The Phase 20a LEFT cardinality rule remains sufficient; no 20b cost-model change was required. The outer-join lowering guard and its verifier path are gone.
-21. Complete uncorrelated-subquery arc. Delivered across Phase 21a/21b: scalar-comparison, IN/NOT IN, and EXISTS/NOT EXISTS use fresh-scope immutable bound subplans and the pinned eager once-per-query 3VL/cardinality oracle. Explicit left SEMI/ANTI algebra has left-only output identity, deterministic interpreted and vectorized hash/NLJ execution, memo identity/printers/EXPLAIN/costing, and conservative transform guards. Top-level Filter EXISTS, NOT EXISTS, and IN decorrelate through proof-bearing memo rules; OR-contained leaves, NOT IN, and scalar forms stay materialized. Residual vectorized subqueries recursively execute once through the vectorized engine, scalar results compile as nullable constants, and IN sets use lookup-only typed hashes plus empty/NULL flags. Exact EXPLAIN goldens, a ten-mode 160-seed full-stack fuzzer, eager error equivalence, and a 100k-row selective-IN benchmark close the arc; the temporary lowering guard and oracle-only verifier path are gone.
-22. Correlated subqueries (future). Define an explicit outer-reference scope model and correlation algebra before adding binding or execution. Phase 21 deliberately rejects correlated names; no decorrelation rule may infer correlation from unresolved identifiers.
+21. Complete subquery arc. Delivered across Phase 21a/21b/21c: scalar-comparison, IN/NOT IN, and EXISTS/NOT EXISTS use immutable bound subplans with explicit lexical-depth correlation metadata. Empty correlation retains the eager once-per-query 3VL/cardinality model; correlated forms use the per-outer-row interpreted specification with nested/grandparent frames and row-ordered scalar errors. Explicit left SEMI/ANTI algebra has left-only output identity, deterministic interpreted/vectorized hash/NLJ execution, memo identity/printers/EXPLAIN/costing, and conservative guards. Uncorrelated and immediate equality-correlated top-level EXISTS, NOT EXISTS, and IN decorrelate through proof-bearing memo rules. Residual correlated forms remain oracle-defined and fail once at physical lowering; uncorrelated residuals still run vectorized recursively. Exact EXPLAIN goldens, eager/per-row error equivalence, and deterministic uncorrelated/correlated fuzz lanes close the story.
+22. Null-aware anti join (future). Extend algebra and execution before considering NOT IN decorrelation; ordinary Anti remains intentionally insufficient for a NULL-bearing right set, where unmatched NOT IN is UNKNOWN rather than TRUE.
 
 ## Phase 1 first tasks
 

ROADMAP.md

@@ -6,5 +6,6 @@ For every optimizer rule, include:
 - Replacement expression.
 - Semantic equivalence argument.
 - Preconditions, especially around NULLs, duplicates, side effects, and ordering.
+- For correlated rules, the explicit correlation-set/depth test, where every outer reference may occur, how substitution maps to join keys, and which residual/result-shaping locations block the rule.
 - For predicate-tree rules, whether conjunct trees move as whole units, whether OR/AND splitting is forbidden, and what measure proves simplification or exploration terminates.
 - Golden query that proves interpreted equality before/after rewrite.

docs/rewrite-rule-template.md

@@ -200,6 +200,53 @@ class InToSemiJoinRule final : public MemoRule {
     [[nodiscard]] bool apply(Memo& memo, GroupId group, const MemoExpression& expression) const override;
 };
 
+// Pattern matched: a top-level Filter EXISTS whose correlated subplan is
+// exactly Project(Filter(Q)) and every outer reference occurs in a whole,
+// top-level `outer_col = inner_col` WHERE conjunct.
+// Replacement expression: remove those conjuncts from Q, project deterministic
+// hidden inner keys, and add SemiJoin(outer, decorrelated-Q, outer=hidden-inner).
+// Semantic equivalence: for each outer row, substituting its correlation values
+// accepts exactly inner rows whose equality keys are TRUE. Equi-SemiJoin tests
+// the same TRUE matches and emits the outer row once iff one exists. A NULL on
+// either key makes equality UNKNOWN, hence no Semi match in both forms.
+// Preconditions: correlations must be immediate-parent/local column equality;
+// non-equality, AND/OR-contained, projection, aggregate, HAVING, ON, nested,
+// DISTINCT/ORDER/LIMIT, and any residual correlation block the rule. Existing
+// projections remain evaluated and bag/order of the preserved side is unchanged.
+// Golden query: `SELECT a FROM t WHERE EXISTS (SELECT b FROM t1 WHERE t1.a=t.a)`.
+class CorrelatedExistsToSemiJoinRule final : public MemoRule {
+public:
+    [[nodiscard]] std::string_view name() const override { return "CorrelatedExistsToSemiJoinRule"; }
+    [[nodiscard]] bool apply(Memo& memo, GroupId group, const MemoExpression& expression) const override;
+};
+
+// Pattern/replacement/preconditions are identical to
+// CorrelatedExistsToSemiJoinRule except NOT EXISTS becomes an equi AntiJoin.
+// Semantic equivalence: Anti emits an outer row exactly when no inner equality
+// is TRUE. FALSE and UNKNOWN are both non-matches, so a NULL correlation value
+// is retained, exactly as per-row NOT EXISTS over an UNKNOWN-rejected WHERE.
+// Golden query: `SELECT a FROM t WHERE NOT EXISTS (SELECT b FROM t1 WHERE t1.a=t.a)`.
+class CorrelatedNotExistsToAntiJoinRule final : public MemoRule {
+public:
+    [[nodiscard]] std::string_view name() const override { return "CorrelatedNotExistsToAntiJoinRule"; }
+    [[nodiscard]] bool apply(Memo& memo, GroupId group, const MemoExpression& expression) const override;
+};
+
+// Pattern matched: the same equality-only correlated subplan owned by a whole
+// top-level `x IN (SELECT c ...)` Filter conjunct.
+// Replacement: an equi SemiJoin with the extracted correlation keys plus x=c.
+// Semantic equivalence: the filter survives only when some correlation-matched
+// row makes x=c TRUE; Semi has the same TRUE match set. NULL x, NULL c, and NULL
+// correlation keys cannot match and are rejected in both forms. NOT IN is
+// excluded because a NULL-bearing right set requires null-aware anti semantics.
+// All shape/precondition guards from the correlated EXISTS rule apply.
+// Golden query: `SELECT b FROM t WHERE b IN (SELECT b FROM t1 WHERE t1.a=t.a)`.
+class CorrelatedInToSemiJoinRule final : public MemoRule {
+public:
+    [[nodiscard]] std::string_view name() const override { return "CorrelatedInToSemiJoinRule"; }
+    [[nodiscard]] bool apply(Memo& memo, GroupId group, const MemoExpression& expression) const override;
+};
+
 // Pattern matched: Filter(P, LeftJoin(L, R, J)) where at least one whole top-level conjunct in P
 // is provably null-rejecting for R's binding identities.
 // Replacement expression: Add Filter(P, InnerJoin(L, R, J)) as an equivalent expression in the

include/optimizer/rewrite.hpp

@@ -25,6 +25,9 @@ struct BoundColumnRef {
     std::string column;
     std::size_t position{0};
     catalog::ColumnType type{catalog::ColumnType::Int64};
+    // Zero identifies the current query block. Positive values identify the
+    // lexical distance to an enclosing query block (1 = immediate parent).
+    std::size_t outer_depth{0};
 };
 
 struct BoundScalarSubquery {
@@ -158,6 +161,9 @@ struct LogicalPlan {
     std::vector<AggregateExpression> aggregate_expressions;
     std::vector<SortKey> sort_keys;
     std::vector<BoundPredicate> predicates;
+    // Set only on a bound subplan root. Empty is the structural Phase 21a
+    // dispatch; non-empty means execution requires the listed outer values.
+    std::vector<BoundColumnRef> correlation_columns;
     JoinKind join_kind{JoinKind::Inner};
     std::size_t limit_count{0};
     std::shared_ptr<LogicalPlan> input;