Merge phase24-null-aware-anti: null-aware anti join — NOT IN decorrelation arc complete

Author: Aly700

ARCHITECTURE.md

@@ -23,7 +23,7 @@
 21. Complete subquery arc. Delivered across Phase 21a/21b/21c: scalar-comparison, IN/NOT IN, and EXISTS/NOT EXISTS use immutable bound subplans with explicit lexical-depth correlation metadata. Empty correlation retains the eager once-per-query 3VL/cardinality model; correlated forms use the per-outer-row interpreted specification with nested/grandparent frames and row-ordered scalar errors. Explicit left SEMI/ANTI algebra has left-only output identity, deterministic interpreted/vectorized hash/NLJ execution, memo identity/printers/EXPLAIN/costing, and conservative guards. Uncorrelated and immediate equality-correlated top-level EXISTS, NOT EXISTS, and IN decorrelate through proof-bearing memo rules. Residual correlated forms remain oracle-defined and fail once at physical lowering; uncorrelated residuals still run vectorized recursively. Exact EXPLAIN goldens, eager/per-row error equivalence, and deterministic uncorrelated/correlated fuzz lanes close the story.
 22. Window functions arc. Delivered across Phase 22a/22b: whole-SELECT-item `ROW_NUMBER`, `RANK`, and `DENSE_RANK` plus whole-partition `SUM`, `COUNT`, `MIN`, and `MAX` parse and bind into one post-HAVING Window node; both interpreted and vectorized execution implement first-appearance partitioning, input-order preservation, stable ranking ties, NULL partition/peer equality, checked/NULL-skipping partition aggregates, and replicated results; Window participates in physical lowering, memo identity, EXPLAIN, extraction, conservative costing, golden coverage, deterministic fuzzing, and a Release benchmark workload while remaining a filter-pushdown barrier, with ROW_NUMBER and whole-partition checked SUM pinning order-changing child transforms because stable ties and intermediate overflow expose input order.
 23. Running window frames. Delivered in Phase 23: aggregate OVER ORDER BY defaults to `RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`, and the same explicit RANGE spelling plus `ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW` bind as distinct frame modes. RANGE publishes one result through each complete peer group and canonically orders checked SUM arguments within peers, making output and overflow category independent of incidental tie order. ROWS publishes each stable input-order prefix and pins order-changing child transforms; a checked Aggregate SUM below any Window pins independently. The oracle, vectorized operator, memo alternatives, deterministic fuzzer, goldens, and standalone benchmark cover the new paths while whole-partition behavior remains unchanged. Every other frame bound or modifier fails as a positioned unsupported frame.
-24. Null-aware anti join (future). Extend algebra and execution before considering NOT IN decorrelation; ordinary Anti remains intentionally insufficient for a NULL-bearing right set, where unmatched NOT IN is UNKNOWN rather than TRUE.
+24. NULL-aware anti join. Delivered in Phase 24: explicit `NullAwareAnti` algebra separates equality-correlation candidate predicates from one structural membership equality and preserves left-only identity plus left-row-major output. The interpreted candidate-set oracle and vectorized grouped hash/NLJ kernels coincide with `NOT IN` 3VL for empty, NULL-bearing, NULL-left, match, and miss cases while ordinary Anti remains byte-identical. Proof-bearing memo rules add uncorrelated and immediate equality-correlated NOT IN alternatives without removing the materialized path; memo identity, EXPLAIN, deterministic costing, transform guards, checked-SUM order pins, trap-matrix goldens, 160-seed fuzz floors, and an optimized checksum-before-timing benchmark complete the arc.
 
 ## Phase 1 first tasks
 

ROADMAP.md

@@ -324,6 +324,22 @@ plan::LogicalPlan bind_decorrelated_semi_query(const execution::Catalog& catalog
     throw std::logic_error("benchmark IN query did not produce a SemiJoin alternative");
 }
 
+plan::LogicalPlan bind_best_null_aware_anti_query(const execution::Catalog& catalog,
+                                                  const std::string& sql) {
+    const auto logical = bind_query(catalog, sql);
+    optimizer::Memo memo;
+    const auto root = memo.insert(logical);
+    const auto explored = optimizer::explore_memo_to_fixpoint(memo, optimizer::default_memo_rules());
+    if (!explored.reached_fixpoint) {
+        throw std::logic_error("benchmark NULL-aware anti memo exploration did not reach fixpoint");
+    }
+    const auto best = memo.extract_best(root, catalog);
+    if (plan::to_string(best).find("NullAwareAntiJoin[") == std::string::npos) {
+        throw std::logic_error("benchmark NOT IN query did not choose its NullAwareAnti alternative");
+    }
+    return best;
+}
+
 std::vector<Workload> make_workloads(const execution::Catalog& catalog) {
     std::vector<Workload> workloads;
     auto add = [&](std::string name, std::string rows, std::string sql) {
@@ -352,6 +368,16 @@ std::vector<Workload> make_workloads(const execution::Catalog& catalog) {
                                      sql,
                                      bind_decorrelated_semi_query(catalog, sql)});
     }
+    {
+        const std::string sql =
+            "SELECT l.payload AS payload FROM join_left AS l WHERE l.k1 NOT IN ("
+            "SELECT r.k1 FROM join_right AS r "
+            "WHERE r.k2 = l.k2 AND r.payload < 10016)";
+        workloads.push_back(Workload{"selective_not_in_null_aware_anti",
+                                     "outer=100000,right_source=256,right_rows=16,survivors=93744",
+                                     sql,
+                                     bind_best_null_aware_anti_query(catalog, sql)});
+    }
     add("aggregate_few_groups",
         "fact=200000,groups=8",
         "SELECT group_few, SUM(value) AS total, COUNT(*) AS n "

benchmarks/sql_bench.cpp

@@ -260,3 +260,23 @@ for `SUM(value) OVER (PARTITION BY bucket ORDER BY sort_key)` across the
 | workload | rows | correctness | interpreted min ms | interpreted median ms | vectorized min ms | vectorized median ms | median speedup |
 |---|---:|---|---:|---:|---:|---:|---:|
 | window_running_sum | fact=200000,partitions=100 | match rows=200000 checksum=0xefdfa729da68f560 | 704.578 | 711.366 | 102.685 | 105.883 | 6.718x |
+
+## Phase 24 Selective NULL-Aware Anti Join (2026-07-16)
+
+Same optimized build, five-repetition min/median, deterministic-data, and
+checksum-before-timing methodology as above. The SQL is a correlated `NOT IN`
+over the 100,000-row `join_left` table. Its equality correlation admits the
+Phase 24 rewrite, and the benchmark harness fails loudly unless memo costing
+chooses `NullAwareAntiJoin`. The right source has 256 rows; its local filter
+retains 16 membership values in one correlation bucket. Because the left key
+cycle ends partway through its final 256-row block, exactly 6,256 rows belong
+to that bucket and 93,744 survive.
+
+| workload | rows | correctness | interpreted min ms | interpreted median ms | vectorized min ms | vectorized median ms | median speedup |
+|---|---:|---|---:|---:|---:|---:|---:|
+| selective_not_in_null_aware_anti | outer=100000,right_source=256,right_rows=16,survivors=93744 | match rows=93744 checksum=0x3aa4b8448334dc4d | 193.038 | 196.903 | 11.762 | 12.043 | 16.350x |
+
+The checksum match is performed before timing and pins the left-only output
+and correlated candidate-set semantics. The vectorized grouped hash kernel is
+16.350x faster on median than the interpreted left-row-major semantic oracle
+for this selective workload.

docs/benchmarks.md

@@ -5,6 +5,7 @@
 
 #include <cstdint>
 #include <cstddef>
+#include <optional>
 #include <string>
 #include <unordered_map>
 #include <utility>
@@ -29,6 +30,7 @@ struct MemoExpression {
     std::vector<plan::WindowExpression> window_expressions;
     std::vector<plan::SortKey> sort_keys;
     std::vector<plan::BoundPredicate> predicates;
+    std::optional<plan::BoundPredicate> null_aware_predicate;
     plan::JoinKind join_kind{plan::JoinKind::Inner};
     std::size_t limit_count{0};
     std::vector<GroupId> children;

include/optimizer/memo.hpp

@@ -200,6 +200,47 @@ class InToSemiJoinRule final : public MemoRule {
     [[nodiscard]] bool apply(Memo& memo, GroupId group, const MemoExpression& expression) const override;
 };
 
+// Pattern matched: one whole top-level `x NOT IN (SELECT c FROM Q)` Filter
+// conjunct whose left operand is not a scalar subquery and whose bound right
+// plan is uncorrelated with exactly one typed final output.
+// Replacement expression: retain the materialized Filter expression and add
+// NullAwareAnti(input, Q) with an empty candidate-predicate list and the
+// structural membership equality `x = c`; residual top-level conjuncts remain
+// above the new join. If Q's projected identity collides with a left identity,
+// a deterministic `__not_in_value_N` Project alias makes the right side
+// explicit before constructing that equality.
+// Semantic equivalence: for each left row, the join candidate bag is exactly
+// Q. An empty Q emits, matching TRUE even for NULL x. For nonempty Q, NULL x
+// makes every equality UNKNOWN and suppresses. A TRUE equality suppresses as
+// NOT IN FALSE; absent a TRUE, any NULL c supplies UNKNOWN and suppresses; only
+// non-NULL x with every non-NULL c unequal yields all FALSE and emits. These
+// cases exhaust SQL 3VL. Right duplicates repeat an existing truth value and
+// cannot multiply the left-only result. The join preserves left row order and
+// executes the right plan once eagerly, so empty-left behavior and positioned
+// checked-expression errors coincide with materialization.
+// Preconditions/guards: only an entire Filter conjunct is inspected; leaves
+// below AND/OR, scalar-subquery operands, correlation, missing plans, or a
+// non-single-output bound shape do not rewrite. A sibling subquery predicate
+// or an embedded subquery in the left relational input also blocks the rule:
+// materialization observes Q's eager execution relative to those other eager
+// errors, and a native join cannot in general reproduce that owner ordering.
+// Binding has already pinned any checked SUM input, and memo ingest copies
+// every node's order permission.
+// Unsupported correlation stays materialized and physical lowering fails at
+// its deterministic source position. Each application removes one NOT IN leaf
+// from the added expression, never creates one, and structural deduplication
+// guarantees termination.
+// Golden query: `SELECT a FROM t WHERE a NOT IN (SELECT a FROM t1)` explores
+// both materialized and NullAwareAnti alternatives; the trap matrix separately
+// pins empty and NULL-bearing right sets.
+class NotInToNullAwareAntiJoinRule final : public MemoRule {
+public:
+    [[nodiscard]] std::string_view name() const override {
+        return "NotInToNullAwareAntiJoinRule";
+    }
+    [[nodiscard]] bool apply(Memo& memo, GroupId group, const MemoExpression& expression) const override;
+};
+
 // Pattern matched: a top-level Filter EXISTS whose correlated subplan is
 // exactly Project(Filter(Q)) and every outer reference occurs in a whole,
 // top-level `outer_col = inner_col` WHERE conjunct.
@@ -247,6 +288,46 @@ class CorrelatedInToSemiJoinRule final : public MemoRule {
     [[nodiscard]] bool apply(Memo& memo, GroupId group, const MemoExpression& expression) const override;
 };
 
+// Pattern matched: a whole top-level `x NOT IN (SELECT c ...)` conjunct whose
+// correlated subplan is exactly the Phase 21c Project(Filter(Scan|Join)) shape,
+// with every outer use covered by an immediate-parent/local column equality.
+// Replacement: remove the correlation WHERE equalities from a copied right
+// plan, append deterministic `__corr_key_N` projections, keep the resulting
+// outer-to-hidden equalities as candidate predicates, and store `x = c`
+// separately as the NullAwareAnti membership equality. A colliding selected
+// right identity receives the same deterministic hidden alias. The original
+// materialized Filter remains in its memo group.
+// Semantic equivalence: for a fixed outer row l, TRUE-only evaluation of the
+// extracted candidate equalities selects exactly the same inner bag C(l) as
+// substituting l into the original WHERE. A NULL on either correlation key
+// makes that equality UNKNOWN and therefore can make C(l) empty; it never acts
+// as a membership NULL. Applying the exhaustive uncorrelated NOT IN argument
+// above to each C(l) proves identical TRUE/FALSE/UNKNOWN survival for every
+// row, including empty per-key bags, NULL membership values, and NULL outer
+// correlation keys. Hidden columns do not escape the left-only join, right
+// duplicates do not multiply rows, and left-row-major order is unchanged.
+// Preconditions/guards: scalar-subquery operands; non-equality, nested,
+// transitive, projection, aggregate, HAVING, JOIN ON, AND/OR-contained, and
+// result-shaped correlation are rejected. Nested subqueries in the right plan
+// and sibling subquery predicates are also rejected because decorrelation
+// would change their eager/per-row error order. The accepted
+// Scan|Join/Filter/Project right slice is therefore pure and has no
+// runtime-error-producing aggregate; all copied order permissions remain
+// structural. Residual shapes stay materialized and fail physical lowering
+// with the positioned guard.
+// The replacement removes one eligible leaf and memo deduplication guarantees
+// termination.
+// Golden query: `SELECT b FROM t WHERE b NOT IN (SELECT t1.b FROM t1 WHERE
+// t1.a=t.a)` executes the original and every native alternative through both
+// engines.
+class CorrelatedNotInToNullAwareAntiJoinRule final : public MemoRule {
+public:
+    [[nodiscard]] std::string_view name() const override {
+        return "CorrelatedNotInToNullAwareAntiJoinRule";
+    }
+    [[nodiscard]] bool apply(Memo& memo, GroupId group, const MemoExpression& expression) const override;
+};
+
 // Pattern matched: Filter(P, LeftJoin(L, R, J)) where at least one whole top-level conjunct in P
 // is provably null-rejecting for R's binding identities.
 // Replacement expression: Add Filter(P, InnerJoin(L, R, J)) as an equivalent expression in the
@@ -294,12 +375,14 @@ class LeftJoinToInnerRule final : public MemoRule {
 // NULL operands yield UNKNOWN and are rejected in either placement. Whole-tree reference analysis
 // prevents unsound OR/AND splitting.
 // Preconditions: The child join must be an INNER join, except that a whole
-// left-only conjunct may move into the preserved child of a SEMI/ANTI join.
-// This exception is sound because Semi/Anti output is a subset of left rows:
+// left-only conjunct may move into the preserved child of a SEMI, ANTI, or
+// NULL-aware ANTI join. This exception is sound because every preserved-left
+// join output is a subset of left rows:
 // filtering a left row before testing match existence keeps exactly the rows
 // that would pass the same predicate after the existence test. Right-only,
 // mixed-side, literal-only, and empty-reference conjuncts remain above a
-// SEMI/ANTI join; no predicate is moved into its match condition or right side.
+// preserved-left join; no predicate is moved into its candidate conditions,
+// NULL-aware membership equality, match condition, or right side.
 // LEFT joins are skipped because pushing a
 // predicate into the null-supplying side, or turning a post-join predicate into an ON predicate,
 // can preserve NULL-extended rows that the original WHERE would reject or reject rows the original

include/optimizer/rewrite.hpp

@@ -7,6 +7,7 @@
 #include <functional>
 #include <memory>
 #include <optional>
+#include <stdexcept>
 #include <string>
 #include <utility>
 #include <variant>
@@ -18,7 +19,7 @@ struct LogicalPlan;
 
 enum class LogicalKind { Scan, Join, Filter, Project, Aggregate, Window, Distinct, Sort, Limit, Explain };
 enum class OrderPermission { Deterministic, Arbitrary };
-enum class JoinKind { Inner, Left, Semi, Anti };
+enum class JoinKind { Inner, Left, Semi, Anti, NullAwareAnti };
 
 struct BoundColumnRef {
     std::string binding;
@@ -174,6 +175,9 @@ struct LogicalPlan {
     std::vector<WindowExpression> window_expressions;
     std::vector<SortKey> sort_keys;
     std::vector<BoundPredicate> predicates;
+    // Structural only for NullAwareAnti. `predicates` identify the candidate
+    // right rows; this equality carries NOT IN's distinct UNKNOWN semantics.
+    std::optional<BoundPredicate> null_aware_predicate;
     // Set only on a bound subplan root. Empty is the structural Phase 21a
     // dispatch; non-empty means execution requires the listed outer values.
     std::vector<BoundColumnRef> correlation_columns;
@@ -196,15 +200,19 @@ struct LogicalPlan {
     }
 
     // INNER/LEFT output identity/order is deterministic: all columns from the
-    // left child followed by all columns from the right child. SEMI/ANTI emit
-    // exactly the left child's identities and order; right identities exist
-    // only while evaluating join predicates. Expressions in a bound logical
-    // plan refer to identities by binding and column name; downstream layers
-    // must not re-resolve parsed SQL names.
+    // left child followed by all columns from the right child. SEMI/ANTI and
+    // NULL-aware ANTI emit exactly the left child's identities and order;
+    // right identities exist only while evaluating join predicates.
+    // Expressions in a bound logical plan refer to identities by binding and
+    // column name; downstream layers must not re-resolve parsed SQL names.
     static LogicalPlan join(std::vector<BoundPredicate> predicates,
                             LogicalPlan left,
                             LogicalPlan right,
                             JoinKind join_kind = JoinKind::Inner) {
+        if (join_kind == JoinKind::NullAwareAnti) {
+            throw std::invalid_argument(
+                "NullAwareAnti must be constructed with an explicit membership equality");
+        }
         LogicalPlan p;
         p.kind = LogicalKind::Join;
         p.join_kind = join_kind;
@@ -214,6 +222,27 @@ struct LogicalPlan {
         return p;
     }
 
+    static LogicalPlan null_aware_anti(BoundPredicate membership,
+                                       LogicalPlan left,
+                                       LogicalPlan right,
+                                       std::vector<BoundPredicate> candidate_predicates = {}) {
+        if (membership.kind != sql::PredicateKind::Comparison ||
+            membership.comparison.op != sql::ComparisonOp::Equal) {
+            throw std::invalid_argument("NullAwareAnti membership predicate must be an equality");
+        }
+        if (membership.comparison.left.type != membership.comparison.right.type) {
+            throw std::invalid_argument("NullAwareAnti membership equality must have matching types");
+        }
+        LogicalPlan p;
+        p.kind = LogicalKind::Join;
+        p.join_kind = JoinKind::NullAwareAnti;
+        p.predicates = std::move(candidate_predicates);
+        p.null_aware_predicate = std::move(membership);
+        p.left = std::make_shared<LogicalPlan>(std::move(left));
+        p.right = std::make_shared<LogicalPlan>(std::move(right));
+        return p;
+    }
+
     static LogicalPlan filter(std::vector<BoundPredicate> predicates, LogicalPlan child) {
         LogicalPlan p;
         p.kind = LogicalKind::Filter;