Skip to content

Commit ca1188f

Browse files
committed
Merge phase21b-decorrelation: subquery arc complete
2 parents bc924a9 + 82e3596 commit ca1188f

19 files changed

Lines changed: 1490 additions & 353 deletions

‎ARCHITECTURE.md‎

Lines changed: 19 additions & 7 deletions
Large diffs are not rendered by default.

‎ROADMAP.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,8 @@
2020
18. EXPLAIN and global HAVING polish. Delivered in Phase 18: `EXPLAIN <select-statement>` parses as an additive wrapper and returns a deterministic one-column `plan` string batch from a shared optimizer-side renderer used by both engines; the report includes the bound logical plan, memo group/rule summary, `extract_best` chosen plan with per-node row/cost estimates, and total cost. HAVING without GROUP BY is now legal over the single global group with the same aggregate-output and 3VL rules as grouped HAVING, including HAVING-only aggregates that are computed and dropped by final Project. Differential/fuzz alternative extraction caps were raised so the current string-widened corpus no longer hits the plan bound.
2121
19. Boolean-mask filter kernels. Delivered in Phase 19: vectorized Filter now evaluates WHERE/HAVING predicate trees as paired `is_true`/`is_known` byte masks using SQL Kleene 3VL algebra, with comparison leaves scanning pre-resolved int64 or string values column-at-a-time and `IS [NOT] NULL` leaves reading validity directly. Scan-adjacent identity selections use a dense full-batch domain, while filtered/non-identity inputs use dense selection-position masks that map back to the immutable input selection. Join residual predicates deliberately remain on the existing row-pair path. Release benchmark evidence in `docs/benchmarks.md` shows the three scan/filter vectorized medians improving with byte-identical correctness checks and no material non-target regression.
2222
20. LEFT/RIGHT OUTER JOIN arc. Delivered across Phase 20a/20b: the parser accepts `LEFT [OUTER] JOIN` and `RIGHT [OUTER] JOIN`; binding normalizes RIGHT to LEFT while stable identities preserve SQL-visible projection order; both interpreted and vectorized engines implement deterministic left-row-major NULL-extension, NULL-key non-matching, and residual-aware matchedness for hash and non-equi joins. `LeftJoinToInnerRule` additively simplifies a filtered LEFT join only under a proof-bearing conservative 3VL null-rejection test, unlocking the existing INNER commute/associate/pushdown exploration without weakening their guards. The deterministic fuzzer now mixes INNER/LEFT/RIGHT chains with actual NULL-bearing key data and verifies standalone rewrites, every memo alternative, and `extract_best` through both engines. The Phase 20a LEFT cardinality rule remains sufficient; no 20b cost-model change was required. The outer-join lowering guard and its verifier path are gone.
23-
21. Uncorrelated subqueries. Phase 21a adds scalar-comparison, IN/NOT IN, and EXISTS/NOT EXISTS parsing and fresh-scope binding, embeds immutable bound logical subplans in expression leaves, and defines scalar cardinality plus complete IN/EXISTS NULL and empty-set semantics in the interpreted oracle. Memo identity hashes and compares opaque subplan trees without exploring them, EXPLAIN renders them under their owner, costing charges them once, and physical lowering rejects them with one exact vectorized-not-supported guard. Deterministic goldens and an oracle-only differential path cover nested, join, aggregate, NULL, empty, LIMIT 0, correlation-error, type-error, and scalar-cardinality cases. Fuzzer generation remains deliberately unchanged in 21a.
24-
22. Subquery decorrelation and execution (Phase 21b). Convert supported subquery forms into explicit semi/anti-join and related scalar algebra with proof-bearing NULL/bag-semantics guards, then add vectorized physical support and deterministic seeded fuzzer generation. Phase 21b revisits the 21a memo-opacity decision only after decorrelated operators make equivalence and costing explicit; correlated subqueries remain out of scope unless a later phase specifies their semantics and scope model.
23+
21. Complete uncorrelated-subquery arc. Delivered across Phase 21a/21b: scalar-comparison, IN/NOT IN, and EXISTS/NOT EXISTS use fresh-scope immutable bound subplans and the pinned eager once-per-query 3VL/cardinality oracle. Explicit left SEMI/ANTI algebra has left-only output identity, deterministic interpreted and vectorized hash/NLJ execution, memo identity/printers/EXPLAIN/costing, and conservative transform guards. Top-level Filter EXISTS, NOT EXISTS, and IN decorrelate through proof-bearing memo rules; OR-contained leaves, NOT IN, and scalar forms stay materialized. Residual vectorized subqueries recursively execute once through the vectorized engine, scalar results compile as nullable constants, and IN sets use lookup-only typed hashes plus empty/NULL flags. Exact EXPLAIN goldens, a ten-mode 160-seed full-stack fuzzer, eager error equivalence, and a 100k-row selective-IN benchmark close the arc; the temporary lowering guard and oracle-only verifier path are gone.
24+
22. Correlated subqueries (future). Define an explicit outer-reference scope model and correlation algebra before adding binding or execution. Phase 21 deliberately rejects correlated names; no decorrelation rule may infer correlation from unresolved identifiers.
2525

2626
## Phase 1 first tasks
2727

‎benchmarks/sql_bench.cpp‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
#include "execution/interpreter.hpp"
22
#include "execution/vectorized.hpp"
3+
#include "optimizer/memo.hpp"
4+
#include "optimizer/rewrite.hpp"
35
#include "sql/binder.hpp"
46

57
#include <algorithm>
@@ -304,6 +306,24 @@ plan::LogicalPlan bind_query(const execution::Catalog& catalog, const std::strin
304306
return sql::bind_select(sql::parse_select(sql), catalog);
305307
}
306308

309+
plan::LogicalPlan bind_decorrelated_semi_query(const execution::Catalog& catalog, const std::string& sql) {
310+
const auto logical = bind_query(catalog, sql);
311+
optimizer::Memo memo;
312+
const auto root = memo.insert(logical);
313+
const auto explored = optimizer::explore_memo_to_fixpoint(memo, optimizer::default_memo_rules());
314+
if (!explored.reached_fixpoint) {
315+
throw std::logic_error("benchmark semi-join memo exploration did not reach fixpoint");
316+
}
317+
const auto alternatives =
318+
memo.extract_alternatives(root, optimizer::AlternativeExtractionOptions{128, 1024});
319+
for (const auto& alternative : alternatives.plans) {
320+
if (plan::to_string(alternative).find("SemiJoin[") != std::string::npos) {
321+
return alternative;
322+
}
323+
}
324+
throw std::logic_error("benchmark IN query did not produce a SemiJoin alternative");
325+
}
326+
307327
std::vector<Workload> make_workloads(const execution::Catalog& catalog) {
308328
std::vector<Workload> workloads;
309329
auto add = [&](std::string name, std::string rows, std::string sql) {
@@ -323,6 +343,15 @@ std::vector<Workload> make_workloads(const execution::Catalog& catalog) {
323343
"left=100000,right=256",
324344
"SELECT l.payload AS left_payload, r.payload AS right_payload "
325345
"FROM join_left AS l JOIN join_right AS r ON l.k1 = r.k1 AND l.k2 = r.k2");
346+
{
347+
const std::string sql =
348+
"SELECT l.payload AS payload FROM join_left AS l "
349+
"WHERE l.k1 IN (SELECT r.k1 FROM join_right AS r WHERE r.payload = 10000)";
350+
workloads.push_back(Workload{"selective_in_semi_join",
351+
"outer=100000,subquery_source=256,subquery_rows=1",
352+
sql,
353+
bind_decorrelated_semi_query(catalog, sql)});
354+
}
326355
add("aggregate_few_groups",
327356
"fact=200000,groups=8",
328357
"SELECT group_few, SUM(value) AS total, COUNT(*) AS n "

‎docs/benchmarks.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -217,3 +217,21 @@ to favorable except for small movements in `aggregate_few_groups` (14.717 ms to
217217
exercise vectorized Filter in this benchmark set, so they are reported as
218218
run-to-run noise rather than attributed wins. Correctness checksums stayed
219219
byte-identical for every workload.
220+
221+
## Phase 21b Selective Semi Join (2026-07-10)
222+
223+
Same optimized build, five-repetition min/median, deterministic-data, and
224+
checksum-before-timing methodology as above. The SQL source is an IN query over
225+
the 100,000-row `join_left` table. The benchmark harness requires and times the
226+
memo-produced equi-SemiJoin alternative, whose subquery scans 256 rows and
227+
selects one key (`r.payload = 10000`). The interpreted and vectorized engines
228+
execute the same decorrelated plan.
229+
230+
| workload | rows | correctness | interpreted min ms | interpreted median ms | vectorized min ms | vectorized median ms | median speedup |
231+
|---|---:|---|---:|---:|---:|---:|---:|
232+
| selective_in_semi_join | outer=100000,subquery_source=256,subquery_rows=1 | match rows=6250 checksum=0x72a0d9bc69515c7e | 11.122 | 11.865 | 5.131 | 5.157 | 2.301x |
233+
234+
The checksum match pins left-only output and duplicate-preserving Semi
235+
semantics before timing. The vectorized lookup-only hash implementation is
236+
2.301x faster on median than the interpreted left-row-major nested-loop oracle
237+
for this selective workload.

‎include/optimizer/rewrite.hpp‎

Lines changed: 66 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,64 @@ class MergeAdjacentFiltersRule final : public Rule, public MemoRule {
142142
[[nodiscard]] bool apply(Memo& memo, GroupId group, const MemoExpression& expression) const override;
143143
};
144144

145+
// Pattern matched: a Filter with a top-level EXISTS subquery conjunct.
146+
// Replacement expression: remove that conjunct and place the Filter input on
147+
// the preserved side of a keyless SemiJoin whose right side is the subquery;
148+
// keep any residual top-level conjuncts above the join.
149+
// Semantic equivalence: EXISTS is TRUE exactly when its independently bound
150+
// right plan has a row. A keyless SemiJoin treats its empty predicate list as
151+
// TRUE for every pair and emits a left row once iff any right row exists, so it
152+
// preserves bag multiplicity and left order exactly.
153+
// Preconditions: only an entire top-level Filter conjunct is matched; EXISTS
154+
// inside AND/OR predicate trees is not inspected or split. The subquery is
155+
// uncorrelated, immutable, pure, and non-null. The rule removes one subquery
156+
// leaf from the owning Filter and never creates one, while structural memo
157+
// deduplication closes repeated exploration.
158+
// Golden query: `SELECT a FROM t WHERE EXISTS (SELECT a FROM t1)`.
159+
class ExistsToSemiJoinRule final : public MemoRule {
160+
public:
161+
[[nodiscard]] std::string_view name() const override { return "ExistsToSemiJoinRule"; }
162+
[[nodiscard]] bool apply(Memo& memo, GroupId group, const MemoExpression& expression) const override;
163+
};
164+
165+
// Pattern matched: a Filter with a top-level NOT EXISTS subquery conjunct.
166+
// Replacement expression: remove that conjunct and place the Filter input on
167+
// the preserved side of a keyless AntiJoin to the subquery, retaining residual
168+
// top-level conjuncts above it.
169+
// Semantic equivalence: NOT EXISTS is TRUE exactly when the right plan is
170+
// empty. Keyless AntiJoin emits each left row once iff no TRUE pair exists;
171+
// right values and NULLs are irrelevant, so no NULL trap exists.
172+
// Preconditions: the predicate itself must be one top-level Filter conjunct;
173+
// trees below AND/OR are opaque. The independently bound subquery is immutable,
174+
// pure, and non-null. Each application removes one eligible leaf and memo
175+
// structural deduplication guarantees finite exploration.
176+
// Golden query: `SELECT a FROM t WHERE NOT EXISTS (SELECT a FROM empty)`.
177+
class NotExistsToAntiJoinRule final : public MemoRule {
178+
public:
179+
[[nodiscard]] std::string_view name() const override { return "NotExistsToAntiJoinRule"; }
180+
[[nodiscard]] bool apply(Memo& memo, GroupId group, const MemoExpression& expression) const override;
181+
};
182+
183+
// Pattern matched: a Filter with a top-level `x IN (SELECT c ...)` conjunct.
184+
// Replacement expression: remove that conjunct and add SemiJoin(input,
185+
// subquery, x = c), retaining residual top-level conjuncts above the join.
186+
// Semantic equivalence: under TRUE-only filtering, IN survives iff at least
187+
// one non-NULL c makes x=c TRUE. Equi-SemiJoin has exactly that match set and
188+
// skips NULL equality keys. When x is NULL, or no value matches but the set has
189+
// a NULL, IN is UNKNOWN while SemiJoin reports no match; both reject the left
190+
// row. Right duplicates cannot multiply a Semi result.
191+
// Preconditions: the IN leaf must be a whole top-level Filter conjunct; IN
192+
// inside OR/AND is never split. NOT IN is excluded because a NULL-bearing set
193+
// makes NOT IN never TRUE while plain AntiJoin would emit unmatched rows.
194+
// Scalar subquery operands are excluded and remain eagerly materialized. The
195+
// subquery has exactly one typed output by binder invariant.
196+
// Golden query: `SELECT a FROM t WHERE a IN (SELECT a FROM t1)`.
197+
class InToSemiJoinRule final : public MemoRule {
198+
public:
199+
[[nodiscard]] std::string_view name() const override { return "InToSemiJoinRule"; }
200+
[[nodiscard]] bool apply(Memo& memo, GroupId group, const MemoExpression& expression) const override;
201+
};
202+
145203
// Pattern matched: Filter(P, LeftJoin(L, R, J)) where at least one whole top-level conjunct in P
146204
// is provably null-rejecting for R's binding identities.
147205
// Replacement expression: Add Filter(P, InnerJoin(L, R, J)) as an equivalent expression in the
@@ -188,7 +246,14 @@ class LeftJoinToInnerRule final : public MemoRule {
188246
// predicate because it is evaluated over the same row pair at a point where both rows are available;
189247
// NULL operands yield UNKNOWN and are rejected in either placement. Whole-tree reference analysis
190248
// prevents unsound OR/AND splitting.
191-
// Preconditions: The child join must be an INNER join. LEFT joins are skipped because pushing a
249+
// Preconditions: The child join must be an INNER join, except that a whole
250+
// left-only conjunct may move into the preserved child of a SEMI/ANTI join.
251+
// This exception is sound because Semi/Anti output is a subset of left rows:
252+
// filtering a left row before testing match existence keeps exactly the rows
253+
// that would pass the same predicate after the existence test. Right-only,
254+
// mixed-side, literal-only, and empty-reference conjuncts remain above a
255+
// SEMI/ANTI join; no predicate is moved into its match condition or right side.
256+
// LEFT joins are skipped because pushing a
192257
// predicate into the null-supplying side, or turning a post-join predicate into an ON predicate,
193258
// can preserve NULL-extended rows that the original WHERE would reject or reject rows the original
194259
// ON would preserve. LeftJoinToInnerRule may first establish an equivalent filtered INNER shape;

‎include/plan/logical_plan.hpp‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ struct LogicalPlan;
1818

1919
enum class LogicalKind { Scan, Join, Filter, Project, Aggregate, Distinct, Sort, Limit, Explain };
2020
enum class OrderPermission { Deterministic, Arbitrary };
21-
enum class JoinKind { Inner, Left };
21+
enum class JoinKind { Inner, Left, Semi, Anti };
2222

2323
struct BoundColumnRef {
2424
std::string binding;
@@ -176,10 +176,12 @@ struct LogicalPlan {
176176
return p;
177177
}
178178

179-
// Join output identity/order is deterministic: all columns from the left
180-
// child followed by all columns from the right child. Expressions in a
181-
// bound logical plan refer to those identities by binding and column name;
182-
// downstream layers must not re-resolve parsed SQL names.
179+
// INNER/LEFT output identity/order is deterministic: all columns from the
180+
// left child followed by all columns from the right child. SEMI/ANTI emit
181+
// exactly the left child's identities and order; right identities exist
182+
// only while evaluating join predicates. Expressions in a bound logical
183+
// plan refer to identities by binding and column name; downstream layers
184+
// must not re-resolve parsed SQL names.
183185
static LogicalPlan join(std::vector<BoundPredicate> predicates,
184186
LogicalPlan left,
185187
LogicalPlan right,

‎include/plan/physical_plan.hpp‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,9 +10,6 @@
1010

1111
namespace plan {
1212

13-
inline constexpr auto kVectorizedSubqueryNotSupported =
14-
"vectorized physical lowering does not support subqueries";
15-
1613
enum class PhysicalKind { Scan, Join, Filter, Project, Aggregate, Distinct, Sort, Limit };
1714

1815
struct PhysicalPlan {

‎src/execution/interpreter.cpp‎

Lines changed: 24 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -814,24 +814,32 @@ storage::ColumnarBatch execute_join(const plan::LogicalPlan& plan, ExecutionCont
814814
const auto right = execute_node(require_right(plan), context);
815815

816816
std::vector<std::string> output_names = left.column_names();
817-
output_names.insert(output_names.end(), right.column_names().begin(), right.column_names().end());
817+
const auto emits_right_columns =
818+
plan.join_kind == plan::JoinKind::Inner || plan.join_kind == plan::JoinKind::Left;
819+
if (emits_right_columns) {
820+
output_names.insert(output_names.end(), right.column_names().begin(), right.column_names().end());
821+
}
818822
std::vector<OutputColumn> output_columns(output_names.size());
819823
for (std::size_t i = 0; i < left.column_names().size(); ++i) {
820824
output_columns[i].type = left.column_type(left.column_names()[i]);
821825
}
822-
for (std::size_t i = 0; i < right.column_names().size(); ++i) {
823-
output_columns[left.column_names().size() + i].type = right.column_type(right.column_names()[i]);
826+
if (emits_right_columns) {
827+
for (std::size_t i = 0; i < right.column_names().size(); ++i) {
828+
output_columns[left.column_names().size() + i].type = right.column_type(right.column_names()[i]);
829+
}
824830
}
825831

826832
auto append_output_row = [&](std::size_t left_row, std::optional<std::size_t> right_row) {
827833
std::size_t output_index = 0;
828834
for (const auto& column_name : left.column_names()) {
829835
output_columns[output_index++].append(cell_at(left, column_name, left_row));
830836
}
831-
for (const auto& column_name : right.column_names()) {
832-
output_columns[output_index++].append(right_row.has_value()
833-
? cell_at(right, column_name, *right_row)
834-
: null_cell(right.column_type(column_name)));
837+
if (emits_right_columns) {
838+
for (const auto& column_name : right.column_names()) {
839+
output_columns[output_index++].append(right_row.has_value()
840+
? cell_at(right, column_name, *right_row)
841+
: null_cell(right.column_type(column_name)));
842+
}
835843
}
836844
};
837845

@@ -847,10 +855,19 @@ storage::ColumnarBatch execute_join(const plan::LogicalPlan& plan, ExecutionCont
847855
}
848856

849857
matched = true;
858+
if (plan.join_kind == plan::JoinKind::Semi) {
859+
append_output_row(left_row, std::nullopt);
860+
break;
861+
}
862+
if (plan.join_kind == plan::JoinKind::Anti) {
863+
break;
864+
}
850865
append_output_row(left_row, right_row);
851866
}
852867
if (!matched && plan.join_kind == plan::JoinKind::Left) {
853868
append_output_row(left_row, std::nullopt);
869+
} else if (!matched && plan.join_kind == plan::JoinKind::Anti) {
870+
append_output_row(left_row, std::nullopt);
854871
}
855872
}
856873

0 commit comments

Comments
 (0)