Skip to content

Commit 2b73435

Browse files
rustyconoverclaude
andcommitted
fix(filter-v2): a boolean column is a predicate on its own
`WHERE flag` and `WHERE NOT flag` were rejected at decode with "predicate root must resolve to BOOLEAN". DuckDB pushes such a predicate down as a bare `column_ref` rather than rewriting it to `flag = true`, and the schema admits that -- `coreExpression` lists `columnRef` first -- so the decoder was stricter than its own published contract, and the idiomatic form of the commonest boolean predicate there is failed for every worker. Three things, each independently wrong: - `FilterParser._is_boolean` enumerated node *kinds* believed to be boolean instead of asking for the resolved *type*. The two agree for every node whose type is structurally fixed -- a comparison is BOOLEAN whatever its operands -- which is why the enumeration looked sufficient. It is not: `column_ref`, `field_ref`, `cast` and `arithmetic` are BOOLEAN or not depending on a type resolved from elsewhere, and all four were refused. The parser had *already* resolved the column against the bind output schema and stored the type on the node; a sibling helper written in the same commit, `expression_type()`, reads it correctly and is used one branch away to type-check `field_ref`. Call it here too. - A bare boolean column left as a generic v2 expression becomes a `V2ExpressionFilter`, which evaluates but renders no SQL -- so a SQL-backed worker silently lost pushdown on it, and on a conjunction containing one it lost pushdown for the whole conjunction. `_v2_filter_view` now projects `flag` and `NOT flag` onto the legacy equality leaf. The rewrite is exact under three-valued logic: `WHERE flag` keeps only TRUE and so does `flag = true`; `WHERE NOT flag` keeps only FALSE and so does `flag = false`. An unresolved or non-boolean column is left alone rather than guessed at. - `to_sql()` raised a bare `ValueError: Unknown filter type` on anything the legacy projection could not reduce. It now raises `FilterSQLUnsupportedError` naming the node and pointing at `apply()`. It still raises rather than skipping, deliberately: DuckDB does **not** re-apply a predicate it pushed into a table function -- measured, not assumed: a scan that ignores its WHERE returns 3255 rows where the predicate admits 1874 -- so a quietly dropped filter is a wrong answer, not a slow one. Nothing covered this. The cross-port conformance corpus has no bare boolean column case, and the extension's own `filter_pushdown/booleans.test` -- added to close a boolean coverage gap -- writes every predicate as `b = true`, `b != true` or `b IN (...)`, never `WHERE b`. Tests added here for both forms, for a conjunction containing one, for the SQL they must render, for the refusal of an unresolved or non-boolean column, and for the no-silent- drop guarantee. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 9aa90f6 commit 2b73435

3 files changed

Lines changed: 263 additions & 7 deletions

File tree

‎tests/test_filter_v2.py‎

Lines changed: 166 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66

77
import json
88
from collections.abc import Mapping
9-
from typing import cast
9+
from typing import Any, cast
1010

1111
import pyarrow as pa
1212
import pytest
@@ -531,3 +531,168 @@ def test_unknown_arrow_extension_metadata_is_rejected() -> None:
531531
_batch(_document([_predicate(_comparison())]), (extension_field, b"opaque")),
532532
output_schema=pa.schema([("n", pa.int64())]),
533533
)
534+
535+
536+
# ---------------------------------------------------------------------------
537+
# A boolean column is a predicate on its own
538+
#
539+
# `WHERE flag` / `WHERE NOT flag` is idiomatic SQL, and DuckDB pushes it down as
540+
# a bare `column_ref` rather than rewriting it to `flag = true`. The schema
541+
# admits that -- `coreExpression` lists `columnRef` first -- but the decoder's
542+
# boolean gate used to enumerate node *kinds* instead of asking for the resolved
543+
# *type*, so it refused every one of them: "predicate root must resolve to
544+
# BOOLEAN", on a node whose type it had already resolved to BOOLEAN and stored.
545+
# ---------------------------------------------------------------------------
546+
547+
_BOOL_FIELDS: list[pa.Field[Any]] = [pa.field("flag", pa.bool_()), pa.field("n", pa.int64())]
548+
_BOOL_SCHEMA = pa.schema(_BOOL_FIELDS)
549+
550+
551+
def _flag(index: int = 0) -> dict[str, object]:
552+
return {"node": "column_ref", "column_index": index, "column_name": "flag"}
553+
554+
555+
def test_bare_boolean_column_is_a_valid_predicate_root() -> None:
556+
"""`WHERE flag` -- the column itself is the predicate."""
557+
filters = deserialize_filters(
558+
_batch(_document([_predicate(_flag())])),
559+
output_schema=_BOOL_SCHEMA,
560+
)
561+
batch = pa.RecordBatch.from_pydict(
562+
{"flag": [True, False, None, True], "n": [1, 2, 3, 4]},
563+
schema=_BOOL_SCHEMA,
564+
)
565+
# evaluate() is the raw three-valued predicate: NULL stays NULL.
566+
assert filters.evaluate(batch).to_pylist() == [True, False, None, True]
567+
# apply() is what selects rows, and there a NULL predicate is not satisfied
568+
# -- the same rows `flag = true` would keep, which is what makes the SQL
569+
# rewrite below a projection rather than a change of meaning.
570+
assert filters.apply(batch).to_pydict()["n"] == [1, 4]
571+
572+
573+
def test_negated_boolean_column_is_a_valid_predicate_root() -> None:
574+
"""`WHERE NOT flag` -- `not` over a bare column, which the gate also refused."""
575+
filters = deserialize_filters(
576+
_batch(_document([_predicate({"node": "not", "expression": _flag()})])),
577+
output_schema=_BOOL_SCHEMA,
578+
)
579+
batch = pa.RecordBatch.from_pydict(
580+
{"flag": [True, False, None, True], "n": [1, 2, 3, 4]},
581+
schema=_BOOL_SCHEMA,
582+
)
583+
assert filters.evaluate(batch).to_pylist() == [False, True, None, False]
584+
# NOT NULL is NULL, so row 3 is not kept -- exactly `flag = false`.
585+
assert filters.apply(batch).to_pydict()["n"] == [2]
586+
587+
588+
def test_boolean_column_predicates_render_as_sql() -> None:
589+
"""They must reach SQL, not just the evaluator.
590+
591+
Left as a generic v2 expression a bare column renders no SQL at all, so a
592+
SQL-backed worker loses pushdown on the commonest boolean predicate there
593+
is -- and DuckDB does not re-apply what it pushed down, so that is a wrong
594+
answer, not a slow one.
595+
"""
596+
positive = deserialize_filters(_batch(_document([_predicate(_flag())])), output_schema=_BOOL_SCHEMA)
597+
where, params = positive.to_sql()
598+
assert where == '"flag" = ?'
599+
assert params == [True]
600+
601+
negative = deserialize_filters(
602+
_batch(_document([_predicate({"node": "not", "expression": _flag()})])),
603+
output_schema=_BOOL_SCHEMA,
604+
)
605+
where, params = negative.to_sql()
606+
assert where == '"flag" = ?'
607+
assert params == [False]
608+
609+
610+
def test_boolean_column_inside_a_conjunction_still_pushes_down() -> None:
611+
"""The shape that actually showed up: `WHERE other = x AND NOT flag`.
612+
613+
One unrenderable child used to cost the whole conjunction its pushdown.
614+
"""
615+
document = _document(
616+
[
617+
_predicate(
618+
{
619+
"node": "and",
620+
"children": [
621+
{
622+
"node": "comparison",
623+
"op": "gt",
624+
"left": _column(name="n", index=1),
625+
"right": {"node": "literal", "value_ref": 0},
626+
},
627+
{"node": "not", "expression": _flag()},
628+
],
629+
}
630+
)
631+
]
632+
)
633+
filters = deserialize_filters(
634+
_batch(document, (pa.field("value_0", pa.int64()), 2)),
635+
output_schema=_BOOL_SCHEMA,
636+
)
637+
where, params = filters.to_sql()
638+
assert where == '("n" > ? AND "flag" = ?)'
639+
assert params == [2, False]
640+
641+
642+
def test_an_unresolved_column_is_not_assumed_boolean() -> None:
643+
"""Without a bind schema the column's type is unknown -- refuse, don't guess."""
644+
with pytest.raises(FilterDeserializationError, match="predicate root"):
645+
deserialize_filters(_batch(_document([_predicate(_flag())])))
646+
647+
648+
def test_a_non_boolean_column_is_still_refused() -> None:
649+
"""`WHERE n` where n is BIGINT is not a predicate."""
650+
with pytest.raises(FilterDeserializationError, match="predicate root"):
651+
deserialize_filters(
652+
_batch(_document([_predicate(_column(name="n", index=1))])),
653+
output_schema=_BOOL_SCHEMA,
654+
)
655+
656+
657+
def test_an_unrenderable_filter_raises_rather_than_vanishing() -> None:
658+
"""`to_sql()` must never quietly drop a predicate it cannot render.
659+
660+
DuckDB does not re-apply a filter it pushed into a table function, so the
661+
worker's WHERE clause is the only thing standing between the caller and
662+
rows the predicate excludes. A skipped filter is therefore a wrong answer,
663+
not a slow one -- which is why this raises instead of degrading.
664+
"""
665+
from vgi.table_filter_pushdown import FilterSQLUnsupportedError
666+
667+
# `n + 1 > 2` -- arithmetic has no legacy leaf shape and no SQL rendering.
668+
document = _document(
669+
[
670+
_predicate(
671+
{
672+
"node": "comparison",
673+
"op": "gt",
674+
"left": {
675+
"node": "arithmetic",
676+
"op": "add",
677+
"left": _column(name="n", index=1),
678+
"right": {"node": "literal", "value_ref": 0},
679+
},
680+
"right": {"node": "literal", "value_ref": 1},
681+
}
682+
)
683+
]
684+
)
685+
filters = deserialize_filters(
686+
_batch(
687+
document,
688+
(pa.field("value_0", pa.int64()), 1),
689+
(pa.field("value_1", pa.int64()), 2),
690+
),
691+
output_schema=_BOOL_SCHEMA,
692+
)
693+
# It evaluates correctly -- there is simply no SQL for it.
694+
batch = pa.RecordBatch.from_pydict({"flag": [True, True, True], "n": [0, 2, 5]}, schema=_BOOL_SCHEMA)
695+
assert filters.apply(batch).to_pydict()["n"] == [2, 5]
696+
697+
with pytest.raises(FilterSQLUnsupportedError, match="no SQL rendering"):
698+
filters.to_sql()

‎vgi/filter_v2.py‎

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -740,11 +740,24 @@ def _predicate(self, obj: dict[str, object], where: str, *, update: bool = False
740740

741741
@staticmethod
742742
def _is_boolean(expression: FilterExpression) -> bool:
743-
if isinstance(expression, Literal):
744-
return pa.types.is_boolean(expression.field.type)
745-
if isinstance(expression, (Comparison, BooleanExpression, Not, IsNull, In, RuntimeFilter)):
746-
return True
747-
return isinstance(expression, Call)
743+
"""Whether *expression* resolves to BOOLEAN.
744+
745+
Ask the type, never the node kind. The two agree for every node whose
746+
type is structurally fixed -- a comparison is BOOLEAN whatever its
747+
operands -- and that is exactly why enumerating kinds looked sufficient.
748+
It is not: ``column_ref``, ``field_ref``, ``cast`` and ``arithmetic``
749+
are BOOLEAN or not depending on a type resolved from elsewhere (the
750+
bind output schema, or a parent expression), and the schema admits all
751+
four as a predicate root -- ``coreExpression`` lists ``columnRef``
752+
first. Enumerating kinds rejected the lot, so the idiomatic
753+
``WHERE flag`` / ``WHERE NOT flag`` -- which DuckDB pushes down as a
754+
bare column reference rather than rewriting to ``flag = true`` -- was
755+
refused as "predicate root must resolve to BOOLEAN" even though the
756+
parser had already resolved the column to BOOLEAN and stored it on the
757+
node.
758+
"""
759+
data_type = expression_type(expression)
760+
return data_type is not None and pa.types.is_boolean(data_type)
748761

749762
def _payload(self, prefix: str, ref: object) -> tuple[pa.Field[Any], pa.Scalar[Any]]:
750763
index = _uint(ref, f"{prefix}_ref")

‎vgi/table_filter_pushdown.py‎

Lines changed: 79 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,9 @@
6363
from vgi.filter_v2 import (
6464
Literal as V2Literal,
6565
)
66+
from vgi.filter_v2 import (
67+
Not as V2Not,
68+
)
6669
from vgi.filter_v2 import (
6770
RuntimeFilter as V2RuntimeFilter,
6871
)
@@ -72,6 +75,9 @@
7275
from vgi.filter_v2 import (
7376
expression_columns as v2_expression_columns,
7477
)
78+
from vgi.filter_v2 import (
79+
expression_type as v2_expression_type,
80+
)
7581

7682
if TYPE_CHECKING:
7783
from collections.abc import Callable, Iterator
@@ -207,6 +213,7 @@ def _make_bool_array(value: bool, length: int) -> pa.BooleanArray:
207213
"OrFilter",
208214
"StructFilter",
209215
"ExpressionFilter",
216+
"FilterSQLUnsupportedError",
210217
"V2ExpressionFilter",
211218
# Expression node classes
212219
"ExpressionNode",
@@ -1638,6 +1645,22 @@ def __repr__(self) -> str:
16381645
# =============================================================================
16391646

16401647

1648+
class FilterSQLUnsupportedError(ValueError):
1649+
"""A filter that `to_sql()` cannot render as a SQL fragment.
1650+
1651+
Raised rather than skipped, and that is the whole point: DuckDB does **not**
1652+
re-apply a predicate it pushed into a table function, so a WHERE clause that
1653+
silently omits one does not return the same rows more slowly -- it returns
1654+
the wrong rows. Degrading quietly here would turn an unrenderable filter
1655+
into a wrong answer with no error anywhere.
1656+
1657+
A worker that catches this has two honest options: evaluate the filters in
1658+
memory instead (`PushdownFilters.apply()` / `evaluate()` handle every v2
1659+
expression), or decline the scan. Subclasses `ValueError` so existing
1660+
callers that catch the old error keep working.
1661+
"""
1662+
1663+
16411664
def _filter_to_sql(
16421665
f: Filter,
16431666
quote: Callable[[str], str],
@@ -1713,8 +1736,19 @@ def _filter_to_sql(
17131736
# Constants are embedded directly in the SQL string.
17141737
return expr.to_sql(f.column_name), []
17151738

1739+
case V2ExpressionFilter(expression=expression):
1740+
# A v2 expression the legacy projection could not reduce to a leaf
1741+
# shape (arithmetic, casts, extension calls, ...). It evaluates
1742+
# correctly -- `V2ExpressionFilter.evaluate()` runs the v2 reference
1743+
# evaluator -- there is simply no SQL rendering for it yet.
1744+
raise FilterSQLUnsupportedError(
1745+
f"filter on {f.column_name!r} is a v2 expression with no SQL rendering "
1746+
f"({type(expression).__name__}); evaluate it with PushdownFilters.apply() "
1747+
f"instead, or the scan will return rows the predicate excludes"
1748+
)
1749+
17161750
case _:
1717-
raise ValueError(f"Unknown filter type: {type(f)}")
1751+
raise FilterSQLUnsupportedError(f"Unknown filter type: {type(f)}")
17181752

17191753

17201754
# =============================================================================
@@ -1787,6 +1821,48 @@ def _pushdown_from_v2_state(state: V2FilterState) -> PushdownFilters:
17871821
)
17881822

17891823

1824+
def _v2_boolean_column_view(expression: V2FilterExpression) -> Filter | None:
1825+
"""Project ``WHERE flag`` / ``WHERE NOT flag`` onto the legacy equality leaf.
1826+
1827+
DuckDB pushes a predicate that *is* a boolean column down as a bare
1828+
``column_ref``, not as ``flag = true``. Left as a generic v2 expression it
1829+
becomes a `V2ExpressionFilter`, which evaluates fine but renders no SQL --
1830+
so every SQL-backed worker loses pushdown on the most ordinary boolean
1831+
predicate there is, and on the commonest shape of all, a conjunction with
1832+
one, it loses pushdown for the whole conjunction.
1833+
1834+
The rewrite is exact rather than approximate, including under NULLs:
1835+
``WHERE flag`` keeps only TRUE (a NULL predicate is not satisfied), and so
1836+
does ``flag = true``; ``WHERE NOT flag`` keeps only FALSE, and so does
1837+
``flag = false``. Three-valued logic makes the two pairs agree on every
1838+
input, which is what lets this be a projection and not a semantic change.
1839+
1840+
Args:
1841+
expression: A v2 expression that may be a bare boolean column or its
1842+
negation.
1843+
1844+
Returns:
1845+
The equivalent `ConstantFilter`, or None when *expression* is neither.
1846+
1847+
"""
1848+
1849+
def boolean_column(node: V2FilterExpression) -> V2ColumnRef | None:
1850+
if not isinstance(node, V2ColumnRef):
1851+
return None
1852+
node_type = v2_expression_type(node)
1853+
# An unresolved column (no bind schema) is not known to be boolean, and
1854+
# guessing here would rewrite a non-boolean predicate into `= true`.
1855+
if node_type is None or not pa.types.is_boolean(node_type):
1856+
return None
1857+
return node
1858+
1859+
if ref := boolean_column(expression):
1860+
return ConstantFilter(ref.column_name, ref.column_index, ComparisonOp.EQ, pa.scalar(True))
1861+
if isinstance(expression, V2Not) and (ref := boolean_column(expression.expression)):
1862+
return ConstantFilter(ref.column_name, ref.column_index, ComparisonOp.EQ, pa.scalar(False))
1863+
return None
1864+
1865+
17901866
def _v2_filter_view(expression: V2FilterExpression, context: EvaluationContext) -> Filter | None:
17911867
"""Project common v2 shapes into the longstanding convenience classes."""
17921868
if isinstance(expression, V2Comparison):
@@ -1802,6 +1878,8 @@ def _v2_filter_view(expression: V2FilterExpression, context: EvaluationContext)
18021878
}
18031879
if op := mapping.get(expression.op):
18041880
return ConstantFilter(operands[0].column_name, operands[0].column_index, op, operands[1].value)
1881+
if boolean_leaf := _v2_boolean_column_view(expression):
1882+
return boolean_leaf
18051883
if isinstance(expression, V2IsNull) and isinstance(expression.expression, V2ColumnRef):
18061884
ref = expression.expression
18071885
if expression.negated:

0 commit comments

Comments
 (0)