Skip SQL pushdown for oversized IN-list predicates

[ptomecek]

Summary

Translating a very large is_in([...]) predicate into a SQL IN (...) clause can exceed a backend's expression-complexity limits. In particular, SQL Server raises error 8632 — "An expression services limit has been reached." Such oversized lists arise naturally when a join materializes one side and pushes its key set down to the other source as an is_in predicate.

Change

• Add DEFAULT_MAX_IN_PREDICATE_SIZE = 4096.
• In the predicate → SQL translation, when an is_in list is longer than the limit, skip translating it: the visitor returns None for that branch, and the existing AND-folding logic keeps any remaining (selective) predicates so they still push down.
• The limit is configurable via convert_predicate_to_sql(...) and apply_polars_io_source_exprs(...).

Why this is safe

Skipping pushdown for an oversized IN never changes results: the IO source already re-applies the full predicate to the fetched rows. Only the pre-filter narrowing (an optimization) is lost, and for the pathological case (a whole key universe) that narrowing was providing little benefit anyway, since co-pushed selective predicates already scope the scan.

Tests

Adds test_oversized_in_predicate_is_not_pushed_down:

• small IN still pushes down,
• oversized IN is dropped,
• oversized IN AND a selective equality keeps the equality pushdown,
• an explicit max_in_predicate_size override re-enables pushdown.

make lint and make format pass; existing SQL-reader tests pass.

[github-actions]

Test Results

1 393 tests  +1   1 345 ✅ +1   40s ⏱️ +2s
    2 suites ±0      48 💤 ±0 
    2 files   ±0       0 ❌ ±0 

Results for commit fddcbd0. ± Comparison against base commit 85adfbb.