Add cast_map to scan_db for server-side column narrowing

[ptomecek]

Summary

Adds an optional cast_map argument to scan_db that casts selected output columns server-side, so filters on a narrowed column push down to the database instead of being evaluated client-side.

Motivation

A Polars IO plugin only receives (with_columns, predicate) from the optimizer. When downstream code casts a scan_db column to a narrower type (for example a column stored as datetime that is logically a date) and then filters on it, Polars will not push the predicate through the type-changing cast — the plugin sees predicate=None, so the filter runs client-side after a full table scan.

cast_map moves the narrowing into the query itself, so the predicate stays pushable.

What it does

scan_db(query, conn, cast_map={"event_ts": pl.Date})

• Wraps the query in an auto-enumerated projecting subquery that CASTs the named columns and passes the rest through, so select * still flows every column (including new upstream ones).
• Reports the target dtype in the schema, so a filter on the cast column pushes down as a normal predicate.

Emitted SQL (T-SQL example):

SELECT ...
FROM (SELECT CAST([event_ts] AS DATE) AS [event_ts], [other_col], ...
      FROM (<original query>) AS __cpl_cast) AS __cpl_subq
WHERE ([event_ts] = '2025-05-30')

Correctness details

• Clauses illegal inside a derived table (a top-level ORDER BY without TOP/OFFSET, or OPTION hints in the T-SQL dialect) are hoisted onto the cast SELECT so the nested subquery stays valid.
• Unsupported target dtypes are rejected with a clear error rather than silently emitting VARCHAR while reporting the requested dtype.
• Decimal precision and scale are preserved.
• A shared polars_dtype_to_sqlglot_type helper is extracted and used by both the predicate translator and the cast wrapping.

Testing

• New tests cover schema reporting, server-side narrowing, date-predicate pushdown, projection pushdown, select * preservation, unknown-column and unsupported-dtype errors, and MSSQL ORDER BY hoisting.
• Full io_sources suite passes; ruff check and ruff format --check are clean.

[github-actions]

Test Results

1 425 tests  +9   1 377 ✅ +9   41s ⏱️ +8s
    2 suites ±0      48 💤 ±0 
    2 files   ±0       0 ❌ ±0 

Results for commit 9ab9dbe. ± Comparison against base commit 4c54917.

♻️ This comment has been updated with latest results.

[NeejWeej]

One concern with casting in SQL is that filters may end up operating on the casted expression—for example:

WHERE CAST(RecordDate AS date) = @date

Applying a function to the filtered column can interfere with index use. Microsoft, for example, explicitly calls out CAST and CONVERT in predicates in its section on [investigating SARGability issues]

This seems similar to the problem already handled by scan_delta: the source has one physical type, while Polars exposes another. It exposes the logical schema, translates predicates toward the physical representation for pushdown, and then converts the output locally.

I wonder if we could use the same pattern here. A timestamp exposed as Date could translate:

pl.col("RecordDate") == date(2025, 5, 30)

into:

WHERE RecordDate >= @start
  AND RecordDate < @next_day

The matching values could then be converted locally, potentially through arrow_odbc’s fetch schema. The current server-side cast is simpler, but this might give us the desired output type while preserving better query plans.

[ptomecek]

Good point, and you're right about the SARGability risk in the general case — wrapping the column in CAST(...) can stop the optimizer from using an index on it.

A couple of reasons I've kept the server-side CAST as the mechanism here rather than a logical/physical range translation:

• It has to stay general. The motivating case is datetime → date, but there are others that a range rewrite doesn't cover — e.g. a numeric id delivered as float that should be an integer. The half-open-interval trick is specific to temporal narrowing, so it wouldn't help those, whereas a plain CAST handles any narrowing uniformly.
• The plan impact is backend dependent, not uniformly bad. Some engines optimize particular conversions — SQL Server, for instance, can still seek on CAST(datetime AS date) — while others fall back to a scan. So the cost depends on the dialect and the specific cast.

Given that, I've gone with keeping cast_map general and documenting the caveat instead: the scan_db docstring and the wiki now spell out that a predicate on a cast column pushes down over its CAST, that this can affect index use, and that the effect is backend dependent — with a note to filter the physical column directly on hot indexed paths.

The range-translation approach you describe (mirroring scan_delta's logical-vs-physical handling, which the range_visitor interval machinery already supports) is a genuinely nicer plan for the temporal case, and I think it's worth doing as a follow-up specialization on top of this — expose date, translate the predicate to a half-open datetime range, and convert the fetched column locally. I'd rather land the general primitive first and layer that optimization on afterward. Does that split seem reasonable to you?

[NeejWeej]

Good point, and you're right about the SARGability risk in the general case — wrapping the column in CAST(...) can stop the optimizer from using an index on it.

A couple of reasons I've kept the server-side CAST as the mechanism here rather than a logical/physical range translation:

• It has to stay general. The motivating case is datetime → date, but there are others that a range rewrite doesn't cover — e.g. a numeric id delivered as float that should be an integer. The half-open-interval trick is specific to temporal narrowing, so it wouldn't help those, whereas a plain CAST handles any narrowing uniformly.
• The plan impact is backend dependent, not uniformly bad. Some engines optimize particular conversions — SQL Server, for instance, can still seek on CAST(datetime AS date) — while others fall back to a scan. So the cost depends on the dialect and the specific cast.

Given that, I've gone with keeping cast_map general and documenting the caveat instead: the scan_db docstring and the wiki now spell out that a predicate on a cast column pushes down over its CAST, that this can affect index use, and that the effect is backend dependent — with a note to filter the physical column directly on hot indexed paths.

The range-translation approach you describe (mirroring scan_delta's logical-vs-physical handling, which the range_visitor interval machinery already supports) is a genuinely nicer plan for the temporal case, and I think it's worth doing as a follow-up specialization on top of this — expose date, translate the predicate to a half-open datetime range, and convert the fetched column locally. I'd rather land the general primitive first and layer that optimization on afterward. Does that split seem reasonable to you?

Makes sense, sounds great!