Add IntervalFilterSpec for validity-interval pushdown - #40
Merged
Conversation
Contributor
hintse
force-pushed
the
feat/interval-filter-spec
branch
4 times, most recently
from
September 4, 2026 09:21
07bc1e1 to
5c0833a
Compare
FilterSpec maps one output column to one source column, which cannot express the overlap predicate needed for validity-interval sources where each row is valid over [start, end]. IntervalFilterSpec pushes the correct overlap (start_col <= hi AND end_col >= lo, adjusted for `closed`) onto the source, mapping the request bounds through value_mapping and optionally pruning windows longer than max_span. The overlap is resolved entirely at the source row, so surviving rows are exactly the overlapping windows. The post-combine predicate is now restricted (via the existing restrict_expr_to_columns) to columns present in the output, so an interval spec's virtual request column does not leak into the source-level filter. This is a no-op for FilterSpec, whose output column combine produces. Signed-off-by: Hin Tse <5867507+hintse@users.noreply.github.com>
hintse
force-pushed
the
feat/interval-filter-spec
branch
from
September 4, 2026 09:29
5c0833a to
68ebbb3
Compare
hintse
marked this pull request as ready for review
September 4, 2026 11:22
ptomecek
reviewed
Sep 4, 2026
ptomecek
left a comment
Collaborator
There was a problem hiding this comment.
Nice addition — the two one-sided range predicates are exactly what pushdown can consume, and the Date-source path looks correct for all four closed modes. One correctness issue on Datetime sources with the half-open modes (details inline), plus a couple of smaller robustness notes. It'd also be good to land at least one in-repo test for IntervalFilterSpec so the module's contract is guarded here rather than only downstream.
hintse
force-pushed
the
feat/interval-filter-spec
branch
from
September 4, 2026 14:22
68ebbb3 to
7e00b2f
Compare
- Fix half-open closed modes on Datetime sources: a date request bound is
now resolved to a full-day datetime comparison point directly, so
left/right/none no longer shift the bound a full day and drop windows
that overlap the boundary day. Endpoint open/closed-ness only selects
the operator.
- Validate closed in __post_init__ (ValueError for anything outside
{both,left,right,none}) so a typo fails fast instead of a late KeyError.
- Document conservative behavior: disjoint requests collapse to their
outer hull, and a dict value_mapping missing a bound leaves that side
unconstrained (over-selects, never drops an overlapping row).
- Add upstream tests: defaults, closed validation, overlap, pushdown
verification, closed x {Date, Datetime} incl. intraday boundary, and
the hull-collapse case.
Signed-off-by: Hin Tse <5867507+hintse@users.noreply.github.com>
hintse
force-pushed
the
feat/interval-filter-spec
branch
from
September 4, 2026 14:40
7e00b2f to
308e717
Compare
ptomecek
approved these changes
Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
FilterSpecmaps one output column to one source column, so it cannot express the overlap predicate needed for validity-interval / history-of-change sources, where each row is valid over[start, end]and a request date range[lo, hi]selects covering windows viastart <= hi AND end >= lo.This adds a first-class
IntervalFilterSpecthat pushes the correct overlap predicate.What it does
For a request range
[lo, hi]extracted from the pushed predicate, it filters the source withend_col >= loandstart_col <= hi(operators adjusted forclosed), mapping the bounds throughvalue_mappingfirst. The overlap is resolved entirely at the source row, so the surviving rows are exactly the overlapping windows and the range predicates push through to SQL/TickStore.Date bounds compared against a
Datetimesource column reuse the existing_extend_dates_to_full_datetimeswidening, so intraday rows on the boundary day are not silently dropped.Notable shared-path change
The post-combine
result_lf.filter(predicate)is now restricted (via the existingrestrict_expr_to_columns) to columns present in the combined output before being applied. An interval spec's request/output column (e.g.date) is supplied by another joined frame and is not a column of the interval source itself, so restricting keeps it from leaking into the source-level filter. This is a no-op forFilterSpec, whose output columncombineproduces.Testing
test_pushdown_combine.py: 92 passed (no regression). Consumer-facing tests (overlap correctness, pushdown verification,closedvariants,value_mapping, Datetime sources, missing columns) live downstream.🤖 Generated with Claude Code