[VibeSWE]

Upstream origin

Ports parseablehq/parseable@af11d36e ("fix: manifest pruning (parseablehq#1828)"). It was cherry-picked with -x, so the first commit keeps upstream's authorship.

Bug: PartialTimeFilter::try_from_expr turned any comparison against a timestamp literal into a manifest-pruning bound. A filter on another column, e.g. created_at >= '2023-12-17T00:00:00', pruned day manifests as if it were a bound on p_timestamp and silently dropped matching rows from those days.

Deviation from upstream

Upstream accepts the bound only if the expression references p_timestamp. That is wrong for streams with a custom time_partition:

• Manifests are bucketed by the time-partition column. catalog::calculate_time_bound / create_manifest group files into day manifests by the min of time_partition (falling back to p_timestamp). Event::process keys staging files by the parsed partition timestamp. query::transform and enterprise::create_time_filter inject the query's time range on that column. Under upstream's check, that injected range no longer prunes anything, so every query on a time-partitioned stream reads every manifest.
• p_timestamp is ingestion time on those streams. It has no relation to the event-time buckets. Upstream still prunes by it, so p_timestamp >= T drops back-filled events whose event time is earlier than their ingestion time. That is the same class of bug the upstream commit fixes.

This PR adds a second commit that accepts a bound only when its left side is a bare column equal to time_partition.unwrap_or(p_timestamp). That is the column manifests are actually keyed by.

Requiring a bare column, instead of upstream's "any column ref" check, keeps pruning for the common query shapes. I checked these with DataFusion 55's optimizer: string literals, now() - interval, to_timestamp(...), BETWEEN, and simple casts. All of them become col <op> TimestampMillisecond(..) with no cast left on the column. Function-wrapped columns such as date_trunc('day', p_timestamp) = X do lose their own bound. The old code pruned on them, but that was unsafe in general: date_trunc('month', c) < '2023-12-16' matches rows up to 2023-12-31, yet it pruned everything from 12-16 on. The request range that is always injected still prunes these queries. time_partition cannot be changed after a stream is created (only its limit can), so the column manifests are keyed by stays the same.

Ramifications reviewed

• extract_primary_filter callers. These are StandardTableProvider::scan, enterprise::utils::fetch_parquet_file_paths (filters on the partition column, so now prunes correctly) and utils::arrow::flight::send_to_ingester (p_timestamp with None, unchanged).
• is_within_staging_window / return_listing_time_filters. Dropping a bound only widens the result (staging gets included, listing covers more), which is safe.
• get_manifest_list / counts build PartialTimeFilters directly, so they're unaffected.
• query::transform always injects the request range on the partition column. table_contains_any_time_filters inspects the unoptimized plan from create_logical_plan. DataFusion's SQL planner builds TableScans with empty filters there, so the function always returns false; it is effectively dead code. As a result, HTTP and alert queries on time-partitioned streams always carry an event-time bound and keep pruning, including queries whose only filter is on p_timestamp. An earlier version of this description said the opposite; that was wrong.
• Callers that bypass transform: a p_timestamp-only scan on a time-partitioned stream now gets no manifest pruning. It also stops sending the scan to the legacy listing path (pre-manifest data), because is_overlapping_query needs a lower bound to do that. This is pre-existing behaviour for any scan without a lower bound, and none of the HTTP or alert paths reach it.

Pre-existing issues found in review (not fixed here)

Five independent review agents (3× Opus 5.5, 2× GPT-6.1 Sol) reviewed this PR. They found no correctness bug introduced by it. They did confirm that manifests, staging files, listing prefixes and hot tier are all keyed by 1-minute slots of the partition column, so a file never spans two days. Pre-existing issues they found:

• Exact pushdown is not enforced everywhere. expr_in_boundary reports any minute-aligned timestamp comparison as Exact, on any column.
  • (a) collect_from_snapshot then truncates the file list for LIMIT by num_rows, assuming every row matches. WHERE created_at >= TIMESTAMP '…:00' LIMIT 100 can return fewer than 100 rows.
  • (b) The in-memory staging MemTable and the legacy ListingTable don't apply Exact filters, so unflushed rows can come back unfiltered.
  • Fixed in fix(query): enforce Exact filters on staging/listing scans and only report Exact for time-partition bounds #65.
• SQL-reachable panic on out-of-range millisecond literals in extract_timestamp_bound: fix(query): don't panic on out-of-range timestamp literals #64.
• Staging on time-partitioned streams. Back-filled events still in staging are invisible to event-time queries ending more than 5 minutes ago (is_within_staging_window, send_to_ingester).
• Log-context API builds its window from p_timestamp on time-partitioned streams.
• get_bin_density counts a file in two adjacent bins when its time falls on a bin edge.
• Distributed: Flight double-counts ingestor results (uploaded files plus staging copied into querier memory), and there is a memory→upload race.

Tests

New unit tests in stream_schema_provider::tests:

• a non-timestamp column (timestamp or string literal) does not prune
• a p_timestamp filter prunes, both bare and table-qualified
• a literal-on-left or != comparison does not prune
• a custom time_partition column prunes (timestamp and string literals)
• p_timestamp and unrelated columns do not prune a time-partitioned stream's manifests

With the column check disabled, the three "does not prune" tests fail, as expected.

• cargo fmt --check: clean
• cargo test (P_BUILD_NO_UI=1): 854 passed, 0 failed
• cargo clippy --all-targets: no warnings in the changed file. -D warnings fails on 21 warnings that are already on main, in files this PR doesn't touch.

🤖 Generated with Claude Code