WHERE on a non-partition timestamp column returns no rows or skips the filter #1826

/about output:

{"version":"v3.2.5","uiVersion":"v3.2.5","commit":"b8fb6ce","deploymentId":"01M5B8EGXJN2V7230HYWFEP7NJ","updateAvailable":false,"latestVersion":null,"oidcActive":false,"license":{"name":"AGPL-3.0-only","version":"v1","plan":"OSS","deploymentInfo":"Managed"},"mode":"Standalone","staging":"/staging","hotTier":"Disabled","grpcPort":8001,"store":{"type":"S3 bucket","path":"http://minio:9000/parseable"},"analytics":{"clarityTag":null}}

Summary

While playing with the self-host version exploring the latest o11y solutions for impending load-bearing projects, I noticed an issue with querying event-time data: a WHERE on an event-time column returns [] on synced data and is ignored on staging data. I expected it to fetch the data I submitted for that range! Is this intentional and thus undocumented, or is there a gap here?

Environment: v3.2.5 (b8fb6ce), standalone, S3 store (MinIO). src/query/ is unchanged on main at a5976d2.

Repro

ts is the event time. The rows arrive on 2026-10-20, so p_timestamp and ts differ by three days.

curl -u admin:$PW -X PUT localhost:8000/api/v1/logstream/verify2-evt
curl -u admin:$PW -H 'X-P-Stream: verify2-evt' -H 'Content-Type: application/json' \
  localhost:8000/api/v1/ingest -d '[
  {"event":"old1","ts":"2026-10-17T01:30:00Z"},
  {"event":"old2","ts":"2026-10-17T02:15:30Z"},
  {"event":"new","ts":"2026-10-20T00:17:40Z"},
  {"event":"no-ts"}]'
# schema: ts is Timestamp(Millisecond, None)

-- POST /api/v1/query with "startTime":"2026-10-19T00:00:00Z", "endTime":"now"
select event, ts from "verify2-evt"
where ts between '2026-10-17T01:00:00' and '2026-10-17T03:00:00'
order by event

# 7 s after ingest (staging)
[{"event":"new","ts":"2026-10-20T00:17:40"},{"event":"no-ts"},{"event":"old1","ts":"2026-10-17T01:30:00"},{"event":"old2","ts":"2026-10-17T02:15:30"}]

# after sync (storage size 1765)
[]

# control, both states: where to_unixtime(ts) between 1792198800 and 1792206000
[{"event":"old1","ts":"2026-10-17T01:30:00"},{"event":"old2","ts":"2026-10-17T02:15:30"}]

After sync, EXPLAIN puts both ts conjuncts in full_filters and plans no object-store file. The to_unixtime control plans date=2026-10-20/hour=00/minute=17/….parquet.

On synced data, timestamp '…', to_timestamp, CAST, arrow_cast and ts < now() - interval '1 day' also return [].

Root cause

extract_timestamp_bound (src/query/stream_schema_provider.rs:1144) checks the column only on the Utf8 arm (:1166), and the TimestampMillisecond and TimestampNanosecond arms (:1158, :1162) accept any column. scan turns the result into PartialTimeFilters through extract_primary_filter (:799). collect_from_snapshot (:853) passes them to Snapshot::manifests (src/catalog/snapshot.rs:42). Manifest bounds are UTC days of p_timestamp (src/catalog/mod.rs:164)... which means a ts range outside the ingest day removes every manifest.

In staging, expr_in_boundary (:1126) calls the same function. A minute-aligned bound on any timestamp column gets Exact from supports_filters_pushdown (:933), so DataFusion removes the filter. The staging MemTable scan (:452) does not filter, so every staged row comes back. With a lower bound of 01:00:01, that conjunct is Inexact and applies = only the upper bound is dropped.

#561 guarded this function on p_timestamp. #683 (bc730ee) replaced that guard with the time-partition check on the Utf8 arm only -> the UI does not depend on the wider match. Prism v3.2.5 sends startTime/endTime and hard-codes p_timestamp in its own predicates, and the archived console used the stream's time_partition, defaulting to p_timestamp.

Workaround + Fix

My current workaround is to compare epoch seconds instead:
where to_unixtime(ts) between <start> and <end>

Suggested fix, having poked around the source:

• Accept the timestamp arms only when the column is p_timestamp or the stream's time partition, as table_contains_any_time_filters (src/query/mod.rs:1103) already does. A check on is_time_partition alone would stop p_timestamp pruning on streams without a time partition.