ORDER BY p_timestamp can return unsorted rows #1827

Have been exploring parseable in my own time, and my agent (Opus 5.5) identified this bug.

ORDER BY p_timestamp can return unsorted rows: scan trusts unverified Parquet footer sort order

Reproduced from source rather than a running instance:

{
  "version": "v3.2.5",
  "commit": "a5976d2e911cdd4391dbc6360b00470281f619f9",
  "note": "main @ 2026-10-15 (#1824); in-process test, release profile, Rust 1.96.0"
}

Description:

Expected: SELECT … ORDER BY p_timestamp DESC NULLS LAST returns rows with non-increasing p_timestamp.

Observed: after concurrent ingestion into one stream and conversion to Parquet, the query (no LIMIT) returns some adjacent rows where a later timestamp follows an earlier one. The gaps are 1–2 ms. Counts and values are correct and no error is raised.

Why this is a bug: ingest stamps p_timestamp (ingest_utils.rs:313) before it takes the stream's writer lock (streams.rs:712). Concurrent requests can therefore be appended out of timestamp order. From there:

• The Parquet writer still declares p_timestamp DESC NULLS LAST in sorting_columns without sorting (streams.rs:1242-1246).
• create_from_parquet_file copies that claim into the manifest's sort_order_id.
• file_groups_are_time_ordered accepts a single-file group on that claim alone (stream_schema_provider.rs:258-292).
• create_parquet_physical_plan then sets with_output_ordering, so DataFusion removes the SortExec.

Minimal reproduction: a deterministic in-process test. It uses the real writer path with no threads:

1. Push three batches into one stream and minute via Stream::push, stamped 1000, then 1002, then 1001. This is the order the writer lock admits racing requests.
2. Run flush and convert_disk_files_to_parquet, then build the manifest entry with create_from_parquet_file.
3. Scan using the ordering flag from partitioned_files.
4. Run ORDER BY p_timestamp DESC NULLS LAST.

Result on main:

ORDER BY p_timestamp DESC NULLS LAST returned (p_timestamp, id) rows [(1001, 11), (1001, 10), (1001, 9), (1001, 8), (1002, 7), (1002, 6), (1002, 5), (1002, 4), (1000, 3), (1000, 2), (1000, 1), (1000, 0)]
plan:
DataSourceExec: file_groups={1 group: [[…/ordering/date=2026-10-17.hour=12.minute=59.….data.….parquet]]}, projection=[id, p_timestamp], output_ordering=[p_timestamp@1 DESC NULLS LAST], file_type=parquet

The plan has no SortExec. Footer flags, min/max stats and a single row group cannot tell this file apart from a truly sorted one.

Related: time-partitioned streams can produce the same false claim without any concurrency, and the legacy ListingTable path also infers ordering from footer sorting_columns.

Here is the prompt I was able to give to an agent (Opus 5.5) to identify the issue. Let me know and I can also open a PR for this

In parseable main, `SELECT … ORDER BY p_timestamp DESC NULLS LAST` (no LIMIT) can
return rows out of order once data has been persisted to Parquet. No error is
raised. We saw it after concurrent ingestion into one stream: counts and values
were correct, but a few adjacent rows had a later p_timestamp after an earlier
one (1–2 ms apart).

Trace the path from HTTP ingest through staging, Parquet conversion and manifest
creation to the query table provider and the physical plan. At each layer, say
what it assumes about row order and whether anything verifies that assumption.
Check EXPLAIN for the affected query.

Deliver:
1. The root cause, with file:line references.
2. A minimal, deterministic regression test that uses the real writer path, with
   no hand-built Parquet metadata and no timing-dependent threads. Run it and
   show that it fails on main.
3. The smallest fix, and show the test passing with it. Data already persisted
   by earlier versions must also return correct results, with or without LIMIT.
   Prefer a fix that changes no on-disk or manifest format; propose any
   performance recovery separately.
4. Other code paths with the same weakness, and the fix's performance cost.
   Keep findings outside this bug out of the fix.

Report what you executed separately from what you only inferred from the source.