[VibeSWE]

Problem

Parseable uploads one small Parquet file per stream per minute, and every query re-reads the same files from object storage. For anything beyond a bare COUNT(*), each file costs about five requests (footer, page index, column chunks), issued one after another per file. When the bucket is far from the server, query time is almost entirely round trips: on a standalone node about 150 ms from its S3-compatible bucket, a 25-tile dashboard made about 6,500 object-store calls per load while the server used 10–240 ms of CPU per query.

Change

• Small-file cache (src/query/small_file_cache.rs). A Parquet file whose manifest size is at most 256 KiB is fetched whole with one bounded GET and kept in memory. Footer, page-index and column reads are then served from that body. Larger files keep DataFusion's reader and its metadata cache unchanged.
• Validation, not trust. On every later request the cached body is revalidated with a conditional GET (If-None-Match). A 304 reuses the bytes; a changed object replaces them; a 404 or error propagates and evicts. Weak or missing ETags are never reused across requests. Nothing validated in one request is reused by another without validating again.
• One validation per file per request. Each query execution gets its own request view; concurrent readers of the same file (DataFusion splits files across partitions once a scan selects more than 1 MiB) share one in-flight validation.
• Bounds. 64 MiB and 4,096 entries of residency by default, O(1) LRU. A scan that selects more small files than the cache can hold uses the original reader, so it cannot thrash the cache. Bodies held by in-flight requests are admitted against a per-request total and a process-wide 64 MiB reservation; once either is exhausted, further scans use the original reader. Worst case for cached bodies with defaults: 128 MiB resident, 192 MiB including validation transients.
• One fewer metadata read per query. scan() captures staging before the first stream.json read, which removes the duplicate read in standalone mode. The ordering invariant (publication precedes staging unlink) is unchanged and has its own tests.
• Dataset listing. The per-stream LIST probes behind GET /api/v1/logstream run up to 16 at a time instead of serially.
• Storage metrics. parseable_storage_read_bytes_total counts bytes actually returned; a 304 is its own outcome with zero bytes instead of being charged the requested range.

Nothing stored in the bucket changes, and no new configuration is required.

Settings

Setting	Default	Meaning
P_SMALL_FILE_CACHE_ENABLED	true	false restores the original reader everywhere
P_SMALL_FILE_CACHE_SIZE	67108864	Residency limit in bytes; 0 also disables the cache

Measurement

A standalone release build against a real S3-compatible bucket about 150 ms away, replaying a 25-tile dashboard (29 requests, 4 in flight) over a fixed past hour holding 60 one-minute files of OTel host metrics. Quiet machine, medians after warm-up:

Build	Dashboard load	Object-store calls per load
main (d6a155f2)	9.8 s	6,528
This branch	5.8 s	1,738

Four queries over that fixed window (a count, two grouped SQL queries, one PromQL range) returned byte-identical responses on both builds. A floating-point SUM varies in its last digit between identical runs on either build, so it is not a useful comparison.

Per one-hour query the count went from 63–393 calls, depending on shape, to about 61: two metadata reads plus one validation per file.

What this does not do

• It still makes one round trip per file per request. Removing that needs a guarantee that published Parquet paths are never rewritten, which nothing enforces today (uploads are unconditional).
• stream.json and manifests are not cached across requests, because another writer can change them.
• The remaining per-request cost is the two serial metadata reads plus one validation wave.

Tests

• Library suite passes (1,393 passed, 7 ignored at the last commit).
• Cache tests through real HTTP against MinIO with a fault-injecting proxy: exact rows after overwrite, delete, and delete-and-recreate; growth and shrink across the size threshold with current and stale manifest sizes; a 503 and an interrupted body during validation; weak and missing ETags; counters cold and warm; disabled-cache request parity with main. These need P_CACHE_TEST_MINIO_ENDPOINT and are skipped visibly without it.
• A regression test through Parseable's query path over files written with the production writer properties (bloom filter and page index), asserting exactly one validation per file per request for filtered SQL, COUNT(*) and PromQL, and that a second request validates again.
• A test that runs in CI without MinIO: an in-memory store behind the metrics layer, the production scan path through request_context, forced byte-range splits and a UNION, asserting one GET per file per request and fresh validation on the next request.
• A ranged reader refuses to recover when an object has grown past its planned size (otherwise row groups beyond the planned ranges would be dropped silently); the test reproduces the missing rows with the guard removed.

Review notes

The first commit was reviewed by two independent reviewers for wrong-data, freshness and failure behaviour; the second commit addresses their twelve findings. The second and third commits were then reviewed independently (no path serves another query's rows or stale bytes; two major findings: silent row loss on a grown ranged file, and unbounded request-view memory), and the fourth commit addresses those findings. The fourth commit has not been independently reviewed. Design notes: docs/performance/validated-small-file-cache.md.

🤖 Generated with Claude Code