[Memory Optimization] Right-size data stream filter buffers

[Arcticae]

Summary

Replace the unconditional 1 MiB reservation in every DataStream::filter_fragment pass with an exact, checked protobuf encoded-size calculation.

The implementation resolves joins and matches, retains the already-fetched header/body data, computes the full output size, allocates once, and encodes with the existing safe Prost helpers. It preserves one output Bytes value per block filter, including empty outputs.

Motivation

DHAT identified the fixed reservation as the dominant allocation pattern:

• 1,038,318 allocations of exactly 1 MiB;
• approximately 1.08 TiB of allocation traffic;
• 91.5% of all allocated bytes;
• approximately 565 MiB simultaneously live at the DHAT heap maximum.

Most filtered responses are substantially smaller than 1 MiB. Repeating this reservation creates allocator traffic and slack that can amplify fragmentation and transparent-huge-page retention even when Rust ownership is bounded.

Implementation

• Add checked encoded-length helpers for header and body fields.
• Use public Prost key, varint, and packed-uint32 length calculations.
• Reuse fetched archived fragments; no second cache or remote read is introduced.
• Allocate BytesMut only after the header decision and all matches are known.
• Keep the existing safe Prost encoding path and assert predicted bytes equal written bytes in debug builds.
• Remove DEFAULT_BLOCKS_BUFFER_SIZE and the fixed 1 MiB reservation.

Behavior preserved

• protobuf bytes and field ordering;
• block/filter ordering;
• one output buffer per block filter;
• empty-output cardinality;
• filter, join, cursor, and response semantics.

Tests

Tests were established before the implementation and initially failed on the missing sizing/encoding helpers.

Coverage includes:

• empty and non-empty header payloads;
• empty, one, and multiple filter IDs;
• multi-byte varints;
• empty, small, and greater-than-1-MiB message bodies;
• multiple protobuf field tags;
• checked overflow without allocating enormous buffers;
• exact encoded bytes and representative item ordering.

Validation run:

• cargo test -p apibara-dna-common --lib — 8 passed;
• cargo check -p apibara-dna-common;
• cargo fmt --all -- --check;
• git diff --check origin/main...HEAD;
• static check confirms no fixed 1 MiB reservation remains.

The full integration suite was also attempted on main, but its existing container fixture cannot pull bitnami/etcd:latest (manifest unknown). The task-specific library tests and compilation pass. Nix was unavailable on the implementation host.

Review and experiment notes

This PR is based directly on main and is independent from the Foyer-budget and response-queue PRs. The three task branches are siblings, not a stack.

Still required before production rollout:

1. Run parent-versus-changed native fast-only A/B tests using the same seed and fresh serving caches.
2. Require byte/cursor correctness, zero workload errors, and at least 95% of control throughput.
3. Run a short DHAT profile and confirm the million-count fixed-1-MiB call site is absent.
4. Compare RSS, anonymous memory, THP, cgroup memory, and reclaim pressure directionally.