Parseable manifest duplication #1739

This is an AI assisted bug report as I was seeing increasing latency in querying logs when parseable pods restarted because of node updates/OOM errors or config updates.

/about output:

{
  "version": "v2.9.3",
  "uiVersion": "v2.9.3",
  "commit": "c54b762",
  "deploymentId": "01KBNFVYPZ46Y5EKD93MQ9GFVN",
  "updateAvailable": false,
  "latestVersion": null,
  "llmActive": false,
  "llmProvider": null,
  "oidcActive": false,
  "license": {
    "name": "AGPL-3.0-only",
    "version": "v1",
    "plan": "OSS",
    "deploymentInfo": "Managed"
  },
  "mode": "Standalone",
  "staging": "/parseable/staging",
  "grpcPort": 8001,
  "store": {
    "type": "S3 bucket",
    "path": "---"
  },
  "analytics": {
    "clarityTag": null
  }

Description:

Expected: when a node writes to a date partition that a different node has already written to, it should append its own ManifestItem to snapshot.manifest_list once, then update that entry in place on subsequent sync cycles. The list should hold one entry per (date partition, writer).

Observed: it appends a new entry on every sync cycle — one per minute — until the UTC date rolls over. On our deployment manifest_list reached 3110 entries across 66 unique manifest paths, making .stream.json 795 KB (99.9% of it manifest_list) and producing 5664 manifest GETs in a five minute window, with a single query fetching the same manifest object 33 times.

Root cause

In src/catalog/mod.rs, process_single_partition finds the entry to update by time range alone:

let pos = meta.snapshot.manifest_list.iter().position(|item| {
    item.time_lower_bound <= partition_lower && partition_lower < item.time_upper_bound
});

handle_existing_partition then checks whether that entry belongs to the current node before allowing an in-place update:

let manifest_file_name = manifest_path("").to_string();
let should_update = manifests[pos].manifest_path.contains(&manifest_file_name);

The two ask different questions. position() returns the first time-overlapping entry, which need not be the current writer's. Once a foreign entry occupies that position, should_update is false, the else branch appends, and the next sync repeats the same lookup with the same result. The current writer never finds its own entry, so it never updates in place.

While one node owns a date for a full day the two checks coincide by accident — there is a single entry and it belongs to that node — which is why stable single-writer deployments never hit this.

Reproduction

1. Run Parseable against an S3 backend and ingest into a stream.
2. Restart the pod mid-day (any change of node identity will do).
3. Poll the snapshot:

aws s3 cp "s3://$BUCKET/$STREAM/.stream/.stream.json" - \
  | jq '.snapshot.manifest_list | {total: length, unique: (group_by(.manifest_path) | length)}'

total climbs by one per minute; unique stays constant.

Evidence

Duplicate count equals the minutes between the writer change and end of that UTC day:

Only 6 of 66 paths were affected — exactly the dates where writer identity changed. Growth is visible directly in .stream.json PUT sizes, one 254-byte entry per sync:

00:37:30  bytes=19047
00:38:30  bytes=19301
00:39:30  bytes=19555

Impact

Query latency. Every duplicate is resolved and fetched during planning. After collapsing to the 66 unique entries, manifest GETs fell from 5664 per five minutes to 51 per three minutes and .stream.json from 795,499 to 19,047 bytes.

Write amplification. The whole snapshot is re-uploaded every sync. At 795 KB that is roughly 1.1 GB/day of PUT traffic.

Corrupted statistics. Per-date rollups appear to sum duplicated entries, so reported figures scale with the duplication factor. 2026-07-15 (1417 duplicates) reported 31,287,815,348 events against 30,810,252 for the adjacent day, a ratio of about 1015, while the stream's lifetime total was 6,159,104. Gauges also go negative once deletions are subtracted from the inflated totals:

parseable_events_ingested{stream="functions"}  -103800687
parseable_storage_size{stream="functions",type="data"}  -16325422799

A per-date value exceeding the lifetime total is a cheap invariant that would have surfaced this much sooner.

Suggested fix

Match writer identity in the lookup rather than testing it afterwards:

let manifest_file_name = manifest_path("").to_string();
let pos = meta.snapshot.manifest_list.iter().position(|item| {
    item.time_lower_bound <= partition_lower
        && partition_lower < item.time_upper_bound
        && item.manifest_path.contains(&manifest_file_name)
});

A node inheriting a partition then finds no match, appends once, and updates in place thereafter — which appears to be what the existing should_update check already intends to support.