Repository navigation
improvement: Use tracking timestamps instead of data timestamps for stale-entry eviction in ACTIVE_OBJECT_STORE_SYNC_FILES #1634
Description
Summary
In src/storage/object_storage.rs, the stale-entry cleanup in collect_upload_results currently uses extract_datetime_from_parquet_path_regex to compare the parquet file's data timestamp (extracted from the filename) against a 5-minute threshold, rather than the time the path was added to ACTIVE_OBJECT_STORE_SYNC_FILES.
Problem
- Historical ingestion (with time partition): entries are immediately eligible for cleanup because the data timestamp (from the filename) is old, potentially causing races where a currently-uploading file gets evicted from the tracking set prematurely.
- Near-real-time ingestion: works correctly since data timestamps closely match wall-clock time.
Proposed Improvement
Change the tracking structure from HashSet<PathBuf> to a HashMap<PathBuf, DateTime<Utc>> (or equivalent), recording Utc::now() when each path is inserted. Update the retain logic to compare now - tracked_at >= Duration::minutes(5) instead of parsing the data timestamp from the filename.
This ensures accurate duration-based eviction regardless of whether data is near-real-time or historical.
Context
- File:
src/storage/object_storage.rs, around lines 1160–1165 - Identified in PR fix: Ingestion staging sync issue #1633: fix: Ingestion staging sync issue #1633
- Review comment: fix: Ingestion staging sync issue #1633 (comment)
- Requested by: @nikhilsinhaparseable
Metadata
Metadata
Assignees
Labels
No labels
Activity
ygndotgg commentedon May 23, 2026
When ingesting historical or backfill data through time-partitioned connector ingestion (e.g. Kafka connector), the object-store sync cleanup logic incorrectly removes newly tracked parquet files from
ACTIVE_OBJECT_STORE_SYNC_FILESbefore they can be uploaded. The root cause is that cleanup parses the event timestamp from the parquet filename and treats it as the upload start time.Affected vs Unaffected Paths
Not affected:
Nonefortime_partition, and OSS rejects ingestion into time-partitioned streams anyway. Staging filenames use current ingestion time, so cleanup age calculation is correct.Affected:
time_partitionmetadata set.time_partitionand passes parsed event timestamps intoStream::push(...).Root Cause
In
src/storage/object_storage.rs, the cleanup logic (collect_upload_resultsor equivalent) does:It then computes:
For time-partitioned ingestion, the filename timestamp is event time, not upload start time. A backfill event with
event_ts: "2024-01-01T10:00:00Z"produces:Cleanup sees an 11-month-old timestamp and evicts the path from active tracking immediately, even if it was inserted seconds ago. The parquet file never gets uploaded.
ygndotgg commentedon May 23, 2026
@nitisht I'd like to work on this issue. Could you assign it to me?