Skip to content

Commit 87dab9a

Browse files
authored
Merge pull request #146 from VibeEngineeringAI/refactor/metastore-metadata-boundary
refactor: route metadata lifecycle operations through the Metastore trait
2 parents edf243c + e0410b0 commit 87dab9a

10 files changed

Lines changed: 493 additions & 33 deletions

File tree

‎docs/metastore-boundary-audit.md‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
# Mutable metadata boundary audit
2+
3+
Baseline: origin/main `d6a155f2`. Scope: stage 0 seams only. No PostgreSQL
4+
implementation, transactions, caching, serialization changes, or race fixes.
5+
6+
A site below is a logical operation in a named symbol (several expressions in
7+
one operation count once). Grouped backend implementations count once per row,
8+
not once per provider. A includes reviewed operations already using the trait;
9+
these are called out so an apparent ObjectStorage helper is not mistaken for a
10+
remaining bypass. Test fixture writes are excluded from production counts.
11+
12+
## Inventory
13+
14+
Paths in the table are relative to `src/`.
15+
16+
| File / symbol | Class | Objects or meaning; disposition |
17+
| --- | --- | --- |
18+
| `parseable/mod.rs::create_stream_and_schema_from_storage`, deletion check | A | `.tombstones/<tenant>/<stream>/.tombstone` HEAD; now `Metastore::is_stream_deleting`. |
19+
| `parseable/mod.rs::create_stream_and_schema_from_storage`, existence check | A | `.stream/*stream.json` discovery; now `Metastore::stream_exists`. Preserves LocalFS's canonical-only predicate and cloud's ingestor acceptance. |
20+
| `parseable/mod.rs::load_tenants`, discovery | A | Root directory candidates standing for tenants; moved unchanged to `list_tenant_candidates`. Includes unrecognized directories for historical cleanup. |
21+
| `parseable/mod.rs::load_tenants`, invalid directory cleanup | A | Whole tenant prefix, including any metadata; now existing `delete_tenant`. Same path and tenant metrics context (`None`). |
22+
| `handlers/http/logstream.rs::delete` | A | Stream prefix, metadata and data together; now `Metastore::delete_stream`, which delegates to the identical provider operation. |
23+
| `handlers/http/modal/query/querier_logstream.rs::delete` | A | Same deletion operation; same reroute. |
24+
| `handlers/http/health_check.rs::readiness_internal` | A | Provider HEAD of `.parseable.json` (LocalFS has its own check); now `check_health`, preserving the provider implementation. |
25+
| `banner.rs::storage_info` | A | Timed `.parseable.json` GET, including timing failed GETs; now `read_latency`, preserving the existing probe. |
26+
| `storage/object_storage.rs::create_stream` | A | Schema then stream JSON; already calls `put_schema` then `put_stream_json`; unchanged. |
27+
| `storage/object_storage.rs::update_time_partition_limit_in_stream` | A | Stream JSON read/overwrite through trait already; unchanged. |
28+
| `storage/object_storage.rs::update_custom_partition_in_stream` | A | Same, custom partition; unchanged. |
29+
| `storage/object_storage.rs::update_log_source_in_stream` | A | Same, log sources; unchanged. |
30+
| `storage/object_storage.rs::update_dataset_tags_and_labels_in_stream` | A | Same, dataset tags/labels; unchanged. |
31+
| `storage/object_storage.rs::update_first_event_in_stream` | A | Same, persisted first event timestamp; unchanged. |
32+
| `storage/object_storage.rs::put_stats` | A | Same, whole stats replacement; unchanged. |
33+
| `storage/object_storage.rs::put_retention` | A | Same, retention configuration; unchanged. |
34+
| `storage/object_storage.rs::upsert_stream_metadata` | A | Mode-specific stream JSON or fallback base read then mode write; already trait. Retains even the original return of base bytes after writing modified config. |
35+
| `storage/object_storage.rs::put_snapshot` | A | Upsert then snapshot replacement via trait; unchanged. |
36+
| `storage/object_storage.rs::create_stream_from_querier` | A | Base stream JSON read, stats/snapshot reset, mode-specific write; already trait. |
37+
| `storage/object_storage.rs::create_stream_from_ingestor` | A | Enumerate ingestor JSONs, merge sources/tags/labels, publish; already trait. |
38+
| `storage/object_storage.rs::create_schema_from_metastore` | A | `fetch_schema` then `put_schema`; already trait. |
39+
| `storage/object_storage.rs::get_log_source_from_storage` | A | Merge source descriptions from all stream JSONs; already trait. |
40+
| `storage/object_storage.rs::commit_schema_to_storage` / `process_schema_files` | A | Schema read/merge/write then memory update and staging cleanup; already trait. No reordering. |
41+
| `storage/store_metadata.rs::resolve_parseable_metadata` / `put_remote_metadata` | A | `.parseable.json`, RBAC, deployment and tenant configuration; already `get/put_parseable_metadata`. |
42+
| `catalog/mod.rs::update_snapshot`, `handle_existing_partition`, `finalize_snapshot_update`, `remove_manifest_from_snapshot` | A | Stream snapshot, manifest reads/writes and stats; already trait (including `put_snapshot` helper). |
43+
| `handlers/http/logstream.rs::{put_stream_hot_tier,delete_stream_hot_tier,get_stream_hot_tier_internal}` | A | Shared hot-tier enablement/config reads and whole stream JSON writes already use trait; local cache config is C below. |
44+
| `storage/retention.rs::action::delete` metadata publication | A | Remove snapshot references and update deleted stats/first event through trait-backed helpers; unchanged. |
45+
| `handlers/http/modal/mod.rs` node registration and HTTP modal node management | A | Node registry JSON; already `put/get/delete_node_metadata`; unchanged. |
46+
| `migration/mod.rs::{run_migration,migration_stream,fetch_or_create_stream_metadata,migrate_stream_metadata,setup_logstream_metadata,put_remote_metadata}` | A | Discover streams, migrate JSON versions, publish schema/stream/deployment metadata; already trait, including helper calls. Retained this policy for all live migration reads/writes. Pure transforms in `metadata_migration.rs` and `stream_metadata_migration.rs` have no storage access. |
47+
| `metastore/metastores/object_store_metastore.rs::{refresh_schema,get_all_schemas}` and `storage/object_storage.rs::schema_objects`, `storage/localfs.rs::schema_objects` | A | Schema enumeration/canonical union publication; raw storage only inside backend implementation/adapters. No application calls to `schema_objects`. |
48+
| `storage/{s3,gcs,azure_blob,localfs}.rs::{stream_exists,list_streams,list_old_streams,get_ingestor_meta_file_paths,try_delete_node_meta,check}` and `object_storage.rs::{stream_exists,get_latency}` | A | Physical metadata predicates, node file enumeration/deletion and diagnostic probes. Provider primitives remain for backend use; active application calls now enter the trait. Legacy unused primitives noted below. |
49+
| `metastore/metastores/object_store_metastore.rs` remaining metadata CRUD | A | Keystones, conversations, alerts, alert history, MTTR, LLM config, targets, outbound policy, dashboards (including conditional tombstones), chats, filters, schemas, manifests, deployment/tenant and node records. All raw object operations are backend implementation details already. |
50+
| `storage/object_storage.rs::{upload_single_parquet_file,validate_uploaded_parquet_file}` | B | Multipart Parquet write, HEAD length validation, failed-upload object cleanup; data durability/size, not catalog authority. Left unchanged. |
51+
| `storage/object_storage.rs::{get_first_and_latest_event_from_storage,extract_timestamp_for_date}` | B | Date/hour/minute directory scans implicitly stand for first/latest event timestamps with minute rounding. Future catalog bounds must preserve rounding. |
52+
| `storage/object_storage.rs::{update_storage_metrics,upload_files_from_staging}` | B | Physical Parquet sizes/date names and publication pipeline. Metrics implicitly stand for ingestion/storage totals; metadata publication already enters trait. |
53+
| `storage/retention.rs::action::delete` partition deletion | A | Mixed daily data/manifest prefix deletion now enters `delete_stream_partition`; the backend performs exactly the original prefix DELETE after snapshot publication, with the same concurrent per-date tasks. |
54+
| `storage/retention.rs::action::delete` date listing | B | Directory names imply retention candidates; remains on storage. A future catalog/GC design must select catalog-native partitions instead. |
55+
| `query/listing_table_builder.rs::ListingTableBuilder::populate_via_listing` | B | Partition prefixes imply legacy file membership; unchanged. |
56+
| `query/stream_schema_provider.rs::{scan,legacy_listing_table,parquet_scan_plan}` | B | Legacy directory-derived membership and Parquet execution; modern metadata fetches already trait. File untouched, as requested. |
57+
| `hottier.rs` download/planning and `hottier/planner.rs::build_work` | B | Remote Parquet fetches and directory-derived minute buckets imply cache work/time ranges; manifests already trait. |
58+
| `parseable/mod.rs::validate_storage` root directory occupancy probe | B | Physical nonempty-store detection (even arbitrary nonmetadata directories affect startup). Metadata existence and stream listing already trait. Kept physical occupancy check; replacing it with catalog listing changes stale-storage detection. |
59+
| `storage/{s3,gcs,azure_blob,localfs}.rs::{list_dates,list_hours,list_minutes,list_dirs_relative,delete_prefix}` | B | Data layout primitives, tenant scoping, metrics, retries and provider errors. Kept as physical adapters; metadata consumers confined to metastore. |
60+
| `storage/field_stats.rs::calculate_field_stats` | B | Derived field statistics ingested as stream data, not a mutable JSON authority; file-derived timestamp remains. |
61+
| `storage/store_metadata.rs::{get_staging_metadata,put_staging_metadata}` | C | Staging `.parseable.json` describes local deployment association; filesystem only, retained. |
62+
| `migration/mod.rs::{get_staging_metadata,put_staging_metadata}` | C | Local migration staging copy; retained. |
63+
| `parseable/streams.rs` schema staging, stream staging metadata and WAL | C | Local ingestion recovery state and schema files; retained. Shared schema commit already trait. |
64+
| `hottier.rs::{read_hot_tier_config,put_hot_tier,delete_hot_tier}` and `hottier/local_state.rs` | C | Local `.hot_tier.json`, cache manifests/runtime state and local filesystem operations; retained. Shared hot-tier enablement/config in stream JSON already trait. |
65+
| `storage/object_storage.rs::list_tombstoned_streams` and unused legacy provider discovery/node helpers | D | No production callers of tombstone discovery, `list_old_streams`, `get_ingestor_meta_file_paths`, or `try_delete_node_meta` outside backend/adapters. Retain dormant compatibility APIs and fixture tests. A future deletion recovery/legacy-layout import feature must expose semantic metastore methods before using them. |
66+
67+
Counts: **33 A, 10 B, 4 C, 1 D** logical operation groups. Nine application
68+
call sites changed; the other A groups already persisted through the trait or
69+
implement its physical adapter.
70+
71+
## Contract additions
72+
73+
Existing methods and signatures are unchanged. PostgreSQL must implement:
74+
75+
```rust
76+
async fn check_health(&self, tenant_id: &Option<String>) -> Result<(), MetastoreError>;
77+
async fn read_latency(&self, tenant_id: &Option<String>) -> std::time::Duration;
78+
async fn stream_exists(&self, stream_name: &str, tenant_id: &Option<String>) -> Result<bool, MetastoreError>;
79+
async fn is_stream_deleting(&self, stream_name: &str, tenant_id: &Option<String>) -> Result<bool, MetastoreError>;
80+
async fn delete_stream(&self, stream_name: &str, tenant_id: &Option<String>) -> Result<(), MetastoreError>;
81+
async fn delete_stream_partition(&self, stream_name: &str, date: &str, tenant_id: &Option<String>) -> Result<(), MetastoreError>;
82+
async fn list_tenant_candidates(&self) -> Result<Vec<String>, MetastoreError>;
83+
```
84+
85+
`delete_stream` and `delete_stream_partition` include physical deletion: PostgreSQL must orchestrate catalog
86+
and data cleanup, just as existing `delete_tenant` already includes both. This
87+
seam introduces no transaction or new deletion protocol. `list_tenant_candidates`
88+
preserves invalid candidates for cleanup; another backend may return only its
89+
registered tenants. The legacy deletion-marker check remains separate from
90+
existence, in the same order. It does not introduce marker publication.
91+
92+
## Races deliberately preserved
93+
94+
All stream-setting/stats/retention/first-event/snapshot updates read and blindly
95+
replace the entire stream JSON. Schema commit and refresh merge schemas then
96+
unconditionally publish; concurrent writers can lose fields. Manifest updates
97+
and stream snapshot publication are separate unconditional writes; retention
98+
can race ingestion, settings, stream deletion and queries. RBAC/deployment
99+
metadata, alert state/history, filters and node registration have blind
100+
whole-record writes or separate discovery/write/delete sequences. Tenant
101+
validation/cleanup races concurrent tenant creation. Check-then-load and
102+
check-then-delete remain racy. No CAS conversion, locks, retries, cache, or
103+
transaction was added. Dashboard CAS behavior is unchanged.
104+
105+
## Guard and evidence
106+
107+
`metastore::boundary_tests` scans production Rust source before the trailing
108+
`#[cfg(test)]` section. Outside the object-store metastore and four provider
109+
adapters it rejects every raw `put_object`, `upload_file`, conditional PUT and
110+
conditional DELETE, regardless of literal/computed key. It permits exactly one
111+
multipart publisher in `object_storage.rs`, exactly one failed-Parquet DELETE
112+
there, and no raw prefix DELETEs in application code. Tests verify
113+
rejection of computed keys and whitespace-split method calls. This is a
114+
conservative source policy, not runtime provenance or Rust type analysis. It
115+
cannot detect mutations through new APIs, raw object_store clients, macros,
116+
code after the first test section, or changes hidden in allowed adapters. The
117+
existing allowed physical publishers still require review if their arguments
118+
change. It does not prove read routing or transaction safety.
119+
120+
Compatibility evidence: the nine application bypasses now delegate to exactly
121+
the original provider operations. Provider errors are unwrapped again at the
122+
rerouted legacy call sites, preserving HTTP error mapping and error text. JSON encoders, paths, payloads, write modes,
123+
operation order and provider implementations are unchanged. The LocalFS
124+
lifecycle fixture compares discovery and existence against raw provider results,
125+
asserts exact JSON bytes survive probes, verifies the separate tombstone survives
126+
stream prefix deletion, and compares missing-stream deletion errors. A second
127+
fixture verifies partition deletion removes both Parquet and manifests while
128+
preserving adjacent-date and other-tenant bytes. Existing
129+
schema-refresh/manifest/dashboard/provider tests remain in the full library
130+
suite. No live service or bucket was used; no container was started. No
131+
main-versus-branch MinIO object diff was performed, so provider integration parity
132+
is supported by unchanged delegates and unit tests, not an end-to-end claim.
133+
134+
Open follow-up: catalog-native deletion/GC, imported legacy file membership,
135+
first/latest timestamp semantics, and removal or semantic exposure of dormant
136+
legacy APIs need later stages. Backend selection and PostgreSQL implementation
137+
are intentionally outside this change.
138+
139+
Final validation on this branch:
140+
141+
- `P_BUILD_NO_UI=1 CARGO_BUILD_JOBS=4 cargo clippy --all-targets`: exit 0;
142+
21 existing library-test warnings, none in changed files.
143+
- `P_BUILD_NO_UI=1 CARGO_BUILD_JOBS=4 cargo test --lib`: exit 0;
144+
**1,379 passed, 0 failed, 4 ignored** (1,383 discovered), 79.65 seconds.
145+
- Both source-guard tests and both LocalFS compatibility fixtures passed.
146+
- `cargo fmt` and `git diff --check` passed. Cargo target remained in this worktree.

‎src/banner.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@ fn status_info(config: &Parseable, scheme: &str, id: Uid) {
8888
/// - Store (path where the data is stored and its latency)
8989
async fn storage_info(config: &Parseable) {
9090
let storage = config.storage();
91-
let latency = storage.get_object_store().get_latency(&None).await;
91+
let latency = config.metastore.read_latency(&None).await;
9292

9393
eprintln!(
9494
"

‎src/handlers/http/health_check.rs‎

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -119,13 +119,7 @@ pub async fn readiness(req: HttpRequest) -> HttpResponse {
119119

120120
pub async fn readiness_internal(tenant_id: &Option<String>) -> StatusCode {
121121
// Check the object store connection
122-
if PARSEABLE
123-
.storage
124-
.get_object_store()
125-
.check(tenant_id)
126-
.await
127-
.is_ok()
128-
{
122+
if PARSEABLE.metastore.check_health(tenant_id).await.is_ok() {
129123
StatusCode::OK
130124
} else {
131125
StatusCode::SERVICE_UNAVAILABLE

‎src/handlers/http/logstream.rs‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -66,10 +66,12 @@ pub async fn delete(
6666
return Err(StreamNotFound(stream_name).into());
6767
}
6868

69-
let objectstore = PARSEABLE.storage.get_object_store();
70-
7169
// Delete from storage
72-
objectstore.delete_stream(&stream_name, &tenant_id).await?;
70+
PARSEABLE
71+
.metastore
72+
.delete_stream(&stream_name, &tenant_id)
73+
.await
74+
.map_err(crate::metastore::MetastoreError::into_storage_error)?;
7375
// Delete from staging
7476
let stream_dir = PARSEABLE.get_or_create_stream(&stream_name, &tenant_id);
7577
if let Err(err) = fs::remove_dir_all(&stream_dir.data_path) {

‎src/handlers/http/modal/query/querier_logstream.rs‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -78,9 +78,12 @@ pub async fn delete(
7878
return Err(StreamNotFound(stream_name.clone()).into());
7979
}
8080

81-
let objectstore = PARSEABLE.storage.get_object_store();
8281
// Delete from storage
83-
objectstore.delete_stream(&stream_name, &tenant_id).await?;
82+
PARSEABLE
83+
.metastore
84+
.delete_stream(&stream_name, &tenant_id)
85+
.await
86+
.map_err(crate::metastore::MetastoreError::into_storage_error)?;
8487
let stream_dir = PARSEABLE.get_or_create_stream(&stream_name, &tenant_id);
8588
if let Err(err) = fs::remove_dir_all(&stream_dir.data_path) {
8689
warn!(

‎src/metastore/metastore_traits.rs‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -381,6 +381,38 @@ pub trait Metastore: std::fmt::Debug + Send + Sync {
381381
node_type: NodeType,
382382
) -> Result<bool, MetastoreError>;
383383
async fn put_node_metadata(&self, obj: &dyn MetastoreObject) -> Result<(), MetastoreError>;
384+
/// Backend health probe; object storage preserves its historical HEAD check.
385+
async fn check_health(&self, tenant_id: &Option<String>) -> Result<(), MetastoreError>;
386+
/// Time a metadata read, including failed reads, for the startup banner.
387+
async fn read_latency(&self, tenant_id: &Option<String>) -> std::time::Duration;
388+
/// Preserve backend-specific stream existence rules without reading bucket metadata at callers.
389+
async fn stream_exists(
390+
&self,
391+
stream_name: &str,
392+
tenant_id: &Option<String>,
393+
) -> Result<bool, MetastoreError>;
394+
/// Whether deletion has begun and lazy loading must be suppressed.
395+
async fn is_stream_deleting(
396+
&self,
397+
stream_name: &str,
398+
tenant_id: &Option<String>,
399+
) -> Result<bool, MetastoreError>;
400+
/// Remove stream metadata and its data, preserving the backend's deletion order.
401+
async fn delete_stream(
402+
&self,
403+
stream_name: &str,
404+
tenant_id: &Option<String>,
405+
) -> Result<(), MetastoreError>;
406+
/// Delete a physical date partition together with its manifest metadata.
407+
/// Snapshot removal is performed by the caller before this operation.
408+
async fn delete_stream_partition(
409+
&self,
410+
stream_name: &str,
411+
date: &str,
412+
tenant_id: &Option<String>,
413+
) -> Result<(), MetastoreError>;
414+
/// Discovery candidates, including invalid entries which startup historically cleans up.
415+
async fn list_tenant_candidates(&self) -> Result<Vec<String>, MetastoreError>;
384416
async fn list_streams(
385417
&self,
386418
tenant_id: &Option<String>,

0 commit comments

Comments
 (0)