|
| 1 | +# Mutable metadata boundary audit |
| 2 | + |
| 3 | +Baseline: origin/main `d6a155f2`. Scope: stage 0 seams only. No PostgreSQL |
| 4 | +implementation, transactions, caching, serialization changes, or race fixes. |
| 5 | + |
| 6 | +A site below is a logical operation in a named symbol (several expressions in |
| 7 | +one operation count once). Grouped backend implementations count once per row, |
| 8 | +not once per provider. A includes reviewed operations already using the trait; |
| 9 | +these are called out so an apparent ObjectStorage helper is not mistaken for a |
| 10 | +remaining bypass. Test fixture writes are excluded from production counts. |
| 11 | + |
| 12 | +## Inventory |
| 13 | + |
| 14 | +Paths in the table are relative to `src/`. |
| 15 | + |
| 16 | +| File / symbol | Class | Objects or meaning; disposition | |
| 17 | +| --- | --- | --- | |
| 18 | +| `parseable/mod.rs::create_stream_and_schema_from_storage`, deletion check | A | `.tombstones/<tenant>/<stream>/.tombstone` HEAD; now `Metastore::is_stream_deleting`. | |
| 19 | +| `parseable/mod.rs::create_stream_and_schema_from_storage`, existence check | A | `.stream/*stream.json` discovery; now `Metastore::stream_exists`. Preserves LocalFS's canonical-only predicate and cloud's ingestor acceptance. | |
| 20 | +| `parseable/mod.rs::load_tenants`, discovery | A | Root directory candidates standing for tenants; moved unchanged to `list_tenant_candidates`. Includes unrecognized directories for historical cleanup. | |
| 21 | +| `parseable/mod.rs::load_tenants`, invalid directory cleanup | A | Whole tenant prefix, including any metadata; now existing `delete_tenant`. Same path and tenant metrics context (`None`). | |
| 22 | +| `handlers/http/logstream.rs::delete` | A | Stream prefix, metadata and data together; now `Metastore::delete_stream`, which delegates to the identical provider operation. | |
| 23 | +| `handlers/http/modal/query/querier_logstream.rs::delete` | A | Same deletion operation; same reroute. | |
| 24 | +| `handlers/http/health_check.rs::readiness_internal` | A | Provider HEAD of `.parseable.json` (LocalFS has its own check); now `check_health`, preserving the provider implementation. | |
| 25 | +| `banner.rs::storage_info` | A | Timed `.parseable.json` GET, including timing failed GETs; now `read_latency`, preserving the existing probe. | |
| 26 | +| `storage/object_storage.rs::create_stream` | A | Schema then stream JSON; already calls `put_schema` then `put_stream_json`; unchanged. | |
| 27 | +| `storage/object_storage.rs::update_time_partition_limit_in_stream` | A | Stream JSON read/overwrite through trait already; unchanged. | |
| 28 | +| `storage/object_storage.rs::update_custom_partition_in_stream` | A | Same, custom partition; unchanged. | |
| 29 | +| `storage/object_storage.rs::update_log_source_in_stream` | A | Same, log sources; unchanged. | |
| 30 | +| `storage/object_storage.rs::update_dataset_tags_and_labels_in_stream` | A | Same, dataset tags/labels; unchanged. | |
| 31 | +| `storage/object_storage.rs::update_first_event_in_stream` | A | Same, persisted first event timestamp; unchanged. | |
| 32 | +| `storage/object_storage.rs::put_stats` | A | Same, whole stats replacement; unchanged. | |
| 33 | +| `storage/object_storage.rs::put_retention` | A | Same, retention configuration; unchanged. | |
| 34 | +| `storage/object_storage.rs::upsert_stream_metadata` | A | Mode-specific stream JSON or fallback base read then mode write; already trait. Retains even the original return of base bytes after writing modified config. | |
| 35 | +| `storage/object_storage.rs::put_snapshot` | A | Upsert then snapshot replacement via trait; unchanged. | |
| 36 | +| `storage/object_storage.rs::create_stream_from_querier` | A | Base stream JSON read, stats/snapshot reset, mode-specific write; already trait. | |
| 37 | +| `storage/object_storage.rs::create_stream_from_ingestor` | A | Enumerate ingestor JSONs, merge sources/tags/labels, publish; already trait. | |
| 38 | +| `storage/object_storage.rs::create_schema_from_metastore` | A | `fetch_schema` then `put_schema`; already trait. | |
| 39 | +| `storage/object_storage.rs::get_log_source_from_storage` | A | Merge source descriptions from all stream JSONs; already trait. | |
| 40 | +| `storage/object_storage.rs::commit_schema_to_storage` / `process_schema_files` | A | Schema read/merge/write then memory update and staging cleanup; already trait. No reordering. | |
| 41 | +| `storage/store_metadata.rs::resolve_parseable_metadata` / `put_remote_metadata` | A | `.parseable.json`, RBAC, deployment and tenant configuration; already `get/put_parseable_metadata`. | |
| 42 | +| `catalog/mod.rs::update_snapshot`, `handle_existing_partition`, `finalize_snapshot_update`, `remove_manifest_from_snapshot` | A | Stream snapshot, manifest reads/writes and stats; already trait (including `put_snapshot` helper). | |
| 43 | +| `handlers/http/logstream.rs::{put_stream_hot_tier,delete_stream_hot_tier,get_stream_hot_tier_internal}` | A | Shared hot-tier enablement/config reads and whole stream JSON writes already use trait; local cache config is C below. | |
| 44 | +| `storage/retention.rs::action::delete` metadata publication | A | Remove snapshot references and update deleted stats/first event through trait-backed helpers; unchanged. | |
| 45 | +| `handlers/http/modal/mod.rs` node registration and HTTP modal node management | A | Node registry JSON; already `put/get/delete_node_metadata`; unchanged. | |
| 46 | +| `migration/mod.rs::{run_migration,migration_stream,fetch_or_create_stream_metadata,migrate_stream_metadata,setup_logstream_metadata,put_remote_metadata}` | A | Discover streams, migrate JSON versions, publish schema/stream/deployment metadata; already trait, including helper calls. Retained this policy for all live migration reads/writes. Pure transforms in `metadata_migration.rs` and `stream_metadata_migration.rs` have no storage access. | |
| 47 | +| `metastore/metastores/object_store_metastore.rs::{refresh_schema,get_all_schemas}` and `storage/object_storage.rs::schema_objects`, `storage/localfs.rs::schema_objects` | A | Schema enumeration/canonical union publication; raw storage only inside backend implementation/adapters. No application calls to `schema_objects`. | |
| 48 | +| `storage/{s3,gcs,azure_blob,localfs}.rs::{stream_exists,list_streams,list_old_streams,get_ingestor_meta_file_paths,try_delete_node_meta,check}` and `object_storage.rs::{stream_exists,get_latency}` | A | Physical metadata predicates, node file enumeration/deletion and diagnostic probes. Provider primitives remain for backend use; active application calls now enter the trait. Legacy unused primitives noted below. | |
| 49 | +| `metastore/metastores/object_store_metastore.rs` remaining metadata CRUD | A | Keystones, conversations, alerts, alert history, MTTR, LLM config, targets, outbound policy, dashboards (including conditional tombstones), chats, filters, schemas, manifests, deployment/tenant and node records. All raw object operations are backend implementation details already. | |
| 50 | +| `storage/object_storage.rs::{upload_single_parquet_file,validate_uploaded_parquet_file}` | B | Multipart Parquet write, HEAD length validation, failed-upload object cleanup; data durability/size, not catalog authority. Left unchanged. | |
| 51 | +| `storage/object_storage.rs::{get_first_and_latest_event_from_storage,extract_timestamp_for_date}` | B | Date/hour/minute directory scans implicitly stand for first/latest event timestamps with minute rounding. Future catalog bounds must preserve rounding. | |
| 52 | +| `storage/object_storage.rs::{update_storage_metrics,upload_files_from_staging}` | B | Physical Parquet sizes/date names and publication pipeline. Metrics implicitly stand for ingestion/storage totals; metadata publication already enters trait. | |
| 53 | +| `storage/retention.rs::action::delete` partition deletion | A | Mixed daily data/manifest prefix deletion now enters `delete_stream_partition`; the backend performs exactly the original prefix DELETE after snapshot publication, with the same concurrent per-date tasks. | |
| 54 | +| `storage/retention.rs::action::delete` date listing | B | Directory names imply retention candidates; remains on storage. A future catalog/GC design must select catalog-native partitions instead. | |
| 55 | +| `query/listing_table_builder.rs::ListingTableBuilder::populate_via_listing` | B | Partition prefixes imply legacy file membership; unchanged. | |
| 56 | +| `query/stream_schema_provider.rs::{scan,legacy_listing_table,parquet_scan_plan}` | B | Legacy directory-derived membership and Parquet execution; modern metadata fetches already trait. File untouched, as requested. | |
| 57 | +| `hottier.rs` download/planning and `hottier/planner.rs::build_work` | B | Remote Parquet fetches and directory-derived minute buckets imply cache work/time ranges; manifests already trait. | |
| 58 | +| `parseable/mod.rs::validate_storage` root directory occupancy probe | B | Physical nonempty-store detection (even arbitrary nonmetadata directories affect startup). Metadata existence and stream listing already trait. Kept physical occupancy check; replacing it with catalog listing changes stale-storage detection. | |
| 59 | +| `storage/{s3,gcs,azure_blob,localfs}.rs::{list_dates,list_hours,list_minutes,list_dirs_relative,delete_prefix}` | B | Data layout primitives, tenant scoping, metrics, retries and provider errors. Kept as physical adapters; metadata consumers confined to metastore. | |
| 60 | +| `storage/field_stats.rs::calculate_field_stats` | B | Derived field statistics ingested as stream data, not a mutable JSON authority; file-derived timestamp remains. | |
| 61 | +| `storage/store_metadata.rs::{get_staging_metadata,put_staging_metadata}` | C | Staging `.parseable.json` describes local deployment association; filesystem only, retained. | |
| 62 | +| `migration/mod.rs::{get_staging_metadata,put_staging_metadata}` | C | Local migration staging copy; retained. | |
| 63 | +| `parseable/streams.rs` schema staging, stream staging metadata and WAL | C | Local ingestion recovery state and schema files; retained. Shared schema commit already trait. | |
| 64 | +| `hottier.rs::{read_hot_tier_config,put_hot_tier,delete_hot_tier}` and `hottier/local_state.rs` | C | Local `.hot_tier.json`, cache manifests/runtime state and local filesystem operations; retained. Shared hot-tier enablement/config in stream JSON already trait. | |
| 65 | +| `storage/object_storage.rs::list_tombstoned_streams` and unused legacy provider discovery/node helpers | D | No production callers of tombstone discovery, `list_old_streams`, `get_ingestor_meta_file_paths`, or `try_delete_node_meta` outside backend/adapters. Retain dormant compatibility APIs and fixture tests. A future deletion recovery/legacy-layout import feature must expose semantic metastore methods before using them. | |
| 66 | + |
| 67 | +Counts: **33 A, 10 B, 4 C, 1 D** logical operation groups. Nine application |
| 68 | +call sites changed; the other A groups already persisted through the trait or |
| 69 | +implement its physical adapter. |
| 70 | + |
| 71 | +## Contract additions |
| 72 | + |
| 73 | +Existing methods and signatures are unchanged. PostgreSQL must implement: |
| 74 | + |
| 75 | +```rust |
| 76 | +async fn check_health(&self, tenant_id: &Option<String>) -> Result<(), MetastoreError>; |
| 77 | +async fn read_latency(&self, tenant_id: &Option<String>) -> std::time::Duration; |
| 78 | +async fn stream_exists(&self, stream_name: &str, tenant_id: &Option<String>) -> Result<bool, MetastoreError>; |
| 79 | +async fn is_stream_deleting(&self, stream_name: &str, tenant_id: &Option<String>) -> Result<bool, MetastoreError>; |
| 80 | +async fn delete_stream(&self, stream_name: &str, tenant_id: &Option<String>) -> Result<(), MetastoreError>; |
| 81 | +async fn delete_stream_partition(&self, stream_name: &str, date: &str, tenant_id: &Option<String>) -> Result<(), MetastoreError>; |
| 82 | +async fn list_tenant_candidates(&self) -> Result<Vec<String>, MetastoreError>; |
| 83 | +``` |
| 84 | + |
| 85 | +`delete_stream` and `delete_stream_partition` include physical deletion: PostgreSQL must orchestrate catalog |
| 86 | +and data cleanup, just as existing `delete_tenant` already includes both. This |
| 87 | +seam introduces no transaction or new deletion protocol. `list_tenant_candidates` |
| 88 | +preserves invalid candidates for cleanup; another backend may return only its |
| 89 | +registered tenants. The legacy deletion-marker check remains separate from |
| 90 | +existence, in the same order. It does not introduce marker publication. |
| 91 | + |
| 92 | +## Races deliberately preserved |
| 93 | + |
| 94 | +All stream-setting/stats/retention/first-event/snapshot updates read and blindly |
| 95 | +replace the entire stream JSON. Schema commit and refresh merge schemas then |
| 96 | +unconditionally publish; concurrent writers can lose fields. Manifest updates |
| 97 | +and stream snapshot publication are separate unconditional writes; retention |
| 98 | +can race ingestion, settings, stream deletion and queries. RBAC/deployment |
| 99 | +metadata, alert state/history, filters and node registration have blind |
| 100 | +whole-record writes or separate discovery/write/delete sequences. Tenant |
| 101 | +validation/cleanup races concurrent tenant creation. Check-then-load and |
| 102 | +check-then-delete remain racy. No CAS conversion, locks, retries, cache, or |
| 103 | +transaction was added. Dashboard CAS behavior is unchanged. |
| 104 | + |
| 105 | +## Guard and evidence |
| 106 | + |
| 107 | +`metastore::boundary_tests` scans production Rust source before the trailing |
| 108 | +`#[cfg(test)]` section. Outside the object-store metastore and four provider |
| 109 | +adapters it rejects every raw `put_object`, `upload_file`, conditional PUT and |
| 110 | +conditional DELETE, regardless of literal/computed key. It permits exactly one |
| 111 | +multipart publisher in `object_storage.rs`, exactly one failed-Parquet DELETE |
| 112 | +there, and no raw prefix DELETEs in application code. Tests verify |
| 113 | +rejection of computed keys and whitespace-split method calls. This is a |
| 114 | +conservative source policy, not runtime provenance or Rust type analysis. It |
| 115 | +cannot detect mutations through new APIs, raw object_store clients, macros, |
| 116 | +code after the first test section, or changes hidden in allowed adapters. The |
| 117 | +existing allowed physical publishers still require review if their arguments |
| 118 | +change. It does not prove read routing or transaction safety. |
| 119 | + |
| 120 | +Compatibility evidence: the nine application bypasses now delegate to exactly |
| 121 | +the original provider operations. Provider errors are unwrapped again at the |
| 122 | +rerouted legacy call sites, preserving HTTP error mapping and error text. JSON encoders, paths, payloads, write modes, |
| 123 | +operation order and provider implementations are unchanged. The LocalFS |
| 124 | +lifecycle fixture compares discovery and existence against raw provider results, |
| 125 | +asserts exact JSON bytes survive probes, verifies the separate tombstone survives |
| 126 | +stream prefix deletion, and compares missing-stream deletion errors. A second |
| 127 | +fixture verifies partition deletion removes both Parquet and manifests while |
| 128 | +preserving adjacent-date and other-tenant bytes. Existing |
| 129 | +schema-refresh/manifest/dashboard/provider tests remain in the full library |
| 130 | +suite. No live service or bucket was used; no container was started. No |
| 131 | +main-versus-branch MinIO object diff was performed, so provider integration parity |
| 132 | +is supported by unchanged delegates and unit tests, not an end-to-end claim. |
| 133 | + |
| 134 | +Open follow-up: catalog-native deletion/GC, imported legacy file membership, |
| 135 | +first/latest timestamp semantics, and removal or semantic exposure of dormant |
| 136 | +legacy APIs need later stages. Backend selection and PostgreSQL implementation |
| 137 | +are intentionally outside this change. |
| 138 | + |
| 139 | +Final validation on this branch: |
| 140 | + |
| 141 | +- `P_BUILD_NO_UI=1 CARGO_BUILD_JOBS=4 cargo clippy --all-targets`: exit 0; |
| 142 | + 21 existing library-test warnings, none in changed files. |
| 143 | +- `P_BUILD_NO_UI=1 CARGO_BUILD_JOBS=4 cargo test --lib`: exit 0; |
| 144 | + **1,379 passed, 0 failed, 4 ignored** (1,383 discovered), 79.65 seconds. |
| 145 | +- Both source-guard tests and both LocalFS compatibility fixtures passed. |
| 146 | +- `cargo fmt` and `git diff --check` passed. Cargo target remained in this worktree. |
0 commit comments