compute: record the export type in mz_compute_exports - #38578
Draft
antiguru wants to merge 1 commit into
Draft
Conversation
Add an `export_type` column to `mz_introspection.mz_compute_exports_per_worker` and to the derived `mz_introspection.mz_compute_exports`. The column carries `index` for index exports and the sink connection name for sink exports, so `materialized_view`, `subscribe`, `copy_to_s3_oneshot`, or `metric_sink`. A consumer asking whether every export on a replica has finished its work needs to know which exports write to storage, because only those reach a write stage. Without the type on the export relation, the only source for that fact is a catalog join through `mz_internal.mz_object_global_ids` to `mz_catalog.mz_materialized_views`. That leaves `mz_introspection` for a fact the replica already has, and it names one catalog type rather than the property that matters, so any other persist-sink-backed export would read as never-writing. The replica knows the type precisely, and `ComputeSinkConnection::name()` already renders the variants. `DataflowDescription::export_types` pairs each export with its type name and becomes the single owner of the vocabulary. The demux keeps the name in `ExportState` so dropping an export retracts the row it inserted. The column is not part of the key, which stays `(export_id, worker_id)`, so `index_by` and the index arity are unchanged. Extends `test/testdrive/introspection-sources.td` with assertions that an index reports `index`, a materialized view reports `materialized_view`, and a live subscribe reports `subscribe`. Closes: CPU-233 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add an
export_typecolumn tomz_introspection.mz_compute_exports_per_workerand to the derivedmz_introspection.mz_compute_exports. The column carriesindexfor index exports and the sink connection name for sink exports, somaterialized_view,subscribe,copy_to_s3_oneshot, ormetric_sink.A consumer asking whether every export on a replica has finished its work needs to know which exports write to storage, because only those reach a write stage. Without the type on the export relation, the only source for that fact is a catalog join through
mz_internal.mz_object_global_idstomz_catalog.mz_materialized_views. That leavesmz_introspectionfor a fact the replica already has, and it names one catalog type rather than the property that matters, so any other persist-sink-backed export would read as never-writing. The replica knows the type precisely, andComputeSinkConnection::name()already renders the variants.DataflowDescription::export_typespairs each export with its type name and becomes the single owner of the vocabulary. The demux keeps the name inExportStateso dropping an export retracts the row it inserted. The column is not part of the key, which stays(export_id, worker_id), soindex_byand the index arity are unchanged. That leaves the per-replica index column positions, theunnest(indkey)rows, and themz_indexescounts alone; what moves is the column listing inmz_catalog_server_index_accounting.slt, the autogeneratedmz_introspection.slt, and the fast-path projection incatalog_server_explain.slt.Extends
test/testdrive/introspection-sources.tdwith assertions that an index reportsindex, a materialized view reportsmaterialized_view, and a live subscribe reportssubscribe.Release notes
This release will add an
export_typecolumn tomz_introspection.mz_compute_exportsandmz_introspection.mz_compute_exports_per_worker, reporting whether a dataflow export is an index, a materialized view, a subscribe, a copy to S3, or a metric sink.Closes: CPU-233
🤖 Generated with Claude Code