Skip to content

compute: record the export type in mz_compute_exports - #38578

Draft
antiguru wants to merge 1 commit into
MaterializeInc:mainfrom
antiguru:cpu-233-export-type
Draft

compute: record the export type in mz_compute_exports#38578
antiguru wants to merge 1 commit into
MaterializeInc:mainfrom
antiguru:cpu-233-export-type

Conversation

@antiguru

@antiguru antiguru commented Aug 31, 2026

Copy link
Copy Markdown
Member

Add an export_type column to mz_introspection.mz_compute_exports_per_worker and to the derived mz_introspection.mz_compute_exports. The column carries index for index exports and the sink connection name for sink exports, so materialized_view, subscribe, copy_to_s3_oneshot, or metric_sink.

A consumer asking whether every export on a replica has finished its work needs to know which exports write to storage, because only those reach a write stage. Without the type on the export relation, the only source for that fact is a catalog join through mz_internal.mz_object_global_ids to mz_catalog.mz_materialized_views. That leaves mz_introspection for a fact the replica already has, and it names one catalog type rather than the property that matters, so any other persist-sink-backed export would read as never-writing. The replica knows the type precisely, and ComputeSinkConnection::name() already renders the variants.

DataflowDescription::export_types pairs each export with its type name and becomes the single owner of the vocabulary. The demux keeps the name in ExportState so dropping an export retracts the row it inserted. The column is not part of the key, which stays (export_id, worker_id), so index_by and the index arity are unchanged. That leaves the per-replica index column positions, the unnest(indkey) rows, and the mz_indexes counts alone; what moves is the column listing in mz_catalog_server_index_accounting.slt, the autogenerated mz_introspection.slt, and the fast-path projection in catalog_server_explain.slt.

Extends test/testdrive/introspection-sources.td with assertions that an index reports index, a materialized view reports materialized_view, and a live subscribe reports subscribe.

Release notes

This release will add an export_type column to mz_introspection.mz_compute_exports and mz_introspection.mz_compute_exports_per_worker, reporting whether a dataflow export is an index, a materialized view, a subscribe, a copy to S3, or a metric sink.

Closes: CPU-233

🤖 Generated with Claude Code

Add an `export_type` column to `mz_introspection.mz_compute_exports_per_worker`
and to the derived `mz_introspection.mz_compute_exports`. The column carries
`index` for index exports and the sink connection name for sink exports, so
`materialized_view`, `subscribe`, `copy_to_s3_oneshot`, or `metric_sink`.

A consumer asking whether every export on a replica has finished its work needs
to know which exports write to storage, because only those reach a write stage.
Without the type on the export relation, the only source for that fact is a
catalog join through `mz_internal.mz_object_global_ids` to
`mz_catalog.mz_materialized_views`. That leaves `mz_introspection` for a fact
the replica already has, and it names one catalog type rather than the property
that matters, so any other persist-sink-backed export would read as
never-writing. The replica knows the type precisely, and
`ComputeSinkConnection::name()` already renders the variants.

`DataflowDescription::export_types` pairs each export with its type name and
becomes the single owner of the vocabulary. The demux keeps the name in
`ExportState` so dropping an export retracts the row it inserted. The column is
not part of the key, which stays `(export_id, worker_id)`, so `index_by` and the
index arity are unchanged.

Extends `test/testdrive/introspection-sources.td` with assertions that an index
reports `index`, a materialized view reports `materialized_view`, and a live
subscribe reports `subscribe`.

Closes: CPU-233

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant