Documentation | Release Notes | Migration Guide | Examples

---

---

Project Overview

SDP-META is a metadata-driven framework for Lakeflow Spark Declarative Pipelines. Define your Bronze and Silver pipelines in a JSON or YAML onboarding file — a single generic Declarative Pipeline reads the resulting DataflowSpec at runtime and builds the full processing graph automatically. No pipeline code to write.

📺 New to SDP-META? Watch the SDP-Meta Deep-Dive: Building Data Pipelines at Scale on Databricks — a video walkthrough of the framework with a Demo.

Who it's for: platform and data engineering teams standardizing repeatable Bronze/Silver pipelines across many datasets — onboarding new feeds through metadata instead of new pipeline code, with consistent data quality, quarantine, CDC, clustering, and sink patterns available through Bundles, CLI, UI, MCP, and agent workflows.

When it's not the best fit: one or two simple pipelines, Gold-layer business modeling, tables that each need unique application logic, a managed connector and downstream logic that already satisfy the complete Bronze/Silver requirement, or a need for a formal support SLA (SDP-META is a Databricks Labs project). See the Introduction for the full positioning.

Components

Metadata Interface

• Onboarding file (JSON or YAML) — sources, targets, CDC config, DQE rules. Examples: demo/conf/json/onboarding.template · demo/conf/yml/onboarding.template.yml
• Data Quality Expectations — per-table JSON or YAML rule files. Examples: demo/conf/json/dqe/customers/ · demo/conf/yml/dqe/customers/
• Silver transformation file — SQL select_exp and where_clause definitions. Examples: demo/conf/json/silver_transformations.json · demo/conf/yml/silver_transformations.yml

Generic Lakeflow Spark Declarative Pipeline

• Reads DataflowSpec at runtime and dynamically wires sources, transformations, expectations, CDC flows, and sinks
• Supports Autoloader, Delta, Kafka, Eventhub, and Snapshot sources
• Applies create_auto_cdc_flow, append_flow, and create_sink based on metadata

High-Level Process Flow

Feature Matrix

Pipeline Capabilities

Feature	Layers
Input sources — Autoloader, Delta, Kafka, Eventhub, Snapshot	Bronze, Silver
Medallion architecture	Bronze → Silver
Bronze ↔ Silver pipeline chaining (layer=bronze_silver)	Both
Custom transformation functions	Bronze, Silver
Data Quality Expectations	Bronze, Silver
Quarantine table	Bronze, Silver
Liquid clustering	Bronze, Bronze Quarantine, Silver
create_auto_cdc_flow — CDC via bronze_cdc_apply_changes	Bronze, Silver
Multi-source CDC (bronze_cdc_apply_changes_flows / silver_cdc_apply_changes_flows)	Bronze, Silver
create_auto_cdc_from_snapshot_flow — Snapshot CDC	Bronze
append_flow — via bronze_append_flows	Bronze
create_sink — Delta and Kafka sinks	Bronze, Silver
Row filters	Bronze, Silver

Deployment & Tooling

Tool	Description
Declarative Automation Bundles	Git-tracked pipelines, dev/prod targets, CI/CD-ready. Commands: bundle-init, bundle-prepare-wheel, bundle-add-flow, bundle-validate. See DAB_README.md.
SDP-META CLI	databricks labs sdp-meta onboard · deploy · bundle-*
SDP-META App	Browser-based UI for onboarding, deployment, and pipeline monitoring
MCP Server	AI-assisted pipeline scaffolding via MCP-capable AI tools (Claude Code, Cursor, Claude Desktop, and others)

Getting Started

SDP-META has three ways to use it — pick the one that matches your role and how far you intend to take it:

1. Declarative Automation Bundle (recommended) — dev/prod targets, git-tracked state, CI/CD-ready. bundle-init --quickstart skips every prompt and gets you a working bundle in one command.
2. Interactive onboard + deploy CLI — kick the tires against a single workspace.
3. SDP-META Databricks App — browser-based UI for non-developers, demos, and click-driven workflows.

See GETTING_STARTED.md for prereqs, install commands, full per-path walkthroughs, local development setup, and troubleshooting. Long form: docs site.

Upgrading from DLT-META

The v0.1.0 release renames DLT-META to SDP-META. Existing onboarding JSON/YAML files and pipeline behavior are unchanged, but new code should move to the new package, CLI, and import path:

Area	Old	New
PyPI package	dlt-meta	databricks-labs-sdp-meta
Labs CLI	databricks labs dlt-meta	databricks labs sdp-meta
Python imports	from dlt_meta import ...	from databricks.labs.sdp_meta import ...

The dlt-meta compatibility package remains available during v0.1.x, but it emits deprecation warnings and receives no new features. Legacy src.* imports are planned for removal in v0.2.0.

See the DLT-META → SDP-META migration guide for the step-by-step plan, compatibility details, and deprecation timeline.

Resources

• Documentation
• Video: SDP-Meta Deep-Dive
• FAQ
• Migration Guide
• Release Notes
• GitHub Issues

Project Support

Please note that all projects released under Databricks Labs are provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements (SLAs). They are provided AS-IS and we do not make any guarantees of any kind. Please do not submit a support ticket relating to any issues arising from the use of these projects.

Any issues discovered through the use of this project should be filed as issues on the Github Repo.
They will be reviewed as time permits, but there are no formal SLAs for support.