Feature/v0.0.10

README.md

@@ -32,10 +32,10 @@ In practice, a single generic pipeline reads the Dataflowspec and uses it to orc
 #### Generic Lakeflow Declarative Pipeline
 
 - Apply appropriate readers based on input metadata
-- Apply data quality rules with DLT expectations
+- Apply data quality rules with Lakeflow Declarative Pipeline expectations
 - Apply CDC apply changes if specified in metadata
 - Builds Lakeflow Declarative Pipeline graph based on input/output metadata
-- Launch Lakeflow Declarative Pipeline pipeline
+- Launch Lakeflow Declarative pipeline
 
 ## High-Level Process Flow:
 
@@ -45,7 +45,7 @@ In practice, a single generic pipeline reads the Dataflowspec and uses it to orc
 
 ![DLT-META Stages](./docs/static/images/dlt-meta_stages.png)
 
-## DLT-META DLT Features support
+## DLT-META Lakeflow Declarative Pipeline Features support
 | Features  | DLT-META Support |
 | ------------- | ------------- |
 | Input data sources  | Autoloader, Delta, Eventhub, Kafka, snapshot  |
@@ -58,7 +58,7 @@ In practice, a single generic pipeline reads the Dataflowspec and uses it to orc
 | [append_flow](https://docs.databricks.com/en/delta-live-tables/flows.html#use-append-flow-to-write-to-a-streaming-table-from-multiple-source-streams) API support | Bronze layer|
 | Liquid cluster support | Bronze, Bronze Quarantine, Silver tables|
 | [DLT-META CLI](https://databrickslabs.github.io/dlt-meta/getting_started/dltmeta_cli/) |  ```databricks labs dlt-meta onboard```, ```databricks labs dlt-meta deploy``` |
-| Bronze and Silver pipeline chaining | Deploy dlt-meta pipeline with ```layer=bronze_silver``` option using Direct publishing mode |
+| Bronze and Silver pipeline chaining | Deploy dlt-meta pipeline with ```layer=bronze_silver``` option using default publishing mode |
 | [create_sink](https://docs.databricks.com/aws/en/dlt-ref/dlt-python-ref-sink) API support |Supported formats:```external delta table , kafka``` Bronze, Silver layers|
 | [Databricks Asset Bundles](https://docs.databricks.com/aws/en/dev-tools/bundles/) | Supported
 | [DLT-META UI](https://github.com/databrickslabs/dlt-meta/tree/main/lakehouse_app#dlt-meta-lakehouse-app-setup) | Uses Databricks Lakehouse DLT-META App
@@ -166,7 +166,7 @@ The command will prompt you to provide pipeline configuration details.
 ![deployingDLTMeta_bronze_silver.gif](docs/static/images/deployingDLTMeta_bronze_silver.gif)
 
 Above deploy cli command will:
-1. Deploy Lakeflow Declarative pipeline with dlt-meta configuration like ```layer```, ```group```, ```dataflowSpec table details``` etc to your databricks workspace
+1. Deploy Lakeflow Declarative Pipeline with dlt-meta configuration like ```layer```, ```group```, ```dataflowSpec table details``` etc to your databricks workspace
 2. Display message: ```dlt-meta pipeline={pipeline_id} created and launched with update_id={pipeline_update_id}, url=https://{databricks workspace url}/#joblist/pipelines/{pipeline_id}```
 3. Pipline URL will automatically open in your defaul browser.
 

demo/README.md

@@ -13,7 +13,7 @@
 ## [DAIS 2023 Session Recording](https://www.youtube.com/watch?v=WYv5haxLlfA)
 This Demo launches Bronze and Silver pipelines with following activities:
 - Customer and Transactions feeds for initial load
-- Adds new feeds Product and Stores to existing Bronze and Silver DLT pipelines with metadata changes.
+- Adds new feeds Product and Stores to existing Bronze and Silver Lakeflow Declarative pipeline with metadata changes.
 - Runs Bronze and Silver pipeline for incremental load for CDC events
 
 ### Steps:

demo/notebooks/afam_cloudfiles_runners/validate.py

@@ -23,7 +23,7 @@
 }
 
 
-log_list.append("Validating DLT Bronze and Silver Table Counts...")
+log_list.append("Validating Lakeflow Declarative Pipeline Bronze and Silver Table Counts...")
 for table, counts in UC_TABLES.items():
     query = spark.sql(f"SELECT count(*) as cnt FROM {table}")
     cnt = query.collect()[0].cnt

demo/notebooks/afam_eventhub_runners/validate.py

@@ -21,7 +21,7 @@
     f"{uc_catalog_name}.{bronze_schema}.bronze_{run_id}_iot_quarantine": 2
 }
 
-log_list.append("Validating DLT EVenthub Bronze Table Counts...")
+log_list.append("Validating Lakeflow Declarative Pipeline EVenthub Bronze Table Counts...")
 tables = UC_TABLES if uc_enabled else NON_UC_TABLES
 for table, counts in tables.items():
     query = spark.sql(f"SELECT count(*) as cnt FROM {table}")

demo/notebooks/dlt_sink_runners/validate.py

@@ -15,7 +15,7 @@
     f"{uc_catalog_name}.{bronze_schema}.bronze_{run_id}_iot_quarantine": 2,
 }
 
-log_list.append("Validating DLT EVenthub Bronze Table Counts...")
+log_list.append("Validating Lakeflow Declarative Pipeline EVenthub Bronze Table Counts...")
 for table, counts in TABLES.items():
     query = spark.sql(f"SELECT count(*) as cnt FROM {table}")
     cnt = query.collect()[0].cnt

docs/content/_index.md

@@ -36,7 +36,7 @@ In practice, a single generic pipeline reads the Dataflowspec and uses it to orc
     - Option#2: [Manual Job](https://databrickslabs.github.io/dlt-meta/getting_started/dltmeta_manual/#onboardjob)
     - option#3: [Databricks Notebook](https://databrickslabs.github.io/dlt-meta/getting_started/dltmeta_manual/#option2-databricks-notebook)
 
-- Dataflow DLT Pipeline
+- Dataflow Lakeflow Declarative Pipeline
     - Option#1: [DLT-META CLI](https://databrickslabs.github.io/dlt-meta/getting_started/dltmeta_cli/#dataflow-dlt-pipeline)
     - Option#2: [DLT-META MANUAL](https://databrickslabs.github.io/dlt-meta/getting_started/dltmeta_manual/#dataflow-dlt-pipeline)
 
@@ -53,7 +53,7 @@ In practice, a single generic pipeline reads the Dataflowspec and uses it to orc
 | [append_flow](https://docs.databricks.com/aws/en/dlt-ref/dlt-python-ref-append-flow) API support | Bronze layer|
 | Liquid cluster support | Bronze, Bronze Quarantine, Silver, Silver Quarantine tables|
 | [DLT-META CLI](https://databrickslabs.github.io/dlt-meta/getting_started/dltmeta_cli/) |  ```databricks labs dlt-meta onboard```, ```databricks labs dlt-meta deploy``` |
-| Bronze and Silver pipeline chaining | Deploy dlt-meta pipeline with ```layer=bronze_silver``` option using Direct publishing mode |
+| Bronze and Silver pipeline chaining | Deploy dlt-meta pipeline with ```layer=bronze_silver``` option using default publishing mode |
 | [DLT Sinks](https://docs.databricks.com/aws/en/dlt/dlt-sinks) | Supported formats:external ```delta table```, ```kafka```.Bronze, Silver layers|
 ## How much does it cost ?
 DLT-META does not have any **direct cost** associated with it other than the cost to run the Databricks Lakeflow Declarative Pipelines 

docs/content/additionals/integration_tests.md

@@ -71,7 +71,7 @@ draft: false
 ,0
 0,Completed Bronze Lakeflow Declarative Pipeline.
 1,Completed Silver Lakeflow Declarative Pipeline.
-2,Validating DLT Bronze and Silver Table Counts...
+2,Validating Lakeflow Declarative Pipeline Bronze and Silver Table Counts...
 3,Validating Counts for Table bronze_7d1d3ccc9e144a85b07c23110ea50133.transactions.
 4,Expected: 10002 Actual: 10002. Passed!
 5,Validating Counts for Table bronze_7d1d3ccc9e144a85b07c23110ea50133.transactions_quarantine.

docs/content/demo/Apply_Changes_From_Snapshot.md

@@ -9,11 +9,11 @@ draft: false
   - This demo will perform following steps
     - Showcase onboarding process for apply changes from snapshot pattern
     - Run onboarding for the bronze stores and products tables, which contains data snapshot data in csv files.
-    - Run Bronze DLT to load initial snapshot (LOAD_1.csv)
+    - Run Bronze Lakeflow Declarative Pipeline to load initial snapshot (LOAD_1.csv)
     - Upload incremental snapshot LOAD_2.csv version=2 for stores and product
-    - Run Bronze DLT to load incremental snapshot (LOAD_2.csv). Stores is scd_type=2 so updated records will expired and added new records with version_number. Products is scd_type=1 so in case records missing for scd_type=1 will be deleted.
+    - Run Bronze Lakeflow Declarative Pipeline to load incremental snapshot (LOAD_2.csv). Stores is scd_type=2 so updated records will expired and added new records with version_number. Products is scd_type=1 so in case records missing for scd_type=1 will be deleted.
     - Upload incremental snapshot LOAD_3.csv version=3 for stores and product
-    - Run Bronze DLT to load incremental snapshot (LOAD_3.csv). Stores is scd_type=2 so updated records will expired and added new records with version_number. Products is scd_type=1 so in case records missing for scd_type=1 will be deleted.
+    - Run Bronze Lakeflow Declarative Pipeline to load incremental snapshot (LOAD_3.csv). Stores is scd_type=2 so updated records will expired and added new records with version_number. Products is scd_type=1 so in case records missing for scd_type=1 will be deleted.
 
 
 ### Steps:

docs/content/demo/DAIS.md

@@ -8,10 +8,10 @@ draft: false
 ### DAIS 2023 DEMO:
 #### [DAIS 2023 Session Recording](https://www.youtube.com/watch?v=WYv5haxLlfA)
 
-This demo showcases DLT-META's capabilities of creating Bronze and Silver DLT pipelines with initial and incremental mode automatically.
+This demo showcases DLT-META's capabilities of creating Bronze and Silver Lakeflow Declarative pipeline with initial and incremental mode automatically.
 - Customer and Transactions feeds for initial load
-- Adds new feeds Product and Stores to existing Bronze and Silver DLT pipelines with metadata changes.
-- Runs Bronze and Silver DLT for incremental load for CDC events
+- Adds new feeds Product and Stores to existing Bronze and Silver Lakeflow Declarative pipeline with metadata changes.
+- Runs Bronze and Silver Lakeflow Declarative Pipeline for incremental load for CDC events
 
 #### Steps to launch DAIS demo in your Databricks workspace:
 1. Launch Command Prompt

docs/content/demo/Techsummit.md

@@ -6,7 +6,7 @@ draft: false
 ---
 
 ### Databricks Tech Summit FY2024 DEMO:
-This demo will launch auto generated tables(100s) inside single bronze and silver DLT pipeline using dlt-meta.
+This demo will launch auto generated tables(100s) inside single bronze and silver Lakeflow Declarative Pipeline using dlt-meta.
 
 1. Launch Command Prompt
 

docs/content/faq/execution.md

@@ -103,7 +103,7 @@ Yes! Please follow below steps:
 - You can run onboarding for additional silver customer_clean table by having [onboarding file](https://github.com/databrickslabs/dlt-meta/blob/main/examples/onboarding_silverfanout.template) and [silver transformation](https://github.com/databrickslabs/dlt-meta/blob/main/examples/silver_transformations_fanout.template) with filter condition for fan out.
 
 - Run onboarding for slilver layer in append mode("overwrite": "False") so it will append to existing silver tables.
-When you launch DLT pipeline it will read silver onboarding and run DLT for bronze source and silver as target
+When you launch Lakeflow Declarative Pipeline it will read silver onboarding and run Lakeflow Declarative Pipeline for bronze source and silver as target
 
 **Q. How can I do type1 or type2 merge to target table?**
 

docs/content/getting_started/dltmeta_cli.md

@@ -76,7 +76,7 @@ draft: false
    4. Job URL will automatically open in your default browser.
 
 
-- Once onboarding jobs is finished deploy `bronze` and `silver` DLT using below command
+- Once onboarding jobs is finished deploy `bronze` and `silver` Lakeflow Declarative Pipeline using below command
 
 ## DLT-META Lakeflow Declarative Pipeline: 
 

docs/content/getting_started/metadatapreperation.md

@@ -26,7 +26,7 @@ The `onboarding.json` file contains links to [silver_transformations.json](https
 | Field | Description |
 | :-----------: | :----------- |
 | data_flow_id | This is unique identifier for pipeline |
-| data_flow_group | This is group identifier for launching multiple pipelines under single DLT |
+| data_flow_group | This is group identifier for launching multiple pipelines under single Lakeflow Declarative Pipeline |
 | source_format | Source format e.g `cloudFiles`, `eventhub`, `kafka`, `delta`, `snapshot` |
 | source_details | This map Type captures all source details for cloudfiles = `source_schema_path`, `source_path_{env}`, `source_catalog`, `source_database`, `source_metadata` For eventhub= `source_schema_path` , `eventhub.accessKeyName`, `eventhub.accessKeySecretName`, `eventhub.name` , `eventhub.secretsScopeName` , `kafka.sasl.mechanism`, `kafka.security.protocol`, `eventhub.namespace`, `eventhub.port`. For Source schema file spark DDL schema format parsing is supported <br> In case of custom schema format then write schema parsing function `bronze_schema_mapper(schema_file_path, spark):Schema` and provide to `OnboardDataflowspec` initialization <br> e.g `onboardDataFlowSpecs = OnboardDataflowspec(spark, dict_obj,bronze_schema_mapper).onboardDataFlowSpecs()`.<br> For cloudFiles option _metadata columns addtiion there is `source_metadata` tag with attributes: `include_autoloader_metadata_column` flag (`True` or `False` value) will add _metadata column to target bronze dataframe, `autoloader_metadata_col_name` if this provided then will be used to rename _metadata to this value otherwise default is `source_metadata`,`select_metadata_cols:{key:value}` will be used to extract columns from _metadata. key is target dataframe column name and value is expression used to add column from _metadata column. <br> for snapshot= `snapshot_format`, `source_path_{env}` |
 | bronze_catalog_{env} | Unity catalog name |         
@@ -39,8 +39,8 @@ The `onboarding.json` file contains links to [silver_transformations.json](https
 | bronze_cdc_apply_changes | Bronze cdc apply changes Json |
 | bronze_apply_changes_from_snapshot | Bronze apply changes from snapshot Json e.g. Mandatory fields: keys=["userId"], scd_type=`1` or `2` optional fields: track_history_column_list=`[col1]`, track_history_except_column_list=`[col2]` | 
 | bronze_table_path_{env} | Bronze table storage path.|
-| bronze_table_properties | DLT table properties map. e.g. `{"pipelines.autoOptimize.managed": "false" , "pipelines.autoOptimize.zOrderCols": "year,month", "pipelines.reset.allowed": "false" }` |
-| bronze_sink | DLT Sink API properties: e.g Delta: `{"name": "bronze_sink","format": "delta","options": {"tableName": "my_catalog.my_schema.my_table"}}`, Kafka:`{"name": "bronze_sink","format": "kafka","options": { "kafka.bootstrap.servers": "host:port","subscribe": "my_topic"}}` |
+| bronze_table_properties | Lakeflow Declarative Pipeline table properties map. e.g. `{"pipelines.autoOptimize.managed": "false" , "pipelines.autoOptimize.zOrderCols": "year,month", "pipelines.reset.allowed": "false" }` |
+| bronze_sink | Lakeflow Declarative Pipeline Sink API properties: e.g Delta: `{"name": "bronze_sink","format": "delta","options": {"tableName": "my_catalog.my_schema.my_table"}}`, Kafka:`{"name": "bronze_sink","format": "kafka","options": { "kafka.bootstrap.servers": "host:port","subscribe": "my_topic"}}` |
 | bronze_data_quality_expectations_json | Bronze table data quality expectations |
 | bronze_catalog_quarantine_{env} | Unity catalog name | 
 | bronze_database_quarantine_{env} | Bronze database for quarantine data which fails expectations. |
@@ -49,7 +49,7 @@ The `onboarding.json` file contains links to [silver_transformations.json](https
 | bronze_quarantine_table_path_{env} | Bronze database for quarantine data which fails expectations. |
 | bronze_quarantine_table_partitions | Bronze quarantine tables partition cols |
 | bronze_quarantine_table_cluster_by | Bronze quarantine tables cluster cols |
-| bronze_quarantine_table_properties | DLT table properties map. e.g. `{"pipelines.autoOptimize.managed": "false" , "pipelines.autoOptimize.zOrderCols": "year,month", "pipelines.reset.allowed": "false" }` |
+| bronze_quarantine_table_properties | Lakeflow Declarative Pipeline table properties map. e.g. `{"pipelines.autoOptimize.managed": "false" , "pipelines.autoOptimize.zOrderCols": "year,month", "pipelines.reset.allowed": "false" }` |
 | bronze_append_flows | Bronze table append flows json. e.g.`"bronze_append_flows":[{"name":"customer_bronze_flow", "create_streaming_table": false,"source_format": "cloudFiles", "source_details": {"source_database": "APP","source_table":"CUSTOMERS", "source_path_dev": "tests/resources/data/customers", "source_schema_path": "tests/resources/schema/customer_schema.ddl"},"reader_options": {"cloudFiles.format": "json","cloudFiles.inferColumnTypes": "true","cloudFiles.rescuedDataColumn": "_rescued_data"},"once": true}]` |
 | silver_catalog_{env} | Unit Catalog name. |
 | silver_database_{env} | Silver database name. |
@@ -59,8 +59,8 @@ The `onboarding.json` file contains links to [silver_transformations.json](https
 | silver_cluster_by | Silver tables cluster by cols list |
 | silver_cdc_apply_changes | Silver cdc apply changes Json |
 | silver_table_path_{env} | Silver table storage path. |
-| silver_table_properties | DLT table properties map. e.g. `{"pipelines.autoOptimize.managed": "false" , "pipelines.autoOptimize.zOrderCols": "year,month", "pipelines.reset.allowed": "false"}` |
-| silver_sink | DLT Sink API properties: e.g Delta:`{"name": "silver_sink","format": "delta","options": {"tableName": "my_catalog.my_schema.my_table"}}`, Kafka:`{"name": "silver_sink","format": "kafka","options": { "kafka.bootstrap.servers": "host:port","subscribe": "my_topic"}}`|
+| silver_table_properties | Lakeflow Declarative Pipeline table properties map. e.g. `{"pipelines.autoOptimize.managed": "false" , "pipelines.autoOptimize.zOrderCols": "year,month", "pipelines.reset.allowed": "false"}` |
+| silver_sink | Lakeflow Declarative Pipeline Sink API properties: e.g Delta:`{"name": "silver_sink","format": "delta","options": {"tableName": "my_catalog.my_schema.my_table"}}`, Kafka:`{"name": "silver_sink","format": "kafka","options": { "kafka.bootstrap.servers": "host:port","subscribe": "my_topic"}}`|
 | silver_transformation_json | Silver table sql transformation json path |
 | silver_data_quality_expectations_json_{env} | Silver table data quality expectations json file path
 | silver_append_flows | Silver table append flows json. e.g.`"silver_append_flows":[{"name":"customer_bronze_flow", 

docs/content/releases/_index.md

@@ -17,7 +17,7 @@ draft: false
 - Fixed issue cluster by not working with bronze append only table [PR](https://github.com/databrickslabs/dlt-meta/issues/197)
 - Fixed issue view name containing period when using DPM [PR](https://github.com/databrickslabs/dlt-meta/issues/169)
 - Fixed issue CLI onboarding overwrite option always set to True [PR](https://github.com/databrickslabs/dlt-meta/issues/163)
-- Fixed issue Silver DLT not creating based on passed database [PR](https://github.com/databrickslabs/dlt-meta/issues/160)
+- Fixed issue Silver Lakeflow Declarative Pipeline not creating based on passed database [PR](https://github.com/databrickslabs/dlt-meta/issues/160)
 - Fixed issue PyPI download stats display [PR](https://github.com/databrickslabs/dlt-meta/issues/200)
 - Fixed issue Silver Data Quality not working [PR](https://github.com/databrickslabs/dlt-meta/issues/156)
 - Fixed issue Removed DPM flag check inside dataflowpipeline [PR](https://github.com/databrickslabs/dlt-meta/issues/177)
@@ -87,7 +87,7 @@ draft: false
 ### 2. Databricks Labs CLI Support ([#28](https://github.com/databrickslabs/dlt-meta/pull/28)) 
 - Added two commands for DLT-META
 - onboard: Captures all onboarding details from command line and launch onboarding job to your databricks workspace
-- deploy: Captures all DLT pipeline details from command line and launch DLT pipeline to your databricks workspace
+- deploy: Captures all Lakeflow Declarative Pipeline details from command line and launch Lakeflow Declarative Pipeline to your databricks workspace
 
 ### Updates 
 - Readme and docs to include above features

integration_tests/README.md

@@ -101,9 +101,9 @@
 11. Output of a successful run should have the following in the file
 ```
 ,0
-0,Completed Bronze DLT Pipeline.
-1,Completed Silver DLT Pipeline.
-2,Validating DLT Bronze and Silver Table Counts...
+0,Completed Bronze Lakeflow Declarative Pipeline.
+1,Completed Silver Lakeflow Declarative Pipeline.
+2,Validating Lakeflow Declarative Pipeline Bronze and Silver Table Counts...
 3,Validating Counts for Table bronze_7d1d3ccc9e144a85b07c23110ea50133.transactions.
 4,Expected: 10002 Actual: 10002. Passed!
 5,Validating Counts for Table bronze_7d1d3ccc9e144a85b07c23110ea50133.transactions_quarantine.