├── LICENSE ├── README.md ├── chapter01 ├── My First DLT Pipeline.py └── README.md ├── chapter02 ├── 02-Publishing Datasets to Unity Catalog.sql ├── 03a-Generate Postgres Table.py ├── 03b-Generate Mock Taxi Trip Data.py ├── 03c-Taxi Trips DLT Pipeline.py └── README.md ├── chapter03 ├── 01-Taxi Trip Data Generator.py ├── 01a-Creating Data Contstraints.sql ├── 02-Enforcing Data Quality.py ├── 03a-Decoupling Expectations.py ├── 03b-Decoupling Expectations Pipeline.py ├── 04-Implementing a Quarantine Table.py └── README.md ├── chapter04 ├── README.md ├── Random Taxi Trip Data Generator.py ├── Taxi Trip Data Pipeline.py └── Update Autoscaling Mode.py ├── chapter05 ├── Data Lineage Demo.sql ├── Dynamic Views Demo.py └── README.md ├── chapter06 ├── Generate Fake Text Documents.py ├── Preprocess Text Documents.py └── README.md ├── chapter07 ├── 01-Lineage Data Generator.py ├── 02-Working with the Data Lineage REST API.py ├── Predicting Carbon Footprint.py └── README.md ├── chapter08 ├── 01-Hello World │ └── main.tf ├── 02-Deploying-DLT-Pipelines │ └── main.tf └── README.md ├── chapter09 ├── 01-Hello World │ ├── databricks.yml │ └── src │ │ └── hello_dab_world.py ├── 02-my-first-dab │ └── my_first_dab │ │ ├── databricks.yml │ │ ├── resources │ │ ├── my_first_dab_job.yml │ │ └── my_first_dab_pipeline.yml │ │ └── src │ │ └── dlt_pipeline.ipynb ├── 03-GitHub Actions │ └── dab_deployment_workflow.yml └── README.md └── chapter10 ├── 01-Pipeline Health Monitoring.py ├── 02-Querying the Event Log.py ├── 03-Creating a Lakehouse Monitor.py ├── 04a-IoT Device Data Generator.py ├── 04b-IoT Device Data Pipeline.py └── README.md /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/README.md -------------------------------------------------------------------------------- /chapter01/My First DLT Pipeline.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter01/My First DLT Pipeline.py -------------------------------------------------------------------------------- /chapter01/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter01/README.md -------------------------------------------------------------------------------- /chapter02/02-Publishing Datasets to Unity Catalog.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter02/02-Publishing Datasets to Unity Catalog.sql -------------------------------------------------------------------------------- /chapter02/03a-Generate Postgres Table.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter02/03a-Generate Postgres Table.py -------------------------------------------------------------------------------- /chapter02/03b-Generate Mock Taxi Trip Data.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter02/03b-Generate Mock Taxi Trip Data.py -------------------------------------------------------------------------------- /chapter02/03c-Taxi Trips DLT Pipeline.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter02/03c-Taxi Trips DLT Pipeline.py -------------------------------------------------------------------------------- /chapter02/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter02/README.md -------------------------------------------------------------------------------- /chapter03/01-Taxi Trip Data Generator.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter03/01-Taxi Trip Data Generator.py -------------------------------------------------------------------------------- /chapter03/01a-Creating Data Contstraints.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter03/01a-Creating Data Contstraints.sql -------------------------------------------------------------------------------- /chapter03/02-Enforcing Data Quality.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter03/02-Enforcing Data Quality.py -------------------------------------------------------------------------------- /chapter03/03a-Decoupling Expectations.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter03/03a-Decoupling Expectations.py -------------------------------------------------------------------------------- /chapter03/03b-Decoupling Expectations Pipeline.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter03/03b-Decoupling Expectations Pipeline.py -------------------------------------------------------------------------------- /chapter03/04-Implementing a Quarantine Table.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter03/04-Implementing a Quarantine Table.py -------------------------------------------------------------------------------- /chapter03/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter03/README.md -------------------------------------------------------------------------------- /chapter04/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter04/README.md -------------------------------------------------------------------------------- /chapter04/Random Taxi Trip Data Generator.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter04/Random Taxi Trip Data Generator.py -------------------------------------------------------------------------------- /chapter04/Taxi Trip Data Pipeline.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter04/Taxi Trip Data Pipeline.py -------------------------------------------------------------------------------- /chapter04/Update Autoscaling Mode.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter04/Update Autoscaling Mode.py -------------------------------------------------------------------------------- /chapter05/Data Lineage Demo.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter05/Data Lineage Demo.sql -------------------------------------------------------------------------------- /chapter05/Dynamic Views Demo.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter05/Dynamic Views Demo.py -------------------------------------------------------------------------------- /chapter05/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter05/README.md -------------------------------------------------------------------------------- /chapter06/Generate Fake Text Documents.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter06/Generate Fake Text Documents.py -------------------------------------------------------------------------------- /chapter06/Preprocess Text Documents.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter06/Preprocess Text Documents.py -------------------------------------------------------------------------------- /chapter06/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter06/README.md -------------------------------------------------------------------------------- /chapter07/01-Lineage Data Generator.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter07/01-Lineage Data Generator.py -------------------------------------------------------------------------------- /chapter07/02-Working with the Data Lineage REST API.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter07/02-Working with the Data Lineage REST API.py -------------------------------------------------------------------------------- /chapter07/Predicting Carbon Footprint.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter07/Predicting Carbon Footprint.py -------------------------------------------------------------------------------- /chapter07/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter07/README.md -------------------------------------------------------------------------------- /chapter08/01-Hello World/main.tf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter08/01-Hello World/main.tf -------------------------------------------------------------------------------- /chapter08/02-Deploying-DLT-Pipelines/main.tf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter08/02-Deploying-DLT-Pipelines/main.tf -------------------------------------------------------------------------------- /chapter08/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter08/README.md -------------------------------------------------------------------------------- /chapter09/01-Hello World/databricks.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter09/01-Hello World/databricks.yml -------------------------------------------------------------------------------- /chapter09/01-Hello World/src/hello_dab_world.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter09/01-Hello World/src/hello_dab_world.py -------------------------------------------------------------------------------- /chapter09/02-my-first-dab/my_first_dab/databricks.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter09/02-my-first-dab/my_first_dab/databricks.yml -------------------------------------------------------------------------------- /chapter09/02-my-first-dab/my_first_dab/resources/my_first_dab_job.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter09/02-my-first-dab/my_first_dab/resources/my_first_dab_job.yml -------------------------------------------------------------------------------- /chapter09/02-my-first-dab/my_first_dab/resources/my_first_dab_pipeline.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter09/02-my-first-dab/my_first_dab/resources/my_first_dab_pipeline.yml -------------------------------------------------------------------------------- /chapter09/02-my-first-dab/my_first_dab/src/dlt_pipeline.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter09/02-my-first-dab/my_first_dab/src/dlt_pipeline.ipynb -------------------------------------------------------------------------------- /chapter09/03-GitHub Actions/dab_deployment_workflow.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter09/03-GitHub Actions/dab_deployment_workflow.yml -------------------------------------------------------------------------------- /chapter09/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter09/README.md -------------------------------------------------------------------------------- /chapter10/01-Pipeline Health Monitoring.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter10/01-Pipeline Health Monitoring.py -------------------------------------------------------------------------------- /chapter10/02-Querying the Event Log.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter10/02-Querying the Event Log.py -------------------------------------------------------------------------------- /chapter10/03-Creating a Lakehouse Monitor.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter10/03-Creating a Lakehouse Monitor.py -------------------------------------------------------------------------------- /chapter10/04a-IoT Device Data Generator.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter10/04a-IoT Device Data Generator.py -------------------------------------------------------------------------------- /chapter10/04b-IoT Device Data Pipeline.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter10/04b-IoT Device Data Pipeline.py -------------------------------------------------------------------------------- /chapter10/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/HEAD/chapter10/README.md --------------------------------------------------------------------------------