├── 1 O4ey_K0ZbsESf8na7OirJg.jpg ├── README.md ├── Screenshot 2023-10-11 225825.png ├── Screenshot 2023-10-12 002249.png ├── Screenshot 2023-10-12 003056.png ├── Screenshot 2023-10-12 004350.png ├── Screenshot 2023-10-12 005837.png ├── Screenshot 2023-10-12 005959.png ├── Screenshot 2023-10-12 011305.png ├── Screenshot 2023-10-12 012305.png ├── Screenshot 2023-10-12 013527.png ├── Screenshot_1.png ├── databricks notebooks ├── databricks_notebooks ├── bronze to silver.ipynb ├── init ├── silver to gold.ipynb └── storagemount.ipynb ├── gif.gif ├── images └── d ├── sdfasd.gif └── sdfasdsxd.gif /1 O4ey_K0ZbsESf8na7OirJg.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/1 O4ey_K0ZbsESf8na7OirJg.jpg -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # End-to-End Azure Data Engineering 3 | 4 | This project demonstrates an end-to-end Azure data engineering solution, starting from a local SQL database and culminating in Power BI reporting, all automated. 5 | 6 | Thanks to [Mr. K Talks Tech](https://www.youtube.com/@mr.ktalkstech) for the project inspiration. 7 | 8 | ## Business Objective 9 | 10 | This project serves as a learning opportunity for common data engineering practices, focusing on ETL pipeline techniques. The skills sharpened here are valuable for small to medium-sized businesses aiming to migrate their local data to the cloud. 11 | 12 | ![Insert Image](https://github.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/blob/main/Screenshot%202023-10-11%20225825.png) 13 | 14 | ## Current Environment 15 | 16 | 17 | 18 | 19 | 20 | - Utilized the AdventureWorks dataset from Microsoft. 21 | - Set up an on-premises Microsoft SQL server on a personal computer. 22 | - Imported the dataset using Microsoft SQL Server Management Studio. 23 | - Created a new user profile, "nik." 24 | - Saved "nik" profile's password credentials as a Secret in Azure Key Vault. 25 | 26 | ![image](https://github.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/blob/main/Screenshot%202023-10-12%20011305.png) 27 | 28 | 29 | ## 1: Data Ingestion 30 | 31 | Data ingestion from the on-premises SQL server to Azure SQL is accomplished via Azure Data Factory. The process involves: 32 | 33 | 1. Installation of Self-Hosted Integration Runtime. 34 | 2. Establishing a connection between Azure Data Factory and the local SQL Server. 35 | 3. Setting up a copy pipeline to transfer all tables from the local SQL server to the Azure Data Lake's "bronze" folder. 36 | 37 | 38 | ![Azure DataFactory](https://github.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/blob/main/Screenshot_1.png) 39 | 40 | 41 | 42 | ## 2: Data Transformation 43 | 44 | After ingesting data into the "bronze" folder, it is transformed following the medallion data lake architecture (bronze, silver, gold). Data transitions through bronze, silver, and ultimately gold, suitable for business reporting tools like Power BI. 45 | 46 | Azure Databricks, using PySpark, is used for these transformations. Data initially stored in parquet format in the "bronze" folder is converted to the delta format as it progresses to "silver" and "gold." This transformation is carried out through Databricks notebooks: 47 | 48 | 1. Mount the storage. 49 | 2. Transform data from "bronze" to "silver" layer. 50 | 3. Further transform data from "silver" to "gold" layer. 51 | 52 | ![Databricks Notebooks](https://github.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/blob/main/sdfasd.gif) 53 | 54 | Azure Data Factory is updated to execute the "bronze" to "silver" and "silver" to "gold" notebooks automatically with each pipeline run. 55 | 56 | ![Completed Pipeline](https://github.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/blob/main/Screenshot%202023-10-12%20002249.png) 57 | 58 | 59 | ## 3: Data Loading 60 | 61 | Data from the "gold" folder is loaded into the Business Intelligence reporting application, Power BI. Azure Synapse is used for this purpose. The steps involve: 62 | 63 | 1. Creating a link from Azure Storage (Gold Folder) to Azure Synapse. 64 | 2. Writing stored procedures to extract table information as a SQL view. 65 | 3. Storing views within a server-less SQL Database in Synapse. 66 | 67 | ![image](https://github.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/blob/main/Screenshot%202023-10-12%20012305.png) 68 | 69 | 70 | ## 4: Data Reporting 71 | 72 | Power BI connects directly to the cloud pipeline using DirectQuery to dynamically update the database. A Power BI report is developed to visualize AdventureWorks dataset data, including sales, product information, and customer gender. 73 | 74 | ![power bi gif](https://github.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/blob/main/gif.gif) 75 | 76 | 77 | 78 | ## 5: Final Pipeline Test 79 | 80 | To verify the end-to-end pipeline, two new customers are added to the local SQL database server. If successful, the pipeline will update, and the Power BI report will dynamically show the new data. The total number of customers should increase from 847 to 849. 81 | 82 | ![completed](https://github.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/blob/main/Screenshot%202023-10-12%20013527.png) 83 | Great success! 84 | 85 | 86 | ## Conclusion and Limitations 87 | 88 | This project demonstrates the ability to create an end-to-end ETL cloud solution using Azure. Some considerations: 89 | 90 | - The dataset used was small (7mb total, 800 rows). This was done to keep compute + storage costs low for myself. 91 | - Multiple applications were employed for a relatively simple task. 92 | - Given the dataset's simplicity, the project could have been managed entirely through Azure Data Factory, with data cleaning done downstream in Power BI. 93 | - The inclusion of Azure Synapse and Databricks was for the sake of self-learning and emulating real-world business pipelines. 94 | -------------------------------------------------------------------------------- /Screenshot 2023-10-11 225825.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/Screenshot 2023-10-11 225825.png -------------------------------------------------------------------------------- /Screenshot 2023-10-12 002249.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/Screenshot 2023-10-12 002249.png -------------------------------------------------------------------------------- /Screenshot 2023-10-12 003056.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/Screenshot 2023-10-12 003056.png -------------------------------------------------------------------------------- /Screenshot 2023-10-12 004350.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/Screenshot 2023-10-12 004350.png -------------------------------------------------------------------------------- /Screenshot 2023-10-12 005837.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/Screenshot 2023-10-12 005837.png -------------------------------------------------------------------------------- /Screenshot 2023-10-12 005959.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/Screenshot 2023-10-12 005959.png -------------------------------------------------------------------------------- /Screenshot 2023-10-12 011305.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/Screenshot 2023-10-12 011305.png -------------------------------------------------------------------------------- /Screenshot 2023-10-12 012305.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/Screenshot 2023-10-12 012305.png -------------------------------------------------------------------------------- /Screenshot 2023-10-12 013527.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/Screenshot 2023-10-12 013527.png -------------------------------------------------------------------------------- /Screenshot_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/Screenshot_1.png -------------------------------------------------------------------------------- /databricks notebooks: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /databricks_notebooks/bronze to silver.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "application/vnd.databricks.v1+cell": { 7 | "cellMetadata": { 8 | "byteLimit": 2048000, 9 | "rowLimit": 10000 10 | }, 11 | "inputWidgets": {}, 12 | "nuid": "0018e4ec-ea86-4dd5-b85e-307bf7dbbeb3", 13 | "showTitle": false, 14 | "title": "" 15 | } 16 | }, 17 | "source": [ 18 | "\n", 19 | "## Transforming All Tables" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": 0, 25 | "metadata": { 26 | "application/vnd.databricks.v1+cell": { 27 | "cellMetadata": { 28 | "byteLimit": 2048000, 29 | "rowLimit": 10000 30 | }, 31 | "inputWidgets": {}, 32 | "nuid": "b98623f4-e542-4af6-9798-99014eede3eb", 33 | "showTitle": false, 34 | "title": "" 35 | } 36 | }, 37 | "outputs": [], 38 | "source": [ 39 | "table_name = []\n", 40 | "\n", 41 | "for i in dbutils.fs.ls('mnt/bronze/SalesLT/'):\n", 42 | " table_name.append(i.name.split('/')[0])" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 0, 48 | "metadata": { 49 | "application/vnd.databricks.v1+cell": { 50 | "cellMetadata": { 51 | "byteLimit": 2048000, 52 | "rowLimit": 10000 53 | }, 54 | "inputWidgets": {}, 55 | "nuid": "78df1697-a6b2-4a95-9624-4ee39ae87e27", 56 | "showTitle": false, 57 | "title": "" 58 | } 59 | }, 60 | "outputs": [ 61 | { 62 | "output_type": "execute_result", 63 | "data": { 64 | "text/plain": [ 65 | "['Address',\n", 66 | " 'Customer',\n", 67 | " 'CustomerAddress',\n", 68 | " 'Product',\n", 69 | " 'ProductCategory',\n", 70 | " 'ProductDescription',\n", 71 | " 'ProductModel',\n", 72 | " 'ProductModelProductDescription',\n", 73 | " 'SalesOrderDetail',\n", 74 | " 'SalesOrderHeader']" 75 | ] 76 | }, 77 | "execution_count": 20, 78 | "metadata": {}, 79 | "output_type": "execute_result" 80 | } 81 | ], 82 | "source": [ 83 | "table_name" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": 0, 89 | "metadata": { 90 | "application/vnd.databricks.v1+cell": { 91 | "cellMetadata": { 92 | "byteLimit": 2048000, 93 | "rowLimit": 10000 94 | }, 95 | "inputWidgets": {}, 96 | "nuid": "801e2647-4acb-415a-b959-47f86d1815e2", 97 | "showTitle": false, 98 | "title": "" 99 | } 100 | }, 101 | "outputs": [], 102 | "source": [ 103 | "from pyspark. sql. functions import from_utc_timestamp, date_format\n", 104 | "from pyspark.sql.types import TimestampType\n", 105 | "\n", 106 | "for i in table_name: \n", 107 | " path = '/mnt/bronze/SalesLT/' + i + '/' + i + '.parquet'\n", 108 | " df = spark.read.format('parquet').load(path)\n", 109 | " column = df.columns\n", 110 | " \n", 111 | " for col in column:\n", 112 | " if \"Date\" in col or \"date\" in col:\n", 113 | " df = df.withColumn(col, date_format(from_utc_timestamp(df[col].cast(TimestampType()), \"UTC\"), \"yyyy-MM-dd\"))\n", 114 | " \n", 115 | " output_path = '/mnt/silver/SalesLT/' +i + '/'\n", 116 | " df.write.format('delta').mode(\"overwrite\").save(output_path)" 117 | ] 118 | }, 119 | { 120 | "cell_type": "code", 121 | "execution_count": 0, 122 | "metadata": { 123 | "application/vnd.databricks.v1+cell": { 124 | "cellMetadata": { 125 | "byteLimit": 2048000, 126 | "rowLimit": 10000 127 | }, 128 | "inputWidgets": {}, 129 | "nuid": "dc148455-12c1-423f-9906-e2971e25dc82", 130 | "showTitle": false, 131 | "title": "" 132 | } 133 | }, 134 | "outputs": [ 135 | { 136 | "output_type": "display_data", 137 | "data": { 138 | "text/html": [ 139 | "
SalesOrderIDRevisionNumberOrderDateDueDateShipDateStatusOnlineOrderFlagSalesOrderNumberPurchaseOrderNumberAccountNumberCustomerIDShipToAddressIDBillToAddressIDShipMethodCreditCardApprovalCodeSubTotalTaxAmtFreightTotalDueCommentrowguidModifiedDate
7177422008-06-012008-06-132008-06-085falseSO71774PO34818628710-4020-0006092984710921092CARGO TRANSPORT 5null880.348470.427922.0087972.7850null89e42cdc-8506-48a2-b89b-eb3e64e3554e2008-06-08
7177622008-06-012008-06-132008-06-085falseSO71776PO1995219205110-4020-00010630072640640CARGO TRANSPORT 5null78.81006.30481.970387.0851null8a3448c5-e677-4158-a29b-dd33069be0b02008-06-08
7178022008-06-012008-06-132008-06-085falseSO71780PO1960417323910-4020-00034030113653653CARGO TRANSPORT 5null38418.68953073.4952960.467242452.6519nulla47665d2-7ac9-4cf3-8a8b-2a38835542842008-06-08
7178222008-06-012008-06-132008-06-085falseSO71782PO1937211474910-4020-0005822948510861086CARGO TRANSPORT 5null39785.33043182.8264994.633343962.7901nullf1be45a5-5c57-4a50-93c6-5f8be44cb7cb2008-06-08
7178322008-06-012008-06-132008-06-085falseSO71783PO1934311360910-4020-00002429957992992CARGO TRANSPORT 5null83858.42616708.67412096.460792663.5609null7db2329e-6446-42a8-8915-9c8370b68ed82008-06-08
7178422008-06-012008-06-132008-06-085falseSO71784PO1928513591910-4020-00044829736659659CARGO TRANSPORT 5null108561.83178684.94652714.0458119960.8240nullca31f324-2c32-4f8d-95eb-596e7f3430272008-06-08
7179622008-06-012008-06-132008-06-085falseSO71796PO1705215966410-4020-0004202966010581058CARGO TRANSPORT 5null57634.63424610.77071440.865963686.2708null917ef5ba-f32d-4563-8588-66db0bcdc8462008-06-08
7179722008-06-012008-06-132008-06-085falseSO71797PO1650113488910-4020-00014229796642642CARGO TRANSPORT 5null78029.68986242.37521950.742286222.8072nullbb3fee84-c8bf-4dd2-bcca-675ab6a11c382008-06-08
7181522008-06-012008-06-132008-06-085falseSO71815PO1302115578510-4020-0002763008910341034CARGO TRANSPORT 5null1141.578291.326328.53951261.4440null2aa5f39b-1096-4a4b-b17b-f10504a397ce2008-06-08
7181622008-06-012008-06-132008-06-085falseSO71816PO1299218044510-4020-0002953002710381038CARGO TRANSPORT 5null3398.1659271.853384.95413754.9733nulle3c189e7-98de-4c40-b6c2-0d1d13f9bb332008-06-08
7183122008-06-012008-06-132008-06-085falseSO71831PO1029511108410-4020-00032230019652652CARGO TRANSPORT 5null2016.3408161.307350.40852228.0566null625d76fc-c26f-4149-bf24-939fb2bccd772008-06-08
7183222008-06-012008-06-132008-06-085falseSO71832PO1035314075610-4020-00008829922639639CARGO TRANSPORT 5null35775.21132862.0169894.380339531.6085nulladdb8620-432a-456e-8470-1bedd4bc34572008-06-08
7184522008-06-012008-06-132008-06-085falseSO71845PO269711936210-4020-0001872993810201020CARGO TRANSPORT 5null41622.05113329.76411040.551345992.3665nulle68f7ee9-c581-45cd-9c4f-386aeda74d842008-06-08
7184622008-06-012008-06-132008-06-085falseSO71846PO237813160410-4020-00063530102669669CARGO TRANSPORT 5null2453.7645196.301261.34412711.4098nulla86d90ad-d1c0-440d-9a57-5b763bf182342008-06-08
7185622008-06-012008-06-132008-06-085falseSO71856PO1653017764710-4020-0006013003310901090CARGO TRANSPORT 5null602.194648.175615.0549665.4251null05fee073-0640-4a3c-914d-fe4ae6da3d432008-06-08
7185822008-06-012008-06-132008-06-085falseSO71858PO1615311227810-4020-0001862965310191019CARGO TRANSPORT 5null13823.70831105.8967345.592715275.1977null5ef091e1-a0af-437d-85ed-0b557c7923f72008-06-08
7186322008-06-012008-06-132008-06-085falseSO71863PO1612416656110-4020-0006492997510981098CARGO TRANSPORT 5null3324.2759265.942183.10693673.3249null3ed03b56-a4bf-4872-9471-bc6c7893eab72008-06-08
7186722008-06-012008-06-132008-06-085falseSO71867PO1305011152910-4020-00016029644643643CARGO TRANSPORT 5null1059.310084.744826.48281170.5376null29743c1b-d3af-4cfe-bd2e-6de436e3398f2008-06-08
7188522008-06-012008-06-132008-06-085falseSO71885PO611913077910-4020-00026829612649649CARGO TRANSPORT 5null550.386044.030913.7597608.1766nullcaad090d-56a6-444e-af24-7bee7605f1202008-06-08
7189522008-06-012008-06-132008-06-085falseSO71895PO377017627310-4020-0001512958410141014CARGO TRANSPORT 5null246.739219.73916.1685272.6468null07a17c0a-7f9a-4413-a035-519e5a4573be2008-06-08
7189722008-06-012008-06-132008-06-085falseSO71897PO478515247910-4020-0002232987710261026CARGO TRANSPORT 5null12685.88991014.8712317.147214017.9083nullf88b3458-e0cf-4248-9b72-1805351463b22008-06-08
7189822008-06-012008-06-132008-06-085falseSO71898PO571319050110-4020-00005229932637637CARGO TRANSPORT 5null63980.98845118.47911599.524770698.9922null96f84d24-3355-43d2-b5a3-55e97c17e58c2008-06-08
7189922008-06-012008-06-132008-06-085falseSO71899PO458214261110-4020-00002529568993993CARGO TRANSPORT 5null2415.6727193.253860.39182669.3183null31d41e8f-6f43-4cae-bee3-3cccb262f2312008-06-08
7190222008-06-012008-06-132008-06-085falseSO71902PO553912516610-4020-00006129929999999CARGO TRANSPORT 5null74058.80785924.70461851.470281834.9826null137850d6-efdf-4de1-920f-5757a86cdaaf2008-06-08
7191522008-06-012008-06-132008-06-085falseSO71915PO234914327510-4020-00000629638989989CARGO TRANSPORT 5null2137.2310170.978553.43082361.6403null0bcb3d9c-9008-4b99-bb45-928adae7be6b2008-06-08
7191722008-06-012008-06-132008-06-085falseSO71917PO2911171810-4020-00030430025651651CARGO TRANSPORT 5null40.90453.27241.022645.1995null6e903ea3-1b9e-4232-94c3-81c15669f8302008-06-08
7192022008-06-012008-06-132008-06-085falseSO71920PO400218985310-4020-0006742998211021102CARGO TRANSPORT 5null2980.7929238.463474.51983293.7761null6228c9cb-1cab-4e32-98ca-d0dae5fe563e2008-06-08
7192322008-06-012008-06-132008-06-085falseSO71923PO17417309610-4020-0002772978110351035CARGO TRANSPORT 5null106.54088.52332.6635117.7276nullf9899f3f-b4b6-4756-b013-96c16be204272008-06-08
7193522008-06-012008-06-132008-06-085falseSO71935PO794614587610-4020-0004382953110611061CARGO TRANSPORT 5null6634.2961530.7437165.85747330.8972null7033c6ec-b12c-45bc-bd96-56efde4c7dd02008-06-08
7193622008-06-012008-06-132008-06-085falseSO71936PO867117038510-4020-00050230050662662CARGO TRANSPORT 5null98278.69107862.29532456.9673108597.9536null119db56a-a97e-414d-b41c-64886fc50ab72008-06-08
7193822008-06-012008-06-132008-06-085falseSO71938PO846818331510-4020-00001629546635635CARGO TRANSPORT 5null88812.86257105.02902220.321698138.2131nulla36ee74a-cf0d-4024-a1ce-4eaffd1f85f02008-06-08
7194622008-06-012008-06-132008-06-085falseSO71946PO896115862910-4020-00046629741660660CARGO TRANSPORT 5null38.95363.11630.973843.0437null10e3129d-657f-46f9-86f5-cedd79b1901c2008-06-08
" 155 | ] 156 | }, 157 | "metadata": { 158 | "application/vnd.databricks.v1+output": { 159 | "addedWidgets": {}, 160 | "aggData": [], 161 | "aggError": "", 162 | "aggOverflow": false, 163 | "aggSchema": [], 164 | "aggSeriesLimitReached": false, 165 | "aggType": "", 166 | "arguments": {}, 167 | "columnCustomDisplayInfos": {}, 168 | "data": [ 169 | [ 170 | 71774, 171 | 2, 172 | "2008-06-01", 173 | "2008-06-13", 174 | "2008-06-08", 175 | 5, 176 | false, 177 | "SO71774", 178 | "PO348186287", 179 | "10-4020-000609", 180 | 29847, 181 | 1092, 182 | 1092, 183 | "CARGO TRANSPORT 5", 184 | null, 185 | "880.3484", 186 | "70.4279", 187 | "22.0087", 188 | "972.7850", 189 | null, 190 | "89e42cdc-8506-48a2-b89b-eb3e64e3554e", 191 | "2008-06-08" 192 | ], 193 | [ 194 | 71776, 195 | 2, 196 | "2008-06-01", 197 | "2008-06-13", 198 | "2008-06-08", 199 | 5, 200 | false, 201 | "SO71776", 202 | "PO19952192051", 203 | "10-4020-000106", 204 | 30072, 205 | 640, 206 | 640, 207 | "CARGO TRANSPORT 5", 208 | null, 209 | "78.8100", 210 | "6.3048", 211 | "1.9703", 212 | "87.0851", 213 | null, 214 | "8a3448c5-e677-4158-a29b-dd33069be0b0", 215 | "2008-06-08" 216 | ], 217 | [ 218 | 71780, 219 | 2, 220 | "2008-06-01", 221 | "2008-06-13", 222 | "2008-06-08", 223 | 5, 224 | false, 225 | "SO71780", 226 | "PO19604173239", 227 | "10-4020-000340", 228 | 30113, 229 | 653, 230 | 653, 231 | "CARGO TRANSPORT 5", 232 | null, 233 | "38418.6895", 234 | "3073.4952", 235 | "960.4672", 236 | "42452.6519", 237 | null, 238 | "a47665d2-7ac9-4cf3-8a8b-2a3883554284", 239 | "2008-06-08" 240 | ], 241 | [ 242 | 71782, 243 | 2, 244 | "2008-06-01", 245 | "2008-06-13", 246 | "2008-06-08", 247 | 5, 248 | false, 249 | "SO71782", 250 | "PO19372114749", 251 | "10-4020-000582", 252 | 29485, 253 | 1086, 254 | 1086, 255 | "CARGO TRANSPORT 5", 256 | null, 257 | "39785.3304", 258 | "3182.8264", 259 | "994.6333", 260 | "43962.7901", 261 | null, 262 | "f1be45a5-5c57-4a50-93c6-5f8be44cb7cb", 263 | "2008-06-08" 264 | ], 265 | [ 266 | 71783, 267 | 2, 268 | "2008-06-01", 269 | "2008-06-13", 270 | "2008-06-08", 271 | 5, 272 | false, 273 | "SO71783", 274 | "PO19343113609", 275 | "10-4020-000024", 276 | 29957, 277 | 992, 278 | 992, 279 | "CARGO TRANSPORT 5", 280 | null, 281 | "83858.4261", 282 | "6708.6741", 283 | "2096.4607", 284 | "92663.5609", 285 | null, 286 | "7db2329e-6446-42a8-8915-9c8370b68ed8", 287 | "2008-06-08" 288 | ], 289 | [ 290 | 71784, 291 | 2, 292 | "2008-06-01", 293 | "2008-06-13", 294 | "2008-06-08", 295 | 5, 296 | false, 297 | "SO71784", 298 | "PO19285135919", 299 | "10-4020-000448", 300 | 29736, 301 | 659, 302 | 659, 303 | "CARGO TRANSPORT 5", 304 | null, 305 | "108561.8317", 306 | "8684.9465", 307 | "2714.0458", 308 | "119960.8240", 309 | null, 310 | "ca31f324-2c32-4f8d-95eb-596e7f343027", 311 | "2008-06-08" 312 | ], 313 | [ 314 | 71796, 315 | 2, 316 | "2008-06-01", 317 | "2008-06-13", 318 | "2008-06-08", 319 | 5, 320 | false, 321 | "SO71796", 322 | "PO17052159664", 323 | "10-4020-000420", 324 | 29660, 325 | 1058, 326 | 1058, 327 | "CARGO TRANSPORT 5", 328 | null, 329 | "57634.6342", 330 | "4610.7707", 331 | "1440.8659", 332 | "63686.2708", 333 | null, 334 | "917ef5ba-f32d-4563-8588-66db0bcdc846", 335 | "2008-06-08" 336 | ], 337 | [ 338 | 71797, 339 | 2, 340 | "2008-06-01", 341 | "2008-06-13", 342 | "2008-06-08", 343 | 5, 344 | false, 345 | "SO71797", 346 | "PO16501134889", 347 | "10-4020-000142", 348 | 29796, 349 | 642, 350 | 642, 351 | "CARGO TRANSPORT 5", 352 | null, 353 | "78029.6898", 354 | "6242.3752", 355 | "1950.7422", 356 | "86222.8072", 357 | null, 358 | "bb3fee84-c8bf-4dd2-bcca-675ab6a11c38", 359 | "2008-06-08" 360 | ], 361 | [ 362 | 71815, 363 | 2, 364 | "2008-06-01", 365 | "2008-06-13", 366 | "2008-06-08", 367 | 5, 368 | false, 369 | "SO71815", 370 | "PO13021155785", 371 | "10-4020-000276", 372 | 30089, 373 | 1034, 374 | 1034, 375 | "CARGO TRANSPORT 5", 376 | null, 377 | "1141.5782", 378 | "91.3263", 379 | "28.5395", 380 | "1261.4440", 381 | null, 382 | "2aa5f39b-1096-4a4b-b17b-f10504a397ce", 383 | "2008-06-08" 384 | ], 385 | [ 386 | 71816, 387 | 2, 388 | "2008-06-01", 389 | "2008-06-13", 390 | "2008-06-08", 391 | 5, 392 | false, 393 | "SO71816", 394 | "PO12992180445", 395 | "10-4020-000295", 396 | 30027, 397 | 1038, 398 | 1038, 399 | "CARGO TRANSPORT 5", 400 | null, 401 | "3398.1659", 402 | "271.8533", 403 | "84.9541", 404 | "3754.9733", 405 | null, 406 | "e3c189e7-98de-4c40-b6c2-0d1d13f9bb33", 407 | "2008-06-08" 408 | ], 409 | [ 410 | 71831, 411 | 2, 412 | "2008-06-01", 413 | "2008-06-13", 414 | "2008-06-08", 415 | 5, 416 | false, 417 | "SO71831", 418 | "PO10295111084", 419 | "10-4020-000322", 420 | 30019, 421 | 652, 422 | 652, 423 | "CARGO TRANSPORT 5", 424 | null, 425 | "2016.3408", 426 | "161.3073", 427 | "50.4085", 428 | "2228.0566", 429 | null, 430 | "625d76fc-c26f-4149-bf24-939fb2bccd77", 431 | "2008-06-08" 432 | ], 433 | [ 434 | 71832, 435 | 2, 436 | "2008-06-01", 437 | "2008-06-13", 438 | "2008-06-08", 439 | 5, 440 | false, 441 | "SO71832", 442 | "PO10353140756", 443 | "10-4020-000088", 444 | 29922, 445 | 639, 446 | 639, 447 | "CARGO TRANSPORT 5", 448 | null, 449 | "35775.2113", 450 | "2862.0169", 451 | "894.3803", 452 | "39531.6085", 453 | null, 454 | "addb8620-432a-456e-8470-1bedd4bc3457", 455 | "2008-06-08" 456 | ], 457 | [ 458 | 71845, 459 | 2, 460 | "2008-06-01", 461 | "2008-06-13", 462 | "2008-06-08", 463 | 5, 464 | false, 465 | "SO71845", 466 | "PO2697119362", 467 | "10-4020-000187", 468 | 29938, 469 | 1020, 470 | 1020, 471 | "CARGO TRANSPORT 5", 472 | null, 473 | "41622.0511", 474 | "3329.7641", 475 | "1040.5513", 476 | "45992.3665", 477 | null, 478 | "e68f7ee9-c581-45cd-9c4f-386aeda74d84", 479 | "2008-06-08" 480 | ], 481 | [ 482 | 71846, 483 | 2, 484 | "2008-06-01", 485 | "2008-06-13", 486 | "2008-06-08", 487 | 5, 488 | false, 489 | "SO71846", 490 | "PO2378131604", 491 | "10-4020-000635", 492 | 30102, 493 | 669, 494 | 669, 495 | "CARGO TRANSPORT 5", 496 | null, 497 | "2453.7645", 498 | "196.3012", 499 | "61.3441", 500 | "2711.4098", 501 | null, 502 | "a86d90ad-d1c0-440d-9a57-5b763bf18234", 503 | "2008-06-08" 504 | ], 505 | [ 506 | 71856, 507 | 2, 508 | "2008-06-01", 509 | "2008-06-13", 510 | "2008-06-08", 511 | 5, 512 | false, 513 | "SO71856", 514 | "PO16530177647", 515 | "10-4020-000601", 516 | 30033, 517 | 1090, 518 | 1090, 519 | "CARGO TRANSPORT 5", 520 | null, 521 | "602.1946", 522 | "48.1756", 523 | "15.0549", 524 | "665.4251", 525 | null, 526 | "05fee073-0640-4a3c-914d-fe4ae6da3d43", 527 | "2008-06-08" 528 | ], 529 | [ 530 | 71858, 531 | 2, 532 | "2008-06-01", 533 | "2008-06-13", 534 | "2008-06-08", 535 | 5, 536 | false, 537 | "SO71858", 538 | "PO16153112278", 539 | "10-4020-000186", 540 | 29653, 541 | 1019, 542 | 1019, 543 | "CARGO TRANSPORT 5", 544 | null, 545 | "13823.7083", 546 | "1105.8967", 547 | "345.5927", 548 | "15275.1977", 549 | null, 550 | "5ef091e1-a0af-437d-85ed-0b557c7923f7", 551 | "2008-06-08" 552 | ], 553 | [ 554 | 71863, 555 | 2, 556 | "2008-06-01", 557 | "2008-06-13", 558 | "2008-06-08", 559 | 5, 560 | false, 561 | "SO71863", 562 | "PO16124166561", 563 | "10-4020-000649", 564 | 29975, 565 | 1098, 566 | 1098, 567 | "CARGO TRANSPORT 5", 568 | null, 569 | "3324.2759", 570 | "265.9421", 571 | "83.1069", 572 | "3673.3249", 573 | null, 574 | "3ed03b56-a4bf-4872-9471-bc6c7893eab7", 575 | "2008-06-08" 576 | ], 577 | [ 578 | 71867, 579 | 2, 580 | "2008-06-01", 581 | "2008-06-13", 582 | "2008-06-08", 583 | 5, 584 | false, 585 | "SO71867", 586 | "PO13050111529", 587 | "10-4020-000160", 588 | 29644, 589 | 643, 590 | 643, 591 | "CARGO TRANSPORT 5", 592 | null, 593 | "1059.3100", 594 | "84.7448", 595 | "26.4828", 596 | "1170.5376", 597 | null, 598 | "29743c1b-d3af-4cfe-bd2e-6de436e3398f", 599 | "2008-06-08" 600 | ], 601 | [ 602 | 71885, 603 | 2, 604 | "2008-06-01", 605 | "2008-06-13", 606 | "2008-06-08", 607 | 5, 608 | false, 609 | "SO71885", 610 | "PO6119130779", 611 | "10-4020-000268", 612 | 29612, 613 | 649, 614 | 649, 615 | "CARGO TRANSPORT 5", 616 | null, 617 | "550.3860", 618 | "44.0309", 619 | "13.7597", 620 | "608.1766", 621 | null, 622 | "caad090d-56a6-444e-af24-7bee7605f120", 623 | "2008-06-08" 624 | ], 625 | [ 626 | 71895, 627 | 2, 628 | "2008-06-01", 629 | "2008-06-13", 630 | "2008-06-08", 631 | 5, 632 | false, 633 | "SO71895", 634 | "PO3770176273", 635 | "10-4020-000151", 636 | 29584, 637 | 1014, 638 | 1014, 639 | "CARGO TRANSPORT 5", 640 | null, 641 | "246.7392", 642 | "19.7391", 643 | "6.1685", 644 | "272.6468", 645 | null, 646 | "07a17c0a-7f9a-4413-a035-519e5a4573be", 647 | "2008-06-08" 648 | ], 649 | [ 650 | 71897, 651 | 2, 652 | "2008-06-01", 653 | "2008-06-13", 654 | "2008-06-08", 655 | 5, 656 | false, 657 | "SO71897", 658 | "PO4785152479", 659 | "10-4020-000223", 660 | 29877, 661 | 1026, 662 | 1026, 663 | "CARGO TRANSPORT 5", 664 | null, 665 | "12685.8899", 666 | "1014.8712", 667 | "317.1472", 668 | "14017.9083", 669 | null, 670 | "f88b3458-e0cf-4248-9b72-1805351463b2", 671 | "2008-06-08" 672 | ], 673 | [ 674 | 71898, 675 | 2, 676 | "2008-06-01", 677 | "2008-06-13", 678 | "2008-06-08", 679 | 5, 680 | false, 681 | "SO71898", 682 | "PO5713190501", 683 | "10-4020-000052", 684 | 29932, 685 | 637, 686 | 637, 687 | "CARGO TRANSPORT 5", 688 | null, 689 | "63980.9884", 690 | "5118.4791", 691 | "1599.5247", 692 | "70698.9922", 693 | null, 694 | "96f84d24-3355-43d2-b5a3-55e97c17e58c", 695 | "2008-06-08" 696 | ], 697 | [ 698 | 71899, 699 | 2, 700 | "2008-06-01", 701 | "2008-06-13", 702 | "2008-06-08", 703 | 5, 704 | false, 705 | "SO71899", 706 | "PO4582142611", 707 | "10-4020-000025", 708 | 29568, 709 | 993, 710 | 993, 711 | "CARGO TRANSPORT 5", 712 | null, 713 | "2415.6727", 714 | "193.2538", 715 | "60.3918", 716 | "2669.3183", 717 | null, 718 | "31d41e8f-6f43-4cae-bee3-3cccb262f231", 719 | "2008-06-08" 720 | ], 721 | [ 722 | 71902, 723 | 2, 724 | "2008-06-01", 725 | "2008-06-13", 726 | "2008-06-08", 727 | 5, 728 | false, 729 | "SO71902", 730 | "PO5539125166", 731 | "10-4020-000061", 732 | 29929, 733 | 999, 734 | 999, 735 | "CARGO TRANSPORT 5", 736 | null, 737 | "74058.8078", 738 | "5924.7046", 739 | "1851.4702", 740 | "81834.9826", 741 | null, 742 | "137850d6-efdf-4de1-920f-5757a86cdaaf", 743 | "2008-06-08" 744 | ], 745 | [ 746 | 71915, 747 | 2, 748 | "2008-06-01", 749 | "2008-06-13", 750 | "2008-06-08", 751 | 5, 752 | false, 753 | "SO71915", 754 | "PO2349143275", 755 | "10-4020-000006", 756 | 29638, 757 | 989, 758 | 989, 759 | "CARGO TRANSPORT 5", 760 | null, 761 | "2137.2310", 762 | "170.9785", 763 | "53.4308", 764 | "2361.6403", 765 | null, 766 | "0bcb3d9c-9008-4b99-bb45-928adae7be6b", 767 | "2008-06-08" 768 | ], 769 | [ 770 | 71917, 771 | 2, 772 | "2008-06-01", 773 | "2008-06-13", 774 | "2008-06-08", 775 | 5, 776 | false, 777 | "SO71917", 778 | "PO29111718", 779 | "10-4020-000304", 780 | 30025, 781 | 651, 782 | 651, 783 | "CARGO TRANSPORT 5", 784 | null, 785 | "40.9045", 786 | "3.2724", 787 | "1.0226", 788 | "45.1995", 789 | null, 790 | "6e903ea3-1b9e-4232-94c3-81c15669f830", 791 | "2008-06-08" 792 | ], 793 | [ 794 | 71920, 795 | 2, 796 | "2008-06-01", 797 | "2008-06-13", 798 | "2008-06-08", 799 | 5, 800 | false, 801 | "SO71920", 802 | "PO4002189853", 803 | "10-4020-000674", 804 | 29982, 805 | 1102, 806 | 1102, 807 | "CARGO TRANSPORT 5", 808 | null, 809 | "2980.7929", 810 | "238.4634", 811 | "74.5198", 812 | "3293.7761", 813 | null, 814 | "6228c9cb-1cab-4e32-98ca-d0dae5fe563e", 815 | "2008-06-08" 816 | ], 817 | [ 818 | 71923, 819 | 2, 820 | "2008-06-01", 821 | "2008-06-13", 822 | "2008-06-08", 823 | 5, 824 | false, 825 | "SO71923", 826 | "PO174173096", 827 | "10-4020-000277", 828 | 29781, 829 | 1035, 830 | 1035, 831 | "CARGO TRANSPORT 5", 832 | null, 833 | "106.5408", 834 | "8.5233", 835 | "2.6635", 836 | "117.7276", 837 | null, 838 | "f9899f3f-b4b6-4756-b013-96c16be20427", 839 | "2008-06-08" 840 | ], 841 | [ 842 | 71935, 843 | 2, 844 | "2008-06-01", 845 | "2008-06-13", 846 | "2008-06-08", 847 | 5, 848 | false, 849 | "SO71935", 850 | "PO7946145876", 851 | "10-4020-000438", 852 | 29531, 853 | 1061, 854 | 1061, 855 | "CARGO TRANSPORT 5", 856 | null, 857 | "6634.2961", 858 | "530.7437", 859 | "165.8574", 860 | "7330.8972", 861 | null, 862 | "7033c6ec-b12c-45bc-bd96-56efde4c7dd0", 863 | "2008-06-08" 864 | ], 865 | [ 866 | 71936, 867 | 2, 868 | "2008-06-01", 869 | "2008-06-13", 870 | "2008-06-08", 871 | 5, 872 | false, 873 | "SO71936", 874 | "PO8671170385", 875 | "10-4020-000502", 876 | 30050, 877 | 662, 878 | 662, 879 | "CARGO TRANSPORT 5", 880 | null, 881 | "98278.6910", 882 | "7862.2953", 883 | "2456.9673", 884 | "108597.9536", 885 | null, 886 | "119db56a-a97e-414d-b41c-64886fc50ab7", 887 | "2008-06-08" 888 | ], 889 | [ 890 | 71938, 891 | 2, 892 | "2008-06-01", 893 | "2008-06-13", 894 | "2008-06-08", 895 | 5, 896 | false, 897 | "SO71938", 898 | "PO8468183315", 899 | "10-4020-000016", 900 | 29546, 901 | 635, 902 | 635, 903 | "CARGO TRANSPORT 5", 904 | null, 905 | "88812.8625", 906 | "7105.0290", 907 | "2220.3216", 908 | "98138.2131", 909 | null, 910 | "a36ee74a-cf0d-4024-a1ce-4eaffd1f85f0", 911 | "2008-06-08" 912 | ], 913 | [ 914 | 71946, 915 | 2, 916 | "2008-06-01", 917 | "2008-06-13", 918 | "2008-06-08", 919 | 5, 920 | false, 921 | "SO71946", 922 | "PO8961158629", 923 | "10-4020-000466", 924 | 29741, 925 | 660, 926 | 660, 927 | "CARGO TRANSPORT 5", 928 | null, 929 | "38.9536", 930 | "3.1163", 931 | "0.9738", 932 | "43.0437", 933 | null, 934 | "10e3129d-657f-46f9-86f5-cedd79b1901c", 935 | "2008-06-08" 936 | ] 937 | ], 938 | "datasetInfos": [], 939 | "dbfsResultPath": null, 940 | "isJsonSchema": true, 941 | "metadata": {}, 942 | "overflow": false, 943 | "plotOptions": { 944 | "customPlotOptions": {}, 945 | "displayType": "table", 946 | "pivotAggregation": null, 947 | "pivotColumns": null, 948 | "xColumns": null, 949 | "yColumns": null 950 | }, 951 | "removedWidgets": [], 952 | "schema": [ 953 | { 954 | "metadata": "{}", 955 | "name": "SalesOrderID", 956 | "type": "\"integer\"" 957 | }, 958 | { 959 | "metadata": "{}", 960 | "name": "RevisionNumber", 961 | "type": "\"integer\"" 962 | }, 963 | { 964 | "metadata": "{}", 965 | "name": "OrderDate", 966 | "type": "\"string\"" 967 | }, 968 | { 969 | "metadata": "{}", 970 | "name": "DueDate", 971 | "type": "\"string\"" 972 | }, 973 | { 974 | "metadata": "{}", 975 | "name": "ShipDate", 976 | "type": "\"string\"" 977 | }, 978 | { 979 | "metadata": "{}", 980 | "name": "Status", 981 | "type": "\"integer\"" 982 | }, 983 | { 984 | "metadata": "{}", 985 | "name": "OnlineOrderFlag", 986 | "type": "\"boolean\"" 987 | }, 988 | { 989 | "metadata": "{}", 990 | "name": "SalesOrderNumber", 991 | "type": "\"string\"" 992 | }, 993 | { 994 | "metadata": "{}", 995 | "name": "PurchaseOrderNumber", 996 | "type": "\"string\"" 997 | }, 998 | { 999 | "metadata": "{}", 1000 | "name": "AccountNumber", 1001 | "type": "\"string\"" 1002 | }, 1003 | { 1004 | "metadata": "{}", 1005 | "name": "CustomerID", 1006 | "type": "\"integer\"" 1007 | }, 1008 | { 1009 | "metadata": "{}", 1010 | "name": "ShipToAddressID", 1011 | "type": "\"integer\"" 1012 | }, 1013 | { 1014 | "metadata": "{}", 1015 | "name": "BillToAddressID", 1016 | "type": "\"integer\"" 1017 | }, 1018 | { 1019 | "metadata": "{}", 1020 | "name": "ShipMethod", 1021 | "type": "\"string\"" 1022 | }, 1023 | { 1024 | "metadata": "{}", 1025 | "name": "CreditCardApprovalCode", 1026 | "type": "\"string\"" 1027 | }, 1028 | { 1029 | "metadata": "{}", 1030 | "name": "SubTotal", 1031 | "type": "\"decimal(19,4)\"" 1032 | }, 1033 | { 1034 | "metadata": "{}", 1035 | "name": "TaxAmt", 1036 | "type": "\"decimal(19,4)\"" 1037 | }, 1038 | { 1039 | "metadata": "{}", 1040 | "name": "Freight", 1041 | "type": "\"decimal(19,4)\"" 1042 | }, 1043 | { 1044 | "metadata": "{}", 1045 | "name": "TotalDue", 1046 | "type": "\"decimal(19,4)\"" 1047 | }, 1048 | { 1049 | "metadata": "{}", 1050 | "name": "Comment", 1051 | "type": "\"string\"" 1052 | }, 1053 | { 1054 | "metadata": "{}", 1055 | "name": "rowguid", 1056 | "type": "\"string\"" 1057 | }, 1058 | { 1059 | "metadata": "{}", 1060 | "name": "ModifiedDate", 1061 | "type": "\"string\"" 1062 | } 1063 | ], 1064 | "type": "table" 1065 | } 1066 | }, 1067 | "output_type": "display_data" 1068 | } 1069 | ], 1070 | "source": [ 1071 | "display(df)" 1072 | ] 1073 | } 1074 | ], 1075 | "metadata": { 1076 | "application/vnd.databricks.v1+notebook": { 1077 | "dashboards": [], 1078 | "language": "python", 1079 | "notebookMetadata": { 1080 | "pythonIndentUnit": 4 1081 | }, 1082 | "notebookName": "bronze to silver", 1083 | "widgets": {} 1084 | } 1085 | }, 1086 | "nbformat": 4, 1087 | "nbformat_minor": 0 1088 | } 1089 | -------------------------------------------------------------------------------- /databricks_notebooks/init: -------------------------------------------------------------------------------- 1 | init 2 | -------------------------------------------------------------------------------- /databricks_notebooks/silver to gold.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "application/vnd.databricks.v1+cell": { 7 | "cellMetadata": { 8 | "byteLimit": 2048000, 9 | "rowLimit": 10000 10 | }, 11 | "inputWidgets": {}, 12 | "nuid": "e84c921d-163d-402c-81f4-06baa137a8b1", 13 | "showTitle": false, 14 | "title": "" 15 | } 16 | }, 17 | "source": [ 18 | "\n", 19 | "## Transforming all the column names of tables" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": 0, 25 | "metadata": { 26 | "application/vnd.databricks.v1+cell": { 27 | "cellMetadata": { 28 | "byteLimit": 2048000, 29 | "rowLimit": 10000 30 | }, 31 | "inputWidgets": {}, 32 | "nuid": "0314b13e-098a-4ad7-89f8-2d6ab766a3ae", 33 | "showTitle": false, 34 | "title": "" 35 | } 36 | }, 37 | "outputs": [], 38 | "source": [ 39 | "table_name = []\n", 40 | "\n", 41 | "for i in dbutils.fs.ls('mnt/silver/SalesLT/'):\n", 42 | " table_name.append(i.name.split('/')[0])" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 0, 48 | "metadata": { 49 | "application/vnd.databricks.v1+cell": { 50 | "cellMetadata": { 51 | "byteLimit": 2048000, 52 | "rowLimit": 10000 53 | }, 54 | "inputWidgets": {}, 55 | "nuid": "3e116833-dcfd-4451-9d02-b7ca3c400b67", 56 | "showTitle": false, 57 | "title": "" 58 | } 59 | }, 60 | "outputs": [ 61 | { 62 | "output_type": "execute_result", 63 | "data": { 64 | "text/plain": [ 65 | "['Address',\n", 66 | " 'Customer',\n", 67 | " 'CustomerAddress',\n", 68 | " 'Product',\n", 69 | " 'ProductCategory',\n", 70 | " 'ProductDescription',\n", 71 | " 'ProductModel',\n", 72 | " 'ProductModelProductDescription',\n", 73 | " 'SalesOrderDetail',\n", 74 | " 'SalesOrderHeader']" 75 | ] 76 | }, 77 | "execution_count": 39, 78 | "metadata": {}, 79 | "output_type": "execute_result" 80 | } 81 | ], 82 | "source": [ 83 | "table_name" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": 0, 89 | "metadata": { 90 | "application/vnd.databricks.v1+cell": { 91 | "cellMetadata": { 92 | "byteLimit": 2048000, 93 | "rowLimit": 10000 94 | }, 95 | "inputWidgets": {}, 96 | "nuid": "312d0f50-5b72-4053-bdec-66062f800dd0", 97 | "showTitle": false, 98 | "title": "" 99 | } 100 | }, 101 | "outputs": [ 102 | { 103 | "output_type": "stream", 104 | "name": "stdout", 105 | "output_type": "stream", 106 | "text": [ 107 | "/mnt/silver/SalesLT/Address\n/mnt/silver/SalesLT/Customer\n/mnt/silver/SalesLT/CustomerAddress\n/mnt/silver/SalesLT/Product\n/mnt/silver/SalesLT/ProductCategory\n/mnt/silver/SalesLT/ProductDescription\n/mnt/silver/SalesLT/ProductModel\n/mnt/silver/SalesLT/ProductModelProductDescription\n/mnt/silver/SalesLT/SalesOrderDetail\n/mnt/silver/SalesLT/SalesOrderHeader\n" 108 | ] 109 | } 110 | ], 111 | "source": [ 112 | "for name in table_name:\n", 113 | " path = '/mnt/silver/SalesLT/' + name \n", 114 | " print(path)\n", 115 | " df = spark.read.format('delta').load(path)\n", 116 | "\n", 117 | " column_names = df.columns\n", 118 | "\n", 119 | " for old_col_name in column_names:\n", 120 | " new_col_name = \"\".join([\"_\" + char if char.isupper() and not old_col_name[i - 1].isupper() else char for i, char in enumerate(old_col_name)]).lstrip(\"_\")\n", 121 | "\n", 122 | " df = df.withColumnRenamed(old_col_name, new_col_name)\n", 123 | " \n", 124 | " output_path = '/mnt/gold/SalesLT/' +name +'/'\n", 125 | " df.write.format('delta').mode(\"overwrite\").save(output_path)" 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": 0, 131 | "metadata": { 132 | "application/vnd.databricks.v1+cell": { 133 | "cellMetadata": { 134 | "byteLimit": 2048000, 135 | "rowLimit": 10000 136 | }, 137 | "inputWidgets": {}, 138 | "nuid": "e54d6a66-4051-485a-adee-7ff167b902b1", 139 | "showTitle": false, 140 | "title": "" 141 | } 142 | }, 143 | "outputs": [ 144 | { 145 | "output_type": "display_data", 146 | "data": { 147 | "text/html": [ 148 | "
Sales_Order_IDRevision_NumberOrder_DateDue_DateShip_DateStatusOnline_Order_FlagSales_Order_NumberPurchase_Order_NumberAccount_NumberCustomer_IDShip_To_Address_IDBill_To_Address_IDShip_MethodCredit_Card_Approval_CodeSub_TotalTax_AmtFreightTotal_DueCommentrowguidModified_Date
7177422008-06-012008-06-132008-06-085falseSO71774PO34818628710-4020-0006092984710921092CARGO TRANSPORT 5null880.348470.427922.0087972.7850null89e42cdc-8506-48a2-b89b-eb3e64e3554e2008-06-08
7177622008-06-012008-06-132008-06-085falseSO71776PO1995219205110-4020-00010630072640640CARGO TRANSPORT 5null78.81006.30481.970387.0851null8a3448c5-e677-4158-a29b-dd33069be0b02008-06-08
7178022008-06-012008-06-132008-06-085falseSO71780PO1960417323910-4020-00034030113653653CARGO TRANSPORT 5null38418.68953073.4952960.467242452.6519nulla47665d2-7ac9-4cf3-8a8b-2a38835542842008-06-08
7178222008-06-012008-06-132008-06-085falseSO71782PO1937211474910-4020-0005822948510861086CARGO TRANSPORT 5null39785.33043182.8264994.633343962.7901nullf1be45a5-5c57-4a50-93c6-5f8be44cb7cb2008-06-08
7178322008-06-012008-06-132008-06-085falseSO71783PO1934311360910-4020-00002429957992992CARGO TRANSPORT 5null83858.42616708.67412096.460792663.5609null7db2329e-6446-42a8-8915-9c8370b68ed82008-06-08
7178422008-06-012008-06-132008-06-085falseSO71784PO1928513591910-4020-00044829736659659CARGO TRANSPORT 5null108561.83178684.94652714.0458119960.8240nullca31f324-2c32-4f8d-95eb-596e7f3430272008-06-08
7179622008-06-012008-06-132008-06-085falseSO71796PO1705215966410-4020-0004202966010581058CARGO TRANSPORT 5null57634.63424610.77071440.865963686.2708null917ef5ba-f32d-4563-8588-66db0bcdc8462008-06-08
7179722008-06-012008-06-132008-06-085falseSO71797PO1650113488910-4020-00014229796642642CARGO TRANSPORT 5null78029.68986242.37521950.742286222.8072nullbb3fee84-c8bf-4dd2-bcca-675ab6a11c382008-06-08
7181522008-06-012008-06-132008-06-085falseSO71815PO1302115578510-4020-0002763008910341034CARGO TRANSPORT 5null1141.578291.326328.53951261.4440null2aa5f39b-1096-4a4b-b17b-f10504a397ce2008-06-08
7181622008-06-012008-06-132008-06-085falseSO71816PO1299218044510-4020-0002953002710381038CARGO TRANSPORT 5null3398.1659271.853384.95413754.9733nulle3c189e7-98de-4c40-b6c2-0d1d13f9bb332008-06-08
7183122008-06-012008-06-132008-06-085falseSO71831PO1029511108410-4020-00032230019652652CARGO TRANSPORT 5null2016.3408161.307350.40852228.0566null625d76fc-c26f-4149-bf24-939fb2bccd772008-06-08
7183222008-06-012008-06-132008-06-085falseSO71832PO1035314075610-4020-00008829922639639CARGO TRANSPORT 5null35775.21132862.0169894.380339531.6085nulladdb8620-432a-456e-8470-1bedd4bc34572008-06-08
7184522008-06-012008-06-132008-06-085falseSO71845PO269711936210-4020-0001872993810201020CARGO TRANSPORT 5null41622.05113329.76411040.551345992.3665nulle68f7ee9-c581-45cd-9c4f-386aeda74d842008-06-08
7184622008-06-012008-06-132008-06-085falseSO71846PO237813160410-4020-00063530102669669CARGO TRANSPORT 5null2453.7645196.301261.34412711.4098nulla86d90ad-d1c0-440d-9a57-5b763bf182342008-06-08
7185622008-06-012008-06-132008-06-085falseSO71856PO1653017764710-4020-0006013003310901090CARGO TRANSPORT 5null602.194648.175615.0549665.4251null05fee073-0640-4a3c-914d-fe4ae6da3d432008-06-08
7185822008-06-012008-06-132008-06-085falseSO71858PO1615311227810-4020-0001862965310191019CARGO TRANSPORT 5null13823.70831105.8967345.592715275.1977null5ef091e1-a0af-437d-85ed-0b557c7923f72008-06-08
7186322008-06-012008-06-132008-06-085falseSO71863PO1612416656110-4020-0006492997510981098CARGO TRANSPORT 5null3324.2759265.942183.10693673.3249null3ed03b56-a4bf-4872-9471-bc6c7893eab72008-06-08
7186722008-06-012008-06-132008-06-085falseSO71867PO1305011152910-4020-00016029644643643CARGO TRANSPORT 5null1059.310084.744826.48281170.5376null29743c1b-d3af-4cfe-bd2e-6de436e3398f2008-06-08
7188522008-06-012008-06-132008-06-085falseSO71885PO611913077910-4020-00026829612649649CARGO TRANSPORT 5null550.386044.030913.7597608.1766nullcaad090d-56a6-444e-af24-7bee7605f1202008-06-08
7189522008-06-012008-06-132008-06-085falseSO71895PO377017627310-4020-0001512958410141014CARGO TRANSPORT 5null246.739219.73916.1685272.6468null07a17c0a-7f9a-4413-a035-519e5a4573be2008-06-08
7189722008-06-012008-06-132008-06-085falseSO71897PO478515247910-4020-0002232987710261026CARGO TRANSPORT 5null12685.88991014.8712317.147214017.9083nullf88b3458-e0cf-4248-9b72-1805351463b22008-06-08
7189822008-06-012008-06-132008-06-085falseSO71898PO571319050110-4020-00005229932637637CARGO TRANSPORT 5null63980.98845118.47911599.524770698.9922null96f84d24-3355-43d2-b5a3-55e97c17e58c2008-06-08
7189922008-06-012008-06-132008-06-085falseSO71899PO458214261110-4020-00002529568993993CARGO TRANSPORT 5null2415.6727193.253860.39182669.3183null31d41e8f-6f43-4cae-bee3-3cccb262f2312008-06-08
7190222008-06-012008-06-132008-06-085falseSO71902PO553912516610-4020-00006129929999999CARGO TRANSPORT 5null74058.80785924.70461851.470281834.9826null137850d6-efdf-4de1-920f-5757a86cdaaf2008-06-08
7191522008-06-012008-06-132008-06-085falseSO71915PO234914327510-4020-00000629638989989CARGO TRANSPORT 5null2137.2310170.978553.43082361.6403null0bcb3d9c-9008-4b99-bb45-928adae7be6b2008-06-08
7191722008-06-012008-06-132008-06-085falseSO71917PO2911171810-4020-00030430025651651CARGO TRANSPORT 5null40.90453.27241.022645.1995null6e903ea3-1b9e-4232-94c3-81c15669f8302008-06-08
7192022008-06-012008-06-132008-06-085falseSO71920PO400218985310-4020-0006742998211021102CARGO TRANSPORT 5null2980.7929238.463474.51983293.7761null6228c9cb-1cab-4e32-98ca-d0dae5fe563e2008-06-08
7192322008-06-012008-06-132008-06-085falseSO71923PO17417309610-4020-0002772978110351035CARGO TRANSPORT 5null106.54088.52332.6635117.7276nullf9899f3f-b4b6-4756-b013-96c16be204272008-06-08
7193522008-06-012008-06-132008-06-085falseSO71935PO794614587610-4020-0004382953110611061CARGO TRANSPORT 5null6634.2961530.7437165.85747330.8972null7033c6ec-b12c-45bc-bd96-56efde4c7dd02008-06-08
7193622008-06-012008-06-132008-06-085falseSO71936PO867117038510-4020-00050230050662662CARGO TRANSPORT 5null98278.69107862.29532456.9673108597.9536null119db56a-a97e-414d-b41c-64886fc50ab72008-06-08
7193822008-06-012008-06-132008-06-085falseSO71938PO846818331510-4020-00001629546635635CARGO TRANSPORT 5null88812.86257105.02902220.321698138.2131nulla36ee74a-cf0d-4024-a1ce-4eaffd1f85f02008-06-08
7194622008-06-012008-06-132008-06-085falseSO71946PO896115862910-4020-00046629741660660CARGO TRANSPORT 5null38.95363.11630.973843.0437null10e3129d-657f-46f9-86f5-cedd79b1901c2008-06-08
" 164 | ] 165 | }, 166 | "metadata": { 167 | "application/vnd.databricks.v1+output": { 168 | "addedWidgets": {}, 169 | "aggData": [], 170 | "aggError": "", 171 | "aggOverflow": false, 172 | "aggSchema": [], 173 | "aggSeriesLimitReached": false, 174 | "aggType": "", 175 | "arguments": {}, 176 | "columnCustomDisplayInfos": {}, 177 | "data": [ 178 | [ 179 | 71774, 180 | 2, 181 | "2008-06-01", 182 | "2008-06-13", 183 | "2008-06-08", 184 | 5, 185 | false, 186 | "SO71774", 187 | "PO348186287", 188 | "10-4020-000609", 189 | 29847, 190 | 1092, 191 | 1092, 192 | "CARGO TRANSPORT 5", 193 | null, 194 | "880.3484", 195 | "70.4279", 196 | "22.0087", 197 | "972.7850", 198 | null, 199 | "89e42cdc-8506-48a2-b89b-eb3e64e3554e", 200 | "2008-06-08" 201 | ], 202 | [ 203 | 71776, 204 | 2, 205 | "2008-06-01", 206 | "2008-06-13", 207 | "2008-06-08", 208 | 5, 209 | false, 210 | "SO71776", 211 | "PO19952192051", 212 | "10-4020-000106", 213 | 30072, 214 | 640, 215 | 640, 216 | "CARGO TRANSPORT 5", 217 | null, 218 | "78.8100", 219 | "6.3048", 220 | "1.9703", 221 | "87.0851", 222 | null, 223 | "8a3448c5-e677-4158-a29b-dd33069be0b0", 224 | "2008-06-08" 225 | ], 226 | [ 227 | 71780, 228 | 2, 229 | "2008-06-01", 230 | "2008-06-13", 231 | "2008-06-08", 232 | 5, 233 | false, 234 | "SO71780", 235 | "PO19604173239", 236 | "10-4020-000340", 237 | 30113, 238 | 653, 239 | 653, 240 | "CARGO TRANSPORT 5", 241 | null, 242 | "38418.6895", 243 | "3073.4952", 244 | "960.4672", 245 | "42452.6519", 246 | null, 247 | "a47665d2-7ac9-4cf3-8a8b-2a3883554284", 248 | "2008-06-08" 249 | ], 250 | [ 251 | 71782, 252 | 2, 253 | "2008-06-01", 254 | "2008-06-13", 255 | "2008-06-08", 256 | 5, 257 | false, 258 | "SO71782", 259 | "PO19372114749", 260 | "10-4020-000582", 261 | 29485, 262 | 1086, 263 | 1086, 264 | "CARGO TRANSPORT 5", 265 | null, 266 | "39785.3304", 267 | "3182.8264", 268 | "994.6333", 269 | "43962.7901", 270 | null, 271 | "f1be45a5-5c57-4a50-93c6-5f8be44cb7cb", 272 | "2008-06-08" 273 | ], 274 | [ 275 | 71783, 276 | 2, 277 | "2008-06-01", 278 | "2008-06-13", 279 | "2008-06-08", 280 | 5, 281 | false, 282 | "SO71783", 283 | "PO19343113609", 284 | "10-4020-000024", 285 | 29957, 286 | 992, 287 | 992, 288 | "CARGO TRANSPORT 5", 289 | null, 290 | "83858.4261", 291 | "6708.6741", 292 | "2096.4607", 293 | "92663.5609", 294 | null, 295 | "7db2329e-6446-42a8-8915-9c8370b68ed8", 296 | "2008-06-08" 297 | ], 298 | [ 299 | 71784, 300 | 2, 301 | "2008-06-01", 302 | "2008-06-13", 303 | "2008-06-08", 304 | 5, 305 | false, 306 | "SO71784", 307 | "PO19285135919", 308 | "10-4020-000448", 309 | 29736, 310 | 659, 311 | 659, 312 | "CARGO TRANSPORT 5", 313 | null, 314 | "108561.8317", 315 | "8684.9465", 316 | "2714.0458", 317 | "119960.8240", 318 | null, 319 | "ca31f324-2c32-4f8d-95eb-596e7f343027", 320 | "2008-06-08" 321 | ], 322 | [ 323 | 71796, 324 | 2, 325 | "2008-06-01", 326 | "2008-06-13", 327 | "2008-06-08", 328 | 5, 329 | false, 330 | "SO71796", 331 | "PO17052159664", 332 | "10-4020-000420", 333 | 29660, 334 | 1058, 335 | 1058, 336 | "CARGO TRANSPORT 5", 337 | null, 338 | "57634.6342", 339 | "4610.7707", 340 | "1440.8659", 341 | "63686.2708", 342 | null, 343 | "917ef5ba-f32d-4563-8588-66db0bcdc846", 344 | "2008-06-08" 345 | ], 346 | [ 347 | 71797, 348 | 2, 349 | "2008-06-01", 350 | "2008-06-13", 351 | "2008-06-08", 352 | 5, 353 | false, 354 | "SO71797", 355 | "PO16501134889", 356 | "10-4020-000142", 357 | 29796, 358 | 642, 359 | 642, 360 | "CARGO TRANSPORT 5", 361 | null, 362 | "78029.6898", 363 | "6242.3752", 364 | "1950.7422", 365 | "86222.8072", 366 | null, 367 | "bb3fee84-c8bf-4dd2-bcca-675ab6a11c38", 368 | "2008-06-08" 369 | ], 370 | [ 371 | 71815, 372 | 2, 373 | "2008-06-01", 374 | "2008-06-13", 375 | "2008-06-08", 376 | 5, 377 | false, 378 | "SO71815", 379 | "PO13021155785", 380 | "10-4020-000276", 381 | 30089, 382 | 1034, 383 | 1034, 384 | "CARGO TRANSPORT 5", 385 | null, 386 | "1141.5782", 387 | "91.3263", 388 | "28.5395", 389 | "1261.4440", 390 | null, 391 | "2aa5f39b-1096-4a4b-b17b-f10504a397ce", 392 | "2008-06-08" 393 | ], 394 | [ 395 | 71816, 396 | 2, 397 | "2008-06-01", 398 | "2008-06-13", 399 | "2008-06-08", 400 | 5, 401 | false, 402 | "SO71816", 403 | "PO12992180445", 404 | "10-4020-000295", 405 | 30027, 406 | 1038, 407 | 1038, 408 | "CARGO TRANSPORT 5", 409 | null, 410 | "3398.1659", 411 | "271.8533", 412 | "84.9541", 413 | "3754.9733", 414 | null, 415 | "e3c189e7-98de-4c40-b6c2-0d1d13f9bb33", 416 | "2008-06-08" 417 | ], 418 | [ 419 | 71831, 420 | 2, 421 | "2008-06-01", 422 | "2008-06-13", 423 | "2008-06-08", 424 | 5, 425 | false, 426 | "SO71831", 427 | "PO10295111084", 428 | "10-4020-000322", 429 | 30019, 430 | 652, 431 | 652, 432 | "CARGO TRANSPORT 5", 433 | null, 434 | "2016.3408", 435 | "161.3073", 436 | "50.4085", 437 | "2228.0566", 438 | null, 439 | "625d76fc-c26f-4149-bf24-939fb2bccd77", 440 | "2008-06-08" 441 | ], 442 | [ 443 | 71832, 444 | 2, 445 | "2008-06-01", 446 | "2008-06-13", 447 | "2008-06-08", 448 | 5, 449 | false, 450 | "SO71832", 451 | "PO10353140756", 452 | "10-4020-000088", 453 | 29922, 454 | 639, 455 | 639, 456 | "CARGO TRANSPORT 5", 457 | null, 458 | "35775.2113", 459 | "2862.0169", 460 | "894.3803", 461 | "39531.6085", 462 | null, 463 | "addb8620-432a-456e-8470-1bedd4bc3457", 464 | "2008-06-08" 465 | ], 466 | [ 467 | 71845, 468 | 2, 469 | "2008-06-01", 470 | "2008-06-13", 471 | "2008-06-08", 472 | 5, 473 | false, 474 | "SO71845", 475 | "PO2697119362", 476 | "10-4020-000187", 477 | 29938, 478 | 1020, 479 | 1020, 480 | "CARGO TRANSPORT 5", 481 | null, 482 | "41622.0511", 483 | "3329.7641", 484 | "1040.5513", 485 | "45992.3665", 486 | null, 487 | "e68f7ee9-c581-45cd-9c4f-386aeda74d84", 488 | "2008-06-08" 489 | ], 490 | [ 491 | 71846, 492 | 2, 493 | "2008-06-01", 494 | "2008-06-13", 495 | "2008-06-08", 496 | 5, 497 | false, 498 | "SO71846", 499 | "PO2378131604", 500 | "10-4020-000635", 501 | 30102, 502 | 669, 503 | 669, 504 | "CARGO TRANSPORT 5", 505 | null, 506 | "2453.7645", 507 | "196.3012", 508 | "61.3441", 509 | "2711.4098", 510 | null, 511 | "a86d90ad-d1c0-440d-9a57-5b763bf18234", 512 | "2008-06-08" 513 | ], 514 | [ 515 | 71856, 516 | 2, 517 | "2008-06-01", 518 | "2008-06-13", 519 | "2008-06-08", 520 | 5, 521 | false, 522 | "SO71856", 523 | "PO16530177647", 524 | "10-4020-000601", 525 | 30033, 526 | 1090, 527 | 1090, 528 | "CARGO TRANSPORT 5", 529 | null, 530 | "602.1946", 531 | "48.1756", 532 | "15.0549", 533 | "665.4251", 534 | null, 535 | "05fee073-0640-4a3c-914d-fe4ae6da3d43", 536 | "2008-06-08" 537 | ], 538 | [ 539 | 71858, 540 | 2, 541 | "2008-06-01", 542 | "2008-06-13", 543 | "2008-06-08", 544 | 5, 545 | false, 546 | "SO71858", 547 | "PO16153112278", 548 | "10-4020-000186", 549 | 29653, 550 | 1019, 551 | 1019, 552 | "CARGO TRANSPORT 5", 553 | null, 554 | "13823.7083", 555 | "1105.8967", 556 | "345.5927", 557 | "15275.1977", 558 | null, 559 | "5ef091e1-a0af-437d-85ed-0b557c7923f7", 560 | "2008-06-08" 561 | ], 562 | [ 563 | 71863, 564 | 2, 565 | "2008-06-01", 566 | "2008-06-13", 567 | "2008-06-08", 568 | 5, 569 | false, 570 | "SO71863", 571 | "PO16124166561", 572 | "10-4020-000649", 573 | 29975, 574 | 1098, 575 | 1098, 576 | "CARGO TRANSPORT 5", 577 | null, 578 | "3324.2759", 579 | "265.9421", 580 | "83.1069", 581 | "3673.3249", 582 | null, 583 | "3ed03b56-a4bf-4872-9471-bc6c7893eab7", 584 | "2008-06-08" 585 | ], 586 | [ 587 | 71867, 588 | 2, 589 | "2008-06-01", 590 | "2008-06-13", 591 | "2008-06-08", 592 | 5, 593 | false, 594 | "SO71867", 595 | "PO13050111529", 596 | "10-4020-000160", 597 | 29644, 598 | 643, 599 | 643, 600 | "CARGO TRANSPORT 5", 601 | null, 602 | "1059.3100", 603 | "84.7448", 604 | "26.4828", 605 | "1170.5376", 606 | null, 607 | "29743c1b-d3af-4cfe-bd2e-6de436e3398f", 608 | "2008-06-08" 609 | ], 610 | [ 611 | 71885, 612 | 2, 613 | "2008-06-01", 614 | "2008-06-13", 615 | "2008-06-08", 616 | 5, 617 | false, 618 | "SO71885", 619 | "PO6119130779", 620 | "10-4020-000268", 621 | 29612, 622 | 649, 623 | 649, 624 | "CARGO TRANSPORT 5", 625 | null, 626 | "550.3860", 627 | "44.0309", 628 | "13.7597", 629 | "608.1766", 630 | null, 631 | "caad090d-56a6-444e-af24-7bee7605f120", 632 | "2008-06-08" 633 | ], 634 | [ 635 | 71895, 636 | 2, 637 | "2008-06-01", 638 | "2008-06-13", 639 | "2008-06-08", 640 | 5, 641 | false, 642 | "SO71895", 643 | "PO3770176273", 644 | "10-4020-000151", 645 | 29584, 646 | 1014, 647 | 1014, 648 | "CARGO TRANSPORT 5", 649 | null, 650 | "246.7392", 651 | "19.7391", 652 | "6.1685", 653 | "272.6468", 654 | null, 655 | "07a17c0a-7f9a-4413-a035-519e5a4573be", 656 | "2008-06-08" 657 | ], 658 | [ 659 | 71897, 660 | 2, 661 | "2008-06-01", 662 | "2008-06-13", 663 | "2008-06-08", 664 | 5, 665 | false, 666 | "SO71897", 667 | "PO4785152479", 668 | "10-4020-000223", 669 | 29877, 670 | 1026, 671 | 1026, 672 | "CARGO TRANSPORT 5", 673 | null, 674 | "12685.8899", 675 | "1014.8712", 676 | "317.1472", 677 | "14017.9083", 678 | null, 679 | "f88b3458-e0cf-4248-9b72-1805351463b2", 680 | "2008-06-08" 681 | ], 682 | [ 683 | 71898, 684 | 2, 685 | "2008-06-01", 686 | "2008-06-13", 687 | "2008-06-08", 688 | 5, 689 | false, 690 | "SO71898", 691 | "PO5713190501", 692 | "10-4020-000052", 693 | 29932, 694 | 637, 695 | 637, 696 | "CARGO TRANSPORT 5", 697 | null, 698 | "63980.9884", 699 | "5118.4791", 700 | "1599.5247", 701 | "70698.9922", 702 | null, 703 | "96f84d24-3355-43d2-b5a3-55e97c17e58c", 704 | "2008-06-08" 705 | ], 706 | [ 707 | 71899, 708 | 2, 709 | "2008-06-01", 710 | "2008-06-13", 711 | "2008-06-08", 712 | 5, 713 | false, 714 | "SO71899", 715 | "PO4582142611", 716 | "10-4020-000025", 717 | 29568, 718 | 993, 719 | 993, 720 | "CARGO TRANSPORT 5", 721 | null, 722 | "2415.6727", 723 | "193.2538", 724 | "60.3918", 725 | "2669.3183", 726 | null, 727 | "31d41e8f-6f43-4cae-bee3-3cccb262f231", 728 | "2008-06-08" 729 | ], 730 | [ 731 | 71902, 732 | 2, 733 | "2008-06-01", 734 | "2008-06-13", 735 | "2008-06-08", 736 | 5, 737 | false, 738 | "SO71902", 739 | "PO5539125166", 740 | "10-4020-000061", 741 | 29929, 742 | 999, 743 | 999, 744 | "CARGO TRANSPORT 5", 745 | null, 746 | "74058.8078", 747 | "5924.7046", 748 | "1851.4702", 749 | "81834.9826", 750 | null, 751 | "137850d6-efdf-4de1-920f-5757a86cdaaf", 752 | "2008-06-08" 753 | ], 754 | [ 755 | 71915, 756 | 2, 757 | "2008-06-01", 758 | "2008-06-13", 759 | "2008-06-08", 760 | 5, 761 | false, 762 | "SO71915", 763 | "PO2349143275", 764 | "10-4020-000006", 765 | 29638, 766 | 989, 767 | 989, 768 | "CARGO TRANSPORT 5", 769 | null, 770 | "2137.2310", 771 | "170.9785", 772 | "53.4308", 773 | "2361.6403", 774 | null, 775 | "0bcb3d9c-9008-4b99-bb45-928adae7be6b", 776 | "2008-06-08" 777 | ], 778 | [ 779 | 71917, 780 | 2, 781 | "2008-06-01", 782 | "2008-06-13", 783 | "2008-06-08", 784 | 5, 785 | false, 786 | "SO71917", 787 | "PO29111718", 788 | "10-4020-000304", 789 | 30025, 790 | 651, 791 | 651, 792 | "CARGO TRANSPORT 5", 793 | null, 794 | "40.9045", 795 | "3.2724", 796 | "1.0226", 797 | "45.1995", 798 | null, 799 | "6e903ea3-1b9e-4232-94c3-81c15669f830", 800 | "2008-06-08" 801 | ], 802 | [ 803 | 71920, 804 | 2, 805 | "2008-06-01", 806 | "2008-06-13", 807 | "2008-06-08", 808 | 5, 809 | false, 810 | "SO71920", 811 | "PO4002189853", 812 | "10-4020-000674", 813 | 29982, 814 | 1102, 815 | 1102, 816 | "CARGO TRANSPORT 5", 817 | null, 818 | "2980.7929", 819 | "238.4634", 820 | "74.5198", 821 | "3293.7761", 822 | null, 823 | "6228c9cb-1cab-4e32-98ca-d0dae5fe563e", 824 | "2008-06-08" 825 | ], 826 | [ 827 | 71923, 828 | 2, 829 | "2008-06-01", 830 | "2008-06-13", 831 | "2008-06-08", 832 | 5, 833 | false, 834 | "SO71923", 835 | "PO174173096", 836 | "10-4020-000277", 837 | 29781, 838 | 1035, 839 | 1035, 840 | "CARGO TRANSPORT 5", 841 | null, 842 | "106.5408", 843 | "8.5233", 844 | "2.6635", 845 | "117.7276", 846 | null, 847 | "f9899f3f-b4b6-4756-b013-96c16be20427", 848 | "2008-06-08" 849 | ], 850 | [ 851 | 71935, 852 | 2, 853 | "2008-06-01", 854 | "2008-06-13", 855 | "2008-06-08", 856 | 5, 857 | false, 858 | "SO71935", 859 | "PO7946145876", 860 | "10-4020-000438", 861 | 29531, 862 | 1061, 863 | 1061, 864 | "CARGO TRANSPORT 5", 865 | null, 866 | "6634.2961", 867 | "530.7437", 868 | "165.8574", 869 | "7330.8972", 870 | null, 871 | "7033c6ec-b12c-45bc-bd96-56efde4c7dd0", 872 | "2008-06-08" 873 | ], 874 | [ 875 | 71936, 876 | 2, 877 | "2008-06-01", 878 | "2008-06-13", 879 | "2008-06-08", 880 | 5, 881 | false, 882 | "SO71936", 883 | "PO8671170385", 884 | "10-4020-000502", 885 | 30050, 886 | 662, 887 | 662, 888 | "CARGO TRANSPORT 5", 889 | null, 890 | "98278.6910", 891 | "7862.2953", 892 | "2456.9673", 893 | "108597.9536", 894 | null, 895 | "119db56a-a97e-414d-b41c-64886fc50ab7", 896 | "2008-06-08" 897 | ], 898 | [ 899 | 71938, 900 | 2, 901 | "2008-06-01", 902 | "2008-06-13", 903 | "2008-06-08", 904 | 5, 905 | false, 906 | "SO71938", 907 | "PO8468183315", 908 | "10-4020-000016", 909 | 29546, 910 | 635, 911 | 635, 912 | "CARGO TRANSPORT 5", 913 | null, 914 | "88812.8625", 915 | "7105.0290", 916 | "2220.3216", 917 | "98138.2131", 918 | null, 919 | "a36ee74a-cf0d-4024-a1ce-4eaffd1f85f0", 920 | "2008-06-08" 921 | ], 922 | [ 923 | 71946, 924 | 2, 925 | "2008-06-01", 926 | "2008-06-13", 927 | "2008-06-08", 928 | 5, 929 | false, 930 | "SO71946", 931 | "PO8961158629", 932 | "10-4020-000466", 933 | 29741, 934 | 660, 935 | 660, 936 | "CARGO TRANSPORT 5", 937 | null, 938 | "38.9536", 939 | "3.1163", 940 | "0.9738", 941 | "43.0437", 942 | null, 943 | "10e3129d-657f-46f9-86f5-cedd79b1901c", 944 | "2008-06-08" 945 | ] 946 | ], 947 | "datasetInfos": [], 948 | "dbfsResultPath": null, 949 | "isJsonSchema": true, 950 | "metadata": {}, 951 | "overflow": false, 952 | "plotOptions": { 953 | "customPlotOptions": {}, 954 | "displayType": "table", 955 | "pivotAggregation": null, 956 | "pivotColumns": null, 957 | "xColumns": null, 958 | "yColumns": null 959 | }, 960 | "removedWidgets": [], 961 | "schema": [ 962 | { 963 | "metadata": "{}", 964 | "name": "Sales_Order_ID", 965 | "type": "\"integer\"" 966 | }, 967 | { 968 | "metadata": "{}", 969 | "name": "Revision_Number", 970 | "type": "\"integer\"" 971 | }, 972 | { 973 | "metadata": "{}", 974 | "name": "Order_Date", 975 | "type": "\"string\"" 976 | }, 977 | { 978 | "metadata": "{}", 979 | "name": "Due_Date", 980 | "type": "\"string\"" 981 | }, 982 | { 983 | "metadata": "{}", 984 | "name": "Ship_Date", 985 | "type": "\"string\"" 986 | }, 987 | { 988 | "metadata": "{}", 989 | "name": "Status", 990 | "type": "\"integer\"" 991 | }, 992 | { 993 | "metadata": "{}", 994 | "name": "Online_Order_Flag", 995 | "type": "\"boolean\"" 996 | }, 997 | { 998 | "metadata": "{}", 999 | "name": "Sales_Order_Number", 1000 | "type": "\"string\"" 1001 | }, 1002 | { 1003 | "metadata": "{}", 1004 | "name": "Purchase_Order_Number", 1005 | "type": "\"string\"" 1006 | }, 1007 | { 1008 | "metadata": "{}", 1009 | "name": "Account_Number", 1010 | "type": "\"string\"" 1011 | }, 1012 | { 1013 | "metadata": "{}", 1014 | "name": "Customer_ID", 1015 | "type": "\"integer\"" 1016 | }, 1017 | { 1018 | "metadata": "{}", 1019 | "name": "Ship_To_Address_ID", 1020 | "type": "\"integer\"" 1021 | }, 1022 | { 1023 | "metadata": "{}", 1024 | "name": "Bill_To_Address_ID", 1025 | "type": "\"integer\"" 1026 | }, 1027 | { 1028 | "metadata": "{}", 1029 | "name": "Ship_Method", 1030 | "type": "\"string\"" 1031 | }, 1032 | { 1033 | "metadata": "{}", 1034 | "name": "Credit_Card_Approval_Code", 1035 | "type": "\"string\"" 1036 | }, 1037 | { 1038 | "metadata": "{}", 1039 | "name": "Sub_Total", 1040 | "type": "\"decimal(19,4)\"" 1041 | }, 1042 | { 1043 | "metadata": "{}", 1044 | "name": "Tax_Amt", 1045 | "type": "\"decimal(19,4)\"" 1046 | }, 1047 | { 1048 | "metadata": "{}", 1049 | "name": "Freight", 1050 | "type": "\"decimal(19,4)\"" 1051 | }, 1052 | { 1053 | "metadata": "{}", 1054 | "name": "Total_Due", 1055 | "type": "\"decimal(19,4)\"" 1056 | }, 1057 | { 1058 | "metadata": "{}", 1059 | "name": "Comment", 1060 | "type": "\"string\"" 1061 | }, 1062 | { 1063 | "metadata": "{}", 1064 | "name": "rowguid", 1065 | "type": "\"string\"" 1066 | }, 1067 | { 1068 | "metadata": "{}", 1069 | "name": "Modified_Date", 1070 | "type": "\"string\"" 1071 | } 1072 | ], 1073 | "type": "table" 1074 | } 1075 | }, 1076 | "output_type": "display_data" 1077 | } 1078 | ], 1079 | "source": [ 1080 | "display(df)" 1081 | ] 1082 | } 1083 | ], 1084 | "metadata": { 1085 | "application/vnd.databricks.v1+notebook": { 1086 | "dashboards": [], 1087 | "language": "python", 1088 | "notebookMetadata": { 1089 | "pythonIndentUnit": 4 1090 | }, 1091 | "notebookName": "silver to gold", 1092 | "widgets": {} 1093 | } 1094 | }, 1095 | "nbformat": 4, 1096 | "nbformat_minor": 0 1097 | } 1098 | -------------------------------------------------------------------------------- /databricks_notebooks/storagemount.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 0, 6 | "metadata": { 7 | "application/vnd.databricks.v1+cell": { 8 | "cellMetadata": { 9 | "byteLimit": 2048000, 10 | "rowLimit": 10000 11 | }, 12 | "inputWidgets": {}, 13 | "nuid": "8d47c7ce-fe9b-4038-ba49-d01580ba2d0a", 14 | "showTitle": false, 15 | "title": "" 16 | } 17 | }, 18 | "outputs": [ 19 | { 20 | "output_type": "display_data", 21 | "data": { 22 | "text/plain": [ 23 | "\u001B[0;31m---------------------------------------------------------------------------\u001B[0m\n", 24 | "\u001B[0;31mExecutionError\u001B[0m Traceback (most recent call last)\n", 25 | "File \u001B[0;32m, line 7\u001B[0m\n", 26 | "\u001B[1;32m 1\u001B[0m configs \u001B[38;5;241m=\u001B[39m {\n", 27 | "\u001B[1;32m 2\u001B[0m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mfs.azure.account.auth.type\u001B[39m\u001B[38;5;124m\"\u001B[39m: \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mCustomAccessToken\u001B[39m\u001B[38;5;124m\"\u001B[39m,\n", 28 | "\u001B[1;32m 3\u001B[0m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mfs.azure.account.custom.token.provider.class\u001B[39m\u001B[38;5;124m\"\u001B[39m: spark\u001B[38;5;241m.\u001B[39mconf\u001B[38;5;241m.\u001B[39mget(\u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mspark.databricks.passthrough.adls.gen2.tokenProviderClassName\u001B[39m\u001B[38;5;124m\"\u001B[39m)\n", 29 | "\u001B[1;32m 4\u001B[0m }\n", 30 | "\u001B[1;32m 6\u001B[0m \u001B[38;5;66;03m# Optionally, you can add to the source URI of your mount point.\u001B[39;00m\n", 31 | "\u001B[0;32m----> 7\u001B[0m dbutils\u001B[38;5;241m.\u001B[39mfs\u001B[38;5;241m.\u001B[39mmount(\n", 32 | "\u001B[1;32m 8\u001B[0m source \u001B[38;5;241m=\u001B[39m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mabfss://bronze@datalakestorage123nik.dfs.core.windows.net/\u001B[39m\u001B[38;5;124m\"\u001B[39m,\n", 33 | "\u001B[1;32m 9\u001B[0m mount_point \u001B[38;5;241m=\u001B[39m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124m/mnt/bronze\u001B[39m\u001B[38;5;124m\"\u001B[39m,\n", 34 | "\u001B[1;32m 10\u001B[0m extra_configs \u001B[38;5;241m=\u001B[39m configs)\n", 35 | "\n", 36 | "File \u001B[0;32m/databricks/python_shell/dbruntime/dbutils.py:362\u001B[0m, in \u001B[0;36mDBUtils.FSHandler.prettify_exception_message..f_with_exception_handling\u001B[0;34m(*args, **kwargs)\u001B[0m\n", 37 | "\u001B[1;32m 360\u001B[0m exc\u001B[38;5;241m.\u001B[39m__context__ \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;01mNone\u001B[39;00m\n", 38 | "\u001B[1;32m 361\u001B[0m exc\u001B[38;5;241m.\u001B[39m__cause__ \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;01mNone\u001B[39;00m\n", 39 | "\u001B[0;32m--> 362\u001B[0m \u001B[38;5;28;01mraise\u001B[39;00m exc\n", 40 | "\n", 41 | "\u001B[0;31mExecutionError\u001B[0m: An error occurred while calling o404.mount.\n", 42 | ": java.rmi.RemoteException: java.lang.IllegalArgumentException: requirement failed: Directory already mounted: /mnt/bronze; nested exception is: \n", 43 | "\tjava.lang.IllegalArgumentException: requirement failed: Directory already mounted: /mnt/bronze\n", 44 | "\tat com.databricks.backend.daemon.data.client.DbfsClient.send0(DbfsClient.scala:135)\n", 45 | "\tat com.databricks.backend.daemon.data.client.DbfsClient.sendIdempotent(DbfsClient.scala:69)\n", 46 | "\tat com.databricks.backend.daemon.dbutils.DBUtilsCore.createOrUpdateMount(DBUtilsCore.scala:1057)\n", 47 | "\tat com.databricks.backend.daemon.dbutils.DBUtilsCore.$anonfun$mount$1(DBUtilsCore.scala:1083)\n", 48 | "\tat com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:571)\n", 49 | "\tat com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:666)\n", 50 | "\tat com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:684)\n", 51 | "\tat com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)\n", 52 | "\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)\n", 53 | "\tat com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:196)\n", 54 | "\tat com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)\n", 55 | "\tat com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)\n", 56 | "\tat com.databricks.backend.daemon.dbutils.FSUtils.withAttributionContext(DBUtilsCore.scala:69)\n", 57 | "\tat com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:470)\n", 58 | "\tat com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:455)\n", 59 | "\tat com.databricks.backend.daemon.dbutils.FSUtils.withAttributionTags(DBUtilsCore.scala:69)\n", 60 | "\tat com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:661)\n", 61 | "\tat com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:580)\n", 62 | "\tat com.databricks.backend.daemon.dbutils.FSUtils.recordOperationWithResultTags(DBUtilsCore.scala:69)\n", 63 | "\tat com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:571)\n", 64 | "\tat com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:540)\n", 65 | "\tat com.databricks.backend.daemon.dbutils.FSUtils.recordOperation(DBUtilsCore.scala:69)\n", 66 | "\tat com.databricks.backend.daemon.dbutils.FSUtils.recordDbutilsFsOp(DBUtilsCore.scala:133)\n", 67 | "\tat com.databricks.backend.daemon.dbutils.DBUtilsCore.mount(DBUtilsCore.scala:1077)\n", 68 | "\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n", 69 | "\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n", 70 | "\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n", 71 | "\tat java.lang.reflect.Method.invoke(Method.java:498)\n", 72 | "\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n", 73 | "\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:397)\n", 74 | "\tat py4j.Gateway.invoke(Gateway.java:306)\n", 75 | "\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n", 76 | "\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n", 77 | "\tat py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:195)\n", 78 | "\tat py4j.ClientServerConnection.run(ClientServerConnection.java:115)\n", 79 | "\tat java.lang.Thread.run(Thread.java:750)\n", 80 | "Caused by: java.lang.IllegalArgumentException: requirement failed: Directory already mounted: /mnt/bronze\n", 81 | "\tat scala.Predef$.require(Predef.scala:281)\n", 82 | "\tat com.databricks.backend.daemon.data.server.DefaultMetadataManager.$anonfun$insertMount$1(MetadataManager.scala:580)\n", 83 | "\tat com.databricks.backend.daemon.data.server.DefaultMetadataManager.$anonfun$modifyAndVerify$2(MetadataManager.scala:948)\n", 84 | "\tat scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)\n", 85 | "\tat com.databricks.backend.daemon.data.server.DefaultMetadataManager.withRetries(MetadataManager.scala:729)\n", 86 | "\tat com.databricks.backend.daemon.data.server.DefaultMetadataManager.modifyAndVerify(MetadataManager.scala:937)\n", 87 | "\tat com.databricks.backend.daemon.data.server.DefaultMetadataManager.insertMount(MetadataManager.scala:588)\n", 88 | "\tat com.databricks.backend.daemon.data.server.handler.MountHandler.receive(MountHandler.scala:120)\n", 89 | "\tat com.databricks.backend.daemon.data.server.session.SessionContext.$anonfun$queryHandlers$1(SessionContext.scala:54)\n", 90 | "\tat com.databricks.backend.daemon.data.server.session.SessionContext.$anonfun$queryHandlers$1$adapted(SessionContext.scala:53)\n", 91 | "\tat scala.collection.immutable.List.foreach(List.scala:431)\n", 92 | "\tat com.databricks.backend.daemon.data.server.session.SessionContext.queryHandlers(SessionContext.scala:53)\n", 93 | "\tat com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$4.$anonfun$applyOrElse$5(DbfsServerBackend.scala:388)\n", 94 | "\tat com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)\n", 95 | "\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)\n", 96 | "\tat com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:216)\n", 97 | "\tat com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)\n", 98 | "\tat com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)\n", 99 | "\tat com.databricks.rpc.ServerBackend.withAttributionContext(ServerBackend.scala:22)\n", 100 | "\tat com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:472)\n", 101 | "\tat com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:455)\n", 102 | "\tat com.databricks.rpc.ServerBackend.withAttributionTags(ServerBackend.scala:22)\n", 103 | "\tat com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$4.applyOrElse(DbfsServerBackend.scala:388)\n", 104 | "\tat com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$4.applyOrElse(DbfsServerBackend.scala:332)\n", 105 | "\tat com.databricks.rpc.ServerBackend.$anonfun$internalReceive0$2(ServerBackend.scala:174)\n", 106 | "\tat com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:200)\n", 107 | "\tat com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:200)\n", 108 | "\tat com.databricks.rpc.ServerBackend.internalReceive0(ServerBackend.scala:171)\n", 109 | "\tat com.databricks.rpc.ServerBackend.$anonfun$internalReceive$1(ServerBackend.scala:147)\n", 110 | "\tat com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:573)\n", 111 | "\tat com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:668)\n", 112 | "\tat com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:686)\n", 113 | "\tat com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)\n", 114 | "\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)\n", 115 | "\tat com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:216)\n", 116 | "\tat com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)\n", 117 | "\tat com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)\n", 118 | "\tat com.databricks.rpc.ServerBackend.withAttributionContext(ServerBackend.scala:22)\n", 119 | "\tat com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:472)\n", 120 | "\tat com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:455)\n", 121 | "\tat com.databricks.rpc.ServerBackend.withAttributionTags(ServerBackend.scala:22)\n", 122 | "\tat com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:663)\n", 123 | "\tat com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:582)\n", 124 | "\tat com.databricks.rpc.ServerBackend.recordOperationWithResultTags(ServerBackend.scala:22)\n", 125 | "\tat com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:573)\n", 126 | "\tat com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:542)\n", 127 | "\tat com.databricks.rpc.ServerBackend.recordOperation(ServerBackend.scala:22)\n", 128 | "\tat com.databricks.rpc.ServerBackend.internalReceive(ServerBackend.scala:147)\n", 129 | "\tat com.databricks.rpc.JettyServer$RequestManager.handleRPC(JettyServer.scala:1020)\n", 130 | "\tat com.databricks.rpc.JettyServer$RequestManager.handleRequestAndRespond(JettyServer.scala:941)\n", 131 | "\tat com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleHttp$6(JettyServer.scala:545)\n", 132 | "\tat com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleHttp$6$adapted(JettyServer.scala:514)\n", 133 | "\tat com.databricks.logging.activity.ActivityContextFactory$.$anonfun$withActivityInternal$3(ActivityContextFactory.scala:398)\n", 134 | "\tat com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)\n", 135 | "\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)\n", 136 | "\tat com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:216)\n", 137 | "\tat com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)\n", 138 | "\tat com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)\n", 139 | "\tat com.databricks.logging.activity.ActivityContextFactory$.withAttributionContext(ActivityContextFactory.scala:53)\n", 140 | "\tat com.databricks.logging.activity.ActivityContextFactory$.withActivityInternal(ActivityContextFactory.scala:398)\n", 141 | "\tat com.databricks.logging.activity.ActivityContextFactory$.withServiceRequestActivity(ActivityContextFactory.scala:154)\n", 142 | "\tat com.databricks.rpc.JettyServer$RequestManager.handleHttp(JettyServer.scala:514)\n", 143 | "\tat com.databricks.rpc.JettyServer$RequestManager.doPost(JettyServer.scala:404)\n", 144 | "\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:665)\n", 145 | "\tat com.databricks.rpc.HttpServletWithPatch.service(HttpServletWithPatch.scala:33)\n", 146 | "\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:750)\n", 147 | "\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)\n", 148 | "\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)\n", 149 | "\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)\n", 150 | "\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n", 151 | "\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n", 152 | "\tat org.eclipse.jetty.server.Server.handle(Server.java:539)\n", 153 | "\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)\n", 154 | "\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n", 155 | "\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n", 156 | "\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n", 157 | "\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n", 158 | "\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n", 159 | "\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n", 160 | "\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n", 161 | "\tat com.databricks.rpc.InstrumentedQueuedThreadPool$$anon$1.$anonfun$run$1(InstrumentedQueuedThreadPool.scala:81)\n", 162 | "\tat scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)\n", 163 | "\tat com.databricks.instrumentation.QueuedThreadPoolInstrumenter.trackActiveThreads(QueuedThreadPoolInstrumenter.scala:66)\n", 164 | "\tat com.databricks.instrumentation.QueuedThreadPoolInstrumenter.trackActiveThreads$(QueuedThreadPoolInstrumenter.scala:63)\n", 165 | "\tat com.databricks.rpc.InstrumentedQueuedThreadPool.trackActiveThreads(InstrumentedQueuedThreadPool.scala:47)\n", 166 | "\tat com.databricks.rpc.InstrumentedQueuedThreadPool$$anon$1.run(InstrumentedQueuedThreadPool.scala:76)\n", 167 | "\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n", 168 | "\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n", 169 | "\t... 1 more\n" 170 | ] 171 | }, 172 | "metadata": { 173 | "application/vnd.databricks.v1+output": { 174 | "arguments": {}, 175 | "data": "\u001B[0;31m---------------------------------------------------------------------------\u001B[0m\n\u001B[0;31mExecutionError\u001B[0m Traceback (most recent call last)\nFile \u001B[0;32m, line 7\u001B[0m\n\u001B[1;32m 1\u001B[0m configs \u001B[38;5;241m=\u001B[39m {\n\u001B[1;32m 2\u001B[0m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mfs.azure.account.auth.type\u001B[39m\u001B[38;5;124m\"\u001B[39m: \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mCustomAccessToken\u001B[39m\u001B[38;5;124m\"\u001B[39m,\n\u001B[1;32m 3\u001B[0m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mfs.azure.account.custom.token.provider.class\u001B[39m\u001B[38;5;124m\"\u001B[39m: spark\u001B[38;5;241m.\u001B[39mconf\u001B[38;5;241m.\u001B[39mget(\u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mspark.databricks.passthrough.adls.gen2.tokenProviderClassName\u001B[39m\u001B[38;5;124m\"\u001B[39m)\n\u001B[1;32m 4\u001B[0m }\n\u001B[1;32m 6\u001B[0m \u001B[38;5;66;03m# Optionally, you can add to the source URI of your mount point.\u001B[39;00m\n\u001B[0;32m----> 7\u001B[0m dbutils\u001B[38;5;241m.\u001B[39mfs\u001B[38;5;241m.\u001B[39mmount(\n\u001B[1;32m 8\u001B[0m source \u001B[38;5;241m=\u001B[39m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mabfss://bronze@datalakestorage123nik.dfs.core.windows.net/\u001B[39m\u001B[38;5;124m\"\u001B[39m,\n\u001B[1;32m 9\u001B[0m mount_point \u001B[38;5;241m=\u001B[39m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124m/mnt/bronze\u001B[39m\u001B[38;5;124m\"\u001B[39m,\n\u001B[1;32m 10\u001B[0m extra_configs \u001B[38;5;241m=\u001B[39m configs)\n\nFile \u001B[0;32m/databricks/python_shell/dbruntime/dbutils.py:362\u001B[0m, in \u001B[0;36mDBUtils.FSHandler.prettify_exception_message..f_with_exception_handling\u001B[0;34m(*args, **kwargs)\u001B[0m\n\u001B[1;32m 360\u001B[0m exc\u001B[38;5;241m.\u001B[39m__context__ \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;01mNone\u001B[39;00m\n\u001B[1;32m 361\u001B[0m exc\u001B[38;5;241m.\u001B[39m__cause__ \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;01mNone\u001B[39;00m\n\u001B[0;32m--> 362\u001B[0m \u001B[38;5;28;01mraise\u001B[39;00m exc\n\n\u001B[0;31mExecutionError\u001B[0m: An error occurred while calling o404.mount.\n: java.rmi.RemoteException: java.lang.IllegalArgumentException: requirement failed: Directory already mounted: /mnt/bronze; nested exception is: \n\tjava.lang.IllegalArgumentException: requirement failed: Directory already mounted: /mnt/bronze\n\tat com.databricks.backend.daemon.data.client.DbfsClient.send0(DbfsClient.scala:135)\n\tat com.databricks.backend.daemon.data.client.DbfsClient.sendIdempotent(DbfsClient.scala:69)\n\tat com.databricks.backend.daemon.dbutils.DBUtilsCore.createOrUpdateMount(DBUtilsCore.scala:1057)\n\tat com.databricks.backend.daemon.dbutils.DBUtilsCore.$anonfun$mount$1(DBUtilsCore.scala:1083)\n\tat com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:571)\n\tat com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:666)\n\tat com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:684)\n\tat com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)\n\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)\n\tat com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:196)\n\tat com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)\n\tat com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)\n\tat com.databricks.backend.daemon.dbutils.FSUtils.withAttributionContext(DBUtilsCore.scala:69)\n\tat com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:470)\n\tat com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:455)\n\tat com.databricks.backend.daemon.dbutils.FSUtils.withAttributionTags(DBUtilsCore.scala:69)\n\tat com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:661)\n\tat com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:580)\n\tat com.databricks.backend.daemon.dbutils.FSUtils.recordOperationWithResultTags(DBUtilsCore.scala:69)\n\tat com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:571)\n\tat com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:540)\n\tat com.databricks.backend.daemon.dbutils.FSUtils.recordOperation(DBUtilsCore.scala:69)\n\tat com.databricks.backend.daemon.dbutils.FSUtils.recordDbutilsFsOp(DBUtilsCore.scala:133)\n\tat com.databricks.backend.daemon.dbutils.DBUtilsCore.mount(DBUtilsCore.scala:1077)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:397)\n\tat py4j.Gateway.invoke(Gateway.java:306)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:195)\n\tat py4j.ClientServerConnection.run(ClientServerConnection.java:115)\n\tat java.lang.Thread.run(Thread.java:750)\nCaused by: java.lang.IllegalArgumentException: requirement failed: Directory already mounted: /mnt/bronze\n\tat scala.Predef$.require(Predef.scala:281)\n\tat com.databricks.backend.daemon.data.server.DefaultMetadataManager.$anonfun$insertMount$1(MetadataManager.scala:580)\n\tat com.databricks.backend.daemon.data.server.DefaultMetadataManager.$anonfun$modifyAndVerify$2(MetadataManager.scala:948)\n\tat scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)\n\tat com.databricks.backend.daemon.data.server.DefaultMetadataManager.withRetries(MetadataManager.scala:729)\n\tat com.databricks.backend.daemon.data.server.DefaultMetadataManager.modifyAndVerify(MetadataManager.scala:937)\n\tat com.databricks.backend.daemon.data.server.DefaultMetadataManager.insertMount(MetadataManager.scala:588)\n\tat com.databricks.backend.daemon.data.server.handler.MountHandler.receive(MountHandler.scala:120)\n\tat com.databricks.backend.daemon.data.server.session.SessionContext.$anonfun$queryHandlers$1(SessionContext.scala:54)\n\tat com.databricks.backend.daemon.data.server.session.SessionContext.$anonfun$queryHandlers$1$adapted(SessionContext.scala:53)\n\tat scala.collection.immutable.List.foreach(List.scala:431)\n\tat com.databricks.backend.daemon.data.server.session.SessionContext.queryHandlers(SessionContext.scala:53)\n\tat com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$4.$anonfun$applyOrElse$5(DbfsServerBackend.scala:388)\n\tat com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)\n\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)\n\tat com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:216)\n\tat com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)\n\tat com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)\n\tat com.databricks.rpc.ServerBackend.withAttributionContext(ServerBackend.scala:22)\n\tat com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:472)\n\tat com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:455)\n\tat com.databricks.rpc.ServerBackend.withAttributionTags(ServerBackend.scala:22)\n\tat com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$4.applyOrElse(DbfsServerBackend.scala:388)\n\tat com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$4.applyOrElse(DbfsServerBackend.scala:332)\n\tat com.databricks.rpc.ServerBackend.$anonfun$internalReceive0$2(ServerBackend.scala:174)\n\tat com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:200)\n\tat com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:200)\n\tat com.databricks.rpc.ServerBackend.internalReceive0(ServerBackend.scala:171)\n\tat com.databricks.rpc.ServerBackend.$anonfun$internalReceive$1(ServerBackend.scala:147)\n\tat com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:573)\n\tat com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:668)\n\tat com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:686)\n\tat com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)\n\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)\n\tat com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:216)\n\tat com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)\n\tat com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)\n\tat com.databricks.rpc.ServerBackend.withAttributionContext(ServerBackend.scala:22)\n\tat com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:472)\n\tat com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:455)\n\tat com.databricks.rpc.ServerBackend.withAttributionTags(ServerBackend.scala:22)\n\tat com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:663)\n\tat com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:582)\n\tat com.databricks.rpc.ServerBackend.recordOperationWithResultTags(ServerBackend.scala:22)\n\tat com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:573)\n\tat com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:542)\n\tat com.databricks.rpc.ServerBackend.recordOperation(ServerBackend.scala:22)\n\tat com.databricks.rpc.ServerBackend.internalReceive(ServerBackend.scala:147)\n\tat com.databricks.rpc.JettyServer$RequestManager.handleRPC(JettyServer.scala:1020)\n\tat com.databricks.rpc.JettyServer$RequestManager.handleRequestAndRespond(JettyServer.scala:941)\n\tat com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleHttp$6(JettyServer.scala:545)\n\tat com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleHttp$6$adapted(JettyServer.scala:514)\n\tat com.databricks.logging.activity.ActivityContextFactory$.$anonfun$withActivityInternal$3(ActivityContextFactory.scala:398)\n\tat com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)\n\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)\n\tat com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:216)\n\tat com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)\n\tat com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)\n\tat com.databricks.logging.activity.ActivityContextFactory$.withAttributionContext(ActivityContextFactory.scala:53)\n\tat com.databricks.logging.activity.ActivityContextFactory$.withActivityInternal(ActivityContextFactory.scala:398)\n\tat com.databricks.logging.activity.ActivityContextFactory$.withServiceRequestActivity(ActivityContextFactory.scala:154)\n\tat com.databricks.rpc.JettyServer$RequestManager.handleHttp(JettyServer.scala:514)\n\tat com.databricks.rpc.JettyServer$RequestManager.doPost(JettyServer.scala:404)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:665)\n\tat com.databricks.rpc.HttpServletWithPatch.service(HttpServletWithPatch.scala:33)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:750)\n\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:539)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\tat com.databricks.rpc.InstrumentedQueuedThreadPool$$anon$1.$anonfun$run$1(InstrumentedQueuedThreadPool.scala:81)\n\tat scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)\n\tat com.databricks.instrumentation.QueuedThreadPoolInstrumenter.trackActiveThreads(QueuedThreadPoolInstrumenter.scala:66)\n\tat com.databricks.instrumentation.QueuedThreadPoolInstrumenter.trackActiveThreads$(QueuedThreadPoolInstrumenter.scala:63)\n\tat com.databricks.rpc.InstrumentedQueuedThreadPool.trackActiveThreads(InstrumentedQueuedThreadPool.scala:47)\n\tat com.databricks.rpc.InstrumentedQueuedThreadPool$$anon$1.run(InstrumentedQueuedThreadPool.scala:76)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\t... 1 more\n", 176 | "errorSummary": "java.rmi.RemoteException: java.lang.IllegalArgumentException: requirement failed: Directory already mounted: /mnt/bronze; nested exception is: ", 177 | "errorTraceType": "ansi", 178 | "metadata": {}, 179 | "type": "ipynbError" 180 | } 181 | }, 182 | "output_type": "display_data" 183 | } 184 | ], 185 | "source": [ 186 | "configs = {\n", 187 | " \"fs.azure.account.auth.type\": \"CustomAccessToken\",\n", 188 | " \"fs.azure.account.custom.token.provider.class\": spark.conf.get(\"spark.databricks.passthrough.adls.gen2.tokenProviderClassName\")\n", 189 | "}\n", 190 | "\n", 191 | "# Optionally, you can add to the source URI of your mount point.\n", 192 | "dbutils.fs.mount(\n", 193 | " source = \"abfss://bronze@datalakestorage123nik.dfs.core.windows.net/\",\n", 194 | " mount_point = \"/mnt/bronze\",\n", 195 | " extra_configs = configs)" 196 | ] 197 | }, 198 | { 199 | "cell_type": "code", 200 | "execution_count": 0, 201 | "metadata": { 202 | "application/vnd.databricks.v1+cell": { 203 | "cellMetadata": { 204 | "byteLimit": 2048000, 205 | "rowLimit": 10000 206 | }, 207 | "inputWidgets": {}, 208 | "nuid": "ef735375-3a97-449d-b13f-3b5fb137e955", 209 | "showTitle": false, 210 | "title": "" 211 | } 212 | }, 213 | "outputs": [ 214 | { 215 | "output_type": "display_data", 216 | "data": { 217 | "text/plain": [ 218 | "" 219 | ] 220 | }, 221 | "metadata": { 222 | "application/vnd.databricks.v1+output": { 223 | "arguments": {}, 224 | "data": "", 225 | "errorSummary": "Command skipped", 226 | "errorTraceType": "ansi", 227 | "metadata": {}, 228 | "type": "ipynbError" 229 | } 230 | }, 231 | "output_type": "display_data" 232 | } 233 | ], 234 | "source": [ 235 | "dbutils.fs.ls(\"/mnt/bronze/SalesLT/\")" 236 | ] 237 | }, 238 | { 239 | "cell_type": "code", 240 | "execution_count": 0, 241 | "metadata": { 242 | "application/vnd.databricks.v1+cell": { 243 | "cellMetadata": { 244 | "byteLimit": 2048000, 245 | "rowLimit": 10000 246 | }, 247 | "inputWidgets": {}, 248 | "nuid": "fbdf9915-ff58-48c8-90b0-2aa7b9d5a2f3", 249 | "showTitle": false, 250 | "title": "" 251 | } 252 | }, 253 | "outputs": [ 254 | { 255 | "output_type": "display_data", 256 | "data": { 257 | "text/plain": [ 258 | "" 259 | ] 260 | }, 261 | "metadata": { 262 | "application/vnd.databricks.v1+output": { 263 | "arguments": {}, 264 | "data": "", 265 | "errorSummary": "Command skipped", 266 | "errorTraceType": "ansi", 267 | "metadata": {}, 268 | "type": "ipynbError" 269 | } 270 | }, 271 | "output_type": "display_data" 272 | } 273 | ], 274 | "source": [ 275 | "configs = {\n", 276 | " \"fs.azure.account.auth.type\": \"CustomAccessToken\",\n", 277 | " \"fs.azure.account.custom.token.provider.class\": spark.conf.get(\"spark.databricks.passthrough.adls.gen2.tokenProviderClassName\")\n", 278 | "}\n", 279 | "\n", 280 | "# Optionally, you can add to the source URI of your mount point.\n", 281 | "dbutils.fs.mount(\n", 282 | " source = \"abfss://silver@datalakestorage123nik.dfs.core.windows.net/\",\n", 283 | " mount_point = \"/mnt/silver\",\n", 284 | " extra_configs = configs)" 285 | ] 286 | }, 287 | { 288 | "cell_type": "code", 289 | "execution_count": 0, 290 | "metadata": { 291 | "application/vnd.databricks.v1+cell": { 292 | "cellMetadata": { 293 | "byteLimit": 2048000, 294 | "rowLimit": 10000 295 | }, 296 | "inputWidgets": {}, 297 | "nuid": "cb26c20c-630b-47a0-9235-122b47d8ae1c", 298 | "showTitle": false, 299 | "title": "" 300 | } 301 | }, 302 | "outputs": [ 303 | { 304 | "output_type": "display_data", 305 | "data": { 306 | "text/plain": [ 307 | "" 308 | ] 309 | }, 310 | "metadata": { 311 | "application/vnd.databricks.v1+output": { 312 | "arguments": {}, 313 | "data": "", 314 | "errorSummary": "Command skipped", 315 | "errorTraceType": "ansi", 316 | "metadata": {}, 317 | "type": "ipynbError" 318 | } 319 | }, 320 | "output_type": "display_data" 321 | } 322 | ], 323 | "source": [ 324 | "configs = {\n", 325 | " \"fs.azure.account.auth.type\": \"CustomAccessToken\",\n", 326 | " \"fs.azure.account.custom.token.provider.class\": spark.conf.get(\"spark.databricks.passthrough.adls.gen2.tokenProviderClassName\")\n", 327 | "}\n", 328 | "\n", 329 | "# Optionally, you can add to the source URI of your mount point.\n", 330 | "dbutils.fs.mount(\n", 331 | " source = \"abfss://gold@datalakestorage123nik.dfs.core.windows.net/\",\n", 332 | " mount_point = \"/mnt/gold\",\n", 333 | " extra_configs = configs)" 334 | ] 335 | } 336 | ], 337 | "metadata": { 338 | "application/vnd.databricks.v1+notebook": { 339 | "dashboards": [], 340 | "language": "python", 341 | "notebookMetadata": { 342 | "pythonIndentUnit": 4 343 | }, 344 | "notebookName": "storagemount", 345 | "widgets": {} 346 | } 347 | }, 348 | "nbformat": 4, 349 | "nbformat_minor": 0 350 | } 351 | -------------------------------------------------------------------------------- /gif.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/gif.gif -------------------------------------------------------------------------------- /images/d: -------------------------------------------------------------------------------- 1 | d 2 | -------------------------------------------------------------------------------- /sdfasd.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/sdfasd.gif -------------------------------------------------------------------------------- /sdfasdsxd.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/sdfasdsxd.gif --------------------------------------------------------------------------------