├── 1 O4ey_K0ZbsESf8na7OirJg.jpg
├── README.md
├── Screenshot 2023-10-11 225825.png
├── Screenshot 2023-10-12 002249.png
├── Screenshot 2023-10-12 003056.png
├── Screenshot 2023-10-12 004350.png
├── Screenshot 2023-10-12 005837.png
├── Screenshot 2023-10-12 005959.png
├── Screenshot 2023-10-12 011305.png
├── Screenshot 2023-10-12 012305.png
├── Screenshot 2023-10-12 013527.png
├── Screenshot_1.png
├── databricks notebooks
├── databricks_notebooks
├── bronze to silver.ipynb
├── init
├── silver to gold.ipynb
└── storagemount.ipynb
├── gif.gif
├── images
└── d
├── sdfasd.gif
└── sdfasdsxd.gif
/1 O4ey_K0ZbsESf8na7OirJg.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/1 O4ey_K0ZbsESf8na7OirJg.jpg
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 |
2 | # End-to-End Azure Data Engineering
3 |
4 | This project demonstrates an end-to-end Azure data engineering solution, starting from a local SQL database and culminating in Power BI reporting, all automated.
5 |
6 | Thanks to [Mr. K Talks Tech](https://www.youtube.com/@mr.ktalkstech) for the project inspiration.
7 |
8 | ## Business Objective
9 |
10 | This project serves as a learning opportunity for common data engineering practices, focusing on ETL pipeline techniques. The skills sharpened here are valuable for small to medium-sized businesses aiming to migrate their local data to the cloud.
11 |
12 | 
13 |
14 | ## Current Environment
15 |
16 |
17 |
18 |
19 |
20 | - Utilized the AdventureWorks dataset from Microsoft.
21 | - Set up an on-premises Microsoft SQL server on a personal computer.
22 | - Imported the dataset using Microsoft SQL Server Management Studio.
23 | - Created a new user profile, "nik."
24 | - Saved "nik" profile's password credentials as a Secret in Azure Key Vault.
25 |
26 | 
27 |
28 |
29 | ## 1: Data Ingestion
30 |
31 | Data ingestion from the on-premises SQL server to Azure SQL is accomplished via Azure Data Factory. The process involves:
32 |
33 | 1. Installation of Self-Hosted Integration Runtime.
34 | 2. Establishing a connection between Azure Data Factory and the local SQL Server.
35 | 3. Setting up a copy pipeline to transfer all tables from the local SQL server to the Azure Data Lake's "bronze" folder.
36 |
37 |
38 | 
39 |
40 |
41 |
42 | ## 2: Data Transformation
43 |
44 | After ingesting data into the "bronze" folder, it is transformed following the medallion data lake architecture (bronze, silver, gold). Data transitions through bronze, silver, and ultimately gold, suitable for business reporting tools like Power BI.
45 |
46 | Azure Databricks, using PySpark, is used for these transformations. Data initially stored in parquet format in the "bronze" folder is converted to the delta format as it progresses to "silver" and "gold." This transformation is carried out through Databricks notebooks:
47 |
48 | 1. Mount the storage.
49 | 2. Transform data from "bronze" to "silver" layer.
50 | 3. Further transform data from "silver" to "gold" layer.
51 |
52 | 
53 |
54 | Azure Data Factory is updated to execute the "bronze" to "silver" and "silver" to "gold" notebooks automatically with each pipeline run.
55 |
56 | 
57 |
58 |
59 | ## 3: Data Loading
60 |
61 | Data from the "gold" folder is loaded into the Business Intelligence reporting application, Power BI. Azure Synapse is used for this purpose. The steps involve:
62 |
63 | 1. Creating a link from Azure Storage (Gold Folder) to Azure Synapse.
64 | 2. Writing stored procedures to extract table information as a SQL view.
65 | 3. Storing views within a server-less SQL Database in Synapse.
66 |
67 | 
68 |
69 |
70 | ## 4: Data Reporting
71 |
72 | Power BI connects directly to the cloud pipeline using DirectQuery to dynamically update the database. A Power BI report is developed to visualize AdventureWorks dataset data, including sales, product information, and customer gender.
73 |
74 | 
75 |
76 |
77 |
78 | ## 5: Final Pipeline Test
79 |
80 | To verify the end-to-end pipeline, two new customers are added to the local SQL database server. If successful, the pipeline will update, and the Power BI report will dynamically show the new data. The total number of customers should increase from 847 to 849.
81 |
82 | 
83 | Great success!
84 |
85 |
86 | ## Conclusion and Limitations
87 |
88 | This project demonstrates the ability to create an end-to-end ETL cloud solution using Azure. Some considerations:
89 |
90 | - The dataset used was small (7mb total, 800 rows). This was done to keep compute + storage costs low for myself.
91 | - Multiple applications were employed for a relatively simple task.
92 | - Given the dataset's simplicity, the project could have been managed entirely through Azure Data Factory, with data cleaning done downstream in Power BI.
93 | - The inclusion of Azure Synapse and Databricks was for the sake of self-learning and emulating real-world business pipelines.
94 |
--------------------------------------------------------------------------------
/Screenshot 2023-10-11 225825.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/Screenshot 2023-10-11 225825.png
--------------------------------------------------------------------------------
/Screenshot 2023-10-12 002249.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/Screenshot 2023-10-12 002249.png
--------------------------------------------------------------------------------
/Screenshot 2023-10-12 003056.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/Screenshot 2023-10-12 003056.png
--------------------------------------------------------------------------------
/Screenshot 2023-10-12 004350.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/Screenshot 2023-10-12 004350.png
--------------------------------------------------------------------------------
/Screenshot 2023-10-12 005837.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/Screenshot 2023-10-12 005837.png
--------------------------------------------------------------------------------
/Screenshot 2023-10-12 005959.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/Screenshot 2023-10-12 005959.png
--------------------------------------------------------------------------------
/Screenshot 2023-10-12 011305.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/Screenshot 2023-10-12 011305.png
--------------------------------------------------------------------------------
/Screenshot 2023-10-12 012305.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/Screenshot 2023-10-12 012305.png
--------------------------------------------------------------------------------
/Screenshot 2023-10-12 013527.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/Screenshot 2023-10-12 013527.png
--------------------------------------------------------------------------------
/Screenshot_1.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/Screenshot_1.png
--------------------------------------------------------------------------------
/databricks notebooks:
--------------------------------------------------------------------------------
1 |
2 |
--------------------------------------------------------------------------------
/databricks_notebooks/bronze to silver.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {
6 | "application/vnd.databricks.v1+cell": {
7 | "cellMetadata": {
8 | "byteLimit": 2048000,
9 | "rowLimit": 10000
10 | },
11 | "inputWidgets": {},
12 | "nuid": "0018e4ec-ea86-4dd5-b85e-307bf7dbbeb3",
13 | "showTitle": false,
14 | "title": ""
15 | }
16 | },
17 | "source": [
18 | "\n",
19 | "## Transforming All Tables"
20 | ]
21 | },
22 | {
23 | "cell_type": "code",
24 | "execution_count": 0,
25 | "metadata": {
26 | "application/vnd.databricks.v1+cell": {
27 | "cellMetadata": {
28 | "byteLimit": 2048000,
29 | "rowLimit": 10000
30 | },
31 | "inputWidgets": {},
32 | "nuid": "b98623f4-e542-4af6-9798-99014eede3eb",
33 | "showTitle": false,
34 | "title": ""
35 | }
36 | },
37 | "outputs": [],
38 | "source": [
39 | "table_name = []\n",
40 | "\n",
41 | "for i in dbutils.fs.ls('mnt/bronze/SalesLT/'):\n",
42 | " table_name.append(i.name.split('/')[0])"
43 | ]
44 | },
45 | {
46 | "cell_type": "code",
47 | "execution_count": 0,
48 | "metadata": {
49 | "application/vnd.databricks.v1+cell": {
50 | "cellMetadata": {
51 | "byteLimit": 2048000,
52 | "rowLimit": 10000
53 | },
54 | "inputWidgets": {},
55 | "nuid": "78df1697-a6b2-4a95-9624-4ee39ae87e27",
56 | "showTitle": false,
57 | "title": ""
58 | }
59 | },
60 | "outputs": [
61 | {
62 | "output_type": "execute_result",
63 | "data": {
64 | "text/plain": [
65 | "['Address',\n",
66 | " 'Customer',\n",
67 | " 'CustomerAddress',\n",
68 | " 'Product',\n",
69 | " 'ProductCategory',\n",
70 | " 'ProductDescription',\n",
71 | " 'ProductModel',\n",
72 | " 'ProductModelProductDescription',\n",
73 | " 'SalesOrderDetail',\n",
74 | " 'SalesOrderHeader']"
75 | ]
76 | },
77 | "execution_count": 20,
78 | "metadata": {},
79 | "output_type": "execute_result"
80 | }
81 | ],
82 | "source": [
83 | "table_name"
84 | ]
85 | },
86 | {
87 | "cell_type": "code",
88 | "execution_count": 0,
89 | "metadata": {
90 | "application/vnd.databricks.v1+cell": {
91 | "cellMetadata": {
92 | "byteLimit": 2048000,
93 | "rowLimit": 10000
94 | },
95 | "inputWidgets": {},
96 | "nuid": "801e2647-4acb-415a-b959-47f86d1815e2",
97 | "showTitle": false,
98 | "title": ""
99 | }
100 | },
101 | "outputs": [],
102 | "source": [
103 | "from pyspark. sql. functions import from_utc_timestamp, date_format\n",
104 | "from pyspark.sql.types import TimestampType\n",
105 | "\n",
106 | "for i in table_name: \n",
107 | " path = '/mnt/bronze/SalesLT/' + i + '/' + i + '.parquet'\n",
108 | " df = spark.read.format('parquet').load(path)\n",
109 | " column = df.columns\n",
110 | " \n",
111 | " for col in column:\n",
112 | " if \"Date\" in col or \"date\" in col:\n",
113 | " df = df.withColumn(col, date_format(from_utc_timestamp(df[col].cast(TimestampType()), \"UTC\"), \"yyyy-MM-dd\"))\n",
114 | " \n",
115 | " output_path = '/mnt/silver/SalesLT/' +i + '/'\n",
116 | " df.write.format('delta').mode(\"overwrite\").save(output_path)"
117 | ]
118 | },
119 | {
120 | "cell_type": "code",
121 | "execution_count": 0,
122 | "metadata": {
123 | "application/vnd.databricks.v1+cell": {
124 | "cellMetadata": {
125 | "byteLimit": 2048000,
126 | "rowLimit": 10000
127 | },
128 | "inputWidgets": {},
129 | "nuid": "dc148455-12c1-423f-9906-e2971e25dc82",
130 | "showTitle": false,
131 | "title": ""
132 | }
133 | },
134 | "outputs": [
135 | {
136 | "output_type": "display_data",
137 | "data": {
138 | "text/html": [
139 | "
SalesOrderID | RevisionNumber | OrderDate | DueDate | ShipDate | Status | OnlineOrderFlag | SalesOrderNumber | PurchaseOrderNumber | AccountNumber | CustomerID | ShipToAddressID | BillToAddressID | ShipMethod | CreditCardApprovalCode | SubTotal | TaxAmt | Freight | TotalDue | Comment | rowguid | ModifiedDate |
---|
71774 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71774 | PO348186287 | 10-4020-000609 | 29847 | 1092 | 1092 | CARGO TRANSPORT 5 | null | 880.3484 | 70.4279 | 22.0087 | 972.7850 | null | 89e42cdc-8506-48a2-b89b-eb3e64e3554e | 2008-06-08 |
71776 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71776 | PO19952192051 | 10-4020-000106 | 30072 | 640 | 640 | CARGO TRANSPORT 5 | null | 78.8100 | 6.3048 | 1.9703 | 87.0851 | null | 8a3448c5-e677-4158-a29b-dd33069be0b0 | 2008-06-08 |
71780 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71780 | PO19604173239 | 10-4020-000340 | 30113 | 653 | 653 | CARGO TRANSPORT 5 | null | 38418.6895 | 3073.4952 | 960.4672 | 42452.6519 | null | a47665d2-7ac9-4cf3-8a8b-2a3883554284 | 2008-06-08 |
71782 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71782 | PO19372114749 | 10-4020-000582 | 29485 | 1086 | 1086 | CARGO TRANSPORT 5 | null | 39785.3304 | 3182.8264 | 994.6333 | 43962.7901 | null | f1be45a5-5c57-4a50-93c6-5f8be44cb7cb | 2008-06-08 |
71783 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71783 | PO19343113609 | 10-4020-000024 | 29957 | 992 | 992 | CARGO TRANSPORT 5 | null | 83858.4261 | 6708.6741 | 2096.4607 | 92663.5609 | null | 7db2329e-6446-42a8-8915-9c8370b68ed8 | 2008-06-08 |
71784 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71784 | PO19285135919 | 10-4020-000448 | 29736 | 659 | 659 | CARGO TRANSPORT 5 | null | 108561.8317 | 8684.9465 | 2714.0458 | 119960.8240 | null | ca31f324-2c32-4f8d-95eb-596e7f343027 | 2008-06-08 |
71796 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71796 | PO17052159664 | 10-4020-000420 | 29660 | 1058 | 1058 | CARGO TRANSPORT 5 | null | 57634.6342 | 4610.7707 | 1440.8659 | 63686.2708 | null | 917ef5ba-f32d-4563-8588-66db0bcdc846 | 2008-06-08 |
71797 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71797 | PO16501134889 | 10-4020-000142 | 29796 | 642 | 642 | CARGO TRANSPORT 5 | null | 78029.6898 | 6242.3752 | 1950.7422 | 86222.8072 | null | bb3fee84-c8bf-4dd2-bcca-675ab6a11c38 | 2008-06-08 |
71815 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71815 | PO13021155785 | 10-4020-000276 | 30089 | 1034 | 1034 | CARGO TRANSPORT 5 | null | 1141.5782 | 91.3263 | 28.5395 | 1261.4440 | null | 2aa5f39b-1096-4a4b-b17b-f10504a397ce | 2008-06-08 |
71816 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71816 | PO12992180445 | 10-4020-000295 | 30027 | 1038 | 1038 | CARGO TRANSPORT 5 | null | 3398.1659 | 271.8533 | 84.9541 | 3754.9733 | null | e3c189e7-98de-4c40-b6c2-0d1d13f9bb33 | 2008-06-08 |
71831 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71831 | PO10295111084 | 10-4020-000322 | 30019 | 652 | 652 | CARGO TRANSPORT 5 | null | 2016.3408 | 161.3073 | 50.4085 | 2228.0566 | null | 625d76fc-c26f-4149-bf24-939fb2bccd77 | 2008-06-08 |
71832 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71832 | PO10353140756 | 10-4020-000088 | 29922 | 639 | 639 | CARGO TRANSPORT 5 | null | 35775.2113 | 2862.0169 | 894.3803 | 39531.6085 | null | addb8620-432a-456e-8470-1bedd4bc3457 | 2008-06-08 |
71845 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71845 | PO2697119362 | 10-4020-000187 | 29938 | 1020 | 1020 | CARGO TRANSPORT 5 | null | 41622.0511 | 3329.7641 | 1040.5513 | 45992.3665 | null | e68f7ee9-c581-45cd-9c4f-386aeda74d84 | 2008-06-08 |
71846 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71846 | PO2378131604 | 10-4020-000635 | 30102 | 669 | 669 | CARGO TRANSPORT 5 | null | 2453.7645 | 196.3012 | 61.3441 | 2711.4098 | null | a86d90ad-d1c0-440d-9a57-5b763bf18234 | 2008-06-08 |
71856 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71856 | PO16530177647 | 10-4020-000601 | 30033 | 1090 | 1090 | CARGO TRANSPORT 5 | null | 602.1946 | 48.1756 | 15.0549 | 665.4251 | null | 05fee073-0640-4a3c-914d-fe4ae6da3d43 | 2008-06-08 |
71858 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71858 | PO16153112278 | 10-4020-000186 | 29653 | 1019 | 1019 | CARGO TRANSPORT 5 | null | 13823.7083 | 1105.8967 | 345.5927 | 15275.1977 | null | 5ef091e1-a0af-437d-85ed-0b557c7923f7 | 2008-06-08 |
71863 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71863 | PO16124166561 | 10-4020-000649 | 29975 | 1098 | 1098 | CARGO TRANSPORT 5 | null | 3324.2759 | 265.9421 | 83.1069 | 3673.3249 | null | 3ed03b56-a4bf-4872-9471-bc6c7893eab7 | 2008-06-08 |
71867 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71867 | PO13050111529 | 10-4020-000160 | 29644 | 643 | 643 | CARGO TRANSPORT 5 | null | 1059.3100 | 84.7448 | 26.4828 | 1170.5376 | null | 29743c1b-d3af-4cfe-bd2e-6de436e3398f | 2008-06-08 |
71885 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71885 | PO6119130779 | 10-4020-000268 | 29612 | 649 | 649 | CARGO TRANSPORT 5 | null | 550.3860 | 44.0309 | 13.7597 | 608.1766 | null | caad090d-56a6-444e-af24-7bee7605f120 | 2008-06-08 |
71895 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71895 | PO3770176273 | 10-4020-000151 | 29584 | 1014 | 1014 | CARGO TRANSPORT 5 | null | 246.7392 | 19.7391 | 6.1685 | 272.6468 | null | 07a17c0a-7f9a-4413-a035-519e5a4573be | 2008-06-08 |
71897 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71897 | PO4785152479 | 10-4020-000223 | 29877 | 1026 | 1026 | CARGO TRANSPORT 5 | null | 12685.8899 | 1014.8712 | 317.1472 | 14017.9083 | null | f88b3458-e0cf-4248-9b72-1805351463b2 | 2008-06-08 |
71898 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71898 | PO5713190501 | 10-4020-000052 | 29932 | 637 | 637 | CARGO TRANSPORT 5 | null | 63980.9884 | 5118.4791 | 1599.5247 | 70698.9922 | null | 96f84d24-3355-43d2-b5a3-55e97c17e58c | 2008-06-08 |
71899 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71899 | PO4582142611 | 10-4020-000025 | 29568 | 993 | 993 | CARGO TRANSPORT 5 | null | 2415.6727 | 193.2538 | 60.3918 | 2669.3183 | null | 31d41e8f-6f43-4cae-bee3-3cccb262f231 | 2008-06-08 |
71902 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71902 | PO5539125166 | 10-4020-000061 | 29929 | 999 | 999 | CARGO TRANSPORT 5 | null | 74058.8078 | 5924.7046 | 1851.4702 | 81834.9826 | null | 137850d6-efdf-4de1-920f-5757a86cdaaf | 2008-06-08 |
71915 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71915 | PO2349143275 | 10-4020-000006 | 29638 | 989 | 989 | CARGO TRANSPORT 5 | null | 2137.2310 | 170.9785 | 53.4308 | 2361.6403 | null | 0bcb3d9c-9008-4b99-bb45-928adae7be6b | 2008-06-08 |
71917 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71917 | PO29111718 | 10-4020-000304 | 30025 | 651 | 651 | CARGO TRANSPORT 5 | null | 40.9045 | 3.2724 | 1.0226 | 45.1995 | null | 6e903ea3-1b9e-4232-94c3-81c15669f830 | 2008-06-08 |
71920 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71920 | PO4002189853 | 10-4020-000674 | 29982 | 1102 | 1102 | CARGO TRANSPORT 5 | null | 2980.7929 | 238.4634 | 74.5198 | 3293.7761 | null | 6228c9cb-1cab-4e32-98ca-d0dae5fe563e | 2008-06-08 |
71923 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71923 | PO174173096 | 10-4020-000277 | 29781 | 1035 | 1035 | CARGO TRANSPORT 5 | null | 106.5408 | 8.5233 | 2.6635 | 117.7276 | null | f9899f3f-b4b6-4756-b013-96c16be20427 | 2008-06-08 |
71935 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71935 | PO7946145876 | 10-4020-000438 | 29531 | 1061 | 1061 | CARGO TRANSPORT 5 | null | 6634.2961 | 530.7437 | 165.8574 | 7330.8972 | null | 7033c6ec-b12c-45bc-bd96-56efde4c7dd0 | 2008-06-08 |
71936 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71936 | PO8671170385 | 10-4020-000502 | 30050 | 662 | 662 | CARGO TRANSPORT 5 | null | 98278.6910 | 7862.2953 | 2456.9673 | 108597.9536 | null | 119db56a-a97e-414d-b41c-64886fc50ab7 | 2008-06-08 |
71938 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71938 | PO8468183315 | 10-4020-000016 | 29546 | 635 | 635 | CARGO TRANSPORT 5 | null | 88812.8625 | 7105.0290 | 2220.3216 | 98138.2131 | null | a36ee74a-cf0d-4024-a1ce-4eaffd1f85f0 | 2008-06-08 |
71946 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71946 | PO8961158629 | 10-4020-000466 | 29741 | 660 | 660 | CARGO TRANSPORT 5 | null | 38.9536 | 3.1163 | 0.9738 | 43.0437 | null | 10e3129d-657f-46f9-86f5-cedd79b1901c | 2008-06-08 |
"
155 | ]
156 | },
157 | "metadata": {
158 | "application/vnd.databricks.v1+output": {
159 | "addedWidgets": {},
160 | "aggData": [],
161 | "aggError": "",
162 | "aggOverflow": false,
163 | "aggSchema": [],
164 | "aggSeriesLimitReached": false,
165 | "aggType": "",
166 | "arguments": {},
167 | "columnCustomDisplayInfos": {},
168 | "data": [
169 | [
170 | 71774,
171 | 2,
172 | "2008-06-01",
173 | "2008-06-13",
174 | "2008-06-08",
175 | 5,
176 | false,
177 | "SO71774",
178 | "PO348186287",
179 | "10-4020-000609",
180 | 29847,
181 | 1092,
182 | 1092,
183 | "CARGO TRANSPORT 5",
184 | null,
185 | "880.3484",
186 | "70.4279",
187 | "22.0087",
188 | "972.7850",
189 | null,
190 | "89e42cdc-8506-48a2-b89b-eb3e64e3554e",
191 | "2008-06-08"
192 | ],
193 | [
194 | 71776,
195 | 2,
196 | "2008-06-01",
197 | "2008-06-13",
198 | "2008-06-08",
199 | 5,
200 | false,
201 | "SO71776",
202 | "PO19952192051",
203 | "10-4020-000106",
204 | 30072,
205 | 640,
206 | 640,
207 | "CARGO TRANSPORT 5",
208 | null,
209 | "78.8100",
210 | "6.3048",
211 | "1.9703",
212 | "87.0851",
213 | null,
214 | "8a3448c5-e677-4158-a29b-dd33069be0b0",
215 | "2008-06-08"
216 | ],
217 | [
218 | 71780,
219 | 2,
220 | "2008-06-01",
221 | "2008-06-13",
222 | "2008-06-08",
223 | 5,
224 | false,
225 | "SO71780",
226 | "PO19604173239",
227 | "10-4020-000340",
228 | 30113,
229 | 653,
230 | 653,
231 | "CARGO TRANSPORT 5",
232 | null,
233 | "38418.6895",
234 | "3073.4952",
235 | "960.4672",
236 | "42452.6519",
237 | null,
238 | "a47665d2-7ac9-4cf3-8a8b-2a3883554284",
239 | "2008-06-08"
240 | ],
241 | [
242 | 71782,
243 | 2,
244 | "2008-06-01",
245 | "2008-06-13",
246 | "2008-06-08",
247 | 5,
248 | false,
249 | "SO71782",
250 | "PO19372114749",
251 | "10-4020-000582",
252 | 29485,
253 | 1086,
254 | 1086,
255 | "CARGO TRANSPORT 5",
256 | null,
257 | "39785.3304",
258 | "3182.8264",
259 | "994.6333",
260 | "43962.7901",
261 | null,
262 | "f1be45a5-5c57-4a50-93c6-5f8be44cb7cb",
263 | "2008-06-08"
264 | ],
265 | [
266 | 71783,
267 | 2,
268 | "2008-06-01",
269 | "2008-06-13",
270 | "2008-06-08",
271 | 5,
272 | false,
273 | "SO71783",
274 | "PO19343113609",
275 | "10-4020-000024",
276 | 29957,
277 | 992,
278 | 992,
279 | "CARGO TRANSPORT 5",
280 | null,
281 | "83858.4261",
282 | "6708.6741",
283 | "2096.4607",
284 | "92663.5609",
285 | null,
286 | "7db2329e-6446-42a8-8915-9c8370b68ed8",
287 | "2008-06-08"
288 | ],
289 | [
290 | 71784,
291 | 2,
292 | "2008-06-01",
293 | "2008-06-13",
294 | "2008-06-08",
295 | 5,
296 | false,
297 | "SO71784",
298 | "PO19285135919",
299 | "10-4020-000448",
300 | 29736,
301 | 659,
302 | 659,
303 | "CARGO TRANSPORT 5",
304 | null,
305 | "108561.8317",
306 | "8684.9465",
307 | "2714.0458",
308 | "119960.8240",
309 | null,
310 | "ca31f324-2c32-4f8d-95eb-596e7f343027",
311 | "2008-06-08"
312 | ],
313 | [
314 | 71796,
315 | 2,
316 | "2008-06-01",
317 | "2008-06-13",
318 | "2008-06-08",
319 | 5,
320 | false,
321 | "SO71796",
322 | "PO17052159664",
323 | "10-4020-000420",
324 | 29660,
325 | 1058,
326 | 1058,
327 | "CARGO TRANSPORT 5",
328 | null,
329 | "57634.6342",
330 | "4610.7707",
331 | "1440.8659",
332 | "63686.2708",
333 | null,
334 | "917ef5ba-f32d-4563-8588-66db0bcdc846",
335 | "2008-06-08"
336 | ],
337 | [
338 | 71797,
339 | 2,
340 | "2008-06-01",
341 | "2008-06-13",
342 | "2008-06-08",
343 | 5,
344 | false,
345 | "SO71797",
346 | "PO16501134889",
347 | "10-4020-000142",
348 | 29796,
349 | 642,
350 | 642,
351 | "CARGO TRANSPORT 5",
352 | null,
353 | "78029.6898",
354 | "6242.3752",
355 | "1950.7422",
356 | "86222.8072",
357 | null,
358 | "bb3fee84-c8bf-4dd2-bcca-675ab6a11c38",
359 | "2008-06-08"
360 | ],
361 | [
362 | 71815,
363 | 2,
364 | "2008-06-01",
365 | "2008-06-13",
366 | "2008-06-08",
367 | 5,
368 | false,
369 | "SO71815",
370 | "PO13021155785",
371 | "10-4020-000276",
372 | 30089,
373 | 1034,
374 | 1034,
375 | "CARGO TRANSPORT 5",
376 | null,
377 | "1141.5782",
378 | "91.3263",
379 | "28.5395",
380 | "1261.4440",
381 | null,
382 | "2aa5f39b-1096-4a4b-b17b-f10504a397ce",
383 | "2008-06-08"
384 | ],
385 | [
386 | 71816,
387 | 2,
388 | "2008-06-01",
389 | "2008-06-13",
390 | "2008-06-08",
391 | 5,
392 | false,
393 | "SO71816",
394 | "PO12992180445",
395 | "10-4020-000295",
396 | 30027,
397 | 1038,
398 | 1038,
399 | "CARGO TRANSPORT 5",
400 | null,
401 | "3398.1659",
402 | "271.8533",
403 | "84.9541",
404 | "3754.9733",
405 | null,
406 | "e3c189e7-98de-4c40-b6c2-0d1d13f9bb33",
407 | "2008-06-08"
408 | ],
409 | [
410 | 71831,
411 | 2,
412 | "2008-06-01",
413 | "2008-06-13",
414 | "2008-06-08",
415 | 5,
416 | false,
417 | "SO71831",
418 | "PO10295111084",
419 | "10-4020-000322",
420 | 30019,
421 | 652,
422 | 652,
423 | "CARGO TRANSPORT 5",
424 | null,
425 | "2016.3408",
426 | "161.3073",
427 | "50.4085",
428 | "2228.0566",
429 | null,
430 | "625d76fc-c26f-4149-bf24-939fb2bccd77",
431 | "2008-06-08"
432 | ],
433 | [
434 | 71832,
435 | 2,
436 | "2008-06-01",
437 | "2008-06-13",
438 | "2008-06-08",
439 | 5,
440 | false,
441 | "SO71832",
442 | "PO10353140756",
443 | "10-4020-000088",
444 | 29922,
445 | 639,
446 | 639,
447 | "CARGO TRANSPORT 5",
448 | null,
449 | "35775.2113",
450 | "2862.0169",
451 | "894.3803",
452 | "39531.6085",
453 | null,
454 | "addb8620-432a-456e-8470-1bedd4bc3457",
455 | "2008-06-08"
456 | ],
457 | [
458 | 71845,
459 | 2,
460 | "2008-06-01",
461 | "2008-06-13",
462 | "2008-06-08",
463 | 5,
464 | false,
465 | "SO71845",
466 | "PO2697119362",
467 | "10-4020-000187",
468 | 29938,
469 | 1020,
470 | 1020,
471 | "CARGO TRANSPORT 5",
472 | null,
473 | "41622.0511",
474 | "3329.7641",
475 | "1040.5513",
476 | "45992.3665",
477 | null,
478 | "e68f7ee9-c581-45cd-9c4f-386aeda74d84",
479 | "2008-06-08"
480 | ],
481 | [
482 | 71846,
483 | 2,
484 | "2008-06-01",
485 | "2008-06-13",
486 | "2008-06-08",
487 | 5,
488 | false,
489 | "SO71846",
490 | "PO2378131604",
491 | "10-4020-000635",
492 | 30102,
493 | 669,
494 | 669,
495 | "CARGO TRANSPORT 5",
496 | null,
497 | "2453.7645",
498 | "196.3012",
499 | "61.3441",
500 | "2711.4098",
501 | null,
502 | "a86d90ad-d1c0-440d-9a57-5b763bf18234",
503 | "2008-06-08"
504 | ],
505 | [
506 | 71856,
507 | 2,
508 | "2008-06-01",
509 | "2008-06-13",
510 | "2008-06-08",
511 | 5,
512 | false,
513 | "SO71856",
514 | "PO16530177647",
515 | "10-4020-000601",
516 | 30033,
517 | 1090,
518 | 1090,
519 | "CARGO TRANSPORT 5",
520 | null,
521 | "602.1946",
522 | "48.1756",
523 | "15.0549",
524 | "665.4251",
525 | null,
526 | "05fee073-0640-4a3c-914d-fe4ae6da3d43",
527 | "2008-06-08"
528 | ],
529 | [
530 | 71858,
531 | 2,
532 | "2008-06-01",
533 | "2008-06-13",
534 | "2008-06-08",
535 | 5,
536 | false,
537 | "SO71858",
538 | "PO16153112278",
539 | "10-4020-000186",
540 | 29653,
541 | 1019,
542 | 1019,
543 | "CARGO TRANSPORT 5",
544 | null,
545 | "13823.7083",
546 | "1105.8967",
547 | "345.5927",
548 | "15275.1977",
549 | null,
550 | "5ef091e1-a0af-437d-85ed-0b557c7923f7",
551 | "2008-06-08"
552 | ],
553 | [
554 | 71863,
555 | 2,
556 | "2008-06-01",
557 | "2008-06-13",
558 | "2008-06-08",
559 | 5,
560 | false,
561 | "SO71863",
562 | "PO16124166561",
563 | "10-4020-000649",
564 | 29975,
565 | 1098,
566 | 1098,
567 | "CARGO TRANSPORT 5",
568 | null,
569 | "3324.2759",
570 | "265.9421",
571 | "83.1069",
572 | "3673.3249",
573 | null,
574 | "3ed03b56-a4bf-4872-9471-bc6c7893eab7",
575 | "2008-06-08"
576 | ],
577 | [
578 | 71867,
579 | 2,
580 | "2008-06-01",
581 | "2008-06-13",
582 | "2008-06-08",
583 | 5,
584 | false,
585 | "SO71867",
586 | "PO13050111529",
587 | "10-4020-000160",
588 | 29644,
589 | 643,
590 | 643,
591 | "CARGO TRANSPORT 5",
592 | null,
593 | "1059.3100",
594 | "84.7448",
595 | "26.4828",
596 | "1170.5376",
597 | null,
598 | "29743c1b-d3af-4cfe-bd2e-6de436e3398f",
599 | "2008-06-08"
600 | ],
601 | [
602 | 71885,
603 | 2,
604 | "2008-06-01",
605 | "2008-06-13",
606 | "2008-06-08",
607 | 5,
608 | false,
609 | "SO71885",
610 | "PO6119130779",
611 | "10-4020-000268",
612 | 29612,
613 | 649,
614 | 649,
615 | "CARGO TRANSPORT 5",
616 | null,
617 | "550.3860",
618 | "44.0309",
619 | "13.7597",
620 | "608.1766",
621 | null,
622 | "caad090d-56a6-444e-af24-7bee7605f120",
623 | "2008-06-08"
624 | ],
625 | [
626 | 71895,
627 | 2,
628 | "2008-06-01",
629 | "2008-06-13",
630 | "2008-06-08",
631 | 5,
632 | false,
633 | "SO71895",
634 | "PO3770176273",
635 | "10-4020-000151",
636 | 29584,
637 | 1014,
638 | 1014,
639 | "CARGO TRANSPORT 5",
640 | null,
641 | "246.7392",
642 | "19.7391",
643 | "6.1685",
644 | "272.6468",
645 | null,
646 | "07a17c0a-7f9a-4413-a035-519e5a4573be",
647 | "2008-06-08"
648 | ],
649 | [
650 | 71897,
651 | 2,
652 | "2008-06-01",
653 | "2008-06-13",
654 | "2008-06-08",
655 | 5,
656 | false,
657 | "SO71897",
658 | "PO4785152479",
659 | "10-4020-000223",
660 | 29877,
661 | 1026,
662 | 1026,
663 | "CARGO TRANSPORT 5",
664 | null,
665 | "12685.8899",
666 | "1014.8712",
667 | "317.1472",
668 | "14017.9083",
669 | null,
670 | "f88b3458-e0cf-4248-9b72-1805351463b2",
671 | "2008-06-08"
672 | ],
673 | [
674 | 71898,
675 | 2,
676 | "2008-06-01",
677 | "2008-06-13",
678 | "2008-06-08",
679 | 5,
680 | false,
681 | "SO71898",
682 | "PO5713190501",
683 | "10-4020-000052",
684 | 29932,
685 | 637,
686 | 637,
687 | "CARGO TRANSPORT 5",
688 | null,
689 | "63980.9884",
690 | "5118.4791",
691 | "1599.5247",
692 | "70698.9922",
693 | null,
694 | "96f84d24-3355-43d2-b5a3-55e97c17e58c",
695 | "2008-06-08"
696 | ],
697 | [
698 | 71899,
699 | 2,
700 | "2008-06-01",
701 | "2008-06-13",
702 | "2008-06-08",
703 | 5,
704 | false,
705 | "SO71899",
706 | "PO4582142611",
707 | "10-4020-000025",
708 | 29568,
709 | 993,
710 | 993,
711 | "CARGO TRANSPORT 5",
712 | null,
713 | "2415.6727",
714 | "193.2538",
715 | "60.3918",
716 | "2669.3183",
717 | null,
718 | "31d41e8f-6f43-4cae-bee3-3cccb262f231",
719 | "2008-06-08"
720 | ],
721 | [
722 | 71902,
723 | 2,
724 | "2008-06-01",
725 | "2008-06-13",
726 | "2008-06-08",
727 | 5,
728 | false,
729 | "SO71902",
730 | "PO5539125166",
731 | "10-4020-000061",
732 | 29929,
733 | 999,
734 | 999,
735 | "CARGO TRANSPORT 5",
736 | null,
737 | "74058.8078",
738 | "5924.7046",
739 | "1851.4702",
740 | "81834.9826",
741 | null,
742 | "137850d6-efdf-4de1-920f-5757a86cdaaf",
743 | "2008-06-08"
744 | ],
745 | [
746 | 71915,
747 | 2,
748 | "2008-06-01",
749 | "2008-06-13",
750 | "2008-06-08",
751 | 5,
752 | false,
753 | "SO71915",
754 | "PO2349143275",
755 | "10-4020-000006",
756 | 29638,
757 | 989,
758 | 989,
759 | "CARGO TRANSPORT 5",
760 | null,
761 | "2137.2310",
762 | "170.9785",
763 | "53.4308",
764 | "2361.6403",
765 | null,
766 | "0bcb3d9c-9008-4b99-bb45-928adae7be6b",
767 | "2008-06-08"
768 | ],
769 | [
770 | 71917,
771 | 2,
772 | "2008-06-01",
773 | "2008-06-13",
774 | "2008-06-08",
775 | 5,
776 | false,
777 | "SO71917",
778 | "PO29111718",
779 | "10-4020-000304",
780 | 30025,
781 | 651,
782 | 651,
783 | "CARGO TRANSPORT 5",
784 | null,
785 | "40.9045",
786 | "3.2724",
787 | "1.0226",
788 | "45.1995",
789 | null,
790 | "6e903ea3-1b9e-4232-94c3-81c15669f830",
791 | "2008-06-08"
792 | ],
793 | [
794 | 71920,
795 | 2,
796 | "2008-06-01",
797 | "2008-06-13",
798 | "2008-06-08",
799 | 5,
800 | false,
801 | "SO71920",
802 | "PO4002189853",
803 | "10-4020-000674",
804 | 29982,
805 | 1102,
806 | 1102,
807 | "CARGO TRANSPORT 5",
808 | null,
809 | "2980.7929",
810 | "238.4634",
811 | "74.5198",
812 | "3293.7761",
813 | null,
814 | "6228c9cb-1cab-4e32-98ca-d0dae5fe563e",
815 | "2008-06-08"
816 | ],
817 | [
818 | 71923,
819 | 2,
820 | "2008-06-01",
821 | "2008-06-13",
822 | "2008-06-08",
823 | 5,
824 | false,
825 | "SO71923",
826 | "PO174173096",
827 | "10-4020-000277",
828 | 29781,
829 | 1035,
830 | 1035,
831 | "CARGO TRANSPORT 5",
832 | null,
833 | "106.5408",
834 | "8.5233",
835 | "2.6635",
836 | "117.7276",
837 | null,
838 | "f9899f3f-b4b6-4756-b013-96c16be20427",
839 | "2008-06-08"
840 | ],
841 | [
842 | 71935,
843 | 2,
844 | "2008-06-01",
845 | "2008-06-13",
846 | "2008-06-08",
847 | 5,
848 | false,
849 | "SO71935",
850 | "PO7946145876",
851 | "10-4020-000438",
852 | 29531,
853 | 1061,
854 | 1061,
855 | "CARGO TRANSPORT 5",
856 | null,
857 | "6634.2961",
858 | "530.7437",
859 | "165.8574",
860 | "7330.8972",
861 | null,
862 | "7033c6ec-b12c-45bc-bd96-56efde4c7dd0",
863 | "2008-06-08"
864 | ],
865 | [
866 | 71936,
867 | 2,
868 | "2008-06-01",
869 | "2008-06-13",
870 | "2008-06-08",
871 | 5,
872 | false,
873 | "SO71936",
874 | "PO8671170385",
875 | "10-4020-000502",
876 | 30050,
877 | 662,
878 | 662,
879 | "CARGO TRANSPORT 5",
880 | null,
881 | "98278.6910",
882 | "7862.2953",
883 | "2456.9673",
884 | "108597.9536",
885 | null,
886 | "119db56a-a97e-414d-b41c-64886fc50ab7",
887 | "2008-06-08"
888 | ],
889 | [
890 | 71938,
891 | 2,
892 | "2008-06-01",
893 | "2008-06-13",
894 | "2008-06-08",
895 | 5,
896 | false,
897 | "SO71938",
898 | "PO8468183315",
899 | "10-4020-000016",
900 | 29546,
901 | 635,
902 | 635,
903 | "CARGO TRANSPORT 5",
904 | null,
905 | "88812.8625",
906 | "7105.0290",
907 | "2220.3216",
908 | "98138.2131",
909 | null,
910 | "a36ee74a-cf0d-4024-a1ce-4eaffd1f85f0",
911 | "2008-06-08"
912 | ],
913 | [
914 | 71946,
915 | 2,
916 | "2008-06-01",
917 | "2008-06-13",
918 | "2008-06-08",
919 | 5,
920 | false,
921 | "SO71946",
922 | "PO8961158629",
923 | "10-4020-000466",
924 | 29741,
925 | 660,
926 | 660,
927 | "CARGO TRANSPORT 5",
928 | null,
929 | "38.9536",
930 | "3.1163",
931 | "0.9738",
932 | "43.0437",
933 | null,
934 | "10e3129d-657f-46f9-86f5-cedd79b1901c",
935 | "2008-06-08"
936 | ]
937 | ],
938 | "datasetInfos": [],
939 | "dbfsResultPath": null,
940 | "isJsonSchema": true,
941 | "metadata": {},
942 | "overflow": false,
943 | "plotOptions": {
944 | "customPlotOptions": {},
945 | "displayType": "table",
946 | "pivotAggregation": null,
947 | "pivotColumns": null,
948 | "xColumns": null,
949 | "yColumns": null
950 | },
951 | "removedWidgets": [],
952 | "schema": [
953 | {
954 | "metadata": "{}",
955 | "name": "SalesOrderID",
956 | "type": "\"integer\""
957 | },
958 | {
959 | "metadata": "{}",
960 | "name": "RevisionNumber",
961 | "type": "\"integer\""
962 | },
963 | {
964 | "metadata": "{}",
965 | "name": "OrderDate",
966 | "type": "\"string\""
967 | },
968 | {
969 | "metadata": "{}",
970 | "name": "DueDate",
971 | "type": "\"string\""
972 | },
973 | {
974 | "metadata": "{}",
975 | "name": "ShipDate",
976 | "type": "\"string\""
977 | },
978 | {
979 | "metadata": "{}",
980 | "name": "Status",
981 | "type": "\"integer\""
982 | },
983 | {
984 | "metadata": "{}",
985 | "name": "OnlineOrderFlag",
986 | "type": "\"boolean\""
987 | },
988 | {
989 | "metadata": "{}",
990 | "name": "SalesOrderNumber",
991 | "type": "\"string\""
992 | },
993 | {
994 | "metadata": "{}",
995 | "name": "PurchaseOrderNumber",
996 | "type": "\"string\""
997 | },
998 | {
999 | "metadata": "{}",
1000 | "name": "AccountNumber",
1001 | "type": "\"string\""
1002 | },
1003 | {
1004 | "metadata": "{}",
1005 | "name": "CustomerID",
1006 | "type": "\"integer\""
1007 | },
1008 | {
1009 | "metadata": "{}",
1010 | "name": "ShipToAddressID",
1011 | "type": "\"integer\""
1012 | },
1013 | {
1014 | "metadata": "{}",
1015 | "name": "BillToAddressID",
1016 | "type": "\"integer\""
1017 | },
1018 | {
1019 | "metadata": "{}",
1020 | "name": "ShipMethod",
1021 | "type": "\"string\""
1022 | },
1023 | {
1024 | "metadata": "{}",
1025 | "name": "CreditCardApprovalCode",
1026 | "type": "\"string\""
1027 | },
1028 | {
1029 | "metadata": "{}",
1030 | "name": "SubTotal",
1031 | "type": "\"decimal(19,4)\""
1032 | },
1033 | {
1034 | "metadata": "{}",
1035 | "name": "TaxAmt",
1036 | "type": "\"decimal(19,4)\""
1037 | },
1038 | {
1039 | "metadata": "{}",
1040 | "name": "Freight",
1041 | "type": "\"decimal(19,4)\""
1042 | },
1043 | {
1044 | "metadata": "{}",
1045 | "name": "TotalDue",
1046 | "type": "\"decimal(19,4)\""
1047 | },
1048 | {
1049 | "metadata": "{}",
1050 | "name": "Comment",
1051 | "type": "\"string\""
1052 | },
1053 | {
1054 | "metadata": "{}",
1055 | "name": "rowguid",
1056 | "type": "\"string\""
1057 | },
1058 | {
1059 | "metadata": "{}",
1060 | "name": "ModifiedDate",
1061 | "type": "\"string\""
1062 | }
1063 | ],
1064 | "type": "table"
1065 | }
1066 | },
1067 | "output_type": "display_data"
1068 | }
1069 | ],
1070 | "source": [
1071 | "display(df)"
1072 | ]
1073 | }
1074 | ],
1075 | "metadata": {
1076 | "application/vnd.databricks.v1+notebook": {
1077 | "dashboards": [],
1078 | "language": "python",
1079 | "notebookMetadata": {
1080 | "pythonIndentUnit": 4
1081 | },
1082 | "notebookName": "bronze to silver",
1083 | "widgets": {}
1084 | }
1085 | },
1086 | "nbformat": 4,
1087 | "nbformat_minor": 0
1088 | }
1089 |
--------------------------------------------------------------------------------
/databricks_notebooks/init:
--------------------------------------------------------------------------------
1 | init
2 |
--------------------------------------------------------------------------------
/databricks_notebooks/silver to gold.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {
6 | "application/vnd.databricks.v1+cell": {
7 | "cellMetadata": {
8 | "byteLimit": 2048000,
9 | "rowLimit": 10000
10 | },
11 | "inputWidgets": {},
12 | "nuid": "e84c921d-163d-402c-81f4-06baa137a8b1",
13 | "showTitle": false,
14 | "title": ""
15 | }
16 | },
17 | "source": [
18 | "\n",
19 | "## Transforming all the column names of tables"
20 | ]
21 | },
22 | {
23 | "cell_type": "code",
24 | "execution_count": 0,
25 | "metadata": {
26 | "application/vnd.databricks.v1+cell": {
27 | "cellMetadata": {
28 | "byteLimit": 2048000,
29 | "rowLimit": 10000
30 | },
31 | "inputWidgets": {},
32 | "nuid": "0314b13e-098a-4ad7-89f8-2d6ab766a3ae",
33 | "showTitle": false,
34 | "title": ""
35 | }
36 | },
37 | "outputs": [],
38 | "source": [
39 | "table_name = []\n",
40 | "\n",
41 | "for i in dbutils.fs.ls('mnt/silver/SalesLT/'):\n",
42 | " table_name.append(i.name.split('/')[0])"
43 | ]
44 | },
45 | {
46 | "cell_type": "code",
47 | "execution_count": 0,
48 | "metadata": {
49 | "application/vnd.databricks.v1+cell": {
50 | "cellMetadata": {
51 | "byteLimit": 2048000,
52 | "rowLimit": 10000
53 | },
54 | "inputWidgets": {},
55 | "nuid": "3e116833-dcfd-4451-9d02-b7ca3c400b67",
56 | "showTitle": false,
57 | "title": ""
58 | }
59 | },
60 | "outputs": [
61 | {
62 | "output_type": "execute_result",
63 | "data": {
64 | "text/plain": [
65 | "['Address',\n",
66 | " 'Customer',\n",
67 | " 'CustomerAddress',\n",
68 | " 'Product',\n",
69 | " 'ProductCategory',\n",
70 | " 'ProductDescription',\n",
71 | " 'ProductModel',\n",
72 | " 'ProductModelProductDescription',\n",
73 | " 'SalesOrderDetail',\n",
74 | " 'SalesOrderHeader']"
75 | ]
76 | },
77 | "execution_count": 39,
78 | "metadata": {},
79 | "output_type": "execute_result"
80 | }
81 | ],
82 | "source": [
83 | "table_name"
84 | ]
85 | },
86 | {
87 | "cell_type": "code",
88 | "execution_count": 0,
89 | "metadata": {
90 | "application/vnd.databricks.v1+cell": {
91 | "cellMetadata": {
92 | "byteLimit": 2048000,
93 | "rowLimit": 10000
94 | },
95 | "inputWidgets": {},
96 | "nuid": "312d0f50-5b72-4053-bdec-66062f800dd0",
97 | "showTitle": false,
98 | "title": ""
99 | }
100 | },
101 | "outputs": [
102 | {
103 | "output_type": "stream",
104 | "name": "stdout",
105 | "output_type": "stream",
106 | "text": [
107 | "/mnt/silver/SalesLT/Address\n/mnt/silver/SalesLT/Customer\n/mnt/silver/SalesLT/CustomerAddress\n/mnt/silver/SalesLT/Product\n/mnt/silver/SalesLT/ProductCategory\n/mnt/silver/SalesLT/ProductDescription\n/mnt/silver/SalesLT/ProductModel\n/mnt/silver/SalesLT/ProductModelProductDescription\n/mnt/silver/SalesLT/SalesOrderDetail\n/mnt/silver/SalesLT/SalesOrderHeader\n"
108 | ]
109 | }
110 | ],
111 | "source": [
112 | "for name in table_name:\n",
113 | " path = '/mnt/silver/SalesLT/' + name \n",
114 | " print(path)\n",
115 | " df = spark.read.format('delta').load(path)\n",
116 | "\n",
117 | " column_names = df.columns\n",
118 | "\n",
119 | " for old_col_name in column_names:\n",
120 | " new_col_name = \"\".join([\"_\" + char if char.isupper() and not old_col_name[i - 1].isupper() else char for i, char in enumerate(old_col_name)]).lstrip(\"_\")\n",
121 | "\n",
122 | " df = df.withColumnRenamed(old_col_name, new_col_name)\n",
123 | " \n",
124 | " output_path = '/mnt/gold/SalesLT/' +name +'/'\n",
125 | " df.write.format('delta').mode(\"overwrite\").save(output_path)"
126 | ]
127 | },
128 | {
129 | "cell_type": "code",
130 | "execution_count": 0,
131 | "metadata": {
132 | "application/vnd.databricks.v1+cell": {
133 | "cellMetadata": {
134 | "byteLimit": 2048000,
135 | "rowLimit": 10000
136 | },
137 | "inputWidgets": {},
138 | "nuid": "e54d6a66-4051-485a-adee-7ff167b902b1",
139 | "showTitle": false,
140 | "title": ""
141 | }
142 | },
143 | "outputs": [
144 | {
145 | "output_type": "display_data",
146 | "data": {
147 | "text/html": [
148 | "Sales_Order_ID | Revision_Number | Order_Date | Due_Date | Ship_Date | Status | Online_Order_Flag | Sales_Order_Number | Purchase_Order_Number | Account_Number | Customer_ID | Ship_To_Address_ID | Bill_To_Address_ID | Ship_Method | Credit_Card_Approval_Code | Sub_Total | Tax_Amt | Freight | Total_Due | Comment | rowguid | Modified_Date |
---|
71774 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71774 | PO348186287 | 10-4020-000609 | 29847 | 1092 | 1092 | CARGO TRANSPORT 5 | null | 880.3484 | 70.4279 | 22.0087 | 972.7850 | null | 89e42cdc-8506-48a2-b89b-eb3e64e3554e | 2008-06-08 |
71776 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71776 | PO19952192051 | 10-4020-000106 | 30072 | 640 | 640 | CARGO TRANSPORT 5 | null | 78.8100 | 6.3048 | 1.9703 | 87.0851 | null | 8a3448c5-e677-4158-a29b-dd33069be0b0 | 2008-06-08 |
71780 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71780 | PO19604173239 | 10-4020-000340 | 30113 | 653 | 653 | CARGO TRANSPORT 5 | null | 38418.6895 | 3073.4952 | 960.4672 | 42452.6519 | null | a47665d2-7ac9-4cf3-8a8b-2a3883554284 | 2008-06-08 |
71782 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71782 | PO19372114749 | 10-4020-000582 | 29485 | 1086 | 1086 | CARGO TRANSPORT 5 | null | 39785.3304 | 3182.8264 | 994.6333 | 43962.7901 | null | f1be45a5-5c57-4a50-93c6-5f8be44cb7cb | 2008-06-08 |
71783 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71783 | PO19343113609 | 10-4020-000024 | 29957 | 992 | 992 | CARGO TRANSPORT 5 | null | 83858.4261 | 6708.6741 | 2096.4607 | 92663.5609 | null | 7db2329e-6446-42a8-8915-9c8370b68ed8 | 2008-06-08 |
71784 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71784 | PO19285135919 | 10-4020-000448 | 29736 | 659 | 659 | CARGO TRANSPORT 5 | null | 108561.8317 | 8684.9465 | 2714.0458 | 119960.8240 | null | ca31f324-2c32-4f8d-95eb-596e7f343027 | 2008-06-08 |
71796 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71796 | PO17052159664 | 10-4020-000420 | 29660 | 1058 | 1058 | CARGO TRANSPORT 5 | null | 57634.6342 | 4610.7707 | 1440.8659 | 63686.2708 | null | 917ef5ba-f32d-4563-8588-66db0bcdc846 | 2008-06-08 |
71797 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71797 | PO16501134889 | 10-4020-000142 | 29796 | 642 | 642 | CARGO TRANSPORT 5 | null | 78029.6898 | 6242.3752 | 1950.7422 | 86222.8072 | null | bb3fee84-c8bf-4dd2-bcca-675ab6a11c38 | 2008-06-08 |
71815 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71815 | PO13021155785 | 10-4020-000276 | 30089 | 1034 | 1034 | CARGO TRANSPORT 5 | null | 1141.5782 | 91.3263 | 28.5395 | 1261.4440 | null | 2aa5f39b-1096-4a4b-b17b-f10504a397ce | 2008-06-08 |
71816 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71816 | PO12992180445 | 10-4020-000295 | 30027 | 1038 | 1038 | CARGO TRANSPORT 5 | null | 3398.1659 | 271.8533 | 84.9541 | 3754.9733 | null | e3c189e7-98de-4c40-b6c2-0d1d13f9bb33 | 2008-06-08 |
71831 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71831 | PO10295111084 | 10-4020-000322 | 30019 | 652 | 652 | CARGO TRANSPORT 5 | null | 2016.3408 | 161.3073 | 50.4085 | 2228.0566 | null | 625d76fc-c26f-4149-bf24-939fb2bccd77 | 2008-06-08 |
71832 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71832 | PO10353140756 | 10-4020-000088 | 29922 | 639 | 639 | CARGO TRANSPORT 5 | null | 35775.2113 | 2862.0169 | 894.3803 | 39531.6085 | null | addb8620-432a-456e-8470-1bedd4bc3457 | 2008-06-08 |
71845 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71845 | PO2697119362 | 10-4020-000187 | 29938 | 1020 | 1020 | CARGO TRANSPORT 5 | null | 41622.0511 | 3329.7641 | 1040.5513 | 45992.3665 | null | e68f7ee9-c581-45cd-9c4f-386aeda74d84 | 2008-06-08 |
71846 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71846 | PO2378131604 | 10-4020-000635 | 30102 | 669 | 669 | CARGO TRANSPORT 5 | null | 2453.7645 | 196.3012 | 61.3441 | 2711.4098 | null | a86d90ad-d1c0-440d-9a57-5b763bf18234 | 2008-06-08 |
71856 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71856 | PO16530177647 | 10-4020-000601 | 30033 | 1090 | 1090 | CARGO TRANSPORT 5 | null | 602.1946 | 48.1756 | 15.0549 | 665.4251 | null | 05fee073-0640-4a3c-914d-fe4ae6da3d43 | 2008-06-08 |
71858 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71858 | PO16153112278 | 10-4020-000186 | 29653 | 1019 | 1019 | CARGO TRANSPORT 5 | null | 13823.7083 | 1105.8967 | 345.5927 | 15275.1977 | null | 5ef091e1-a0af-437d-85ed-0b557c7923f7 | 2008-06-08 |
71863 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71863 | PO16124166561 | 10-4020-000649 | 29975 | 1098 | 1098 | CARGO TRANSPORT 5 | null | 3324.2759 | 265.9421 | 83.1069 | 3673.3249 | null | 3ed03b56-a4bf-4872-9471-bc6c7893eab7 | 2008-06-08 |
71867 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71867 | PO13050111529 | 10-4020-000160 | 29644 | 643 | 643 | CARGO TRANSPORT 5 | null | 1059.3100 | 84.7448 | 26.4828 | 1170.5376 | null | 29743c1b-d3af-4cfe-bd2e-6de436e3398f | 2008-06-08 |
71885 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71885 | PO6119130779 | 10-4020-000268 | 29612 | 649 | 649 | CARGO TRANSPORT 5 | null | 550.3860 | 44.0309 | 13.7597 | 608.1766 | null | caad090d-56a6-444e-af24-7bee7605f120 | 2008-06-08 |
71895 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71895 | PO3770176273 | 10-4020-000151 | 29584 | 1014 | 1014 | CARGO TRANSPORT 5 | null | 246.7392 | 19.7391 | 6.1685 | 272.6468 | null | 07a17c0a-7f9a-4413-a035-519e5a4573be | 2008-06-08 |
71897 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71897 | PO4785152479 | 10-4020-000223 | 29877 | 1026 | 1026 | CARGO TRANSPORT 5 | null | 12685.8899 | 1014.8712 | 317.1472 | 14017.9083 | null | f88b3458-e0cf-4248-9b72-1805351463b2 | 2008-06-08 |
71898 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71898 | PO5713190501 | 10-4020-000052 | 29932 | 637 | 637 | CARGO TRANSPORT 5 | null | 63980.9884 | 5118.4791 | 1599.5247 | 70698.9922 | null | 96f84d24-3355-43d2-b5a3-55e97c17e58c | 2008-06-08 |
71899 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71899 | PO4582142611 | 10-4020-000025 | 29568 | 993 | 993 | CARGO TRANSPORT 5 | null | 2415.6727 | 193.2538 | 60.3918 | 2669.3183 | null | 31d41e8f-6f43-4cae-bee3-3cccb262f231 | 2008-06-08 |
71902 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71902 | PO5539125166 | 10-4020-000061 | 29929 | 999 | 999 | CARGO TRANSPORT 5 | null | 74058.8078 | 5924.7046 | 1851.4702 | 81834.9826 | null | 137850d6-efdf-4de1-920f-5757a86cdaaf | 2008-06-08 |
71915 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71915 | PO2349143275 | 10-4020-000006 | 29638 | 989 | 989 | CARGO TRANSPORT 5 | null | 2137.2310 | 170.9785 | 53.4308 | 2361.6403 | null | 0bcb3d9c-9008-4b99-bb45-928adae7be6b | 2008-06-08 |
71917 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71917 | PO29111718 | 10-4020-000304 | 30025 | 651 | 651 | CARGO TRANSPORT 5 | null | 40.9045 | 3.2724 | 1.0226 | 45.1995 | null | 6e903ea3-1b9e-4232-94c3-81c15669f830 | 2008-06-08 |
71920 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71920 | PO4002189853 | 10-4020-000674 | 29982 | 1102 | 1102 | CARGO TRANSPORT 5 | null | 2980.7929 | 238.4634 | 74.5198 | 3293.7761 | null | 6228c9cb-1cab-4e32-98ca-d0dae5fe563e | 2008-06-08 |
71923 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71923 | PO174173096 | 10-4020-000277 | 29781 | 1035 | 1035 | CARGO TRANSPORT 5 | null | 106.5408 | 8.5233 | 2.6635 | 117.7276 | null | f9899f3f-b4b6-4756-b013-96c16be20427 | 2008-06-08 |
71935 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71935 | PO7946145876 | 10-4020-000438 | 29531 | 1061 | 1061 | CARGO TRANSPORT 5 | null | 6634.2961 | 530.7437 | 165.8574 | 7330.8972 | null | 7033c6ec-b12c-45bc-bd96-56efde4c7dd0 | 2008-06-08 |
71936 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71936 | PO8671170385 | 10-4020-000502 | 30050 | 662 | 662 | CARGO TRANSPORT 5 | null | 98278.6910 | 7862.2953 | 2456.9673 | 108597.9536 | null | 119db56a-a97e-414d-b41c-64886fc50ab7 | 2008-06-08 |
71938 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71938 | PO8468183315 | 10-4020-000016 | 29546 | 635 | 635 | CARGO TRANSPORT 5 | null | 88812.8625 | 7105.0290 | 2220.3216 | 98138.2131 | null | a36ee74a-cf0d-4024-a1ce-4eaffd1f85f0 | 2008-06-08 |
71946 | 2 | 2008-06-01 | 2008-06-13 | 2008-06-08 | 5 | false | SO71946 | PO8961158629 | 10-4020-000466 | 29741 | 660 | 660 | CARGO TRANSPORT 5 | null | 38.9536 | 3.1163 | 0.9738 | 43.0437 | null | 10e3129d-657f-46f9-86f5-cedd79b1901c | 2008-06-08 |
"
164 | ]
165 | },
166 | "metadata": {
167 | "application/vnd.databricks.v1+output": {
168 | "addedWidgets": {},
169 | "aggData": [],
170 | "aggError": "",
171 | "aggOverflow": false,
172 | "aggSchema": [],
173 | "aggSeriesLimitReached": false,
174 | "aggType": "",
175 | "arguments": {},
176 | "columnCustomDisplayInfos": {},
177 | "data": [
178 | [
179 | 71774,
180 | 2,
181 | "2008-06-01",
182 | "2008-06-13",
183 | "2008-06-08",
184 | 5,
185 | false,
186 | "SO71774",
187 | "PO348186287",
188 | "10-4020-000609",
189 | 29847,
190 | 1092,
191 | 1092,
192 | "CARGO TRANSPORT 5",
193 | null,
194 | "880.3484",
195 | "70.4279",
196 | "22.0087",
197 | "972.7850",
198 | null,
199 | "89e42cdc-8506-48a2-b89b-eb3e64e3554e",
200 | "2008-06-08"
201 | ],
202 | [
203 | 71776,
204 | 2,
205 | "2008-06-01",
206 | "2008-06-13",
207 | "2008-06-08",
208 | 5,
209 | false,
210 | "SO71776",
211 | "PO19952192051",
212 | "10-4020-000106",
213 | 30072,
214 | 640,
215 | 640,
216 | "CARGO TRANSPORT 5",
217 | null,
218 | "78.8100",
219 | "6.3048",
220 | "1.9703",
221 | "87.0851",
222 | null,
223 | "8a3448c5-e677-4158-a29b-dd33069be0b0",
224 | "2008-06-08"
225 | ],
226 | [
227 | 71780,
228 | 2,
229 | "2008-06-01",
230 | "2008-06-13",
231 | "2008-06-08",
232 | 5,
233 | false,
234 | "SO71780",
235 | "PO19604173239",
236 | "10-4020-000340",
237 | 30113,
238 | 653,
239 | 653,
240 | "CARGO TRANSPORT 5",
241 | null,
242 | "38418.6895",
243 | "3073.4952",
244 | "960.4672",
245 | "42452.6519",
246 | null,
247 | "a47665d2-7ac9-4cf3-8a8b-2a3883554284",
248 | "2008-06-08"
249 | ],
250 | [
251 | 71782,
252 | 2,
253 | "2008-06-01",
254 | "2008-06-13",
255 | "2008-06-08",
256 | 5,
257 | false,
258 | "SO71782",
259 | "PO19372114749",
260 | "10-4020-000582",
261 | 29485,
262 | 1086,
263 | 1086,
264 | "CARGO TRANSPORT 5",
265 | null,
266 | "39785.3304",
267 | "3182.8264",
268 | "994.6333",
269 | "43962.7901",
270 | null,
271 | "f1be45a5-5c57-4a50-93c6-5f8be44cb7cb",
272 | "2008-06-08"
273 | ],
274 | [
275 | 71783,
276 | 2,
277 | "2008-06-01",
278 | "2008-06-13",
279 | "2008-06-08",
280 | 5,
281 | false,
282 | "SO71783",
283 | "PO19343113609",
284 | "10-4020-000024",
285 | 29957,
286 | 992,
287 | 992,
288 | "CARGO TRANSPORT 5",
289 | null,
290 | "83858.4261",
291 | "6708.6741",
292 | "2096.4607",
293 | "92663.5609",
294 | null,
295 | "7db2329e-6446-42a8-8915-9c8370b68ed8",
296 | "2008-06-08"
297 | ],
298 | [
299 | 71784,
300 | 2,
301 | "2008-06-01",
302 | "2008-06-13",
303 | "2008-06-08",
304 | 5,
305 | false,
306 | "SO71784",
307 | "PO19285135919",
308 | "10-4020-000448",
309 | 29736,
310 | 659,
311 | 659,
312 | "CARGO TRANSPORT 5",
313 | null,
314 | "108561.8317",
315 | "8684.9465",
316 | "2714.0458",
317 | "119960.8240",
318 | null,
319 | "ca31f324-2c32-4f8d-95eb-596e7f343027",
320 | "2008-06-08"
321 | ],
322 | [
323 | 71796,
324 | 2,
325 | "2008-06-01",
326 | "2008-06-13",
327 | "2008-06-08",
328 | 5,
329 | false,
330 | "SO71796",
331 | "PO17052159664",
332 | "10-4020-000420",
333 | 29660,
334 | 1058,
335 | 1058,
336 | "CARGO TRANSPORT 5",
337 | null,
338 | "57634.6342",
339 | "4610.7707",
340 | "1440.8659",
341 | "63686.2708",
342 | null,
343 | "917ef5ba-f32d-4563-8588-66db0bcdc846",
344 | "2008-06-08"
345 | ],
346 | [
347 | 71797,
348 | 2,
349 | "2008-06-01",
350 | "2008-06-13",
351 | "2008-06-08",
352 | 5,
353 | false,
354 | "SO71797",
355 | "PO16501134889",
356 | "10-4020-000142",
357 | 29796,
358 | 642,
359 | 642,
360 | "CARGO TRANSPORT 5",
361 | null,
362 | "78029.6898",
363 | "6242.3752",
364 | "1950.7422",
365 | "86222.8072",
366 | null,
367 | "bb3fee84-c8bf-4dd2-bcca-675ab6a11c38",
368 | "2008-06-08"
369 | ],
370 | [
371 | 71815,
372 | 2,
373 | "2008-06-01",
374 | "2008-06-13",
375 | "2008-06-08",
376 | 5,
377 | false,
378 | "SO71815",
379 | "PO13021155785",
380 | "10-4020-000276",
381 | 30089,
382 | 1034,
383 | 1034,
384 | "CARGO TRANSPORT 5",
385 | null,
386 | "1141.5782",
387 | "91.3263",
388 | "28.5395",
389 | "1261.4440",
390 | null,
391 | "2aa5f39b-1096-4a4b-b17b-f10504a397ce",
392 | "2008-06-08"
393 | ],
394 | [
395 | 71816,
396 | 2,
397 | "2008-06-01",
398 | "2008-06-13",
399 | "2008-06-08",
400 | 5,
401 | false,
402 | "SO71816",
403 | "PO12992180445",
404 | "10-4020-000295",
405 | 30027,
406 | 1038,
407 | 1038,
408 | "CARGO TRANSPORT 5",
409 | null,
410 | "3398.1659",
411 | "271.8533",
412 | "84.9541",
413 | "3754.9733",
414 | null,
415 | "e3c189e7-98de-4c40-b6c2-0d1d13f9bb33",
416 | "2008-06-08"
417 | ],
418 | [
419 | 71831,
420 | 2,
421 | "2008-06-01",
422 | "2008-06-13",
423 | "2008-06-08",
424 | 5,
425 | false,
426 | "SO71831",
427 | "PO10295111084",
428 | "10-4020-000322",
429 | 30019,
430 | 652,
431 | 652,
432 | "CARGO TRANSPORT 5",
433 | null,
434 | "2016.3408",
435 | "161.3073",
436 | "50.4085",
437 | "2228.0566",
438 | null,
439 | "625d76fc-c26f-4149-bf24-939fb2bccd77",
440 | "2008-06-08"
441 | ],
442 | [
443 | 71832,
444 | 2,
445 | "2008-06-01",
446 | "2008-06-13",
447 | "2008-06-08",
448 | 5,
449 | false,
450 | "SO71832",
451 | "PO10353140756",
452 | "10-4020-000088",
453 | 29922,
454 | 639,
455 | 639,
456 | "CARGO TRANSPORT 5",
457 | null,
458 | "35775.2113",
459 | "2862.0169",
460 | "894.3803",
461 | "39531.6085",
462 | null,
463 | "addb8620-432a-456e-8470-1bedd4bc3457",
464 | "2008-06-08"
465 | ],
466 | [
467 | 71845,
468 | 2,
469 | "2008-06-01",
470 | "2008-06-13",
471 | "2008-06-08",
472 | 5,
473 | false,
474 | "SO71845",
475 | "PO2697119362",
476 | "10-4020-000187",
477 | 29938,
478 | 1020,
479 | 1020,
480 | "CARGO TRANSPORT 5",
481 | null,
482 | "41622.0511",
483 | "3329.7641",
484 | "1040.5513",
485 | "45992.3665",
486 | null,
487 | "e68f7ee9-c581-45cd-9c4f-386aeda74d84",
488 | "2008-06-08"
489 | ],
490 | [
491 | 71846,
492 | 2,
493 | "2008-06-01",
494 | "2008-06-13",
495 | "2008-06-08",
496 | 5,
497 | false,
498 | "SO71846",
499 | "PO2378131604",
500 | "10-4020-000635",
501 | 30102,
502 | 669,
503 | 669,
504 | "CARGO TRANSPORT 5",
505 | null,
506 | "2453.7645",
507 | "196.3012",
508 | "61.3441",
509 | "2711.4098",
510 | null,
511 | "a86d90ad-d1c0-440d-9a57-5b763bf18234",
512 | "2008-06-08"
513 | ],
514 | [
515 | 71856,
516 | 2,
517 | "2008-06-01",
518 | "2008-06-13",
519 | "2008-06-08",
520 | 5,
521 | false,
522 | "SO71856",
523 | "PO16530177647",
524 | "10-4020-000601",
525 | 30033,
526 | 1090,
527 | 1090,
528 | "CARGO TRANSPORT 5",
529 | null,
530 | "602.1946",
531 | "48.1756",
532 | "15.0549",
533 | "665.4251",
534 | null,
535 | "05fee073-0640-4a3c-914d-fe4ae6da3d43",
536 | "2008-06-08"
537 | ],
538 | [
539 | 71858,
540 | 2,
541 | "2008-06-01",
542 | "2008-06-13",
543 | "2008-06-08",
544 | 5,
545 | false,
546 | "SO71858",
547 | "PO16153112278",
548 | "10-4020-000186",
549 | 29653,
550 | 1019,
551 | 1019,
552 | "CARGO TRANSPORT 5",
553 | null,
554 | "13823.7083",
555 | "1105.8967",
556 | "345.5927",
557 | "15275.1977",
558 | null,
559 | "5ef091e1-a0af-437d-85ed-0b557c7923f7",
560 | "2008-06-08"
561 | ],
562 | [
563 | 71863,
564 | 2,
565 | "2008-06-01",
566 | "2008-06-13",
567 | "2008-06-08",
568 | 5,
569 | false,
570 | "SO71863",
571 | "PO16124166561",
572 | "10-4020-000649",
573 | 29975,
574 | 1098,
575 | 1098,
576 | "CARGO TRANSPORT 5",
577 | null,
578 | "3324.2759",
579 | "265.9421",
580 | "83.1069",
581 | "3673.3249",
582 | null,
583 | "3ed03b56-a4bf-4872-9471-bc6c7893eab7",
584 | "2008-06-08"
585 | ],
586 | [
587 | 71867,
588 | 2,
589 | "2008-06-01",
590 | "2008-06-13",
591 | "2008-06-08",
592 | 5,
593 | false,
594 | "SO71867",
595 | "PO13050111529",
596 | "10-4020-000160",
597 | 29644,
598 | 643,
599 | 643,
600 | "CARGO TRANSPORT 5",
601 | null,
602 | "1059.3100",
603 | "84.7448",
604 | "26.4828",
605 | "1170.5376",
606 | null,
607 | "29743c1b-d3af-4cfe-bd2e-6de436e3398f",
608 | "2008-06-08"
609 | ],
610 | [
611 | 71885,
612 | 2,
613 | "2008-06-01",
614 | "2008-06-13",
615 | "2008-06-08",
616 | 5,
617 | false,
618 | "SO71885",
619 | "PO6119130779",
620 | "10-4020-000268",
621 | 29612,
622 | 649,
623 | 649,
624 | "CARGO TRANSPORT 5",
625 | null,
626 | "550.3860",
627 | "44.0309",
628 | "13.7597",
629 | "608.1766",
630 | null,
631 | "caad090d-56a6-444e-af24-7bee7605f120",
632 | "2008-06-08"
633 | ],
634 | [
635 | 71895,
636 | 2,
637 | "2008-06-01",
638 | "2008-06-13",
639 | "2008-06-08",
640 | 5,
641 | false,
642 | "SO71895",
643 | "PO3770176273",
644 | "10-4020-000151",
645 | 29584,
646 | 1014,
647 | 1014,
648 | "CARGO TRANSPORT 5",
649 | null,
650 | "246.7392",
651 | "19.7391",
652 | "6.1685",
653 | "272.6468",
654 | null,
655 | "07a17c0a-7f9a-4413-a035-519e5a4573be",
656 | "2008-06-08"
657 | ],
658 | [
659 | 71897,
660 | 2,
661 | "2008-06-01",
662 | "2008-06-13",
663 | "2008-06-08",
664 | 5,
665 | false,
666 | "SO71897",
667 | "PO4785152479",
668 | "10-4020-000223",
669 | 29877,
670 | 1026,
671 | 1026,
672 | "CARGO TRANSPORT 5",
673 | null,
674 | "12685.8899",
675 | "1014.8712",
676 | "317.1472",
677 | "14017.9083",
678 | null,
679 | "f88b3458-e0cf-4248-9b72-1805351463b2",
680 | "2008-06-08"
681 | ],
682 | [
683 | 71898,
684 | 2,
685 | "2008-06-01",
686 | "2008-06-13",
687 | "2008-06-08",
688 | 5,
689 | false,
690 | "SO71898",
691 | "PO5713190501",
692 | "10-4020-000052",
693 | 29932,
694 | 637,
695 | 637,
696 | "CARGO TRANSPORT 5",
697 | null,
698 | "63980.9884",
699 | "5118.4791",
700 | "1599.5247",
701 | "70698.9922",
702 | null,
703 | "96f84d24-3355-43d2-b5a3-55e97c17e58c",
704 | "2008-06-08"
705 | ],
706 | [
707 | 71899,
708 | 2,
709 | "2008-06-01",
710 | "2008-06-13",
711 | "2008-06-08",
712 | 5,
713 | false,
714 | "SO71899",
715 | "PO4582142611",
716 | "10-4020-000025",
717 | 29568,
718 | 993,
719 | 993,
720 | "CARGO TRANSPORT 5",
721 | null,
722 | "2415.6727",
723 | "193.2538",
724 | "60.3918",
725 | "2669.3183",
726 | null,
727 | "31d41e8f-6f43-4cae-bee3-3cccb262f231",
728 | "2008-06-08"
729 | ],
730 | [
731 | 71902,
732 | 2,
733 | "2008-06-01",
734 | "2008-06-13",
735 | "2008-06-08",
736 | 5,
737 | false,
738 | "SO71902",
739 | "PO5539125166",
740 | "10-4020-000061",
741 | 29929,
742 | 999,
743 | 999,
744 | "CARGO TRANSPORT 5",
745 | null,
746 | "74058.8078",
747 | "5924.7046",
748 | "1851.4702",
749 | "81834.9826",
750 | null,
751 | "137850d6-efdf-4de1-920f-5757a86cdaaf",
752 | "2008-06-08"
753 | ],
754 | [
755 | 71915,
756 | 2,
757 | "2008-06-01",
758 | "2008-06-13",
759 | "2008-06-08",
760 | 5,
761 | false,
762 | "SO71915",
763 | "PO2349143275",
764 | "10-4020-000006",
765 | 29638,
766 | 989,
767 | 989,
768 | "CARGO TRANSPORT 5",
769 | null,
770 | "2137.2310",
771 | "170.9785",
772 | "53.4308",
773 | "2361.6403",
774 | null,
775 | "0bcb3d9c-9008-4b99-bb45-928adae7be6b",
776 | "2008-06-08"
777 | ],
778 | [
779 | 71917,
780 | 2,
781 | "2008-06-01",
782 | "2008-06-13",
783 | "2008-06-08",
784 | 5,
785 | false,
786 | "SO71917",
787 | "PO29111718",
788 | "10-4020-000304",
789 | 30025,
790 | 651,
791 | 651,
792 | "CARGO TRANSPORT 5",
793 | null,
794 | "40.9045",
795 | "3.2724",
796 | "1.0226",
797 | "45.1995",
798 | null,
799 | "6e903ea3-1b9e-4232-94c3-81c15669f830",
800 | "2008-06-08"
801 | ],
802 | [
803 | 71920,
804 | 2,
805 | "2008-06-01",
806 | "2008-06-13",
807 | "2008-06-08",
808 | 5,
809 | false,
810 | "SO71920",
811 | "PO4002189853",
812 | "10-4020-000674",
813 | 29982,
814 | 1102,
815 | 1102,
816 | "CARGO TRANSPORT 5",
817 | null,
818 | "2980.7929",
819 | "238.4634",
820 | "74.5198",
821 | "3293.7761",
822 | null,
823 | "6228c9cb-1cab-4e32-98ca-d0dae5fe563e",
824 | "2008-06-08"
825 | ],
826 | [
827 | 71923,
828 | 2,
829 | "2008-06-01",
830 | "2008-06-13",
831 | "2008-06-08",
832 | 5,
833 | false,
834 | "SO71923",
835 | "PO174173096",
836 | "10-4020-000277",
837 | 29781,
838 | 1035,
839 | 1035,
840 | "CARGO TRANSPORT 5",
841 | null,
842 | "106.5408",
843 | "8.5233",
844 | "2.6635",
845 | "117.7276",
846 | null,
847 | "f9899f3f-b4b6-4756-b013-96c16be20427",
848 | "2008-06-08"
849 | ],
850 | [
851 | 71935,
852 | 2,
853 | "2008-06-01",
854 | "2008-06-13",
855 | "2008-06-08",
856 | 5,
857 | false,
858 | "SO71935",
859 | "PO7946145876",
860 | "10-4020-000438",
861 | 29531,
862 | 1061,
863 | 1061,
864 | "CARGO TRANSPORT 5",
865 | null,
866 | "6634.2961",
867 | "530.7437",
868 | "165.8574",
869 | "7330.8972",
870 | null,
871 | "7033c6ec-b12c-45bc-bd96-56efde4c7dd0",
872 | "2008-06-08"
873 | ],
874 | [
875 | 71936,
876 | 2,
877 | "2008-06-01",
878 | "2008-06-13",
879 | "2008-06-08",
880 | 5,
881 | false,
882 | "SO71936",
883 | "PO8671170385",
884 | "10-4020-000502",
885 | 30050,
886 | 662,
887 | 662,
888 | "CARGO TRANSPORT 5",
889 | null,
890 | "98278.6910",
891 | "7862.2953",
892 | "2456.9673",
893 | "108597.9536",
894 | null,
895 | "119db56a-a97e-414d-b41c-64886fc50ab7",
896 | "2008-06-08"
897 | ],
898 | [
899 | 71938,
900 | 2,
901 | "2008-06-01",
902 | "2008-06-13",
903 | "2008-06-08",
904 | 5,
905 | false,
906 | "SO71938",
907 | "PO8468183315",
908 | "10-4020-000016",
909 | 29546,
910 | 635,
911 | 635,
912 | "CARGO TRANSPORT 5",
913 | null,
914 | "88812.8625",
915 | "7105.0290",
916 | "2220.3216",
917 | "98138.2131",
918 | null,
919 | "a36ee74a-cf0d-4024-a1ce-4eaffd1f85f0",
920 | "2008-06-08"
921 | ],
922 | [
923 | 71946,
924 | 2,
925 | "2008-06-01",
926 | "2008-06-13",
927 | "2008-06-08",
928 | 5,
929 | false,
930 | "SO71946",
931 | "PO8961158629",
932 | "10-4020-000466",
933 | 29741,
934 | 660,
935 | 660,
936 | "CARGO TRANSPORT 5",
937 | null,
938 | "38.9536",
939 | "3.1163",
940 | "0.9738",
941 | "43.0437",
942 | null,
943 | "10e3129d-657f-46f9-86f5-cedd79b1901c",
944 | "2008-06-08"
945 | ]
946 | ],
947 | "datasetInfos": [],
948 | "dbfsResultPath": null,
949 | "isJsonSchema": true,
950 | "metadata": {},
951 | "overflow": false,
952 | "plotOptions": {
953 | "customPlotOptions": {},
954 | "displayType": "table",
955 | "pivotAggregation": null,
956 | "pivotColumns": null,
957 | "xColumns": null,
958 | "yColumns": null
959 | },
960 | "removedWidgets": [],
961 | "schema": [
962 | {
963 | "metadata": "{}",
964 | "name": "Sales_Order_ID",
965 | "type": "\"integer\""
966 | },
967 | {
968 | "metadata": "{}",
969 | "name": "Revision_Number",
970 | "type": "\"integer\""
971 | },
972 | {
973 | "metadata": "{}",
974 | "name": "Order_Date",
975 | "type": "\"string\""
976 | },
977 | {
978 | "metadata": "{}",
979 | "name": "Due_Date",
980 | "type": "\"string\""
981 | },
982 | {
983 | "metadata": "{}",
984 | "name": "Ship_Date",
985 | "type": "\"string\""
986 | },
987 | {
988 | "metadata": "{}",
989 | "name": "Status",
990 | "type": "\"integer\""
991 | },
992 | {
993 | "metadata": "{}",
994 | "name": "Online_Order_Flag",
995 | "type": "\"boolean\""
996 | },
997 | {
998 | "metadata": "{}",
999 | "name": "Sales_Order_Number",
1000 | "type": "\"string\""
1001 | },
1002 | {
1003 | "metadata": "{}",
1004 | "name": "Purchase_Order_Number",
1005 | "type": "\"string\""
1006 | },
1007 | {
1008 | "metadata": "{}",
1009 | "name": "Account_Number",
1010 | "type": "\"string\""
1011 | },
1012 | {
1013 | "metadata": "{}",
1014 | "name": "Customer_ID",
1015 | "type": "\"integer\""
1016 | },
1017 | {
1018 | "metadata": "{}",
1019 | "name": "Ship_To_Address_ID",
1020 | "type": "\"integer\""
1021 | },
1022 | {
1023 | "metadata": "{}",
1024 | "name": "Bill_To_Address_ID",
1025 | "type": "\"integer\""
1026 | },
1027 | {
1028 | "metadata": "{}",
1029 | "name": "Ship_Method",
1030 | "type": "\"string\""
1031 | },
1032 | {
1033 | "metadata": "{}",
1034 | "name": "Credit_Card_Approval_Code",
1035 | "type": "\"string\""
1036 | },
1037 | {
1038 | "metadata": "{}",
1039 | "name": "Sub_Total",
1040 | "type": "\"decimal(19,4)\""
1041 | },
1042 | {
1043 | "metadata": "{}",
1044 | "name": "Tax_Amt",
1045 | "type": "\"decimal(19,4)\""
1046 | },
1047 | {
1048 | "metadata": "{}",
1049 | "name": "Freight",
1050 | "type": "\"decimal(19,4)\""
1051 | },
1052 | {
1053 | "metadata": "{}",
1054 | "name": "Total_Due",
1055 | "type": "\"decimal(19,4)\""
1056 | },
1057 | {
1058 | "metadata": "{}",
1059 | "name": "Comment",
1060 | "type": "\"string\""
1061 | },
1062 | {
1063 | "metadata": "{}",
1064 | "name": "rowguid",
1065 | "type": "\"string\""
1066 | },
1067 | {
1068 | "metadata": "{}",
1069 | "name": "Modified_Date",
1070 | "type": "\"string\""
1071 | }
1072 | ],
1073 | "type": "table"
1074 | }
1075 | },
1076 | "output_type": "display_data"
1077 | }
1078 | ],
1079 | "source": [
1080 | "display(df)"
1081 | ]
1082 | }
1083 | ],
1084 | "metadata": {
1085 | "application/vnd.databricks.v1+notebook": {
1086 | "dashboards": [],
1087 | "language": "python",
1088 | "notebookMetadata": {
1089 | "pythonIndentUnit": 4
1090 | },
1091 | "notebookName": "silver to gold",
1092 | "widgets": {}
1093 | }
1094 | },
1095 | "nbformat": 4,
1096 | "nbformat_minor": 0
1097 | }
1098 |
--------------------------------------------------------------------------------
/databricks_notebooks/storagemount.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": 0,
6 | "metadata": {
7 | "application/vnd.databricks.v1+cell": {
8 | "cellMetadata": {
9 | "byteLimit": 2048000,
10 | "rowLimit": 10000
11 | },
12 | "inputWidgets": {},
13 | "nuid": "8d47c7ce-fe9b-4038-ba49-d01580ba2d0a",
14 | "showTitle": false,
15 | "title": ""
16 | }
17 | },
18 | "outputs": [
19 | {
20 | "output_type": "display_data",
21 | "data": {
22 | "text/plain": [
23 | "\u001B[0;31m---------------------------------------------------------------------------\u001B[0m\n",
24 | "\u001B[0;31mExecutionError\u001B[0m Traceback (most recent call last)\n",
25 | "File \u001B[0;32m, line 7\u001B[0m\n",
26 | "\u001B[1;32m 1\u001B[0m configs \u001B[38;5;241m=\u001B[39m {\n",
27 | "\u001B[1;32m 2\u001B[0m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mfs.azure.account.auth.type\u001B[39m\u001B[38;5;124m\"\u001B[39m: \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mCustomAccessToken\u001B[39m\u001B[38;5;124m\"\u001B[39m,\n",
28 | "\u001B[1;32m 3\u001B[0m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mfs.azure.account.custom.token.provider.class\u001B[39m\u001B[38;5;124m\"\u001B[39m: spark\u001B[38;5;241m.\u001B[39mconf\u001B[38;5;241m.\u001B[39mget(\u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mspark.databricks.passthrough.adls.gen2.tokenProviderClassName\u001B[39m\u001B[38;5;124m\"\u001B[39m)\n",
29 | "\u001B[1;32m 4\u001B[0m }\n",
30 | "\u001B[1;32m 6\u001B[0m \u001B[38;5;66;03m# Optionally, you can add to the source URI of your mount point.\u001B[39;00m\n",
31 | "\u001B[0;32m----> 7\u001B[0m dbutils\u001B[38;5;241m.\u001B[39mfs\u001B[38;5;241m.\u001B[39mmount(\n",
32 | "\u001B[1;32m 8\u001B[0m source \u001B[38;5;241m=\u001B[39m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mabfss://bronze@datalakestorage123nik.dfs.core.windows.net/\u001B[39m\u001B[38;5;124m\"\u001B[39m,\n",
33 | "\u001B[1;32m 9\u001B[0m mount_point \u001B[38;5;241m=\u001B[39m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124m/mnt/bronze\u001B[39m\u001B[38;5;124m\"\u001B[39m,\n",
34 | "\u001B[1;32m 10\u001B[0m extra_configs \u001B[38;5;241m=\u001B[39m configs)\n",
35 | "\n",
36 | "File \u001B[0;32m/databricks/python_shell/dbruntime/dbutils.py:362\u001B[0m, in \u001B[0;36mDBUtils.FSHandler.prettify_exception_message..f_with_exception_handling\u001B[0;34m(*args, **kwargs)\u001B[0m\n",
37 | "\u001B[1;32m 360\u001B[0m exc\u001B[38;5;241m.\u001B[39m__context__ \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;01mNone\u001B[39;00m\n",
38 | "\u001B[1;32m 361\u001B[0m exc\u001B[38;5;241m.\u001B[39m__cause__ \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;01mNone\u001B[39;00m\n",
39 | "\u001B[0;32m--> 362\u001B[0m \u001B[38;5;28;01mraise\u001B[39;00m exc\n",
40 | "\n",
41 | "\u001B[0;31mExecutionError\u001B[0m: An error occurred while calling o404.mount.\n",
42 | ": java.rmi.RemoteException: java.lang.IllegalArgumentException: requirement failed: Directory already mounted: /mnt/bronze; nested exception is: \n",
43 | "\tjava.lang.IllegalArgumentException: requirement failed: Directory already mounted: /mnt/bronze\n",
44 | "\tat com.databricks.backend.daemon.data.client.DbfsClient.send0(DbfsClient.scala:135)\n",
45 | "\tat com.databricks.backend.daemon.data.client.DbfsClient.sendIdempotent(DbfsClient.scala:69)\n",
46 | "\tat com.databricks.backend.daemon.dbutils.DBUtilsCore.createOrUpdateMount(DBUtilsCore.scala:1057)\n",
47 | "\tat com.databricks.backend.daemon.dbutils.DBUtilsCore.$anonfun$mount$1(DBUtilsCore.scala:1083)\n",
48 | "\tat com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:571)\n",
49 | "\tat com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:666)\n",
50 | "\tat com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:684)\n",
51 | "\tat com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)\n",
52 | "\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)\n",
53 | "\tat com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:196)\n",
54 | "\tat com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)\n",
55 | "\tat com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)\n",
56 | "\tat com.databricks.backend.daemon.dbutils.FSUtils.withAttributionContext(DBUtilsCore.scala:69)\n",
57 | "\tat com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:470)\n",
58 | "\tat com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:455)\n",
59 | "\tat com.databricks.backend.daemon.dbutils.FSUtils.withAttributionTags(DBUtilsCore.scala:69)\n",
60 | "\tat com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:661)\n",
61 | "\tat com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:580)\n",
62 | "\tat com.databricks.backend.daemon.dbutils.FSUtils.recordOperationWithResultTags(DBUtilsCore.scala:69)\n",
63 | "\tat com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:571)\n",
64 | "\tat com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:540)\n",
65 | "\tat com.databricks.backend.daemon.dbutils.FSUtils.recordOperation(DBUtilsCore.scala:69)\n",
66 | "\tat com.databricks.backend.daemon.dbutils.FSUtils.recordDbutilsFsOp(DBUtilsCore.scala:133)\n",
67 | "\tat com.databricks.backend.daemon.dbutils.DBUtilsCore.mount(DBUtilsCore.scala:1077)\n",
68 | "\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n",
69 | "\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n",
70 | "\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n",
71 | "\tat java.lang.reflect.Method.invoke(Method.java:498)\n",
72 | "\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n",
73 | "\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:397)\n",
74 | "\tat py4j.Gateway.invoke(Gateway.java:306)\n",
75 | "\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n",
76 | "\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n",
77 | "\tat py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:195)\n",
78 | "\tat py4j.ClientServerConnection.run(ClientServerConnection.java:115)\n",
79 | "\tat java.lang.Thread.run(Thread.java:750)\n",
80 | "Caused by: java.lang.IllegalArgumentException: requirement failed: Directory already mounted: /mnt/bronze\n",
81 | "\tat scala.Predef$.require(Predef.scala:281)\n",
82 | "\tat com.databricks.backend.daemon.data.server.DefaultMetadataManager.$anonfun$insertMount$1(MetadataManager.scala:580)\n",
83 | "\tat com.databricks.backend.daemon.data.server.DefaultMetadataManager.$anonfun$modifyAndVerify$2(MetadataManager.scala:948)\n",
84 | "\tat scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)\n",
85 | "\tat com.databricks.backend.daemon.data.server.DefaultMetadataManager.withRetries(MetadataManager.scala:729)\n",
86 | "\tat com.databricks.backend.daemon.data.server.DefaultMetadataManager.modifyAndVerify(MetadataManager.scala:937)\n",
87 | "\tat com.databricks.backend.daemon.data.server.DefaultMetadataManager.insertMount(MetadataManager.scala:588)\n",
88 | "\tat com.databricks.backend.daemon.data.server.handler.MountHandler.receive(MountHandler.scala:120)\n",
89 | "\tat com.databricks.backend.daemon.data.server.session.SessionContext.$anonfun$queryHandlers$1(SessionContext.scala:54)\n",
90 | "\tat com.databricks.backend.daemon.data.server.session.SessionContext.$anonfun$queryHandlers$1$adapted(SessionContext.scala:53)\n",
91 | "\tat scala.collection.immutable.List.foreach(List.scala:431)\n",
92 | "\tat com.databricks.backend.daemon.data.server.session.SessionContext.queryHandlers(SessionContext.scala:53)\n",
93 | "\tat com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$4.$anonfun$applyOrElse$5(DbfsServerBackend.scala:388)\n",
94 | "\tat com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)\n",
95 | "\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)\n",
96 | "\tat com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:216)\n",
97 | "\tat com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)\n",
98 | "\tat com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)\n",
99 | "\tat com.databricks.rpc.ServerBackend.withAttributionContext(ServerBackend.scala:22)\n",
100 | "\tat com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:472)\n",
101 | "\tat com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:455)\n",
102 | "\tat com.databricks.rpc.ServerBackend.withAttributionTags(ServerBackend.scala:22)\n",
103 | "\tat com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$4.applyOrElse(DbfsServerBackend.scala:388)\n",
104 | "\tat com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$4.applyOrElse(DbfsServerBackend.scala:332)\n",
105 | "\tat com.databricks.rpc.ServerBackend.$anonfun$internalReceive0$2(ServerBackend.scala:174)\n",
106 | "\tat com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:200)\n",
107 | "\tat com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:200)\n",
108 | "\tat com.databricks.rpc.ServerBackend.internalReceive0(ServerBackend.scala:171)\n",
109 | "\tat com.databricks.rpc.ServerBackend.$anonfun$internalReceive$1(ServerBackend.scala:147)\n",
110 | "\tat com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:573)\n",
111 | "\tat com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:668)\n",
112 | "\tat com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:686)\n",
113 | "\tat com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)\n",
114 | "\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)\n",
115 | "\tat com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:216)\n",
116 | "\tat com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)\n",
117 | "\tat com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)\n",
118 | "\tat com.databricks.rpc.ServerBackend.withAttributionContext(ServerBackend.scala:22)\n",
119 | "\tat com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:472)\n",
120 | "\tat com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:455)\n",
121 | "\tat com.databricks.rpc.ServerBackend.withAttributionTags(ServerBackend.scala:22)\n",
122 | "\tat com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:663)\n",
123 | "\tat com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:582)\n",
124 | "\tat com.databricks.rpc.ServerBackend.recordOperationWithResultTags(ServerBackend.scala:22)\n",
125 | "\tat com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:573)\n",
126 | "\tat com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:542)\n",
127 | "\tat com.databricks.rpc.ServerBackend.recordOperation(ServerBackend.scala:22)\n",
128 | "\tat com.databricks.rpc.ServerBackend.internalReceive(ServerBackend.scala:147)\n",
129 | "\tat com.databricks.rpc.JettyServer$RequestManager.handleRPC(JettyServer.scala:1020)\n",
130 | "\tat com.databricks.rpc.JettyServer$RequestManager.handleRequestAndRespond(JettyServer.scala:941)\n",
131 | "\tat com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleHttp$6(JettyServer.scala:545)\n",
132 | "\tat com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleHttp$6$adapted(JettyServer.scala:514)\n",
133 | "\tat com.databricks.logging.activity.ActivityContextFactory$.$anonfun$withActivityInternal$3(ActivityContextFactory.scala:398)\n",
134 | "\tat com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)\n",
135 | "\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)\n",
136 | "\tat com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:216)\n",
137 | "\tat com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)\n",
138 | "\tat com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)\n",
139 | "\tat com.databricks.logging.activity.ActivityContextFactory$.withAttributionContext(ActivityContextFactory.scala:53)\n",
140 | "\tat com.databricks.logging.activity.ActivityContextFactory$.withActivityInternal(ActivityContextFactory.scala:398)\n",
141 | "\tat com.databricks.logging.activity.ActivityContextFactory$.withServiceRequestActivity(ActivityContextFactory.scala:154)\n",
142 | "\tat com.databricks.rpc.JettyServer$RequestManager.handleHttp(JettyServer.scala:514)\n",
143 | "\tat com.databricks.rpc.JettyServer$RequestManager.doPost(JettyServer.scala:404)\n",
144 | "\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:665)\n",
145 | "\tat com.databricks.rpc.HttpServletWithPatch.service(HttpServletWithPatch.scala:33)\n",
146 | "\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:750)\n",
147 | "\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)\n",
148 | "\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)\n",
149 | "\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)\n",
150 | "\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n",
151 | "\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n",
152 | "\tat org.eclipse.jetty.server.Server.handle(Server.java:539)\n",
153 | "\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)\n",
154 | "\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n",
155 | "\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n",
156 | "\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n",
157 | "\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n",
158 | "\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n",
159 | "\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n",
160 | "\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n",
161 | "\tat com.databricks.rpc.InstrumentedQueuedThreadPool$$anon$1.$anonfun$run$1(InstrumentedQueuedThreadPool.scala:81)\n",
162 | "\tat scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)\n",
163 | "\tat com.databricks.instrumentation.QueuedThreadPoolInstrumenter.trackActiveThreads(QueuedThreadPoolInstrumenter.scala:66)\n",
164 | "\tat com.databricks.instrumentation.QueuedThreadPoolInstrumenter.trackActiveThreads$(QueuedThreadPoolInstrumenter.scala:63)\n",
165 | "\tat com.databricks.rpc.InstrumentedQueuedThreadPool.trackActiveThreads(InstrumentedQueuedThreadPool.scala:47)\n",
166 | "\tat com.databricks.rpc.InstrumentedQueuedThreadPool$$anon$1.run(InstrumentedQueuedThreadPool.scala:76)\n",
167 | "\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n",
168 | "\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n",
169 | "\t... 1 more\n"
170 | ]
171 | },
172 | "metadata": {
173 | "application/vnd.databricks.v1+output": {
174 | "arguments": {},
175 | "data": "\u001B[0;31m---------------------------------------------------------------------------\u001B[0m\n\u001B[0;31mExecutionError\u001B[0m Traceback (most recent call last)\nFile \u001B[0;32m, line 7\u001B[0m\n\u001B[1;32m 1\u001B[0m configs \u001B[38;5;241m=\u001B[39m {\n\u001B[1;32m 2\u001B[0m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mfs.azure.account.auth.type\u001B[39m\u001B[38;5;124m\"\u001B[39m: \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mCustomAccessToken\u001B[39m\u001B[38;5;124m\"\u001B[39m,\n\u001B[1;32m 3\u001B[0m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mfs.azure.account.custom.token.provider.class\u001B[39m\u001B[38;5;124m\"\u001B[39m: spark\u001B[38;5;241m.\u001B[39mconf\u001B[38;5;241m.\u001B[39mget(\u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mspark.databricks.passthrough.adls.gen2.tokenProviderClassName\u001B[39m\u001B[38;5;124m\"\u001B[39m)\n\u001B[1;32m 4\u001B[0m }\n\u001B[1;32m 6\u001B[0m \u001B[38;5;66;03m# Optionally, you can add to the source URI of your mount point.\u001B[39;00m\n\u001B[0;32m----> 7\u001B[0m dbutils\u001B[38;5;241m.\u001B[39mfs\u001B[38;5;241m.\u001B[39mmount(\n\u001B[1;32m 8\u001B[0m source \u001B[38;5;241m=\u001B[39m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mabfss://bronze@datalakestorage123nik.dfs.core.windows.net/\u001B[39m\u001B[38;5;124m\"\u001B[39m,\n\u001B[1;32m 9\u001B[0m mount_point \u001B[38;5;241m=\u001B[39m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124m/mnt/bronze\u001B[39m\u001B[38;5;124m\"\u001B[39m,\n\u001B[1;32m 10\u001B[0m extra_configs \u001B[38;5;241m=\u001B[39m configs)\n\nFile \u001B[0;32m/databricks/python_shell/dbruntime/dbutils.py:362\u001B[0m, in \u001B[0;36mDBUtils.FSHandler.prettify_exception_message..f_with_exception_handling\u001B[0;34m(*args, **kwargs)\u001B[0m\n\u001B[1;32m 360\u001B[0m exc\u001B[38;5;241m.\u001B[39m__context__ \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;01mNone\u001B[39;00m\n\u001B[1;32m 361\u001B[0m exc\u001B[38;5;241m.\u001B[39m__cause__ \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;01mNone\u001B[39;00m\n\u001B[0;32m--> 362\u001B[0m \u001B[38;5;28;01mraise\u001B[39;00m exc\n\n\u001B[0;31mExecutionError\u001B[0m: An error occurred while calling o404.mount.\n: java.rmi.RemoteException: java.lang.IllegalArgumentException: requirement failed: Directory already mounted: /mnt/bronze; nested exception is: \n\tjava.lang.IllegalArgumentException: requirement failed: Directory already mounted: /mnt/bronze\n\tat com.databricks.backend.daemon.data.client.DbfsClient.send0(DbfsClient.scala:135)\n\tat com.databricks.backend.daemon.data.client.DbfsClient.sendIdempotent(DbfsClient.scala:69)\n\tat com.databricks.backend.daemon.dbutils.DBUtilsCore.createOrUpdateMount(DBUtilsCore.scala:1057)\n\tat com.databricks.backend.daemon.dbutils.DBUtilsCore.$anonfun$mount$1(DBUtilsCore.scala:1083)\n\tat com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:571)\n\tat com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:666)\n\tat com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:684)\n\tat com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)\n\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)\n\tat com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:196)\n\tat com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)\n\tat com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)\n\tat com.databricks.backend.daemon.dbutils.FSUtils.withAttributionContext(DBUtilsCore.scala:69)\n\tat com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:470)\n\tat com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:455)\n\tat com.databricks.backend.daemon.dbutils.FSUtils.withAttributionTags(DBUtilsCore.scala:69)\n\tat com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:661)\n\tat com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:580)\n\tat com.databricks.backend.daemon.dbutils.FSUtils.recordOperationWithResultTags(DBUtilsCore.scala:69)\n\tat com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:571)\n\tat com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:540)\n\tat com.databricks.backend.daemon.dbutils.FSUtils.recordOperation(DBUtilsCore.scala:69)\n\tat com.databricks.backend.daemon.dbutils.FSUtils.recordDbutilsFsOp(DBUtilsCore.scala:133)\n\tat com.databricks.backend.daemon.dbutils.DBUtilsCore.mount(DBUtilsCore.scala:1077)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:397)\n\tat py4j.Gateway.invoke(Gateway.java:306)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:195)\n\tat py4j.ClientServerConnection.run(ClientServerConnection.java:115)\n\tat java.lang.Thread.run(Thread.java:750)\nCaused by: java.lang.IllegalArgumentException: requirement failed: Directory already mounted: /mnt/bronze\n\tat scala.Predef$.require(Predef.scala:281)\n\tat com.databricks.backend.daemon.data.server.DefaultMetadataManager.$anonfun$insertMount$1(MetadataManager.scala:580)\n\tat com.databricks.backend.daemon.data.server.DefaultMetadataManager.$anonfun$modifyAndVerify$2(MetadataManager.scala:948)\n\tat scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)\n\tat com.databricks.backend.daemon.data.server.DefaultMetadataManager.withRetries(MetadataManager.scala:729)\n\tat com.databricks.backend.daemon.data.server.DefaultMetadataManager.modifyAndVerify(MetadataManager.scala:937)\n\tat com.databricks.backend.daemon.data.server.DefaultMetadataManager.insertMount(MetadataManager.scala:588)\n\tat com.databricks.backend.daemon.data.server.handler.MountHandler.receive(MountHandler.scala:120)\n\tat com.databricks.backend.daemon.data.server.session.SessionContext.$anonfun$queryHandlers$1(SessionContext.scala:54)\n\tat com.databricks.backend.daemon.data.server.session.SessionContext.$anonfun$queryHandlers$1$adapted(SessionContext.scala:53)\n\tat scala.collection.immutable.List.foreach(List.scala:431)\n\tat com.databricks.backend.daemon.data.server.session.SessionContext.queryHandlers(SessionContext.scala:53)\n\tat com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$4.$anonfun$applyOrElse$5(DbfsServerBackend.scala:388)\n\tat com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)\n\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)\n\tat com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:216)\n\tat com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)\n\tat com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)\n\tat com.databricks.rpc.ServerBackend.withAttributionContext(ServerBackend.scala:22)\n\tat com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:472)\n\tat com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:455)\n\tat com.databricks.rpc.ServerBackend.withAttributionTags(ServerBackend.scala:22)\n\tat com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$4.applyOrElse(DbfsServerBackend.scala:388)\n\tat com.databricks.backend.daemon.data.server.DbfsServerBackend$$anonfun$receive$4.applyOrElse(DbfsServerBackend.scala:332)\n\tat com.databricks.rpc.ServerBackend.$anonfun$internalReceive0$2(ServerBackend.scala:174)\n\tat com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:200)\n\tat com.databricks.rpc.ServerBackend$$anonfun$commonReceive$1.applyOrElse(ServerBackend.scala:200)\n\tat com.databricks.rpc.ServerBackend.internalReceive0(ServerBackend.scala:171)\n\tat com.databricks.rpc.ServerBackend.$anonfun$internalReceive$1(ServerBackend.scala:147)\n\tat com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:573)\n\tat com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:668)\n\tat com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:686)\n\tat com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)\n\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)\n\tat com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:216)\n\tat com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)\n\tat com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)\n\tat com.databricks.rpc.ServerBackend.withAttributionContext(ServerBackend.scala:22)\n\tat com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:472)\n\tat com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:455)\n\tat com.databricks.rpc.ServerBackend.withAttributionTags(ServerBackend.scala:22)\n\tat com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:663)\n\tat com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:582)\n\tat com.databricks.rpc.ServerBackend.recordOperationWithResultTags(ServerBackend.scala:22)\n\tat com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:573)\n\tat com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:542)\n\tat com.databricks.rpc.ServerBackend.recordOperation(ServerBackend.scala:22)\n\tat com.databricks.rpc.ServerBackend.internalReceive(ServerBackend.scala:147)\n\tat com.databricks.rpc.JettyServer$RequestManager.handleRPC(JettyServer.scala:1020)\n\tat com.databricks.rpc.JettyServer$RequestManager.handleRequestAndRespond(JettyServer.scala:941)\n\tat com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleHttp$6(JettyServer.scala:545)\n\tat com.databricks.rpc.JettyServer$RequestManager.$anonfun$handleHttp$6$adapted(JettyServer.scala:514)\n\tat com.databricks.logging.activity.ActivityContextFactory$.$anonfun$withActivityInternal$3(ActivityContextFactory.scala:398)\n\tat com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)\n\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)\n\tat com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:216)\n\tat com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)\n\tat com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)\n\tat com.databricks.logging.activity.ActivityContextFactory$.withAttributionContext(ActivityContextFactory.scala:53)\n\tat com.databricks.logging.activity.ActivityContextFactory$.withActivityInternal(ActivityContextFactory.scala:398)\n\tat com.databricks.logging.activity.ActivityContextFactory$.withServiceRequestActivity(ActivityContextFactory.scala:154)\n\tat com.databricks.rpc.JettyServer$RequestManager.handleHttp(JettyServer.scala:514)\n\tat com.databricks.rpc.JettyServer$RequestManager.doPost(JettyServer.scala:404)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:665)\n\tat com.databricks.rpc.HttpServletWithPatch.service(HttpServletWithPatch.scala:33)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:750)\n\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:539)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\tat com.databricks.rpc.InstrumentedQueuedThreadPool$$anon$1.$anonfun$run$1(InstrumentedQueuedThreadPool.scala:81)\n\tat scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)\n\tat com.databricks.instrumentation.QueuedThreadPoolInstrumenter.trackActiveThreads(QueuedThreadPoolInstrumenter.scala:66)\n\tat com.databricks.instrumentation.QueuedThreadPoolInstrumenter.trackActiveThreads$(QueuedThreadPoolInstrumenter.scala:63)\n\tat com.databricks.rpc.InstrumentedQueuedThreadPool.trackActiveThreads(InstrumentedQueuedThreadPool.scala:47)\n\tat com.databricks.rpc.InstrumentedQueuedThreadPool$$anon$1.run(InstrumentedQueuedThreadPool.scala:76)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\t... 1 more\n",
176 | "errorSummary": "java.rmi.RemoteException: java.lang.IllegalArgumentException: requirement failed: Directory already mounted: /mnt/bronze; nested exception is: ",
177 | "errorTraceType": "ansi",
178 | "metadata": {},
179 | "type": "ipynbError"
180 | }
181 | },
182 | "output_type": "display_data"
183 | }
184 | ],
185 | "source": [
186 | "configs = {\n",
187 | " \"fs.azure.account.auth.type\": \"CustomAccessToken\",\n",
188 | " \"fs.azure.account.custom.token.provider.class\": spark.conf.get(\"spark.databricks.passthrough.adls.gen2.tokenProviderClassName\")\n",
189 | "}\n",
190 | "\n",
191 | "# Optionally, you can add to the source URI of your mount point.\n",
192 | "dbutils.fs.mount(\n",
193 | " source = \"abfss://bronze@datalakestorage123nik.dfs.core.windows.net/\",\n",
194 | " mount_point = \"/mnt/bronze\",\n",
195 | " extra_configs = configs)"
196 | ]
197 | },
198 | {
199 | "cell_type": "code",
200 | "execution_count": 0,
201 | "metadata": {
202 | "application/vnd.databricks.v1+cell": {
203 | "cellMetadata": {
204 | "byteLimit": 2048000,
205 | "rowLimit": 10000
206 | },
207 | "inputWidgets": {},
208 | "nuid": "ef735375-3a97-449d-b13f-3b5fb137e955",
209 | "showTitle": false,
210 | "title": ""
211 | }
212 | },
213 | "outputs": [
214 | {
215 | "output_type": "display_data",
216 | "data": {
217 | "text/plain": [
218 | ""
219 | ]
220 | },
221 | "metadata": {
222 | "application/vnd.databricks.v1+output": {
223 | "arguments": {},
224 | "data": "",
225 | "errorSummary": "Command skipped",
226 | "errorTraceType": "ansi",
227 | "metadata": {},
228 | "type": "ipynbError"
229 | }
230 | },
231 | "output_type": "display_data"
232 | }
233 | ],
234 | "source": [
235 | "dbutils.fs.ls(\"/mnt/bronze/SalesLT/\")"
236 | ]
237 | },
238 | {
239 | "cell_type": "code",
240 | "execution_count": 0,
241 | "metadata": {
242 | "application/vnd.databricks.v1+cell": {
243 | "cellMetadata": {
244 | "byteLimit": 2048000,
245 | "rowLimit": 10000
246 | },
247 | "inputWidgets": {},
248 | "nuid": "fbdf9915-ff58-48c8-90b0-2aa7b9d5a2f3",
249 | "showTitle": false,
250 | "title": ""
251 | }
252 | },
253 | "outputs": [
254 | {
255 | "output_type": "display_data",
256 | "data": {
257 | "text/plain": [
258 | ""
259 | ]
260 | },
261 | "metadata": {
262 | "application/vnd.databricks.v1+output": {
263 | "arguments": {},
264 | "data": "",
265 | "errorSummary": "Command skipped",
266 | "errorTraceType": "ansi",
267 | "metadata": {},
268 | "type": "ipynbError"
269 | }
270 | },
271 | "output_type": "display_data"
272 | }
273 | ],
274 | "source": [
275 | "configs = {\n",
276 | " \"fs.azure.account.auth.type\": \"CustomAccessToken\",\n",
277 | " \"fs.azure.account.custom.token.provider.class\": spark.conf.get(\"spark.databricks.passthrough.adls.gen2.tokenProviderClassName\")\n",
278 | "}\n",
279 | "\n",
280 | "# Optionally, you can add to the source URI of your mount point.\n",
281 | "dbutils.fs.mount(\n",
282 | " source = \"abfss://silver@datalakestorage123nik.dfs.core.windows.net/\",\n",
283 | " mount_point = \"/mnt/silver\",\n",
284 | " extra_configs = configs)"
285 | ]
286 | },
287 | {
288 | "cell_type": "code",
289 | "execution_count": 0,
290 | "metadata": {
291 | "application/vnd.databricks.v1+cell": {
292 | "cellMetadata": {
293 | "byteLimit": 2048000,
294 | "rowLimit": 10000
295 | },
296 | "inputWidgets": {},
297 | "nuid": "cb26c20c-630b-47a0-9235-122b47d8ae1c",
298 | "showTitle": false,
299 | "title": ""
300 | }
301 | },
302 | "outputs": [
303 | {
304 | "output_type": "display_data",
305 | "data": {
306 | "text/plain": [
307 | ""
308 | ]
309 | },
310 | "metadata": {
311 | "application/vnd.databricks.v1+output": {
312 | "arguments": {},
313 | "data": "",
314 | "errorSummary": "Command skipped",
315 | "errorTraceType": "ansi",
316 | "metadata": {},
317 | "type": "ipynbError"
318 | }
319 | },
320 | "output_type": "display_data"
321 | }
322 | ],
323 | "source": [
324 | "configs = {\n",
325 | " \"fs.azure.account.auth.type\": \"CustomAccessToken\",\n",
326 | " \"fs.azure.account.custom.token.provider.class\": spark.conf.get(\"spark.databricks.passthrough.adls.gen2.tokenProviderClassName\")\n",
327 | "}\n",
328 | "\n",
329 | "# Optionally, you can add to the source URI of your mount point.\n",
330 | "dbutils.fs.mount(\n",
331 | " source = \"abfss://gold@datalakestorage123nik.dfs.core.windows.net/\",\n",
332 | " mount_point = \"/mnt/gold\",\n",
333 | " extra_configs = configs)"
334 | ]
335 | }
336 | ],
337 | "metadata": {
338 | "application/vnd.databricks.v1+notebook": {
339 | "dashboards": [],
340 | "language": "python",
341 | "notebookMetadata": {
342 | "pythonIndentUnit": 4
343 | },
344 | "notebookName": "storagemount",
345 | "widgets": {}
346 | }
347 | },
348 | "nbformat": 4,
349 | "nbformat_minor": 0
350 | }
351 |
--------------------------------------------------------------------------------
/gif.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/gif.gif
--------------------------------------------------------------------------------
/images/d:
--------------------------------------------------------------------------------
1 | d
2 |
--------------------------------------------------------------------------------
/sdfasd.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/sdfasd.gif
--------------------------------------------------------------------------------
/sdfasdsxd.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Nikhil-Pickle/End-to-End-Azure-Data-Engineering/a4c66847a67bf108223e6b11dfd1ba337306659f/sdfasdsxd.gif
--------------------------------------------------------------------------------