├── README.md ├── content-packs.md ├── hands-on-labs ├── lab-01 │ ├── ProductQuantityForecast.pbix │ └── README.md ├── lab-02 │ ├── README.md │ └── TransactionStatistics.pbix ├── lab-03 │ └── README.md ├── lab-04 │ └── README.md ├── media │ ├── asaworkspace-deploy-configure.png │ ├── asaworkspace-deploy-pbi-linked-service-01.png │ ├── asaworkspace-deploy-pbi-linked-service-02.png │ ├── asaworkspace-deploy-pbi-linked-service-03.png │ ├── asaworkspace-deploy-progress.png │ ├── asaworkspace-deploy-tag.png │ ├── cleanup-01.png │ ├── cleanup-02.png │ ├── cloudshell-configure-01.png │ ├── cloudshell-configure-02.png │ ├── cloudshell-configure-03.png │ ├── cloudshell-configure-04.png │ ├── cloudshell-create-storage-01.png │ ├── cloudshell-create-storage-02.png │ ├── cloudshell-setup-01.png │ ├── cloudshell-setup-02.png │ ├── cloudshell-setup-03.png │ ├── cloudshell-setup-04.png │ ├── issue-details.png │ ├── issue-find-section.png │ ├── issue-new-issue.png │ ├── lab-01-deploy-configure.png │ ├── lab-01-deploy-progress.png │ ├── lab-01-ex-01-task-01-configure-linked-service.png │ ├── lab-01-ex-01-task-01-keyvault-secret.png │ ├── lab-01-ex-01-task-01-mlworkspace-permissions.png │ ├── lab-01-ex-01-task-01-new-linked-service.png │ ├── lab-01-ex-01-task-01-publish-linked-service.png │ ├── lab-01-ex-01-task-01-service-principal-clientid.png │ ├── lab-01-ex-01-task-01-service-principal.png │ ├── lab-01-ex-01-task-02-create-spark-table.png │ ├── lab-01-ex-01-task-02-ml-menu.png │ ├── lab-01-ex-02-task-01-ml-menu.png │ ├── lab-01-ex-02-task-01-model-type.png │ ├── lab-01-ex-02-task-01-open-in-notebook.png │ ├── lab-01-ex-02-task-01-regressio-model-configuration.png │ ├── lab-01-ex-02-task-01-started-notification.png │ ├── lab-01-ex-02-task-01-submit-notification.png │ ├── lab-01-ex-02-task-01-trigger-experiment.png │ ├── lab-01-ex-02-task-02-best-mode-explained.png │ ├── lab-01-ex-02-task-02-experiment-run.png │ ├── lab-01-ex-02-task-02-model-registry.png │ ├── lab-01-ex-02-task-02-open-aml-workspace.png │ ├── lab-01-ex-02-task-02-run-details.png │ ├── lab-01-ex-03-task-01-deploy-model.png │ ├── lab-01-ex-03-task-01-explore-table.png │ ├── lab-01-ex-03-task-01-forecast-stored-procedure.png │ ├── lab-01-ex-03-task-01-map-columns.png │ ├── lab-01-ex-03-task-01-run-forecast.png │ ├── lab-01-ex-03-task-01-select-model.png │ ├── lab-01-ex-03-task-02-connect-to-model.png │ ├── lab-01-ex-03-task-02-new-spark-table.png │ ├── lab-01-ex-03-task-02-text-analytics-code.png │ ├── lab-01-ex-03-task-02-text-analytics-mappings.png │ ├── lab-01-ex-03-task-02-text-analytics-model.png │ ├── lab-01-ex-03-task-02-text-analytics-results.png │ ├── lab-01-ex-03-task-03-create-pipeline.png │ ├── lab-01-ex-03-task-03-pipeline-mapping.png │ ├── lab-01-ex-03-task-03-pipeline-sink.png │ ├── lab-01-ex-03-task-03-pipeline-source.png │ ├── lab-01-ex-03-task-03-pipeline-staging.png │ ├── lab-01-ex-03-task-03-pipeline-stored-procedure-01.png │ ├── lab-01-ex-03-task-03-pipeline-stored-procedure-02.png │ ├── lab-01-ex-03-task-03-pipeline-test.png │ ├── lab-01-ex-04-task-01-credentials-in-power-bi-desktop.png │ ├── lab-01-ex-04-task-01-server-in-power-bi-desktop.png │ ├── lab-01-ex-04-task-01-view-report.png │ ├── lab-01-ex-04-task-02-create-trigger.png │ ├── lab-02-deploy-configure.png │ ├── lab-02-deploy-progress.png │ ├── lab-02-ex-01-task-01-configure-linked-service.png │ ├── lab-02-ex-01-task-01-database-permissions.png │ ├── lab-02-ex-01-task-01-keyvault-secret.png │ ├── lab-02-ex-01-task-01-new-linked-service.png │ ├── lab-02-ex-01-task-01-publish-linked-service.png │ ├── lab-02-ex-01-task-01-service-principal-clientid.png │ ├── lab-02-ex-01-task-01-service-principal.png │ ├── lab-02-ex-01-task-02-load-table.png │ ├── lab-02-ex-01-task-02-notebook-from-table.png │ ├── lab-02-ex-02-task-01-load-sales-telemetry.png │ ├── lab-02-ex-02-task-02-create-indexes.png │ ├── lab-02-ex-02-task-02-explain-hyperspace-01.png │ ├── lab-02-ex-02-task-02-explain-hyperspace-02.png │ ├── lab-02-ex-02-task-02-explain-hyperspace-03.png │ ├── lab-02-ex-02-task-02-explain-hyperspace-04.png │ ├── lab-02-ex-02-task-02-initialize-hyperspace.png │ ├── lab-02-ex-03-task-01-execute-sql-script.png │ ├── lab-02-ex-03-task-01-new-sql-script.png │ ├── lab-02-ex-03-task-02-credentials-in-power-bi-desktop.png │ ├── lab-02-ex-03-task-02-server-in-power-bi-desktop.png │ ├── lab-02-ex-03-task-02-view-report.png │ ├── lab-03-deploy-configure.png │ ├── lab-03-deploy-progress.png │ ├── lab-03-ex-01-task-01-run-adf-pipeline.png │ ├── lab-03-ex-01-task-01-view-pipeline-result.png │ ├── lab-03-ex-01-task-02-check-uploaded-file.png │ ├── lab-03-ex-01-task-02-create-adfartifacts-folder.png │ ├── lab-03-ex-01-task-02-display-file-content.png │ ├── lab-03-ex-01-task-02-download-support-files.png │ ├── lab-03-ex-01-task-02-upload-file.png │ ├── lab-03-ex-02-task-01-create-synapse-artifact.png │ ├── lab-04 │ │ ├── lab04-ex1-task1-read-cdm-01.png │ │ ├── lab04-ex1-task1-read-cdm-02.png │ │ ├── lab04-ex1-task1-read-cdm-03.png │ │ ├── lab04-ex1-task1-read-cdm-04.png │ │ ├── lab04-ex1-task1-read-cdm-05.png │ │ ├── lab04-ex1-task1-read-cdm-06.png │ │ ├── lab04-ex1-task1-read-cdm-07.png │ │ ├── lab04-ex1-task1-read-cdm-08.png │ │ ├── lab04-ex1-task2-spark-read-cdm-01.png │ │ ├── lab04-ex2-task1-update-cdm-01.png │ │ ├── lab04-ex2-task1-update-cdm-02.png │ │ ├── lab04-ex2-task2-update-cdm-01.png │ │ ├── lab04-ex2-task2-update-cdm-02.png │ │ ├── lab04-ex2-task3-spark-write-cdm-01.png │ │ ├── lab04-ex2-task3-spark-write-cdm-02.png │ │ ├── lab04-ex3-task1-pipeline-cdm-01.png │ │ ├── lab04-ex3-task1-pipeline-cdm-02.png │ │ ├── lab04-ex3-task1-pipeline-cdm-03.png │ │ ├── lab04-ex3-task1-pipeline-cdm-04.png │ │ ├── lab04-ex3-task1-pipeline-cdm-05.png │ │ └── lab04-ex3-task1-pipeline-cdm-06.png │ ├── video-01.png │ ├── video-02.png │ └── video-03.png └── setup │ ├── arm │ ├── asaga-workspace-core.json │ ├── asaga-workspace-lab-01.json │ ├── asaga-workspace-lab-02.json │ └── asaga-workspace-lab-03.json │ ├── asa-workspace-deploy.md │ ├── automation │ ├── environment-setup.ps1 │ ├── lab-01-setup.ps1 │ ├── lab-02-setup.ps1 │ └── lab-03-setup.ps1 │ ├── cleanup.md │ ├── datasets │ ├── wwi02_date_adls.json │ ├── wwi02_date_asa.json │ ├── wwi02_poc_customer_adls.json │ ├── wwi02_poc_customer_asa.json │ ├── wwi02_product_adls.json │ ├── wwi02_product_asa.json │ ├── wwi02_sale_small_adls.json │ ├── wwi02_sale_small_asa.json │ ├── wwi02_sale_small_product_quantity_forecast_adls.json │ ├── wwi02_sale_small_product_quantity_forecast_asa.json │ ├── wwi02_sale_small_product_review_adls.json │ ├── wwi02_sale_small_product_review_asa.json │ ├── wwi02_sale_small_stats_adls.json │ ├── wwi02_sale_small_stats_asa.json │ ├── wwi02_sale_small_telemetry_ade.json │ └── wwi02_sale_small_telemetry_adls.json │ ├── lab-01-deploy.md │ ├── lab-02-deploy.md │ ├── lab-03-deploy.md │ ├── pipelines │ ├── import_poc_customer_data.json │ ├── import_sale_small_product_review_data.json │ ├── import_sale_small_stats_data.json │ ├── import_sale_small_telemetry_data.json │ └── load_sql_pool_from_data_lake.json │ ├── solliance-synapse-automation │ └── solliance-synapse-automation.psm1 │ ├── sql │ ├── 01-create-logins.sql │ ├── 02-create-users.sql │ ├── 03-create-schemas.sql │ ├── 04-create-tables-in-wwi-schema.sql │ └── 05-create-tables-lab-03.sql │ └── templates │ ├── blob_storage_linked_service.json │ ├── cosmos_db_linked_service.json │ ├── data_explorer_key_vault_linked_service.json │ ├── data_lake_key_vault_linked_service.json │ ├── data_lake_linked_service.json │ ├── integration_runtime.json │ ├── key_vault_linked_service.json │ ├── spark_notebook.json │ ├── spark_notebook_session.json │ ├── sql_pool_key_vault_linked_service.json │ ├── sql_pool_script.json │ └── sql_script.json ├── presentation ├── SynapseWhatIsNew.pdf └── SynapseWhatIsNew.pptx ├── report-issues.md └── videos ├── git-and-workspace-management.md ├── monitoring.md └── sqldw-synapse-workspace.md /README.md: -------------------------------------------------------------------------------- 1 | # Azure Synapse Analytics GA Content Packs 2 | 3 | ## Lab descriptions 4 | 5 | Id | Name | Description | Useful links 6 | ---|---|---|--- 7 | Lab 01 | [The Integrated Machine Learning process in Synapse Analytics](./hands-on-labs/lab-01/README.md) | This lab demonstrates the integrated, end-to-end Azure Machine Learning and Azure Cognitive Services experience in Azure Synapse Analytics. You will learn how to connect an Azure Synapse Analytics workspace to an Azure Machine Learning workspace using a Linked Service and then trigger an Automated ML experiment that uses data from a Spark table. You will also learn how to use trained models from Azure Machine Learning or Azure Cognitive Services to enrich data in a SQL pool table and then serve prediction results using Power BI.

After completing the lab, you will understand the main steps of an end-to-end Machine Learning process that build on top of the integration between Azure Synapse Analytics and Azure Machine Learning. | https://docs.microsoft.com/en-us/azure/synapse-analytics/machine-learning/quickstart-integrate-azure-machine-learning
https://docs.microsoft.com/en-us/azure/synapse-analytics/machine-learning/tutorial-sql-pool-model-scoring-wizard 8 | Lab 02 | [Working with Apache Spark in Synapse Analytics](./hands-on-labs/lab-02/README.md) | This lab demonstrates the experience of working with Apache Spark in Azure Synapse Analytics. You will learn how to connect an Azure Synapse Analytics workspace to an Azure Data Explorer workspace using a Linked Service and then load data from one of its databases using a Spark notebook. You will also learn how to use libraries like Hyperspace and MSSparkUtil to optimize the experience of working with Data Lake storage accounts from Spark notebooks. In addition to Data Explorer and Data Lake storage, the data enrichment process will also use historical data from a SQL Pool. In the end, you will learn how to publish the enriched data back into the Data Lake and consume it with the SQL Built-in Pool and Power BI.

After completing the lab, you will understand the main steps of an end-to-end data enrichment process that uses Spark in an Azure Synapse Analytics workspace. | https://docs.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-overview
https://techcommunity.microsoft.com/t5/azure-data-explorer/announcing-azure-data-explorer-data-connector-for-azure-synapse/ba-p/1743868 9 | Lab 03 | [Migrating Data Factory Pipelines to Synapse Analytics Pipelines](./hands-on-labs/lab-03/README.md) | This lab demonstrates the experience of migrating an existing Azure Data Factory pipeline to an Azure Synapse Analytics Pipeline. You will learn how to script major ADF artifacts like linked services, datasets, activities, and pipelines. You will also learn how to import these artifacts into Azure Synapse Analytics. In the end, you will learn to validate, test, and trigger the imported Azure Synapse Pipeline.

After completing the lab, you will understand the main steps of a migration process from Azure Data Factory pipelines to Azure Synapse Analytics pipelines. | https://docs.microsoft.com/en-us/dotnet/api/azure.analytics.synapse.artifacts?view=azure-dotnet-preview 10 | Lab 04 | [Working with the Common Data Model in Synapse Analytics](./hands-on-labs/lab-04/README.md) | This lab demonstrates using Synapse Analytics Spark to work with the Common Data Model. | https://docs.microsoft.com/en-us/common-data-model/ 11 | 12 | ## Videos 13 | 14 | Name | Description 15 | --- | --- 16 | [Adding an Azure Synapse Analytics workspace to an existing Azure SQL Data Warehouse instance](./videos/sqldw-synapse-workspace.md) | This video provides an overview on how to add an Azure Synapse Analytics workspace on top of an existing Azure Synapse Analytics (formerly SQL DW) instance. 17 | [Azure Synapse Analytics Git integration and workspace management](./videos/git-and-workspace-management.md) | This video provides an overview of the new Git integration features in Synapse Analytics as well as some of other features related to workspace management (folders and worskpace access control). 18 | [Azure Synapse Analytics monitoring](./videos/monitoring.md) | This video provides an overview of the new monitoring capabilities in Synapse Analytics. It covers Az Monitor integration as well as SQL and Spark pool monitoring. 19 | 20 | ## Report issues 21 | 22 | In case you encounter any issues with the content in this repository, please follow the [How to report issues](./report-issues.md) guideline. We will try to address them as soon as possible. Please check your open issues to learn about their status. 23 | -------------------------------------------------------------------------------- /content-packs.md: -------------------------------------------------------------------------------- 1 | # Readiness content packs for Azure Synapse Analytics features released at GA 2 | 3 | The following content packs will be released: 4 | - Pre-GA content pack 5 | - Post-GA content pack 6 | 7 | The following table shows the mapping of categories and topics to content packs: 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 |
Category / Topic mapping to content packs
 
CategoryTopicPre-GA CPPost-GA CPDetails
Migration from ADF and SQL DWADF import to Synapse PipelineMarkdown / Lab 03
Synapse workspace addition on top of existing SQL DWMarkdown / Video 01
Git integrationCI/CD and Git IntegrationMarkdown / Video 02
Artifact - FoldersMarkdown / Video 02
Artifact - RenameMarkdown / Video 02
Workspace access controlMarkdown / Video 02Admin, data engineer/scientist, data/business analyst
Linked ServicesIntegration - PurviewMarkdown / Video 06
Integration - AMLMarkdown / Lab 01
Integration - Cognitive ServicesMarkdown / Lab 01
Integration - Azure Data ExplorerMarkdown / Lab 02https://docs.microsoft.com/en-us/azure/synapse-analytics/quickstart-connect-azure-data-explorer
Synapse LinkSynapse Link with MongoDB APIMarkdown / Video 07
Synapse Link SQL serverless - CosmosDBMarkdown / Video 03 (low priority)https://docs.microsoft.com/en-us/azure/cosmos-db/synapse-link
https://docs.microsoft.com/en-us/azure/synapse-analytics/sql/query-cosmos-db-analytical-store
DevelopmentData Wrangling using Power QueryMarkdown / Video 08
What does it mean to run Apache Spark in Synapse - benefits, unique capabilitiesMarkdown / Lab 02https://docs.microsoft.com/en-us/azure/synapse-analytics/metadata/overview
MSSparkUtil libraryMarkdown / Lab 02https://docs.microsoft.com/en-us/azure/synapse-analytics/spark/microsoft-spark-utilities?pivots=programming-language-python
Hummingbird libraryMarkdown / (just mention in the lab this is an alternative)
Metastore - Spark and serverless SQL poolMarkdown / Lab 02https://docs.microsoft.com/en-us/azure/synapse-analytics/metadata/database
127 | https://docs.microsoft.com/en-us/azure/synapse-analytics/metadata/table
HyperspaceMarkdown / Lab 02https://github.com/microsoft/hyperspace
https://docs.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-performance-hyperspace?pivots=programming-language-csharp
Spark CDM connectorTBDhttps://github.com/Azure/spark-cdm-connector
MonitoringMonitoring - Az Monitor IntegrationMarkdown / Video 05
Monitoring - SQL Request DetailsMarkdown / Video 05
Monitoring - Spark poolsMarkdown / Video 05
Monitoring  - dedicated SQL poolMarkdown / Video 05
168 | 169 | -------------------------------------------------------------------------------- /hands-on-labs/lab-01/ProductQuantityForecast.pbix: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/lab-01/ProductQuantityForecast.pbix -------------------------------------------------------------------------------- /hands-on-labs/lab-02/TransactionStatistics.pbix: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/lab-02/TransactionStatistics.pbix -------------------------------------------------------------------------------- /hands-on-labs/lab-03/README.md: -------------------------------------------------------------------------------- 1 | # Lab 03 - Migrating Data Factory Pipelines to Synapse Analytics Pipelines 2 | 3 | This lab demonstrates the experience of migrating an existing Azure Data Factory pipeline to an Azure Synapse Analytics Pipeline. You will learn how to script major ADF artifacts like linked services, datasets, activities, and pipelines. You will also learn how to import these artifacts into Azure Synapse Analytics. In the end, you will learn to validate, test, and trigger the imported Azure Synapse Pipeline. 4 | 5 | After completing the lab, you will understand the main steps of a migration process from Azure Data Factory pipelines to Azure Synapse Analytics pipelines. 6 | 7 | >**NOTE**: 8 | > 9 | >There are more automated approaches possible to migrate existing ADF v2 artifacts to Synapse Analytics Pipeline. In this lab, we use a more manual approach to enable you to understand the details of the process. 10 | 11 | This lab has the following structure: 12 | 13 | - [Before the hands-on lab](#before-the-hands-on-lab) 14 | - [Task 1 - Create and configure the Azure Synapse Analytics workspace](#task-1---create-and-configure-the-azure-synapse-analytics-workspace) 15 | - [Task 2 - Create and configure additional resources for this lab](#task-2---create-and-configure-additional-resources-for-this-lab) 16 | - [Exercise 1 - Script an Azure Data Factory (ADF) pipeline](#exercise-1---script-an-azure-data-factory-adf-pipeline) 17 | - [Task 1 - View and run the ADF pipeline](#task-1---view-and-run-the-adf-pipeline) 18 | - [Task 2 - Script an ADF pipeline and all its related components](#task-2---script-an-adf-pipeline-and-all-its-related-components) 19 | - [Exercise 2 - Import a scripted ADF pipeline into Azure Synapse](#exercise-2---import-a-scripted-adf-pipeline-into-azure-synapse) 20 | - [Task 1 - Import linked services](#task-1---import-linked-services) 21 | - [Task 2 - Import datasets](#task-2---import-datasets) 22 | - [Task 3 - Import a pipeline](#task-3---import-a-pipeline) 23 | - [Exercise 3 - Test the imported pipeline](#exercise-3---test-the-imported-pipeline) 24 | - [Task 1 - Validate and run the imported pipeline](#task-1---validate-and-run-the-imported-pipeline) 25 | - [After the hands-on lab](#after-the-hands-on-lab) 26 | - [Resources](#resources) 27 | - [Report issues](#report-issues) 28 | 29 | ## Before the hands-on lab 30 | 31 | Before stepping through the exercises in this lab, make sure you have properly configured your Azure Synapse Analytics workspace. Perform the tasks below to configure the workspace. 32 | 33 | ### Task 1 - Create and configure the Azure Synapse Analytics workspace 34 | 35 | >**NOTE** 36 | > 37 | >If you have already created and configured the Synapse Analytics workspace while running one of the other labs available in this repo, you must not perform this task again and you can move on to the next task. The labs are designed to share the Synapse Analytics workspace, so you only need to create it once. 38 | 39 | Follow the instructions in [Deploy your Azure Synapse Analytics workspace](./../setup/asa-workspace-deploy.md) to create and configure the workspace. 40 | 41 | ### Task 2 - Create and configure additional resources for this lab 42 | 43 | Follow the instructions in [Deploy resources for Lab 03](./../setup/lab-03-deploy.md) to deploy additional resources for this lab. Once deployment is complete, you are ready to proceed with the exercises in this lab. 44 | 45 | ## Exercise 1 - Script an Azure Data Factory (ADF) pipeline 46 | 47 | In this exercise you will run an existing Azure Data Factory v2 pipeline and then script its components. This will prepare the necessary inputs to re-create the pipeline as a Synapse Analytics Pipeline in your workspace. 48 | 49 | ### Task 1 - View and run the ADF pipeline 50 | 51 | In the Azure Portal, select your Azure Data Factory v2 workspace and then select `Author & Monitor` to load the ADF designer. In the designer, select the `Author` hub on the left side, select the `Pipelines` section and then select the `Lab 03 - Import Sales Stats` pipeline. Start the pipeline in debug mode by selecting the `Debug` option on the toolbar and wait until the pipeline completes successfully. 52 | 53 | ![View and run ADF v2 pipeline](./../media/lab-03-ex-01-task-01-run-adf-pipeline.png) 54 | 55 | In the Azure Portal, select your Synapse Analytics workspace and then select `Open Synapse Studio` to load Synapse Studio. In Synapse Studio, select the `Develop` hub on the left side, open a new SQL script and run the following statement on `SQLPool01`: 56 | 57 | ```sql 58 | SELECT 59 | COUNT(*) 60 | FROM 61 | wwi.SaleStatistic 62 | ``` 63 | 64 | You should get a count of around 27.000 records which proves the ADF pipeline executed successfully. 65 | 66 | ![View ADF v2 pipeline results](./../media/lab-03-ex-01-task-01-view-pipeline-result.png) 67 | 68 | Return to the ADF designer and explore the structure of the pipeline. Notice the following ADF objects: 69 | 70 | - **Pipelines**: `Lab 03 - Import Sales Stats` 71 | - **Datasets**: `wwi02_sale_small_stats_adls`, `wwi02_sale_small_stats_asa` 72 | - **Linked services**: `asagakeyvault01`, `asagadatalake01`, `asagasqlpool01` (available in the `Manage` hub) 73 | 74 | The remainder of this exercise covers the procedure to get the JSON templates associated with these objects. 75 | 76 | ### Task 2 - Script an ADF pipeline and all its related components 77 | 78 | Open an Azure Cloud Shell instance and run `Connect-AzAccount -DeviceCode` to make sure you are authenticated with the right account. 79 | 80 | Run the following commands to create a working folder: 81 | 82 | ```cmd 83 | md adfartifacts 84 | cd adfartifacts 85 | ``` 86 | 87 | ![Create artifacts folder](./../media/lab-03-ex-01-task-02-create-adfartifacts-folder.png) 88 | 89 | In the ADF designer, select the `...` to the right of the `Lab 03 - Import Sales Stats` pipeline (appears when you hover over the pipeline name) and then select `Download support files`. 90 | 91 | ![Download support files](./../media/lab-03-ex-01-task-02-download-support-files.png) 92 | 93 | This will download a file named `Lab 03 - Import Sales Stats_support_live.zip` to your local computer. Once the download is complete, upload the file to the Azure Cloud Shell. Select the `Upload/download files` option from the Cloud Shell toolbar and select the ZIP file to upload. 94 | 95 | ![Upload ZIP file to Cloud Shell](./../media/lab-03-ex-01-task-02-upload-file.png) 96 | 97 | Once the upload is complete, go back to the home folder (run `cd ..`), run the `dir` command to ensure the file exists in your home folder. 98 | 99 | ![Check uploaded file](./../media/lab-03-ex-01-task-02-check-uploaded-file.png) 100 | 101 | Next, run the following command to extract the content of the archive: 102 | 103 | ```powershell 104 | Expand-Archive "Lab 03 - Import Sales Stats_support_live.zip" -DestinationPath ".\adfartifacts" -Force 105 | ``` 106 | 107 | Check the content of the various files that have been extracted. The ADF artifact definitions share the same schema with Synapse Analytics, so they can be imported into the Synapse Analytics workspace without any change. 108 | 109 | Use the `Get-Content` command to display the content of files in the `adfartifacts` folder: 110 | 111 | ```powershell 112 | Get-Content -Path ".\adfartifacts\linkedService\asagakeyvault01_adf.json" 113 | ``` 114 | 115 | ![Display JSON file content](./../media/lab-03-ex-01-task-02-display-file-content.png) 116 | 117 | Next, you will use these files to create the Synapse Analytics equivalents of the ADF v2 objects. 118 | 119 | ## Exercise 2 - Import a scripted ADF pipeline into Azure Synapse 120 | 121 | In this exercise you will use the exported ADF v2 artifacts (in the form of JSON files) to create their Synapse Analytics equivalents. 122 | 123 | ### Task 1 - Import linked services 124 | 125 | To make sure the current subscription content is correctly set, first run the following command: 126 | 127 | ```powershell 128 | $context = Get-AzContext 129 | $context 130 | ``` 131 | 132 | If the subscription name displayed does not match the name of the subscription where your Synapse Analytics workspace was created, run the following command (do not forget to replace `` with the name of your subscription): 133 | 134 | ```powershell 135 | $context = Select-AzSubscription -SubscriptionName "" 136 | $context 137 | ``` 138 | 139 | Before starting to import Azure Synapse Analytics artifacts, you need to make sure you have the `Az.Synapse` PowerShell module installed in your Cloud Shell. Run the following command to install the module: 140 | 141 | ```powershell 142 | Install-Module Az.Synapse 143 | Import-Module Az.Synapse 144 | ``` 145 | 146 | Run the following commands to create the three linked services in Synapse Analytics (remember to replace `` with the value you specified during the Synapse Analytics workspace deployment): 147 | 148 | ```powershell 149 | 150 | Set-AzSynapseLinkedService -WorkspaceName asagaworkspace -Name asagakeyvault01_adf -DefinitionFile ".\adfartifacts\linkedService\asagakeyvault01_adf.json" -DefaultProfile $context 151 | Set-AzSynapseLinkedService -WorkspaceName asagaworkspace -Name asagadatalake01_adf -DefinitionFile ".\adfartifacts\linkedService\asagadatalake01_adf.json" -DefaultProfile $context 152 | Set-AzSynapseLinkedService -WorkspaceName asagaworkspace -Name asagasqlpool01_adf -DefinitionFile ".\adfartifacts\linkedService\asagasqlpool01_adf.json" -DefaultProfile $context 153 | 154 | ``` 155 | 156 | After each command, you should see an output similar to this: 157 | 158 | ![Synapse Analytics artifact created](./../media/lab-03-ex-02-task-01-create-synapse-artifact.png) 159 | 160 | ### Task 2 - Import datasets 161 | 162 | Run the following commands to create the two datasets in Synapse Analytics (remember to replace `` with the value you specified during the Synapse Analytics workspace deployment): 163 | 164 | ```powershell 165 | Set-AzSynapseDataset -WorkspaceName asagaworkspace -Name wwi02_sale_small_stats_adls -DefinitionFile ".\adfartifacts\dataset\wwi02_sale_small_stats_adls.json" -DefaultProfile $context 166 | Set-AzSynapseDataset -WorkspaceName asagaworkspace -Name wwi02_sale_small_stats_asa -DefinitionFile ".\adfartifacts\dataset\wwi02_sale_small_stats_asa.json" -DefaultProfile $context 167 | ``` 168 | 169 | ### Task 3 - Import a pipeline 170 | 171 | Run the following command to create the pipeline in Synapse Analytics (remember to replace `` with the value you specified during the Synapse Analytics workspace deployment): 172 | 173 | ```powershell 174 | Set-AzSynapsePipeline -WorkspaceName asagaworkspace -Name "Lab 03 - Import Sales Stats" -DefinitionFile ".\adfartifacts\pipeline\Lab 03 - Import Sales Stats.json" -DefaultProfile $context 175 | ``` 176 | 177 | ## Exercise 3 - Test the imported pipeline 178 | 179 | In this exercise, you will test the imported Synapse Analytics pipeline in ADF. 180 | 181 | ### Task 1 - Validate and run the imported pipeline 182 | 183 | In Synapse Studio, run the `Lab 03 - Import Sale Stats` pipeline. 184 | 185 | ## After the hands-on lab 186 | 187 | Follow the instructions in [Clean-up your subscription](./../setup/cleanup.md) to clean-up your environment after the hands-on lab. 188 | 189 | ## Resources 190 | 191 | To learn more about the topics covered in this lab, use these resources: 192 | 193 | - [Azure.Analytics.Synapse.Artifacts Namespace 194 | Classes](https://docs.microsoft.com/en-us/dotnet/api/azure.analytics.synapse.artifacts?view=azure-dotnet-preview) 195 | 196 | ## Report issues 197 | 198 | In case you encounter any issues with the content in this repository, please follow the [How to report issues](./../../report-issues.md) guideline. We will try to address them as soon as possible. Please check your open issues to learn about their status. 199 | -------------------------------------------------------------------------------- /hands-on-labs/lab-04/README.md: -------------------------------------------------------------------------------- 1 | # Lab 04 - Working with the Common Data Model in Synapse Analytics 2 | 3 | This lab demonstrates using Synapse Analytics Spark to work with the Common Data Model. 4 | 5 | After completing the lab, you will know how to interact with the Common Data Model in Synapse Analytics. 6 | 7 | - [Lab 04 - Working with the Common Data Model in Synapse Analytics](#lab-04---working-with-the-common-data-model-in-synapse-analytics) 8 | - [Before the hands-on lab](#before-the-hands-on-lab) 9 | - [Task 1 - Create and configure the Azure Synapse Analytics workspace](#task-1---create-and-configure-the-azure-synapse-analytics-workspace) 10 | - [Task 2 - Create and configure additional resources for this lab](#task-2---create-and-configure-additional-resources-for-this-lab) 11 | - [Exercise 1 - Read data that exists in the CDM](#exercise-1---read-data-that-exists-in-the-cdm) 12 | - [Task 1 - Explore CDM data in the Synapse workspace data lake](#task-1---explore-cdm-data-in-the-synapse-workspace-data-lake) 13 | - [Task 2 - Load Spark dataframes from entities in CDM folders](#task-2---load-spark-dataframes-from-entities-in-cdm-folders) 14 | - [Exercise 2 - Update data that exists in the CDM](#exercise-2---update-data-that-exists-in-the-cdm) 15 | - [Task 1 - Update CDM data using dataframe schemas](#task-1---update-cdm-data-using-dataframe-schemas) 16 | - [Task 2 - Update CDM data using CDM entity definitions](#task-2---update-cdm-data-using-cdm-entity-definitions) 17 | - [Task 3 - Use Spark to transform incoming raw data to CDM data](#task-3---use-spark-to-transform-incoming-raw-data-to-cdm-data) 18 | - [Task 3 A - creating CDM with implicit manifest](#task-3-a---creating-cdm-with-implicit-manifest) 19 | - [Task 3 B - creating CDM with explicit manifest](#task-3-b---creating-cdm-with-explicit-manifest) 20 | - [Exercise 3 - Create an end-to-end CDM-based data pipeline](#exercise-3---create-an-end-to-end-cdm-based-data-pipeline) 21 | - [Task 1 - Automate raw data ingestion and coversion to CDM using Synapse Pipelines](#task-1---automate-raw-data-ingestion-and-coversion-to-cdm-using-synapse-pipelines) 22 | - [After the hands-on lab](#after-the-hands-on-lab) 23 | - [Resources](#resources) 24 | - [Report issues](#report-issues) 25 | 26 | ## Before the hands-on lab 27 | 28 | Before stepping through the exercises in this lab, make sure you have properly configured your Azure Synapse Analytics workspace. Perform the tasks below to configure the workspace. 29 | 30 | ### Task 1 - Create and configure the Azure Synapse Analytics workspace 31 | 32 | >**NOTE** 33 | > 34 | >If you have already created and configured the Synapse Analytics workspace while running one of the other labs available in this repo, you must not perform this task again and you can move on to the next task. The labs are designed to share the Synapse Analytics workspace, so you only need to create it once. 35 | 36 | Follow the instructions in [Deploy your Azure Synapse Analytics workspace](./../setup/asa-workspace-deploy.md) to create and configure the workspace. 37 | 38 | ### Task 2 - Create and configure additional resources for this lab 39 | 40 | Details coming soon ... 41 | 42 | ## Exercise 1 - Read data that exists in the CDM 43 | 44 | ### Task 1 - Explore CDM data in the Synapse workspace data lake 45 | 46 | Our CDM data is already stored on Azure Data Lake Gen 2. 47 | 48 | We have to open the corresponding container/folder: 49 | 50 | ![Open the DataLake folder with CDM data](./../media/lab-04/lab04-ex1-task1-read-cdm-01.png) 51 | 52 | To view the actual data, we can use a SQL query. Right click on the corresponding data folder and choose `New SQL Script` then choose `Select TOP 100 rows` 53 | 54 | ![Note the presence of the CDM manifest](./../media/lab-04/lab04-ex1-task1-read-cdm-02.png) 55 | 56 | You may right click and choose `Preview` to visualize contents for the `default.manifest.cdm.json` file. 57 | 58 | This file points to our entity which is part of our subfolder `SaleSmall`, see below. 59 | 60 | ![Note the presence of the CDM entity](./../media/lab-04/lab04-ex1-task1-read-cdm-03.png) 61 | 62 | You may right click and choose `Preview` to visualize contents for the `SaleSmall.cdm.json` file. 63 | 64 | ![Check the manifest of our CDM entity](./../media/lab-04/lab04-ex1-task1-read-cdm-04.png) 65 | 66 | The data itself is stored in a subfolder: `2020-12-12`, see below. 67 | 68 | ![Use SQL query to select CDM data](./../media/lab-04/lab04-ex1-task1-read-cdm-05.png) 69 | 70 | Click the `Run` button in the toolbar to execute the SQL query. Once executed, results are visible in the lower pane. 71 | 72 | ![View the available CDM data](./../media/lab-04/lab04-ex1-task1-read-cdm-06.png) 73 | 74 | Alternately we can load the CDM data in a Spark dataframe 75 | 76 | ![Use spark to select CDM data](./../media/lab-04/lab04-ex1-task1-read-cdm-07.png) 77 | 78 | Choose a Spark pool first, then click the `Run` button in the toolbar to run the notebook. Once executed, results are visible in the lower pane. 79 | 80 | ![View the available CDM data](./../media/lab-04/lab04-ex1-task1-read-cdm-08.png) 81 | 82 | ### Task 2 - Load Spark dataframes from entities in CDM folders 83 | 84 | We will load the existing CDM data into Spark dataframes. Open a notebook and use the following python code to load the CDM data: 85 | 86 | ```python 87 | from pyspark.sql.types import * 88 | from pyspark.sql import functions, Row 89 | from decimal import Decimal 90 | from datetime import datetime 91 | 92 | 93 | storageAccountName = "asadatalake01.dfs.core.windows.net" 94 | container = "wwi-02" 95 | outputContainer = "wwi-02" 96 | 97 | abfssRoot = "abfss://" + outputContainer + "@" + storageAccountName 98 | 99 | folder1 = "/cdm-data/input" 100 | 101 | #read CDM entities 102 | df = (spark.read.format("com.microsoft.cdm") 103 | .option("storage", storageAccountName) 104 | .option("manifestPath", container + folder1 + "/salesmall/default.manifest.cdm.json") 105 | .option("entity", "SaleSmall") 106 | .load()) 107 | 108 | df.printSchema() 109 | 110 | df.select("*").show() 111 | ``` 112 | 113 | The loaded CDM data is now displayed as spark dataframes in the lower pane. 114 | 115 | The dataframe schema matches the structure of our CDM entity. 116 | 117 | ![Load the available CDM data via Spark](./../media/lab-04/lab04-ex1-task2-spark-read-cdm-01.png) 118 | 119 | Note how our manifest file `default.manifest.cdm.json` points to our model 120 | 121 | ```json 122 | ... 123 | "entities" : [ 124 | { 125 | "type" : "LocalEntity", 126 | "entityName" : "SaleSmall", 127 | "entityPath" : "SaleSmall/SaleSmall.cdm.json/SaleSmall", 128 | } 129 | ] 130 | ... 131 | ``` 132 | 133 | which in turn contains our entity's definition `SaleSmall`. 134 | 135 | Some contents of `SaleSmall.cdm.json` is skipped for brevity: 136 | 137 | ```json 138 | { 139 | "definitions" : [ 140 | { 141 | "entityName" : "SaleSmall", 142 | "attributeContext" : { 143 | "type" : "entity", 144 | "name" : "SaleSmall", 145 | "definition" : "resolvedFrom/SaleSmall", 146 | "contents" : [ 147 | ... 148 | ] 149 | }, 150 | "hasAttributes" : [ 151 | { 152 | "name" : "TransactionId", 153 | "dataFormat" : "String" 154 | }, 155 | { 156 | "name" : "CustomerId", 157 | "dataFormat" : "Int32" 158 | }, 159 | { 160 | "name" : "ProductId", 161 | "dataFormat" : "Int16" 162 | }, 163 | { 164 | "name" : "Quantity", 165 | "dataFormat" : "Byte" 166 | }, 167 | { 168 | "name" : "Price", 169 | "dataFormat" : "Decimal" 170 | }, 171 | { 172 | "name" : "TotalAmount", 173 | "dataFormat" : "Decimal" 174 | }, 175 | { 176 | "name" : "TransactionDate", 177 | "dataFormat" : "Int32" 178 | }, 179 | { 180 | "name" : "ProfitAmount", 181 | "dataFormat" : "Decimal" 182 | }, 183 | { 184 | "name" : "Hour", 185 | "dataFormat" : "Byte" 186 | }, 187 | { 188 | "name" : "Minute", 189 | "dataFormat" : "Byte" 190 | }, 191 | { 192 | "name" : "StoreId", 193 | "dataFormat" : "Int16" 194 | } 195 | ] 196 | } 197 | ] 198 | } 199 | ``` 200 | 201 | ## Exercise 2 - Update data that exists in the CDM 202 | 203 | ### Task 1 - Update CDM data using dataframe schemas 204 | 205 | We will update existing CDM data by modifying the Spark dataframe schema. 206 | 207 | ```python 208 | from pyspark.sql.types import * 209 | from pyspark.sql import functions as f, Row 210 | 211 | from decimal import Decimal 212 | from datetime import datetime 213 | 214 | 215 | storageAccountName = "asadatalake01.dfs.core.windows.net" 216 | container = "wwi-02" 217 | outputContainer = "wwi-02" 218 | 219 | abfssRoot = "abfss://" + outputContainer + "@" + storageAccountName 220 | 221 | folder1 = "/cdm-data/input" 222 | folder2 = "/cdm-data/update-implicit" 223 | 224 | #read CDM 225 | df = (spark.read.format("com.microsoft.cdm") 226 | .option("storage", storageAccountName) 227 | .option("manifestPath", container + folder1 + "/salesmall/default.manifest.cdm.json") 228 | .option("entity", "SaleSmall") 229 | .load()) 230 | 231 | df.printSchema() 232 | df.select("*").show() 233 | 234 | #update dataframe/schema 235 | df2 = df.withColumn("x4", f.lit(0)) 236 | df2.printSchema() 237 | df2.select("*").show() 238 | 239 | #write CDM; entity manifest is implicitly created based on df schema 240 | (df2.write.format("com.microsoft.cdm") 241 | .option("storage", storageAccountName) 242 | .option("manifestPath", container + folder2 + "/salesmall/default.manifest.cdm.json") 243 | .option("entity", "SaleSmall") 244 | .option("format", "parquet") 245 | .option("compression", "gzip") 246 | .save()) 247 | 248 | readDf2 = (spark.read.format("com.microsoft.cdm") 249 | .option("storage", storageAccountName) 250 | .option("manifestPath", container + folder2 + "/salesmall/default.manifest.cdm.json") 251 | .option("entity", "SaleSmall") 252 | .load()) 253 | 254 | readDf2.select("*").show() 255 | ``` 256 | 257 | This is now the new schema as shown by the spark dataframe: 258 | 259 | ![New df schema](./../media/lab-04/lab04-ex2-task1-update-cdm-01.png) 260 | 261 | Note that the CDM manifest was updated: 262 | 263 | ![New CDM manifest](./../media/lab-04/lab04-ex2-task1-update-cdm-02.png) 264 | 265 | ### Task 2 - Update CDM data using CDM entity definitions 266 | 267 | We will update existing CDM data by using a modified CDM manifest. 268 | 269 | ```python 270 | from pyspark.sql.types import * 271 | from pyspark.sql import functions as f, Row 272 | 273 | from decimal import Decimal 274 | from datetime import datetime 275 | 276 | 277 | storageAccountName = "asadatalake01.dfs.core.windows.net" 278 | container = "wwi-02" 279 | outputContainer = "wwi-02" 280 | 281 | abfssRoot = "abfss://" + outputContainer + "@" + storageAccountName 282 | 283 | folder1 = "/cdm-data/input" 284 | folder2 = "/cdm-data/update-explicit" 285 | 286 | #read CDM 287 | df = (spark.read.format("com.microsoft.cdm") 288 | .option("storage", storageAccountName) 289 | .option("manifestPath", container + folder1 + "/salesmall/default.manifest.cdm.json") 290 | .option("entity", "SaleSmall") 291 | .load()) 292 | 293 | df.printSchema() 294 | df.select("*").show() 295 | 296 | #update dataframe/schema 297 | df2 = df.withColumn("x5", f.lit(0)) 298 | df2.printSchema() 299 | df2.select("*").show() 300 | 301 | #write CDM; entity manifest is explicitly passed as SaleSmall-Modified.cdm.json 302 | (df2.write.format("com.microsoft.cdm") 303 | .option("storage", storageAccountName) 304 | .option("manifestPath", container + folder2 + "/salesmall/default.manifest.cdm.json") 305 | .option("entity", "SaleSmall") 306 | .option("entityDefinitionModelRoot", container + folder1) #root folder for our own CDM models 307 | .option("entityDefinitionPath", "/salesmall/SaleSmall/SaleSmall-Modified.cdm.json/SaleSmall") #relative path for our own CDM model 308 | .option("format", "parquet") 309 | .option("compression", "gzip") 310 | .save()) 311 | 312 | readDf2 = (spark.read.format("com.microsoft.cdm") 313 | .option("storage", storageAccountName) 314 | .option("manifestPath", container + folder2 + "/salesmall/default.manifest.cdm.json") 315 | .option("entity", "SaleSmall") 316 | .load()) 317 | 318 | readDf2.select("*").show() 319 | ``` 320 | 321 | This is now the new schema as shown by the spark dataframe: 322 | 323 | ![New df schema](./../media/lab-04/lab04-ex2-task2-update-cdm-01.png) 324 | 325 | Note that the CDM manifest contains a member that matches our new column. 326 | 327 | ![New CDM manifest](./../media/lab-04/lab04-ex2-task2-update-cdm-02.png) 328 | 329 | 330 | 331 | ### Task 3 - Use Spark to transform incoming raw data to CDM data 332 | 333 | #### Task 3 A - creating CDM with implicit manifest 334 | 335 | We will write new CDM data based on existing Spark dataframes. 336 | 337 | We will infer the CDM manifest based on the dataframe's schema. 338 | 339 | Open a notebook and use the following python code to save the CDM data: 340 | 341 | ```python 342 | storageAccountName = "asadatalake01.dfs.core.windows.net" 343 | container = "wwi-02" 344 | outputContainer = "wwi-02" 345 | 346 | abfssRoot = "abfss://" + outputContainer + "@" + storageAccountName 347 | 348 | # WARNING: if output folder exists, writer will fail with a java.lang.NullPointerException 349 | folder1 = "/cdm-data/output-implicit" 350 | 351 | from pyspark.sql.types import * 352 | from pyspark.sql import functions, Row 353 | from decimal import Decimal 354 | from datetime import datetime 355 | 356 | df = spark.read.parquet(abfssRoot + "/sale-small/Year=2019/Quarter=Q4/Month=12/*/*.parquet") 357 | df.show(10) 358 | df.printSchema() 359 | 360 | #save dataframe using CDM format; entity manifest is implicitly created based on df schema 361 | (df.write.format("com.microsoft.cdm") 362 | .option("storage", storageAccountName) 363 | .option("manifestPath", container + folder1 + "/salesmall/default.manifest.cdm.json") 364 | .option("entity", "SaleSmall") 365 | .option("format", "parquet") 366 | .option("compression", "gzip") 367 | .save()) 368 | 369 | readDf = (spark.read.format("com.microsoft.cdm") 370 | .option("storage", storageAccountName) 371 | .option("manifestPath", container + folder1 + "/salesmall/default.manifest.cdm.json") 372 | .option("entity", "SaleSmall") 373 | .load()) 374 | 375 | readDf.select("*").show() 376 | ``` 377 | ![Save new CDM data, use df schema](./../media/lab-04/lab04-ex2-task3-spark-write-cdm-01.png) 378 | 379 | 380 | #### Task 3 B - creating CDM with explicit manifest 381 | 382 | Writing new CDM data based on existing CDM models is just as easy. 383 | 384 | The CDM manifest is explicit here, user specified. 385 | 386 | Open a notebook and use the following python code to save the CDM data: 387 | 388 | ```python 389 | storageAccountName = "asadatalake01.dfs.core.windows.net" 390 | container = "wwi-02" 391 | outputContainer = "wwi-02" 392 | 393 | abfssRoot = "abfss://" + outputContainer + "@" + storageAccountName 394 | 395 | # WARNING: if output folder exists, writer will fail with a java.lang.NullPointerException 396 | folderInput = "/cdm-data/input" 397 | folder2 = "/cdm-data/output-explicit" 398 | 399 | from pyspark.sql.types import * 400 | from pyspark.sql import functions, Row 401 | from decimal import Decimal 402 | from datetime import datetime 403 | 404 | df2 = spark.read.parquet(abfssRoot + "/sale-small/Year=2019/Quarter=Q4/Month=12/*/*.parquet") 405 | df2.show(10) 406 | df2.printSchema() 407 | 408 | #save dataframe using CDM format; entity manifest is explicitly passed as SaleSmall/SaleSmall.cdm.json 409 | (df2.write.format("com.microsoft.cdm") 410 | .option("storage", storageAccountName) 411 | .option("manifestPath", container + folder2 + "/salesmall/default.manifest.cdm.json") 412 | .option("entity", "SaleSmall") 413 | .option("entityDefinitionModelRoot", container + folderInput) #root folder for our own CDM models 414 | .option("entityDefinitionPath", "/salesmall/SaleSmall/SaleSmall.cdm.json/SaleSmall") #relative path for our own CDM model 415 | .mode("overwrite") 416 | .save()) 417 | 418 | readDf2 = (spark.read.format("com.microsoft.cdm") 419 | .option("storage", storageAccountName) 420 | .option("manifestPath", container + folder2 + "/salesmall/default.manifest.cdm.json") 421 | .option("entity", "SaleSmall") 422 | .option("entityDefinitionModelRoot", container + folderInput) 423 | .load()) 424 | 425 | readDf2.select("*").show() 426 | ``` 427 | 428 | ![Save new CDM data, use CDM model](./../media/lab-04/lab04-ex2-task3-spark-write-cdm-02.png) 429 | 430 | ## Exercise 3 - Create an end-to-end CDM-based data pipeline 431 | 432 | ### Task 1 - Automate raw data ingestion and coversion to CDM using Synapse Pipelines 433 | 434 | ![Create a new pipeline](./../media/lab-04/lab04-ex3-task1-pipeline-cdm-01.png) 435 | 436 | Go to the `Integrate` hub, press the `+` button then choose `Pipeline` to create a new pipeline. 437 | 438 | ![Configure the pipeline](./../media/lab-04/lab04-ex3-task1-pipeline-cdm-02.png) 439 | 440 | Name your pipeline, then pick an activity from the `Activities` list: drag a `Notebook` element found under the `Synapse` group. 441 | 442 | ![Configure the activity](./../media/lab-04/lab04-ex3-task1-pipeline-cdm-03.png) 443 | 444 | In the lower pane, choose the `Settings` tab and select the notebook you want to be used by our pipeline. 445 | 446 | In our example we chose the notebook that converts raw data to cdm format. 447 | 448 | ![Automate the pipeline](./../media/lab-04/lab04-ex3-task1-pipeline-cdm-04.png) 449 | 450 | Press `Add trigger` to use a condition that will automatically launch your new pipeline. 451 | 452 | ![Creating the trigger](./../media/lab-04/lab04-ex3-task1-pipeline-cdm-05.png) 453 | 454 | Press `New` to create a new trigger. 455 | 456 | ![Configure the trigger](./../media/lab-04/lab04-ex3-task1-pipeline-cdm-06.png) 457 | 458 | * Name your trigger, and choose its type as `Event`. 459 | * Choose the Azure subscription, Storage account name, Container name. 460 | * Setup a path prefix that will be used to detect changes in our datalake in a specific location. 461 | * Check the `Blob created` Event to make sure that pipeline is launched when new blob appears in your blob path. 462 | 463 | Save your trigger, then click on the `Validate` button and then press `Publish` to save your new pipeline. 464 | 465 | We chose to use a trigger that detects changes in our datalake. 466 | Which means that everytime we get new data in our datalake, our pipeline will run. 467 | In our case, the notebook will be executed, and will convert the raw data to the CDM format. 468 | 469 | ## After the hands-on lab 470 | 471 | Follow the instructions in [Clean-up your subscription](./../setup/cleanup.md) to clean-up your environment after the hands-on lab. 472 | 473 | ## Resources 474 | 475 | To learn more about the topics covered in this lab, use these resources: 476 | 477 | - [Azure.Analytics.Synapse.Artifacts Namespace Classes](https://docs.microsoft.com/en-us/dotnet/api/azure.analytics.synapse.artifacts?view=azure-dotnet-preview) 478 | - [Using the Spark CDM Connector](https://github.com/Azure/spark-cdm-connector/blob/master/documentation/overview.md) 479 | - [Common Data Model (CDM) Schema](https://github.com/microsoft/CDM) 480 | 481 | ## Report issues 482 | 483 | In case you encounter any issues with the content in this repository, please follow the [How to report issues](./../../report-issues.md) guideline. We will try to address them as soon as possible. Please check your open issues to learn about their status. 484 | -------------------------------------------------------------------------------- /hands-on-labs/media/asaworkspace-deploy-configure.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/asaworkspace-deploy-configure.png -------------------------------------------------------------------------------- /hands-on-labs/media/asaworkspace-deploy-pbi-linked-service-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/asaworkspace-deploy-pbi-linked-service-01.png -------------------------------------------------------------------------------- /hands-on-labs/media/asaworkspace-deploy-pbi-linked-service-02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/asaworkspace-deploy-pbi-linked-service-02.png -------------------------------------------------------------------------------- /hands-on-labs/media/asaworkspace-deploy-pbi-linked-service-03.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/asaworkspace-deploy-pbi-linked-service-03.png -------------------------------------------------------------------------------- /hands-on-labs/media/asaworkspace-deploy-progress.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/asaworkspace-deploy-progress.png -------------------------------------------------------------------------------- /hands-on-labs/media/asaworkspace-deploy-tag.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/asaworkspace-deploy-tag.png -------------------------------------------------------------------------------- /hands-on-labs/media/cleanup-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/cleanup-01.png -------------------------------------------------------------------------------- /hands-on-labs/media/cleanup-02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/cleanup-02.png -------------------------------------------------------------------------------- /hands-on-labs/media/cloudshell-configure-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/cloudshell-configure-01.png -------------------------------------------------------------------------------- /hands-on-labs/media/cloudshell-configure-02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/cloudshell-configure-02.png -------------------------------------------------------------------------------- /hands-on-labs/media/cloudshell-configure-03.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/cloudshell-configure-03.png -------------------------------------------------------------------------------- /hands-on-labs/media/cloudshell-configure-04.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/cloudshell-configure-04.png -------------------------------------------------------------------------------- /hands-on-labs/media/cloudshell-create-storage-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/cloudshell-create-storage-01.png -------------------------------------------------------------------------------- /hands-on-labs/media/cloudshell-create-storage-02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/cloudshell-create-storage-02.png -------------------------------------------------------------------------------- /hands-on-labs/media/cloudshell-setup-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/cloudshell-setup-01.png -------------------------------------------------------------------------------- /hands-on-labs/media/cloudshell-setup-02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/cloudshell-setup-02.png -------------------------------------------------------------------------------- /hands-on-labs/media/cloudshell-setup-03.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/cloudshell-setup-03.png -------------------------------------------------------------------------------- /hands-on-labs/media/cloudshell-setup-04.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/cloudshell-setup-04.png -------------------------------------------------------------------------------- /hands-on-labs/media/issue-details.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/issue-details.png -------------------------------------------------------------------------------- /hands-on-labs/media/issue-find-section.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/issue-find-section.png -------------------------------------------------------------------------------- /hands-on-labs/media/issue-new-issue.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/issue-new-issue.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-deploy-configure.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-deploy-configure.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-deploy-progress.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-deploy-progress.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-01-task-01-configure-linked-service.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-01-task-01-configure-linked-service.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-01-task-01-keyvault-secret.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-01-task-01-keyvault-secret.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-01-task-01-mlworkspace-permissions.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-01-task-01-mlworkspace-permissions.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-01-task-01-new-linked-service.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-01-task-01-new-linked-service.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-01-task-01-publish-linked-service.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-01-task-01-publish-linked-service.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-01-task-01-service-principal-clientid.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-01-task-01-service-principal-clientid.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-01-task-01-service-principal.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-01-task-01-service-principal.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-01-task-02-create-spark-table.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-01-task-02-create-spark-table.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-01-task-02-ml-menu.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-01-task-02-ml-menu.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-02-task-01-ml-menu.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-02-task-01-ml-menu.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-02-task-01-model-type.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-02-task-01-model-type.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-02-task-01-open-in-notebook.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-02-task-01-open-in-notebook.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-02-task-01-regressio-model-configuration.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-02-task-01-regressio-model-configuration.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-02-task-01-started-notification.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-02-task-01-started-notification.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-02-task-01-submit-notification.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-02-task-01-submit-notification.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-02-task-01-trigger-experiment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-02-task-01-trigger-experiment.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-02-task-02-best-mode-explained.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-02-task-02-best-mode-explained.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-02-task-02-experiment-run.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-02-task-02-experiment-run.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-02-task-02-model-registry.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-02-task-02-model-registry.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-02-task-02-open-aml-workspace.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-02-task-02-open-aml-workspace.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-02-task-02-run-details.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-02-task-02-run-details.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-03-task-01-deploy-model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-03-task-01-deploy-model.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-03-task-01-explore-table.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-03-task-01-explore-table.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-03-task-01-forecast-stored-procedure.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-03-task-01-forecast-stored-procedure.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-03-task-01-map-columns.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-03-task-01-map-columns.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-03-task-01-run-forecast.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-03-task-01-run-forecast.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-03-task-01-select-model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-03-task-01-select-model.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-03-task-02-connect-to-model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-03-task-02-connect-to-model.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-03-task-02-new-spark-table.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-03-task-02-new-spark-table.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-03-task-02-text-analytics-code.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-03-task-02-text-analytics-code.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-03-task-02-text-analytics-mappings.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-03-task-02-text-analytics-mappings.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-03-task-02-text-analytics-model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-03-task-02-text-analytics-model.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-03-task-02-text-analytics-results.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-03-task-02-text-analytics-results.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-03-task-03-create-pipeline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-03-task-03-create-pipeline.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-03-task-03-pipeline-mapping.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-03-task-03-pipeline-mapping.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-03-task-03-pipeline-sink.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-03-task-03-pipeline-sink.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-03-task-03-pipeline-source.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-03-task-03-pipeline-source.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-03-task-03-pipeline-staging.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-03-task-03-pipeline-staging.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-03-task-03-pipeline-stored-procedure-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-03-task-03-pipeline-stored-procedure-01.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-03-task-03-pipeline-stored-procedure-02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-03-task-03-pipeline-stored-procedure-02.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-03-task-03-pipeline-test.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-03-task-03-pipeline-test.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-04-task-01-credentials-in-power-bi-desktop.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-04-task-01-credentials-in-power-bi-desktop.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-04-task-01-server-in-power-bi-desktop.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-04-task-01-server-in-power-bi-desktop.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-04-task-01-view-report.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-04-task-01-view-report.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-01-ex-04-task-02-create-trigger.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-01-ex-04-task-02-create-trigger.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-deploy-configure.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-deploy-configure.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-deploy-progress.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-deploy-progress.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-01-task-01-configure-linked-service.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-01-task-01-configure-linked-service.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-01-task-01-database-permissions.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-01-task-01-database-permissions.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-01-task-01-keyvault-secret.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-01-task-01-keyvault-secret.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-01-task-01-new-linked-service.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-01-task-01-new-linked-service.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-01-task-01-publish-linked-service.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-01-task-01-publish-linked-service.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-01-task-01-service-principal-clientid.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-01-task-01-service-principal-clientid.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-01-task-01-service-principal.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-01-task-01-service-principal.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-01-task-02-load-table.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-01-task-02-load-table.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-01-task-02-notebook-from-table.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-01-task-02-notebook-from-table.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-02-task-01-load-sales-telemetry.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-02-task-01-load-sales-telemetry.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-02-task-02-create-indexes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-02-task-02-create-indexes.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-02-task-02-explain-hyperspace-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-02-task-02-explain-hyperspace-01.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-02-task-02-explain-hyperspace-02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-02-task-02-explain-hyperspace-02.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-02-task-02-explain-hyperspace-03.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-02-task-02-explain-hyperspace-03.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-02-task-02-explain-hyperspace-04.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-02-task-02-explain-hyperspace-04.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-02-task-02-initialize-hyperspace.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-02-task-02-initialize-hyperspace.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-03-task-01-execute-sql-script.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-03-task-01-execute-sql-script.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-03-task-01-new-sql-script.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-03-task-01-new-sql-script.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-03-task-02-credentials-in-power-bi-desktop.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-03-task-02-credentials-in-power-bi-desktop.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-03-task-02-server-in-power-bi-desktop.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-03-task-02-server-in-power-bi-desktop.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-02-ex-03-task-02-view-report.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-02-ex-03-task-02-view-report.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-03-deploy-configure.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-03-deploy-configure.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-03-deploy-progress.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-03-deploy-progress.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-03-ex-01-task-01-run-adf-pipeline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-03-ex-01-task-01-run-adf-pipeline.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-03-ex-01-task-01-view-pipeline-result.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-03-ex-01-task-01-view-pipeline-result.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-03-ex-01-task-02-check-uploaded-file.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-03-ex-01-task-02-check-uploaded-file.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-03-ex-01-task-02-create-adfartifacts-folder.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-03-ex-01-task-02-create-adfartifacts-folder.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-03-ex-01-task-02-display-file-content.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-03-ex-01-task-02-display-file-content.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-03-ex-01-task-02-download-support-files.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-03-ex-01-task-02-download-support-files.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-03-ex-01-task-02-upload-file.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-03-ex-01-task-02-upload-file.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-03-ex-02-task-01-create-synapse-artifact.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-03-ex-02-task-01-create-synapse-artifact.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex1-task1-read-cdm-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex1-task1-read-cdm-01.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex1-task1-read-cdm-02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex1-task1-read-cdm-02.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex1-task1-read-cdm-03.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex1-task1-read-cdm-03.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex1-task1-read-cdm-04.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex1-task1-read-cdm-04.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex1-task1-read-cdm-05.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex1-task1-read-cdm-05.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex1-task1-read-cdm-06.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex1-task1-read-cdm-06.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex1-task1-read-cdm-07.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex1-task1-read-cdm-07.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex1-task1-read-cdm-08.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex1-task1-read-cdm-08.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex1-task2-spark-read-cdm-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex1-task2-spark-read-cdm-01.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex2-task1-update-cdm-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex2-task1-update-cdm-01.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex2-task1-update-cdm-02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex2-task1-update-cdm-02.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex2-task2-update-cdm-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex2-task2-update-cdm-01.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex2-task2-update-cdm-02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex2-task2-update-cdm-02.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex2-task3-spark-write-cdm-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex2-task3-spark-write-cdm-01.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex2-task3-spark-write-cdm-02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex2-task3-spark-write-cdm-02.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex3-task1-pipeline-cdm-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex3-task1-pipeline-cdm-01.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex3-task1-pipeline-cdm-02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex3-task1-pipeline-cdm-02.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex3-task1-pipeline-cdm-03.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex3-task1-pipeline-cdm-03.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex3-task1-pipeline-cdm-04.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex3-task1-pipeline-cdm-04.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex3-task1-pipeline-cdm-05.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex3-task1-pipeline-cdm-05.png -------------------------------------------------------------------------------- /hands-on-labs/media/lab-04/lab04-ex3-task1-pipeline-cdm-06.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/lab-04/lab04-ex3-task1-pipeline-cdm-06.png -------------------------------------------------------------------------------- /hands-on-labs/media/video-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/video-01.png -------------------------------------------------------------------------------- /hands-on-labs/media/video-02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/video-02.png -------------------------------------------------------------------------------- /hands-on-labs/media/video-03.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/hands-on-labs/media/video-03.png -------------------------------------------------------------------------------- /hands-on-labs/setup/arm/asaga-workspace-core.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#", 3 | "contentVersion": "1.0.0.0", 4 | "parameters": { 5 | "uniqueSuffix": { 6 | "type": "String", 7 | "metadata": { 8 | "description": "Suffix added to all resource name to make them unique." 9 | } 10 | }, 11 | "sqlAdministratorLoginPassword": { 12 | "type": "securestring", 13 | "metadata": { 14 | "description": "Password for SQL Admin" 15 | } 16 | } 17 | }, 18 | "variables": { 19 | "location": "[resourceGroup().location]", 20 | "sqlAdministratorLogin": "asaga.sql.admin", 21 | "workspaceName": "[concat('asagaworkspace', parameters('uniqueSuffix'))]", 22 | "adlsStorageAccountName": "[concat('asagadatalake', parameters('uniqueSuffix'))]", 23 | "defaultDataLakeStorageFilesystemName": "workspace", 24 | "sqlPoolName": "SQLPool01", 25 | "sparkPoolName": "SparkPool01", 26 | "sqlServerSKU": "DW100c", 27 | "storageBlobDataContributorRoleID": "ba92f5b4-2d11-453d-a403-e96b0029c9fe", 28 | "storageBlobDataOwnerRoleID": "b7e6dc6d-f1e8-4753-8033-0f276bb0955b", 29 | "defaultDataLakeStorageAccountUrl": "[concat('https://', variables('adlsStorageAccountName'), '.dfs.core.windows.net')]", 30 | "sparkAutoScaleEnabled": "true", 31 | "sparkMinNodeCount": "3", 32 | "sparkMaxNodeCount": "4", 33 | "sparkNodeCount": "0", 34 | "sparkNodeSizeFamily": "MemoryOptimized", 35 | "sparkNodeSize": "Small", 36 | "sparkAutoPauseEnabled": "true", 37 | "sparkAutoPauseDelayInMinutes": "15", 38 | "sparkVersion": "2.4", 39 | "keyVaultName": "[concat('asagakeyvault', parameters('uniqueSuffix'))]" 40 | }, 41 | "resources": [ 42 | { 43 | "type": "Microsoft.KeyVault/vaults", 44 | "apiVersion": "2018-02-14", 45 | "name": "[variables('keyVaultName')]", 46 | "location": "[variables('location')]", 47 | "properties": { 48 | "tenantId": "[subscription().tenantId]", 49 | "sku": { 50 | "name": "standard", 51 | "family": "A" 52 | }, 53 | "accessPolicies": [] 54 | } 55 | }, 56 | { 57 | "type": "Microsoft.KeyVault/vaults/secrets", 58 | "apiVersion": "2016-10-01", 59 | "name": "[concat(variables('keyVaultName'), '/SQL-USER-ASA')]", 60 | "dependsOn": [ 61 | "[resourceId('Microsoft.KeyVault/vaults', variables('keyVaultName'))]" 62 | ], 63 | "properties": { 64 | "contentType": "text/plain", 65 | "value": "[parameters('sqlAdministratorLoginPassword')]", 66 | "attributes": { 67 | "enabled": true 68 | } 69 | } 70 | }, 71 | { 72 | "type": "Microsoft.Storage/storageAccounts", 73 | "apiVersion": "2019-06-01", 74 | "name": "[variables('adlsStorageAccountName')]", 75 | "location": "[variables('location')]", 76 | "dependsOn": [], 77 | "tags": {}, 78 | "sku": { 79 | "name": "Standard_RAGRS", 80 | "tier": "Standard" 81 | }, 82 | "kind": "StorageV2", 83 | "properties": { 84 | "accessTier": "Hot", 85 | "supportsHttpsTrafficOnly": "true", 86 | "isHnsEnabled": "true", 87 | "largeFileSharesState": "Disabled" 88 | }, 89 | "resources": [ 90 | { 91 | "type": "blobServices/containers", 92 | "apiVersion": "2018-07-01", 93 | "name": "[concat('default/', variables('defaultDataLakeStorageFilesystemName'))]", 94 | "dependsOn": [ 95 | "[concat('Microsoft.Storage/storageAccounts/', variables('adlsStorageAccountName'))]" 96 | ] 97 | } 98 | ] 99 | }, 100 | { 101 | "type": "Microsoft.Storage/storageAccounts/blobServices", 102 | "apiVersion": "2019-06-01", 103 | "name": "[concat(variables('adlsStorageAccountName'), '/default')]", 104 | "dependsOn": [ 105 | "[resourceId('Microsoft.Storage/storageAccounts', variables('adlsStorageAccountName'))]" 106 | ], 107 | "sku": { 108 | "name": "Standard_RAGRS", 109 | "tier": "Standard" 110 | }, 111 | "properties": { 112 | "cors": { 113 | "corsRules": [] 114 | }, 115 | "deleteRetentionPolicy": { 116 | "enabled": false 117 | } 118 | } 119 | }, 120 | { 121 | "type": "Microsoft.Storage/storageAccounts/fileServices", 122 | "apiVersion": "2019-06-01", 123 | "name": "[concat(variables('adlsStorageAccountName'), '/default')]", 124 | "dependsOn": [ 125 | "[resourceId('Microsoft.Storage/storageAccounts', variables('adlsStorageAccountName'))]" 126 | ], 127 | "sku": { 128 | "name": "Standard_RAGRS", 129 | "tier": "Standard" 130 | }, 131 | "properties": { 132 | "cors": { 133 | "corsRules": [] 134 | } 135 | } 136 | }, 137 | { 138 | "type": "Microsoft.Synapse/workspaces", 139 | "apiVersion": "2019-06-01-preview", 140 | "name": "[variables('workspaceName')]", 141 | "location": "[variables('location')]", 142 | "dependsOn": [ 143 | "[concat('Microsoft.Storage/storageAccounts/', variables('adlsStorageAccountName'), '/blobServices/default/containers/', variables('defaultDataLakeStorageFilesystemName'))]" 144 | ], 145 | "identity": { 146 | "type": "SystemAssigned" 147 | }, 148 | "properties": { 149 | "managedResourceGroupName": "[concat(resourceGroup().name, '-internal')]", 150 | "defaultDataLakeStorage": { 151 | "accountUrl": "[variables('defaultDataLakeStorageAccountUrl')]", 152 | "filesystem": "[variables('defaultDataLakeStorageFilesystemName')]" 153 | }, 154 | "sqlAdministratorLogin": "[variables('sqlAdministratorLogin')]", 155 | "sqlAdministratorLoginPassword": "[parameters('sqlAdministratorLoginPassword')]" 156 | }, 157 | "resources": [ 158 | { 159 | "apiVersion": "2019-06-01-preview", 160 | "dependsOn": [ 161 | "[concat('Microsoft.Synapse/workspaces/', variables('workspaceName'))]" 162 | ], 163 | "location": "[variables('location')]", 164 | "name": "default", 165 | "properties": { 166 | "grantSqlControlToManagedIdentity": { 167 | "desiredState": "Enabled" 168 | } 169 | }, 170 | "type": "managedIdentitySqlControlSettings" 171 | } 172 | ] 173 | }, 174 | { 175 | "type": "Microsoft.Synapse/workspaces/firewallRules", 176 | "apiVersion": "2019-06-01-preview", 177 | "name": "[concat(variables('workspaceName'), '/allowAll')]", 178 | "dependsOn": [ 179 | "[concat('Microsoft.Synapse/workspaces/', variables('workspaceName'))]" 180 | ], 181 | "properties": { 182 | "startIpAddress": "0.0.0.0", 183 | "endIpAddress": "255.255.255.255" 184 | } 185 | }, 186 | { 187 | "type": "Microsoft.Synapse/workspaces/firewallRules", 188 | "apiVersion": "2019-06-01-preview", 189 | "name": "[concat(variables('workspaceName'), '/AllowAllWindowsAzureIps')]", 190 | "dependsOn": [ 191 | "[resourceId('Microsoft.Synapse/workspaces', variables('workspaceName'))]" 192 | ], 193 | "properties": { 194 | "startIpAddress": "0.0.0.0", 195 | "endIpAddress": "0.0.0.0" 196 | } 197 | }, 198 | { 199 | "type": "Microsoft.Synapse/workspaces/integrationruntimes", 200 | "apiVersion": "2019-06-01-preview", 201 | "name": "[concat(variables('workspaceName'), '/AutoResolveIntegrationRuntime')]", 202 | "dependsOn": [ 203 | "[concat('Microsoft.Synapse/workspaces/', variables('workspaceName'))]" 204 | ], 205 | "properties": { 206 | "type": "Managed", 207 | "typeProperties": { 208 | "computeProperties": { 209 | "location": "AutoResolve" 210 | } 211 | } 212 | } 213 | }, 214 | { 215 | "type": "Microsoft.Storage/storageAccounts/blobServices/containers/providers/roleAssignments", 216 | "apiVersion": "2018-09-01-preview", 217 | "name": "[concat(variables('adlsStorageAccountName'), '/default/', variables('defaultDataLakeStorageFilesystemName'), '/Microsoft.Authorization/', guid(concat(resourceGroup().id, '/', variables('storageBlobDataOwnerRoleID'), '/', variables('workspaceName'))))]", 218 | "location": "[variables('location')]", 219 | "dependsOn": [ 220 | "[concat('Microsoft.Synapse/workspaces/', variables('workspaceName'))]" 221 | ], 222 | "properties": { 223 | "roleDefinitionId": "[resourceId('Microsoft.Authorization/roleDefinitions', variables('storageBlobDataOwnerRoleID'))]", 224 | "principalId": "[reference(concat('Microsoft.Synapse/workspaces/', variables('workspaceName')), '2019-06-01-preview', 'Full').identity.principalId]", 225 | "principalType": "ServicePrincipal" 226 | } 227 | }, 228 | { 229 | "type": "Microsoft.Authorization/roleAssignments", 230 | "apiVersion": "2018-09-01-preview", 231 | "name": "[guid(concat(resourceGroup().id, '/', variables('storageBlobDataOwnerRoleID'), '/', variables('workspaceName'), '2'))]", 232 | "location": "[variables('location')]", 233 | "dependsOn": [ 234 | "[concat('Microsoft.Synapse/workspaces/', variables('workspaceName'))]" 235 | ], 236 | "properties": { 237 | "roleDefinitionId": "[resourceId('Microsoft.Authorization/roleDefinitions', variables('storageBlobDataOwnerRoleID'))]", 238 | "principalId": "[reference(concat('Microsoft.Synapse/workspaces/', variables('workspaceName')), '2019-06-01-preview', 'Full').identity.principalId]", 239 | "principalType": "ServicePrincipal" 240 | } 241 | }, 242 | { 243 | "type": "Microsoft.Synapse/workspaces/bigDataPools", 244 | "apiVersion": "2019-06-01-preview", 245 | "name": "[concat(variables('workspaceName'), '/', variables('sparkPoolName'))]", 246 | "location": "[variables('location')]", 247 | "dependsOn": [ 248 | "[concat('Microsoft.Synapse/workspaces/', variables('workspaceName'))]" 249 | ], 250 | "properties": { 251 | "nodeCount": "[variables('sparkNodeCount')]", 252 | "nodeSizeFamily": "[variables('sparkNodeSizeFamily')]", 253 | "nodeSize": "[variables('sparkNodeSize')]", 254 | "autoScale": { 255 | "enabled": "[variables('sparkAutoScaleEnabled')]", 256 | "minNodeCount": "[variables('sparkMinNodeCount')]", 257 | "maxNodeCount": "[variables('sparkMaxNodeCount')]" 258 | }, 259 | "autoPause": { 260 | "enabled": "[variables('sparkAutoPauseEnabled')]", 261 | "delayInMinutes": "[variables('sparkAutoPauseDelayInMinutes')]" 262 | }, 263 | "sparkVersion": "[variables('sparkVersion')]" 264 | } 265 | }, 266 | { 267 | "type": "Microsoft.Synapse/workspaces/sqlPools", 268 | "apiVersion": "2019-06-01-preview", 269 | "name": "[concat(variables('workspaceName'), '/', variables('sqlPoolName'))]", 270 | "location": "[variables('location')]", 271 | "dependsOn": [ 272 | "[concat('Microsoft.Synapse/workspaces/', variables('workspaceName'))]", 273 | "[concat('Microsoft.Synapse/workspaces/', variables('workspaceName'), '/bigDataPools/', variables('sparkPoolName'))]" 274 | ], 275 | "sku": { 276 | "name": "[variables('sqlServerSKU')]" 277 | }, 278 | "properties": { 279 | "createMode": "Default", 280 | "collation": "SQL_Latin1_General_CP1_CI_AS" 281 | } 282 | }, 283 | { 284 | "type": "Microsoft.Storage/storageAccounts/blobServices/containers", 285 | "apiVersion": "2019-06-01", 286 | "name": "[concat(variables('adlsStorageAccountName'), '/default/staging')]", 287 | "dependsOn": [ 288 | "[resourceId('Microsoft.Storage/storageAccounts/blobServices', variables('adlsStorageAccountName'), 'default')]", 289 | "[resourceId('Microsoft.Storage/storageAccounts', variables('adlsStorageAccountName'))]" 290 | ], 291 | "properties": { 292 | "publicAccess": "None" 293 | } 294 | }, 295 | { 296 | "type": "Microsoft.Storage/storageAccounts/blobServices/containers", 297 | "apiVersion": "2019-06-01", 298 | "name": "[concat(variables('adlsStorageAccountName'), '/default/wwi-02')]", 299 | "dependsOn": [ 300 | "[resourceId('Microsoft.Storage/storageAccounts/blobServices', variables('adlsStorageAccountName'), 'default')]", 301 | "[resourceId('Microsoft.Storage/storageAccounts', variables('adlsStorageAccountName'))]" 302 | ], 303 | "properties": { 304 | "publicAccess": "None" 305 | } 306 | } 307 | ], 308 | "outputs": {} 309 | } 310 | -------------------------------------------------------------------------------- /hands-on-labs/setup/arm/asaga-workspace-lab-01.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", 3 | "contentVersion": "1.0.0.0", 4 | "variables": { 5 | "location": "[resourceGroup().location]", 6 | "uniqueSuffix": "[resourceGroup().tags['DeploymentId']]", 7 | "workspaceName": "[concat('asagamachinelearning', variables('uniqueSuffix'))]", 8 | "sku": "enterprise", 9 | "storageAccountName": "[concat('asagastorage', variables('uniqueSuffix'))]", 10 | "storageAccountType": "Standard_LRS", 11 | "keyVaultName": "[concat('asagakeyvault', variables('uniqueSuffix'))]", 12 | "applicationInsightsName": "[concat('asagaappinsights', variables('uniqueSuffix'))]", 13 | "containerRegistryName": "[concat('asagacontainerregistry', variables('uniqueSuffix'))]", 14 | "tenantId": "[subscription().tenantId]", 15 | "storageAccount": "[resourceId(resourceGroup().name, 'Microsoft.Storage/storageAccounts', variables('storageAccountName'))]", 16 | "keyVault": "[resourceId(resourceGroup().name, 'Microsoft.KeyVault/vaults', variables('keyVaultName'))]", 17 | "applicationInsights": "[resourceId(resourceGroup().name, 'Microsoft.Insights/components', variables('applicationInsightsName'))]", 18 | "containerRegistry": "[resourceId(resourceGroup().name, 'Microsoft.ContainerRegistry/registries', variables('containerRegistryName'))]", 19 | "cognitiveServicesName": "[concat('asagacognitiveservices', variables('uniqueSuffix'))]" 20 | }, 21 | "resources": [ 22 | { 23 | "type": "Microsoft.Storage/storageAccounts", 24 | "apiVersion": "2019-04-01", 25 | "name": "[variables('storageAccountName')]", 26 | "location": "[variables('location')]", 27 | "sku": { 28 | "name": "[variables('storageAccountType')]" 29 | }, 30 | "kind": "StorageV2", 31 | "properties": { 32 | "encryption": { 33 | "services": { 34 | "blob": { 35 | "enabled": true 36 | }, 37 | "file": { 38 | "enabled": true 39 | } 40 | }, 41 | "keySource": "Microsoft.Storage" 42 | }, 43 | "supportsHttpsTrafficOnly": true 44 | } 45 | }, 46 | { 47 | "type": "Microsoft.Insights/components", 48 | "apiVersion": "2018-05-01-preview", 49 | "name": "[variables('applicationInsightsName')]", 50 | "location": "[if(or(equals(variables('location'),'eastus2'), equals(variables('location'),'westcentralus')),'southcentralus',variables('location'))]", 51 | "kind": "web", 52 | "properties": { 53 | "Application_Type": "web" 54 | } 55 | }, 56 | { 57 | "type": "Microsoft.ContainerRegistry/registries", 58 | "apiVersion": "2019-12-01-preview", 59 | "name": "[variables('containerRegistryName')]", 60 | "location": "[variables('location')]", 61 | "comments": "Container registry for Azure Machine Learning docker images", 62 | "tags": { 63 | "displayName": "Container Registry", 64 | "container.registry": "[variables('containerRegistryName')]" 65 | }, 66 | "sku": { 67 | "name": "Standard", 68 | "tier": "Standard" 69 | } 70 | }, 71 | { 72 | "type": "Microsoft.MachineLearningServices/workspaces", 73 | "apiVersion": "2020-03-01", 74 | "name": "[variables('workspaceName')]", 75 | "location": "[variables('location')]", 76 | "dependsOn": [ 77 | "[variables('storageAccount')]", 78 | "[variables('applicationInsights')]" 79 | ], 80 | "identity": { 81 | "type": "systemAssigned" 82 | }, 83 | "sku": { 84 | "tier": "[variables('sku')]", 85 | "name": "[variables('sku')]" 86 | }, 87 | "properties": { 88 | "friendlyName": "[variables('workspaceName')]", 89 | "storageAccount": "[variables('storageAccount')]", 90 | "keyVault": "[variables('keyVault')]", 91 | "applicationInsights": "[variables('applicationInsights')]", 92 | "containerRegistry": "[if(empty(variables('containerRegistryName')), json('null'), variables('containerRegistry'))]" 93 | } 94 | }, 95 | { 96 | "type": "Microsoft.CognitiveServices/accounts", 97 | "apiVersion": "2017-04-18", 98 | "name": "[variables('cognitiveServicesName')]", 99 | "location": "[variables('location')]", 100 | "sku": { 101 | "name": "S0" 102 | }, 103 | "kind": "CognitiveServices", 104 | "properties": { 105 | "apiProperties": {}, 106 | "customSubDomainName": "[variables('cognitiveServicesName')]", 107 | "publicNetworkAccess": "Enabled" 108 | } 109 | } 110 | ] 111 | } 112 | -------------------------------------------------------------------------------- /hands-on-labs/setup/arm/asaga-workspace-lab-02.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#", 3 | "contentVersion": "1.0.0.0", 4 | "variables": { 5 | "location": "[resourceGroup().location]", 6 | "uniqueSuffix": "[resourceGroup().tags['DeploymentId']]", 7 | "clusterName": "[concat('asagadataexpl', variables('uniqueSuffix'))]" 8 | }, 9 | "resources": [ 10 | { 11 | "type": "Microsoft.Kusto/Clusters", 12 | "apiVersion": "2020-06-14", 13 | "name": "[variables('clusterName')]", 14 | "location": "[variables('location')]", 15 | "sku": { 16 | "name": "Dev(No SLA)_Standard_E2a_v4", 17 | "tier": "Basic", 18 | "capacity": 1 19 | }, 20 | "properties": { 21 | "trustedExternalTenants": [ 22 | { 23 | "value": "*" 24 | } 25 | ], 26 | "enableDiskEncryption": false, 27 | "enableStreamingIngest": false, 28 | "enablePurge": false, 29 | "enableDoubleEncryption": false 30 | } 31 | }, 32 | { 33 | "type": "Microsoft.Kusto/Clusters/Databases", 34 | "apiVersion": "2020-06-14", 35 | "name": "[concat(variables('clusterName'), '/ASA-Data-Explorer-DB-01')]", 36 | "location": "[variables('location')]", 37 | "dependsOn": [ 38 | "[resourceId('Microsoft.Kusto/Clusters', variables('clusterName'))]" 39 | ], 40 | "kind": "ReadWrite", 41 | "properties": { 42 | "softDeletePeriod": "P3650D", 43 | "hotCachePeriod": "P31D" 44 | } 45 | } 46 | ] 47 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/arm/asaga-workspace-lab-03.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", 3 | "contentVersion": "1.0.0.0", 4 | "variables": { 5 | "location": "[resourceGroup().location]", 6 | "uniqueSuffix": "[resourceGroup().tags['DeploymentId']]", 7 | "dataFactoryName": "[concat('asagadatafactory', variables('uniqueSuffix'))]" 8 | }, 9 | "resources": [ 10 | { 11 | "apiVersion": "2018-06-01", 12 | "name": "[variables('dataFactoryName')]", 13 | "location": "[variables('location')]", 14 | "type": "Microsoft.DataFactory/factories", 15 | "identity": { 16 | "type": "SystemAssigned" 17 | }, 18 | "properties": {} 19 | } 20 | ] 21 | } 22 | -------------------------------------------------------------------------------- /hands-on-labs/setup/asa-workspace-deploy.md: -------------------------------------------------------------------------------- 1 | # Deploy your Azure Synapse Analytics workspace 2 | 3 | ## Pre-requisites for deployment 4 | 5 | The following requirements must be met before the deployment: 6 | 7 | - A resource group (this will be provided during the deployment process). 8 | 9 | >**IMPORTANT** 10 | > 11 | >In case you didn't create the resource group yourself, make sure your account has the `Owner` role on the resource group. 12 | > 13 | >Also, your account (i.e. the Azure AD account used to deploy the Azure Synapse Analytics workspace) must have permissions to create new resource groups in the subscription (this is required because Synapse Analytics requires an additional resource group to keep various hidden artifacts; this resource group is created during the deployment process). 14 | 15 | - A unique suffix to be used when generating the name of the workspace. All workspaces deployed using the templates in this repo are named `asagaworkspace`, where `` gets replaced with the value you provide. Make sure the unique suffix is specific enough to avoid potential naming collisions (i.e. avoid using common values like `01`, `1`, `test`, etc.). Make sure you remember the unique suffix as you need to use it for additional configuration once the Azure Synapse Analytics workspace deployment is complete. 16 | - A password for the SQL admin account of the workspace. Make sure you save the password in a secure location (like a password manager) as you will need to use it later. 17 | - A GitHub account to access the content packs repository. 18 | - A Power BI Pro subscription attached to the Azure AD account you will use to setup the Synapse Analytics workspace. In case you do not have a paid Power BI Pro subscription, you can get a 60 days trial by signing in to `https://powerbi.com` with your account and selecting `Try free`. 19 | - A Power BI Pro workspace (for details about creating a workspace in Power BI, see [Create the new workspaces in Power BI](https://docs.microsoft.com/en-us/azure/synapse-analytics/quickstart-power-bi)). 20 | 21 | ## Configure the Azure Cloud Shell 22 | 23 | >**NOTE** 24 | > 25 | >If Cloud Shell is already configured, you can skip this section entirely and advance to [Deploy the Synapse Analytics workspace](#deploy-the-synapse-analytics-workspace). 26 | 27 | In the Azure Portal, navigate to your resource group and create a new storage account to be used in the Cloud Shell configuration process (make sure the resource type you create is `Storage account`). In the newly created storage account, select `File shares` (under the `File service` settings group) and create a new file share. 28 | 29 | Next, select the Cloud Shell icon (located in the top right part of the page) and then select `PowerShell`: 30 | 31 | ![Cloud Shell configuration start](./../media/cloudshell-configure-01.png) 32 | 33 | Select your subscription under `Subscription` if it's not already selected, and then select `Show advanced settings`: 34 | 35 | ![Cloud Shell configuration advanced settings](./../media/cloudshell-configure-02.png) 36 | 37 | Provide values for the following fields: 38 | 39 | - **Cloud Shell region**: the same region as the region of your resource group. 40 | - **Resource group**: select `Use existing` and then select you resource group from the list. 41 | - **Storage account**: select `Use existing` and then select the storage account you created above. 42 | - **File share**: select `Use existing` and then select the file share you created above. 43 | 44 | Select `Attach storage` once all the values are in place. 45 | 46 | ![Cloud Shell configuration advanced settings values](./../media/cloudshell-configure-03.png) 47 | 48 | Once configuration is complete, you should get an instance of Cloud Shell: 49 | 50 | ![Cloud Shell](./../media/cloudshell-configure-04.png) 51 | 52 | ## Deploy the Synapse Analytics workspace 53 | 54 | Click the `Deploy to Azure` button below to start the deployment process. 55 | 56 | [![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fsolliancenet%2Fazure-synapse-analytics-ga-content-packs%2Fmain%2Fhands-on-labs%2Fsetup%2Farm%2Fasaga-workspace-core.json%3Ftoken%3DAA2FKXRAGLJK2Q5PS7UV6QC7ZZAS2) 57 | 58 | You should see next the `Custom deployment` screen where you need to provide the following (see [Pre-requisites for deployment](#pre-requisites-for-deployment) above for details): 59 | 60 | - The resource group where the Synapse Analytics workspace will be deployed. 61 | - The unique suffix used to generate the name of the workspace (**NOTE**: Make sure this value has a **maximum** length of **9 characters**). 62 | - The password for the SQL Administrator account. 63 | 64 | Select `Review + create` to validate the settings. 65 | 66 | ![Synapse Analytics workspace deployment configuration](../media/asaworkspace-deploy-configure.png) 67 | 68 | Once the validation is passed, select `Create` to start the deployment. You should see next an indication of the deployment progress: 69 | 70 | ![Synapse Analytics workspace deployment progress](./../media/asaworkspace-deploy-progress.png) 71 | 72 | Wait until the deployment completes successfully before proceeding to the next step. 73 | 74 | ## Tag your resource group with the unique suffix 75 | 76 | In the Azure Portal, navigate to the resource group you used to deploy the Synapse Analytics workspace (see [Pre-requisites for deployment](#pre-requisites-for-deployment) above for details). 77 | 78 | Select the `Tags` section and add a new tag named `DeploymentId`. Use the unique suffix as the value of the tag and then select `Apply` to save it. 79 | 80 | ![Synapse Analytics workspace resource group tagging](./../media/asaworkspace-deploy-tag.png) 81 | 82 | The deployment of your Synapse Analytics workspace is now complete. Next, you will deploy the artifacts required by the labs into the newly created Synapse Analytics workspace. 83 | 84 | ## Run the global setup script in Cloud Shell 85 | 86 | In the Azure Portal, navigate to the resource group you used to deploy the Synapse Analytics workspace (see [Pre-requisites for deployment](#pre-requisites-for-deployment) above for details) and start a Cloud Shell instance (see [Configure the Azure Cloud Shell](#configure-the-azure-cloud-shell) above for details). 87 | 88 | Once the Cloud Shell instance becomes available, run ```az login``` to make sure the correct account and subscription context are set: 89 | 90 | ![Cloud Shell login](./../media/cloudshell-setup-01.png) 91 | 92 | Clone the content packs repository into the `asa` local folder using 93 | 94 | ```cmd 95 | git clone https://github.com/solliancenet/azure-synapse-analytics-ga-content-packs asa 96 | ``` 97 | 98 | If GIT asks for credentials, provide your GitHub username and password. 99 | 100 | >**IMPORTANT** 101 | > 102 | >If your GitHub account has two-factor authentication activated, you need to provide a PAT (Personal Access Token) instead your password. For more details, read the [Creating a personal access token](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token) section in GitHub Docs. 103 | > 104 | >When pasting your password or PAT into the Cloud Shell window, make sure you are familiar with the supported key combinations (Shift-INS for Windows and Cmd-V for Mac). For more details, see [Using the Azure Cloud Shell window](https://docs.microsoft.com/en-us/azure/cloud-shell/using-the-shell-window#copy-and-paste). 105 | 106 | Once the repository is successfully cloned, you shoud see a result similar to this: 107 | 108 | ![Cloud Shell git clone repository](./../media/cloudshell-setup-02.png) 109 | 110 | Change your current directory using 111 | 112 | ```cmd 113 | cd asa/hands-on-labs/setup/automation 114 | ``` 115 | 116 | and then start the setup script using 117 | 118 | ```powershell 119 | .\environment-setup.ps1 120 | ``` 121 | 122 | Make sure the selected subscription is the one that contains the resource group where you deployed the Synapse Analytics workspace: 123 | 124 | ![Cloud Shell select subscription](./../media/cloudshell-setup-03.png) 125 | 126 | Enter the name of the resource group where you deployed the Synapse Analytics workspace: 127 | 128 | ![Cloud Shell select resource group](./../media/cloudshell-setup-04.png) 129 | 130 | The setup script will now proceed to create all necessary Synapse Analytics artifacts in your environment. 131 | 132 | The process should take 5 to 10 minutes to finish. Wait until the setup script is finished before proceeding to the next steps. 133 | 134 | ## Connect the Azure Synapse Analytics workspace to your Power BI workspace 135 | 136 | In the Azure Portal, navigate to your resource group, open the Synapse workspace resource (should be named `asagaworkspace` where `` is the one you specified when creating the workspace), and then open Synapse Studio. 137 | 138 | In Synapse Studio, select the `Manage` hub on the left side, select `Linked Services`, and then select `+ New` to start creating a new linked service. Select `Connect to Power BI` to start configuring the linked service (if the `Connect to Power BI` option does not show up, enter `Power BI` in the search box, select `Power BI` and then select `Continue`). 139 | 140 | ![Start configuring a new Power BI linked service](./../media/asaworkspace-deploy-pbi-linked-service-01.png) 141 | 142 | In the `New linked service (Power BI)` dialog enter settings as follows: 143 | 144 | - **Name**: enter `asagapowerbi` (where `` is the one you specified when creating the Synapse Analytics workspace). 145 | - **Tenant**: ensure the correct tenant is selected (the one that contains your Azure AD account). 146 | - **Workspace name**: select the Power BI workspace you want to use. 147 | 148 | Select `Create` to create the Power BI linked service. 149 | 150 | ![Power BI linked service configuration](./../media/asaworkspace-deploy-pbi-linked-service-02.png) 151 | 152 | After the linked service is successfully created, select the `Develop` hub on the left side and expand the `Power BI` section. You should see your Power BI workspace listed. 153 | 154 | ![Check Power BI linked service configuration](./../media/asaworkspace-deploy-pbi-linked-service-03.png) 155 | 156 | ## Report issues 157 | 158 | In case you encounter any issues with the content in this repository, please follow the [How to report issues](./../../report-issues.md) guideline. We will try to address them as soon as possible. Please check your open issues to learn about their status. 159 | -------------------------------------------------------------------------------- /hands-on-labs/setup/automation/lab-01-setup.ps1: -------------------------------------------------------------------------------- 1 | $InformationPreference = "Continue" 2 | 3 | if(Get-Module -Name solliance-synapse-automation){ 4 | Remove-Module solliance-synapse-automation 5 | } 6 | Import-Module "..\solliance-synapse-automation" 7 | 8 | #Different approach to run automation in Cloud Shell 9 | $subs = Get-AzSubscription | Select-Object -ExpandProperty Name 10 | if($subs.GetType().IsArray -and $subs.length -gt 1){ 11 | $subOptions = [System.Collections.ArrayList]::new() 12 | for($subIdx=0; $subIdx -lt $subs.length; $subIdx++){ 13 | $opt = New-Object System.Management.Automation.Host.ChoiceDescription "$($subs[$subIdx])", "Selects the $($subs[$subIdx]) subscription." 14 | $subOptions.Add($opt) 15 | } 16 | $selectedSubIdx = $host.ui.PromptForChoice('Enter the desired Azure Subscription for this lab','Copy and paste the name of the subscription to make your choice.', $subOptions.ToArray(),0) 17 | $selectedSubName = $subs[$selectedSubIdx] 18 | Write-Information "Selecting the $selectedSubName subscription" 19 | Select-AzSubscription -SubscriptionName $selectedSubName 20 | } 21 | 22 | $resourceGroupName = Read-Host "Enter the resource group name"; 23 | 24 | $userName = ((az ad signed-in-user show) | ConvertFrom-JSON).UserPrincipalName 25 | 26 | $artifactsPath = "..\..\" 27 | $reportsPath = "..\reports" 28 | $templatesPath = "..\templates" 29 | $datasetsPath = "..\datasets" 30 | $dataflowsPath = "..\dataflows" 31 | $pipelinesPath = "..\pipelines" 32 | $sqlScriptsPath = "..\sql" 33 | 34 | 35 | Write-Information "Using $resourceGroupName"; 36 | 37 | $resourceGroup = Get-AzResourceGroup -Name $resourceGroupName 38 | $uniqueId = $resourceGroup.Tags["DeploymentId"] 39 | $location = $resourceGroup.Location 40 | $subscriptionId = (Get-AzContext).Subscription.Id 41 | $tenantId = (Get-AzContext).Tenant.Id 42 | $global:logindomain = (Get-AzContext).Tenant.Id; 43 | 44 | $workspaceName = "asagaworkspace$($uniqueId)" 45 | $dataLakeAccountName = "asagadatalake$($uniqueId)" 46 | $keyVaultName = "asagakeyvault$($uniqueId)" 47 | $keyVaultSQLUserSecretName = "SQL-USER-ASA" 48 | $sqlPoolName = "SQLPool01" 49 | $integrationRuntimeName = "AutoResolveIntegrationRuntime" 50 | $sparkPoolName = "SparkPool01" 51 | $global:sqlEndpoint = "$($workspaceName).sql.azuresynapse.net" 52 | $global:sqlUser = "asaga.sql.admin" 53 | 54 | $global:synapseToken = "" 55 | $global:synapseSQLToken = "" 56 | $global:managementToken = "" 57 | $global:powerbiToken = ""; 58 | 59 | $global:tokenTimes = [ordered]@{ 60 | Synapse = (Get-Date -Year 1) 61 | SynapseSQL = (Get-Date -Year 1) 62 | Management = (Get-Date -Year 1) 63 | PowerBI = (Get-Date -Year 1) 64 | } 65 | 66 | $cognitiveServicesAccountName = "asagacognitiveservices$($uniqueId)" 67 | 68 | $cognitiveServicesKeys = Get-AzCognitiveServicesAccountKey -ResourceGroupName $resourceGroupName -name $cognitiveServicesAccountName 69 | 70 | $secretValue = ConvertTo-SecureString $cognitiveServicesKeys.Key1 -AsPlainText -Force 71 | Set-AzKeyVaultSecret -VaultName $keyVaultName -Name "ASA-GA-COGNITIVE-SERVICES" -SecretValue $secretValue 72 | 73 | 74 | Write-Information "Create data sets for Product Quantity Forecast data load in SQL pool $($sqlPoolName)" 75 | 76 | $loadingDatasets = @{ 77 | wwi02_sale_small_product_quantity_forecast_adls = $dataLakeAccountName 78 | wwi02_sale_small_product_quantity_forecast_asa = $sqlPoolName.ToLower() 79 | } 80 | 81 | foreach ($dataset in $loadingDatasets.Keys) { 82 | Write-Information "Creating dataset $($dataset)" 83 | $result = Create-Dataset -DatasetsPath $datasetsPath -WorkspaceName $workspaceName -Name $dataset -LinkedServiceName $loadingDatasets[$dataset] 84 | Wait-ForOperation -WorkspaceName $workspaceName -OperationId $result.operationId 85 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/automation/lab-02-setup.ps1: -------------------------------------------------------------------------------- 1 | $InformationPreference = "Continue" 2 | 3 | if(Get-Module -Name solliance-synapse-automation){ 4 | Remove-Module solliance-synapse-automation 5 | } 6 | Import-Module "..\solliance-synapse-automation" 7 | 8 | #Different approach to run automation in Cloud Shell 9 | $subs = Get-AzSubscription | Select-Object -ExpandProperty Name 10 | if($subs.GetType().IsArray -and $subs.length -gt 1){ 11 | $subOptions = [System.Collections.ArrayList]::new() 12 | for($subIdx=0; $subIdx -lt $subs.length; $subIdx++){ 13 | $opt = New-Object System.Management.Automation.Host.ChoiceDescription "$($subs[$subIdx])", "Selects the $($subs[$subIdx]) subscription." 14 | $subOptions.Add($opt) 15 | } 16 | $selectedSubIdx = $host.ui.PromptForChoice('Enter the desired Azure Subscription for this lab','Copy and paste the name of the subscription to make your choice.', $subOptions.ToArray(),0) 17 | $selectedSubName = $subs[$selectedSubIdx] 18 | Write-Information "Selecting the $selectedSubName subscription" 19 | Select-AzSubscription -SubscriptionName $selectedSubName 20 | } 21 | 22 | $resourceGroupName = Read-Host "Enter the resource group name"; 23 | 24 | $userName = ((az ad signed-in-user show) | ConvertFrom-JSON).UserPrincipalName 25 | 26 | $artifactsPath = "..\..\" 27 | $reportsPath = "..\reports" 28 | $templatesPath = "..\templates" 29 | $datasetsPath = "..\datasets" 30 | $dataflowsPath = "..\dataflows" 31 | $pipelinesPath = "..\pipelines" 32 | $sqlScriptsPath = "..\sql" 33 | 34 | 35 | Write-Information "Using $resourceGroupName"; 36 | 37 | $resourceGroup = Get-AzResourceGroup -Name $resourceGroupName 38 | $uniqueId = $resourceGroup.Tags["DeploymentId"] 39 | $location = $resourceGroup.Location 40 | $subscriptionId = (Get-AzContext).Subscription.Id 41 | $tenantId = (Get-AzContext).Tenant.Id 42 | $global:logindomain = (Get-AzContext).Tenant.Id; 43 | 44 | $workspaceName = "asagaworkspace$($uniqueId)" 45 | $dataLakeAccountName = "asagadatalake$($uniqueId)" 46 | $keyVaultName = "asagakeyvault$($uniqueId)" 47 | $keyVaultSQLUserSecretName = "SQL-USER-ASA" 48 | $sqlPoolName = "SQLPool01" 49 | $integrationRuntimeName = "AutoResolveIntegrationRuntime" 50 | $sparkPoolName = "SparkPool01" 51 | $global:sqlEndpoint = "$($workspaceName).sql.azuresynapse.net" 52 | $global:sqlUser = "asaga.sql.admin" 53 | 54 | $global:synapseToken = "" 55 | $global:synapseSQLToken = "" 56 | $global:managementToken = "" 57 | $global:powerbiToken = ""; 58 | 59 | $global:tokenTimes = [ordered]@{ 60 | Synapse = (Get-Date -Year 1) 61 | SynapseSQL = (Get-Date -Year 1) 62 | Management = (Get-Date -Year 1) 63 | PowerBI = (Get-Date -Year 1) 64 | } 65 | 66 | Write-Information "Creating the SalesTelemetry table" 67 | 68 | $kustoClusterName = "asagadataexpl$($uniqueId)" 69 | $kustoDatabaseName = "ASA-Data-Explorer-DB-01" 70 | $kustoStatement = ".create table SalesTelemetry ( CustomerId:int32, ProductId:int32, Timestamp:datetime, Url:string)" 71 | 72 | $token = ((az account get-access-token --resource https://help.kusto.windows.net) | ConvertFrom-Json).accessToken 73 | $body = "{ db: ""$kustoDatabaseName"", csl: ""$kustoStatement"" }" 74 | Invoke-RestMethod -Uri https://$kustoClusterName.$($location).kusto.windows.net/v1/rest/mgmt -Method POST -Body $body -Headers @{ Authorization="Bearer $token" } -ContentType "application/json" 75 | 76 | # Set the Azure Synapse Analytics GA Labs service principal as admin on the Kusto database 77 | 78 | Write-Information "Making the service principal 'Azure Synapse Analytics GA Labs' an admin on the Kusto database" 79 | $app = ((az ad sp list --display-name "Azure Synapse Analytics GA Labs") | ConvertFrom-Json)[0] 80 | $kustoStatement = ".add database ['$($kustoDatabaseName)'] admins ('aadapp=$($app.appId)')" 81 | $body = "{ db: ""$kustoDatabaseName"", csl: ""$kustoStatement"" }" 82 | Invoke-RestMethod -Uri https://$kustoClusterName.$($location).kusto.windows.net/v1/rest/mgmt -Method POST -Body $body -Headers @{ Authorization="Bearer $token" } -ContentType "application/json" 83 | 84 | 85 | Write-Information "Create linked service for Kusto database $($kustoDatabaseName)" 86 | 87 | $linkedServiceName = $kustoClusterName.ToLower() 88 | $result = Create-DataExplorerKeyVaultLinkedService -TemplatesPath $templatesPath -WorkspaceName $workspaceName -Name $linkedServiceName -DataExplorerClusterName "$($kustoClusterName).$($location)" ` 89 | -DataExplorerDatabaseName $kustoDatabaseName -AADTenantId $tenantId -AADServicePrincipalId $app.appId -KeyVaultLinkedServiceName $keyVaultName -SecretName "ASA-GA-LABS" 90 | Wait-ForOperation -WorkspaceName $workspaceName -OperationId $result.operationId 91 | 92 | 93 | 94 | Write-Information "Create data sets for loading sales telemetry data into $($kustoDatabaseName) Kusto database" 95 | 96 | $loadingDatasets = @{ 97 | wwi02_sale_small_telemetry_adls = $dataLakeAccountName 98 | wwi02_sale_small_telemetry_ade = $linkedServiceName 99 | } 100 | 101 | foreach ($dataset in $loadingDatasets.Keys) { 102 | Write-Information "Creating dataset $($dataset)" 103 | $result = Create-Dataset -DatasetsPath $datasetsPath -WorkspaceName $workspaceName -Name $dataset -LinkedServiceName $loadingDatasets[$dataset] 104 | Wait-ForOperation -WorkspaceName $workspaceName -OperationId $result.operationId 105 | } 106 | 107 | Write-Information "Create pipeline to load sales telemetry data into the Data Explorer database" 108 | 109 | $params = @{} 110 | $loadingPipelineName = "Setup - Import sales telemetry data" 111 | $fileName = "import_sale_small_telemetry_data" 112 | 113 | Write-Information "Creating pipeline $($loadingPipelineName)" 114 | 115 | $result = Create-Pipeline -PipelinesPath $pipelinesPath -WorkspaceName $workspaceName -Name $loadingPipelineName -FileName $fileName -Parameters $params 116 | Wait-ForOperation -WorkspaceName $workspaceName -OperationId $result.operationId 117 | 118 | Write-Information "Running pipeline $($loadingPipelineName)" 119 | 120 | $result = Run-Pipeline -WorkspaceName $workspaceName -Name $loadingPipelineName 121 | $result = Wait-ForPipelineRun -WorkspaceName $workspaceName -RunId $result.runId 122 | $result 123 | 124 | Write-Information "Deleting pipeline $($loadingPipelineName)" 125 | 126 | $result = Delete-ASAObject -WorkspaceName $workspaceName -Category "pipelines" -Name $loadingPipelineName 127 | Wait-ForOperation -WorkspaceName $workspaceName -OperationId $result.operationId 128 | 129 | foreach ($dataset in $loadingDatasets.Keys) { 130 | Write-Information "Deleting dataset $($dataset)" 131 | $result = Delete-ASAObject -WorkspaceName $workspaceName -Category "datasets" -Name $dataset 132 | Wait-ForOperation -WorkspaceName $workspaceName -OperationId $result.operationId 133 | } 134 | 135 | 136 | 137 | Write-Information "Deleting linked service for Kusto database" 138 | 139 | $result = Delete-ASAObject -WorkspaceName $workspaceName -Category "linkedservices" -Name $linkedServiceName 140 | Wait-ForOperation -WorkspaceName $workspaceName -OperationId $result.operationId -------------------------------------------------------------------------------- /hands-on-labs/setup/automation/lab-03-setup.ps1: -------------------------------------------------------------------------------- 1 | $InformationPreference = "Continue" 2 | 3 | if(Get-Module -Name solliance-synapse-automation){ 4 | Remove-Module solliance-synapse-automation 5 | } 6 | Import-Module "..\solliance-synapse-automation" 7 | 8 | #Different approach to run automation in Cloud Shell 9 | $subs = Get-AzSubscription | Select-Object -ExpandProperty Name 10 | if($subs.GetType().IsArray -and $subs.length -gt 1){ 11 | $subOptions = [System.Collections.ArrayList]::new() 12 | for($subIdx=0; $subIdx -lt $subs.length; $subIdx++){ 13 | $opt = New-Object System.Management.Automation.Host.ChoiceDescription "$($subs[$subIdx])", "Selects the $($subs[$subIdx]) subscription." 14 | $subOptions.Add($opt) 15 | } 16 | $selectedSubIdx = $host.ui.PromptForChoice('Enter the desired Azure Subscription for this lab','Copy and paste the name of the subscription to make your choice.', $subOptions.ToArray(),0) 17 | $selectedSubName = $subs[$selectedSubIdx] 18 | Write-Information "Selecting the $selectedSubName subscription" 19 | Select-AzSubscription -SubscriptionName $selectedSubName 20 | } 21 | 22 | $resourceGroupName = Read-Host "Enter the resource group name"; 23 | 24 | $userName = ((az ad signed-in-user show) | ConvertFrom-JSON).UserPrincipalName 25 | 26 | $artifactsPath = "..\..\" 27 | $reportsPath = "..\reports" 28 | $templatesPath = "..\templates" 29 | $datasetsPath = "..\datasets" 30 | $dataflowsPath = "..\dataflows" 31 | $pipelinesPath = "..\pipelines" 32 | $sqlScriptsPath = "..\sql" 33 | 34 | 35 | Write-Information "Using $resourceGroupName"; 36 | 37 | $resourceGroup = Get-AzResourceGroup -Name $resourceGroupName 38 | $uniqueId = $resourceGroup.Tags["DeploymentId"] 39 | $location = $resourceGroup.Location 40 | $subscriptionId = (Get-AzContext).Subscription.Id 41 | $tenantId = (Get-AzContext).Tenant.Id 42 | $global:logindomain = (Get-AzContext).Tenant.Id; 43 | 44 | $workspaceName = "asagaworkspace$($uniqueId)" 45 | $dataLakeAccountName = "asagadatalake$($uniqueId)" 46 | $keyVaultName = "asagakeyvault$($uniqueId)" 47 | $keyVaultSQLUserSecretName = "SQL-USER-ASA" 48 | $sqlPoolName = "SQLPool01" 49 | $integrationRuntimeName = "AutoResolveIntegrationRuntime" 50 | $sparkPoolName = "SparkPool01" 51 | $global:sqlEndpoint = "$($workspaceName).sql.azuresynapse.net" 52 | $global:sqlUser = "asaga.sql.admin" 53 | 54 | $global:synapseToken = "" 55 | $global:synapseSQLToken = "" 56 | $global:managementToken = "" 57 | $global:powerbiToken = ""; 58 | 59 | $global:tokenTimes = [ordered]@{ 60 | Synapse = (Get-Date -Year 1) 61 | SynapseSQL = (Get-Date -Year 1) 62 | Management = (Get-Date -Year 1) 63 | PowerBI = (Get-Date -Year 1) 64 | } 65 | 66 | $dataFactoryAccountName = "asagadatafactory$($uniqueId)" 67 | 68 | $dataLakeAccountKey = List-StorageAccountKeys -SubscriptionId $subscriptionId -ResourceGroupName $resourceGroupName -Name $dataLakeAccountName 69 | 70 | $secretValue = ConvertTo-SecureString $dataLakeAccountKey -AsPlainText -Force 71 | Set-AzKeyVaultSecret -VaultName $keyVaultName -Name "ASA-GA-DATA-LAKE" -SecretValue $secretValue 72 | 73 | Write-Output "Data Factory v2 account name is $($dataFactoryAccountName)" 74 | $dataFactoryServicePrincipal = (Get-AzADServicePrincipal -DisplayName $dataFactoryAccountName) 75 | Set-AzKeyVaultAccessPolicy -ResourceGroupName $resourceGroupName -VaultName $keyVaultName -ObjectId $dataFactoryServicePrincipal.Id -PermissionsToSecrets set,delete,get,list 76 | 77 | # Create Azure Data Factory v2 linked services 78 | 79 | $template = Get-Content -Path "$($templatesPath)/key_vault_linked_service.json" 80 | $templateContent = $template.Replace("#LINKED_SERVICE_NAME#", "asagakeyvault01_adf").Replace("#KEY_VAULT_NAME#", $keyVaultName) 81 | Set-Content -Path .\temp.json -Value $templateContent 82 | Set-AzDataFactoryV2LinkedService -DataFactoryName $dataFactoryAccountName ` 83 | -ResourceGroupName $resourceGroupName -Name "asagakeyvault01_adf" ` 84 | -DefinitionFile ".\temp.json" -Force 85 | Remove-Item -Path .\temp.json -Force 86 | 87 | 88 | $template = Get-Content -Path "$($templatesPath)/data_lake_key_vault_linked_service.json" 89 | $templateContent = $template.Replace("#LINKED_SERVICE_NAME#", "asagadatalake01_adf").Replace("#DATA_LAKE_ACCOUNT_NAME#", $dataLakeAccountName).Replace("#KEY_VAULT_LINKED_SERVICE_NAME#", "asagakeyvault01_adf").Replace("#SECRET_NAME#", "ASA-GA-DATA-LAKE") 90 | Set-Content -Path .\temp.json -Value $templateContent 91 | Set-AzDataFactoryV2LinkedService -DataFactoryName $dataFactoryAccountName ` 92 | -ResourceGroupName $resourceGroupName -Name "asagadatalake01_adf" ` 93 | -DefinitionFile ".\temp.json" -Force 94 | Remove-Item -Path .\temp.json -Force 95 | 96 | 97 | $template = Get-Content -Path "$($templatesPath)/sql_pool_key_vault_linked_service.json" 98 | $templateContent = $template.Replace("#LINKED_SERVICE_NAME#", "asagasqlpool01_adf").Replace("#WORKSPACE_NAME#", $workspaceName).Replace("#DATABASE_NAME#", "SQLPool01").Replace("#USER_NAME#", "asaga.sql.admin").Replace("#KEY_VAULT_LINKED_SERVICE_NAME#", "asagakeyvault01_adf").Replace("#SECRET_NAME#", "SQL-USER-ASA") 99 | Set-Content -Path .\temp.json -Value $templateContent 100 | Set-AzDataFactoryV2LinkedService -DataFactoryName $dataFactoryAccountName ` 101 | -ResourceGroupName $resourceGroupName -Name "asagasqlpool01_adf" ` 102 | -DefinitionFile ".\temp.json" -Force 103 | Remove-Item -Path .\temp.json -Force 104 | 105 | # Create tables in SQL pool 106 | Write-Information "Create tables for Lab 03 in $($sqlPoolName)" 107 | 108 | #$secret = Get-AzKeyVaultSecret -VaultName $keyVaultName -Name SQL-USER-ASA 109 | #$global:sqlPassword = $secret.SecretValue | ConvertFrom-SecureString -AsPlainText 110 | $global:sqlPassword = Get-AzKeyVaultSecret -VaultName $keyVaultName -Name SQL-USER-ASA -AsPlainText 111 | 112 | $params = @{} 113 | $result = Execute-SQLScriptFile -SQLScriptsPath $sqlScriptsPath -WorkspaceName $workspaceName -SQLPoolName $sqlPoolName -FileName "05-create-tables-lab-03" -Parameters $params 114 | $result 115 | 116 | 117 | # Create datasets 118 | 119 | Set-AzDataFactoryV2Dataset -DataFactoryName $dataFactoryAccountName ` 120 | -ResourceGroupName $resourceGroupName -Name "wwi02_sale_small_stats_adls" ` 121 | -DefinitionFile "$($datasetsPath)\wwi02_sale_small_stats_adls.json" -Force 122 | 123 | 124 | Set-AzDataFactoryV2Dataset -DataFactoryName $dataFactoryAccountName ` 125 | -ResourceGroupName $resourceGroupName -Name "wwi02_sale_small_stats_asa" ` 126 | -DefinitionFile "$($datasetsPath)\wwi02_sale_small_stats_asa.json" -Force 127 | 128 | 129 | # Create pipeline 130 | 131 | $template = Get-Content -Path "$($pipelinesPath)/import_sale_small_stats_data.json" 132 | $templateContent = $template.Replace("#BLOB_STORAGE_LINKED_SERVICE_NAME#", "asagadatalake01_adf") 133 | Set-Content -Path .\temp.json -Value $templateContent 134 | Set-AzDataFactoryV2Pipeline -DataFactoryName $dataFactoryAccountName ` 135 | -ResourceGroupName $resourceGroupName -Name "Lab 03 - Import Sales Stats" ` 136 | -DefinitionFile ".\temp.json" -Force 137 | Remove-Item -Path .\temp.json -Force -------------------------------------------------------------------------------- /hands-on-labs/setup/cleanup.md: -------------------------------------------------------------------------------- 1 | # Clean-up your subscription 2 | 3 | This section provides instructions on how to clean-up your subscription after you complete one or more hands-on labs. 4 | 5 | >**IMPORTANT** 6 | > 7 | >If you plan to run other labs that are available in this repo as well, you should defer the clean-up process until you complete all the labs you are interested in. The labs are designed to share the same Synapse Analytics workspace so you are not required to setup the workspace separately for each lab. Keep in mind though that once provisioned, resources generate costs, so, if you are not planning to run labs in sequence (i.e. there will be longer times between labs) you should either pause resources or perform this clean-up procedure and re-provision the environment at a later time. 8 | > 9 | >In case you decide to pause resources, follow these guidelines to minimize costs: 10 | > 11 | > - Ensure you pause all SQL Pools in the Synapse Analytics workspace 12 | > - Ensure you stop the Azure Data Explorer cluster (if aplicable). 13 | 14 | To clean-up your subscription, you need to delete the resource group you provided when deploying the Synapse Analytics workpace and the internal resource group that was automatically created during the deployment process (see [Deploy your Synapse Analytics workspace](./deploy-asa-workspace.md) for details). 15 | 16 | To delete the resource group you provided when deploying the Synapse Analytics workspace, open the Azure Portal, navigate to the resource group, and then select `Delete resource group`. 17 | 18 | ![Delete main resource group](./../media/cleanup-01.png) 19 | 20 | The name of the internal resource group that was automaticall created during the deployment process is `-internal`, where `` is the name of the resource group you provided when deploying the Synapse Analytics workspace. To delete the internal resource group, open the Azure Portal, navigate to the resource group, and the select `Delete resource group`: 21 | 22 | ![Delete internal resource group](./../media/cleanup-02.png) 23 | 24 | Your subscription has been now successfully cleaned-up. 25 | 26 | ## Report issues 27 | 28 | In case you encounter any issues with the content in this repository, please follow the [How to report issues](./../../report-issues.md) guideline. We will try to address them as soon as possible. Please check your open issues to learn about their status. 29 | -------------------------------------------------------------------------------- /hands-on-labs/setup/datasets/wwi02_date_adls.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "wwi02_date_adls", 3 | "properties": { 4 | "linkedServiceName": { 5 | "referenceName": "#LINKED_SERVICE_NAME#", 6 | "type": "LinkedServiceReference" 7 | }, 8 | "annotations": [], 9 | "type": "DelimitedText", 10 | "typeProperties": { 11 | "location": { 12 | "type": "AzureBlobFSLocation", 13 | "fileName": "generator-date.csv", 14 | "folderPath": "data-generators", 15 | "fileSystem": "wwi-02" 16 | }, 17 | "columnDelimiter": ",", 18 | "escapeChar": "\\", 19 | "quoteChar": "\"" 20 | }, 21 | "schema": [ 22 | { 23 | "type": "String" 24 | }, 25 | { 26 | "type": "String" 27 | }, 28 | { 29 | "type": "String" 30 | }, 31 | { 32 | "type": "String" 33 | }, 34 | { 35 | "type": "String" 36 | } 37 | ] 38 | }, 39 | "type": "Microsoft.Synapse/workspaces/datasets" 40 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/datasets/wwi02_date_asa.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "wwi02_date_asa", 3 | "properties": { 4 | "linkedServiceName": { 5 | "referenceName": "#LINKED_SERVICE_NAME#", 6 | "type": "LinkedServiceReference" 7 | }, 8 | "annotations": [], 9 | "type": "AzureSqlDWTable", 10 | "schema": [ 11 | { 12 | "name": "DateId", 13 | "type": "int", 14 | "precision": 10 15 | }, 16 | { 17 | "name": "Day", 18 | "type": "tinyint", 19 | "precision": 3 20 | }, 21 | { 22 | "name": "Month", 23 | "type": "tinyint", 24 | "precision": 3 25 | }, 26 | { 27 | "name": "Quarter", 28 | "type": "tinyint", 29 | "precision": 3 30 | }, 31 | { 32 | "name": "Year", 33 | "type": "smallint", 34 | "precision": 5 35 | } 36 | ], 37 | "typeProperties": { 38 | "schema": "wwi", 39 | "table": "Date" 40 | } 41 | }, 42 | "type": "Microsoft.Synapse/workspaces/datasets" 43 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/datasets/wwi02_poc_customer_adls.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "wwi02_poc_customer_adls", 3 | "properties": { 4 | "linkedServiceName": { 5 | "referenceName": "#LINKED_SERVICE_NAME#", 6 | "type": "LinkedServiceReference" 7 | }, 8 | "annotations": [], 9 | "type": "DelimitedText", 10 | "typeProperties": { 11 | "location": { 12 | "type": "AzureBlobFSLocation", 13 | "fileName": "generator-customer-clean.csv", 14 | "folderPath": "data-generators", 15 | "fileSystem": "wwi-02" 16 | }, 17 | "columnDelimiter": ",", 18 | "escapeChar": "\\", 19 | "firstRowAsHeader": true, 20 | "quoteChar": "'" 21 | }, 22 | "schema": [ 23 | { 24 | "name": "CustomerId", 25 | "type": "String" 26 | }, 27 | { 28 | "name": "FirstName", 29 | "type": "String" 30 | }, 31 | { 32 | "name": "MiddleInitial", 33 | "type": "String" 34 | }, 35 | { 36 | "name": "LastName", 37 | "type": "String" 38 | }, 39 | { 40 | "name": "FullName", 41 | "type": "String" 42 | }, 43 | { 44 | "name": "Gender", 45 | "type": "String" 46 | }, 47 | { 48 | "name": "Age", 49 | "type": "String" 50 | }, 51 | { 52 | "name": "BirthDate", 53 | "type": "String" 54 | }, 55 | { 56 | "name": "Address", 57 | "type": "String" 58 | }, 59 | { 60 | "name": "Mobile", 61 | "type": "String" 62 | }, 63 | { 64 | "name": "Email", 65 | "type": "String" 66 | } 67 | ] 68 | } 69 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/datasets/wwi02_poc_customer_asa.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "wwi02_poc_customer_asa", 3 | "properties": { 4 | "linkedServiceName": { 5 | "referenceName": "#LINKED_SERVICE_NAME#", 6 | "type": "LinkedServiceReference" 7 | }, 8 | "annotations": [], 9 | "type": "AzureSqlDWTable", 10 | "schema": [ 11 | { 12 | "name": "CustomerId", 13 | "type": "int", 14 | "precision": 10 15 | }, 16 | { 17 | "name": "FirstName", 18 | "type": "nvarchar" 19 | }, 20 | { 21 | "name": "MiddleInitial", 22 | "type": "nvarchar" 23 | }, 24 | { 25 | "name": "LastName", 26 | "type": "nvarchar" 27 | }, 28 | { 29 | "name": "FullName", 30 | "type": "nvarchar" 31 | }, 32 | { 33 | "name": "Gender", 34 | "type": "nvarchar" 35 | }, 36 | { 37 | "name": "Age", 38 | "type": "int", 39 | "precision": 10 40 | }, 41 | { 42 | "name": "BirthDate", 43 | "type": "date" 44 | }, 45 | { 46 | "name": "Address_PostalCode", 47 | "type": "nvarchar" 48 | }, 49 | { 50 | "name": "Address_Street", 51 | "type": "nvarchar" 52 | }, 53 | { 54 | "name": "Address_City", 55 | "type": "nvarchar" 56 | }, 57 | { 58 | "name": "Address_Country", 59 | "type": "nvarchar" 60 | }, 61 | { 62 | "name": "Mobile", 63 | "type": "nvarchar" 64 | }, 65 | { 66 | "name": "Email", 67 | "type": "nvarchar" 68 | } 69 | ], 70 | "typeProperties": { 71 | "schema": "wwi", 72 | "table": "Customer" 73 | } 74 | } 75 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/datasets/wwi02_product_adls.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "wwi02_product_adls", 3 | "properties": { 4 | "linkedServiceName": { 5 | "referenceName": "#LINKED_SERVICE_NAME#", 6 | "type": "LinkedServiceReference" 7 | }, 8 | "annotations": [], 9 | "type": "DelimitedText", 10 | "typeProperties": { 11 | "location": { 12 | "type": "AzureBlobFSLocation", 13 | "fileName": "generator-product.csv", 14 | "folderPath": "data-generators/generator-product", 15 | "fileSystem": "wwi-02" 16 | }, 17 | "columnDelimiter": ",", 18 | "escapeChar": "\\", 19 | "quoteChar": "\"" 20 | }, 21 | "schema": [ 22 | { 23 | "type": "String" 24 | }, 25 | { 26 | "type": "String" 27 | }, 28 | { 29 | "type": "String" 30 | }, 31 | { 32 | "type": "String" 33 | } 34 | ] 35 | }, 36 | "type": "Microsoft.Synapse/workspaces/datasets" 37 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/datasets/wwi02_product_asa.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "wwi02_product_asa", 3 | "properties": { 4 | "linkedServiceName": { 5 | "referenceName": "#LINKED_SERVICE_NAME#", 6 | "type": "LinkedServiceReference" 7 | }, 8 | "annotations": [], 9 | "type": "AzureSqlDWTable", 10 | "schema": [ 11 | { 12 | "name": "ProductId", 13 | "type": "smallint", 14 | "precision": 5 15 | }, 16 | { 17 | "name": "Seasonality", 18 | "type": "tinyint", 19 | "precision": 3 20 | }, 21 | { 22 | "name": "Price", 23 | "type": "decimal", 24 | "precision": 6, 25 | "scale": 2 26 | }, 27 | { 28 | "name": "Profit", 29 | "type": "decimal", 30 | "precision": 6, 31 | "scale": 2 32 | } 33 | ], 34 | "typeProperties": { 35 | "schema": "wwi", 36 | "table": "Product" 37 | } 38 | }, 39 | "type": "Microsoft.Synapse/workspaces/datasets" 40 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/datasets/wwi02_sale_small_adls.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "wwi02_sale_small_adls", 3 | "properties": { 4 | "linkedServiceName": { 5 | "referenceName": "#LINKED_SERVICE_NAME#", 6 | "type": "LinkedServiceReference" 7 | }, 8 | "annotations": [], 9 | "type": "Parquet", 10 | "typeProperties": { 11 | "location": { 12 | "type": "AzureBlobFSLocation", 13 | "folderPath": "sale-small", 14 | "fileSystem": "wwi-02" 15 | }, 16 | "compressionCodec": "snappy" 17 | }, 18 | "schema": [ 19 | { 20 | "name": "TransactionId", 21 | "type": "UTF8" 22 | }, 23 | { 24 | "name": "CustomerId", 25 | "type": "INT32" 26 | }, 27 | { 28 | "name": "ProductId", 29 | "type": "INT_16" 30 | }, 31 | { 32 | "name": "Quantity", 33 | "type": "INT_8" 34 | }, 35 | { 36 | "name": "Price", 37 | "type": "DECIMAL", 38 | "precision": 38, 39 | "scale": 18 40 | }, 41 | { 42 | "name": "TotalAmount", 43 | "type": "DECIMAL", 44 | "precision": 38, 45 | "scale": 18 46 | }, 47 | { 48 | "name": "TransactionDate", 49 | "type": "INT32" 50 | }, 51 | { 52 | "name": "ProfitAmount", 53 | "type": "DECIMAL", 54 | "precision": 38, 55 | "scale": 18 56 | }, 57 | { 58 | "name": "Hour", 59 | "type": "INT_8" 60 | }, 61 | { 62 | "name": "Minute", 63 | "type": "INT_8" 64 | }, 65 | { 66 | "name": "StoreId", 67 | "type": "INT_16" 68 | }, 69 | { 70 | "name": "Year", 71 | "type": "UTF8" 72 | }, 73 | { 74 | "name": "Quarter", 75 | "type": "UTF8" 76 | }, 77 | { 78 | "name": "Month", 79 | "type": "UTF8" 80 | }, 81 | { 82 | "name": "Day", 83 | "type": "UTF8" 84 | } 85 | ] 86 | }, 87 | "type": "Microsoft.Synapse/workspaces/datasets" 88 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/datasets/wwi02_sale_small_asa.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "wwi02_sale_small_asa", 3 | "properties": { 4 | "linkedServiceName": { 5 | "referenceName": "#LINKED_SERVICE_NAME#", 6 | "type": "LinkedServiceReference" 7 | }, 8 | "annotations": [], 9 | "type": "AzureSqlDWTable", 10 | "schema": [ 11 | { 12 | "name": "TransactionId", 13 | "type": "uniqueidentifier" 14 | }, 15 | { 16 | "name": "CustomerId", 17 | "type": "int", 18 | "precision": 10 19 | }, 20 | { 21 | "name": "ProductId", 22 | "type": "smallint", 23 | "precision": 5 24 | }, 25 | { 26 | "name": "Quantity", 27 | "type": "tinyint", 28 | "precision": 3 29 | }, 30 | { 31 | "name": "Price", 32 | "type": "decimal", 33 | "precision": 9, 34 | "scale": 2 35 | }, 36 | { 37 | "name": "TotalAmount", 38 | "type": "decimal", 39 | "precision": 9, 40 | "scale": 2 41 | }, 42 | { 43 | "name": "TransactionDateId", 44 | "type": "int", 45 | "precision": 10 46 | }, 47 | { 48 | "name": "ProfitAmount", 49 | "type": "decimal", 50 | "precision": 9, 51 | "scale": 2 52 | }, 53 | { 54 | "name": "Hour", 55 | "type": "tinyint", 56 | "precision": 3 57 | }, 58 | { 59 | "name": "Minute", 60 | "type": "tinyint", 61 | "precision": 3 62 | }, 63 | { 64 | "name": "StoreId", 65 | "type": "smallint", 66 | "precision": 5 67 | } 68 | ], 69 | "typeProperties": { 70 | "schema": "wwi", 71 | "table": "Sale" 72 | } 73 | }, 74 | "type": "Microsoft.Synapse/workspaces/datasets" 75 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/datasets/wwi02_sale_small_product_quantity_forecast_adls.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "wwi02_sale_small_product_quantity_forecast_adls", 3 | "properties": { 4 | "linkedServiceName": { 5 | "referenceName": "#LINKED_SERVICE_NAME#", 6 | "type": "LinkedServiceReference" 7 | }, 8 | "annotations": [], 9 | "type": "DelimitedText", 10 | "typeProperties": { 11 | "location": { 12 | "type": "AzureBlobFSLocation", 13 | "folderPath": "sale-small-product-quantity-forecast", 14 | "fileSystem": "wwi-02" 15 | }, 16 | "columnDelimiter": ",", 17 | "escapeChar": "\\", 18 | "firstRowAsHeader": true, 19 | "quoteChar": "\"" 20 | }, 21 | "schema": [ 22 | { 23 | "name": "ProductId", 24 | "type": "String" 25 | }, 26 | { 27 | "name": "TransactionDate", 28 | "type": "String" 29 | }, 30 | { 31 | "name": "Hour", 32 | "type": "String" 33 | }, 34 | { 35 | "name": "TotalQuantity", 36 | "type": "String" 37 | } 38 | ] 39 | } 40 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/datasets/wwi02_sale_small_product_quantity_forecast_asa.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "wwi02_sale_small_product_quantity_forecast_asa", 3 | "properties": { 4 | "linkedServiceName": { 5 | "referenceName": "#LINKED_SERVICE_NAME#", 6 | "type": "LinkedServiceReference" 7 | }, 8 | "annotations": [], 9 | "type": "AzureSqlDWTable", 10 | "schema": [ 11 | { 12 | "name": "ProductId", 13 | "type": "int", 14 | "precision": 10 15 | }, 16 | { 17 | "name": "TransactionDate", 18 | "type": "int", 19 | "precision": 10 20 | }, 21 | { 22 | "name": "Hour", 23 | "type": "int", 24 | "precision": 10 25 | }, 26 | { 27 | "name": "TotalQuantity", 28 | "type": "int", 29 | "precision": 10 30 | } 31 | ], 32 | "typeProperties": { 33 | "schema": "wwi", 34 | "table": "ProductQuantityForecast" 35 | } 36 | } 37 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/datasets/wwi02_sale_small_product_review_adls.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "wwi02_sale_small_product_review_adls", 3 | "properties": { 4 | "linkedServiceName": { 5 | "referenceName": "#LINKED_SERVICE_NAME#", 6 | "type": "LinkedServiceReference" 7 | }, 8 | "annotations": [], 9 | "type": "DelimitedText", 10 | "typeProperties": { 11 | "location": { 12 | "type": "AzureBlobFSLocation", 13 | "fileName": "ProductReviews.csv", 14 | "folderPath": "sale-small-product-reviews", 15 | "fileSystem": "wwi-02" 16 | }, 17 | "columnDelimiter": ",", 18 | "escapeChar": "\\", 19 | "firstRowAsHeader": true, 20 | "quoteChar": "\"" 21 | }, 22 | "schema": [ 23 | { 24 | "name": "UserId", 25 | "type": "String" 26 | }, 27 | { 28 | "name": " ProductId", 29 | "type": "String" 30 | }, 31 | { 32 | "name": " ReviewText", 33 | "type": "String" 34 | }, 35 | { 36 | "name": " ReviewDate", 37 | "type": "String" 38 | } 39 | ] 40 | } 41 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/datasets/wwi02_sale_small_product_review_asa.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "wwi02_sale_small_product_review_asa", 3 | "properties": { 4 | "linkedServiceName": { 5 | "referenceName": "#LINKED_SERVICE_NAME#", 6 | "type": "LinkedServiceReference" 7 | }, 8 | "annotations": [], 9 | "type": "AzureSqlDWTable", 10 | "schema": [ 11 | { 12 | "name": "UserId", 13 | "type": "int", 14 | "precision": 10 15 | }, 16 | { 17 | "name": "ProductId", 18 | "type": "int", 19 | "precision": 10 20 | }, 21 | { 22 | "name": "ReviewText", 23 | "type": "nvarchar" 24 | }, 25 | { 26 | "name": "ReviewDate", 27 | "type": "datetime", 28 | "precision": 23, 29 | "scale": 3 30 | } 31 | ], 32 | "typeProperties": { 33 | "schema": "wwi", 34 | "table": "ProductReview" 35 | } 36 | } 37 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/datasets/wwi02_sale_small_stats_adls.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "wwi02_sale_small_stats_adls", 3 | "properties": { 4 | "linkedServiceName": { 5 | "referenceName": "asagadatalake01_adf", 6 | "type": "LinkedServiceReference" 7 | }, 8 | "annotations": [], 9 | "type": "Parquet", 10 | "typeProperties": { 11 | "location": { 12 | "type": "AzureBlobFSLocation", 13 | "folderPath": "sale-small-stats-final", 14 | "fileSystem": "wwi-02" 15 | }, 16 | "compressionCodec": "snappy" 17 | }, 18 | "schema": [ 19 | { 20 | "name": "CustomerId", 21 | "type": "INT32" 22 | }, 23 | { 24 | "name": "TransactionId", 25 | "type": "UTF8" 26 | }, 27 | { 28 | "name": "TransactionTime", 29 | "type": "INT96" 30 | }, 31 | { 32 | "name": "TotalClicksToPurchase", 33 | "type": "INT64" 34 | }, 35 | { 36 | "name": "TotalSecondsToPurchase", 37 | "type": "INT64" 38 | }, 39 | { 40 | "name": "Age", 41 | "type": "INT32" 42 | } 43 | ] 44 | } 45 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/datasets/wwi02_sale_small_stats_asa.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "wwi02_sale_small_stats_asa", 3 | "properties": { 4 | "linkedServiceName": { 5 | "referenceName": "asagasqlpool01_adf", 6 | "type": "LinkedServiceReference" 7 | }, 8 | "annotations": [], 9 | "type": "AzureSqlDWTable", 10 | "schema": [ 11 | { 12 | "name": "CustomerId", 13 | "type": "int", 14 | "precision": 10 15 | }, 16 | { 17 | "name": "TransactionId", 18 | "type": "uniqueidentifier" 19 | }, 20 | { 21 | "name": "TransactionDate", 22 | "type": "datetime", 23 | "precision": 23, 24 | "scale": 3 25 | }, 26 | { 27 | "name": "TotalClicksToPurchase", 28 | "type": "bigint", 29 | "precision": 19 30 | }, 31 | { 32 | "name": "TotalSecondsToPurchase", 33 | "type": "bigint", 34 | "precision": 19 35 | }, 36 | { 37 | "name": "Age", 38 | "type": "int", 39 | "precision": 10 40 | } 41 | ], 42 | "typeProperties": { 43 | "schema": "wwi", 44 | "table": "SaleStatistic" 45 | } 46 | } 47 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/datasets/wwi02_sale_small_telemetry_ade.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "wwi02_sale_small_telemetry_ade", 3 | "properties": { 4 | "linkedServiceName": { 5 | "referenceName": "#LINKED_SERVICE_NAME#", 6 | "type": "LinkedServiceReference" 7 | }, 8 | "annotations": [], 9 | "type": "AzureDataExplorerTable", 10 | "schema": [], 11 | "typeProperties": { 12 | "table": "SalesTelemetry" 13 | } 14 | }, 15 | "type": "Microsoft.Synapse/workspaces/datasets" 16 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/datasets/wwi02_sale_small_telemetry_adls.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "wwi02_sale_small_telemetry", 3 | "properties": { 4 | "linkedServiceName": { 5 | "referenceName": "#LINKED_SERVICE_NAME#", 6 | "type": "LinkedServiceReference" 7 | }, 8 | "annotations": [], 9 | "type": "DelimitedText", 10 | "typeProperties": { 11 | "location": { 12 | "type": "AzureBlobFSLocation", 13 | "folderPath": "sale-small-telemetry", 14 | "fileSystem": "wwi-02" 15 | }, 16 | "columnDelimiter": ",", 17 | "escapeChar": "\\", 18 | "firstRowAsHeader": true, 19 | "quoteChar": "\"" 20 | }, 21 | "schema": [ 22 | { 23 | "name": "CustomerId", 24 | "type": "String" 25 | }, 26 | { 27 | "name": "ProductId", 28 | "type": "String" 29 | }, 30 | { 31 | "name": "Timestamp", 32 | "type": "String" 33 | }, 34 | { 35 | "name": "Url", 36 | "type": "String" 37 | } 38 | ] 39 | }, 40 | "type": "Microsoft.Synapse/workspaces/datasets" 41 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/lab-01-deploy.md: -------------------------------------------------------------------------------- 1 | # Deploy resources for Lab 01 2 | 3 | ## Deploy services for the lab 4 | 5 | Click the `Deploy to Azure` button below to start the deployment process. 6 | 7 | [![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fsolliancenet%2Fazure-synapse-analytics-ga-content-packs%2Fmain%2Fhands-on-labs%2Fsetup%2Farm%2Fasaga-workspace-lab-01.json%3Ftoken%3DAA2FKXX5X2DEGNWNV6LNIT27ZYETC) 8 | 9 | You should see next the `Custom deployment` screen where you need to provide the resource group where the Synapse Analytics workspace was deployed (see [Pre-requisites for deployment](./asa-workspace-deploy.md#pre-requisites-for-deployment) for details): 10 | 11 | Select `Review + create` to validate the settings. 12 | 13 | ![Synapse Analytics workspace deployment configuration](../media/lab-01-deploy-configure.png) 14 | 15 | Once the validation is passed, select `Create` to start the deployment. You should see next an indication of the deployment progress: 16 | 17 | ![Synapse Analytics workspace deployment progress](./../media/lab-01-deploy-progress.png) 18 | 19 | Wait until the deployment completes successfully before proceeding to the next step. 20 | 21 | In the Azure Portal, navigate to the resource group you used to deploy the Synapse Analytics workspace (see [Pre-requisites for deployment](./asa-workspace-deploy.md#pre-requisites-for-deployment) for details) and start a Cloud Shell instance (see [Configure the Azure Cloud Shell](#configure-the-azure-cloud-shell) above for details). 22 | 23 | Once the Cloud Shell instance becomes available, run ```az login``` to make sure the correct account and subscription context are set: 24 | 25 | ![Cloud Shell login](./../media/cloudshell-setup-01.png) 26 | 27 | Run the following command to make sure the Git repository has been correctly cloned (see [Run the global setup script in Cloud Shell](./asa-workspace-deploy.md#run-the-global-setup-script-in-cloud-shell) for details): 28 | 29 | ```cmd 30 | dir 31 | ``` 32 | 33 | Change your current directory using 34 | 35 | ```cmd 36 | cd asa/hands-on-labs/setup/automation 37 | ``` 38 | 39 | and then start the setup script using 40 | 41 | ```powershell 42 | .\lab-01-setup.ps1 43 | ``` 44 | 45 | Make sure the selected subscription is the one that contains the resource group where you deployed the Synapse Analytics workspace: 46 | 47 | ![Cloud Shell select subscription](./../media/cloudshell-setup-03.png) 48 | 49 | Enter the name of the resource group where you deployed the Synapse Analytics workspace: 50 | 51 | ![Cloud Shell select resource group](./../media/cloudshell-setup-04.png) 52 | 53 | The setup script will now proceed to create all necesary Synapse Analytics artifacts in your environment. 54 | 55 | The process should take a few of minutes to finish. Once it completes successfully, you have completed the deployment of the lab. 56 | 57 | ## Report issues 58 | 59 | In case you encounter any issues with the content in this repository, please follow the [How to report issues](./../../report-issues.md) guideline. We will try to address them as soon as possible. Please check your open issues to learn about their status. 60 | -------------------------------------------------------------------------------- /hands-on-labs/setup/lab-02-deploy.md: -------------------------------------------------------------------------------- 1 | # Deploy resources for Lab 02 2 | 3 | ## Deploy services for the lab 4 | 5 | Click the `Deploy to Azure` button below to start the deployment process. 6 | 7 | [![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fsolliancenet%2Fazure-synapse-analytics-ga-content-packs%2Fmain%2Fhands-on-labs%2Fsetup%2Farm%2Fasaga-workspace-lab-02.json%3Ftoken%3DAA2FKXURSMJAZDFEIRRDZVK7Z3WVC) 8 | 9 | You should see next the `Custom deployment` screen where you need to provide the resource group where the Synapse Analytics workspace was deployed (see [Pre-requisites for deployment](./asa-workspace-deploy.md#pre-requisites-for-deployment) for details): 10 | 11 | Select `Review + create` to validate the settings. 12 | 13 | ![Synapse Analytics workspace deployment configuration](../media/lab-02-deploy-configure.png) 14 | 15 | Once the validation is passed, select `Create` to start the deployment. You should see next an indication of the deployment progress: 16 | 17 | ![Synapse Analytics workspace deployment progress](./../media/lab-02-deploy-progress.png) 18 | 19 | Wait until the deployment completes successfully before proceeding to the next step. 20 | 21 | In the Azure Portal, navigate to the resource group you used to deploy the Synapse Analytics workspace (see [Pre-requisites for deployment](./asa-workspace-deploy.md#pre-requisites-for-deployment) for details) and start a Cloud Shell instance (see [Configure the Azure Cloud Shell](#configure-the-azure-cloud-shell) above for details). 22 | 23 | Once the Cloud Shell instance becomes available, run ```az login``` to make sure the correct account and subscription context are set: 24 | 25 | ![Cloud Shell login](./../media/cloudshell-setup-01.png) 26 | 27 | Run the following command to make sure the Git repository has been correctly cloned (see [Run the global setup script in Cloud Shell](./asa-workspace-deploy.md#run-the-global-setup-script-in-cloud-shell) for details): 28 | 29 | ```cmd 30 | dir 31 | ``` 32 | 33 | Change your current directory using 34 | 35 | ```cmd 36 | cd asa/hands-on-labs/setup/automation 37 | ``` 38 | 39 | and then start the setup script using 40 | 41 | ```powershell 42 | .\lab-02-setup.ps1 43 | ``` 44 | 45 | Make sure the selected subscription is the one that contains the resource group where you deployed the Synapse Analytics workspace: 46 | 47 | ![Cloud Shell select subscription](./../media/cloudshell-setup-03.png) 48 | 49 | Enter the name of the resource group where you deployed the Synapse Analytics workspace: 50 | 51 | ![Cloud Shell select resource group](./../media/cloudshell-setup-04.png) 52 | 53 | The setup script will now proceed to create all necesary Synapse Analytics artifacts in your environment. 54 | 55 | The process should take a few minutes to finish. Once it completes successfully, you have completed the deployment of the lab. 56 | 57 | ## Report issues 58 | 59 | In case you encounter any issues with the content in this repository, please follow the [How to report issues](./../../report-issues.md) guideline. We will try to address them as soon as possible. Please check your open issues to learn about their status. 60 | -------------------------------------------------------------------------------- /hands-on-labs/setup/lab-03-deploy.md: -------------------------------------------------------------------------------- 1 | # Deploy resources for Lab 03 2 | 3 | ## Deploy services for the lab 4 | 5 | Click the `Deploy to Azure` button below to start the deployment process. 6 | 7 | [![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fsolliancenet%2Fazure-synapse-analytics-ga-content-packs%2Fmain%2Fhands-on-labs%2Fsetup%2Farm%2Fasaga-workspace-lab-03.json%3Ftoken%3DAA2FKXVXJ2VR6GUPTNMY3RK7Z4Z3C) 8 | 9 | You should see next the `Custom deployment` screen where you need to provide the resource group where the Synapse Analytics workspace was deployed (see [Pre-requisites for deployment](./asa-workspace-deploy.md#pre-requisites-for-deployment) for details): 10 | 11 | Select `Review + create` to validate the settings. 12 | 13 | ![Synapse Analytics workspace deployment configuration](../media/lab-03-deploy-configure.png) 14 | 15 | Once the validation is passed, select `Create` to start the deployment. You should see next an indication of the deployment progress: 16 | 17 | ![Synapse Analytics workspace deployment progress](./../media/lab-03-deploy-progress.png) 18 | 19 | Wait until the deployment completes successfully before proceeding to the next step. 20 | 21 | In the Azure Portal, navigate to the resource group you used to deploy the Synapse Analytics workspace (see [Pre-requisites for deployment](./asa-workspace-deploy.md#pre-requisites-for-deployment) for details) and start a Cloud Shell instance (see [Configure the Azure Cloud Shell](#configure-the-azure-cloud-shell) above for details). 22 | 23 | Once the Cloud Shell instance becomes available, run ```az login``` to make sure the correct account and subscription context are set: 24 | 25 | ![Cloud Shell login](./../media/cloudshell-setup-01.png) 26 | 27 | Run the following command to make sure the Git repository has been correctly cloned (see [Run the global setup script in Cloud Shell](./asa-workspace-deploy.md#run-the-global-setup-script-in-cloud-shell) for details): 28 | 29 | ```cmd 30 | dir 31 | ``` 32 | 33 | Change your current directory using 34 | 35 | ```cmd 36 | cd asa/hands-on-labs/setup/automation 37 | ``` 38 | 39 | and then start the setup script using 40 | 41 | ```powershell 42 | .\lab-03-setup.ps1 43 | ``` 44 | 45 | Make sure the selected subscription is the one that contains the resource group where you deployed the Synapse Analytics workspace: 46 | 47 | ![Cloud Shell select subscription](./../media/cloudshell-setup-03.png) 48 | 49 | Enter the name of the resource group where you deployed the Synapse Analytics workspace: 50 | 51 | ![Cloud Shell select resource group](./../media/cloudshell-setup-04.png) 52 | 53 | The setup script will now proceed to create all necesary Synapse Analytics artifacts in your environment. 54 | 55 | The process should take a few minutes to finish. Once it completes successfully, you have completed the deployment of the lab. 56 | 57 | ## Report issues 58 | 59 | In case you encounter any issues with the content in this repository, please follow the [How to report issues](./../../report-issues.md) guideline. We will try to address them as soon as possible. Please check your open issues to learn about their status. 60 | -------------------------------------------------------------------------------- /hands-on-labs/setup/pipelines/import_poc_customer_data.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "Setup - Copy PoC Customer Data", 3 | "properties": { 4 | "activities": [ 5 | { 6 | "name": "Copy Customer", 7 | "type": "Copy", 8 | "dependsOn": [], 9 | "policy": { 10 | "timeout": "7.00:00:00", 11 | "retry": 0, 12 | "retryIntervalInSeconds": 30, 13 | "secureOutput": false, 14 | "secureInput": false 15 | }, 16 | "userProperties": [], 17 | "typeProperties": { 18 | "source": { 19 | "type": "DelimitedTextSource", 20 | "storeSettings": { 21 | "type": "AzureBlobFSReadSettings", 22 | "recursive": true 23 | }, 24 | "formatSettings": { 25 | "type": "DelimitedTextReadSettings" 26 | } 27 | }, 28 | "sink": { 29 | "type": "SqlDWSink", 30 | "allowPolyBase": true, 31 | "polyBaseSettings": { 32 | "rejectValue": 0, 33 | "rejectType": "value", 34 | "useTypeDefault": true 35 | } 36 | }, 37 | "enableStaging": true, 38 | "stagingSettings": { 39 | "linkedServiceName": { 40 | "referenceName": "#BLOB_STORAGE_LINKED_SERVICE_NAME#", 41 | "type": "LinkedServiceReference" 42 | }, 43 | "path": "staging" 44 | }, 45 | "translator": { 46 | "type": "TabularTranslator", 47 | "mappings": [ 48 | { 49 | "source": { 50 | "name": "CustomerId", 51 | "type": "String" 52 | }, 53 | "sink": { 54 | "name": "CustomerId", 55 | "type": "Int32" 56 | } 57 | }, 58 | { 59 | "source": { 60 | "name": "FirstName", 61 | "type": "String" 62 | }, 63 | "sink": { 64 | "name": "FirstName", 65 | "type": "String" 66 | } 67 | }, 68 | { 69 | "source": { 70 | "name": "MiddleInitial", 71 | "type": "String" 72 | }, 73 | "sink": { 74 | "name": "MiddleInitial", 75 | "type": "String" 76 | } 77 | }, 78 | { 79 | "source": { 80 | "name": "LastName", 81 | "type": "String" 82 | }, 83 | "sink": { 84 | "name": "LastName", 85 | "type": "String" 86 | } 87 | }, 88 | { 89 | "source": { 90 | "name": "FullName", 91 | "type": "String" 92 | }, 93 | "sink": { 94 | "name": "FullName", 95 | "type": "String" 96 | } 97 | }, 98 | { 99 | "source": { 100 | "name": "Gender", 101 | "type": "String" 102 | }, 103 | "sink": { 104 | "name": "Gender", 105 | "type": "String" 106 | } 107 | }, 108 | { 109 | "source": { 110 | "name": "Age", 111 | "type": "String" 112 | }, 113 | "sink": { 114 | "name": "Age", 115 | "type": "Int32" 116 | } 117 | }, 118 | { 119 | "source": { 120 | "name": "BirthDate", 121 | "type": "String" 122 | }, 123 | "sink": { 124 | "name": "BirthDate", 125 | "type": "DateTime" 126 | } 127 | }, 128 | { 129 | "source": { 130 | "name": "Address_PostalCode", 131 | "type": "String" 132 | }, 133 | "sink": { 134 | "name": "Address_PostalCode", 135 | "type": "String" 136 | } 137 | }, 138 | { 139 | "source": { 140 | "name": "Address_Street", 141 | "type": "String" 142 | }, 143 | "sink": { 144 | "name": "Address_Street", 145 | "type": "String" 146 | } 147 | }, 148 | { 149 | "source": { 150 | "name": "Address_City", 151 | "type": "String" 152 | }, 153 | "sink": { 154 | "name": "Address_City", 155 | "type": "String" 156 | } 157 | }, 158 | { 159 | "source": { 160 | "name": "Address_Country", 161 | "type": "String" 162 | }, 163 | "sink": { 164 | "name": "Address_Country", 165 | "type": "String" 166 | } 167 | }, 168 | { 169 | "source": { 170 | "name": "Mobile", 171 | "type": "String" 172 | }, 173 | "sink": { 174 | "name": "Mobile", 175 | "type": "String" 176 | } 177 | }, 178 | { 179 | "source": { 180 | "name": "Email", 181 | "type": "String" 182 | }, 183 | "sink": { 184 | "name": "Email", 185 | "type": "String" 186 | } 187 | } 188 | ] 189 | } 190 | }, 191 | "inputs": [ 192 | { 193 | "referenceName": "wwi02_poc_customer_adls", 194 | "type": "DatasetReference" 195 | } 196 | ], 197 | "outputs": [ 198 | { 199 | "referenceName": "wwi02_poc_customer_asa", 200 | "type": "DatasetReference" 201 | } 202 | ] 203 | } 204 | ], 205 | "annotations": [] 206 | } 207 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/pipelines/import_sale_small_product_review_data.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "Setup - Import product reviews data", 3 | "properties": { 4 | "activities": [ 5 | { 6 | "name": "Copy Product Reviews", 7 | "type": "Copy", 8 | "dependsOn": [], 9 | "policy": { 10 | "timeout": "7.00:00:00", 11 | "retry": 0, 12 | "retryIntervalInSeconds": 30, 13 | "secureOutput": false, 14 | "secureInput": false 15 | }, 16 | "userProperties": [], 17 | "typeProperties": { 18 | "source": { 19 | "type": "DelimitedTextSource", 20 | "storeSettings": { 21 | "type": "AzureBlobFSReadSettings", 22 | "recursive": true 23 | }, 24 | "formatSettings": { 25 | "type": "DelimitedTextReadSettings" 26 | } 27 | }, 28 | "sink": { 29 | "type": "SqlDWSink", 30 | "allowPolyBase": true, 31 | "polyBaseSettings": { 32 | "rejectValue": 0, 33 | "rejectType": "value", 34 | "useTypeDefault": true 35 | } 36 | }, 37 | "enableStaging": true, 38 | "stagingSettings": { 39 | "linkedServiceName": { 40 | "referenceName": "#BLOB_STORAGE_LINKED_SERVICE_NAME#", 41 | "type": "LinkedServiceReference" 42 | }, 43 | "path": "staging" 44 | }, 45 | "translator": { 46 | "type": "TabularTranslator", 47 | "mappings": [ 48 | { 49 | "source": { 50 | "name": "UserId", 51 | "type": "String" 52 | }, 53 | "sink": { 54 | "name": "UserId", 55 | "type": "Int32" 56 | } 57 | }, 58 | { 59 | "source": { 60 | "name": " ProductId", 61 | "type": "String" 62 | }, 63 | "sink": { 64 | "name": "ProductId", 65 | "type": "Int32" 66 | } 67 | }, 68 | { 69 | "source": { 70 | "name": " ReviewText", 71 | "type": "String" 72 | }, 73 | "sink": { 74 | "name": "ReviewText", 75 | "type": "String" 76 | } 77 | }, 78 | { 79 | "source": { 80 | "name": " ReviewDate", 81 | "type": "String" 82 | }, 83 | "sink": { 84 | "name": "ReviewDate", 85 | "type": "DateTime" 86 | } 87 | } 88 | ] 89 | } 90 | }, 91 | "inputs": [ 92 | { 93 | "referenceName": "wwi02_sale_small_product_review_adls", 94 | "type": "DatasetReference" 95 | } 96 | ], 97 | "outputs": [ 98 | { 99 | "referenceName": "wwi02_sale_small_product_review_asa", 100 | "type": "DatasetReference" 101 | } 102 | ] 103 | } 104 | ], 105 | "annotations": [] 106 | } 107 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/pipelines/import_sale_small_stats_data.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "Lab 03 - Import Sales Stats", 3 | "properties": { 4 | "activities": [ 5 | { 6 | "name": "Copy Sales Stats", 7 | "type": "Copy", 8 | "dependsOn": [], 9 | "policy": { 10 | "timeout": "7.00:00:00", 11 | "retry": 0, 12 | "retryIntervalInSeconds": 30, 13 | "secureOutput": false, 14 | "secureInput": false 15 | }, 16 | "userProperties": [], 17 | "typeProperties": { 18 | "source": { 19 | "type": "ParquetSource", 20 | "storeSettings": { 21 | "type": "AzureBlobFSReadSettings", 22 | "recursive": true, 23 | "wildcardFileName": "*.parquet" 24 | } 25 | }, 26 | "sink": { 27 | "type": "SqlDWSink", 28 | "preCopyScript": "truncate table wwi.SaleStatistic", 29 | "allowPolyBase": true, 30 | "polyBaseSettings": { 31 | "rejectValue": 0, 32 | "rejectType": "value", 33 | "useTypeDefault": true 34 | }, 35 | "disableMetricsCollection": false 36 | }, 37 | "enableStaging": true, 38 | "stagingSettings": { 39 | "linkedServiceName": { 40 | "referenceName": "#BLOB_STORAGE_LINKED_SERVICE_NAME#", 41 | "type": "LinkedServiceReference" 42 | }, 43 | "path": "staging" 44 | }, 45 | "translator": { 46 | "type": "TabularTranslator", 47 | "mappings": [ 48 | { 49 | "source": { 50 | "name": "CustomerId", 51 | "type": "Int32" 52 | }, 53 | "sink": { 54 | "name": "CustomerId", 55 | "type": "Int32" 56 | } 57 | }, 58 | { 59 | "source": { 60 | "name": "TransactionId", 61 | "type": "String" 62 | }, 63 | "sink": { 64 | "name": "TransactionId", 65 | "type": "Guid" 66 | } 67 | }, 68 | { 69 | "source": { 70 | "name": "TransactionTime", 71 | "type": "DateTime" 72 | }, 73 | "sink": { 74 | "name": "TransactionDate", 75 | "type": "DateTime" 76 | } 77 | }, 78 | { 79 | "source": { 80 | "name": "TotalClicksToPurchase", 81 | "type": "Int64" 82 | }, 83 | "sink": { 84 | "name": "TotalClicksToPurchase", 85 | "type": "Int64" 86 | } 87 | }, 88 | { 89 | "source": { 90 | "name": "TotalSecondsToPurchase", 91 | "type": "Int64" 92 | }, 93 | "sink": { 94 | "name": "TotalSecondsToPurchase", 95 | "type": "Int64" 96 | } 97 | }, 98 | { 99 | "source": { 100 | "name": "Age", 101 | "type": "Int32" 102 | }, 103 | "sink": { 104 | "name": "Age", 105 | "type": "Int32" 106 | } 107 | } 108 | ] 109 | } 110 | }, 111 | "inputs": [ 112 | { 113 | "referenceName": "wwi02_sale_small_stats_adls", 114 | "type": "DatasetReference" 115 | } 116 | ], 117 | "outputs": [ 118 | { 119 | "referenceName": "wwi02_sale_small_stats_asa", 120 | "type": "DatasetReference" 121 | } 122 | ] 123 | } 124 | ], 125 | "annotations": [] 126 | } 127 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/pipelines/import_sale_small_telemetry_data.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "Setup - Import sales telemetry data", 3 | "properties": { 4 | "activities": [ 5 | { 6 | "name": "Copy Telemetry", 7 | "type": "Copy", 8 | "dependsOn": [], 9 | "policy": { 10 | "timeout": "7.00:00:00", 11 | "retry": 0, 12 | "retryIntervalInSeconds": 30, 13 | "secureOutput": false, 14 | "secureInput": false 15 | }, 16 | "userProperties": [], 17 | "typeProperties": { 18 | "source": { 19 | "type": "DelimitedTextSource", 20 | "storeSettings": { 21 | "type": "AzureBlobFSReadSettings", 22 | "recursive": true, 23 | "wildcardFolderPath": "sale-small-telemetry", 24 | "wildcardFileName": "*.csv" 25 | }, 26 | "formatSettings": { 27 | "type": "DelimitedTextReadSettings" 28 | } 29 | }, 30 | "sink": { 31 | "type": "AzureDataExplorerSink" 32 | }, 33 | "enableStaging": false, 34 | "translator": { 35 | "type": "TabularTranslator", 36 | "mappings": [ 37 | { 38 | "source": { 39 | "name": "CustomerId", 40 | "type": "String", 41 | "physicalType": "String" 42 | }, 43 | "sink": { 44 | "name": "CustomerId", 45 | "type": "Int32" 46 | } 47 | }, 48 | { 49 | "source": { 50 | "name": "ProductId", 51 | "type": "String", 52 | "physicalType": "String" 53 | }, 54 | "sink": { 55 | "name": "ProductId", 56 | "type": "Int32" 57 | } 58 | }, 59 | { 60 | "source": { 61 | "name": "Timestamp", 62 | "type": "String", 63 | "physicalType": "String" 64 | }, 65 | "sink": { 66 | "name": "Timestamp", 67 | "type": "DateTime" 68 | } 69 | }, 70 | { 71 | "source": { 72 | "name": "Url", 73 | "type": "String", 74 | "physicalType": "String" 75 | }, 76 | "sink": { 77 | "name": "Url", 78 | "type": "String" 79 | } 80 | } 81 | ], 82 | "typeConversion": true, 83 | "typeConversionSettings": { 84 | "allowDataTruncation": true, 85 | "treatBooleanAsNumber": false 86 | } 87 | } 88 | }, 89 | "inputs": [ 90 | { 91 | "referenceName": "wwi02_sale_small_telemetry_adls", 92 | "type": "DatasetReference" 93 | } 94 | ], 95 | "outputs": [ 96 | { 97 | "referenceName": "wwi02_sale_small_telemetry_ade", 98 | "type": "DatasetReference" 99 | } 100 | ] 101 | } 102 | ], 103 | "annotations": [] 104 | }, 105 | "type": "Microsoft.Synapse/workspaces/pipelines" 106 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/sql/01-create-logins.sql: -------------------------------------------------------------------------------- 1 | create login [asaga.sql.highperf] with password = '#PASSWORD#' -------------------------------------------------------------------------------- /hands-on-labs/setup/sql/02-create-users.sql: -------------------------------------------------------------------------------- 1 | create user [asaga.sql.highperf] for login [asaga.sql.highperf] 2 | execute sp_addrolemember 'db_owner', 'asaga.sql.highperf' 3 | execute sp_addrolemember 'staticrc80', 'asaga.sql.highperf' -------------------------------------------------------------------------------- /hands-on-labs/setup/sql/03-create-schemas.sql: -------------------------------------------------------------------------------- 1 | create schema wwi 2 | go -------------------------------------------------------------------------------- /hands-on-labs/setup/sql/04-create-tables-in-wwi-schema.sql: -------------------------------------------------------------------------------- 1 | IF OBJECT_ID(N'[wwi].[Product]', N'U') IS NOT NULL 2 | DROP TABLE [wwi].[Product] 3 | 4 | CREATE TABLE [wwi].[Product] 5 | ( 6 | ProductId SMALLINT NOT NULL, 7 | Seasonality TINYINT NOT NULL, 8 | Price DECIMAL(6,2), 9 | Profit DECIMAL(6,2) 10 | ) 11 | WITH 12 | ( 13 | DISTRIBUTION = REPLICATE 14 | ) 15 | 16 | IF OBJECT_ID(N'[wwi].[Date]', N'U') IS NOT NULL 17 | DROP TABLE [wwi].[Date] 18 | 19 | CREATE TABLE [wwi].[Date] 20 | ( 21 | DateId int not null, 22 | Day tinyint not null, 23 | Month tinyint not null, 24 | Quarter tinyint not null, 25 | Year smallint not null 26 | ) 27 | WITH 28 | ( 29 | DISTRIBUTION = REPLICATE 30 | ) 31 | 32 | 33 | IF OBJECT_ID(N'[wwi].[Customer]', N'U') IS NOT NULL 34 | DROP TABLE [wwi].[Customer] 35 | 36 | CREATE TABLE wwi.Customer 37 | ( 38 | CustomerId INT NOT NULL 39 | ,FirstName NVARCHAR(50) NOT NULL 40 | ,MiddleInitial NVARCHAR(10) NULL 41 | ,LastName NVARCHAR(50) NOT NULL 42 | ,FullName NVARCHAR(110) NOT NULL 43 | ,Gender NVARCHAR(15) NULL 44 | ,Age INT NULL 45 | ,BirthDate DATE NULL 46 | ,Address_PostalCode NVARCHAR(200) NULL 47 | ,Address_Street NVARCHAR(2000) NULL 48 | ,Address_City NVARCHAR(2000) NULL 49 | ,Address_Country NVARCHAR(2000) NULL 50 | ,Mobile NVARCHAR(50) NULL 51 | ,Email NVARCHAR(50) NULL 52 | ) 53 | WITH 54 | ( 55 | DISTRIBUTION = REPLICATE 56 | ) 57 | 58 | IF OBJECT_ID(N'[wwi].[Sale]', N'U') IS NOT NULL 59 | DROP TABLE [wwi].[Sale] 60 | 61 | CREATE TABLE [wwi].[Sale] 62 | ( 63 | [TransactionId] [uniqueidentifier] NOT NULL, 64 | [CustomerId] [int] NOT NULL, 65 | [ProductId] [smallint] NOT NULL, 66 | [Quantity] [tinyint] NOT NULL, 67 | [Price] [decimal](9,2) NOT NULL, 68 | [TotalAmount] [decimal](9,2) NOT NULL, 69 | [TransactionDateId] [int] NOT NULL, 70 | [ProfitAmount] [decimal](9,2) NOT NULL, 71 | [Hour] [tinyint] NOT NULL, 72 | [Minute] [tinyint] NOT NULL, 73 | [StoreId] [smallint] NOT NULL 74 | ) 75 | WITH 76 | ( 77 | DISTRIBUTION = HASH ( [CustomerId] ), 78 | CLUSTERED COLUMNSTORE INDEX, 79 | PARTITION 80 | ( 81 | [TransactionDateId] RANGE RIGHT FOR VALUES ( 82 | 20190101, 20190201, 20190301, 20190401, 20190501, 20190601, 20190701, 20190801, 20190901, 20191001, 20191101, 20191201) 83 | ) 84 | ) 85 | 86 | IF OBJECT_ID(N'[wwi].[ProductQuantityForecast]', N'U') IS NOT NULL 87 | DROP TABLE [wwi].[ProductQuantityForecast] 88 | 89 | CREATE TABLE [wwi].[ProductQuantityForecast] 90 | ( 91 | [ProductId] [int] NOT NULL, 92 | [TransactionDate] [int] NOT NULL, 93 | [Hour] [int] NOT NULL, 94 | [TotalQuantity] [int] NOT NULL 95 | ) 96 | WITH 97 | ( 98 | DISTRIBUTION = REPLICATE 99 | ) 100 | 101 | INSERT INTO [wwi].[ProductQuantityForecast] VALUES (100, 20201209, 10, 0) 102 | INSERT INTO [wwi].[ProductQuantityForecast] VALUES (200, 20201209, 10, 0) 103 | INSERT INTO [wwi].[ProductQuantityForecast] VALUES (300, 20201209, 10, 0) 104 | INSERT INTO [wwi].[ProductQuantityForecast] VALUES (400, 20201209, 10, 0) 105 | INSERT INTO [wwi].[ProductQuantityForecast] VALUES (500, 20201209, 10, 0) 106 | INSERT INTO [wwi].[ProductQuantityForecast] VALUES (600, 20201209, 10, 0) 107 | INSERT INTO [wwi].[ProductQuantityForecast] VALUES (700, 20201209, 10, 0) 108 | INSERT INTO [wwi].[ProductQuantityForecast] VALUES (800, 20201209, 10, 0) 109 | INSERT INTO [wwi].[ProductQuantityForecast] VALUES (900, 20201209, 10, 0) 110 | INSERT INTO [wwi].[ProductQuantityForecast] VALUES (1000, 20201209, 10, 0) 111 | INSERT INTO [wwi].[ProductQuantityForecast] VALUES (1100, 20201209, 10, 0) 112 | INSERT INTO [wwi].[ProductQuantityForecast] VALUES (1200, 20201209, 10, 0) 113 | 114 | 115 | IF OBJECT_ID(N'[wwi].[ProductReview]', N'U') IS NOT NULL 116 | DROP TABLE [wwi].[ProductReview] 117 | 118 | CREATE TABLE [wwi].[ProductReview] 119 | ( 120 | [UserId] [int] NOT NULL, 121 | [ProductId] [int] NOT NULL, 122 | [ReviewText] nvarchar(1000) NOT NULL, 123 | [ReviewDate] [datetime] NOT NULL 124 | ) 125 | WITH 126 | ( 127 | DISTRIBUTION = REPLICATE 128 | ) -------------------------------------------------------------------------------- /hands-on-labs/setup/sql/05-create-tables-lab-03.sql: -------------------------------------------------------------------------------- 1 | 2 | IF OBJECT_ID(N'[wwi].[SaleStatistic]', N'U') IS NOT NULL 3 | DROP TABLE [wwi].[SaleStatistic] 4 | 5 | CREATE TABLE [wwi].[SaleStatistic] 6 | ( 7 | [CustomerId] [int] NOT NULL, 8 | [TransactionId] [uniqueidentifier] NOT NULL, 9 | [TransactionDate] [datetime] NOT NULL, 10 | [TotalClicksToPurchase] [bigint] NOT NULL, 11 | [TotalSecondsToPurchase] [bigint] NOT NULL, 12 | [Age] [int] NOT NULL 13 | ) 14 | WITH 15 | ( 16 | DISTRIBUTION = REPLICATE 17 | ) -------------------------------------------------------------------------------- /hands-on-labs/setup/templates/blob_storage_linked_service.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "#LINKED_SERVICE_NAME#", 3 | "type": "Microsoft.Synapse/workspaces/linkedservices", 4 | "properties": { 5 | "annotations": [], 6 | "type": "AzureBlobStorage", 7 | "typeProperties": { 8 | "connectionString": "DefaultEndpointsProtocol=https;AccountName=#STORAGE_ACCOUNT_NAME#;AccountKey=#STORAGE_ACCOUNT_KEY#;EndpointSuffix=core.windows.net;" 9 | } 10 | } 11 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/templates/cosmos_db_linked_service.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "#LINKED_SERVICE_NAME#", 3 | "properties": { 4 | "annotations": [], 5 | "type": "CosmosDb", 6 | "typeProperties": { 7 | "connectionString": "AccountEndpoint=https://#COSMOSDB_ACCOUNT_NAME#.documents.azure.com:443/;Database=#COSMOSDB_DATABASE_NAME#;AccountKey=\"#COSMOSDB_ACCOUNT_KEY#\"" 8 | } 9 | } 10 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/templates/data_explorer_key_vault_linked_service.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "#LINKED_SERVICE_NAME#", 3 | "type": "Microsoft.Synapse/workspaces/linkedservices", 4 | "properties": { 5 | "annotations": [], 6 | "type": "AzureDataExplorer", 7 | "typeProperties": { 8 | "endpoint": "https://#DATA_EXPLORER_CLUSTER#.kusto.windows.net", 9 | "tenant": "#AAD_TENANT_ID#", 10 | "servicePrincipalId": "#AAD_SERVICE_PRINCIPAL_ID#", 11 | "servicePrincipalKey": { 12 | "type": "AzureKeyVaultSecret", 13 | "store": { 14 | "referenceName": "#KEY_VAULT_LINKED_SERVICE_NAME#", 15 | "type": "LinkedServiceReference" 16 | }, 17 | "secretName": "#SECRET_NAME#" 18 | }, 19 | "database": "#DATA_EXPLORER_DATABASE#" 20 | } 21 | } 22 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/templates/data_lake_key_vault_linked_service.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "#LINKED_SERVICE_NAME#", 3 | "properties": { 4 | "annotations": [], 5 | "type": "AzureBlobFS", 6 | "typeProperties": { 7 | "url": "https://#DATA_LAKE_ACCOUNT_NAME#.dfs.core.windows.net", 8 | "accountKey": { 9 | "type": "AzureKeyVaultSecret", 10 | "store": { 11 | "referenceName": "#KEY_VAULT_LINKED_SERVICE_NAME#", 12 | "type": "LinkedServiceReference" 13 | }, 14 | "secretName": "#SECRET_NAME#" 15 | } 16 | } 17 | } 18 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/templates/data_lake_linked_service.json: -------------------------------------------------------------------------------- 1 | { 2 | "name":"#LINKED_SERVICE_NAME#", 3 | "properties": { 4 | "annotations": [], 5 | "type":"AzureBlobFS", 6 | "typeProperties": { 7 | "url":"https://#STORAGE_ACCOUNT_NAME#.dfs.core.windows.net", 8 | "accountKey": { 9 | "type": "SecureString", 10 | "value":"#STORAGE_ACCOUNT_KEY#" 11 | } 12 | } 13 | } 14 | } 15 | -------------------------------------------------------------------------------- /hands-on-labs/setup/templates/integration_runtime.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "#INTEGRATION_RUNTIME_NAME#", 3 | "properties": { 4 | "type": "Managed", 5 | "typeProperties": { 6 | "computeProperties": { 7 | "location": "AutoResolve", 8 | "dataFlowProperties": { 9 | "computeType": "MemoryOptimized", 10 | "coreCount": #CORE_COUNT#, 11 | "timeToLive": #TIME_TO_LIVE# 12 | } 13 | } 14 | } 15 | } 16 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/templates/key_vault_linked_service.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "#LINKED_SERVICE_NAME#", 3 | "type": "Microsoft.Synapse/workspaces/linkedservices", 4 | "properties": { 5 | "annotations": [], 6 | "type": "AzureKeyVault", 7 | "typeProperties": { 8 | "baseUrl": "https://#KEY_VAULT_NAME#.vault.azure.net/" 9 | } 10 | } 11 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/templates/spark_notebook.json: -------------------------------------------------------------------------------- 1 | { 2 | "name":"#NOTEBOOK_NAME#", 3 | "properties":{ 4 | "nbformat":4, 5 | "nbformat_minor":2, 6 | "bigDataPool":{ 7 | "referenceName":"#SPARK_POOL_NAME#", 8 | "type":"BigDataPoolReference" 9 | }, 10 | "sessionProperties":{ 11 | "driverMemory":"28g", 12 | "driverCores":4, 13 | "executorMemory":"28g", 14 | "executorCores":4, 15 | "numExecutors":2 16 | }, 17 | "metadata":{ 18 | "language_info":{ 19 | "name":"python" 20 | }, 21 | "a365ComputeOptions":{ 22 | "id":"/subscriptions/#SUBSCRIPTION_ID#/resourceGroups/#RESOURCE_GROUP_NAME#/providers/Microsoft.Synapse/workspaces/#WORKSPACE_NAME#/bigDataPools/#SPARK_POOL_NAME#", 23 | "name":"#SPARK_POOL_NAME#", 24 | "type":"Spark", 25 | "endpoint":"https://#WORKSPACE_NAME#.dev.azuresynapse.net/livyApi/versions/2019-11-01-preview/sparkPools/#SPARK_POOL_NAME#", 26 | "auth":{ 27 | "type":"AAD", 28 | "authResource":"https://dev.azuresynapse.net" 29 | }, 30 | "sparkVersion":"2.4", 31 | "nodeCount":3, 32 | "cores":4, 33 | "memory":28, 34 | "extraHeader":{ 35 | 36 | } 37 | }, 38 | "sessionKeepAliveTimeout":60 39 | }, 40 | "cells":[ 41 | ] 42 | } 43 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/templates/spark_notebook_session.json: -------------------------------------------------------------------------------- 1 | { 2 | "kind":"pyspark", 3 | "name":"#SPARK_SESSION_NAME#", 4 | "driverMemory":"56g", 5 | "driverCores":8, 6 | "executorMemory":"56g", 7 | "executorCores":8, 8 | "numExecutors":4 9 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/templates/sql_pool_key_vault_linked_service.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "#LINKED_SERVICE_NAME#", 3 | "properties": { 4 | "annotations": [], 5 | "type": "AzureSqlDW", 6 | "typeProperties": { 7 | "connectionString": "Integrated Security=False;Encrypt=True;Connection Timeout=30;Data Source=#WORKSPACE_NAME#.sql.azuresynapse.net;Initial Catalog=#DATABASE_NAME#;User ID=#USER_NAME#", 8 | "password": { 9 | "type": "AzureKeyVaultSecret", 10 | "store": { 11 | "referenceName": "#KEY_VAULT_LINKED_SERVICE_NAME#", 12 | "type": "LinkedServiceReference" 13 | }, 14 | "secretName": "#SECRET_NAME#" 15 | } 16 | } 17 | } 18 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/templates/sql_pool_script.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "#SQL_SCRIPT_NAME#", 3 | "properties": { 4 | "content": { 5 | "query": "#SQL_SCRIPT#", 6 | "metadata": { 7 | "language": "sql" 8 | }, 9 | "currentConnection": { 10 | "name": "#SQL_POOL_NAME#", 11 | "type": "SqlPool" 12 | } 13 | }, 14 | "type": "SqlQuery" 15 | } 16 | } -------------------------------------------------------------------------------- /hands-on-labs/setup/templates/sql_script.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "#SQL_SCRIPT_NAME#", 3 | "properties": { 4 | "content": { 5 | "query": "", 6 | "metadata": { 7 | "language":"sql" 8 | }, 9 | "currentConnection": { 10 | "name": "SQLPool01", 11 | "type":"SqlPool" 12 | } 13 | }, 14 | "type": "SqlQuery" 15 | } 16 | } -------------------------------------------------------------------------------- /presentation/SynapseWhatIsNew.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/presentation/SynapseWhatIsNew.pdf -------------------------------------------------------------------------------- /presentation/SynapseWhatIsNew.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/azure-synapse-analytics-ga-content-packs/da408021ec6f77898d14a5823aac501e63d40550/presentation/SynapseWhatIsNew.pptx -------------------------------------------------------------------------------- /report-issues.md: -------------------------------------------------------------------------------- 1 | # How to report issues 2 | 3 | We are maintaining the content in this repo using GitHub's issue management process. To submit a new issue, please follow these steps: 4 | 5 | - Open the repo page in GitHub and select the `Issues` section. 6 | 7 | ![Open issues section in GitHub](./hands-on-labs/media/issue-find-section.png) 8 | 9 | - Select `New issue` to create a new issue. 10 | 11 | ![Create new issue in GitHub](./hands-on-labs/media/issue-new-issue.png) 12 | 13 | - Provide a short, relevant title for the issue (that also makes it easy for you to find it when multiple issues are open in the same time). Also, provide the detailed description of the issue and mention the lab number, exercise number, and task number. It is usually very helpful to attach one or more screenshots with the context in which the issue manifests itself (you can attach files by either pasting them directly or by dragging and dropping). Select `Submit new issue` once you are done. 14 | 15 | ![Fill issue details](./hands-on-labs/media/issue-details.png) 16 | 17 | >**NOTE**: 18 | > 19 | >Once your issue is submitted, you can return any time to check it's status. It is recommended to check it periodically, as we might respond with comments to the issue's thread (either explaining things or asking for more details in case they are needed). -------------------------------------------------------------------------------- /videos/git-and-workspace-management.md: -------------------------------------------------------------------------------- 1 | # Azure Synapse Analytics Git integration and workspace management 2 | 3 | This video provides an overview of the new Git integration features in Synapse Analytics as well as some of other features related to workspace management (folders and worskpace access control). The following topics are covered: 4 | 5 | - Configuring a Git repository for a Synapse Analytics workspace 6 | - Differences between the collaboration and publish branches 7 | - Committing Synapse artifacts to the Git repo 8 | - Publishing into the publish branch 9 | - Deploying from the publish branch into a production Synapse workspace using an Azure DevOps release pipeine (and the `Synapse workspace deployment` DevOps extension) 10 | - Organizing Synapse artifacts using folders 11 | - New security roles available in Synapse Analytics 12 | 13 | Click on the image below to start the video. 14 | 15 | [![Azure Synapse Analytics Git integration and workspace management](./../hands-on-labs/media/video-02.png)](https://msit.microsoftstream.com/video/00fba0ff-0400-b9eb-3a44-f1eb34ec8faf) 16 | 17 | ## Report issues 18 | 19 | In case you encounter any issues with the content in this repository, please follow the [How to report issues](./../report-issues.md) guideline. We will try to address them as soon as possible. Please check your open issues to learn about their status. 20 | -------------------------------------------------------------------------------- /videos/monitoring.md: -------------------------------------------------------------------------------- 1 | # Azure Synapse Analytics monitoring 2 | 3 | This video provides an overview of the new monitoring capabilities in Synapse Analytics. It covers Az Monitor integration as well as SQL and Spark pool monitoring. 4 | 5 | Click on the image below to start the video. 6 | 7 | [![Synapse Analytics monitoring](./../hands-on-labs/media/video-03.png)](https://msit.microsoftstream.com/video/7f32a1ff-0400-b9eb-303d-f1eb35a5ea8c) 8 | 9 | ## Report issues 10 | 11 | In case you encounter any issues with the content in this repository, please follow the [How to report issues](./../report-issues.md) guideline. We will try to address them as soon as possible. Please check your open issues to learn about their status. 12 | -------------------------------------------------------------------------------- /videos/sqldw-synapse-workspace.md: -------------------------------------------------------------------------------- 1 | # Adding an Azure Synapse Analytics workspace to an existing Azure SQL Data Warehouse instance 2 | 3 | This video provides an overview on how to add an Azure Synapse Analytics workspace on top of an existing Azure Synapse Analytics (formerly SQL DW) instance. A logical SQL server with two dedicated SQL pools (SQL Data Warehouses) gets a full Synapse Analytics workspace added on top of it. SQL pool data availability and some specific Synapse Analytics workspace details are also showcased. 4 | 5 | Click on the image below to start the video. 6 | 7 | [![Adding an Azure Synapse Analytics workspace to an existing Azure SQL Data Warehouse instance](./../hands-on-labs/media/video-01.png)](https://msit.microsoftstream.com/video/00fba0ff-0400-b9eb-48a0-f1eb34ebc78c) 8 | 9 | ## Report issues 10 | 11 | In case you encounter any issues with the content in this repository, please follow the [How to report issues](./../report-issues.md) guideline. We will try to address them as soon as possible. Please check your open issues to learn about their status. 12 | --------------------------------------------------------------------------------