218 |
219 | 9. Replace **
220 |
221 | 222 | ### Part 7 – Queries running slowly 223 | Your user comes to you and says “My query is running slow. I’m not sure why, because I’m not selecting very many rows. Can you help me figure out why? 224 | 225 | 1. Open a PowerShell window.
226 | 2. Change directory to **Query Performance Tuning lab** content folder.
227 | 3. Change directory to **Lab** sub folder.
228 | 4. Run **RunExercise.ps1** script with following parameters. This will execute a query on your server and show the result
229 | `` 230 | .\RunExercise.ps1 -Name Exercise1 -Type Slow 231 | `` 232 |
233 | 234 | 5. Open Query editor of SQL Data Warehouse in Azure Portal.
235 |  236 | 237 | 6. Check the query execution details with using DMVs.
238 | 239 | ``` 240 | SELECT * FROM sys.dm_pdw_exec_requests 241 | WHERE [Label] like 'Exercise1 | Slow%' 242 | ORDER BY submit_time DESC 243 | ``` 244 | 245 | 7. You can use the labels to search for your specific query. Powershell window shows the “Label” that was used during query execution.
246 |
247 | 8. Look for most recent execution of Exercise 1 query (“Running” or “Completed”)
248 | 9. Once you’ve identified the problematic query ID for this scenario, take a deeper look into it by using dm_pdw_request_steps:
249 | 250 | ``` 251 | SELECT * FROM sys.dm_pdw_request_steps 252 | WHERE request_id = 'QID####' 253 | ORDER BY step_index 254 | ``` 255 | 256 | 10. After running these queries, come up with a hypothesis about why the operation may be taking a long time. What are the longest running operations? What might they tell you about how the workflow is structured?
257 |
258 | 259 | ``` 260 | Hint: Moving data will typically be one of the most expensive operations within SQL DW. How is this query moving data and how could it move it more effectively? See the Appendix for more info about data movement types.
261 | Hint: Why is this query moving data? What is the source of the moved data? 262 | Hint: If you’re still stuck, look at your tables in the object explorer (or sys.tables) – what’s different about the table the user is querying? Can you find it where you expect in the object explorer tree? Why not? What type of table is this table?
263 | ``` 264 | 265 | 11. After you have a firm understanding of this slow scenario, run the same query with Fast optionThis will execute a query on your server and show the result.
266 | `` 267 | .\RunExercise.ps1 -Name Exercise1 -Type Fast 268 | `` 269 |
270 | 271 | 12. Run the following querys and check the results
272 | 273 | ``` 274 | SELECT * FROM sys.dm_pdw_exec_requests 275 | WHERE [Label] like 'Exercise1 | Fast%' 276 | ORDER BY submit_time DESC 277 | 278 | SELECT * FROM sys.dm_pdw_request_steps 279 | WHERE request_id = 'QID####' 280 | ORDER BY step_index 281 | ``` 282 | 283 | 13. Compare the results.
284 | 285 |
286 |
287 | 288 | ### Discussion 289 | After steps 1-5, you should have taken a look at the query that was being executed and think about what kind of plan you may expect to see.
290 | At step 10, you should have been able to find your query in exec_requests using the label and determine: Request_ID, total elapsed time, resource class, and first 4,000 characters of the command text. Looking at the command you can see that you are running a select count_big(*) against the 'dbo.lineitem_0' table. You may want to look in object explorer or sys.tables at this point to see a few general things: what type of table is it? If it's distributed, what is the distribution key/column? Are there multi column statistics? Have they been updated? If it's a view, what are the underlying tables? These are all general things you will want to know for most query performance issues.
291 | At step 11, you want to pick out the long-running step based on total_elapsed_time, which in this case was a HadoopRoundRobinOperation. If you look up this movement, it is querying data in an external table and storing the results in PDW in a round robin table. Also notice that the row count is large at about 60 million rows. This is because when copying data from an external table we copy a full version of the table into SQLDW - the predicate is not used to trim this data down yet.
292 | At step 14, you should be able to use exec_requests in the same way you did before to get the query text and see that we are now querying table 'dbo.lineitem_3'. You should look at sys.tables or object explorer in SSMS/SSDT to see the differences between this table and the previous one. You should notice that lineitem_3 is a distributed table whereas Lineitem_2 was an external table. The new plan in request_steps no longer has the Hadoop shuffle because the data is already in SQLDW.
293 | This should illustrate that when you are going to be using external tables repeatedly, it is much more efficient if you first import the table(s) into SQLDW then run the queries on the local tables. Otherwise, every query that touches the external table will have to import the table into tempdb as part of the plan before being able to execute the query. 294 | 295 | ## Part 8 – Joins 296 | Now that you’ve got the hang of things, let’s try the same process on the next exercise. 297 | Again, your user comes to you with questions, complaining that they are joining two of their most important tables together, and SQL DW just isn’t performing as well as they had expected. 298 | 299 | 1. Open a PowerShell window.
300 | 2. Change directory to Query Performance Tuning lab content folder.
301 | 3. Change directory to Lab sub folder.
302 | 4. Run “RunExercise.ps1” script with following parameters
303 | `` 304 | .\RunExercise.ps1 -Name Exercise2 -Type Slow 305 | `` 306 |
307 | 5. Open Query editor of SQL Data Warehouse in Azure Portal.
308 | 6. Check the query execution details with using DMVs.
309 |
310 | 7. You can use the labels to search for your specific query. Powershell window shows the “Label” that was used during query execution.
311 | 8. Look for most recent execution of Exercise 2 query (“Running” or “Completed”)
312 |
313 | 9. Once you’ve identified the problematic query ID for this scenario, take a deeper look into it by using dm_pdw_request_steps:
314 | Some steps of the DSQL plan are mostly overhead and can generally be ignored for purposes of optimizing the plan.
315 | These steps include the RandomIDOperation and the creation of the temporary tables for DMS.
316 | It can often help to add additional predicates to the above query to remove some of the overhead steps thus allowing you to focus on the heavy lifting operations. AND operation_type NOT IN ('RandomIDOperation') AND command NOT LIKE 'CREATE %' AND command NOT LIKE 'DROP %'
317 |
318 | 10. Check the steps and determine which one(s) might be the problematic steps.
319 |
320 | 11. Run the same query with Fast option.
321 | `` 322 | .\RunExercise.ps1 -Name Exercise2 -Type Fast” 323 | `` 324 |
325 | 12. Compare the 2 query execution plans and determine what would be the reason for query slowness.
326 |
327 |
328 | ``` 329 | Hint: Look at the tables that are being joined with the query. Take a look at the table distribution types in the SSMS object explorer. The icon for each table tells you if the table is hash distributed or round robin distributed? What occurs when two round robin tables are joined? 330 | ``` 331 | 332 | ### Discussion: 333 | You should have followed the same workflow as you did in exercise 1 up until step 9. In Step 9 we introduce some helpful additions to the query to filter out some of the SQL OnOperations that are not helpful for troubleshooting. This makes request_steps easier to read.
334 | At step 10, you can see that we are performing 5 broadcast moves and 1 shuffle move. Most of the time was spent in the shuffle move, but the large rowcount on the first broadcast is a point of interest, because remember we generally do not want to be broadcasting large tables.
335 | At step 12, you are comparing the fast plan to the slow plan. You can see in the fast plan that we now have 4 broadcast moves (instead of 5) and 1 shuffle. The table that is no longer being broadcasted is that large table we noticed in step 10. We can get the tables being queries from exec_requests, then look at sys.tables or object explorer to see what kind of tables they are. You will see that the fast version has all hash distributed tables, while the slow version has round robin tables.
336 | In general, you want large fact tables to be distributed tables. In this query both the orders and lineitem tables are large fact tables. If we are joining them together then it is best if they are distribution-compatible, which means distributed on the same key. This way each distribution has just a small slice of data to work with. In the fast version, both of these tables are distributed on orderkey. Round robin tables are never distribution-compatible, so the slow plan has to perform some sort of movement, like a broadcast, to make them distribution compatible before performing the join. The fast version shuffle will be faster because of the smaller input data volume. 337 | 338 | ## Part 9 – Troubleshooting Nuke 339 | Again, your user comes to you with questions, saying “I’ve just loaded my data and my queries are running slow than on SQL Server! What am I missing here?” 340 | 341 | 1. Open a PowerShell window.
342 | 2. Change directory to Query Performance Tuning lab content folder.
343 | 3. Change directory to Lab sub folder.
344 | 4. Run “RunExercise.ps1” script with following parameters
345 | `` 346 | .\RunExercise.ps1 -Name Exercise3 -Type Slow 347 | `` 348 |
349 | 5. Open Query editor of SQL Data Warehouse in Azure Portal.
350 | 6. Check the query execution details with using DMVs.
351 |
352 | 7. You can use the labels to search for your specific query. Powershell window shows the “Label” that was used during query execution.
353 | 8. Look for most recent execution of Exercise 3 query (“Running” or “Completed”)
354 |
355 | 9. Once you’ve identified the problematic query ID for this scenario, take a deeper look into it by using dm_pdw_request_steps:
356 |
357 | 10. Check the steps and determine which one(s) might be the problematic steps.
358 |
359 | 11. Run the same query with Fast option.
360 | `` 361 | .\RunExercise.ps1 -Name Exercise3 -Type Fast 362 | `` 363 |
364 | 12. Compare the 2 query execution plans and determine what would be the reason for query slowness.
365 |
366 |
367 | 368 | ``` 369 | Hint: Look at our best practices (in order) to narrow down what issues cause queries to run slowly.
370 | Hint: The “orders” table is one of the two largest tables and generally too big for a broadcast move. Why did the optimizer choose to create a copy of these rows on all nodes?
371 | Hint: If you’re still having trouble, look at the orders table in the object explorer tree to see what statistics are on the tables in this query. You can see the SQL that is running in the DMV sys.dm_pdw_exec_requests.
372 | ``` 373 | 374 | ### Discussion 375 | You should be able to reach step 10 using the same method you did in the first 2 exercises. In step 10, you should look through the steps for the longest-running step. In this case it's the BroadcastMove that took over 30 seconds. Next you can see that the rowcount for this broadcast is 60 million rows. This is a red flag because broadcasting such a large table will be very expensive. You should have used the original query or the exec_requests DMV to get the command and see what tables you are querying.
376 | In step 12, you can see that this large broadcast is no longer there. If you compare the table types in the slow version to the tables in the fast version you will see that they are the same. However in the fast version you can see in object explorer that there are statistics on the distribution and join columns.
377 | Further, if you run the query provided to get details about the table, you will see that for lineitem_1, the CTL_row_count is 1,000, but the cmp_row_count is ~60 million. 1,000 is the default value for statistics on the control node, so this means that statistics were never manually created. The distributed plan was created assuming it could broadcast this table because there were only 1,000 rows, but in reality there were 60 million rows, which caused our long-running step.
378 | This illustrates how the absence of statistics or statistics not being up to date can affect the distributed plan.
379 | 380 | ## Part 10 – Query performance improvements 381 | Now that your user has got all of their data loaded and organized they are trying out some of their more complex queries. Check this exercise to see if there are any improvements they can make to decrease their query runtime. 382 | 383 | 1. Open a PowerShell window.
384 | 2. Change directory to Query Performance Tuning lab content folder.
385 | 3. Change directory to Lab sub folder.
386 | 4. Run “RunExercise.ps1” script with following parameters
387 | `` 388 | .\RunExercise.ps1 -Name Exercise4 -Type Slow 389 | `` 390 |
391 | 5. Open Query editor of SQL Data Warehouse in Azure Portal.
392 | 6. Check the query execution details with using DMVs.
393 |
394 | 7. You can use the labels to search for your specific query. Powershell window shows the “Label” that was used during query execution
395 | 8. Look for most recent execution of Exercise 4 query (“Running” or “Completed”)
396 |
397 | 9. Check the steps and determine which one(s) might be the problematic steps.
398 |
399 | 10. Run the same query with Fast option.
400 | `` 401 | .\RunExercise.ps1 -Name Exercise4 -Type Fast 402 | `` 403 |
404 | 11. Compare the 2 query execution plans and determine what would be the reason for query slowness.
405 |
406 |
407 | 408 | ``` 409 | Hint: In this example, the query plan is optimal. This query could benefit if it was given more memory. How much memory has been allocated to this query? How can you use sys.dm_pdw_exec_requests to determine the memory grant? How can you change the memory allocation for a query? 410 | ``` 411 | 412 | ## Discussion 413 | In this exercise you will follow the same method to get to step 9. This time we see that the plan is a single step plan - a return operation. From a distributed plan perspective this is ideal because no data movement occurred. We simply were able to run the distributed queries on each distribution and return the results.
414 | For a query like this we cannot improve the MPP plan, so the next option to look at is resource class. You should have noticed that exec_requests shows that the query was running in smallRC. Certain queries will benefit from the larger memory allocation of larger resource classes and this usually requires testing to find the ideal balance of resource class usage for a query vs concurrency.
415 | Once you are at step 11, you should have looked at exec_requests and noticed that it was now running in LargeRC and the execution time was faster. These test queries are pretty fast running because the execution time is low, but for larger queries this can make a big difference. The default resource class is small. Remember, as you increase resource class you also decrease concurrency, so testing is required.
416 | If you want to change the resource class for a query you would use sp_addrolemember and sp_droprolemember 417 | 418 | ### Part 11: Query analysis 419 | Now that you’ve helped your user with some of their initial issues, they’re beginning to bring some of their analysts onto the system – but some analysts are complaining that their queries are taking very long to run or don’t seem to be running at all. 420 | 421 | 1. Open a PowerShell window.
422 | 2. Change directory to Query Performance Tuning lab content folder.
423 | 3. Change directory to Lab sub folder.
424 | 4. Run “RunExercise.ps1” script with following parameters
425 | `` 426 | “.\RunExercise.ps1 -Name Exercise5 -Type Slow” 427 | `` 428 |
429 | 5. This script will create a workload simulation on your server. It will create 20 background jobs which will send queries to your system.
430 | 6. It will wait for 60 seconds for all background jobs properly starts and then will start your problematic query.
431 | 7. Open Query editor of SQL Data Warehouse in Azure Portal.
432 | 8. Check all the active queries.
433 | 434 | ``` 435 | SELECT * FROM sys.dm_pdw_exec_requests 436 | WHERE status not in ('completed', 'failed', 'cancelled') AND session_id <> session_id() 437 | ORDER BY request_id DESC; 438 | ``` 439 | 440 | 9. Can you tell what is happening?
441 | ``` 442 | Hint: What is the state of this query in sys.dm_pdw_exec_requests? Why? 443 | Hint: Run this query. What does it tell you about the state of the query? 444 | Hint: What can be changed to ensure these small queries run? After you investigate connect to ‘demo5_fast’ to see the changes in action. 445 | ``` 446 | 10. You need to kill the background jobs before continuing.
447 | 11. Cancel the running process on current PowerShell window.
448 | 12. Run “.\Kill.ps1”
449 |
450 | 13. Make sure you close the PowerShell window.
451 | 14. Open a PowerShell window.
452 | 15. Change directory to Query Performance Tuning lab content folder.
453 | 16. Change directory to Lab sub folder.
454 | 17. Run “RunExercise.ps1” script with following parameters Run the same query with Fast option.
455 | `` 456 | “.\RunExercise.ps1 -Name Exercise5 -Type Fast” 457 | `` 458 | 459 |
460 | 18. Compare the 2 query execution plans.
461 | 19. You need to kill the background jobs before continuing.
462 | 20. Cancel the running process on current PowerShell window.
463 | 21. Run “.\Kill.ps1” 464 |
465 | 22. Make sure you close the PowerShell window. 466 | 467 | ## Discussion 468 | This exercise tries to simulate an active workload on your data warehouse. It creates 20 background sessions which sends queries constantly.
469 | When you reached to step 9 on Slow execution exercise, you will notice that your query is in the queue (“Suspended”) and does not “Running”. You can check wait stats and see that what is your query is waiting on “UserConcurrencyResourceType” which means that it is waiting for enough concurrency slots become available.
470 | When you check dm_pdw_exec_requests you will notice that this query is running on largerc resource class. In previous example we talk about using higher resource classes allow your query to have more memory resources. But this will result in more memory consumption from the overall system and resulted in less concurrent query executions. So you need to be careful about which resource classes you are using for executing your queries. Always test your queries with your actual workload.
471 | On faster version of this exercise you will notice that your queries might again queued but once there is enough concurrency slots available it will go through the execution. You will see that your query runs 3 times but at every execution it waits on the queue. You can check the queue waiting time by comparing start_time and submit_time in dm_pdw_exec_requests DMV. 472 | -------------------------------------------------------------------------------- /LAB01-Ingesting Data into a Blob with ADF.md: -------------------------------------------------------------------------------- 1 | # Lab: Extract and Load with ADF 2 | 3 | ## Table of Contents 4 | 5 | Summary
6 | Pre-requisites
7 | Scenario
8 | Part 1 – Create an Azure Data Factory (v2)
9 | Part 2 – Connect ADF to a code repository to begin using the ADF GUI (Optional)
10 | Part 3 – Setting up the Connections in the ADF GUI (Azure SQL Database -> Blob)
11 | Part 4 – Setting up the Copy Activity in the ADF GUI (Azure SQL Database -> Blob)
12 | Part 5 – Using Parameters and Triggers (scheduling) in ADF GUI
13 | 14 | ## Summary 15 | This tutorial walks through creating a pipeline copy activity to copy a file from a S3 storage location to an Azure blob storage container, so we can prepare the file to be processed later for transformation.
16 | 17 | In this lab we will:
18 | • Show the graphical user interface for creating a pipeline
19 | • Copy CSV file via a Copy Activity
20 | • Copy Structed data from SQL Azure via a Copy Activity
21 | • Use parameters to make the pipeline easy to change and more reusable
22 | 23 | ## Prerequisites 24 | • Azure Subscription with rights to use/deploy Azure services, and X of Azure credit
25 | • An storage account with a container (container)
26 | • Azure Data Factory
27 | • Visual Studio Team Services Git project (optional)
28 | 29 | ## Scenario 30 | ### Part 1 – Create an Azure Data Factory (v2) 31 | We are going to use the portal to create the Azure Data Factory. 32 | 1. Go to Azure Portal- https://portal.azure.com.
33 | 2. Select **New** on the left menu, select **Analytics**, and then select **Data Factory**.
34 |
35 | 3. On the New data factory page, enter **ad-flab-adf** for Name. 36 | 4. Make sure you select **Version** as **V2 **
37 |
38 | 39 | `` 40 | Note: The name of the Azure data factory must be globally unique. Please modify the name if the Name validation fails. 41 | `` 42 | 43 | 5. After the creation is complete, you see the **Data Factory** page. Select the **Author & Monitor** tile to start the **Azure Data Factory** user interface (UI) application on a separate tab.
44 |
45 | 6. Go to **g1stor Azure Storage** in azure portal 46 |
47 | 7. Create **Blob Container** in **g1stor** Azure Storage as per below instructions:
48 | a. Name: **inputsql**
49 | b. Public access level: **Container**
50 | and Click **OK**
51 |
52 | 53 | ### Part 2 – Connect ADF to a code repository to begin using the ADF GUI (Optional) 54 | One option to be able to sync our code is to connect ADF to a code repository. This section walks through the steps to connect ADF to a Visual Studio Team Services Git project, so we can save our code for later re-use. Note that this is not required but a recommended best practice. 55 | 56 | 1. Navigate to the **Azure portal** within your web browser and navigate to https://portal.azure.com 57 | 2. Open the Azure Data Factory blade **adflab-adf**. 58 | 3. In the Overview Blade, Click on **Author and Monitor**
59 |
60 | 4. Click the **Set up Code Repository** button to begin connecting this Azure data factory to a code repository.
61 |
62 | 5. You can either create a new VSTS account for this lab or use an existing one. You can create a new one -[Here](https://app.vsaex.visualstudio.com/signup/?mkt=en-US&acquisitionType=byDefault) and configure it in ADF GUI under repository settings. You need to create/ select a Project under this Account.
63 |
64 | 6. The Repository Settings pane will appear on the right.
65 |
66 | 7. Select **Use Existing** in select working branch page and click the **Save** button when you have verified your settings. 67 | 68 | 69 | ### Part 3 – Setting up the Connections in the ADF GUI (Azure SQL Database -> Blob) 70 | We now want to use the GUI to create another copy activity in the same pipeline to copy the Data from Azure SQL DB to Azure blob storage to be ready for transformation along with the earlier CSV file. Our first step is setting up the connections and linked services need for the source and destination. 71 | 72 | 1. In the Left Menu click the **Connections** menu item. 73 | 2. In the right pane you need to **Select** working branch as **Master**.
74 |
75 | 3. Click the **+New** button under **Linked** Services.
76 |
77 | 4. Click on the **Azure SQL Database** tile. 78 |
79 | 5. Click **Continue**. 80 | 6. In below image, you should see the properties to configure the **Azure SQL Database** account. We will name this linked service AzureSqlData-base-Source and using the Default runtime. We will use a predefined Azure SQL Database for this lab. 81 | 82 | * Account Selection Method -> **Manual** 83 | * Fully qualified domain name ->**adlabserver.database.windows.net** 84 | * Database name -> **adflab** 85 | * User name -> **lab_user** 86 | * Password -> **P@ssw0rd** 87 | 88 | 7. Click the **Test Connection** to verify settings are entered correctly.
89 |
90 | 8. Click **Finish**. 91 | 92 | ### Part 4 – Setting up the Copy Activity in the ADF GUI (Azure SQL Database -> Blob) 93 | We now want to use the GUI to create a Copy Activity in the pipeline to move the files from the Azure SQL Database as source to our Azure storage destination. 94 | 95 | 1. Click the **Pipeline** in the left menu and then click the **+** sign for adding the pipepline. 96 | 2. In the **Pipeline GUI**, drag the Copy activity (under Move & Transform) to the empty pane above General. 97 | 3. Rename the activity to **AzureSQL-toAzureBlob**.
98 |
99 | 4. Click **Save**. 100 | 5. Click the **Source** Tab in the **Copy Activity GUI**. 101 | 6. Click the **+New** button next to Source **Dataset**.
102 | 7. You should now see the list of source **dataset** connectors. 103 | 8. Choose the **Azure SQL dataset** and click **Finish**. 104 |
105 | 9. You should now set the connection property information as following. 106 | 107 | * Name this **datasetAzureSqlTable**
108 | * Select Table -> **SalesLT.Customer**
109 | * Click **Preview Data** to preview the first several data rows.
110 |
111 | 10. Click on **Finish**. 112 | 11. Click on **CopyPipeline** and then click **AzureSQLtoAzureBlob** Activity. 113 | 12. Select **Source -> Use Query -> Query** 114 | 115 | `` 116 | SELECT CustomerID, CompanyName, SalesPerson, ModifiedDate 117 | FROM [SalesLT].[Customer] 118 | `` 119 | 120 | `` 121 | Note: This Query may change based on your table selection. 122 | ``
123 |
124 | 13. Click on **Preview data** to ensure the query works.
125 | 14. Click back on the **CopyPipeline**.
126 | 15. Click the **AzureSQLtoAzureBlob** copy activity.
127 | 16. Click the **Sink** Tab in the Copy Activity GUI.
128 | 17. Click the **+New** button next to **Source Dataset**.
129 | 18. You should now see the list of sink dataset connectors.
130 | 19. Choose the **Azure Blob storage** dataset and click **Finish**.
131 |
132 | 20. Select the format as **DelimitedText** and click on **Conitnue**.
133 |
134 | 21. Name the dataset as **datasetBlobfromSQL**
135 | 22. Drop down for Linked Service and Click on **+New**.
136 |
137 | 23. Fill the instructions as following:
138 |
139 | 24. **Test Connection** and Click the **Finish** button. 140 | 25. Set the properties. Click browse button and drill down to the **inputsql** container.
141 |
142 | `` 143 | Note: Make sure **inputsql** container exists or else create one first. 144 | `` 145 | 146 | 26. Click the **Finish** button. 147 | 27. Navigate to the **CopyPipeline** 148 | 28. **Save** the changes. Click the **Debug** icon at the top menu to test and run our copy activity.
149 | 150 | ### Part 5 – Using Parameters and Triggers (scheduling) in ADF GUI 151 | You can define parameters at the pipeline level and pass arguments while you're invoking the pipeline on-demand or from a trigger. Activities can consume the arguments that are passed to the pipeline. Using parameters, you can build more flexible pipelines. 152 | And triggers can be used to execute the pipelines on a schedule or on-demand. 153 | 154 | 1. Navigate to **CopyPipeline -> Parameters**. Add new parameter. 155 | 2. Name it as **filename**, let the **Value** be empty.
156 |
157 | 3. Click **Save** 158 | 4. Navigate to **datasetBlobfromSQL -> Parameters -> File Name**, and set the value as ``@pipeline().parameters.filename``
159 |
160 | 8. Navigate to the **CopyPipeline**, do a Test Run.
161 |
162 | 9. It will ask for an input parameter. Enter appropriate name and this will be used as the file name in sink.
163 |
164 | 10. Click on **Finish** (and **Publish**). This will write the changes to Master. 165 | 11. Click on **CopyPipeline -> Triggers -> Add** new trigger.
166 |
167 | 12. Enter the trigger properties accordingly. In this case, we create a daily tumbling window trigger.
168 |
169 | Set Start, End time for **Trigger**. Check **Activated** check-box. 170 | 13. Click **Save**. 171 | 14. In the **Trigger** Run Parameter window,Set **fileName ->** ``copyfromsql_@{formatDateTime(trigger().outputs.windowStartTime, 'yyyy-MM-dd')}`` 172 | 173 | **Note**: Expressions can be changed based on requirements. 174 | 175 | 15. Click **Finish**. 176 | Make sure you **‘Publish’** for the trigger to activated. 177 | 16. Navigate to Monitoring section to see pipeline runs.
178 |
179 | 17. We can find the appropriate parameters being passed during each triggered run. 180 | 18. On Successful run of the CopyPipeline, navigate to the storage locations using Storage Explorer or Azure Portal (Storage Account), to verify the files copied. The filename would be defined by the parameter -> fileName. 181 | 182 | -------------------------------------------------------------------------------- /LAB02-Ingesting_Data_into_a_ADLSGen2_with_ADF.md: -------------------------------------------------------------------------------- 1 | # Ingesting Data into a Azure Data Lake Storage Gen2 with Azure Data Factory 2 | 3 | ## Table of Contents 4 | 5 | Summary
6 | Pre-requisite
7 | Scenario
8 | Part 1 – Create an Azure Data Factory (v2)
9 | Part 2 – Connect ADF to a code repository to begin using the ADF GUI
10 | Part 3 – Setting up the Connections in the ADF GUI (Azure SQL Database -> ADLS)
11 | Part 4 – Setting up the Copy Activity in the ADF GUI (Azure SQL Database -> ADLS)
12 | Part 5 – Using Parameters and Triggers (scheduling) in ADF GUI 13 | 14 | ### Summary 15 | This tutorial walks through creating a pipeline copy activity to copy a file from a S3 storage location to an Azure Data Lake Storage Gen2 container, so we can prepare the file to be processed later for transformation. 16 | 17 | In this lab we will:
18 | • Show the graphical user interface for creating a pipeline
19 | • Copy CSV file via a Copy Activity
20 | • Copy Structed data from SQL Azure via a Copy Activity
21 | • Use parameters to make the pipeline easy to change and more reusable
22 | 23 | ### Prerequisites 24 | • Azure Subscription with rights to use/deploy Azure services, and X of Azure credit
25 | • An ADLS Gen2 storage account with a filesystem (container)
26 | • Azure Data Factory
27 | • Visual Studio Team Services Git project (optional)
28 | 29 | ### Part 1 – Create an Azure Data Factory (v2) 30 | We are going to use the portal to create the Azure Data Factory. 31 | 1. Go to Azure Portal- https://portal.azure.com.
32 | 2. Select **New** on the left menu, select **Analytics**, and then select **Data Factory**.
33 |
34 | 3. On the New data factory page, enter **ad-flab-adf** for Name. 35 | 4. Make sure you select **Version** as **V2**
36 |
37 | 38 | `` 39 | Note: The name of the Azure data factory must be globally unique. Please modify the name if the Name validation fails. 40 | `` 41 | 42 | 5. After the creation is complete, you see the **Data Factory** page. Select the **Author & Monitor** tile to start the **Azure Data Factory** user interface (UI) application on a separate tab.
43 |
44 | 6. Go to **ADLS Gen2 storage account** in azure portal 45 |
46 | 7. Create **File Systems** in **adlsg2** storage account as per below instructions:
47 | a. Name: **inputsql** and Click **OK** 48 |
49 | 50 | ### Part 2 – Connect ADF to a code repository to begin using the ADF GUI (Optional) 51 | One option to be able to sync our code is to connect ADF to a code repository. This section walks through the steps to connect ADF to a Visual Studio Team Services Git project, so we can save our code for later re-use. Note that this is not required but a recommended best practice. 52 | 53 | 1. Navigate to the **Azure portal** within your web browser and navigate to https://portal.azure.com 54 | 2. Open the Azure Data Factory blade **adflab-adf**. 55 | 3. In the Overview Blade, Click on **Author and Monitor**
56 |
57 | 4. Click the **Set up Code Repository** button to begin connecting this Azure data factory to a code repository.
58 |
59 | 5. You can either create a new VSTS account for this lab or use an existing one. You can create a new one -[Here](https://app.vsaex.visualstudio.com/signup/?mkt=en-US&acquisitionType=byDefault) and configure it in ADF GUI under repository settings. You need to create/ select a Project under this Account.
60 |
61 | 6. The Repository Settings pane will appear on the right.
62 |
63 | 7. Select **Use Existing** in select working branch page and click the **Save** button when you have verified your settings. 64 | 65 | 66 | ### Part 3 – Setting up the Connections in the ADF GUI (Azure SQL Database -> ADLS) 67 | We now want to use the GUI to create another copy activity in the same pipeline to copy the Data from Azure SQL DB to Azure blob storage to be ready for transformation along with the earlier CSV file. Our first step is setting up the connections and linked services need for the source and destination. 68 | 69 | 1. In the Left Menu go to **Author**click the **Connections** menu item. 70 | 2. In the right pane you need to **Select** working branch as **Master**.
71 |
72 | 3. Click the **+New** button under **Linked** Services.
73 |
74 | 4. Click on the **Azure SQL Database** tile. 75 |
76 | 5. Click **Continue**. 77 | 6. In below image, you should see the properties to configure the **Azure SQL Database** account. We will name this linked service AzureSqlData-base-Source and using the Default runtime. We will use a predefined Azure SQL Database for this lab. 78 | 79 | * Account Selection Method -> **Manual** 80 | * Fully qualified domain name ->**adlabserver.database.windows.net** 81 | * Database name -> **adflab** 82 | * User name -> **lab_user** 83 | * Password -> **P@ssw0rd** 84 | 85 | 7. Click the **Test Connection** to verify settings are entered correctly.
86 |
87 | 8. Click **Finish**. 88 | 89 | ### Part 4 – Setting up the Copy Activity in the ADF GUI (Azure SQL Database -> ADLS) 90 | We now want to use the GUI to create a Copy Activity in the pipeline to move the files from the Azure SQL Database as source to our Azure storage destination. 91 | 92 | 1. Click the **Pipeline** in the left menu and then click the **+** sign for adding the pipepline. 93 | 2. In the **Pipeline GUI**, drag the Copy activity (under Move & Transform) to the empty pane above General. 94 | 3. Rename the activity to **AzureSQLtoADLSGen2**.
95 | 4. Click **Save**. 96 | 5. Click the **Source** Tab in the **Copy Activity GUI**. 97 |
98 | 6. Click the **+New** button next to Source **Dataset**.
99 | 7. You should now see the list of source **dataset** connectors. 100 | 8. Choose the **Azure SQL dataset** and click **Finish**. 101 |
102 | 9. You should now set the connection property information as following. 103 | 104 | * Name this **datasetAzureSqlTable**
105 | * Select Table -> **SalesLT.Customer**
106 | * Click **Preview Data** to preview the first several data rows.
107 |
108 | 10. Click on **Finish**. 109 | 11. Click on **CopyPipeline** and then click **AzureSQLtoADLSGen2** Activity. 110 | 12. Select **Source -> Use Query -> Query** 111 | 112 | `` 113 | SELECT CustomerID, CompanyName, SalesPerson, ModifiedDate 114 | FROM [SalesLT].[Customer] 115 | `` 116 | 117 | `` 118 | Note: This Query may change based on your table selection. 119 | ``
120 |
121 | 13. Click on **Preview data** to ensure the query works.
122 | 14. Click back on the **CopyPipeline**.
123 | 15. Click the **AzureSQLtoADLSGen2** copy activity.
124 | 16. Click the **Sink** Tab in the Copy Activity GUI.
125 | 17. Click the **+New** button next to **Sink Dataset**.
126 |
127 | 18. You should now see the list of sink dataset connectors.
128 | 19. Choose the **Azure Data lake Storage Gen2** dataset and click **Continue**. 129 |
130 | 20. Select the format as **DelimitedText** and click on **Conitnue**.
131 |
132 | 21. Name the dataset as **datasetADLSgen2fromSQL**
133 | 22. Drop down for **Linked Service** and Click on **+New**.
134 | 23. Fill the instructions as following:
135 |
136 | 24. **Test Connection** and Click the **Finish** button. 137 | 25. Set the properties. Click browse button and drill down to the **inputsql** container.
138 |
139 | `` 140 | Note: Make sure **inputsql** container exists or else create one first. 141 | `` 142 | 143 | 26. Click the **Finish** button. 144 | 27. Navigate to the **CopyPipeline** 145 | 28. **Save** the changes. Click the **Debug** icon at the top menu to test and run our copy activity.
146 | 147 | ### Part 5 – Using Parameters and Triggers (scheduling) in ADF GUI 148 | You can define parameters at the pipeline level and pass arguments while you're invoking the pipeline on-demand or from a trigger. Activities can consume the arguments that are passed to the pipeline. Using parameters, you can build more flexible pipelines. 149 | And triggers can be used to execute the pipelines on a schedule or on-demand. 150 | 151 | 1. Navigate to **CopyPipeline -> Parameters**. Add new parameter. 152 | 2. Name it as **filename**, let the **Value** be empty.
153 |
154 | 3. Click **Save** 155 | 4. Navigate to **datasetADLSgen2fromSQL -> Parameters -> filename**, and set the value as ``@pipeline().parameters.filename``
156 |
157 | 5. Navigate to the **CopyPipeline** and **Debug** it.
158 |
159 | 6. It will ask for an input parameter. Enter appropriate name and this will be used as the file name in sink.``@pipeline().parameters.filename``
160 |
161 | 7. Click on **Finish** (and **Publish**). This will write the changes to Master. 162 | 10. Click on **CopyPipeline -> Triggers -> Add** new trigger.
163 | 9. Enter the trigger properties accordingly. In this case, we create a daily tumbling window trigger.
164 |
165 | Set Start, End time for **Trigger**. Check **Activated** check-box. 166 | 11. Click **Save**. 167 | 12. In the **Trigger** Run Parameter window,Set **fileName ->** ``copyfromsql_@{formatDateTime(trigger().outputs.windowStartTime, 'yyyy-MM-dd')}`` 168 | 169 | `` 170 | Note: Expressions can be changed based on requirements. 171 | `` 172 | 173 | 13. Click **Finish**. 174 | Make sure you **‘Publish’** for the trigger to activated. 175 | 14. Navigate to Monitoring section to see pipeline runs.
176 |
177 | 15. We can find the appropriate parameters being passed during each triggered run. 178 | 16. On Successful run of the CopyPipeline, navigate to the storage locations using Storage Explorer or Azure Portal (Storage Account), to verify the files copied. The filename would be defined by the parameter -> fileName. 179 | -------------------------------------------------------------------------------- /Lab03-Data loading best practices.md: -------------------------------------------------------------------------------- 1 | # Lab03: Migrating a SQL Server to SQL DW 2 | 3 | ## Table of Contents 4 | 5 | Summary
6 | Pre-requisites
7 | Scenario
8 | Part 1 – Loading data into Azure SQL Data Warehouse
9 | Part 2 – Load Dimension tables
10 | Part 3 – Create Partitioned Fact Table
11 | Part 4 – Load data into partitioned staging tables from WASB
12 | Part 5 – Copy Data into Correctly formatted tables via CTAS
13 | Part 6 – Dynamic Management Views
14 | **IMPORTANT: AVOID INCURRING EXTRA CHARGES - PAUSE YOUR SUBSCRIPTION RESOURCES** 15 | 16 | 17 | ## Summary 18 | This tutorial walks through loading best practices for a SQL Data Warehouse. 19 | In this lab we will:
20 | • Create necessary objects to connect to Windows Azure Blob Storage (WASB) container
21 | • Load Dimension tables from a WASB container using best practices on distribution and indexes
22 | • Load a Fact table data from WASB container into a partitioned table via staging tables
23 | • Gain insight into the loading process via DMVs
24 | `` 25 | The full set of scripts needed for this lab can be found on our VM under **Day-1\05.SQLDW** - Loading lab 26 | `` 27 | 28 | ## Pre-requisites 29 | • Azure Subscription with rights to use/deploy Azure services, and $200+ of Azure credit
30 | • Software Requirement:
31 | o Visual Studio 2017 (SQL ops studio)
32 | o Portal query editor
33 | o SQL Server Data Tools 2016 (SSDT).
34 | • SQL Data Warehouse
35 | • Data in Windows Azure Storage Blob
36 | • Access via portal to the storage blob
37 | 38 | ## Scenario 39 | During the Airlift, we will to Part 1 together. In the lab we will use demo data that is already provided on a shared blob storage. 40 | 41 | ### Part 1 – Loading Blob storage data into Azure SQL Data Warehouse 42 | We have created our SQL Data Warehouse and now we want to load data into it. We can do this through the traditional ways of ETL and tooling such as SQL Server Integration Services or third-party tooling. However, today we are going to use Polybase. Your source data has been precreated and is in your Azure Blob Storage account.< 43 | 44 | 1. From your virtual machine navigate to the **Azure portal** within the web browser which should be open from the last exercise. If not, open the browser and navigate to https://portal.azure.com
45 | 2. Open the **Azure SQL Data Warehouse** blade from the tile on the portal dashboard (you pinned it in the earlier exercise).
46 |
47 | 3. Looking at the **Overview** blade you can see the **Common Tasks** as shown in the screen shot.
48 |
49 | 4. Click the **Open in Visual Studio** button.
50 |
51 | ``` 52 | Note: Before opening Visual Studio click on Configure your firewall to make sure that your ClientIP has been added to the rules. 53 | ``` 54 | 55 | 5. Click **Open Link** on the dialog box that ap-pears. **Visual Studio** will now launch.
56 |
57 | 6. Sign in with your given **Azure Credentials**.
58 |
59 | 7. Fill in the **password** specified in **Environment Detail Page**.
60 |
61 |
62 | 8. Click **Connect**.
63 | 9. **Expand** the object tree within the **SQL Server** object explorer pane.
64 | 10. Right click the database name and select **New Query**. A new query window will open
65 |
66 | 11. Open the **Setup** file that can be found in the **LabContent** folder in your drive C:\ under **Day-1\05.SQLDW** - Loading lab from visual studio.
67 |
68 |
69 | 12. Copy the content of **Setup** script and paste it in new query window.
70 |
71 | 13. Execute the **Query**. 72 | 73 | ### Part 2 – Load Dimension tables 74 | Now, we have created our external data source we can query and load the data we have in the Azure Blob Store.In the following lab we will load dimension tables into our SQL DW. Dimension tables are often a good first step because they are relatively small and this will allow you to gain an understanding of how to load data into SQL DW from WASB. 75 | 76 | 1. Open the **Dimensions** file that can be found in the **LabContent** folder in your drive C:\ under **Day-1\05.SQLDW** - Loading lab from visual studio.
77 |
78 |
79 | 2. Copy the **Dimensions.sql** script and replace it with the existing script in query window.
80 |
81 | 2. Execute the **Query**.
82 | 3. Go through the Content inside the dimension for understanding the script:
83 | a. Using the following script to create an external table called Aircraft_IMPORT 84 | ``` 85 | Note: 86 | • data_source - References the Ex-ternal Data Source that you want to read from 87 | 88 | • File_format - References the File Format that the data is in 89 | 90 | • location - Specifies the directory lo-cation that you want to read from. PolyBase traverses all childern di-rectories and files from a stated filepath. 91 | ``` 92 | ``` 93 | CREATE EXTERNAL TABLE Aircraft_IMPORT 94 | ([id] [int] NULL, 95 | [TailNum] [varchar](15) NULL, 96 | [Type] [varchar](50) NULL, 97 | [Manufacturer] [varchar](50) NULL, 98 | [IssueDate] [varchar](15) NULL, 99 | [Model] [varchar](20) NULL, 100 | [Status] [char](5) NULL, 101 | [AircraftType] [varchar](30) NULL, 102 | [EngineType] [varchar](20) NULL, 103 | [Year] [smallint] NULL) 104 | WITH 105 | ( 106 | DATA_SOURCE = MastData_Stor, 107 | FILE_FORMAT = pipe, 108 | LOCATION = 'aircraft' 109 | ) 110 | ``` 111 | 112 | b. Use the following **CTAS** script to create the table and load data
113 | ``` 114 | Note: 115 | * Make sure that you select * From Aircraft_IMPORT you just created.
116 | * Run the following script to update Statstics
117 | * Auto update statistics can take care of automatically updating single column stats, but in this case it is multi-column stats 118 | ``` 119 | ``` 120 | CREATE TABLE Dim_Aircraft 121 | WITH 122 | ( 123 | DISTRIBUTION = ROUND_ROBIN 124 | , CLUSTERED INDEX (id) 125 | ) 126 | AS SELECT * FROM Aircraft_IMPORT 127 | CREATE STATISTICS Aircraft_Stat 128 | ON 129 | Dim_Aircraft (id, type, manufacturer) 130 | ``` 131 | 3. Remainder code will load the all dimension tables. 132 | 133 | ### Part 3 – Create Partitioned Fact Table 134 | To effectively leverage a partition swap load, a table has to exist with an exisiting partition scheme. To do this you must create an empty table with a partitioning scheme. 135 | 136 | 1. To create an empty table partioned by DateID. Open the **2 - Create Fact Table** file that can be found in the **LabContent** folder in your drive C:\ under **Day-1\05.SQLDW** - Loading lab from visual studio.
137 |
138 |
139 | 2. Copy the 2 - **Create Fact Table.dsql** script and replace it with existing content in query window.
140 |
141 | 3. To load the staging tables from WASB into SQDL DW. 142 | 4. Open the **3 - InitialFactLoad.dsql** file that can be found in the **LabContent** folder in your drive C:\ under **Day-1\05.SQLDW** - Loading lab from visual studio.
143 |
144 |
145 | 5. Copy the 3 - **InitialFactLoad.dsql** script and replace it with existing content in query window
146 |
147 | ``` 148 | Note: 149 | • We are using Round_Robin distribution and a Heap because we want to ensure that the load occurs as quickly as possible. Remember ELT. 150 | • Use CTAS external table and create a local table. 151 | ``` 152 | 153 | ### Part 4 – Load data into partitioned staging tables from WASB 154 | In the next set of steps we are going to take the staging tables we created in part 3 and prep the data for a partition switch. 155 | 156 | 1. Open the **4 - PartitionStagingTables.dsql** file that can be found in the **LabContent** folder in your drive C:\ under **Day-1\05.SQLDW** - Loading lab from visual studio.
157 |
158 |
159 | 2. To complete the staging table prep. Copy the script and replace it with existing content in query window
160 |
161 | 162 | ### Part 5 – Copy Data into Correctly formatted tables via CTAS 163 | Now that we have a set of partitioned tables and an empty fact table, we can start doing partition switches into the table. 164 | 165 | 1. The next script that you will run loops through the partitioned tables and dynamically switches the partitions. Because this operation is on the metadata, there is relatively little downtime for the amount of data "loaded" into the production fact table. 166 | 167 | Open the 5 -**LoadWithPartitionSwitch.dsql** file that can be found in the **LabContent** folder in your drive C:\ under **Day-1\05.SQLDW** - Loading lab from visual studio.
168 |
169 |
170 | 2. To switch the partitions on your empty fact table. Run the following script that is part of 5 -**LoadWithPartitionSwitch.dsql** script and replace it with existing content in query window. 171 |
172 | -------------------------------------------------------------------------------- /Lab04 - SQLDW Query Performance Tuning.md: -------------------------------------------------------------------------------- 1 | # SQLDW Query Performance Tuning 2 | 3 | ### Table of Contents 4 | Summary
5 | Pre-requisites: What you need to get started
6 | Part 1: Queries running slowly
7 | Part 2: Joins
8 | Part 3: Troubleshooting
9 | Part 4: Query performance improvements
10 | Part 5: Query analysis 11 | 12 | ## Summary 13 | Over the course of this lab you will look for inefficiencies that are causing sub-optimal performance.
14 | The tables in this lab use the TPCH database.
15 | In each of the examples, there will be two versions of same query. One that is running a poor performing scenario (denoted as slow) and one that is running a more optimal scenario (denoted fast). Your goal should be to study the slow scenario using the DMV queries and SSMS object explorer to decide what is wrong with the slow scenario. Once you feel you understand the issue in the ‘slow’ scenario, connect to the ‘fast’ version to verify your findings. 16 | 17 | ### SQL DW Resources 18 | These articles to help you solve the scenarios presented in this lab.
19 | • Query investigation: https://azure.microsoft.com/en-us/documentation/articles/sql-data-warehouse-managemonitor/
20 | • Best Practices: https://azure.microsoft.com/en-us/documentation/articles/sql-data-warehouse-best-practices/ 21 | 22 | ## Pre-requisites: What you need to get started 23 | 24 | This lab requires you to load a new data source to the Azure Data Warehouse server created in the previous labs. Please follow below steps to load the sample dataset to your server. 25 | The login that you use for running this script should have “Create Login” permission on your server! 26 | This script will create multiple versions of customer, orders, lineitem, part, partsupp, supplier, nation and region tables. These tables will be used during your lab. 27 | You will also edit the PowerShell script and add your server and database names. This will be used during exercises. 28 | 29 | 1. Open a PowerShell window in your virtual machine.
30 | 2. Change directory to Query Performance Tuning lab content folder with given command:
31 | `` 32 | cd "C:\LabContent\Analytics-Airlift-master\Day 1\07.SQLDW - Query tuning lab\Prep" 33 | `` 34 | 3. Change directory to Prep sub folder.
35 | 4. Run ``.\PrepLab.ps1`` script with your Azure Data Warehouse details. It will take around 10-15 minutes.
36 |
37 | 5. Run the following command for changing the PowerShell execution policies for Windows computers.
38 | `` 39 | Set-ExecutionPolicy -Scope Process -ExecutionPolicy ByPass 40 | `` 41 |
42 | 43 | 6. Open your **Query Performance Tuning Lab** content folder.
44 | 7. Change directory to **C:\LabContent\Analytics-Airlift-master\Day 1\07.SQLDW - Query tuning lab\Lab>** sub folder.
45 | 8. Edit “RunExercise.ps1” script.
46 | 9. Replace **
47 |
48 | 10. Replace **
49 |
50 | 51 | ## Part 1: Queries running slowly 52 | Your user comes to you and says “My query is running slow. I’m not sure why, because I’m not selecting very many rows. Can you help me figure out why? 53 | 54 | 1. Open a PowerShell window.
55 | 2. Change directory to **Query Performance Tuning lab** content folder.
56 | 3. Change directory to Lab sub folder.
57 | 4. Run **RunExercise.ps1** script with following parameters. This will execute a query on your server and show the result
58 | 59 | `` 60 | .\RunExercise.ps1 -Name Exercise1 -Type Slow 61 | `` 62 |
63 | 64 | 5. Open Query editor of SQL Data Warehouse in Azure Portal.
65 |
66 | 67 | 6. Check the query execution details with using DMVs.
68 | ``` 69 | SELECT * FROM sys.dm_pdw_exec_requests 70 | WHERE [Label] like 'Exercise1 | Slow%' 71 | ORDER BY submit_time DESC 72 | ``` 73 | 7. You can use the labels to search for your specific query. Powershell window shows the “Label” that was used during query execution.
74 |
75 | 8. Look for most recent execution of Exercise 1 query (“Running” or “Completed”)
76 | 9. Once you’ve identified the problematic query ID for this scenario, take a deeper look into it by using dm_pdw_request_steps:
77 | 78 | ``` 79 | SELECT * FROM sys.dm_pdw_request_steps 80 | WHERE request_id = 'QID####' 81 | ORDER BY step_index 82 | ``` 83 | 84 | 10. After running these queries, come up with a hypothesis about why the operation may be taking a long time. What are the longest running operations? What might they tell you about how the workflow is structured?
85 |
86 | ``` 87 | Hint: Moving data will typically be one of the most expensive operations within SQL DW. How is this query moving data and how could it move it more effectively? See the Appendix for more info about data movement types.
88 | Hint: Why is this query moving data? What is the source of the moved data? 89 | Hint: If you’re still stuck, look at your tables in the object explorer (or sys.tables) – what’s different about the table the user is querying? Can you find it where you expect in the object explorer tree? Why not? What type of table is this table?
90 | ``` 91 | 11. After you have a firm understanding of this slow scenario, run the same query with Fast optionThis will execute a query on your server and show the result.
92 | `` 93 | .\RunExercise.ps1 -Name Exercise1 -Type Fast 94 | `` 95 |
96 | 12. Run the following queries and check the results
97 | ``` 98 | SELECT * FROM sys.dm_pdw_exec_requests 99 | WHERE [Label] like 'Exercise1 | Fast%' 100 | ORDER BY submit_time DESC 101 | 102 | SELECT * FROM sys.dm_pdw_request_steps 103 | WHERE request_id = 'QID####' 104 | ORDER BY step_index 105 | ``` 106 | 13. Compare the results.
107 | 108 |
109 |
110 | 111 | ### Discussion 112 | After steps 1-5, you should have taken a look at the query that was being executed and think about what kind of plan you may expect to see.
113 | At step 10, you should have been able to find your query in exec_requests using the label and determine: Request_ID, total elapsed time, resource class, and first 4,000 characters of the command text. Looking at the command you can see that you are running a select count_big(*) against the 'dbo.lineitem_0' table. You may want to look in object explorer or sys.tables at this point to see a few general things: what type of table is it? If it's distributed, what is the distribution key/column? Are there multi column statistics? Have they been updated? If it's a view, what are the underlying tables? These are all general things you will want to know for most query performance issues.
114 | At step 11, you want to pick out the long-running step based on total_elapsed_time, which in this case was a HadoopRoundRobinOperation. If you look up this movement, it is querying data in an external table and storing the results in PDW in a round robin table. Also notice that the row count is large at about 60 million rows. This is because when copying data from an external table we copy a full version of the table into SQLDW - the predicate is not used to trim this data down yet.
115 | At step 14, you should be able to use exec_requests in the same way you did before to get the query text and see that we are now querying table 'dbo.lineitem_3'. You should look at sys.tables or object explorer in SSMS/SSDT to see the differences between this table and the previous one. You should notice that lineitem_3 is a distributed table whereas Lineitem_2 was an external table. The new plan in request_steps no longer has the Hadoop shuffle because the data is already in SQLDW.
116 | This should illustrate that when you are going to be using external tables repeatedly, it is much more efficient if you first import the table(s) into SQLDW then run the queries on the local tables. Otherwise, every query that touches the external table will have to import the table into tempdb as part of the plan before being able to execute the query. 117 | 118 | ## Part 2: Joins 119 | Now that you’ve got the hang of things, let’s try the same process on the next exercise. 120 | Again, your user comes to you with questions, complaining that they are joining two of their most important tables together, and SQL DW just isn’t performing as well as they had expected. 121 | 122 | 1. Open a PowerShell window.
123 | 2. Change directory to Query Performance Tuning lab content folder.
124 | 3. Change directory to Lab sub folder.
125 | 4. Run “RunExercise.ps1” script with following parameters
126 | `` 127 | .\RunExercise.ps1 -Name Exercise2 -Type Slow 128 | `` 129 |
130 | 5. Open Query editor of SQL Data Warehouse in Azure Portal.
131 | 6. Check the query execution details with using DMVs.
132 | ``` 133 | SELECT * FROM sys.dm_pdw_exec_requests 134 | WHERE [Label] like 'Exercise2 | Slow%' 135 | ORDER BY submit_time DESC 136 | ``` 137 | 138 | 7. You can use the labels to search for your specific query. Powershell window shows the “Label” that was used during query execution.
139 | 8. Look for most recent execution of Exercise 2 query (“Running” or “Completed”)
140 |
141 | 9. Once you’ve identified the problematic query ID for this scenario, take a deeper look into it by using dm_pdw_request_steps:
142 | Some steps of the DSQL plan are mostly overhead and can generally be ignored for purposes of optimizing the plan.
143 | These steps include the RandomIDOperation and the creation of the temporary tables for DMS.
144 | It can often help to add additional predicates to the above query to remove some of the overhead steps thus allowing you to focus on the heavy lifting operations
145 | ``` 146 | SELECT * FROM sys.dm_pdw_request_steps 147 | WHERE request_id = 'QID####' 148 | ORDER BY step_index 149 | ``` 150 | 10. Check the steps and determine which one(s) might be the problematic steps.
151 |
152 | 11. Run the same query with Fast option.
153 | 154 | `` 155 | .\RunExercise.ps1 -Name Exercise2 -Type Fast 156 | `` 157 |
158 | 12. Compare the 2 query execution plans and determine what would be the reason for query slowness.
159 |
160 |
161 | ``` 162 | SELECT * FROM sys.dm_pdw_exec_requests 163 | WHERE [Label] like 'Exercise2 | Fast%' 164 | ORDER BY submit_time DESC 165 | 166 | SELECT * FROM sys.dm_pdw_request_steps 167 | WHERE request_id = 'QID####' 168 | ORDER BY step_index 169 | 170 | ``` 171 | 172 | `` 173 | Hint: Look at the tables that are being joined with the query. Take a look at the table distribution types in the SSMS object explorer. The icon for each table tells you if the table is hash distributed or round robin distributed? What occurs when two round robin tables are joined? 174 | `` 175 | 176 | ### Discussion: 177 | You should have followed the same workflow as you did in exercise 1 up until step 9. In Step 9 we introduce some helpful additions to the query to filter out some of the SQL OnOperations that are not helpful for troubleshooting. This makes request_steps easier to read.
178 | At step 10, you can see that we are performing 5 broadcast moves and 1 shuffle move. Most of the time was spent in the shuffle move, but the large rowcount on the first broadcast is a point of interest, because remember we generally do not want to be broadcasting large tables.
179 | At step 12, you are comparing the fast plan to the slow plan. You can see in the fast plan that we now have 4 broadcast moves (instead of 5) and 1 shuffle. The table that is no longer being broadcasted is that large table we noticed in step 10. We can get the tables being queries from exec_requests, then look at sys.tables or object explorer to see what kind of tables they are. You will see that the fast version has all hash distributed tables, while the slow version has round robin tables.
180 | In general, you want large fact tables to be distributed tables. In this query both the orders and lineitem tables are large fact tables. If we are joining them together then it is best if they are distribution-compatible, which means distributed on the same key. This way each distribution has just a small slice of data to work with. In the fast version, both of these tables are distributed on orderkey. Round robin tables are never distribution-compatible, so the slow plan has to perform some sort of movement, like a broadcast, to make them distribution compatible before performing the join. The fast version shuffle will be faster because of the smaller input data volume. 181 | 182 | ## Part 3: Troubleshooting Nuke 183 | Again, your user comes to you with questions, saying “I’ve just loaded my data and my queries are running slow than on SQL Server! What am I missing here?” 184 | 185 | 1. Open a PowerShell window.
186 | 2. Change directory to Query Performance Tuning lab content folder.
187 | 3. Change directory to Lab sub folder.
188 | 4.Run **RunExercise.ps1** script with following parameters
189 | `` 190 | .\RunExercise.ps1 -Name Exercise3 -Type Slow 191 | `` 192 |
193 | 5. Open Query editor of SQL Data Warehouse in Azure Portal.
194 | 6. Check the query execution details with using DMVs.
195 | ``` 196 | SELECT * FROM sys.dm_pdw_exec_requests 197 | WHERE [Label] like 'Exercise3 | Slow%' 198 | ORDER BY submit_time DESC 199 | 200 | ``` 201 | 7. You can use the labels to search for your specific query. Powershell window shows the “Label” that was used during query execution.
202 | 8. Look for most recent execution of Exercise 3 query (“Running” or “Completed”)
203 |
204 | 9. Once you’ve identified the problematic query ID for this scenario, take a deeper look into it by using dm_pdw_request_steps:
205 | ``` 206 | SELECT * FROM sys.dm_pdw_request_steps 207 | WHERE request_id = 'QID####' 208 | ORDER BY step_index---- 209 | 210 | ``` 211 | 10. Check the steps and determine which one(s) might be the problematic steps.
212 |
213 | 11. Run the same query with Fast option.
214 | `` 215 | .\RunExercise.ps1 -Name Exercise3 -Type Fast 216 | `` 217 |
218 | 12. Compare the 2 query execution plans and determine what would be the reason for query slowness.
219 | ``` 220 | SELECT * FROM sys.dm_pdw_exec_requests 221 | WHERE [Label] like 'Exercise3 | Fast%' 222 | ORDER BY submit_time DESC 223 | 224 | SELECT * FROM sys.dm_pdw_request_steps 225 | WHERE request_id = 'QID####' 226 | ORDER BY step_index 227 | ``` 228 | ``` 229 | Hint: Look at our best practices (in order) to narrow down what issues cause queries to run slowly.
230 | Hint: The “orders” table is one of the two largest tables and generally too big for a broadcast move. Why did the optimizer choose to create a copy of these rows on all nodes?
231 | Hint: If you’re still having trouble, look at the orders table in the object explorer tree to see what statistics are on the tables in this query. You can see the SQL that is running in the DMV sys.dm_pdw_exec_requests.
232 | ``` 233 | ### Discussion 234 | You should be able to reach step 10 using the same method you did in the first 2 exercises. In step 10, you should look through the steps for the longest-running step. In this case it's the BroadcastMove that took over 30 seconds. Next you can see that the rowcount for this broadcast is 60 million rows. This is a red flag because broadcasting such a large table will be very expensive. You should have used the original query or the exec_requests DMV to get the command and see what tables you are querying.
235 | In step 12, you can see that this large broadcast is no longer there. If you compare the table types in the slow version to the tables in the fast version you will see that they are the same. However in the fast version you can see in object explorer that there are statistics on the distribution and join columns.
236 | Further, if you run the query provided to get details about the table, you will see that for lineitem_1, the CTL_row_count is 1,000, but the cmp_row_count is ~60 million. 1,000 is the default value for statistics on the control node, so this means that statistics were never manually created. The distributed plan was created assuming it could broadcast this table because there were only 1,000 rows, but in reality there were 60 million rows, which caused our long-running step.
237 | This illustrates how the absence of statistics or statistics not being up to date can affect the distributed plan.
238 | 239 | ## Part 4: Query performance improvements 240 | Now that your user has got all of their data loaded and organized they are trying out some of their more complex queries. Check this exercise to see if there are any improvements they can make to decrease their query runtime. 241 | 242 | 1. Open a PowerShell window.
243 | 2. Change directory to Query Performance Tuning lab content folder.
244 | 3. Change directory to Lab sub folder.
245 | 4. Run “RunExercise.ps1” script with following parameters
246 | `` 247 | .\RunExercise.ps1 -Name Exercise4 -Type Slow 248 | `` 249 |
250 | 5. Open Query editor of SQL Data Warehouse in Azure Portal.
251 | 6. Check the query execution details with using DMVs.
252 | ``` 253 | SELECT * FROM sys.dm_pdw_exec_requests 254 | WHERE [Label] like 'Exercise4 | Slow%' 255 | ORDER BY submit_time DESC 256 | ``` 257 |
258 | 7. You can use the labels to search for your specific query. Powershell window shows the “Label” that was used during query execution
259 | 8. Look for most recent execution of Exercise 4 query (“Running” or “Completed”)
260 |
261 | 9. Check the steps and determine which one(s) might be the problematic steps.
262 | ``` 263 | SELECT * FROM sys.dm_pdw_request_steps 264 | WHERE request_id = 'QID####' 265 | ORDER BY step_index 266 | ``` 267 |
268 | 10. Run the same query with Fast option.
269 | `` 270 | .\RunExercise.ps1 -Name Exercise4 -Type Fast 271 | `` 272 |
273 | 11. Compare the 2 query execution plans and determine what would be the reason for query slowness.
274 | ``` 275 | SELECT * FROM sys.dm_pdw_exec_requests 276 | WHERE [Label] like 'Exercise4 | Fast%' 277 | ORDER BY submit_time DESC 278 | 279 | SELECT * FROM sys.dm_pdw_request_steps 280 | WHERE request_id = 'QID####' 281 | ORDER BY step_index 282 | 283 | ``` 284 | ``` 285 | Hint: In this example, the query plan is optimal. This query could benefit if it was given more memory. How much memory has been allocated to this query? How can you use sys.dm_pdw_exec_requests to determine the memory grant? How can you change the memory allocation for a query? 286 | ``` 287 | 288 | ## Discussion 289 | In this exercise you will follow the same method to get to step 9. This time we see that the plan is a single step plan - a return operation. From a distributed plan perspective this is ideal because no data movement occurred. We simply were able to run the distributed queries on each distribution and return the results.
290 | For a query like this we cannot improve the MPP plan, so the next option to look at is resource class. You should have noticed that exec_requests shows that the query was running in smallRC. Certain queries will benefit from the larger memory allocation of larger resource classes and this usually requires testing to find the ideal balance of resource class usage for a query vs concurrency.
291 | Once you are at step 11, you should have looked at exec_requests and noticed that it was now running in LargeRC and the execution time was faster. These test queries are pretty fast running because the execution time is low, but for larger queries this can make a big difference. The default resource class is small. Remember, as you increase resource class you also decrease concurrency, so testing is required.
292 | If you want to change the resource class for a query you would use sp_addrolemember and sp_droprolemember 293 | 294 | ### Part 5: Query analysis 295 | Now that you’ve helped your user with some of their initial issues, they’re beginning to bring some of their analysts onto the system – but some analysts are complaining that their queries are taking very long to run or don’t seem to be running at all. 296 | 297 | 1. Open a PowerShell window.
298 | 2. Change directory to Query Performance Tuning lab content folder.
299 | 3. Change directory to Lab sub folder.
300 | 4. Run ``RunExercise.ps1`` script with following parameters
301 | `` 302 | .\RunExercise.ps1 -Name Exercise5 -Type Slow 303 | `` 304 |
305 | 5. This script will create a workload simulation on your server. It will create 20 background jobs which will send queries to your system.
306 | 6. It will wait for 60 seconds for all background jobs properly starts and then will start your problematic query.
307 | 7. Open Query editor of SQL Data Warehouse in Azure Portal.
308 | 8. Check all the active queries.
309 | ``` 310 | SELECT * FROM sys.dm_pdw_exec_requests 311 | WHERE status not in ('completed', 'failed', 'cancelled') AND session_id <> session_id() 312 | ORDER BY request_id DESC; 313 | 314 | ``` 315 | 9. Can you tell what is happening?
316 | `` 317 | Hint: What is the state of this query in sys.dm_pdw_exec_requests? Why? 318 | Hint: Run this query. What does it tell you about the state of the query? 319 | Hint: What can be changed to ensure these small queries run? After you investigate connect to ‘demo5_fast’ to see the changes in action. 320 | `` 321 | 10. You need to kill the background jobs before continuing.
322 | 11. Cancel the running process on current PowerShell window.
323 | 12. Run ``.\Kill.ps1``
324 |
325 | 13. Make sure you close the PowerShell window.
326 | 14. Open a PowerShell window.
327 | 15. Change directory to Query Performance Tuning lab content folder.
328 | 16. Change directory to Lab sub folder.
329 | 17. Run “RunExercise.ps1” script with following parameters Run the same query with Fast option.
330 | `` 331 | “.\RunExercise.ps1 -Name Exercise5 -Type Fast” 332 | `` 333 |
334 | 18. Compare the 2 query execution plans.
335 | 19. You need to kill the background jobs before continuing.
336 | 20. Cancel the running process on current PowerShell window.
337 | 21. Run “.\Kill.ps1” 338 |
339 | 22. Make sure you close the PowerShell window. 340 | 341 | ## Discussion 342 | This exercise tries to simulate an active workload on your data warehouse. It creates 20 background sessions which sends queries constantly.
343 | When you reached to step 9 on Slow execution exercise, you will notice that your query is in the queue (“Suspended”) and does not “Running”. You can check wait stats and see that what is your query is waiting on “UserConcurrencyResourceType” which means that it is waiting for enough concurrency slots become available.
344 | When you check dm_pdw_exec_requests you will notice that this query is running on largerc resource class. In previous example we talk about using higher resource classes allow your query to have more memory resources. But this will result in more memory consumption from the overall system and resulted in less concurrent query executions. So you need to be careful about which resource classes you are using for executing your queries. Always test your queries with your actual workload.
345 | On faster version of this exercise you will notice that your queries might again queued but once there is enough concurrency slots available it will go through the execution. You will see that your query runs 3 times but at every execution it waits on the queue. You can check the queue waiting time by comparing start_time and submit_time in dm_pdw_exec_requests DMV. 346 | 347 | 348 | -------------------------------------------------------------------------------- /Lab05 - ETL with Azure Databricks – Setup Instructions.md: -------------------------------------------------------------------------------- 1 | # ETL with Azure Databricks – Setup Instructions 2 | 3 | ## Overview 4 | In this lab, you will create an end-to-end pipeline containing Lookup, Copy and Databricks notebook activities in ADF.
5 | • Lookup or GetMetadata activity is used to ensure the source dataset is ready for downstream consumption, before triggering the copy and analytics job.
6 | • Copy activity, copies the source file/ dataset to the sink storage. The sink storage is mounted as DBFS in the databricks notebook so that the dataset can be directly consumed by Spark.
7 | • Databricks notebook activity, triggers the databricks notebook that transforms the dataset, and adds it to a processed folder/ SQL DW.
8 | We will not create a scheduled trigger in this template for simplicity, and you may add that if required. 9 |
10 | 11 | ### Prerequisite: 12 | 1. Create a blob storage account and a container called **sinkdata** to be used as sink. Keep a note of the storage account name, container name and Access key will be referenced later in the template.
13 |
14 | 2. Go to **Storage Explorer (preview)** -> **BLOB CONTAINERS** -> **sinkdata** -> **+ New Folder** and give instructions as below:
15 | a. Name: **staged_sink** Click **Ok**
16 |
17 | 3. Now, Upload **Product.csv** file in that **staged_sink** folder. You will get this file in "C:\staged_sink"
18 |
19 | 4. Ensure you have an **Azure Databricks workspace** or create a new one.
20 | 21 | * 4.1- Import the below **Transform** notebook to the databricks workspace. (need not be the same location as below, just remember the path for later usage)
22 | ACTION REQUIRED – Import the notebook for ETL. Import the notebook from URL. Please enter the below URL in the URL field: **https://adflabstaging1.blob.core.windows.net/share/Transformations.html**. Select **Import**.
23 |
24 | * 4.2- Now let’s update the **Transformation** notebook with your storage connection information (name and access key). Go to command 5 in the imported notebook above, replace it with the below code snippet after replacing the highlighted values. Ensure this is the same storage account created earlier and contains the sinkdata container.
25 |
26 | * 4.3 Generate a databricks access token for **ADF** to access databricks. Save the access token for later use in creating a databricks linked service, which looks something like **'dapi32db32cbb4w6eee18b7d87e45exxxxxx'** 27 |
28 |
29 | 30 | ### Create Linked Services and datasets 31 | 1. Create new **linked services** in ADF UI by going to **Connections** -> **Linked services** -> **+ new**
32 | 33 | * 1.1- **Source** – for accessing source data. You can use the public blob storage containing the source files for this sample.
34 | Select **Blob Storage**, use the below **SAS URI** to connect to source storage (read-only access).
35 | ``` 36 | https://storagewithdataformdw.blob.core.windows.net/?sv=2019-02-02&ss=bfqt&srt=sco&sp=rwdlacup&se=2020-05-30T18:40:47Z&st=2020-01-03T10:40:47Z&sip=0.0.0.0-255.255.255.255&spr=https,http&sig=u4%2BQnUWSsVw64B2%2FHDCvPsfRDx9QsLxgJZ0sBRyRCPc%3D 37 | ``` 38 |
39 | * 1.2- **Sink** – for copying data into.
40 | Create a new linked service. Select a storage created in the **prerequisite 1**, in the sink linked service.
41 |
42 | * 1.3- **Databricks** – for connecting to databricks cluster
43 | Create a **databricks** linked service using **access key generated** in **prerequisite 4.c**. If you have an interactive cluster, you may select that. (in this case we use a New job cluster option)
44 |
45 |
46 | 2. Create **datasets**
47 | 48 | * 2.1- Add **Dataset** as shown in image: 49 |
50 | * 2.2- Create **'sourceAvailability_Dataset'** to check if source data is available.
51 | * 2.2.1 Select a data store as **Azure Blob Storage** 52 |
53 | * 2.2.2- Select the fromat type as **DelimitedText** of your data. 54 |
55 | * 2.2.3- Now, create as **'sourceAvailability_Dataset'** and follow instructions as below:
56 | **Linked service** - select **sourceBlob_LS**
57 | **File path** - **data/source/_success**
58 |
59 | 3. Source **dataset** – for copying the source data (using binary copy)
60 | * 3.1- Select a data store as **Azure Blob Storage**
61 |
62 | * 3.2- Select the fromat type as **Binary** of your data.
63 |
64 | * 3.3 Now, create as **'sourceFiles_Dataset'** and follow instructions as below:
65 | **Linked service** - select **sourceBlob_LS**
66 | **File path** - **data/source/Product.csv**
67 |
68 | 4. **Sink dataset** – for copying into the sink destination location
69 | * 4.1- Select a data store as **Azure Blob Storage**
70 |
71 | * 4.2- Select the format type as **Binary** of your data.
72 |
73 | * 4.3- Now, create as **'sinkRawFiles_Dataset'** and follow instructions as below:
74 | **Linked service** - select **sinkBlob_LS**
75 | **File path** - **sinkdata/staged_sink**
76 |
77 | 78 | ### Create activities 79 | a. Create a Lookup activity **Availability flag** for doing a Source Availability check (Lookup or GetMetadata can be used). Select **sourceAvailability_Dataset** created in 2.a.
80 |
81 | b. Create a **Copy** activity **file-to-blob** for copying dataset from source to sink. In this case the data is binary file. Reference the below screenshots for **source** and **sink** configurations in the copy activity.
82 |
83 |
84 | c. Define **pipeline parameters**
85 | ``` 86 | inputPath = /staged_sink 87 | outputPath = /processed_sink 88 | filename = Product.csv 89 | 90 | ``` 91 |
92 | d. Create a databricks **activity**
93 | • Select the **linked** service created in 1.c
94 |
95 | • Configure the settings. Create **Base Parameters** as shown in the screenshot and create parameters to be passed to the databricks notebook from **ADF****. Browse and select the correct notebook path uploaded in prerequisite 2.
96 | ``` 97 | input = @pipeline().parameters.inputPath 98 | output = @pipeline().parameters.outputPath 99 | filename = @pipeline().parameters.fileName 100 | pipelineRunid = @pipeline().RunId 101 | 102 | 103 | ``` 104 |
105 | • Run the **pipeline**. You can find link to databricks logs for more detailed spark logs.
106 |
107 | • You can also verify the data file using **storage explorer**. (For correlating with Data Factory pipeline runs, we appended pipeline run id from data factory to the output folder. This way you can track back the files generated via each run)
108 |
109 | -------------------------------------------------------------------------------- /Lab06-ETL with Azure Data Factory– Setup Instructions.md: -------------------------------------------------------------------------------- 1 | # ETL with Azure Data Factory (Dataflow) – Setup Instructions 2 | 3 | ## Overview 4 | 5 | In this tutorial, you'll use the Azure Data Factory user interface (UX) to create a pipeline that copies and transforms data from an Azure Blob Storage to an Blob Storage sink using mapping data flow. The configuration pattern in this tutorial can be expanded upon when transforming data using mapping data flow. 6 | 7 | Also, we will go through the necessary dataflow transformations and actions to return a DataFrame which satisfies following requirements: 8 | * Select just the product_id, category, brand, model, size and price columns 9 | * Rename product_id to prodID 10 | * Sort by the price column in descending order 11 | * Create new column doublePrice, and multiply the price column by 2. This is for demonstration, in reality you would populate the field using some business logic to make it more meaningful 12 | * Write the transformed DataFrame in CSV as Product.csv 13 | 14 |
15 |
16 | ## Exercise 1.1 - Create Linked Service and Datasets
17 |
18 | 1. In Azure Portal, go to **Resource Group** and **Click** on azure data factory.19 |
20 | 2. Select **Author & Monitor** to launch the Data Factory UI in a separate tab.
21 |
22 | 1. Create new **linked services** in ADF UI by going to **Connections** -> **Linked services** -> **+ new**
23 | a. **Source** – for accessing source data. You can use the public blob storage containing the source files for this sample.
24 | Select **Blob Storage**, use the below **Account Key** to connect to source storage.
25 | ``` 26 | Storage Account Name: <-Given in your environment page-> 27 | Storage Account Key: <-Given in your environment page-> 28 | ``` 29 |
30 | b. **Sink** – for copying data into.
31 | Create a new linked service. Select a storage created in the **prerequisite**, in the sink linked service.
32 |
33 | 2. Create **datasets**
34 | a. Add **Dataset** as shown in image:
35 |
36 | b. Create **'sourceFiles_Dataset'** to check if source data is available.
37 | i. Select a data store as **Azure Blob Storage** 38 |
39 | ii. Select the fromat type as **DelimitedText** of your data. 40 |
41 | iii. Now, create as **'sourceFiles_Dataset'** and follow instructions as below:
42 | **Linked service** - select **sourceBlob_LS**
43 | **File path** - **data/source/Product.csv**
44 | Select the **First Row as Header**
45 |
46 | c. **Sink dataset** – for copying into the sink destination location
47 | i. Select a data store as **Azure Blob Storage**
48 |
49 | ii. Select the fromat type as **DelimitedText** of your data.
50 |
51 | iii. Now, create as **'sinkRawFiles_Dataset'** and follow instructions as below:
52 | **Linked service** - select **sinkBlob_LS**
53 | **File path** - **sinkdata**
54 | Select the **First Row as Header**
55 |
56 | 57 | ## Exercise 1.2 Build a mapping data flow with six transformations 58 | 59 | 1. In the factory top bar, slide the Data Flow debug slider on. Debug mode allows for interactive testing of transformation logic against a live Spark cluster. Data Flow clusters take 5-7 minutes to warm up and users are recommended to turn on debug first if they plan to do Data Flow development. For more information, see Debug Mode.
60 |
61 | 2. In Factory Resouces, select Data flows and add **New Data flow**. Select **Mapping Data Flow** while creating DataFlow. Once you create your Data Flow, you'll be automatically sent to the data flow canvas. In this step, you'll build a data flow that takes the Product.csv in Blob storage.
62 |
63 |
64 | 3. In the data flow canvas, add a source by clicking on the **Add Source** box
65 | * Name: **Give any name**
66 |
67 | 4. Name your source **Source**. Select **sourceFiles_Dataset** from dropdown.
68 |
69 | 5. If your debug cluster has started, go to the Data Preview tab of the source transformation and click Refresh to get a snapshot of the data. You can use data preview to verify your transformation is configured correctly.
70 | 6. Next to your source node on the data flow canvas, click on the plus icon to add a new transformation. The first transformation you're adding is a **Derived Colunm**. The Derived Column transformation in ADF Data Flows is a multi-use transformation. While it is generally used for writing expressions for data transformation, you can also use it for data type casting and you can even modify metadata with it.
71 |
72 | 7. Name your **Derived Colunm** transformation **Casting**. Add Colunm and define the expression as shown below image:
73 |
74 | 8. Add below expression in **Expression for field**
75 | `` 76 | toInteger(product_id) 77 | `` 78 |
79 |
80 | 9. Add other columns **product_id**, **category**, **brand**, **model**, **size**, and **price**. Add new colunm as **doublePrice** and in expression field give **toFloat(price)** as a value.
81 |
82 | 10. The next transformation you'll add is an **Select** transformation under **Schema modifier**.
83 |
84 | 11. Using select transformation will Rename the **product_id** to **prodID**. Name your **Select** transformation as **UpdateName**.
85 |
86 | 12. The next transformation you'll add is an **Derived Colunm** transformation. Will use this transformation for doublePrice, and multiply the price column by 2. This is for demonstration, in reality you would populate the field using some business logic to make it more meaningful.
87 |
88 | 13. Name the transformation as **Double Price**. In **Colunms** section, select **doublePrice**.
89 |
90 | 14. Add following expression and save the expression:
91 | 92 | `` 93 | multiply(price, 2) 94 | `` 95 |
96 | 15. The next transformation you'll add is an **sort** transformation under **Row modifier**. Will sort the price column in descending order.
97 | 16. Name the transformation as **Sort** and sort the condition as below:
98 |
99 | 17. Next, you want to add a **Sink** transformation under Destination.
100 |
101 | 18. Name your Output stream as **Sink**. Select the **sinkRawFiles_Dataset** dataset from drop down.
102 |
103 | 19. Click on **Publish All** for saving the all solution. 104 | 105 | Now you've finished building your data flow. You're ready to run it in your pipeline. 106 | 107 | ## Exercise 3 Running and monitoring the Data Flow 108 | 109 | You can debug a pipeline before you publish it. In this step, you're going to trigger a debug run of the data flow pipeline. While data preview doesn't write data, a debug run will write data to your sink destination.
110 | 111 | 1. Add Pipeline **Factory Resources**.
112 |
113 | 2. In the Activities pane, expand the **Move and Transform** accordion. Drag and drop the Data Flow activity from the pane to the pipeline canvas.
114 |
115 | 3. In the Adding Data Flow pop-up, select **Use existing Data Flow** and select **dataflow** from drop down. Click Finish when done.
116 |
117 | 4. Go to the pipeline canvas. Click **Debug** to trigger a debug run.
118 |
119 | 5. Pipeline debug of **Data Flow** activities uses the active debug cluster but still take at least a minute to initialize. You can track the progress via the Output tab. Once the run is successful, click on the eyeglasses icon to open the monitoring pane. 120 |
121 | 6. In the monitoring pane, you can see the number of rows and time spent in each transformation step. 122 |
123 | 7. Click on a transformation to get detailed information about the columns and partitioning of the data. 124 |
125 | 8. Go back to your **Azure portal** and open **Blob Storage** -> **Containers** -> **sinkdata**. Download the **Product.csv** using the **URL** and check the CSV file result
126 |
127 |
128 | -------------------------------------------------------------------------------- /ModernDataWarehouse-Analytics-In-A-Day.md: -------------------------------------------------------------------------------- 1 | # Modern Data Warehouse (Analytics in a Day) 2 | 3 | ### Table of Contents 4 | Summary
5 | Part 1 – Loading Blob storage data into Azure SQL Data Warehouse
6 | Part 2 – Load Dimension tables
7 | Part 3 – Create Partitioned Fact Table
8 | Part 4 – Load data into partitioned staging tables from WASB
9 | Part 5 – Copy Data into Correctly formatted tables via CTASr
10 | Part 6 – Load the sample dataset to your server
11 | Part 7 – Queries running slowly
12 | Part 8 – Joins
13 | Part 9 – Troubleshooting Nuke
14 | Part 10 – Query performance improvements
15 | Part 11: Query analysis 16 | 17 | ## Summary 18 | Over the course of this lab you will look for inefficiencies that are causing sub-optimal performance.
19 | The tables in this lab use the TPCH database.
20 | In each of the examples, there will be two versions of same query. One that is running a poor performing scenario (denoted as slow) and one that is running a more optimal scenario (denoted fast). Your goal should be to study the slow scenario using the DMV queries and SSMS object explorer to decide what is wrong with the slow scenario. Once you feel you understand the issue in the ‘slow’ scenario, connect to the ‘fast’ version to verify your findings. 21 | 22 | ### SQL DW Resources 23 | These articles to help you solve the scenarios presented in this lab.
24 | • Query investigation: https://azure.microsoft.com/en-us/documentation/articles/sql-data-warehouse-managemonitor/
25 | • Best Practices: https://azure.microsoft.com/en-us/documentation/articles/sql-data-warehouse-best-practices/ 26 | 27 | ### Part 1 – Loading Blob storage data into Azure SQL Data Warehouse 28 | We have created our SQL Data Warehouse and now we want to load data into it. We can do this through the traditional ways of ETL and tooling such as SQL Server Integration Services or third-party tooling. However, today we are going to use Polybase. Your source data has been precreated and is in your Azure Blob Storage account.< 29 | 30 | 1. From virtual machine that is given in your environment, navigate to the **Azure portal** open the browser and navigate to https://portal.azure.com
31 | 2. Open the **Azure SQL Data Warehouse** blade from the tile on the portal dashboard.
32 |
33 | 3. Looking at the **Overview** blade you can see the **Common Tasks** as shown in the screen shot.
34 |
35 | 4. Click the **Open in Visual Studio** button.
36 |
37 | ``` 38 | Note: Before opening Visual Studio click on Configure your firewall to make sure that your ClientIP has been added to the rules. 39 | ``` 40 | 41 | 5. Click **Open Link** on the dialog box that ap-pears. **Visual Studio** will now launch.
42 |
43 | 6. Sign in with your given **Azure Credentials**.
44 |
45 | 7. Fill in the **password** specified in **Environment Detail Page**.
46 |
47 |
48 | 8. Click **Connect**.
49 | 9. **Expand** the object tree within the **SQL Server** object explorer pane.
50 | 10. Right click the database name and select **New Query**. A new query window will open
51 |
52 | 11. Open the **Setup** file that can be found in the **LabContent** folder in your drive C:\ under **Day-1\05.SQLDW** - Loading lab from visual studio.
53 |
54 |
55 | 12. Copy the content of **Setup** script and paste it in new query window.
56 |
57 | 13. Execute the **Query**. 58 | 59 | ### Part 2 – Load Dimension tables 60 | Now, we have created our external data source we can query and load the data we have in the Azure Blob Store.In the following lab we will load dimension tables into our SQL DW. Dimension tables are often a good first step because they are relatively small and this will allow you to gain an understanding of how to load data into SQL DW from WASB. 61 | 62 | 1. Open the **Dimensions** file that can be found in the **LabContent** folder in your drive C:\ under **Day-1\05.SQLDW** - Loading lab from visual studio.
63 |
64 |
65 | 2. Copy the **Dimensions.sql** script and replace it with the existing script in query window.
66 |
67 | 2. Execute the **Query**.
68 | 3. Go through the Content inside the dimension for understanding the script:
69 | a. Using the following script to create an external table called Aircraft_IMPORT 70 | ``` 71 | Note: 72 | • data_source - References the Ex-ternal Data Source that you want to read from 73 | 74 | • File_format - References the File Format that the data is in 75 | 76 | • location - Specifies the directory lo-cation that you want to read from. PolyBase traverses all childern di-rectories and files from a stated filepath. 77 | ``` 78 | ``` 79 | CREATE EXTERNAL TABLE Aircraft_IMPORT 80 | ([id] [int] NULL, 81 | [TailNum] [varchar](15) NULL, 82 | [Type] [varchar](50) NULL, 83 | [Manufacturer] [varchar](50) NULL, 84 | [IssueDate] [varchar](15) NULL, 85 | [Model] [varchar](20) NULL, 86 | [Status] [char](5) NULL, 87 | [AircraftType] [varchar](30) NULL, 88 | [EngineType] [varchar](20) NULL, 89 | [Year] [smallint] NULL) 90 | WITH 91 | ( 92 | DATA_SOURCE = MastData_Stor, 93 | FILE_FORMAT = pipe, 94 | LOCATION = 'aircraft' 95 | ) 96 | ``` 97 | 98 | b. Use the following **CTAS** script to create the table and load data
99 | ``` 100 | Note: 101 | * Make sure that you select * From Aircraft_IMPORT you just created.
102 | * Run the following script to update Statstics
103 | * Auto update statistics can take care of automatically updating single column stats, but in this case it is multi-column stats 104 | ``` 105 | ``` 106 | CREATE TABLE Dim_Aircraft 107 | WITH 108 | ( 109 | DISTRIBUTION = ROUND_ROBIN 110 | , CLUSTERED INDEX (id) 111 | ) 112 | AS SELECT * FROM Aircraft_IMPORT 113 | CREATE STATISTICS Aircraft_Stat 114 | ON 115 | Dim_Aircraft (id, type, manufacturer) 116 | ``` 117 | 3. Remainder code will load the all dimension tables. 118 | 119 | ### Part 3 – Create Partitioned Fact Table 120 | To effectively leverage a partition swap load, a table has to exist with an exisiting partition scheme. To do this you must create an empty table with a partitioning scheme. 121 | 122 | 1. To create an empty table partioned by DateID. Open the **2 - Create Fact Table** file that can be found in the **LabContent** folder in your drive C:\ under **Day-1\05.SQLDW** - Loading lab from visual studio.
123 |
124 |
125 | 2. Copy the 2 - **Create Fact Table.dsql** script and replace it with existing content in query window.
126 |
127 | 3. To load the staging tables from WASB into SQDL DW. 128 | 4. Open the **3 - InitialFactLoad.dsql** file that can be found in the **LabContent** folder in your drive C:\ under **Day-1\05.SQLDW** - Loading lab from visual studio.
129 |
130 |
131 | 5. Copy the 3 - **InitialFactLoad.dsql** script and replace it with existing content in query window
132 |
133 | ``` 134 | Note: 135 | • We are using Round_Robin distribution and a Heap because we want to ensure that the load occurs as quickly as possible. Remember ELT. 136 | • Use CTAS external table and create a local table. 137 | ``` 138 | 139 | ### Part 4 – Load data into partitioned staging tables from WASB 140 | In the next set of steps we are going to take the staging tables we created in part 3 and prep the data for a partition switch. 141 | 142 | 1. Open the **4 - PartitionStagingTables.dsql** file that can be found in the **LabContent** folder in your drive C:\ under **Day-1\05.SQLDW** - Loading lab from visual studio.
143 |
144 |
145 | 2. To complete the staging table prep. Copy the script and replace it with existing content in query window
146 |
147 | 148 | ### Part 5 – Copy Data into Correctly formatted tables via CTAS 149 | Now that we have a set of partitioned tables and an empty fact table, we can start doing partition switches into the table. 150 | 151 | 1. The next script that you will run loops through the partitioned tables and dynamically switches the partitions. Because this operation is on the metadata, there is relatively little downtime for the amount of data "loaded" into the production fact table. 152 | 153 | Open the 5 -**LoadWithPartitionSwitch.dsql** file that can be found in the **LabContent** folder in your drive C:\ under **Day-1\05.SQLDW** - Loading lab from visual studio.
154 |
155 |
156 | 2. To switch the partitions on your empty fact table. Run the following script that is part of 5 -**LoadWithPartitionSwitch.dsql** script and replace it with existing content in query window. 157 |
158 | 159 | ## Part 6 – Load the sample dataset to your server 160 | 161 | This part requires you to load a new data source to the Azure Data Warehouse server created in the previous parts. Please follow below steps to load the sample dataset to your server. The login that you use for running this script should have “Create Login” permission on your server! This script will create multiple versions of customer, orders, lineitem, part, partsupp, supplier, nation and region tables. These tables will be used during your lab. You will also edit the PowerShell script and add your server and database names. This will be used during exercises 162 | 163 | 1. Open a PowerShell window in your virtual machine.
164 | 2. Change directory to **Query Performance Tuning** in lab content folder using given command:
165 | 166 | `` 167 | cd "C:\LabContent\Analytics-Airlift-master\Day 1\07.SQLDW - Query tuning lab\Prep" 168 | `` 169 |
170 | 3. Run **PrepLab.ps1** script with you Azure Data Warehouse details. This will take 10-15 minutes.
171 |
172 | 4. Run the following command for changing the PowerShell execution policies for Windows computers.
173 | 174 | `` 175 | Set-ExecutionPolicy -Scope Process -ExecutionPolicy ByPass 176 | `` 177 |
178 | 179 | 5. Open your **Query Performance Tuning Lab** content folder.
180 | 6. Change directory to **C:\LabContent\Analytics-Airlift-master\Day 1\07.SQLDW - Query tuning lab\Lab>** sub folder.
181 | 7. Edit “RunExercise.ps1” script.
182 | 8. Replace **
183 |
184 | 9. Replace **
185 |
186 | 187 | ### Part 7 – Queries running slowly 188 | Your user comes to you and says “My query is running slow. I’m not sure why, because I’m not selecting very many rows. Can you help me figure out why? 189 | 190 | 1. Open a PowerShell window.
191 | 2. Change directory to **Query Performance Tuning lab** content folder.
192 | 3. Change directory to **Lab** sub folder.
193 | 4. Run **RunExercise.ps1** script with following parameters. This will execute a query on your server and show the result
194 | 195 | `` 196 | .\RunExercise.ps1 -Name Exercise1 -Type Slow 197 | `` 198 |
199 | 200 | 5. Open Query editor of SQL Data Warehouse in Azure Portal.
201 |
202 | 203 | 6. Check the query execution details with using DMVs.
204 | ``` 205 | SELECT * FROM sys.dm_pdw_exec_requests 206 | WHERE [Label] like 'Exercise1 | Slow%' 207 | ORDER BY submit_time DESC 208 | ``` 209 | 7. You can use the labels to search for your specific query. Powershell window shows the “Label” that was used during query execution.
210 |
211 | 8. Look for most recent execution of Exercise 1 query (“Running” or “Completed”)
212 | 9. Once you’ve identified the problematic query ID for this scenario, take a deeper look into it by using dm_pdw_request_steps:
213 | 214 | ``` 215 | SELECT * FROM sys.dm_pdw_request_steps 216 | WHERE request_id = 'QID####' 217 | ORDER BY step_index 218 | ``` 219 | 220 | 10. After running these queries, come up with a hypothesis about why the operation may be taking a long time. What are the longest running operations? What might they tell you about how the workflow is structured?
221 |
222 | ``` 223 | Hint: Moving data will typically be one of the most expensive operations within SQL DW. How is this query moving data and how could it move it more effectively? See the Appendix for more info about data movement types.
224 | Hint: Why is this query moving data? What is the source of the moved data? 225 | Hint: If you’re still stuck, look at your tables in the object explorer (or sys.tables) – what’s different about the table the user is querying? Can you find it where you expect in the object explorer tree? Why not? What type of table is this table?
226 | ``` 227 | 11. After you have a firm understanding of this slow scenario, run the same query with Fast optionThis will execute a query on your server and show the result.
228 | `` 229 | .\RunExercise.ps1 -Name Exercise1 -Type Fast 230 | `` 231 |
232 | 12. Run the following querys and check the results
233 | ``` 234 | SELECT * FROM sys.dm_pdw_exec_requests 235 | WHERE [Label] like 'Exercise1 | Fast%' 236 | ORDER BY submit_time DESC 237 | 238 | SELECT * FROM sys.dm_pdw_request_steps 239 | WHERE request_id = 'QID####' 240 | ORDER BY step_index 241 | ``` 242 | 13. Compare the results.
243 | 244 |
245 |
246 | 247 | ### Discussion 248 | After steps 1-5, you should have taken a look at the query that was being executed and think about what kind of plan you may expect to see.
249 | At step 10, you should have been able to find your query in exec_requests using the label and determine: Request_ID, total elapsed time, resource class, and first 4,000 characters of the command text. Looking at the command you can see that you are running a select count_big(*) against the 'dbo.lineitem_0' table. You may want to look in object explorer or sys.tables at this point to see a few general things: what type of table is it? If it's distributed, what is the distribution key/column? Are there multi column statistics? Have they been updated? If it's a view, what are the underlying tables? These are all general things you will want to know for most query performance issues.
250 | At step 11, you want to pick out the long-running step based on total_elapsed_time, which in this case was a HadoopRoundRobinOperation. If you look up this movement, it is querying data in an external table and storing the results in PDW in a round robin table. Also notice that the row count is large at about 60 million rows. This is because when copying data from an external table we copy a full version of the table into SQLDW - the predicate is not used to trim this data down yet.
251 | At step 14, you should be able to use exec_requests in the same way you did before to get the query text and see that we are now querying table 'dbo.lineitem_3'. You should look at sys.tables or object explorer in SSMS/SSDT to see the differences between this table and the previous one. You should notice that lineitem_3 is a distributed table whereas Lineitem_2 was an external table. The new plan in request_steps no longer has the Hadoop shuffle because the data is already in SQLDW.
252 | This should illustrate that when you are going to be using external tables repeatedly, it is much more efficient if you first import the table(s) into SQLDW then run the queries on the local tables. Otherwise, every query that touches the external table will have to import the table into tempdb as part of the plan before being able to execute the query. 253 | 254 | ## Part 8 – Joins 255 | Now that you’ve got the hang of things, let’s try the same process on the next exercise. 256 | Again, your user comes to you with questions, complaining that they are joining two of their most important tables together, and SQL DW just isn’t performing as well as they had expected. 257 | 258 | 1. Open a PowerShell window.
259 | 2. Change directory to Query Performance Tuning lab content folder.
260 | 3. Change directory to Lab sub folder.
261 | 4. Run “RunExercise.ps1” script with following parameters
262 | `` 263 | .\RunExercise.ps1 -Name Exercise2 -Type Slow 264 | `` 265 |
266 | 5. Open Query editor of SQL Data Warehouse in Azure Portal.
267 | 6. Check the query execution details with using DMVs.
268 |
269 | 7. You can use the labels to search for your specific query. Powershell window shows the “Label” that was used during query execution.
270 | 8. Look for most recent execution of Exercise 2 query (“Running” or “Completed”)
271 |
272 | 9. Once you’ve identified the problematic query ID for this scenario, take a deeper look into it by using dm_pdw_request_steps:
273 | Some steps of the DSQL plan are mostly overhead and can generally be ignored for purposes of optimizing the plan.
274 | These steps include the RandomIDOperation and the creation of the temporary tables for DMS.
275 | It can often help to add additional predicates to the above query to remove some of the overhead steps thus allowing you to focus on the heavy lifting operations. AND operation_type NOT IN ('RandomIDOperation') AND command NOT LIKE 'CREATE %' AND command NOT LIKE 'DROP %'
276 |
277 | 10. Check the steps and determine which one(s) might be the problematic steps.
278 |
279 | 11. Run the same query with Fast option.
280 | 281 | `` 282 | .\RunExercise.ps1 -Name Exercise2 -Type Fast” 283 | `` 284 |
285 | 12. Compare the 2 query execution plans and determine what would be the reason for query slowness.
286 |
287 |
288 | `` 289 | Hint: Look at the tables that are being joined with the query. Take a look at the table distribution types in the SSMS object explorer. The icon for each table tells you if the table is hash distributed or round robin distributed? What occurs when two round robin tables are joined? 290 | `` 291 | 292 | ### Discussion: 293 | You should have followed the same workflow as you did in exercise 1 up until step 9. In Step 9 we introduce some helpful additions to the query to filter out some of the SQL OnOperations that are not helpful for troubleshooting. This makes request_steps easier to read.
294 | At step 10, you can see that we are performing 5 broadcast moves and 1 shuffle move. Most of the time was spent in the shuffle move, but the large rowcount on the first broadcast is a point of interest, because remember we generally do not want to be broadcasting large tables.
295 | At step 12, you are comparing the fast plan to the slow plan. You can see in the fast plan that we now have 4 broadcast moves (instead of 5) and 1 shuffle. The table that is no longer being broadcasted is that large table we noticed in step 10. We can get the tables being queries from exec_requests, then look at sys.tables or object explorer to see what kind of tables they are. You will see that the fast version has all hash distributed tables, while the slow version has round robin tables.
296 | In general, you want large fact tables to be distributed tables. In this query both the orders and lineitem tables are large fact tables. If we are joining them together then it is best if they are distribution-compatible, which means distributed on the same key. This way each distribution has just a small slice of data to work with. In the fast version, both of these tables are distributed on orderkey. Round robin tables are never distribution-compatible, so the slow plan has to perform some sort of movement, like a broadcast, to make them distribution compatible before performing the join. The fast version shuffle will be faster because of the smaller input data volume. 297 | 298 | ## Part 9 – Troubleshooting Nuke 299 | Again, your user comes to you with questions, saying “I’ve just loaded my data and my queries are running slow than on SQL Server! What am I missing here?” 300 | 301 | 1. Open a PowerShell window.
302 | 2. Change directory to Query Performance Tuning lab content folder.
303 | 3. Change directory to Lab sub folder.
304 | 4. Run “RunExercise.ps1” script with following parameters
305 | `` 306 | .\RunExercise.ps1 -Name Exercise3 -Type Slow 307 | `` 308 |
309 | 5. Open Query editor of SQL Data Warehouse in Azure Portal.
310 | 6. Check the query execution details with using DMVs.
311 |
312 | 7. You can use the labels to search for your specific query. Powershell window shows the “Label” that was used during query execution.
313 | 8. Look for most recent execution of Exercise 3 query (“Running” or “Completed”)
314 |
315 | 9. Once you’ve identified the problematic query ID for this scenario, take a deeper look into it by using dm_pdw_request_steps:
316 |
317 | 10. Check the steps and determine which one(s) might be the problematic steps.
318 |
319 | 11. Run the same query with Fast option.
320 | `` 321 | .\RunExercise.ps1 -Name Exercise3 -Type Fast 322 | `` 323 |
324 | 12. Compare the 2 query execution plans and determine what would be the reason for query slowness.
325 |
326 |
327 | ``` 328 | Hint: Look at our best practices (in order) to narrow down what issues cause queries to run slowly.
329 | Hint: The “orders” table is one of the two largest tables and generally too big for a broadcast move. Why did the optimizer choose to create a copy of these rows on all nodes?
330 | Hint: If you’re still having trouble, look at the orders table in the object explorer tree to see what statistics are on the tables in this query. You can see the SQL that is running in the DMV sys.dm_pdw_exec_requests.
331 | ``` 332 | ### Discussion 333 | You should be able to reach step 10 using the same method you did in the first 2 exercises. In step 10, you should look through the steps for the longest-running step. In this case it's the BroadcastMove that took over 30 seconds. Next you can see that the rowcount for this broadcast is 60 million rows. This is a red flag because broadcasting such a large table will be very expensive. You should have used the original query or the exec_requests DMV to get the command and see what tables you are querying.
334 | In step 12, you can see that this large broadcast is no longer there. If you compare the table types in the slow version to the tables in the fast version you will see that they are the same. However in the fast version you can see in object explorer that there are statistics on the distribution and join columns.
335 | Further, if you run the query provided to get details about the table, you will see that for lineitem_1, the CTL_row_count is 1,000, but the cmp_row_count is ~60 million. 1,000 is the default value for statistics on the control node, so this means that statistics were never manually created. The distributed plan was created assuming it could broadcast this table because there were only 1,000 rows, but in reality there were 60 million rows, which caused our long-running step.
336 | This illustrates how the absence of statistics or statistics not being up to date can affect the distributed plan.
337 | 338 | ## Part 10 – Query performance improvements 339 | Now that your user has got all of their data loaded and organized they are trying out some of their more complex queries. Check this exercise to see if there are any improvements they can make to decrease their query runtime. 340 | 341 | 1. Open a PowerShell window.
342 | 2. Change directory to Query Performance Tuning lab content folder.
343 | 3. Change directory to Lab sub folder.
344 | 4. Run “RunExercise.ps1” script with following parameters
345 | `` 346 | .\RunExercise.ps1 -Name Exercise4 -Type Slow 347 | `` 348 |
349 | 5. Open Query editor of SQL Data Warehouse in Azure Portal.
350 | 6. Check the query execution details with using DMVs.
351 |
352 | 7. You can use the labels to search for your specific query. Powershell window shows the “Label” that was used during query execution
353 | 8. Look for most recent execution of Exercise 4 query (“Running” or “Completed”)
354 |
355 | 9. Check the steps and determine which one(s) might be the problematic steps.
356 |
357 | 10. Run the same query with Fast option.
358 | `` 359 | .\RunExercise.ps1 -Name Exercise4 -Type Fast 360 | `` 361 |
362 | 11. Compare the 2 query execution plans and determine what would be the reason for query slowness.
363 |
364 |
365 | ``` 366 | Hint: In this example, the query plan is optimal. This query could benefit if it was given more memory. How much memory has been allocated to this query? How can you use sys.dm_pdw_exec_requests to determine the memory grant? How can you change the memory allocation for a query? 367 | ``` 368 | 369 | ## Discussion 370 | In this exercise you will follow the same method to get to step 9. This time we see that the plan is a single step plan - a return operation. From a distributed plan perspective this is ideal because no data movement occurred. We simply were able to run the distributed queries on each distribution and return the results.
371 | For a query like this we cannot improve the MPP plan, so the next option to look at is resource class. You should have noticed that exec_requests shows that the query was running in smallRC. Certain queries will benefit from the larger memory allocation of larger resource classes and this usually requires testing to find the ideal balance of resource class usage for a query vs concurrency.
372 | Once you are at step 11, you should have looked at exec_requests and noticed that it was now running in LargeRC and the execution time was faster. These test queries are pretty fast running because the execution time is low, but for larger queries this can make a big difference. The default resource class is small. Remember, as you increase resource class you also decrease concurrency, so testing is required.
373 | If you want to change the resource class for a query you would use sp_addrolemember and sp_droprolemember 374 | 375 | ### Part 11: Query analysis 376 | Now that you’ve helped your user with some of their initial issues, they’re beginning to bring some of their analysts onto the system – but some analysts are complaining that their queries are taking very long to run or don’t seem to be running at all. 377 | 378 | 1. Open a PowerShell window.
379 | 2. Change directory to Query Performance Tuning lab content folder.
380 | 3. Change directory to Lab sub folder.
381 | 4. Run “RunExercise.ps1” script with following parameters
382 | `` 383 | “.\RunExercise.ps1 -Name Exercise5 -Type Slow” 384 | `` 385 |
386 | 5. This script will create a workload simulation on your server. It will create 20 background jobs which will send queries to your system.
387 | 6. It will wait for 60 seconds for all background jobs properly starts and then will start your problematic query.
388 | 7. Open Query editor of SQL Data Warehouse in Azure Portal.
389 | 8. Check all the active queries.
390 | ``` 391 | SELECT * FROM sys.dm_pdw_exec_requests 392 | WHERE status not in ('completed', 'failed', 'cancelled') AND session_id <> session_id() 393 | ORDER BY request_id DESC; 394 | 395 | ``` 396 | 9. Can you tell what is happening?
397 | `` 398 | Hint: What is the state of this query in sys.dm_pdw_exec_requests? Why? 399 | Hint: Run this query. What does it tell you about the state of the query? 400 | Hint: What can be changed to ensure these small queries run? After you investigate connect to ‘demo5_fast’ to see the changes in action. 401 | `` 402 | 10. You need to kill the background jobs before continuing.
403 | 11. Cancel the running process on current PowerShell window.
404 | 12. Run “.\Kill.ps1”
405 |
406 | 13. Make sure you close the PowerShell window.
407 | 14. Open a PowerShell window.
408 | 15. Change directory to Query Performance Tuning lab content folder.
409 | 16. Change directory to Lab sub folder.
410 | 17. Run “RunExercise.ps1” script with following parameters Run the same query with Fast option.
411 | `` 412 | “.\RunExercise.ps1 -Name Exercise5 -Type Fast” 413 | `` 414 |
415 | 18. Compare the 2 query execution plans.
416 | 19. You need to kill the background jobs before continuing.
417 | 20. Cancel the running process on current PowerShell window.
418 | 21. Run “.\Kill.ps1” 419 |
420 | 22. Make sure you close the PowerShell window. 421 | 422 | ## Discussion 423 | This exercise tries to simulate an active workload on your data warehouse. It creates 20 background sessions which sends queries constantly.
424 | When you reached to step 9 on Slow execution exercise, you will notice that your query is in the queue (“Suspended”) and does not “Running”. You can check wait stats and see that what is your query is waiting on “UserConcurrencyResourceType” which means that it is waiting for enough concurrency slots become available.
425 | When you check dm_pdw_exec_requests you will notice that this query is running on largerc resource class. In previous example we talk about using higher resource classes allow your query to have more memory resources. But this will result in more memory consumption from the overall system and resulted in less concurrent query executions. So you need to be careful about which resource classes you are using for executing your queries. Always test your queries with your actual workload.
426 | On faster version of this exercise you will notice that your queries might again queued but once there is enough concurrency slots available it will go through the execution. You will see that your query runs 3 times but at every execution it waits on the queue. You can check the queue waiting time by comparing start_time and submit_time in dm_pdw_exec_requests DMV. 427 | 428 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Analytics-Airlift 2 | 3 | ## Agenda 4 | Please note: This is a rough agenda, and the schedule is subject to change pending class activities, breaks, and interactions. 5 | 6 | - ADF - Data ingestion lab 7 | - [Lab 1.1: LAB-Ingesting Data into a Blob with ADF](https://github.com/SpektraSystems/Analytics-Airlift/blob/master/LAB01-Ingesting%20Data%20into%20a%20Blob%20with%20ADF.md) 8 | - [Lab 1.2: LAB-Ingesting_Data_into_a_ADLSGen2_with_ADF](https://github.com/SpektraSystems/Analytics-Airlift/blob/master/LAB02-Ingesting_Data_into_a_ADLSGen2_with_ADF.md) 9 | - SQLDW - Loading lab 10 | - [Lab 1.3: Data loading best practices Final](https://github.com/SpektraSystems/Analytics-Airlift/blob/master/Lab03-Data%20loading%20best%20practices.md) 11 | - SQLDW - Query tuning lab 12 | - [Lab 1.4: SQLDW - Query Performance Tuning](https://github.com/SpektraSystems/Analytics-Airlift/blob/master/Lab04%20-%20SQLDW%20Query%20Performance%20Tuning.md) 13 | - Databricks - ETL 14 | - [Lab 1.5: ETL with Azure Databricks – Setup Instructions.md](https://github.com/SpektraSystems/Analytics-Airlift/blob/master/Lab05%20-%20ETL%20with%20Azure%20Databricks%20%E2%80%93%20Setup%20Instructions.md) 15 | - DataFlow - ETL 16 | - [Lab 1.6: ETL with Dataflow – Setup Instructions.md](https://github.com/SpektraSystems/Analytics-Airlift/blob/master/Lab06-ETL%20with%20Azure%20Data%20Factory%E2%80%93%20Setup%20Instructions.md) 17 | -------------------------------------------------------------------------------- /images/Partition.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/Partition.jpg -------------------------------------------------------------------------------- /images/adf1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf1.jpg -------------------------------------------------------------------------------- /images/adf10.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf10.jpg -------------------------------------------------------------------------------- /images/adf11.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf11.jpg -------------------------------------------------------------------------------- /images/adf12.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf12.jpg -------------------------------------------------------------------------------- /images/adf13.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf13.jpg -------------------------------------------------------------------------------- /images/adf14.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf14.jpg -------------------------------------------------------------------------------- /images/adf15.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf15.jpg -------------------------------------------------------------------------------- /images/adf16.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf16.jpg -------------------------------------------------------------------------------- /images/adf17.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf17.jpg -------------------------------------------------------------------------------- /images/adf18.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf18.jpg -------------------------------------------------------------------------------- /images/adf19.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf19.jpg -------------------------------------------------------------------------------- /images/adf2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf2.jpg -------------------------------------------------------------------------------- /images/adf20.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf20.jpg -------------------------------------------------------------------------------- /images/adf21.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf21.jpg -------------------------------------------------------------------------------- /images/adf22.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf22.jpg -------------------------------------------------------------------------------- /images/adf23.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf23.jpg -------------------------------------------------------------------------------- /images/adf24.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf24.jpg -------------------------------------------------------------------------------- /images/adf25.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf25.jpg -------------------------------------------------------------------------------- /images/adf26.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf26.jpg -------------------------------------------------------------------------------- /images/adf27.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf27.jpg -------------------------------------------------------------------------------- /images/adf28.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf28.jpg -------------------------------------------------------------------------------- /images/adf3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf3.jpg -------------------------------------------------------------------------------- /images/adf4.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf4.jpg -------------------------------------------------------------------------------- /images/adf5.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf5.jpg -------------------------------------------------------------------------------- /images/adf6.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf6.jpg -------------------------------------------------------------------------------- /images/adf7.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf7.jpg -------------------------------------------------------------------------------- /images/adf8.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf8.jpg -------------------------------------------------------------------------------- /images/adf9.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adf9.jpg -------------------------------------------------------------------------------- /images/adls.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adls.jpg -------------------------------------------------------------------------------- /images/adls1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adls1.jpg -------------------------------------------------------------------------------- /images/adls2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adls2.jpg -------------------------------------------------------------------------------- /images/adls3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adls3.jpg -------------------------------------------------------------------------------- /images/adls4.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adls4.jpg -------------------------------------------------------------------------------- /images/adls5.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adls5.jpg -------------------------------------------------------------------------------- /images/adls6.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/adls6.jpg -------------------------------------------------------------------------------- /images/connect.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/connect.png -------------------------------------------------------------------------------- /images/create.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/create.jpg -------------------------------------------------------------------------------- /images/createmasterkey.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/createmasterkey.png -------------------------------------------------------------------------------- /images/debug.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/debug.jpg -------------------------------------------------------------------------------- /images/dimensions.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/dimensions.jpg -------------------------------------------------------------------------------- /images/dsd: -------------------------------------------------------------------------------- 1 | sdd 2 | -------------------------------------------------------------------------------- /images/etl01.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl01.jpg -------------------------------------------------------------------------------- /images/etl02.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl02.jpg -------------------------------------------------------------------------------- /images/etl03.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl03.jpg -------------------------------------------------------------------------------- /images/etl1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl1.jpg -------------------------------------------------------------------------------- /images/etl10.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl10.jpg -------------------------------------------------------------------------------- /images/etl11.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl11.jpg -------------------------------------------------------------------------------- /images/etl12.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl12.jpg -------------------------------------------------------------------------------- /images/etl13.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl13.jpg -------------------------------------------------------------------------------- /images/etl14.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl14.jpg -------------------------------------------------------------------------------- /images/etl15.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl15.jpg -------------------------------------------------------------------------------- /images/etl16.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl16.jpg -------------------------------------------------------------------------------- /images/etl17.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl17.jpg -------------------------------------------------------------------------------- /images/etl18.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl18.jpg -------------------------------------------------------------------------------- /images/etl19.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl19.jpg -------------------------------------------------------------------------------- /images/etl2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl2.jpg -------------------------------------------------------------------------------- /images/etl20.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl20.jpg -------------------------------------------------------------------------------- /images/etl21.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl21.jpg -------------------------------------------------------------------------------- /images/etl22.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl22.jpg -------------------------------------------------------------------------------- /images/etl23.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl23.jpg -------------------------------------------------------------------------------- /images/etl24.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl24.jpg -------------------------------------------------------------------------------- /images/etl26.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl26.jpg -------------------------------------------------------------------------------- /images/etl27.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl27.jpg -------------------------------------------------------------------------------- /images/etl28.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl28.jpg -------------------------------------------------------------------------------- /images/etl29.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl29.jpg -------------------------------------------------------------------------------- /images/etl3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl3.jpg -------------------------------------------------------------------------------- /images/etl30.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl30.jpg -------------------------------------------------------------------------------- /images/etl31.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl31.jpg -------------------------------------------------------------------------------- /images/etl32.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl32.jpg -------------------------------------------------------------------------------- /images/etl33.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl33.jpg -------------------------------------------------------------------------------- /images/etl34.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl34.jpg -------------------------------------------------------------------------------- /images/etl35.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl35.jpg -------------------------------------------------------------------------------- /images/etl36.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl36.jpg -------------------------------------------------------------------------------- /images/etl37.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl37.jpg -------------------------------------------------------------------------------- /images/etl38.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl38.jpg -------------------------------------------------------------------------------- /images/etl39.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl39.jpg -------------------------------------------------------------------------------- /images/etl4.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl4.jpg -------------------------------------------------------------------------------- /images/etl5.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl5.jpg -------------------------------------------------------------------------------- /images/etl6.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl6.jpg -------------------------------------------------------------------------------- /images/etl7.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl7.jpg -------------------------------------------------------------------------------- /images/etl8.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl8.jpg -------------------------------------------------------------------------------- /images/etl9.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/etl9.jpg -------------------------------------------------------------------------------- /images/ex01.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex01.jpg -------------------------------------------------------------------------------- /images/ex02.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex02.jpg -------------------------------------------------------------------------------- /images/ex03.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex03.jpg -------------------------------------------------------------------------------- /images/ex04.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex04.jpg -------------------------------------------------------------------------------- /images/ex05.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex05.jpg -------------------------------------------------------------------------------- /images/ex07.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex07.jpg -------------------------------------------------------------------------------- /images/ex08.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex08.jpg -------------------------------------------------------------------------------- /images/ex09.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex09.jpg -------------------------------------------------------------------------------- /images/ex10.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex10.jpg -------------------------------------------------------------------------------- /images/ex11.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex11.jpg -------------------------------------------------------------------------------- /images/ex12.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex12.jpg -------------------------------------------------------------------------------- /images/ex13.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex13.jpg -------------------------------------------------------------------------------- /images/ex14.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex14.jpg -------------------------------------------------------------------------------- /images/ex15.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex15.jpg -------------------------------------------------------------------------------- /images/ex16.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex16.jpg -------------------------------------------------------------------------------- /images/ex17.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex17.jpg -------------------------------------------------------------------------------- /images/ex18.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex18.jpg -------------------------------------------------------------------------------- /images/ex19.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex19.jpg -------------------------------------------------------------------------------- /images/ex20.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex20.jpg -------------------------------------------------------------------------------- /images/ex21.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex21.jpg -------------------------------------------------------------------------------- /images/ex22.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex22.jpg -------------------------------------------------------------------------------- /images/ex23.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex23.jpg -------------------------------------------------------------------------------- /images/ex24.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex24.jpg -------------------------------------------------------------------------------- /images/ex25.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex25.jpg -------------------------------------------------------------------------------- /images/ex26.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex26.jpg -------------------------------------------------------------------------------- /images/ex27.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex27.jpg -------------------------------------------------------------------------------- /images/ex28.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex28.jpg -------------------------------------------------------------------------------- /images/ex29.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex29.jpg -------------------------------------------------------------------------------- /images/ex3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex3.jpg -------------------------------------------------------------------------------- /images/ex30.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ex30.jpg -------------------------------------------------------------------------------- /images/gen1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/gen1.jpg -------------------------------------------------------------------------------- /images/gen2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/gen2.jpg -------------------------------------------------------------------------------- /images/gen3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/gen3.jpg -------------------------------------------------------------------------------- /images/initial.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/initial.jpg -------------------------------------------------------------------------------- /images/kill.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/kill.jpg -------------------------------------------------------------------------------- /images/kill1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/kill1.jpg -------------------------------------------------------------------------------- /images/ld.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ld.jpg -------------------------------------------------------------------------------- /images/ld0.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ld0.jpg -------------------------------------------------------------------------------- /images/ld1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ld1.jpg -------------------------------------------------------------------------------- /images/ld10.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ld10.jpg -------------------------------------------------------------------------------- /images/ld11.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ld11.jpg -------------------------------------------------------------------------------- /images/ld12.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ld12.jpg -------------------------------------------------------------------------------- /images/ld13.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ld13.jpg -------------------------------------------------------------------------------- /images/ld14.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ld14.jpg -------------------------------------------------------------------------------- /images/ld2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ld2.jpg -------------------------------------------------------------------------------- /images/ld3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ld3.jpg -------------------------------------------------------------------------------- /images/ld4.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ld4.jpg -------------------------------------------------------------------------------- /images/ld5.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ld5.jpg -------------------------------------------------------------------------------- /images/ld6.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ld6.jpg -------------------------------------------------------------------------------- /images/ld7.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ld7.jpg -------------------------------------------------------------------------------- /images/ld8.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ld8.jpg -------------------------------------------------------------------------------- /images/ld9.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/ld9.jpg -------------------------------------------------------------------------------- /images/load.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/load.jpg -------------------------------------------------------------------------------- /images/monitor.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/monitor.jpg -------------------------------------------------------------------------------- /images/monitor1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/monitor1.jpg -------------------------------------------------------------------------------- /images/newquerey.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/newquerey.png -------------------------------------------------------------------------------- /images/openinvisualstudio.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/openinvisualstudio.png -------------------------------------------------------------------------------- /images/output.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/output.jpg -------------------------------------------------------------------------------- /images/password.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/password.jpg -------------------------------------------------------------------------------- /images/product.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/product.jpg -------------------------------------------------------------------------------- /images/psscript.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/psscript.png -------------------------------------------------------------------------------- /images/query1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/query1.png -------------------------------------------------------------------------------- /images/query2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/query2.png -------------------------------------------------------------------------------- /images/query3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/query3.png -------------------------------------------------------------------------------- /images/query4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/query4.png -------------------------------------------------------------------------------- /images/query5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/query5.png -------------------------------------------------------------------------------- /images/query6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/query6.png -------------------------------------------------------------------------------- /images/resume.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/resume.jpg -------------------------------------------------------------------------------- /images/resumedw.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/resumedw.jpg -------------------------------------------------------------------------------- /images/rg.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/rg.jpg -------------------------------------------------------------------------------- /images/rg1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/rg1.jpg -------------------------------------------------------------------------------- /images/set1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/set1.jpg -------------------------------------------------------------------------------- /images/sink.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sink.jpg -------------------------------------------------------------------------------- /images/sql01.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql01.jpg -------------------------------------------------------------------------------- /images/sql02.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql02.jpg -------------------------------------------------------------------------------- /images/sql03.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql03.jpg -------------------------------------------------------------------------------- /images/sql04.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql04.jpg -------------------------------------------------------------------------------- /images/sql05.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql05.jpg -------------------------------------------------------------------------------- /images/sql06.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql06.jpg -------------------------------------------------------------------------------- /images/sql1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql1.jpg -------------------------------------------------------------------------------- /images/sql10.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql10.jpg -------------------------------------------------------------------------------- /images/sql11.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql11.jpg -------------------------------------------------------------------------------- /images/sql12.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql12.jpg -------------------------------------------------------------------------------- /images/sql13.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql13.jpg -------------------------------------------------------------------------------- /images/sql14.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql14.jpg -------------------------------------------------------------------------------- /images/sql15.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql15.jpg -------------------------------------------------------------------------------- /images/sql16.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql16.jpg -------------------------------------------------------------------------------- /images/sql17.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql17.jpg -------------------------------------------------------------------------------- /images/sql18.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql18.jpg -------------------------------------------------------------------------------- /images/sql19.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql19.jpg -------------------------------------------------------------------------------- /images/sql2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql2.jpg -------------------------------------------------------------------------------- /images/sql21.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql21.jpg -------------------------------------------------------------------------------- /images/sql22.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql22.jpg -------------------------------------------------------------------------------- /images/sql23.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql23.jpg -------------------------------------------------------------------------------- /images/sql24.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql24.jpg -------------------------------------------------------------------------------- /images/sql25.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql25.jpg -------------------------------------------------------------------------------- /images/sql26.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql26.jpg -------------------------------------------------------------------------------- /images/sql27.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql27.jpg -------------------------------------------------------------------------------- /images/sql28.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql28.jpg -------------------------------------------------------------------------------- /images/sql29.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql29.jpg -------------------------------------------------------------------------------- /images/sql3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql3.jpg -------------------------------------------------------------------------------- /images/sql30.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql30.jpg -------------------------------------------------------------------------------- /images/sql31.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql31.jpg -------------------------------------------------------------------------------- /images/sql32.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql32.jpg -------------------------------------------------------------------------------- /images/sql33.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql33.jpg -------------------------------------------------------------------------------- /images/sql34.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql34.jpg -------------------------------------------------------------------------------- /images/sql35.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql35.jpg -------------------------------------------------------------------------------- /images/sql4.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql4.jpg -------------------------------------------------------------------------------- /images/sql5.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql5.jpg -------------------------------------------------------------------------------- /images/sql6.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql6.jpg -------------------------------------------------------------------------------- /images/sql7.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql7.jpg -------------------------------------------------------------------------------- /images/sql8.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql8.jpg -------------------------------------------------------------------------------- /images/sql9.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sql9.jpg -------------------------------------------------------------------------------- /images/sqldwimage1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/sqldwimage1.png -------------------------------------------------------------------------------- /images/start.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/start.jpg -------------------------------------------------------------------------------- /images/startvm.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/startvm.jpg -------------------------------------------------------------------------------- /images/store1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/store1.jpg -------------------------------------------------------------------------------- /images/store2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/store2.jpg -------------------------------------------------------------------------------- /images/tilogin.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/tilogin.png -------------------------------------------------------------------------------- /images/visualstudio.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpektraSystems/Analytics-Airlift/f5494c6cb39b7291ebd91dea4146ff4fed9ed7a3/images/visualstudio.png --------------------------------------------------------------------------------