├── 2018 └── power-bi-cumulative-total-nuances │ └── Cumulative Totals.pbix ├── 2020 ├── a-user-friendly-way-to-deal-with-role-playing-dimensions-in-power-bi │ └── Power BI - Role Playing Dimensions.pbix ├── automating-power-bi-deployments │ ├── General - Connect.ps1 │ ├── General - PoSh Script Template.ps1 │ ├── Reports - Change Data Source Credentials.ps1 │ ├── Reports - Deploy Report.ps1 │ ├── Reports - Trigger Refresh.ps1 │ ├── Reports - Update Parameter.ps1 │ ├── Reports - Update Refresh Schedule.ps1 │ ├── Workspaces - Assign Permissions.ps1 │ └── Workspaces - Create Workspace.ps1 ├── change-password-expiration-policies-for-service-accounts-in-o365 │ └── Users - Change Password Expiration Policy.ps1 └── restore-a-power-bi-workspace │ ├── Workspaces - List Workspaces.ps1 │ └── Workspaces - Restore Workspace.ps1 ├── 2021 ├── azure-ad-assign-administrator-roles-with-powershell │ └── Roles - Assign Administrator Role.ps1 ├── fixing-incorrect-drill-through-results-in-analysis-services │ ├── Fixing Drillthrough Results.sln │ ├── Tabular Project - Part 1 │ │ ├── Model.bim │ │ ├── Model.bim.layout │ │ ├── Model.bim_MartinSchoombee.settings │ │ └── Tabular Project - Part 1.smproj │ └── Tabular Project - Part 2 │ │ ├── Model.bim │ │ ├── Model.bim.layout │ │ ├── Model.bim_MartinSchoombee.settings │ │ └── Tabular Project - Part 2.smproj └── using-calculation-groups-with-role-playing-dimensions │ ├── Tabular Project │ ├── Model.bim │ ├── Model.bim.layout │ ├── Model.bim_MartinSchoombee.settings │ └── Tabular Project.smproj │ └── Using Calculation Groups with Role Playing Dimensions.sln ├── 2022 └── creating-development-environments-in-azure │ ├── ARM - Deploy VM.ps1 │ └── ARM Template - VM.json ├── 2024 └── building-a-framework-for-orchestration-in-azure-data-factory │ ├── Data Factory │ ├── 1 - Linked Services │ │ ├── Linked Service Template - Azure Key Vault.jsonc │ │ ├── Linked Service Template - Azure SQL Database.jsonc │ │ └── Linked Service Template - Source - Azure SQL Database.jsonc │ ├── 2 - Datasets │ │ ├── Dataset Template - Azure SQL Database.jsonc │ │ ├── Dataset Template - Azure SQL Table.jsonc │ │ └── Dataset Template - Source - Azure SQL Database.jsonc │ ├── 3 - Pipelines │ │ ├── Pipeline Template - Orchestrator - Execute Task.jsonc │ │ ├── Pipeline Template - Orchestrator - Get Tasks To Execute.jsonc │ │ ├── Pipeline Template - Orchestrator - Main.jsonc │ │ └── Pipeline Template - Worker - Copy Data - Azure SQL.jsonc │ ├── Readme - Orchestrators.txt │ └── Readme - Workers.txt │ └── Metadata │ ├── GetExecutionSequences.sql │ ├── GetTaskQueries.sql │ ├── GetTasksToExecute.sql │ └── Metadata Tables.sql └── readme /2018/power-bi-cumulative-total-nuances/Cumulative Totals.pbix: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mschoombee/Blog/d03e18fcedf00ebfcade330fb19ae4aa975211ed/2018/power-bi-cumulative-total-nuances/Cumulative Totals.pbix -------------------------------------------------------------------------------- /2020/a-user-friendly-way-to-deal-with-role-playing-dimensions-in-power-bi/Power BI - Role Playing Dimensions.pbix: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mschoombee/Blog/d03e18fcedf00ebfcade330fb19ae4aa975211ed/2020/a-user-friendly-way-to-deal-with-role-playing-dimensions-in-power-bi/Power BI - Role Playing Dimensions.pbix -------------------------------------------------------------------------------- /2020/automating-power-bi-deployments/General - Connect.ps1: -------------------------------------------------------------------------------- 1 | <# 2 | ** Check the execution policy for the current machine/user, and set it to unrestricted (if required) ** 3 | 4 | Get-ExecutionPolicy -List 5 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope LocalMachine #if you want to set it for the machine 6 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process #if you want to set it for the current process 7 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser #if you want to set it for current user only 8 | #> 9 | 10 | <# 11 | ** Install and import the following Power BI management cmdlets (if required) ** 12 | 13 | Install-Module -Name MicrosoftPowerBIMgmt -AllowClobber 14 | Import-Module MicrosoftPowerBIMgmt 15 | #> 16 | 17 | <# 18 | Script Notes 19 | ----------------------------------------------------------------------------------------- 20 | 21 | ** This script shows the different ways to connect to a Power BI tenant. It isn't meant to 22 | run as a unit, so use if for testing purposes only. 23 | 24 | 25 | Technical References 26 | ----------------------------------------------------------------------------------------- 27 | 28 | Power BI cmdlets 29 | https://docs.microsoft.com/en-us/powershell/power-bi/overview?view=powerbi-ps 30 | 31 | Power BI REST APIs 32 | https://docs.microsoft.com/en-us/rest/api/power-bi/ 33 | #> 34 | 35 | <#------------------------------------------------------------------------------------------- 36 | 37 | Interactive Connection - Running this cmdlet will prompt the user to connect 38 | 39 | -------------------------------------------------------------------------------------------#> 40 | 41 | Connect-PowerBIServiceAccount 42 | 43 | 44 | 45 | <#------------------------------------------------------------------------------------------- 46 | 47 | Automated Connection with Username & Password - Running the code below will connect without user prompts 48 | 49 | ** The account you're using will need Power BI administrator permissions 50 | ** The method below will not work if MFA is required for the account 51 | 52 | -------------------------------------------------------------------------------------------#> 53 | 54 | #Variables 55 | $PbiUser = "me@me.com" 56 | $PbiPassword = "MyPassword" 57 | 58 | #Create secure string and credential for username & password 59 | $PbiSecurePassword = ConvertTo-SecureString $PbiPassword -Force -AsPlainText 60 | $PbiCredential = New-Object Management.Automation.PSCredential($PbiUser, $PbiSecurePassword) 61 | 62 | #Connect to the Power BI service 63 | Connect-PowerBIServiceAccount -Credential $PbiCredential 64 | 65 | 66 | 67 | <#------------------------------------------------------------------------------------------- 68 | 69 | Automated Connection with Service Principal - Running the code below will connect without user prompts 70 | 71 | ** The Service Principal will need Power BI administrator permissions 72 | 73 | -------------------------------------------------------------------------------------------#> 74 | 75 | #Variables 76 | $AppId = "MyAppId" 77 | $TenantId = "MyServicePrincipalTenantId" 78 | $ClientSecret = "MyClientSecret" 79 | 80 | #Create secure string & credential for application id and client secret 81 | $PbiSecurePassword = ConvertTo-SecureString $ClientSecret -Force -AsPlainText 82 | $PbiCredential = New-Object Management.Automation.PSCredential($AppId, $PbiSecurePassword) 83 | 84 | #Connect to the Power BI service 85 | Connect-PowerBIServiceAccount -ServicePrincipal -TenantId $TenantId -Credential $PbiCredential 86 | 87 | -------------------------------------------------------------------------------- /2020/automating-power-bi-deployments/General - PoSh Script Template.ps1: -------------------------------------------------------------------------------- 1 | <# 2 | ** Check the execution policy for the current machine/user, and set it to unrestricted (if required) ** 3 | 4 | Get-ExecutionPolicy -List 5 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope LocalMachine #if you want to set it for the machine 6 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process #if you want to set it for the current process 7 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser #if you want to set it for current user only 8 | #> 9 | 10 | <# 11 | ** Install and import the following cmdlets (if required) ** 12 | 13 | Install-Module -Name -AllowClobber 14 | Import-Module 15 | #> 16 | 17 | <# 18 | Script Notes 19 | ----------------------------------------------------------------------------------------- 20 | 21 | ** This script will... 22 | 23 | ** Execute this script in the following way in a PowerShell session (run as admin) 24 | ex. &'c:\PowerShell\...\General - PoSh Script Template.ps1' 25 | 26 | 27 | Technical References 28 | ----------------------------------------------------------------------------------------- 29 | 30 | 31 | #> 32 | 33 | <#------------------------------------------------------------------------------------------- 34 | 35 | Global actions & variables 36 | 37 | -------------------------------------------------------------------------------------------#> 38 | 39 | $ErrorActionPreference = "Stop" 40 | Write-Host "**NOTE: THIS SCRIPT WILL STOP EXECUTING AFTER THE FIRST ERROR**"`r`n -ForegroundColor Yellow 41 | 42 | 43 | Try 44 | { 45 | 46 | <#------------------------------------------------------------------------------------------- 47 | 48 | Section 1 49 | 50 | -------------------------------------------------------------------------------------------#> 51 | 52 | 53 | 54 | } 55 | 56 | Catch 57 | { 58 | Write-Host ""`n 59 | Write-Host "An error has occurred!!" -ForegroundColor Red 60 | Write-Host "Error Line Number : $($_.InvocationInfo.ScriptLineNumber)" -ForegroundColor Red 61 | Write-Host "Error Command : $($_.InvocationInfo.Line.Trim())" -ForegroundColor Red 62 | Write-Host "Error Message : $($_.Exception.Message)"`n -ForegroundColor Red 63 | Write-Host "Terminating script execution..."`n 64 | } 65 | 66 | Exit 67 | 68 | 69 | -------------------------------------------------------------------------------- /2020/automating-power-bi-deployments/Reports - Change Data Source Credentials.ps1: -------------------------------------------------------------------------------- 1 | <# 2 | ** Check the execution policy for the current machine/user, and set it to unrestricted (if required) ** 3 | 4 | Get-ExecutionPolicy -List 5 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope LocalMachine #if you want to set it for the machine 6 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process #if you want to set it for the current process 7 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser #if you want to set it for current user only 8 | #> 9 | 10 | <# 11 | ** Install and import the following Power BI management cmdlets (if required) ** 12 | 13 | Install-Module -Name MicrosoftPowerBIMgmt -AllowClobber 14 | Import-Module MicrosoftPowerBIMgmt 15 | #> 16 | 17 | <# 18 | Script Notes 19 | ----------------------------------------------------------------------------------------- 20 | 21 | ** This script will update credentials (username & password) for all SQL Server data sources in a report. 22 | 23 | ** Execute this script in the following way in a PowerShell session (run as admin) 24 | ex. &'c:\PowerShell\Power BI\Reports - Change Data Source Credentials.ps1' -WorkspaceName 'MyWorkspace' -ReportName 'MyReportName' -DataSourceUser 'MyDataSourceUserName' -DataSourcePassword 'MyDataSourcePassword' 25 | 26 | 27 | Technical References 28 | ----------------------------------------------------------------------------------------- 29 | 30 | Power BI cmdlets 31 | https://docs.microsoft.com/en-us/powershell/power-bi/overview?view=powerbi-ps 32 | 33 | Power BI REST APIs 34 | https://docs.microsoft.com/en-us/rest/api/power-bi/ 35 | #> 36 | 37 | <#------------------------------------------------------------------------------------------- 38 | 39 | Parameters 40 | 41 | -------------------------------------------------------------------------------------------#> 42 | 43 | param 44 | ( 45 | [Parameter(Mandatory=$true)] 46 | [ValidateNotNullOrEmpty()] 47 | [String] 48 | $WorkspaceName, 49 | 50 | [Parameter(Mandatory=$true)] 51 | [ValidateNotNullOrEmpty()] 52 | [String] 53 | $ReportName, 54 | 55 | [Parameter(Mandatory=$true)] 56 | [ValidateNotNullOrEmpty()] 57 | [String] 58 | $DataSourceUser, 59 | 60 | [Parameter(Mandatory=$true)] 61 | [ValidateNotNullOrEmpty()] 62 | [String] 63 | $DataSourcePassword 64 | ) 65 | 66 | <#------------------------------------------------------------------------------------------- 67 | 68 | Global actions & variables 69 | 70 | -------------------------------------------------------------------------------------------#> 71 | 72 | $ErrorActionPreference = "Stop" 73 | Write-Host "**NOTE: THIS SCRIPT WILL STOP EXECUTING AFTER THE FIRST ERROR**"`r`n -ForegroundColor Yellow 74 | 75 | 76 | try { 77 | 78 | <#------------------------------------------------------------------------------------------- 79 | 80 | Connect to Power BI 81 | 82 | -------------------------------------------------------------------------------------------#> 83 | 84 | #Connect to Power BI tenant 85 | Connect-PowerBIServiceAccount | Out-Null 86 | 87 | 88 | <#------------------------------------------------------------------------------------------- 89 | 90 | Retrieve workspace, report and data source objects 91 | 92 | -------------------------------------------------------------------------------------------#> 93 | 94 | #Retrieve the workspace 95 | $WorkspaceObject = (Get-PowerBIWorkspace -Scope Organization -Name $WorkspaceName) 96 | 97 | #Retrieve the report 98 | $PbiReportObject = (Get-PowerBIReport -Workspace $WorkspaceObject -Name $ReportName) 99 | 100 | #Retrieve all data sources 101 | $PbiDataSourcesObject = (Get-PowerBIDatasource -DatasetId $PbiReportObject.DatasetId -Scope Organization) 102 | 103 | 104 | <#------------------------------------------------------------------------------------------- 105 | 106 | Iterate through data sources and update credentials for all SQL Server sources 107 | 108 | -------------------------------------------------------------------------------------------#> 109 | 110 | foreach ($DataSource in $PbiDataSourcesObject) { 111 | 112 | #Store the data source id in a variable (for ease of use later) 113 | $DataSourceId = $DataSource.DatasourceId 114 | 115 | #API url for data source 116 | $ApiUrl = "gateways/" + $DataSource.GatewayId + "/datasources/" + $DataSourceId 117 | 118 | #Format username and password, replacing escape characters for the body of the request 119 | $FormattedDataSourceUser = $DataSourceUser.Replace("\", "\\") 120 | $FormattedDataSourcePassword = $DataSourcePassword.Replace("\", "\\") 121 | 122 | #Build the request body 123 | $ApiRequestBody = @" 124 | { 125 | "credentialDetails": { 126 | "credentialType": "Basic", 127 | "credentials": "{\"credentialData\":[{\"name\":\"username\", \"value\":\"$FormattedDataSourceUser\"}, {\"name\":\"password\", \"value\":\"$FormattedDataSourcePassword\"}]}", 128 | "encryptedConnection": "Encrypted", 129 | "encryptionAlgorithm": "None", 130 | "privacyLevel": "Organizational" 131 | } 132 | } 133 | "@ 134 | 135 | 136 | #If it's a sql server source, change the username/password 137 | if ($DataSource.DatasourceType = "Sql") { 138 | 139 | #Update username & password 140 | Invoke-PowerBIRestMethod -Url $ApiUrl -Method Patch -Body ("$ApiRequestBody") 141 | 142 | Write-Output "Credentials for data source ""$DataSourceId"" successfully updated..." `n 143 | } 144 | 145 | } 146 | } 147 | 148 | catch { 149 | Write-Host ""`n 150 | Write-Host "An error has occurred!!" -ForegroundColor Red 151 | Write-Host "Error Line Number : $($_.InvocationInfo.ScriptLineNumber)" -ForegroundColor Red 152 | Write-Host "Error Command : $($_.InvocationInfo.Line.Trim())" -ForegroundColor Red 153 | Write-Host "Error Message : $($_.Exception.Message)"`n -ForegroundColor Red 154 | Write-Host "Terminating script execution..."`n 155 | } 156 | 157 | exit 158 | 159 | 160 | -------------------------------------------------------------------------------- /2020/automating-power-bi-deployments/Reports - Deploy Report.ps1: -------------------------------------------------------------------------------- 1 | <# 2 | ** Check the execution policy for the current machine/user, and set it to unrestricted (if required) ** 3 | 4 | Get-ExecutionPolicy -List 5 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope LocalMachine #if you want to set it for the machine 6 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process #if you want to set it for the current process 7 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser #if you want to set it for current user only 8 | #> 9 | 10 | <# 11 | ** Install and import the following Power BI management cmdlets (if required) ** 12 | 13 | Install-Module -Name MicrosoftPowerBIMgmt -AllowClobber 14 | Import-Module MicrosoftPowerBIMgmt 15 | #> 16 | 17 | <# 18 | Script Notes 19 | ----------------------------------------------------------------------------------------- 20 | 21 | ** This script will deploy a pbix file (report/dataset combination), overwriting it if it 22 | already exists. 23 | 24 | ** Note that we're ignoring an edge-case "error" in this script, if more than one reports 25 | are associated with the dataset we're attempting to overwrite. 26 | 27 | 28 | ** Execute this script in the following way in a PowerShell session (run as admin) 29 | ex. &'c:\PowerShell\Power BI\Reports - Deploy Report.ps1' -WorkspaceName 'MyWorkspace' -PbixFilePath 'c:\my report.pbix' -ReportName 'MyReportName' 30 | 31 | 32 | Technical References 33 | ----------------------------------------------------------------------------------------- 34 | 35 | Power BI cmdlets 36 | https://docs.microsoft.com/en-us/powershell/power-bi/overview?view=powerbi-ps 37 | 38 | Power BI REST APIs 39 | https://docs.microsoft.com/en-us/rest/api/power-bi/ 40 | #> 41 | 42 | <#------------------------------------------------------------------------------------------- 43 | 44 | Parameters 45 | 46 | -------------------------------------------------------------------------------------------#> 47 | 48 | param 49 | ( 50 | [Parameter(Mandatory=$true)] 51 | [ValidateNotNullOrEmpty()] 52 | [String] 53 | $WorkspaceName, 54 | 55 | [Parameter(Mandatory=$true)] 56 | [ValidateNotNullOrEmpty()] 57 | [String] 58 | $PbixFilePath, 59 | 60 | [Parameter(Mandatory=$true)] 61 | [ValidateNotNullOrEmpty()] 62 | [String] 63 | $ReportName 64 | ) 65 | 66 | <#------------------------------------------------------------------------------------------- 67 | 68 | Global actions & variables 69 | 70 | -------------------------------------------------------------------------------------------#> 71 | 72 | $ErrorActionPreference = "Stop" 73 | Write-Host "**NOTE: THIS SCRIPT WILL STOP EXECUTING AFTER THE FIRST ERROR**"`r`n -ForegroundColor Yellow 74 | 75 | 76 | try { 77 | 78 | <#------------------------------------------------------------------------------------------- 79 | 80 | Connect to Power BI 81 | 82 | -------------------------------------------------------------------------------------------#> 83 | 84 | #Connect to Power BI tenant 85 | Connect-PowerBIServiceAccount | Out-Null 86 | 87 | 88 | <#------------------------------------------------------------------------------------------- 89 | 90 | Deploy .pbix file 91 | 92 | -------------------------------------------------------------------------------------------#> 93 | 94 | #Retrieve the workspace 95 | $WorkspaceObject = (Get-PowerBIWorkspace -Scope Organization -Name $WorkspaceName) 96 | 97 | try { 98 | 99 | #Deploy pbix file 100 | New-PowerBIReport -Workspace $WorkspaceObject -Path $PbixFilePath -Name $ReportName -ConflictAction CreateOrOverwrite | Out-Null 101 | } 102 | 103 | catch { 104 | 105 | #This "error" means that there were more than one report associated with the dataset, so we're going to ignore it and continuing 106 | if ($_.Exception.Message -ccontains "Sequence contains more than one element") { 107 | 108 | Write-Output "More than one report was associated with this dataset. We're ignoring the ""error"" and continuing..." `n 109 | 110 | #It's a non-terminating error, but we'll clear it to be safe 111 | $error.Clear() 112 | 113 | } 114 | 115 | #A real error occurred, so throw it 116 | else { 117 | 118 | throw 119 | 120 | } 121 | } 122 | 123 | #Retrieve the newly deployed report 124 | $PbiReportObject = (Get-PowerBIReport -Workspace $WorkspaceObject -Name $ReportName) 125 | 126 | #Check to see if it's been deployed successfully 127 | if ($PbiReportObject) { 128 | 129 | Write-Output "Report ""$ReportName"" successfully deployed..." `n 130 | } 131 | 132 | else { 133 | 134 | #throw an error if it hasn't been deployed successfully (i.e. the variable is empty) 135 | throw "Hmmm...something went wrong. The report was not deployed :-/" 136 | } 137 | } 138 | 139 | catch { 140 | Write-Host ""`n 141 | Write-Host "An error has occurred!!" -ForegroundColor Red 142 | Write-Host "Error Line Number : $($_.InvocationInfo.ScriptLineNumber)" -ForegroundColor Red 143 | Write-Host "Error Command : $($_.InvocationInfo.Line.Trim())" -ForegroundColor Red 144 | Write-Host "Error Message : $($_.Exception.Message)"`n -ForegroundColor Red 145 | Write-Host "Terminating script execution..."`n 146 | } 147 | 148 | exit 149 | 150 | 151 | -------------------------------------------------------------------------------- /2020/automating-power-bi-deployments/Reports - Trigger Refresh.ps1: -------------------------------------------------------------------------------- 1 | <# 2 | ** Check the execution policy for the current machine/user, and set it to unrestricted (if required) ** 3 | 4 | Get-ExecutionPolicy -List 5 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope LocalMachine #if you want to set it for the machine 6 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process #if you want to set it for the current process 7 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser #if you want to set it for current user only 8 | #> 9 | 10 | <# 11 | ** Install and import the following Power BI management cmdlets (if required) ** 12 | 13 | Install-Module -Name MicrosoftPowerBIMgmt -AllowClobber 14 | Import-Module MicrosoftPowerBIMgmt 15 | #> 16 | 17 | <# 18 | Script Notes 19 | ----------------------------------------------------------------------------------------- 20 | 21 | ** This script will trigger a dataset refresh. 22 | 23 | ** Execute this script in the following way in a PowerShell session (run as admin) 24 | ex. &'c:\PowerShell\Power BI\Reports - Trigger Refresh.ps1' -WorkspaceName 'MyWorkspace' -DatasetName 'MyDatasetName' 25 | 26 | 27 | Technical References 28 | ----------------------------------------------------------------------------------------- 29 | 30 | Power BI cmdlets 31 | https://docs.microsoft.com/en-us/powershell/power-bi/overview?view=powerbi-ps 32 | 33 | Power BI REST APIs 34 | https://docs.microsoft.com/en-us/rest/api/power-bi/ 35 | #> 36 | 37 | <#------------------------------------------------------------------------------------------- 38 | 39 | Parameters 40 | 41 | -------------------------------------------------------------------------------------------#> 42 | 43 | param 44 | ( 45 | [Parameter(Mandatory=$true)] 46 | [ValidateNotNullOrEmpty()] 47 | [String] 48 | $WorkspaceName, 49 | 50 | [Parameter(Mandatory=$true)] 51 | [ValidateNotNullOrEmpty()] 52 | [String] 53 | $DatasetName 54 | ) 55 | 56 | <#------------------------------------------------------------------------------------------- 57 | 58 | Global actions & variables 59 | 60 | -------------------------------------------------------------------------------------------#> 61 | 62 | $ErrorActionPreference = "Stop" 63 | Write-Host "**NOTE: THIS SCRIPT WILL STOP EXECUTING AFTER THE FIRST ERROR**"`r`n -ForegroundColor Yellow 64 | 65 | 66 | try { 67 | 68 | <#------------------------------------------------------------------------------------------- 69 | 70 | Connect to Power BI 71 | 72 | -------------------------------------------------------------------------------------------#> 73 | 74 | #Connect to Power BI tenant 75 | Connect-PowerBIServiceAccount | Out-Null 76 | 77 | 78 | <#------------------------------------------------------------------------------------------- 79 | 80 | Retrieve workspace and report objects 81 | 82 | -------------------------------------------------------------------------------------------#> 83 | 84 | #Retrieve the workspace 85 | $WorkspaceObject = (Get-PowerBIWorkspace -Scope Organization -Name $WorkspaceName) 86 | 87 | #Retrieve the dataset 88 | $PbiDatasetObject = (Get-PowerBIDataset -Scope Organization -Workspace $WorkspaceObject -Name $DatasetName) 89 | 90 | 91 | <#------------------------------------------------------------------------------------------- 92 | 93 | Trigger dataset refresh 94 | 95 | -------------------------------------------------------------------------------------------#> 96 | 97 | #API url for the refresh 98 | $ApiUrl = "groups/" + $WorkspaceObject.Id + "/datasets/" + $PbiDatasetObject.Id + "/refreshes" 99 | 100 | $ApiRequestBody = @" 101 | { 102 | "type": "Full", 103 | "notifyOption": "MailOnCompletion" 104 | } 105 | "@ 106 | 107 | 108 | #Trigger refresh 109 | Invoke-PowerBIRestMethod -Url $ApiUrl -Method Post -Body ("$ApiRequestBody") -Verbose 110 | } 111 | 112 | catch { 113 | Write-Host ""`n 114 | Write-Host "An error has occurred!!" -ForegroundColor Red 115 | Write-Host "Error Line Number : $($_.InvocationInfo.ScriptLineNumber)" -ForegroundColor Red 116 | Write-Host "Error Command : $($_.InvocationInfo.Line.Trim())" -ForegroundColor Red 117 | Write-Host "Error Message : $($_.Exception.Message)"`n -ForegroundColor Red 118 | Write-Host "Terminating script execution..."`n 119 | } 120 | 121 | exit 122 | -------------------------------------------------------------------------------- /2020/automating-power-bi-deployments/Reports - Update Parameter.ps1: -------------------------------------------------------------------------------- 1 | <# 2 | ** Check the execution policy for the current machine/user, and set it to unrestricted (if required) ** 3 | 4 | Get-ExecutionPolicy -List 5 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope LocalMachine #if you want to set it for the machine 6 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process #if you want to set it for the current process 7 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser #if you want to set it for current user only 8 | #> 9 | 10 | <# 11 | ** Install and import the following Power BI management cmdlets (if required) ** 12 | 13 | Install-Module -Name MicrosoftPowerBIMgmt -AllowClobber 14 | Import-Module MicrosoftPowerBIMgmt 15 | #> 16 | 17 | <# 18 | Script Notes 19 | ----------------------------------------------------------------------------------------- 20 | 21 | ** This script will update a dataset parameter "Customer Name" in a report. 22 | 23 | ** Execute this script in the following way in a PowerShell session (run as admin) 24 | ex. &'c:\PowerShell\Power BI\Reports - Update Parameters.ps1' -WorkspaceName 'MyWorkspace' -ReportName 'MyReportName' -CustomerName 'MyCustomerName' 25 | 26 | 27 | Technical References 28 | ----------------------------------------------------------------------------------------- 29 | 30 | Power BI cmdlets 31 | https://docs.microsoft.com/en-us/powershell/power-bi/overview?view=powerbi-ps 32 | 33 | Power BI REST APIs 34 | https://docs.microsoft.com/en-us/rest/api/power-bi/ 35 | #> 36 | 37 | <#------------------------------------------------------------------------------------------- 38 | 39 | Parameters 40 | 41 | -------------------------------------------------------------------------------------------#> 42 | 43 | param 44 | ( 45 | [Parameter(Mandatory=$true)] 46 | [ValidateNotNullOrEmpty()] 47 | [String] 48 | $WorkspaceName, 49 | 50 | [Parameter(Mandatory=$true)] 51 | [ValidateNotNullOrEmpty()] 52 | [String] 53 | $ReportName, 54 | 55 | [Parameter(Mandatory=$true)] 56 | [ValidateNotNullOrEmpty()] 57 | [String] 58 | $CustomerName 59 | ) 60 | 61 | <#------------------------------------------------------------------------------------------- 62 | 63 | Global actions & variables 64 | 65 | -------------------------------------------------------------------------------------------#> 66 | 67 | $ErrorActionPreference = "Stop" 68 | Write-Host "**NOTE: THIS SCRIPT WILL STOP EXECUTING AFTER THE FIRST ERROR**"`r`n -ForegroundColor Yellow 69 | 70 | 71 | try { 72 | 73 | <#------------------------------------------------------------------------------------------- 74 | 75 | Connect to Power BI 76 | 77 | -------------------------------------------------------------------------------------------#> 78 | 79 | #Connect to Power BI tenant 80 | Connect-PowerBIServiceAccount | Out-Null 81 | 82 | 83 | <#------------------------------------------------------------------------------------------- 84 | 85 | Retrieve workspace and report objects 86 | 87 | -------------------------------------------------------------------------------------------#> 88 | 89 | #Retrieve the workspace 90 | $WorkspaceObject = (Get-PowerBIWorkspace -Scope Organization -Name $WorkspaceName) 91 | 92 | #Retrieve the report 93 | $PbiReportObject = (Get-PowerBIReport -Workspace $WorkspaceObject -Name $ReportName) 94 | 95 | 96 | <#------------------------------------------------------------------------------------------- 97 | 98 | Update parameters 99 | 100 | -------------------------------------------------------------------------------------------#> 101 | 102 | #API url for parameters 103 | $ApiUrl = "groups/" + $WorkspaceObject.Id + "/datasets/" + $PbiReportObject.DatasetId + "/Default.UpdateParameters" 104 | 105 | #Build the request body 106 | $ApiRequestBody = @" 107 | { 108 | "updateDetails": [ 109 | { 110 | "name": "Customer Name", "newValue": "$CustomerName" 111 | } 112 | ] 113 | } 114 | "@ 115 | 116 | #Update parameters 117 | Invoke-PowerBIRestMethod -Url $ApiUrl -Method Post -Body ("$ApiRequestBody") 118 | 119 | Write-Output "Parameter ""Customer Name"" in report ""$ReportName"" sucessfully updated to ""$CustomerName""..." `n 120 | } 121 | 122 | catch { 123 | Write-Host ""`n 124 | Write-Host "An error has occurred!!" -ForegroundColor Red 125 | Write-Host "Error Line Number : $($_.InvocationInfo.ScriptLineNumber)" -ForegroundColor Red 126 | Write-Host "Error Command : $($_.InvocationInfo.Line.Trim())" -ForegroundColor Red 127 | Write-Host "Error Message : $($_.Exception.Message)"`n -ForegroundColor Red 128 | Write-Host "Terminating script execution..."`n 129 | } 130 | 131 | exit 132 | 133 | 134 | -------------------------------------------------------------------------------- /2020/automating-power-bi-deployments/Reports - Update Refresh Schedule.ps1: -------------------------------------------------------------------------------- 1 | <# 2 | ** Check the execution policy for the current machine/user, and set it to unrestricted (if required) ** 3 | 4 | Get-ExecutionPolicy -List 5 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope LocalMachine #if you want to set it for the machine 6 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process #if you want to set it for the current process 7 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser #if you want to set it for current user only 8 | #> 9 | 10 | <# 11 | ** Install and import the following Power BI management cmdlets (if required) ** 12 | 13 | Install-Module -Name MicrosoftPowerBIMgmt -AllowClobber 14 | Import-Module MicrosoftPowerBIMgmt 15 | #> 16 | 17 | <# 18 | Script Notes 19 | ----------------------------------------------------------------------------------------- 20 | 21 | ** This script will update the refresh schedule (once a day, at the specified time) of a report dataset. 22 | 23 | ** Execute this script in the following way in a PowerShell session (run as admin) 24 | ex. &'c:\PowerShell\Power BI\Reports - Update Refresh Schedule.ps1' -WorkspaceName 'MyWorkspace' -ReportName 'MyReportName' -Timezone 'Mountain Standard Time' -RefreshTime '04:30' 25 | 26 | 27 | Technical References 28 | ----------------------------------------------------------------------------------------- 29 | 30 | Power BI cmdlets 31 | https://docs.microsoft.com/en-us/powershell/power-bi/overview?view=powerbi-ps 32 | 33 | Power BI REST APIs 34 | https://docs.microsoft.com/en-us/rest/api/power-bi/ 35 | #> 36 | 37 | <#------------------------------------------------------------------------------------------- 38 | 39 | Parameters 40 | 41 | -------------------------------------------------------------------------------------------#> 42 | 43 | param 44 | ( 45 | [Parameter(Mandatory=$true)] 46 | [ValidateNotNullOrEmpty()] 47 | [String] 48 | $WorkspaceName, 49 | 50 | [Parameter(Mandatory=$true)] 51 | [ValidateNotNullOrEmpty()] 52 | [String] 53 | $ReportName, 54 | 55 | [Parameter(Mandatory=$true)] 56 | [ValidateNotNullOrEmpty()] 57 | [String] 58 | $Timezone, 59 | 60 | [Parameter(Mandatory=$true)] 61 | [ValidateNotNullOrEmpty()] 62 | [String] 63 | $RefreshTime 64 | ) 65 | 66 | <#------------------------------------------------------------------------------------------- 67 | 68 | Global actions & variables 69 | 70 | -------------------------------------------------------------------------------------------#> 71 | 72 | $ErrorActionPreference = "Stop" 73 | Write-Host "**NOTE: THIS SCRIPT WILL STOP EXECUTING AFTER THE FIRST ERROR**"`r`n -ForegroundColor Yellow 74 | 75 | 76 | try { 77 | 78 | <#------------------------------------------------------------------------------------------- 79 | 80 | Connect to Power BI 81 | 82 | -------------------------------------------------------------------------------------------#> 83 | 84 | #Connect to Power BI tenant 85 | Connect-PowerBIServiceAccount | Out-Null 86 | 87 | 88 | <#------------------------------------------------------------------------------------------- 89 | 90 | Retrieve workspace and report objects 91 | 92 | -------------------------------------------------------------------------------------------#> 93 | 94 | #Retrieve the workspace 95 | $WorkspaceObject = (Get-PowerBIWorkspace -Scope Organization -Name $WorkspaceName) 96 | 97 | #Retrieve the report 98 | $PbiReportObject = (Get-PowerBIReport -Workspace $WorkspaceObject -Name $ReportName) 99 | 100 | 101 | <#------------------------------------------------------------------------------------------- 102 | 103 | Update the dataset refresh schedule 104 | 105 | -------------------------------------------------------------------------------------------#> 106 | 107 | #API url for the refresh schedule 108 | $ApiUrl = "groups/" + $WorkspaceObject.Id + "/datasets/" + $PbiReportObject.DatasetId + "/refreshSchedule" 109 | 110 | $ApiRequestBody = @" 111 | { 112 | "value": { 113 | "days": [ 114 | "Sunday", 115 | "Monday", 116 | "Tuesday", 117 | "Wednesday", 118 | "Thursday", 119 | "Friday", 120 | "Saturday" 121 | ], 122 | "times": [ 123 | "$RefreshTime" 124 | ], 125 | "notifyOption": "MailOnFailure", 126 | "localTimeZoneId": "$Timezone", 127 | "enabled": true 128 | } 129 | } 130 | "@ 131 | 132 | 133 | #Update refresh schedule 134 | Invoke-PowerBIRestMethod -Url $ApiUrl -Method Patch -Body ("$ApiRequestBody") 135 | 136 | Write-Output "Data refresh schedule for report ""$ReportName"" updated successfully to once a day at $RefreshTime in the $Timezone time zone..." `n 137 | } 138 | 139 | catch { 140 | Write-Host ""`n 141 | Write-Host "An error has occurred!!" -ForegroundColor Red 142 | Write-Host "Error Line Number : $($_.InvocationInfo.ScriptLineNumber)" -ForegroundColor Red 143 | Write-Host "Error Command : $($_.InvocationInfo.Line.Trim())" -ForegroundColor Red 144 | Write-Host "Error Message : $($_.Exception.Message)"`n -ForegroundColor Red 145 | Write-Host "Terminating script execution..."`n 146 | } 147 | 148 | exit 149 | 150 | 151 | -------------------------------------------------------------------------------- /2020/automating-power-bi-deployments/Workspaces - Assign Permissions.ps1: -------------------------------------------------------------------------------- 1 | <# 2 | ** Check the execution policy for the current machine/user, and set it to unrestricted (if required) ** 3 | 4 | Get-ExecutionPolicy -List 5 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope LocalMachine #if you want to set it for the machine 6 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process #if you want to set it for the current process 7 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser #if you want to set it for current user only 8 | #> 9 | 10 | <# 11 | ** Install and import the following Power BI management cmdlets (if required) ** 12 | 13 | Install-Module -Name MicrosoftPowerBIMgmt -AllowClobber 14 | Import-Module MicrosoftPowerBIMgmt 15 | #> 16 | 17 | <# 18 | Script Notes 19 | ----------------------------------------------------------------------------------------- 20 | 21 | ** This script will assign user, group or app permissions to a workspace. 22 | ** You can assign the following permissions: Admin, Contributor, Member, None, Viewer 23 | ** Assigning the "None" permission will effectively remove the user/group/app permission 24 | 25 | ** Execute this script in the following way in a PowerShell session (run as admin) 26 | ex. &'c:\PowerShell\Power BI\Workspaces - Assign Permissions.ps1' -WorkspaceName 'MyWorkspace' -Users 'User1@me.com|Admin;User2@me.com|Member' -Groups 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx|Contributor' -Apps 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx|Contributor' 27 | 28 | 29 | Technical References 30 | ----------------------------------------------------------------------------------------- 31 | 32 | Power BI cmdlets 33 | https://docs.microsoft.com/en-us/powershell/power-bi/overview?view=powerbi-ps 34 | 35 | Power BI REST APIs 36 | https://docs.microsoft.com/en-us/rest/api/power-bi/ 37 | #> 38 | 39 | <#------------------------------------------------------------------------------------------- 40 | 41 | Parameters 42 | 43 | -------------------------------------------------------------------------------------------#> 44 | 45 | param 46 | ( 47 | [Parameter(Mandatory=$true)] 48 | [ValidateNotNullOrEmpty()] 49 | [String] 50 | $WorkspaceName, 51 | 52 | [Parameter(Mandatory=$false)] 53 | [ValidateNotNullOrEmpty()] 54 | [String] 55 | $Users, 56 | 57 | [Parameter(Mandatory=$false)] 58 | [ValidateNotNullOrEmpty()] 59 | [String] 60 | $Groups, 61 | 62 | [Parameter(Mandatory=$false)] 63 | [ValidateNotNullOrEmpty()] 64 | [String] 65 | $Apps 66 | ) 67 | 68 | <#------------------------------------------------------------------------------------------- 69 | 70 | Global actions & variables 71 | 72 | -------------------------------------------------------------------------------------------#> 73 | 74 | $ErrorActionPreference = "Stop" 75 | Write-Host "**NOTE: THIS SCRIPT WILL STOP EXECUTING AFTER THE FIRST ERROR**"`r`n -ForegroundColor Yellow 76 | 77 | 78 | try { 79 | 80 | <#------------------------------------------------------------------------------------------- 81 | 82 | Connect to Power BI 83 | 84 | -------------------------------------------------------------------------------------------#> 85 | 86 | #Connect to Power BI tenant 87 | Connect-PowerBIServiceAccount | Out-Null 88 | 89 | 90 | <#------------------------------------------------------------------------------------------- 91 | 92 | Retrieve workspace & set global variables 93 | 94 | -------------------------------------------------------------------------------------------#> 95 | 96 | #Retrieve the workspace 97 | $WorkspaceObject = (Get-PowerBIWorkspace -Scope Organization -Name $WorkspaceName) 98 | 99 | 100 | #API url for the workspace users/groups/apps 101 | $ApiUrl = "groups/" + $WorkspaceObject.Id + "/users" 102 | 103 | #Get a list of all workspace users/groups/apps, to see if we need to add or update the user 104 | $WorkspaceUsers = (Invoke-PowerBIRestMethod -Url $ApiUrl -Method Get) | ConvertFrom-Json 105 | 106 | 107 | <#------------------------------------------------------------------------------------------- 108 | 109 | Add/Update workspace users 110 | 111 | -------------------------------------------------------------------------------------------#> 112 | 113 | #Check if the optional parameter was provided 114 | if ($Users -ne "") { 115 | 116 | #Split the string and iterate through the items (users) 117 | foreach ($UserRolePair in $Users.Split(";")) { 118 | 119 | $UserName = $UserRolePair.Split("|")[0] 120 | $UserRole = $UserRolePair.Split("|")[1] 121 | 122 | #API request body for user permissions 123 | $ApiRequestBody = @" 124 | { 125 | "groupUserAccessRight": "$UserRole", 126 | "identifier": "$UserName", 127 | "principalType": "User" 128 | } 129 | "@ 130 | 131 | #See if the user already exists in the list of workspace users/groups/apps, and if so only update its role 132 | if ($UserName -cin $WorkspaceUsers.Value.identifier) { 133 | 134 | if ($UserRole -eq "None") { 135 | 136 | #Remove the user 137 | Invoke-PowerBIRestMethod -Url ($ApiUrl + "/$UserName") -Method Delete 138 | 139 | Write-Output "User ""$UserName"" was removed from the workspace ""$WorkspaceName""..." `n 140 | } 141 | else { 142 | 143 | Write-Output "User ""$UserName"" already exists in workspace ""$WorkspaceName"". Updating its role only..." 144 | 145 | #Update the user's role 146 | Invoke-PowerBIRestMethod -Url $ApiUrl -Method Put -Body ("$ApiRequestBody") 147 | 148 | Write-Output "User ""$UserName"" granted the ""$UserRole"" role..." `n 149 | } 150 | 151 | } 152 | 153 | #User doesn't exist, so we add it 154 | else { 155 | 156 | if ($UserRole -eq "None") { 157 | 158 | #User doesn't exist, so there's nothing to do 159 | Write-Output "User ""$UserName"" cannot be removed from the workspace ""$WorkspaceName"" because it doesn't exist..." `n 160 | } 161 | else { 162 | 163 | #Add the user and assign the role 164 | Invoke-PowerBIRestMethod -Url $ApiUrl -Method Post -Body ("$ApiRequestBody") 165 | 166 | Write-Output "User ""$UserName"" added to workspace ""$WorkspaceName"" as ""$UserRole""..." `n 167 | } 168 | } 169 | } 170 | } 171 | 172 | 173 | <#------------------------------------------------------------------------------------------- 174 | 175 | Add/Update workspace groups 176 | 177 | -------------------------------------------------------------------------------------------#> 178 | 179 | #Check if the optional parameter was provided 180 | if ($Groups -ne "") { 181 | 182 | #Split the string and iterate through the items (groups) 183 | foreach ($GroupRolePair in $Groups.Split(";")) { 184 | 185 | $GroupGuid = $GroupRolePair.Split("|")[0] 186 | $GroupRole = $GroupRolePair.Split("|")[1] 187 | 188 | #API request body for group permissions 189 | $ApiRequestBody = @" 190 | { 191 | "groupUserAccessRight": "$GroupRole", 192 | "identifier": "$GroupGuid", 193 | "principalType": "Group" 194 | } 195 | "@ 196 | 197 | #See if the group already exists in the list of workspace users/groups/apps, and if so only update its role 198 | if ($GroupGuid -cin $WorkspaceUsers.Value.identifier) { 199 | 200 | if ($GroupRole -eq "None") { 201 | 202 | #Remove the group 203 | Invoke-PowerBIRestMethod -Url ($ApiUrl + "/$GroupGuid") -Method Delete 204 | 205 | Write-Output "Group ""$GroupGuid"" was removed from the workspace ""$WorkspaceName""..." `n 206 | } 207 | else { 208 | 209 | Write-Output "Group ""$GroupGuid"" already exists in workspace ""$WorkspaceName"". Updating its role only..." 210 | 211 | #Update the group's role 212 | Invoke-PowerBIRestMethod -Url $ApiUrl -Method Put -Body ("$ApiRequestBody") 213 | 214 | Write-Output "Group ""$GroupGuid"" granted the ""$GroupRole"" role..." `n 215 | } 216 | } 217 | 218 | #Group doesn't exist, so we add it 219 | else { 220 | 221 | if ($GroupRole -eq "None") { 222 | 223 | #Group doesn't exist, so there's nothing to do 224 | Write-Output "Group ""$GroupGuid"" cannot be removed from the workspace ""$WorkspaceName"" because it doesn't exist..." `n 225 | } 226 | else { 227 | 228 | #Add the group and assign the role 229 | Invoke-PowerBIRestMethod -Url $ApiUrl -Method Post -Body ("$ApiRequestBody") 230 | 231 | Write-Output "Group ""$GroupGuid"" added to workspace ""$WorkspaceName"" as ""$GroupRole""..." `n 232 | } 233 | } 234 | } 235 | } 236 | 237 | 238 | <#------------------------------------------------------------------------------------------- 239 | 240 | Add/Update workspace service principals (apps) 241 | 242 | -------------------------------------------------------------------------------------------#> 243 | 244 | #Check if the optional parameter was provided 245 | if ($Apps -ne "") { 246 | 247 | #Split the string and iterate through the items (apps) 248 | foreach ($AppRolePair in $Apps.Split(";")) { 249 | 250 | $AppGuid = $AppRolePair.Split("|")[0] 251 | $AppRole = $AppRolePair.Split("|")[1] 252 | 253 | #API request body for app permissions 254 | $ApiRequestBody = @" 255 | { 256 | "groupUserAccessRight": "$AppRole", 257 | "identifier": "$AppGuid", 258 | "principalType": "App" 259 | } 260 | "@ 261 | 262 | #See if the app already exists in the list of workspace users/groups, and if so only update its role 263 | if ($AppGuid -cin $WorkspaceUsers.Value.identifier) { 264 | 265 | if ($AppRole -eq "None") { 266 | 267 | #Remove the app 268 | Invoke-PowerBIRestMethod -Url ($ApiUrl + "/$AppGuid") -Method Delete 269 | 270 | Write-Output "App ""$AppGuid"" was removed from the workspace ""$WorkspaceName""..." `n 271 | } 272 | else { 273 | 274 | Write-Output "App ""$AppGuid"" already exists in workspace ""$WorkspaceName"". Updating its role only..." 275 | 276 | #Update the app's role 277 | Invoke-PowerBIRestMethod -Url $ApiUrl -Method Put -Body ("$ApiRequestBody") 278 | 279 | Write-Output "App ""$AppGuid"" granted the ""$AppRole"" role..." `n 280 | } 281 | } 282 | 283 | #App doesn't exist, so we add it 284 | else { 285 | 286 | if ($AppRole -eq "None") { 287 | 288 | #App doesn't exist, so there's nothing to do 289 | Write-Output "App ""$AppGuid"" cannot be removed from the workspace ""$WorkspaceName"" because it doesn't exist..." `n 290 | } 291 | else { 292 | 293 | #Add the app and assign the role 294 | Invoke-PowerBIRestMethod -Url $ApiUrl -Method Post -Body ("$ApiRequestBody") -Verbose 295 | 296 | Write-Output "App ""$AppGuid"" added to workspace ""$WorkspaceName"" as ""$AppRole""..." `n 297 | } 298 | } 299 | } 300 | } 301 | } 302 | 303 | catch { 304 | Write-Host ""`n 305 | Write-Host "An error has occurred!!" -ForegroundColor Red 306 | Write-Host "Error Line Number : $($_.InvocationInfo.ScriptLineNumber)" -ForegroundColor Red 307 | Write-Host "Error Command : $($_.InvocationInfo.Line.Trim())" -ForegroundColor Red 308 | Write-Host "Error Message : $($_.Exception.Message)"`n -ForegroundColor Red 309 | Write-Host "Terminating script execution..."`n 310 | } 311 | 312 | exit 313 | 314 | 315 | -------------------------------------------------------------------------------- /2020/automating-power-bi-deployments/Workspaces - Create Workspace.ps1: -------------------------------------------------------------------------------- 1 | <# 2 | ** Check the execution policy for the current machine/user, and set it to unrestricted (if required) ** 3 | 4 | Get-ExecutionPolicy -List 5 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope LocalMachine #if you want to set it for the machine 6 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process #if you want to set it for the current process 7 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser #if you want to set it for current user only 8 | #> 9 | 10 | <# 11 | ** Install and import the following Power BI management cmdlets (if required) ** 12 | 13 | Install-Module -Name MicrosoftPowerBIMgmt -AllowClobber 14 | Import-Module MicrosoftPowerBIMgmt 15 | #> 16 | 17 | <# 18 | Script Notes 19 | ----------------------------------------------------------------------------------------- 20 | 21 | ** This script will create a workspace, restoring it if it already exists in a deleted state. 22 | 23 | ** Note that if restored, all datasets and reports will be there as well, but 24 | refresh schedules will be disabled and you may have to reenter credentials and parameters. 25 | 26 | 27 | ** Execute this script in the following way in a PowerShell session (run as admin) 28 | ex. &'c:\PowerShell\Power BI\Workspaces - Create Workspace.ps1' -WorkspaceName 'MyWorkspace' -WorkspaceDescription 'MyDescription' -AdminUser 'me@me.com' 29 | 30 | 31 | Technical References 32 | ----------------------------------------------------------------------------------------- 33 | 34 | Power BI cmdlets 35 | https://docs.microsoft.com/en-us/powershell/power-bi/overview?view=powerbi-ps 36 | 37 | Power BI REST APIs 38 | https://docs.microsoft.com/en-us/rest/api/power-bi/ 39 | #> 40 | 41 | <#------------------------------------------------------------------------------------------- 42 | 43 | Parameters 44 | 45 | -------------------------------------------------------------------------------------------#> 46 | 47 | param 48 | ( 49 | [Parameter(Mandatory=$true)] 50 | [ValidateNotNullOrEmpty()] 51 | [String] 52 | $WorkspaceName, 53 | 54 | [Parameter(Mandatory=$false)] 55 | [AllowNull()] 56 | [AllowEmptyString()] 57 | [String] 58 | $WorkspaceDescription = $null, 59 | 60 | [Parameter(Mandatory=$true)] 61 | [ValidateNotNullOrEmpty()] 62 | [String] 63 | $AdminUser 64 | ) 65 | 66 | <#------------------------------------------------------------------------------------------- 67 | 68 | Global actions & variables 69 | 70 | -------------------------------------------------------------------------------------------#> 71 | 72 | $ErrorActionPreference = "Stop" 73 | Write-Host "**NOTE: THIS SCRIPT WILL STOP EXECUTING AFTER THE FIRST ERROR**"`r`n -ForegroundColor Yellow 74 | 75 | 76 | try { 77 | 78 | <#------------------------------------------------------------------------------------------- 79 | 80 | Connect to Power BI 81 | 82 | -------------------------------------------------------------------------------------------#> 83 | 84 | #Connect to Power BI tenant 85 | Connect-PowerBIServiceAccount | Out-Null 86 | 87 | 88 | <#------------------------------------------------------------------------------------------- 89 | 90 | Create the Workspace 91 | 92 | -------------------------------------------------------------------------------------------#> 93 | 94 | #Retrieve the workspace 95 | $WorkspaceObject = (Get-PowerBIWorkspace -Scope Organization -Name $WorkspaceName -WarningAction SilentlyContinue) 96 | 97 | #Uncomment the next line if you want to see the details of the workspace 98 | #Write-Output $WorkspaceObject 99 | 100 | 101 | #Create the workspace, or restore it if it's in a deleted state 102 | if (!$WorkspaceObject) { 103 | 104 | #Some user-friendly messages :-) 105 | Write-Output "Workspace ""$WorkspaceName"" does not exist. Creating it now..." `n 106 | 107 | #Create new workspace 108 | New-PowerBIWorkspace -Name $WorkspaceName -OutVariable WorkspaceObject | Out-Null 109 | 110 | #Return completion message if it was created successfully 111 | if ($WorkspaceObject){ 112 | 113 | Write-Output "Workspace ""$WorkspaceName"" successfully created..." `n 114 | } 115 | } 116 | 117 | else { 118 | 119 | #Found the workspace and with a "deleted" status. 120 | if ($WorkspaceObject.State -eq "Deleted") { 121 | 122 | Write-Output "Found it! Restoring workspace ""$WorkspaceName"" now..." 123 | 124 | #Restore workspace 125 | Restore-PowerBIWorkspace -Id $WorkspaceObject.Id -RestoredName $WorkspaceName -AdminUserPrincipalName $AdminUser -WarningAction SilentlyContinue 126 | 127 | Write-Output "Workspace ""$WorkspaceName"" restored successfully..." `n 128 | } 129 | 130 | else { 131 | 132 | #Nothing to do here... 133 | Write-Output "Workspace ""$WorkspaceName"" exists and in an active state. Nothing to do here..." `n 134 | } 135 | } 136 | 137 | <#------------------------------------------------------------------------------------------- 138 | 139 | Update the description (if provided) 140 | 141 | -------------------------------------------------------------------------------------------#> 142 | 143 | if (($WorkspaceDescription -eq $null) -or ($WorkspaceDescription -eq "")) { 144 | 145 | #Description not provided...do nothing 146 | Write-Output "Workspace description not provided. All done." `n 147 | } 148 | 149 | else { 150 | 151 | #Update the description 152 | Set-PowerBIWorkspace -Scope Organization -Id $WorkspaceObject.Id -Description $WorkspaceDescription -WarningAction SilentlyContinue 153 | 154 | Write-Output "Workspace ""$WorkspaceName"" description updated to ""$WorkspaceDescription"". All done." `n 155 | } 156 | } 157 | 158 | catch { 159 | Write-Host ""`n 160 | Write-Host "An error has occurred!!" -ForegroundColor Red 161 | Write-Host "Error Line Number : $($_.InvocationInfo.ScriptLineNumber)" -ForegroundColor Red 162 | Write-Host "Error Command : $($_.InvocationInfo.Line.Trim())" -ForegroundColor Red 163 | Write-Host "Error Message : $($_.Exception.Message)"`n -ForegroundColor Red 164 | Write-Host "Terminating script execution..."`n 165 | } 166 | 167 | exit 168 | 169 | 170 | -------------------------------------------------------------------------------- /2020/change-password-expiration-policies-for-service-accounts-in-o365/Users - Change Password Expiration Policy.ps1: -------------------------------------------------------------------------------- 1 | <# 2 | ** Check the execution policy for the current machine/user, and set it to unrestricted (if required) ** 3 | 4 | Get-ExecutionPolicy -List 5 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope LocalMachine #if you want to set it for the machine 6 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process #if you want to set it for the current process 7 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser #if you want to set it for current user only 8 | #> 9 | 10 | <# 11 | ** Install and import the following Power BI management cmdlets (if required) ** 12 | 13 | Install-Module -Name AzureAD -AllowClobber 14 | Import-Module AzureAD 15 | #> 16 | 17 | <# 18 | Script Notes 19 | ----------------------------------------------------------------------------------------- 20 | 21 | ** This script will check if a user's password is set to expire, and optionally change it 22 | to not expire. You'll typically use this for service accounts, as there is no other way 23 | to change the default password policy for specific accounts. 24 | 25 | ** Execute this script in the following way in a PowerShell session (run as admin) 26 | ex. &'c:\PowerShell\Azure AD\Users - Change Password Expiration Policy.ps1' -UserGuid '00000000-0000-0000-0000-000000000000' -DisablePasswordExpiration 'No' 27 | #> 28 | 29 | <#------------------------------------------------------------------------------------------- 30 | 31 | Parameters 32 | 33 | -------------------------------------------------------------------------------------------#> 34 | 35 | Param 36 | ( 37 | [Parameter(Mandatory=$true)] 38 | [ValidateNotNullOrEmpty()] 39 | [String] 40 | $UserGuid, 41 | 42 | [Parameter(Mandatory=$false)] 43 | [ValidateNotNullOrEmpty()] 44 | [ValidateSet("Yes","No")] 45 | [String] 46 | $DisablePasswordExpiration = "No" 47 | ) 48 | 49 | <#------------------------------------------------------------------------------------------- 50 | 51 | Global actions & variables 52 | 53 | -------------------------------------------------------------------------------------------#> 54 | 55 | $ErrorActionPreference = "Stop" 56 | Write-Host "**NOTE: THIS SCRIPT WILL STOP EXECUTING AFTER THE FIRST ERROR**"`r`n -ForegroundColor Yellow 57 | 58 | 59 | Try 60 | { 61 | 62 | <#------------------------------------------------------------------------------------------- 63 | 64 | Connect to Azure AD 65 | 66 | -------------------------------------------------------------------------------------------#> 67 | 68 | #Connect to azure ad tenant 69 | Connect-AzureAD | Out-Null 70 | 71 | #Get the user's password expiration policy info 72 | Get-AzureADUser -ObjectId $UserGuid | Select-Object UserprincipalName, @{N="PasswordNeverExpires";E={$_.PasswordPolicies -contains "DisablePasswordExpiration"}} 73 | 74 | #Set the user's password to not expire 75 | if ($DisablePasswordExpiration -eq "Yes") 76 | { 77 | Set-AzureADUser -ObjectId $UserGuid -PasswordPolicies DisablePasswordExpiration 78 | Write-Output `n "User password policy set to not expire..." `n 79 | 80 | #Get the user's password expiration policy info 81 | Get-AzureADUser -ObjectId $UserGuid | Select-Object UserprincipalName,@{N="PasswordNeverExpires";E={$_.PasswordPolicies -contains "DisablePasswordExpiration"}} 82 | } 83 | 84 | } 85 | 86 | Catch 87 | { 88 | Write-Host ""`n 89 | Write-Host "An error has occurred!!" -ForegroundColor Red 90 | Write-Host "Error Line Number : $($_.InvocationInfo.ScriptLineNumber)" -ForegroundColor Red 91 | Write-Host "Error Command : $($_.InvocationInfo.Line.Trim())" -ForegroundColor Red 92 | Write-Host "Error Message : $($_.Exception.Message)"`n -ForegroundColor Red 93 | Write-Host "Terminating script execution..."`n 94 | } 95 | 96 | Exit 97 | 98 | 99 | -------------------------------------------------------------------------------- /2020/restore-a-power-bi-workspace/Workspaces - List Workspaces.ps1: -------------------------------------------------------------------------------- 1 | <# 2 | ** Check the execution policy for the current machine/user, and set it to unrestricted (if required) ** 3 | 4 | Get-ExecutionPolicy -List 5 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope LocalMachine #if you want to set it for the machine 6 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process #if you want to set it for the current process 7 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser #if you want to set it for current user only 8 | #> 9 | 10 | <# 11 | ** Install and import the following Power BI management cmdlets (if required) ** 12 | 13 | Install-Module -Name MicrosoftPowerBIMgmt -AllowClobber 14 | Import-Module MicrosoftPowerBIMgmt 15 | #> 16 | 17 | <# 18 | Script Notes 19 | ----------------------------------------------------------------------------------------- 20 | 21 | ** This script will list all workspaces in a Power BI tenant. 22 | 23 | ** Execute this script in the following way in a PowerShell session (run as admin) 24 | ex. &'c:\PowerShell\Power BI\Workspaces - List Workspaces.ps1' 25 | 26 | 27 | Technical References 28 | ----------------------------------------------------------------------------------------- 29 | 30 | Power BI cmdlets 31 | https://docs.microsoft.com/en-us/powershell/power-bi/overview?view=powerbi-ps 32 | 33 | Power BI REST APIs 34 | https://docs.microsoft.com/en-us/rest/api/power-bi/ 35 | #> 36 | 37 | <#------------------------------------------------------------------------------------------- 38 | 39 | Global actions & variables 40 | 41 | -------------------------------------------------------------------------------------------#> 42 | 43 | $ErrorActionPreference = "Stop" 44 | Write-Host "**NOTE: THIS SCRIPT WILL STOP EXECUTING AFTER THE FIRST ERROR**"`r`n -ForegroundColor Yellow 45 | 46 | 47 | Try 48 | { 49 | 50 | <#------------------------------------------------------------------------------------------- 51 | 52 | Connect to Power BI & List Workspaces 53 | 54 | -------------------------------------------------------------------------------------------#> 55 | 56 | #Connect to Power BI tenant 57 | Connect-PowerBIServiceAccount | Out-Null 58 | 59 | #Retrieve workspaces 60 | Get-PowerBIWorkspace -Scope Organization -All 61 | 62 | } 63 | 64 | Catch 65 | { 66 | Write-Host ""`n 67 | Write-Host "An error has occurred!!" -ForegroundColor Red 68 | Write-Host "Error Line Number : $($_.InvocationInfo.ScriptLineNumber)" -ForegroundColor Red 69 | Write-Host "Error Command : $($_.InvocationInfo.Line.Trim())" -ForegroundColor Red 70 | Write-Host "Error Message : $($_.Exception.Message)"`n -ForegroundColor Red 71 | Write-Host "Terminating script execution..."`n 72 | } 73 | 74 | Exit 75 | 76 | 77 | -------------------------------------------------------------------------------- /2020/restore-a-power-bi-workspace/Workspaces - Restore Workspace.ps1: -------------------------------------------------------------------------------- 1 | <# 2 | ** Check the execution policy for the current machine/user, and set it to unrestricted (if required) ** 3 | 4 | Get-ExecutionPolicy -List 5 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope LocalMachine #if you want to set it for the machine 6 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process #if you want to set it for the current process 7 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser #if you want to set it for current user only 8 | #> 9 | 10 | <# 11 | ** Install and import the following Power BI management cmdlets (if required) ** 12 | 13 | Install-Module -Name MicrosoftPowerBIMgmt -AllowClobber 14 | Import-Module MicrosoftPowerBIMgmt 15 | #> 16 | 17 | <# 18 | Script Notes 19 | ----------------------------------------------------------------------------------------- 20 | 21 | ** This script will restore a workspace, using the original name of the workspace. 22 | 23 | ** Note that after restoring it, all data sets and reports will be there as well, but 24 | refresh schedules will be disabled and you may have to reenter credentials and parameters. 25 | 26 | 27 | ** Execute this script in the following way in a PowerShell session (run as admin) 28 | ex. &'c:\PowerShell\Power BI\Workspaces - Restore Workspace.ps1' -WorkspaceName 'MyWorkspace' -AdminUser 'me@me.com' 29 | 30 | 31 | Technical References 32 | ----------------------------------------------------------------------------------------- 33 | 34 | Power BI cmdlets 35 | https://docs.microsoft.com/en-us/powershell/power-bi/overview?view=powerbi-ps 36 | 37 | Power BI REST APIs 38 | https://docs.microsoft.com/en-us/rest/api/power-bi/ 39 | #> 40 | 41 | <#------------------------------------------------------------------------------------------- 42 | 43 | Parameters 44 | 45 | -------------------------------------------------------------------------------------------#> 46 | 47 | Param 48 | ( 49 | [Parameter(Mandatory=$true)] 50 | [ValidateNotNullOrEmpty()] 51 | [String] 52 | $WorkspaceName, 53 | 54 | [Parameter(Mandatory=$true)] 55 | [ValidateNotNullOrEmpty()] 56 | [String] 57 | $AdminUser 58 | ) 59 | 60 | <#------------------------------------------------------------------------------------------- 61 | 62 | Global actions & variables 63 | 64 | -------------------------------------------------------------------------------------------#> 65 | 66 | $ErrorActionPreference = "Stop" 67 | Write-Host "**NOTE: THIS SCRIPT WILL STOP EXECUTING AFTER THE FIRST ERROR**"`r`n -ForegroundColor Yellow 68 | 69 | 70 | Try { 71 | 72 | <#------------------------------------------------------------------------------------------- 73 | 74 | Connect to Power BI 75 | 76 | -------------------------------------------------------------------------------------------#> 77 | 78 | #Connect to Power BI tenant 79 | Connect-PowerBIServiceAccount | Out-Null 80 | 81 | 82 | <#------------------------------------------------------------------------------------------- 83 | 84 | Restore the Workspace 85 | 86 | -------------------------------------------------------------------------------------------#> 87 | 88 | #Retrieve the workspace 89 | $WorkspaceObject = (Get-PowerBIWorkspace -Scope Organization -Name $WorkspaceName -WarningAction SilentlyContinue) 90 | #Write-Output $WorkspaceObject 91 | 92 | 93 | #Restore the workspace if it's in a deleted state 94 | if (!$WorkspaceObject) { 95 | Write-Output "Workspace ""$WorkspaceName"" does not exist. Nothing to do here..." `n 96 | } 97 | 98 | else { 99 | #Found the workspace and with a "deleted" status. 100 | if ($WorkspaceObject.State -eq "Deleted") { 101 | Write-Output "Found it! Restoring workspace ""$WorkspaceName"" now..." 102 | 103 | #Restore workspace 104 | Restore-PowerBIWorkspace -Id $WorkspaceObject.Id -RestoredName $WorkspaceName -AdminUserPrincipalName $AdminUser -WarningAction SilentlyContinue 105 | 106 | Write-Output "Workspace ""$WorkspaceName"" restored successfully..." `n 107 | } 108 | 109 | else { 110 | #Nothing to do here... 111 | Write-Output "Workspace ""$WorkspaceName"" exists and in an active state. Nothing to do here..." `n 112 | } 113 | } 114 | } 115 | 116 | Catch { 117 | Write-Host ""`n 118 | Write-Host "An error has occurred!!" -ForegroundColor Red 119 | Write-Host "Error Line Number : $($_.InvocationInfo.ScriptLineNumber)" -ForegroundColor Red 120 | Write-Host "Error Command : $($_.InvocationInfo.Line.Trim())" -ForegroundColor Red 121 | Write-Host "Error Message : $($_.Exception.Message)"`n -ForegroundColor Red 122 | Write-Host "Terminating script execution..."`n 123 | } 124 | 125 | Exit 126 | 127 | 128 | -------------------------------------------------------------------------------- /2021/azure-ad-assign-administrator-roles-with-powershell/Roles - Assign Administrator Role.ps1: -------------------------------------------------------------------------------- 1 | <# 2 | ** Check the execution policy for the current machine/user, and set it to unrestricted (if required) ** 3 | 4 | Get-ExecutionPolicy -List 5 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope LocalMachine #if you want to set it for the machine 6 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process #if you want to set it for the current process 7 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser #if you want to set it for current user only 8 | #> 9 | 10 | <# 11 | ** Install and import the following cmdlets (if required) ** 12 | 13 | Install-Module -Name AzureAD -AllowClobber 14 | Import-Module AzureAD 15 | #> 16 | 17 | <# 18 | Script Notes 19 | ----------------------------------------------------------------------------------------- 20 | 21 | ** This script will assign an Administrator role in Azure AD to a user or service principal, enabling the role if needed from the template. 22 | You can use either the email address or display name of the user, and display name or app id of the service principal. 23 | 24 | ** Execute this script in the following way in a PowerShell session (run as admin) 25 | ex. &'c:\PowerShell\Azure AD\Roles - Assign Administrator Role.ps1' AdminRole 'Intune Administrator' -User 'me@me.com' 26 | #> 27 | 28 | <#------------------------------------------------------------------------------------------- 29 | 30 | Parameters 31 | 32 | -------------------------------------------------------------------------------------------#> 33 | 34 | Param 35 | ( 36 | [Parameter(Mandatory=$true)] 37 | [ValidateNotNullOrEmpty()] 38 | [String] 39 | $AdminRole, 40 | 41 | [Parameter(Mandatory=$true)] 42 | [ValidateNotNullOrEmpty()] 43 | [String] 44 | $User 45 | ) 46 | 47 | <#------------------------------------------------------------------------------------------- 48 | 49 | Global actions & variables 50 | 51 | -------------------------------------------------------------------------------------------#> 52 | 53 | $ErrorActionPreference = "Stop" 54 | Write-Host "**NOTE: THIS SCRIPT WILL STOP EXECUTING AFTER THE FIRST ERROR**"`r`n -ForegroundColor Yellow 55 | 56 | 57 | try { 58 | 59 | <#------------------------------------------------------------------------------------------- 60 | 61 | Connect to Azure AD 62 | 63 | -------------------------------------------------------------------------------------------#> 64 | 65 | #Connect to Azure AD tenant 66 | Connect-AzureAD | Out-Null 67 | 68 | 69 | <#------------------------------------------------------------------------------------------- 70 | 71 | Get the admin role, or enable it if necessary 72 | 73 | -------------------------------------------------------------------------------------------#> 74 | 75 | #Get the AD role 76 | $AdminRoleObject = (Get-AzureADDirectoryRole | where {$_.DisplayName -eq $AdminRole}) 77 | 78 | if (-not $AdminRoleObject) { 79 | 80 | #Role doesn't exist in the tenant, so let's get the template 81 | $AdminRoleTemplateObject = (Get-AzureADDirectoryRoleTemplate | where {$_.DisplayName -eq $AdminRole}) 82 | 83 | if ($AdminRoleTemplateObject) { 84 | 85 | #Found it...now enable it 86 | Enable-AzureADDirectoryRole -RoleTemplateId $AdminRoleTemplateObject.ObjectId | Out-Null 87 | 88 | Write-Output `n "The template for the ""$AdminRole"" role was found and successfully enabled." 89 | 90 | #Assign it to the object now that we've enabled it 91 | $AdminRoleObject = (Get-AzureADDirectoryRole | where {$_.DisplayName -eq $AdminRole}) 92 | } 93 | else { 94 | 95 | #Couldn't find the role or template...time to throw an error 96 | throw "Could not find the admin role or template :-/" 97 | } 98 | 99 | } 100 | 101 | 102 | <#------------------------------------------------------------------------------------------- 103 | 104 | Get the user or service principal and assign role 105 | 106 | -------------------------------------------------------------------------------------------#> 107 | 108 | #Attempt to get the user by either principal or display name 109 | $UserObject = (Get-AzureADUser | where {($_.UserPrincipalName -eq $User) -or ($_.DisplayName -eq $User)}) 110 | 111 | #If the user isn't found, try to find a service principal with that name or App Id 112 | if (-not $UserObject) { 113 | 114 | $UserObject = (Get-AzureADServicePrincipal | where {($_.AppId -eq $User) -or ($_.DisplayName -eq $User)}) 115 | 116 | if (-not $UserObject) { 117 | 118 | #Couldn't find the user or service principal...time to throw an error 119 | throw "Could not find the user or service principal :-/" 120 | 121 | } 122 | else { 123 | 124 | #Assign the role to the service principal 125 | Add-AzureADDirectoryRoleMember -ObjectId $AdminRoleObject.ObjectId -RefObjectId $UserObject.ObjectId 126 | 127 | Write-Output `n "The admin role ""$AdminRole"" role was successfully assigned to service principal ""$User""." 128 | 129 | } 130 | 131 | } 132 | else { 133 | 134 | #Assign the role to the user 135 | Add-AzureADDirectoryRoleMember -ObjectId $AdminRoleObject.ObjectId -RefObjectId $UserObject.ObjectId 136 | 137 | Write-Output `n "The admin role ""$AdminRole"" role was successfully assigned to user ""$User""." 138 | 139 | } 140 | 141 | } 142 | 143 | catch { 144 | Write-Host ""`n 145 | Write-Host "An error has occurred!!" -ForegroundColor Red 146 | Write-Host "Error Line Number : $($_.InvocationInfo.ScriptLineNumber)" -ForegroundColor Red 147 | Write-Host "Error Command : $($_.InvocationInfo.Line.Trim())" -ForegroundColor Red 148 | Write-Host "Error Message : $($_.Exception.Message)"`n -ForegroundColor Red 149 | Write-Host "Terminating script execution..."`n 150 | } 151 | 152 | exit 153 | 154 | 155 | -------------------------------------------------------------------------------- /2021/fixing-incorrect-drill-through-results-in-analysis-services/Fixing Drillthrough Results.sln: -------------------------------------------------------------------------------- 1 |  2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio Version 16 4 | VisualStudioVersion = 16.0.30907.101 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{6870E480-7721-4708-BFB8-9AE898AA21B3}") = "Tabular Project - Part 1", "Tabular Project - Part 1\Tabular Project - Part 1.smproj", "{8CE414BB-95B2-4C99-9E03-51BA72086E22}" 7 | EndProject 8 | Project("{6870E480-7721-4708-BFB8-9AE898AA21B3}") = "Tabular Project - Part 2", "Tabular Project - Part 2\Tabular Project - Part 2.smproj", "{CE1AA93F-9448-40CF-BB16-9BEEBEFBE10E}" 9 | EndProject 10 | Global 11 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 12 | Development|x86 = Development|x86 13 | EndGlobalSection 14 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 15 | {8CE414BB-95B2-4C99-9E03-51BA72086E22}.Development|x86.ActiveCfg = Development|x86 16 | {8CE414BB-95B2-4C99-9E03-51BA72086E22}.Development|x86.Build.0 = Development|x86 17 | {8CE414BB-95B2-4C99-9E03-51BA72086E22}.Development|x86.Deploy.0 = Development|x86 18 | {CE1AA93F-9448-40CF-BB16-9BEEBEFBE10E}.Development|x86.ActiveCfg = Development|x86 19 | {CE1AA93F-9448-40CF-BB16-9BEEBEFBE10E}.Development|x86.Build.0 = Development|x86 20 | {CE1AA93F-9448-40CF-BB16-9BEEBEFBE10E}.Development|x86.Deploy.0 = Development|x86 21 | EndGlobalSection 22 | GlobalSection(SolutionProperties) = preSolution 23 | HideSolutionNode = FALSE 24 | EndGlobalSection 25 | GlobalSection(ExtensibilityGlobals) = postSolution 26 | SolutionGuid = {3090B76F-4BB9-4D5F-88A6-D19437CDB3B3} 27 | EndGlobalSection 28 | EndGlobal 29 | -------------------------------------------------------------------------------- /2021/fixing-incorrect-drill-through-results-in-analysis-services/Tabular Project - Part 1/Model.bim: -------------------------------------------------------------------------------- 1 | { 2 | "name": "SemanticModel", 3 | "compatibilityLevel": 1500, 4 | "model": { 5 | "culture": "en-US", 6 | "dataSources": [ 7 | { 8 | "type": "structured", 9 | "name": "AdventureWorksDW2019", 10 | "connectionDetails": { 11 | "protocol": "tds", 12 | "address": { 13 | "server": "PETER", 14 | "database": "AdventureWorksDW2019" 15 | }, 16 | "authentication": null, 17 | "query": null 18 | }, 19 | "credential": { 20 | "AuthenticationKind": "UsernamePassword", 21 | "kind": "SQL", 22 | "path": "peter;AdventureWorksDW2019", 23 | "Username": "sa", 24 | "EncryptConnection": false, 25 | "PrivacySetting": "Organizational" 26 | } 27 | } 28 | ], 29 | "tables": [ 30 | { 31 | "name": "Date", 32 | "dataCategory": "Time", 33 | "columns": [ 34 | { 35 | "name": "Date Key", 36 | "dataType": "int64", 37 | "isHidden": true, 38 | "sourceColumn": "DateKey", 39 | "formatString": "0", 40 | "summarizeBy": "none" 41 | }, 42 | { 43 | "name": "Date", 44 | "dataType": "dateTime", 45 | "isKey": true, 46 | "sourceColumn": "FullDateAlternateKey", 47 | "formatString": "M/d/yyyy", 48 | "annotations": [ 49 | { 50 | "name": "Format", 51 | "value": "" 52 | } 53 | ] 54 | }, 55 | { 56 | "name": "Year", 57 | "dataType": "int64", 58 | "sourceColumn": "CalendarYear", 59 | "formatString": "0", 60 | "summarizeBy": "none" 61 | }, 62 | { 63 | "name": "Quarter", 64 | "dataType": "int64", 65 | "sourceColumn": "CalendarQuarter", 66 | "formatString": "0", 67 | "summarizeBy": "none" 68 | }, 69 | { 70 | "name": "Month Number", 71 | "dataType": "int64", 72 | "sourceColumn": "MonthNumberOfYear", 73 | "formatString": "0", 74 | "summarizeBy": "none" 75 | }, 76 | { 77 | "name": "Month Name", 78 | "dataType": "string", 79 | "sourceColumn": "EnglishMonthName", 80 | "sortByColumn": "Month Number" 81 | }, 82 | { 83 | "type": "calculated", 84 | "name": "Month Key", 85 | "dataType": "int64", 86 | "isDataTypeInferred": false, 87 | "isHidden": true, 88 | "expression": " LEFT('Date'[Date Key], 6)", 89 | "formatString": "0", 90 | "summarizeBy": "none" 91 | }, 92 | { 93 | "type": "calculated", 94 | "name": "Month", 95 | "dataType": "string", 96 | "isDataTypeInferred": true, 97 | "expression": " LEFT('Date'[Month Name], 3) & \" \" & 'Date'[Year]", 98 | "sortByColumn": "Month Key" 99 | } 100 | ], 101 | "partitions": [ 102 | { 103 | "name": "Partition", 104 | "dataView": "full", 105 | "source": { 106 | "type": "m", 107 | "expression": [ 108 | "let", 109 | " Source = Value.NativeQuery(AdventureWorksDW2019, #\"Source Query - Date\")", 110 | "in", 111 | " Source" 112 | ] 113 | } 114 | } 115 | ] 116 | }, 117 | { 118 | "name": "Product", 119 | "columns": [ 120 | { 121 | "name": "Product Key", 122 | "dataType": "int64", 123 | "isHidden": true, 124 | "isKey": true, 125 | "sourceColumn": "ProductKey", 126 | "formatString": "0", 127 | "summarizeBy": "none" 128 | }, 129 | { 130 | "name": "Product Code", 131 | "dataType": "string", 132 | "sourceColumn": "ProductAlternateKey" 133 | }, 134 | { 135 | "name": "Product Category Key", 136 | "dataType": "int64", 137 | "isHidden": true, 138 | "sourceColumn": "ProductCategoryKey", 139 | "formatString": "0", 140 | "summarizeBy": "none" 141 | }, 142 | { 143 | "name": "Product Category", 144 | "dataType": "string", 145 | "sourceColumn": "EnglishProductCategoryName" 146 | }, 147 | { 148 | "name": "Product Sub-Category Key", 149 | "dataType": "int64", 150 | "isHidden": true, 151 | "sourceColumn": "ProductSubcategoryKey", 152 | "formatString": "0", 153 | "summarizeBy": "none" 154 | }, 155 | { 156 | "name": "Product Sub-Category", 157 | "dataType": "string", 158 | "sourceColumn": "EnglishProductSubcategoryName" 159 | }, 160 | { 161 | "name": "Product Name", 162 | "dataType": "string", 163 | "sourceColumn": "EnglishProductName" 164 | }, 165 | { 166 | "name": "List Price - Source", 167 | "dataType": "decimal", 168 | "isHidden": true, 169 | "sourceColumn": "ListPrice", 170 | "formatString": "\\$#,0.00;(\\$#,0.00);\\$#,0.00", 171 | "summarizeBy": "average", 172 | "annotations": [ 173 | { 174 | "name": "Format", 175 | "value": "" 176 | } 177 | ] 178 | } 179 | ], 180 | "partitions": [ 181 | { 182 | "name": "Partition", 183 | "dataView": "full", 184 | "source": { 185 | "type": "m", 186 | "expression": [ 187 | "let", 188 | " Source = Value.NativeQuery(AdventureWorksDW2019, #\"Source Query - Product\")", 189 | "in", 190 | " Source" 191 | ] 192 | } 193 | } 194 | ], 195 | "measures": [ 196 | { 197 | "name": "Avg List Price", 198 | "expression": " AVERAGE('Product'[List Price - Source])", 199 | "formatString": "\\$#,0.00;(\\$#,0.00);\\$#,0.00", 200 | "annotations": [ 201 | { 202 | "name": "Format", 203 | "value": "" 204 | } 205 | ] 206 | } 207 | ] 208 | }, 209 | { 210 | "name": "Internet Sales", 211 | "columns": [ 212 | { 213 | "name": "Product Key", 214 | "dataType": "int64", 215 | "isHidden": true, 216 | "sourceColumn": "ProductKey", 217 | "formatString": "0", 218 | "summarizeBy": "none" 219 | }, 220 | { 221 | "name": "Order Date Key", 222 | "dataType": "int64", 223 | "isHidden": true, 224 | "sourceColumn": "OrderDateKey", 225 | "formatString": "0", 226 | "summarizeBy": "none" 227 | }, 228 | { 229 | "name": "Due Date Key", 230 | "dataType": "int64", 231 | "isHidden": true, 232 | "sourceColumn": "DueDateKey", 233 | "formatString": "0", 234 | "summarizeBy": "none" 235 | }, 236 | { 237 | "name": "Ship Date Key", 238 | "dataType": "int64", 239 | "isHidden": true, 240 | "sourceColumn": "ShipDateKey", 241 | "formatString": "0", 242 | "summarizeBy": "none" 243 | }, 244 | { 245 | "name": "Order Number", 246 | "dataType": "string", 247 | "sourceColumn": "SalesOrderNumber" 248 | }, 249 | { 250 | "name": "Order Line Number", 251 | "dataType": "int64", 252 | "sourceColumn": "SalesOrderLineNumber", 253 | "formatString": "0", 254 | "summarizeBy": "none" 255 | }, 256 | { 257 | "name": "Quantity - Source", 258 | "dataType": "int64", 259 | "isHidden": true, 260 | "sourceColumn": "OrderQuantity", 261 | "formatString": "0", 262 | "summarizeBy": "sum" 263 | }, 264 | { 265 | "name": "Sales Amount - Source", 266 | "dataType": "decimal", 267 | "isHidden": true, 268 | "sourceColumn": "SalesAmount", 269 | "formatString": "\\$#,0.00;(\\$#,0.00);\\$#,0.00", 270 | "summarizeBy": "sum", 271 | "annotations": [ 272 | { 273 | "name": "Format", 274 | "value": "" 275 | } 276 | ] 277 | } 278 | ], 279 | "partitions": [ 280 | { 281 | "name": "Partition", 282 | "dataView": "full", 283 | "source": { 284 | "type": "m", 285 | "expression": [ 286 | "let", 287 | " Source = Value.NativeQuery(AdventureWorksDW2019, #\"Source Query - Internet Sales\")", 288 | "in", 289 | " Source" 290 | ] 291 | } 292 | } 293 | ], 294 | "measures": [ 295 | { 296 | "name": "# Orders", 297 | "expression": " DISTINCTCOUNT('Internet Sales'[Order Number])", 298 | "formatString": "#,0", 299 | "detailRowsDefinition": { 300 | "expression": "DETAILROWS([Internet Sales Drillthrough - Default])" 301 | } 302 | }, 303 | { 304 | "name": "# Orders Due", 305 | "expression": [ 306 | " ", 307 | "CALCULATE", 308 | "(", 309 | " DISTINCTCOUNT('Internet Sales'[Order Number])", 310 | ", USERELATIONSHIP('Internet Sales'[Due Date Key], 'Date'[Date Key])", 311 | ")" 312 | ], 313 | "formatString": "#,0", 314 | "detailRowsDefinition": { 315 | "expression": "DETAILROWS([Internet Sales Drillthrough - By Due Date])" 316 | } 317 | }, 318 | { 319 | "name": "# Orders Shipped", 320 | "expression": [ 321 | " ", 322 | "CALCULATE", 323 | "(", 324 | " DISTINCTCOUNT('Internet Sales'[Order Number])", 325 | ", USERELATIONSHIP('Internet Sales'[Ship Date Key], 'Date'[Date Key])", 326 | ")" 327 | ], 328 | "formatString": "#,0", 329 | "detailRowsDefinition": { 330 | "expression": "DETAILROWS([Internet Sales Drillthrough - By Ship Date])" 331 | } 332 | }, 333 | { 334 | "name": "Quantity", 335 | "expression": " SUM('Internet Sales'[Quantity - Source])", 336 | "formatString": "#,0", 337 | "detailRowsDefinition": { 338 | "expression": "DETAILROWS([Internet Sales Drillthrough - Default])" 339 | } 340 | }, 341 | { 342 | "name": "Sales Amount", 343 | "expression": " SUM('Internet Sales'[Sales Amount - Source])", 344 | "formatString": "\\$#,0.00;(\\$#,0.00);\\$#,0.00", 345 | "detailRowsDefinition": { 346 | "expression": "DETAILROWS([Internet Sales Drillthrough - Default])" 347 | }, 348 | "annotations": [ 349 | { 350 | "name": "Format", 351 | "value": "" 352 | } 353 | ] 354 | }, 355 | { 356 | "name": "Sales Amount (by Due Date)", 357 | "expression": [ 358 | " ", 359 | "CALCULATE", 360 | "(", 361 | " SUM('Internet Sales'[Sales Amount - Source])", 362 | ", USERELATIONSHIP('Internet Sales'[Due Date Key], 'Date'[Date Key])", 363 | ")" 364 | ], 365 | "formatString": "\\$#,0.00;(\\$#,0.00);\\$#,0.00", 366 | "detailRowsDefinition": { 367 | "expression": "DETAILROWS([Internet Sales Drillthrough - By Due Date])" 368 | }, 369 | "annotations": [ 370 | { 371 | "name": "Format", 372 | "value": "" 373 | } 374 | ] 375 | }, 376 | { 377 | "name": "Sales Amount (by Ship Date)", 378 | "expression": [ 379 | " ", 380 | "CALCULATE", 381 | "(", 382 | " SUM('Internet Sales'[Sales Amount - Source])", 383 | ", USERELATIONSHIP('Internet Sales'[Ship Date Key], 'Date'[Date Key])", 384 | ")" 385 | ], 386 | "formatString": "\\$#,0.00;(\\$#,0.00);\\$#,0.00", 387 | "detailRowsDefinition": { 388 | "expression": "DETAILROWS([Internet Sales Drillthrough - By Ship Date])" 389 | }, 390 | "annotations": [ 391 | { 392 | "name": "Format", 393 | "value": "" 394 | } 395 | ] 396 | }, 397 | { 398 | "name": "Internet Sales Drillthrough - Default", 399 | "expression": " 1", 400 | "isHidden": true, 401 | "detailRowsDefinition": { 402 | "expression": [ 403 | "SELECTCOLUMNS", 404 | "(", 405 | " 'Internet Sales'", 406 | ", \"Order Number\", 'Internet Sales'[Order Number]", 407 | ", \"Order Line Number\", 'Internet Sales'[Order Line Number]", 408 | ", \"Order Date\"", 409 | ", DATE", 410 | " (", 411 | " LEFT('Internet Sales'[Order Date Key], 4)", 412 | " , MID('Internet Sales'[Order Date Key], 5, 2)", 413 | " , RIGHT('Internet Sales'[Order Date Key], 2)", 414 | " )", 415 | ", \"Due Date\"", 416 | ", DATE", 417 | " (", 418 | " LEFT('Internet Sales'[Due Date Key], 4)", 419 | " , MID('Internet Sales'[Due Date Key], 5, 2)", 420 | " , RIGHT('Internet Sales'[Due Date Key], 2)", 421 | " )", 422 | ", \"Ship Date\"", 423 | ", DATE", 424 | " (", 425 | " LEFT('Internet Sales'[Ship Date Key], 4)", 426 | " , MID('Internet Sales'[Ship Date Key], 5, 2)", 427 | " , RIGHT('Internet Sales'[Ship Date Key], 2)", 428 | " )", 429 | ", \"Quantity\", 'Internet Sales'[Quantity - Source]", 430 | ", \"Sales Amount\", 'Internet Sales'[Sales Amount - Source]", 431 | ")" 432 | ] 433 | } 434 | }, 435 | { 436 | "name": "Internet Sales Drillthrough - By Due Date", 437 | "expression": " 1", 438 | "isHidden": true, 439 | "detailRowsDefinition": { 440 | "expression": [ 441 | "CALCULATETABLE", 442 | "(", 443 | " DETAILROWS('Internet Sales'[Internet Sales Drillthrough - Default])", 444 | ", USERELATIONSHIP('Internet Sales'[Due Date Key], 'Date'[Date Key])", 445 | ")" 446 | ] 447 | } 448 | }, 449 | { 450 | "name": "Internet Sales Drillthrough - By Ship Date", 451 | "expression": " 1", 452 | "isHidden": true, 453 | "detailRowsDefinition": { 454 | "expression": [ 455 | "CALCULATETABLE", 456 | "(", 457 | " DETAILROWS('Internet Sales'[Internet Sales Drillthrough - Default])", 458 | ", USERELATIONSHIP('Internet Sales'[Ship Date Key], 'Date'[Date Key])", 459 | ")" 460 | ] 461 | } 462 | } 463 | ] 464 | } 465 | ], 466 | "relationships": [ 467 | { 468 | "name": "9061b661-7bb3-4087-ab76-c082c8bf8905", 469 | "fromTable": "Internet Sales", 470 | "fromColumn": "Order Date Key", 471 | "toTable": "Date", 472 | "toColumn": "Date Key" 473 | }, 474 | { 475 | "name": "9581210a-de40-485b-ae6e-75439ca4d6c0", 476 | "fromTable": "Internet Sales", 477 | "fromColumn": "Due Date Key", 478 | "toTable": "Date", 479 | "toColumn": "Date Key", 480 | "isActive": false 481 | }, 482 | { 483 | "name": "4729bf6d-1ff0-48a9-b6ce-caf55f946f12", 484 | "fromTable": "Internet Sales", 485 | "fromColumn": "Ship Date Key", 486 | "toTable": "Date", 487 | "toColumn": "Date Key", 488 | "isActive": false 489 | }, 490 | { 491 | "name": "d4c86d6c-f091-48da-8293-48539bd6de9f", 492 | "fromTable": "Internet Sales", 493 | "fromColumn": "Product Key", 494 | "toTable": "Product", 495 | "toColumn": "Product Key" 496 | } 497 | ], 498 | "expressions": [ 499 | { 500 | "name": "Source Query - Product", 501 | "kind": "m", 502 | "expression": [ 503 | "let", 504 | " Source = \"select#(tab)prd.[ProductKey]#(lf) ,#(tab)prd.[ProductAlternateKey]#(lf)#(tab),#(tab)cat.[ProductCategoryKey]#(lf)#(tab),#(tab)cat.[EnglishProductCategoryName]#(lf) ,#(tab)sub.[ProductSubcategoryKey]#(lf)#(tab),#(tab)sub.[EnglishProductSubcategoryName]#(lf)#(tab),#(tab)prd.[EnglishProductName]#(lf)#(tab),#(tab)prd.[ListPrice]#(lf) #(lf)from#(tab)[AdventureWorksDW2019].[dbo].[DimProduct] prd#(lf)left outer join [AdventureWorksDW2019].[dbo].[DimProductSubcategory] sub on sub.[ProductSubcategoryKey] = prd.[ProductSubcategoryKey]#(lf)left outer join [AdventureWorksDW2019].[dbo].[DimProductCategory] cat on cat.[ProductCategoryKey] = sub.[ProductCategoryKey]#(lf);\"", 505 | "in", 506 | " Source" 507 | ] 508 | }, 509 | { 510 | "name": "Source Query - Internet Sales", 511 | "kind": "m", 512 | "expression": [ 513 | "let", 514 | " Source = \"select#(tab)[ProductKey]#(lf) ,#(tab)[OrderDateKey]#(lf) ,#(tab)[DueDateKey]#(lf) ,#(tab)[ShipDateKey]#(lf) ,#(tab)[SalesOrderNumber]#(lf) ,#(tab)[SalesOrderLineNumber]#(lf) ,#(tab)[OrderQuantity]#(lf) ,#(tab)[SalesAmount]#(lf) #(lf)from#(tab)[AdventureWorksDW2019].[dbo].[FactInternetSales]#(lf);\"", 515 | "in", 516 | " Source" 517 | ] 518 | }, 519 | { 520 | "name": "Source Query - Date", 521 | "kind": "m", 522 | "expression": [ 523 | "let", 524 | " Source = \"select#(tab)[DateKey]#(lf) ,#(tab)[FullDateAlternateKey]#(lf)#(tab),#(tab)[CalendarYear]#(lf)#(tab),#(tab)[CalendarQuarter]#(lf)#(tab),#(tab)[MonthNumberOfYear]#(lf)#(tab),#(tab)[EnglishMonthName] #(lf) #(lf)from#(tab)[AdventureWorksDW2019].[dbo].[DimDate]#(lf);\"", 525 | "in", 526 | " Source" 527 | ] 528 | } 529 | ], 530 | "annotations": [ 531 | { 532 | "name": "ClientCompatibilityLevel", 533 | "value": "600" 534 | } 535 | ] 536 | }, 537 | "id": "SemanticModel" 538 | } -------------------------------------------------------------------------------- /2021/fixing-incorrect-drill-through-results-in-analysis-services/Tabular Project - Part 1/Model.bim_MartinSchoombee.settings: -------------------------------------------------------------------------------- 1 | localhost:61329dbOnDisktrueDoNotKeepBackupSinglefalsefalsefalsefalsefalsefalsefalse600 -------------------------------------------------------------------------------- /2021/fixing-incorrect-drill-through-results-in-analysis-services/Tabular Project - Part 1/Tabular Project - Part 1.smproj: -------------------------------------------------------------------------------- 1 |  2 | 3 | Development 4 | 2.0 5 | {8CE414BB-95B2-4C99-9E03-51BA72086E22} 6 | Exe 7 | MyRootNamespace 8 | MyAssemblyName 9 | false 10 | bin\ 11 | Tabular Project - Part 1 12 | 13 | 14 | bin\ 15 | localhost 16 | Developer 17 | Unknown 18 | Tabular Project - Part 1 19 | Model 20 | Default 21 | False 22 | Default 23 | InMemory 24 | Default 25 | 26 | 27 | localhost 28 | Developer 29 | Unknown 30 | Tabular Project - Part 1 31 | Model 32 | Default 33 | False 34 | Default 35 | InMemory 36 | Default 37 | 38 | 39 | 40 | 41 | Code 42 | 43 | 44 | 45 | -------------------------------------------------------------------------------- /2021/fixing-incorrect-drill-through-results-in-analysis-services/Tabular Project - Part 2/Model.bim_MartinSchoombee.settings: -------------------------------------------------------------------------------- 1 | localhost:50501dbOnDisktrueDoNotKeepBackupSinglefalsefalsefalsefalsefalsefalsefalse600 -------------------------------------------------------------------------------- /2021/fixing-incorrect-drill-through-results-in-analysis-services/Tabular Project - Part 2/Tabular Project - Part 2.smproj: -------------------------------------------------------------------------------- 1 |  2 | 3 | Development 4 | 2.0 5 | {ce1aa93f-9448-40cf-bb16-9beebefbe10e} 6 | Exe 7 | MyRootNamespace 8 | MyAssemblyName 9 | false 10 | bin\ 11 | Tabular Project - Part 2 12 | 13 | 14 | bin\ 15 | localhost 16 | Developer 17 | Unknown 18 | Tabular Project - Part 2 19 | Model 20 | Default 21 | False 22 | Default 23 | InMemory 24 | Default 25 | 26 | 27 | localhost 28 | Developer 29 | Unknown 30 | Tabular Project - Part 2 31 | Model 32 | Default 33 | False 34 | Default 35 | InMemory 36 | Default 37 | 38 | 39 | 40 | 41 | Code 42 | 43 | 44 | 45 | -------------------------------------------------------------------------------- /2021/using-calculation-groups-with-role-playing-dimensions/Tabular Project/Model.bim_MartinSchoombee.settings: -------------------------------------------------------------------------------- 1 | localhost:50820dbOnDisktrueDoNotKeepBackupSinglefalsefalsefalsefalsefalsefalsefalse600 -------------------------------------------------------------------------------- /2021/using-calculation-groups-with-role-playing-dimensions/Tabular Project/Tabular Project.smproj: -------------------------------------------------------------------------------- 1 |  2 | 3 | Development 4 | 2.0 5 | {8CE414BB-95B2-4C99-9E03-51BA72086E22} 6 | Exe 7 | MyRootNamespace 8 | MyAssemblyName 9 | false 10 | bin\ 11 | Tabular Project 12 | 13 | 14 | bin\ 15 | localhost 16 | Developer 17 | Unknown 18 | Tabular Project 19 | Model 20 | Default 21 | False 22 | Default 23 | InMemory 24 | Default 25 | 26 | 27 | localhost 28 | Developer 29 | Unknown 30 | Tabular Project 31 | Model 32 | Default 33 | False 34 | Default 35 | InMemory 36 | Default 37 | 38 | 39 | 40 | 41 | Code 42 | 43 | 44 | 45 | -------------------------------------------------------------------------------- /2021/using-calculation-groups-with-role-playing-dimensions/Using Calculation Groups with Role Playing Dimensions.sln: -------------------------------------------------------------------------------- 1 |  2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio Version 16 4 | VisualStudioVersion = 16.0.30907.101 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{6870E480-7721-4708-BFB8-9AE898AA21B3}") = "Tabular Project", "Tabular Project\Tabular Project.smproj", "{8CE414BB-95B2-4C99-9E03-51BA72086E22}" 7 | EndProject 8 | Global 9 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 10 | Development|x86 = Development|x86 11 | EndGlobalSection 12 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 13 | {8CE414BB-95B2-4C99-9E03-51BA72086E22}.Development|x86.ActiveCfg = Development|x86 14 | {8CE414BB-95B2-4C99-9E03-51BA72086E22}.Development|x86.Build.0 = Development|x86 15 | {8CE414BB-95B2-4C99-9E03-51BA72086E22}.Development|x86.Deploy.0 = Development|x86 16 | EndGlobalSection 17 | GlobalSection(SolutionProperties) = preSolution 18 | HideSolutionNode = FALSE 19 | EndGlobalSection 20 | GlobalSection(ExtensibilityGlobals) = postSolution 21 | SolutionGuid = {439DE876-4810-4A18-8295-326EE1A9ABC3} 22 | EndGlobalSection 23 | EndGlobal 24 | -------------------------------------------------------------------------------- /2022/creating-development-environments-in-azure/ARM - Deploy VM.ps1: -------------------------------------------------------------------------------- 1 | <# 2 | ** Check the execution policy for the current machine/user, and set it to unrestricted (if required) ** 3 | 4 | Get-ExecutionPolicy -List 5 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope LocalMachine #if you want to set it for the machine 6 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process #if you want to set it for the current process 7 | Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser #if you want to set it for current user only 8 | #> 9 | 10 | <# 11 | ** Install and import the following Azure Cmdlets (if required) ** 12 | 13 | Install-Module -Name Az -AllowClobber 14 | Import-Module Az 15 | #> 16 | 17 | <# 18 | Script Notes 19 | ----------------------------------------------------------------------------------------- 20 | 21 | ** This script will use the referenced ARM template and create a development VM. 22 | 23 | ** Execute this script in the following way in a PowerShell session (run as admin) 24 | ex. &'D:\Resources\PowerShell\Azure RM\ARM - Deploy VM.ps1' -CustomerName 'Customer1' -CustomerPrefix 'c1' -Location 'East US 2' -AdminUser 'mschoombee' -AdminPassword 'xxxxx' -SecurityRuleIpAddress '127.0.0.1' -TemplateFile 'D:\\Dropbox\\28twelve\Resources\\Azure RM\\ARM Template - VM - Development.json' 25 | #> 26 | 27 | <#------------------------------------------------------------------------------------------- 28 | 29 | Parameters 30 | 31 | -------------------------------------------------------------------------------------------#> 32 | 33 | Param 34 | ( 35 | [Parameter(Mandatory=$true)] 36 | [ValidateNotNullOrEmpty()] 37 | [String] 38 | $SubscriptionId, 39 | 40 | [Parameter(Mandatory=$true)] 41 | [ValidateNotNullOrEmpty()] 42 | [String] 43 | $CustomerName, 44 | 45 | [Parameter(Mandatory=$true)] 46 | [ValidateNotNullOrEmpty()] 47 | [String] 48 | $CustomerPrefix, 49 | 50 | [Parameter(Mandatory=$true)] 51 | [ValidateNotNullOrEmpty()] 52 | [String] 53 | $Location, 54 | 55 | [Parameter(Mandatory=$true)] 56 | [ValidateNotNullOrEmpty()] 57 | [String] 58 | $AdminUserName, 59 | 60 | [Parameter(Mandatory=$true)] 61 | [ValidateNotNullOrEmpty()] 62 | [String] 63 | $AdminPassword, 64 | 65 | [Parameter(Mandatory=$true)] 66 | [ValidateNotNullOrEmpty()] 67 | [String] 68 | $SecurityRuleIpAddress, 69 | 70 | [Parameter(Mandatory=$true)] 71 | [ValidateNotNullOrEmpty()] 72 | [String] 73 | $TemplateFile 74 | ) 75 | 76 | <#------------------------------------------------------------------------------------------- 77 | 78 | Global actions & variables 79 | 80 | -------------------------------------------------------------------------------------------#> 81 | 82 | $ErrorActionPreference = "Stop" 83 | Write-Host "**NOTE: THIS SCRIPT WILL STOP EXECUTING AFTER THE FIRST ERROR**"`r`n -ForegroundColor Yellow 84 | 85 | 86 | try 87 | { 88 | 89 | <#------------------------------------------------------------------------------------------- 90 | 91 | Connect to Azure Tenant 92 | 93 | -------------------------------------------------------------------------------------------#> 94 | 95 | #Connect to azure tenant and set context to subscription 96 | Connect-AzAccount | Out-Null 97 | Set-AzContext -SubscriptionId $SubscriptionId | Out-Null 98 | 99 | 100 | <#------------------------------------------------------------------------------------------- 101 | 102 | Create Resource Group 103 | 104 | -------------------------------------------------------------------------------------------#> 105 | 106 | #Derive the resource group name 107 | $ResourceGroupName = "28t-rg-$CustomerPrefix" 108 | 109 | #Create or update the resource group 110 | New-AzResourceGroup -Name $ResourceGroupName -Location $Location -Tag @{Customer="$CustomerName"} -Force 111 | 112 | 113 | <#------------------------------------------------------------------------------------------- 114 | 115 | Deploy Development VM 116 | 117 | -------------------------------------------------------------------------------------------#> 118 | $DeploymentDate = (Get-Date -format "yyyyMMdd") 119 | $DeploymentName = "$DeploymentDate-VM-28t-vm-$CustomerPrefix-dev" 120 | 121 | #ARM template parameter object 122 | $ParameterObject = @{ 123 | "_SubscriptionId" = "$SubscriptionId" 124 | "_CustomerName" = "$CustomerName" 125 | "_CustomerPrefix" = "$CustomerPrefix" 126 | "_Location" = "$Location" 127 | "_AdminUserName" = "$AdminUserName" 128 | "_AdminPassword" = "$AdminPassword" 129 | "_SecurityRuleIpAddress" = "$SecurityRuleIpAddress" 130 | } 131 | 132 | #Parameters for the cmdlet 133 | $Parameters = @{ 134 | "Name" = "$DeploymentName" 135 | "TemplateFile" = "$TemplateFile" 136 | "TemplateParameterObject" = $ParameterObject 137 | "Verbose" = $true 138 | } 139 | 140 | #Create deployment 141 | New-AzResourceGroupDeployment -ResourceGroupName "$ResourceGroupName" @Parameters 142 | 143 | } 144 | 145 | catch 146 | { 147 | Write-Host ""`n 148 | Write-Host "An error has occurred!!" -ForegroundColor Red 149 | Write-Host "Error Line Number : $($_.InvocationInfo.ScriptLineNumber)" -ForegroundColor Red 150 | Write-Host "Error Command : $($_.InvocationInfo.Line.Trim())" -ForegroundColor Red 151 | Write-Host "Error Message : $($_.Exception.Message)"`n -ForegroundColor Red 152 | Write-Host "Terminating script execution..."`n 153 | } 154 | 155 | exit 156 | 157 | 158 | -------------------------------------------------------------------------------- /2022/creating-development-environments-in-azure/ARM Template - VM.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", 3 | "contentVersion": "1.0.0.0", 4 | "parameters": { 5 | "_SubscriptionId" : { 6 | "defaultValue": "00000000-0000-0000-0000-000000000000", 7 | "type": "String", 8 | "metadata": { 9 | "description": "Azure Subscription Id." 10 | } 11 | }, 12 | "_CustomerName": { 13 | "defaultValue": "TestCustomer", 14 | "type": "String", 15 | "metadata": { 16 | "description": "Name of the customer. This will be used in a tag for easy reference." 17 | } 18 | }, 19 | "_CustomerPrefix": { 20 | "defaultValue": "tc", 21 | "type": "String", 22 | "metadata": { 23 | "description": "Customer prefix. This will be used in the names of resources for easy reference." 24 | } 25 | }, 26 | "_Location": { 27 | "defaultValue": "West US", 28 | "type": "String", 29 | "metadata": { 30 | "description": "Name of the location for the VM." 31 | } 32 | }, 33 | "_AdminUserName": { 34 | "defaultValue": "MyAdminUser", 35 | "type": "String", 36 | "metadata": { 37 | "description": "Local administrator account user name." 38 | } 39 | }, 40 | "_AdminPassword": { 41 | "defaultValue": "MyAdminPassword", 42 | "type": "String", 43 | "metadata": { 44 | "description": "Local administrator account password." 45 | } 46 | }, 47 | "_SecurityRuleIpAddress": { 48 | "defaultValue": "127.0.0.1", 49 | "type": "String", 50 | "metadata": { 51 | "description": "IP Address to grant RDP access to." 52 | } 53 | } 54 | }, 55 | "variables": { 56 | "_ResourceGroupName": "[concat('28t-rg-', parameters('_CustomerPrefix'))]", 57 | "_NetworkSecurityGroupName": "[concat('28t-', parameters('_CustomerPrefix'), '-nsg-1')]", 58 | "_VirtualNetworkName": "[concat('28t-', parameters('_CustomerPrefix'), '-vnet-1')]", 59 | "_VirtualSubnetName": "[concat('28t-', parameters('_CustomerPrefix'), '-subnet-1')]", 60 | "_VirtualMachineName": "[concat('28t-vm-', parameters('_CustomerPrefix'), '-dev')]", 61 | "_VirtualMachineSize": "Standard_D4as_v4", 62 | "_ImageSku": "20h2-pron-g2", 63 | "_OsDiskName": "[concat(variables('_VirtualMachineName'), '-osdisk-1')]", 64 | "_DataDiskName": "[concat(variables('_VirtualMachineName'), '-datadisk-1')]", 65 | "_NetworkInterfaceName": "[concat(variables('_VirtualMachineName'), '-nic-1')]", 66 | "_PublicIpAddressName": "[concat(variables('_VirtualMachineName'), '-ip-1')]", 67 | "_ShutdownScheduleName": "[concat('shutdown-computevm-', variables('_VirtualMachineName'))]", 68 | "_ShutdownTime": "1900", 69 | "_ShutdownTimeZone": "US Mountain Standard Time" 70 | }, 71 | "resources": [ 72 | { 73 | "type": "Microsoft.Network/networkSecurityGroups", 74 | "apiVersion": "2020-11-01", 75 | "name": "[variables('_NetworkSecurityGroupName')]", 76 | "location": "[parameters('_Location')]", 77 | "tags": { 78 | "Customer": "[parameters('_CustomerName')]" 79 | }, 80 | "properties": { 81 | "securityRules": [ 82 | { 83 | "name": "Allow-PETER-RDP", 84 | "properties": { 85 | "protocol": "TCP", 86 | "sourcePortRange": "*", 87 | "destinationPortRange": "3389", 88 | "sourceAddressPrefix": "[parameters('_SecurityRuleIpAddress')]", 89 | "destinationAddressPrefix": "*", 90 | "access": "Allow", 91 | "priority": 100, 92 | "direction": "Inbound", 93 | "sourcePortRanges": [], 94 | "destinationPortRanges": [], 95 | "sourceAddressPrefixes": [], 96 | "destinationAddressPrefixes": [] 97 | } 98 | } 99 | ] 100 | } 101 | }, 102 | { 103 | "type": "Microsoft.Network/publicIPAddresses", 104 | "apiVersion": "2020-11-01", 105 | "name": "[variables('_PublicIpAddressName')]", 106 | "location": "[parameters('_Location')]", 107 | "tags": { 108 | "Customer": "[parameters('_CustomerName')]" 109 | }, 110 | "properties": { 111 | "publicIPAddressVersion": "IPv4", 112 | "publicIPAllocationMethod": "Static" 113 | } 114 | }, 115 | { 116 | "type": "Microsoft.Network/virtualNetworks", 117 | "apiVersion": "2020-11-01", 118 | "name": "[variables('_VirtualNetworkName')]", 119 | "location": "[parameters('_Location')]", 120 | "tags": { 121 | "Customer": "[parameters('_CustomerName')]" 122 | }, 123 | "properties": { 124 | "addressSpace": { 125 | "addressPrefixes": [ 126 | "10.2.0.0/28" 127 | ] 128 | }, 129 | "subnets": [ 130 | { 131 | "name": "[variables('_VirtualSubnetName')]", 132 | "properties": { 133 | "addressPrefix": "10.2.0.0/28", 134 | "delegations": [], 135 | "privateEndpointNetworkPolicies": "Enabled", 136 | "privateLinkServiceNetworkPolicies": "Enabled" 137 | } 138 | } 139 | ], 140 | "virtualNetworkPeerings": [], 141 | "enableDdosProtection": false 142 | } 143 | }, 144 | { 145 | "type": "Microsoft.Network/virtualNetworks/subnets", 146 | "apiVersion": "2020-11-01", 147 | "name": "[concat(variables('_VirtualNetworkName'), '/', variables('_VirtualSubnetName'))]", 148 | "dependsOn": [ 149 | "[resourceId('Microsoft.Network/virtualNetworks', variables('_VirtualNetworkName'))]" 150 | ], 151 | "properties": { 152 | "addressPrefix": "10.2.0.0/28", 153 | "delegations": [], 154 | "privateEndpointNetworkPolicies": "Enabled", 155 | "privateLinkServiceNetworkPolicies": "Enabled" 156 | } 157 | }, 158 | { 159 | "type": "Microsoft.Network/networkInterfaces", 160 | "apiVersion": "2020-11-01", 161 | "name": "[variables('_NetworkInterfaceName')]", 162 | "location": "[parameters('_Location')]", 163 | "dependsOn": [ 164 | "[resourceId('Microsoft.Network/publicIPAddresses', variables('_PublicIpAddressName'))]", 165 | "[resourceId('Microsoft.Network/virtualNetworks/subnets', variables('_VirtualNetworkName'), variables('_VirtualSubnetName'))]", 166 | "[resourceId('Microsoft.Network/networkSecurityGroups', variables('_NetworkSecurityGroupName'))]" 167 | ], 168 | "tags": { 169 | "Customer": "[parameters('_CustomerName')]" 170 | }, 171 | "properties": { 172 | "ipConfigurations": [ 173 | { 174 | "name": "ipconfig1", 175 | "properties": { 176 | "privateIPAddress": "10.2.0.4", 177 | "privateIPAllocationMethod": "Dynamic", 178 | "publicIPAddress": { 179 | "id": "[resourceId('Microsoft.Network/publicIPAddresses', variables('_PublicIpAddressName'))]" 180 | }, 181 | "subnet": { 182 | "id": "[resourceId('Microsoft.Network/virtualNetworks/subnets', variables('_VirtualNetworkName'), variables('_VirtualSubnetName'))]" 183 | }, 184 | "primary": true, 185 | "privateIPAddressVersion": "IPv4" 186 | } 187 | } 188 | ], 189 | "dnsSettings": { 190 | "dnsServers": [] 191 | }, 192 | "enableAcceleratedNetworking": false, 193 | "enableIPForwarding": false, 194 | "networkSecurityGroup": { 195 | "id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('_NetworkSecurityGroupName'))]" 196 | } 197 | } 198 | }, 199 | { 200 | "type": "Microsoft.Compute/disks", 201 | "apiVersion": "2020-12-01", 202 | "name": "[variables('_DataDiskName')]", 203 | "location": "[parameters('_Location')]", 204 | "tags": { 205 | "Customer": "[parameters('_CustomerName')]" 206 | }, 207 | "sku": { 208 | "name": "Standard_LRS", 209 | "tier": "Standard" 210 | }, 211 | "properties": { 212 | "creationData": { 213 | "createOption": "Empty" 214 | }, 215 | "diskSizeGB": 256, 216 | "diskIOPSReadWrite": 500, 217 | "diskMBpsReadWrite": 60, 218 | "encryption": { 219 | "type": "EncryptionAtRestWithPlatformKey" 220 | }, 221 | "diskState": "Reserved", 222 | "networkAccessPolicy": "AllowAll" 223 | } 224 | }, 225 | { 226 | "type": "Microsoft.Compute/virtualMachines", 227 | "apiVersion": "2021-03-01", 228 | "name": "[variables('_VirtualMachineName')]", 229 | "location": "[parameters('_Location')]", 230 | "dependsOn": [ 231 | "[resourceId('Microsoft.Network/networkInterfaces', variables('_NetworkInterfaceName'))]" 232 | ], 233 | "tags": { 234 | "Customer": "[parameters('_CustomerName')]" 235 | }, 236 | "properties": { 237 | "hardwareProfile": { 238 | "vmSize": "[variables('_VirtualMachineSize')]" 239 | }, 240 | "osProfile": { 241 | "computerName": "[variables('_VirtualMachineName')]", 242 | "adminUsername": "[parameters('_AdminUserName')]", 243 | "adminPassword": "[parameters('_AdminPassword')]", 244 | "windowsConfiguration": { 245 | "provisionVMAgent": true, 246 | "enableAutomaticUpdates": true, 247 | "patchSettings": { 248 | "patchMode": "AutomaticByOS", 249 | "enableHotpatching": false 250 | } 251 | }, 252 | "allowExtensionOperations": true 253 | }, 254 | "storageProfile": { 255 | "imageReference": { 256 | "publisher": "MicrosoftWindowsDesktop", 257 | "offer": "Windows-10", 258 | "sku": "[variables('_ImageSku')]", 259 | "version": "latest" 260 | }, 261 | "osDisk": { 262 | "osType": "Windows", 263 | "name": "[variables('_OsDiskName')]", 264 | "createOption": "FromImage", 265 | "caching": "ReadWrite", 266 | "managedDisk": { 267 | "storageAccountType": "Standard_LRS" 268 | } 269 | }, 270 | "dataDisks": [ 271 | { 272 | "lun": 0, 273 | "name": "[variables('_DataDiskName')]", 274 | "createOption": "Attach", 275 | "caching": "ReadWrite", 276 | "writeAcceleratorEnabled": false, 277 | "managedDisk": { 278 | "id": "[concat('/subscriptions/', parameters('_SubscriptionId'), '/resourceGroups/', variables('_ResourceGroupName'), '/providers/Microsoft.Compute/disks/', variables('_DataDiskName'))]" 279 | }, 280 | "toBeDetached": false 281 | } 282 | ] 283 | }, 284 | "networkProfile": { 285 | "networkInterfaces": [ 286 | { 287 | "id": "[resourceId('Microsoft.Network/networkInterfaces', variables('_NetworkInterfaceName'))]" 288 | } 289 | ] 290 | }, 291 | "diagnosticsProfile": { 292 | "bootDiagnostics": { 293 | "enabled": true 294 | } 295 | }, 296 | "licenseType": "Windows_Client" 297 | }, 298 | "resources": [ 299 | { 300 | "type": "Microsoft.DevTestLab/schedules", 301 | "apiVersion": "2018-09-15", 302 | "name": "[variables('_ShutdownScheduleName')]", 303 | "location": "[parameters('_Location')]", 304 | "tags": { 305 | "Customer": "[parameters('_CustomerName')]" 306 | }, 307 | "dependsOn": [ 308 | "[concat('Microsoft.Compute/virtualMachines/', variables('_VirtualMachineName'))]" 309 | ], 310 | "properties": { 311 | "status": "Enabled", 312 | "taskType": "ComputeVmShutdownTask", 313 | "dailyRecurrence": { 314 | "time": "[variables('_ShutdownTime')]" 315 | }, 316 | "timeZoneId": "[variables('_ShutdownTimeZone')]", 317 | "notificationSettings": { 318 | "status": "Enabled", 319 | "timeInMinutes": 30, 320 | "emailRecipient": "me@me.com", 321 | "notificationLocale": "en" 322 | }, 323 | "targetResourceId": "[resourceId('Microsoft.Compute/virtualMachines', variables('_VirtualMachineName'))]" 324 | } 325 | } 326 | ] 327 | } 328 | ] 329 | } 330 | -------------------------------------------------------------------------------- /2024/building-a-framework-for-orchestration-in-azure-data-factory/Data Factory/1 - Linked Services/Linked Service Template - Azure Key Vault.jsonc: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", 3 | "contentVersion": "1.0.0.0", 4 | "resources": [ 5 | { 6 | "type": "Microsoft.DataFactory/factories/linkedservices", 7 | "name": "@[DataFactory]/@[LinkedService]", 8 | "apiVersion": "2018-06-01", 9 | "properties": { 10 | "annotations": [ 11 | "FrameworkVersion=1.0" 12 | ], 13 | "type": "AzureKeyVault", 14 | "typeProperties": { 15 | "baseUrl": "@[Url]" 16 | } 17 | } 18 | } 19 | ] 20 | } -------------------------------------------------------------------------------- /2024/building-a-framework-for-orchestration-in-azure-data-factory/Data Factory/1 - Linked Services/Linked Service Template - Azure SQL Database.jsonc: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", 3 | "contentVersion": "1.0.0.0", 4 | "resources": [ 5 | { 6 | "type": "Microsoft.DataFactory/factories/linkedservices", 7 | "name": "@[DataFactory]/@[LinkedService]", 8 | "apiVersion": "2018-06-01", 9 | "dependsOn": [], 10 | "properties": { 11 | "annotations": [ 12 | "FrameworkVersion=1.0" 13 | ], 14 | "connectVia": { 15 | "type": "IntegrationRuntimeReference", 16 | "referenceName": "@[IntegrationRuntime]" 17 | }, 18 | "type": "AzureSqlDatabase", 19 | /* 20 | --------------------------------------------------------- 21 | Connection String - Stored in Key Vault 22 | Authentication - Managed Identity 23 | --------------------------------------------------------- 24 | */ 25 | "typeProperties": { 26 | "connectionString": { 27 | "type": "AzureKeyVaultSecret", 28 | "store": { 29 | "referenceName": "@[KeyVault]", 30 | "type": "LinkedServiceReference" 31 | }, 32 | "secretName": "@[KeyVaultSecret]" 33 | } 34 | } 35 | /* 36 | --------------------------------------------------------- 37 | Connection String - Stored in Key Vault 38 | Authentication - Service Principal 39 | --------------------------------------------------------- 40 | "typeProperties": { 41 | "connectionString": { 42 | "type": "AzureKeyVaultSecret", 43 | "store": { 44 | "referenceName": "@[KeyVault]", 45 | "type": "LinkedServiceReference" 46 | }, 47 | "secretName": "@[KeyVaultSecret]" 48 | }, 49 | "tenant": "@[ActiveDirectoryTenantId]", 50 | "servicePrincipalId": "@[ServicePrincipalId]", 51 | "servicePrincipalKey": { 52 | "type": "AzureKeyVaultSecret", 53 | "store": { 54 | "referenceName": "@[KeyVault]", 55 | "type": "LinkedServiceReference" 56 | }, 57 | "secretName": "@[ServicePrincipalSecret]" 58 | } 59 | }, 60 | */ 61 | /* 62 | --------------------------------------------------------- 63 | Connection String - Explicitly Typed 64 | Authentication - Managed Identity 65 | --------------------------------------------------------- 66 | "typeProperties": { 67 | "connectionString": "@[ConnectionString]" 68 | }, 69 | */ 70 | /* 71 | --------------------------------------------------------- 72 | Connection String - Explicitly Typed 73 | Authentication - Service Principal 74 | --------------------------------------------------------- 75 | "typeProperties": { 76 | "connectionString": "@[ConnectionString]", 77 | "tenant": "@[ActiveDirectoryTenantId]", 78 | "servicePrincipalId": "@[ServicePrincipalId]", 79 | "servicePrincipalKey": { 80 | "type": "AzureKeyVaultSecret", 81 | "store": { 82 | "referenceName": "@[KeyVault]", 83 | "type": "LinkedServiceReference" 84 | }, 85 | "secretName": "@[ServicePrincipalSecret]" 86 | } 87 | }, 88 | */ 89 | /* 90 | --------------------------------------------------------- 91 | Connection String - Explicitly Typed 92 | Authentication - SQL Authentication (stored in Key Vault) 93 | --------------------------------------------------------- 94 | "typeProperties": { 95 | "connectionString": "@[ConnectionString]", 96 | "password": { 97 | "type": "AzureKeyVaultSecret", 98 | "store": { 99 | "referenceName": "@[KeyVault]", 100 | "type": "LinkedServiceReference" 101 | }, 102 | "secretName": "@[KeyVaultSecret]" 103 | } 104 | }, 105 | */ 106 | } 107 | } 108 | ] 109 | } -------------------------------------------------------------------------------- /2024/building-a-framework-for-orchestration-in-azure-data-factory/Data Factory/1 - Linked Services/Linked Service Template - Source - Azure SQL Database.jsonc: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", 3 | "contentVersion": "1.0.0.0", 4 | "resources": [ 5 | { 6 | "type": "Microsoft.DataFactory/factories/linkedservices", 7 | "name": "@[DataFactory]/@[LinkedService]", 8 | "apiVersion": "2018-06-01", 9 | "dependsOn": [], 10 | "properties": { 11 | "parameters": { 12 | "KeyVaultSecret": { 13 | "type": "string", 14 | "defaultValue": "@[KeyVaultSecret]" 15 | } 16 | }, 17 | "annotations": [ 18 | "FrameworkVersion=1.0" 19 | ], 20 | "connectVia": { 21 | "type": "IntegrationRuntimeReference", 22 | "referenceName": "@[IntegrationRuntime]" 23 | }, 24 | "type": "AzureSqlDatabase", 25 | /* 26 | --------------------------------------------------------- 27 | Connection String - Stored in Key Vault 28 | Authentication - Managed Identity 29 | --------------------------------------------------------- 30 | */ 31 | "typeProperties": { 32 | "connectionString": { 33 | "type": "AzureKeyVaultSecret", 34 | "store": { 35 | "referenceName": "@[KeyVault]", 36 | "type": "LinkedServiceReference" 37 | }, 38 | "secretName": { 39 | "value": "@linkedService().KeyVaultSecret", 40 | "type": "Expression" 41 | } 42 | } 43 | } 44 | /* 45 | --------------------------------------------------------- 46 | Connection String - Stored in Key Vault 47 | Authentication - Service Principal 48 | --------------------------------------------------------- 49 | "typeProperties": { 50 | "connectionString": { 51 | "type": "AzureKeyVaultSecret", 52 | "store": { 53 | "referenceName": "@[KeyVault]", 54 | "type": "LinkedServiceReference" 55 | }, 56 | "secretName": { 57 | "value": "@linkedService().KeyVaultSecret", 58 | "type": "Expression" 59 | } 60 | }, 61 | "tenant": "@[ActiveDirectoryTenantId]", 62 | "servicePrincipalId": "@[ServicePrincipalId]", 63 | "servicePrincipalKey": { 64 | "type": "AzureKeyVaultSecret", 65 | "store": { 66 | "referenceName": "@[KeyVault]", 67 | "type": "LinkedServiceReference" 68 | }, 69 | "secretName": "@[ServicePrincipalSecret]" 70 | } 71 | }, 72 | */ 73 | /* 74 | --------------------------------------------------------- 75 | Connection String - Explicitly Typed 76 | Authentication - Managed Identity 77 | --------------------------------------------------------- 78 | "typeProperties": { 79 | "connectionString": "@[ConnectionString]" 80 | }, 81 | */ 82 | /* 83 | --------------------------------------------------------- 84 | Connection String - Explicitly Typed 85 | Authentication - Service Principal 86 | --------------------------------------------------------- 87 | "typeProperties": { 88 | "connectionString": "@[ConnectionString]", 89 | "tenant": "@[ActiveDirectoryTenantId]", 90 | "servicePrincipalId": "@[ServicePrincipalId]", 91 | "servicePrincipalKey": { 92 | "type": "AzureKeyVaultSecret", 93 | "store": { 94 | "referenceName": "@[KeyVault]", 95 | "type": "LinkedServiceReference" 96 | }, 97 | "secretName": { 98 | "value": "@linkedService().KeyVaultSecret", 99 | "type": "Expression" 100 | } 101 | } 102 | }, 103 | */ 104 | /* 105 | --------------------------------------------------------- 106 | Connection String - Explicitly Typed 107 | Authentication - SQL Authentication (stored in Key Vault) 108 | --------------------------------------------------------- 109 | "typeProperties": { 110 | "connectionString": "@[ConnectionString]", 111 | "password": { 112 | "type": "AzureKeyVaultSecret", 113 | "store": { 114 | "referenceName": "@[KeyVault]", 115 | "type": "LinkedServiceReference" 116 | }, 117 | "secretName": { 118 | "value": "@linkedService().KeyVaultSecret", 119 | "type": "Expression" 120 | } 121 | } 122 | }, 123 | */ 124 | } 125 | } 126 | ] 127 | } -------------------------------------------------------------------------------- /2024/building-a-framework-for-orchestration-in-azure-data-factory/Data Factory/2 - Datasets/Dataset Template - Azure SQL Database.jsonc: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", 3 | "contentVersion": "1.0.0.0", 4 | "resources": [ 5 | { 6 | "type": "Microsoft.DataFactory/factories/datasets", 7 | "name": "@[DataFactory]/@[Dataset]", 8 | "apiVersion": "2018-06-01", 9 | "dependsOn": [], 10 | "properties": { 11 | "annotations": [ 12 | "FrameworkVersion=1.0", 13 | "Type=Azure SQL Database" 14 | ], 15 | "description": "Generic dataset to an Azure SQL database, used when executing a query or stored proc.", 16 | "folder": { 17 | "name": "@[Folder]" 18 | }, 19 | "type": "AzureSqlTable", 20 | "linkedServiceName": { 21 | "referenceName": "@[LinkedService]", 22 | "type": "LinkedServiceReference" 23 | }, 24 | "schema": [] 25 | } 26 | } 27 | ] 28 | } -------------------------------------------------------------------------------- /2024/building-a-framework-for-orchestration-in-azure-data-factory/Data Factory/2 - Datasets/Dataset Template - Azure SQL Table.jsonc: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", 3 | "contentVersion": "1.0.0.0", 4 | "resources": [ 5 | { 6 | "type": "Microsoft.DataFactory/factories/datasets", 7 | "name": "@[DataFactory]/@[Dataset]", 8 | "apiVersion": "2018-06-01", 9 | "dependsOn": [], 10 | "properties": { 11 | "annotations": [ 12 | "FrameworkVersion=1.0", 13 | "Type=Azure SQL Table" 14 | ], 15 | "description": "Dataset specifically for an Azure SQL database table.", 16 | "folder": { 17 | "name": "@[Folder]" 18 | }, 19 | "type": "AzureSqlTable", 20 | "linkedServiceName": { 21 | "referenceName": "@[LinkedService]", 22 | "type": "LinkedServiceReference" 23 | }, 24 | "typeProperties": { 25 | "schema": { 26 | "value": "@dataset().SchemaName", 27 | "type": "Expression" 28 | }, 29 | "table": { 30 | "value": "@dataset().TableName", 31 | "type": "Expression" 32 | } 33 | }, 34 | "parameters": { 35 | "SchemaName": { 36 | "type": "string" 37 | }, 38 | "TableName": { 39 | "type": "string" 40 | } 41 | }, 42 | "schema": [] 43 | } 44 | } 45 | ] 46 | } -------------------------------------------------------------------------------- /2024/building-a-framework-for-orchestration-in-azure-data-factory/Data Factory/2 - Datasets/Dataset Template - Source - Azure SQL Database.jsonc: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", 3 | "contentVersion": "1.0.0.0", 4 | "resources": [ 5 | { 6 | "type": "Microsoft.DataFactory/factories/datasets", 7 | "name": "@[DataFactory]/@[Dataset]", 8 | "apiVersion": "2018-06-01", 9 | "dependsOn": [], 10 | "properties": { 11 | "annotations": [ 12 | "FrameworkVersion=1.0", 13 | "Type=Azure SQL Database" 14 | ], 15 | "description": "Generic dataset to an Azure SQL database, used when executing a query or stored proc.", 16 | "folder": { 17 | "name": "@[Folder]" 18 | }, 19 | "parameters": { 20 | "KeyVaultSecret": { 21 | "type": "string" 22 | } 23 | }, 24 | "type": "AzureSqlTable", 25 | "linkedServiceName": { 26 | "referenceName": "@[LinkedService]", 27 | "type": "LinkedServiceReference", 28 | "parameters": { 29 | "KeyVaultSecret": { 30 | "value": "@dataset().KeyVaultSecret", 31 | "type": "Expression" 32 | } 33 | } 34 | }, 35 | "schema": [] 36 | } 37 | } 38 | ] 39 | } -------------------------------------------------------------------------------- /2024/building-a-framework-for-orchestration-in-azure-data-factory/Data Factory/3 - Pipelines/Pipeline Template - Orchestrator - Get Tasks To Execute.jsonc: -------------------------------------------------------------------------------- 1 | { 2 | "type": "Microsoft.DataFactory/factories/pipelines", 3 | "name": "@[DataFactory]/Orchestrator - Get Tasks To Execute", 4 | "apiVersion": "2018-06-01", 5 | "dependsOn": [], 6 | "properties": { 7 | "folder": { 8 | "name": "Orchestrators" 9 | }, 10 | "annotations": [ 11 | "FrameworkVersion=1.0" 12 | ], 13 | "parameters": { 14 | "SubscriptionId": { 15 | "type": "string" 16 | }, 17 | "ResourceGroup": { 18 | "type": "string" 19 | }, 20 | "Environment": { 21 | "type": "string" 22 | }, 23 | "ProcessName": { 24 | "type": "string" 25 | }, 26 | "TaskName": { 27 | "type": "string" 28 | }, 29 | "ExecutionSequence": { 30 | "type": "int" 31 | }, 32 | "OrchestratorRunId": { 33 | "type": "string" 34 | } 35 | }, 36 | "description": "Gets all tasks to execute, based on the parameters provided. Each process will be executed according to its execution sequence number. Processes with the same execution sequence number will be processed in parallel.", 37 | "activities": [ 38 | { 39 | "name": "Get Tasks To Execute", 40 | "description": "Gets all the tasks to execute in parallel from the [ETL].[Task] table...based on the process, task and execution sequence number provided as parameters to this pipeline.", 41 | "type": "Lookup", 42 | "dependsOn": [], 43 | "policy": { 44 | "timeout": "0.00:10:00", 45 | "retry": 0, 46 | "retryIntervalInSeconds": 30, 47 | "secureOutput": false, 48 | "secureInput": false 49 | }, 50 | "userProperties": [], 51 | "typeProperties": { 52 | "source": { 53 | "type": "AzureSqlSource", 54 | "sqlReaderStoredProcedureName": "[[ETL].[GetTasksToExecute]", 55 | "storedProcedureParameters": { 56 | "Environment": { 57 | "type": "String", 58 | "value": { 59 | "value": "@pipeline().parameters.Environment", 60 | "type": "Expression" 61 | } 62 | }, 63 | "ProcessName": { 64 | "type": "String", 65 | "value": { 66 | "value": "@pipeline().parameters.ProcessName", 67 | "type": "Expression" 68 | } 69 | }, 70 | "TaskName": { 71 | "type": "String", 72 | "value": { 73 | "value": "@pipeline().parameters.TaskName", 74 | "type": "Expression" 75 | } 76 | }, 77 | "ExecutionSequence": { 78 | "type": "Int32", 79 | "value": { 80 | "value": "@pipeline().parameters.ExecutionSequence", 81 | "type": "Expression" 82 | } 83 | } 84 | }, 85 | "queryTimeout": "00:10:00", 86 | "partitionOption": "None" 87 | }, 88 | "dataset": { 89 | "referenceName": "@[Dataset]", 90 | "type": "DatasetReference" 91 | }, 92 | "firstRowOnly": false 93 | } 94 | }, 95 | { 96 | "name": "For Each Task", 97 | "type": "ForEach", 98 | "dependsOn": [ 99 | { 100 | "activity": "Get Tasks To Execute", 101 | "dependencyConditions": [ 102 | "Succeeded" 103 | ] 104 | } 105 | ], 106 | "userProperties": [], 107 | "typeProperties": { 108 | "items": { 109 | "value": "@activity('Get Tasks To Execute').output.value", 110 | "type": "Expression" 111 | }, 112 | "isSequential": false, 113 | "batchCount": 5, 114 | "activities": [ 115 | { 116 | "name": "Execute Task", 117 | "description": "Executes the pipeline that will facilitate the execution of the task.", 118 | "type": "ExecutePipeline", 119 | "dependsOn": [], 120 | "userProperties": [], 121 | "typeProperties": { 122 | "pipeline": { 123 | "referenceName": "Orchestrator - Execute Task", 124 | "type": "PipelineReference" 125 | }, 126 | "waitOnCompletion": true, 127 | "parameters": { 128 | "SubscriptionId": { 129 | "value": "@pipeline().parameters.SubscriptionId", 130 | "type": "Expression" 131 | }, 132 | "ResourceGroup": { 133 | "value": "@pipeline().parameters.ResourceGroup", 134 | "type": "Expression" 135 | }, 136 | "TaskKey": { 137 | "value": "@item().TaskKey", 138 | "type": "Expression" 139 | }, 140 | "DataFactoryPipeline": { 141 | "value": "@item().DataFactoryPipeline", 142 | "type": "Expression" 143 | }, 144 | "OrchestratorRunId": { 145 | "value": "@pipeline().parameters.OrchestratorRunId", 146 | "type": "Expression" 147 | } 148 | } 149 | } 150 | } 151 | ] 152 | } 153 | } 154 | ] 155 | } 156 | } -------------------------------------------------------------------------------- /2024/building-a-framework-for-orchestration-in-azure-data-factory/Data Factory/3 - Pipelines/Pipeline Template - Orchestrator - Main.jsonc: -------------------------------------------------------------------------------- 1 | { 2 | "type": "Microsoft.DataFactory/factories/pipelines", 3 | "name": "@[DataFactory]/Orchestrator - Main", 4 | "apiVersion": "2018-06-01", 5 | "dependsOn": [], 6 | "properties": { 7 | "folder": { 8 | "name": "Orchestrators" 9 | }, 10 | "annotations": [ 11 | "FrameworkVersion=1.0" 12 | ], 13 | "parameters": { 14 | "SubscriptionId": { 15 | "type": "string", 16 | "defaultValue": "@[SubscriptionId]" 17 | }, 18 | "ResourceGroup": { 19 | "type": "string", 20 | "defaultValue": "@[ResourceGroup]" 21 | }, 22 | "Environment": { 23 | "type": "string", 24 | "defaultValue": "Development" 25 | }, 26 | "ProcessName": { 27 | "type": "string", 28 | "defaultValue": "All" 29 | }, 30 | "TaskName": { 31 | "type": "string", 32 | "defaultValue": "All" 33 | } 34 | }, 35 | "description": "This pipeline is the entry-point for process and/or task execution. \n\nIt orchestrates the execution by first getting a list of sequence numbers to execute, and then executing those tasks in parallel.", 36 | "activities": [ 37 | { 38 | "name": "Get Execution Sequences", 39 | "description": "Gets a list of unique execution sequence numbers of the tasks to execute, based on the process and task provided as parameters to this pipeline.", 40 | "type": "Lookup", 41 | "dependsOn": [], 42 | "policy": { 43 | "timeout": "0.00:10:00", 44 | "retry": 0, 45 | "retryIntervalInSeconds": 30, 46 | "secureOutput": false, 47 | "secureInput": false 48 | }, 49 | "userProperties": [], 50 | "typeProperties": { 51 | "source": { 52 | "type": "AzureSqlSource", 53 | "sqlReaderStoredProcedureName": "[[ETL].[GetExecutionSequences]", 54 | "storedProcedureParameters": { 55 | "Environment": { 56 | "type": "String", 57 | "value": { 58 | "value": "@pipeline().parameters.Environment", 59 | "type": "Expression" 60 | } 61 | }, 62 | "ProcessName": { 63 | "type": "String", 64 | "value": { 65 | "value": "@pipeline().parameters.ProcessName", 66 | "type": "Expression" 67 | } 68 | }, 69 | "TaskName": { 70 | "type": "String", 71 | "value": { 72 | "value": "@pipeline().parameters.TaskName", 73 | "type": "Expression" 74 | } 75 | } 76 | }, 77 | "queryTimeout": "00:10:00", 78 | "partitionOption": "None" 79 | }, 80 | "dataset": { 81 | "referenceName": "@[Dataset]", 82 | "type": "DatasetReference" 83 | }, 84 | "firstRowOnly": false 85 | } 86 | }, 87 | { 88 | "name": "For Each Execution Sequence", 89 | "type": "ForEach", 90 | "dependsOn": [ 91 | { 92 | "activity": "Get Execution Sequences", 93 | "dependencyConditions": [ 94 | "Succeeded" 95 | ] 96 | } 97 | ], 98 | "userProperties": [], 99 | "typeProperties": { 100 | "items": { 101 | "value": "@activity('Get Execution Sequences').output.value", 102 | "type": "Expression" 103 | }, 104 | "isSequential": true, 105 | "activities": [ 106 | { 107 | "name": "Get Tasks to Execute", 108 | "description": "Executes the pipeline that will retrieve all tasks with this sequence number.", 109 | "type": "ExecutePipeline", 110 | "dependsOn": [], 111 | "userProperties": [], 112 | "typeProperties": { 113 | "pipeline": { 114 | "referenceName": "Orchestrator - Get Tasks to Execute", 115 | "type": "PipelineReference" 116 | }, 117 | "waitOnCompletion": true, 118 | "parameters": { 119 | "SubscriptionId": { 120 | "value": "@pipeline().parameters.SubscriptionId", 121 | "type": "Expression" 122 | }, 123 | "ResourceGroup": { 124 | "value": "@pipeline().parameters.ResourceGroup", 125 | "type": "Expression" 126 | }, 127 | "Environment": { 128 | "value": "@pipeline().parameters.Environment", 129 | "type": "Expression" 130 | }, 131 | "ProcessName": { 132 | "value": "@pipeline().parameters.ProcessName", 133 | "type": "Expression" 134 | }, 135 | "TaskName": { 136 | "value": "@pipeline().parameters.TaskName", 137 | "type": "Expression" 138 | }, 139 | "ExecutionSequence": { 140 | "value": "@item().ExecutionSequence", 141 | "type": "Expression" 142 | }, 143 | "OrchestratorRunId": { 144 | "value": "@pipeline().RunId", 145 | "type": "Expression" 146 | } 147 | } 148 | } 149 | } 150 | ] 151 | } 152 | } 153 | ] 154 | } 155 | } -------------------------------------------------------------------------------- /2024/building-a-framework-for-orchestration-in-azure-data-factory/Data Factory/3 - Pipelines/Pipeline Template - Worker - Copy Data - Azure SQL.jsonc: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", 3 | "contentVersion": "1.0.0.0", 4 | "resources": [ 5 | { 6 | "type": "Microsoft.DataFactory/factories/pipelines", 7 | "name": "@[DataFactory]/Copy Data - @[SourceSystem]", 8 | "apiVersion": "2018-06-01", 9 | "dependsOn": [], 10 | "properties": { 11 | "folder": { 12 | "name": "Workers" 13 | }, 14 | "annotations": [ 15 | "FrameworkVersion=1.0" 16 | ], 17 | "concurrency": 5, 18 | "parameters": { 19 | "TaskKey": { 20 | "type": "int" 21 | } 22 | }, 23 | "activities": [ 24 | { 25 | "name": "Get Task Queries", 26 | "description": "Gets all the source queries for the task we are executing, from the [ETL].[TaskQuery] table.", 27 | "type": "Lookup", 28 | "dependsOn": [], 29 | "policy": { 30 | "timeout": "0.00:10:00", 31 | "retry": 0, 32 | "retryIntervalInSeconds": 30, 33 | "secureOutput": false, 34 | "secureInput": false 35 | }, 36 | "userProperties": [], 37 | "typeProperties": { 38 | "source": { 39 | "type": "AzureSqlSource", 40 | "sqlReaderStoredProcedureName": "[[ETL].[GetTaskQueries]", 41 | "storedProcedureParameters": { 42 | "TaskKey": { 43 | "type": "Int32", 44 | "value": { 45 | "value": "@pipeline().parameters.TaskKey", 46 | "type": "Expression" 47 | } 48 | }, 49 | "TaskName": { 50 | "type": "String", 51 | "value": null 52 | } 53 | }, 54 | "queryTimeout": "00:10:00", 55 | "partitionOption": "None" 56 | }, 57 | "dataset": { 58 | "referenceName": "@[AdminDataset]", 59 | "type": "DatasetReference" 60 | }, 61 | "firstRowOnly": false 62 | } 63 | }, 64 | { 65 | "name": "Truncate Staging Table", 66 | "type": "Script", 67 | "dependsOn": [ 68 | { 69 | "activity": "Get Task Queries", 70 | "dependencyConditions": [ 71 | "Succeeded" 72 | ] 73 | } 74 | ], 75 | "policy": { 76 | "timeout": "0.00:10:00", 77 | "retry": 0, 78 | "retryIntervalInSeconds": 30, 79 | "secureOutput": false, 80 | "secureInput": false 81 | }, 82 | "userProperties": [], 83 | "linkedServiceName": { 84 | "referenceName": "@[StagingLinkedService]", 85 | "type": "LinkedServiceReference" 86 | }, 87 | "typeProperties": { 88 | "scripts": [ 89 | { 90 | "type": "NonQuery", 91 | "text": { 92 | "value": "truncate table [@{activity('Get Task Queries').output.value[0].TargetSchema}].[@{activity('Get Task Queries').output.value[0].TargetTable}];", 93 | "type": "Expression" 94 | } 95 | } 96 | ], 97 | "scriptBlockExecutionTimeout": "00:10:00" 98 | } 99 | }, 100 | { 101 | "name": "For Each Task Query", 102 | "type": "ForEach", 103 | "dependsOn": [ 104 | { 105 | "activity": "Truncate Staging Table", 106 | "dependencyConditions": [ 107 | "Succeeded" 108 | ] 109 | } 110 | ], 111 | "userProperties": [], 112 | "typeProperties": { 113 | "items": { 114 | "value": "@activity('Get Task Queries').output.value", 115 | "type": "Expression" 116 | }, 117 | "isSequential": true, 118 | "activities": [ 119 | { 120 | "name": "Load Staging Table", 121 | "description": "Load the source data into the staging table.", 122 | "type": "Copy", 123 | "dependsOn": [], 124 | "policy": { 125 | "timeout": "0.00:10:00", 126 | "retry": 0, 127 | "retryIntervalInSeconds": 30, 128 | "secureOutput": false, 129 | "secureInput": false 130 | }, 131 | "userProperties": [ 132 | { 133 | "name": "Source", 134 | "value": "@item().SourceQuery" 135 | }, 136 | { 137 | "name": "Target", 138 | "value": "@{item().TargetSchema}.@{item().TargetTable}" 139 | } 140 | ], 141 | "typeProperties": { 142 | "source": { 143 | "type": "AzureSqlSource", 144 | "sqlReaderQuery": { 145 | "value": "@item().SourceQuery", 146 | "type": "Expression" 147 | }, 148 | "queryTimeout": "00:30:00", 149 | "partitionOption": "None" 150 | }, 151 | "sink": { 152 | "type": "AzureSqlSink", 153 | "writeBatchSize": 50000, 154 | "writeBatchTimeout": "00:30:00", 155 | "writeBehavior": "insert", 156 | "sqlWriterUseTableLock": false, 157 | "disableMetricsCollection": false 158 | }, 159 | "enableStaging": false, 160 | "dataIntegrationUnits": 2, 161 | "translator": { 162 | "type": "Expression", 163 | "value": "@json(item().ColumnMapping)" 164 | } 165 | }, 166 | "inputs": [ 167 | { 168 | "referenceName": "@[SourceDataset]", 169 | "type": "DatasetReference", 170 | "parameters": { 171 | "KeyVaultSecret": "@item().SourceKeyVaultSecret" 172 | } 173 | } 174 | ], 175 | "outputs": [ 176 | { 177 | "referenceName": "@[StagingTableDataset]", 178 | "type": "DatasetReference", 179 | "parameters": { 180 | "SchemaName": { 181 | "value": "@item().TargetSchema", 182 | "type": "Expression" 183 | }, 184 | "TableName": { 185 | "value": "@item().TargetTable", 186 | "type": "Expression" 187 | } 188 | } 189 | } 190 | ] 191 | } 192 | ] 193 | } 194 | } 195 | ] 196 | } 197 | } 198 | ] 199 | } -------------------------------------------------------------------------------- /2024/building-a-framework-for-orchestration-in-azure-data-factory/Data Factory/Readme - Orchestrators.txt: -------------------------------------------------------------------------------- 1 | ---------------------------------------------------------------------------------------------------------------------------- 2 | **** Data Factory ARM Templates - Deployment Guide **** 3 | ---------------------------------------------------------------------------------------------------------------------------- 4 | 5 | The ARM templates in this repo can be deployed either via the Azure portal, or by using PowerShell. More information on how 6 | to do that can be found here: 7 | 8 | https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/quickstart-create-templates-use-the-portal 9 | 10 | https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/deploy-powershell 11 | 12 | 13 | *** PLEASE NOTE *** 14 | If an object (Linked Service, Dataset or Pipeline) with the same name exists, it may be replaced when you deploy any of these 15 | templates. 16 | 17 | 18 | To deploy the Orchestrator pipeline templates, you will need to have the following in place: 19 | 20 | 1. A Data Factory. 21 | 2. Integration Runtime(s) that can facilitate connectivity to your data source, target and metadata. 22 | 3. Metadata tables and stored procedures. 23 | 4. A Linked Service for Azure Key Vault, if you are planning to store connections strings there. 24 | * See section "Linked Service Template - Azure Key Vault.jsonc". 25 | 5. A Linked Service for the metadata database. 26 | * See section "Linked Service Template - Azure SQL Database.jsonc". 27 | 6. A generic Dataset for the metadata database, to execute queries and/or stored procedures. 28 | * See section "Dataset Template - Azure SQL Database.jsonc". 29 | 7. Appropriate permissions to use the Data Factory API. 30 | The Data Factory's managed identity needs the "Data Factory Contributor" role in the Data Factory resource in Azure. 31 | 32 | 33 | Because of dependencies the orchestrator pipelines have to be deployed in the order in which they appear in this document. 34 | Linked Services & Datasets can be skipped if already deployed. 35 | 36 | 37 | ---------------------------------------------------------------------------------------------------------------------------- 38 | * Linked Service Template - Azure Key Vault.jsonc * 39 | ---------------------------------------------------------------------------------------------------------------------------- 40 | 41 | This ARM template will deploy a Linked Service for Azure Key Vault, which you will need if you want to store the 42 | connection strings (or passwords) of other data sources in it. 43 | 44 | 45 | The following resources have to exist before you attempt to deploy the template: 46 | 47 | ** Data Factory 48 | ** Integration Runtime 49 | 50 | 51 | Find the following text and replace with your own values before deployment: 52 | 53 | @[DataFactory] - The name of your Data Factory resource in Azure. 54 | 55 | @[LinkedService] - The name of the Linked Service you are attempting to create. If it already exists, it 56 | may be replaced. 57 | 58 | @[Url] - URL to your Key Vault. It can be found in the overview section of the resource in the 59 | Azure portal. 60 | 61 | 62 | 63 | 64 | ---------------------------------------------------------------------------------------------------------------------------- 65 | * Linked Service Template - Azure SQL Database.jsonc * 66 | ---------------------------------------------------------------------------------------------------------------------------- 67 | 68 | This ARM template will deploy a Linked Service for an Azure SQL Database. The Linked Service can be used for the target 69 | and/or metadata databases. 70 | 71 | By default this template will use a connection string stored in Azure Key Vault, and will use the managed identity of the 72 | Data Factory to authenticate against the data source. A few other options have been provided in the ARM template and have 73 | been commented out. Replace the relevant section with your choice of authentication method before deployment, or skip this 74 | step and create the Linked Service manually. 75 | 76 | 77 | The following resources have to exist before you attempt to deploy the template: 78 | 79 | ** Data Factory 80 | ** Integration Runtime 81 | ** Azure Key Vault Linked Service (if you use the default authentication option) 82 | 83 | 84 | Find the following text and replace with your own values before deployment: 85 | 86 | @[DataFactory] - The name of your Data Factory resource in Azure. 87 | 88 | @[LinkedService] - The name of the Linked Service you are attempting to create. If it already exists, it 89 | may be replaced. 90 | 91 | @[IntegrationRuntime] - Name of the Integration Runtime (IR) that will be used by the Linked Service. 92 | 93 | @[KeyVault] - The name of the Linked Service in ADF that points to your Azure Key Vault, where connection 94 | details are stored. 95 | 96 | @[KeyVaultSecret] - Name of the secret in Azure Key Vault that contains the connection string (or password, 97 | if that authentication option is used). 98 | 99 | @[ActiveDirectoryTenantId] - Unique identifier (GUID) of the Active Directory (or Entra) tenant, if the Service Principal 100 | option is used for authentication. 101 | 102 | @[ServicePrincipalId] - Unique identifier (GUID) of the Service Principal, if that option is used for 103 | authentication. 104 | 105 | @[ServicePrincipalSecret] - Name of the Azure Key Vault secret that contains the Service Principal key, if that 106 | option is used for authentication. 107 | 108 | @[ConnectionString] - Connection string to the data source, if that option is used. 109 | 110 | 111 | 112 | 113 | ---------------------------------------------------------------------------------------------------------------------------- 114 | * Dataset Template - Azure SQL Database.jsonc * 115 | ---------------------------------------------------------------------------------------------------------------------------- 116 | 117 | This ARM template will deploy a generic Dataset associated with an Azure SQL Database, that can be used to execute stored 118 | procedures or queries. If you have separate Azure SQL Linked Services for both your metadata and target databases in Azure 119 | SQL, then you will need to deploy a Dataset for each. 120 | 121 | 122 | The following resources have to exist before you attempt to deploy the template: 123 | 124 | ** Data Factory 125 | ** Integration Runtime 126 | ** Linked Service 127 | 128 | 129 | Find the following text and replace with your own values before deployment: 130 | 131 | @[DataFactory] - The name of your Data Factory resource in Azure. 132 | 133 | @[Dataset] - The name of the Dataset you are attempting to create. If it already exists, it 134 | may be replaced. 135 | 136 | @[Folder] - The folder in which you want this Dataset to be placed. The folder will be created 137 | if it doesn't exist. 138 | 139 | @[LinkedService] - The name of the Linked Service this Dataset should be associated with. 140 | 141 | 142 | 143 | 144 | ---------------------------------------------------------------------------------------------------------------------------- 145 | * Pipeline Template - Orchestrator - Execute Task.jsonc * 146 | ---------------------------------------------------------------------------------------------------------------------------- 147 | 148 | This ARM template will deploy a orchestration pipeline that initiates a worker pipeline (via the API), and monitor the 149 | execution until there is a result. 150 | 151 | The naming convention for the pipeline will by default be "Orchestrator - Execute Task", and a folder named "Orchestrators" will 152 | be created if it doesn't exist. 153 | 154 | 155 | The following resources have to exist before you attempt to deploy the template: 156 | 157 | ** Data Factory 158 | ** Linked Service for the metadata database 159 | ** Generic Dataset for the metadata database, to execute queries or stored procedures 160 | ** Metadata tables and stored procedures 161 | 162 | 163 | Before deploying the template, you will need to find the following text and replace with your own values: 164 | 165 | @[DataFactory] - The name of your Data Factory resource in Azure. 166 | 167 | 168 | 169 | 170 | ---------------------------------------------------------------------------------------------------------------------------- 171 | * Pipeline Template - Orchestrator - Get Tasks to Execute.jsonc * 172 | ---------------------------------------------------------------------------------------------------------------------------- 173 | 174 | This ARM template will deploy a orchestration pipeline that gets all tasks with a certain sequence order, which will ultimately 175 | be executed in parallel. 176 | 177 | The naming convention for the pipeline will by default be "Orchestrator - Get Tasks to Execute", and a folder named "Orchestrators" will 178 | be created if it doesn't exist. 179 | 180 | 181 | The following resources have to exist before you attempt to deploy the template: 182 | 183 | ** Data Factory 184 | ** Linked Service for the metadata database 185 | ** Generic Dataset for the metadata database, to execute queries or stored procedures 186 | ** Metadata tables and stored procedures 187 | ** "Orchestrator - Execute Task" pipeline 188 | 189 | 190 | Before deploying the template, you will need to find the following text and replace with your own values: 191 | 192 | @[DataFactory] - The name of your Data Factory resource in Azure. 193 | 194 | @[Dataset] - The name of the Dataset that points to the database where the metadata is stored. This 195 | Dataset will be used to execute stored procedures to extract the required metadata. 196 | 197 | 198 | 199 | 200 | ---------------------------------------------------------------------------------------------------------------------------- 201 | * Pipeline Template - Orchestrator - Main.jsonc * 202 | ---------------------------------------------------------------------------------------------------------------------------- 203 | 204 | This ARM template will deploy a orchestration pipeline that will be the main entry point for all executions. 205 | 206 | The naming convention for the pipeline will by default be "Orchestrator - Main", and a folder named "Orchestrators" will 207 | be created if it doesn't exist. 208 | 209 | 210 | The following resources have to exist before you attempt to deploy the template: 211 | 212 | ** Data Factory 213 | ** Linked Service for the metadata database 214 | ** Generic Dataset for the metadata database, to execute queries or stored procedures 215 | ** Metadata tables and stored procedures 216 | ** "Orchestrator - Get Tasks to Execute" pipeline 217 | 218 | 219 | Before deploying the template, you will need to find the following text and replace with your own values: 220 | 221 | @[DataFactory] - The name of your Data Factory resource in Azure. 222 | 223 | @[SubscriptionId] - Unique ID (GUID) of the Azure subscription in which our Data Factory resource 224 | was created. 225 | 226 | @[ResourceGroup] - Name of the Resource Group in which your Data Factory was created. 227 | 228 | @[Dataset] - The name of the Dataset that points to the database where the metadata is stored. This 229 | Dataset will be used to execute stored procedures to extract the required metadata. 230 | -------------------------------------------------------------------------------- /2024/building-a-framework-for-orchestration-in-azure-data-factory/Data Factory/Readme - Workers.txt: -------------------------------------------------------------------------------- 1 | ---------------------------------------------------------------------------------------------------------------------------- 2 | **** Data Factory ARM Templates - Deployment Guide **** 3 | ---------------------------------------------------------------------------------------------------------------------------- 4 | 5 | The ARM templates in this repo can be deployed either via the Azure portal, or by using PowerShell. More information on how 6 | to do that can be found here: 7 | 8 | https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/quickstart-create-templates-use-the-portal 9 | 10 | https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/deploy-powershell 11 | 12 | 13 | *** PLEASE NOTE *** 14 | If an object (Linked Service, Dataset or Pipeline) with the same name exists, it may be replaced when you deploy any of these 15 | templates. 16 | 17 | 18 | To deploy the Worker pipeline template, you will need to have the following in place: 19 | 20 | 1. A Data Factory. 21 | 2. Integration Runtime(s) that can facilitate connectivity to your data source, target and metadata. 22 | 3. Metadata tables and stored procedures. 23 | 4. A Linked Service for Azure Key Vault, if you are planning to store connections strings there. 24 | * See section "Linked Service Template - Azure Key Vault.jsonc". 25 | 5. A Linked Service for the metadata database. 26 | * See section "Linked Service Template - Azure SQL Database.jsonc". 27 | 6. A Linked Service for the target. 28 | * The only template provided here is for an Azure SQL Database as target. 29 | * See section "Linked Service Template - Azure SQL Database.jsonc". 30 | 7. A Linked Service for the source. 31 | * The only template provided here is for an Azure SQL source. 32 | * See section "Linked Service Template - Source - Azure SQL Database.jsonc". 33 | 8. A generic Dataset for the metadata database, to execute queries and/or stored procedures. 34 | * See section "Dataset Template - Azure SQL Database.jsonc". 35 | 9. Two Datasets for the target (staging) database. 36 | * Generic Dataset to execute queries and/or stored procedures (see section "Dataset Template - Azure SQL Database.jsonc"). 37 | * Parameterized Dataset to use as sink (target) in Copy activities (see section "Dataset Template - Azure SQL Table.jsonc"). 38 | 10. A Dataset for the source. 39 | * See section "Dataset Template - Source - Azure SQL Database.jsonc" for details. 40 | 41 | 42 | See section "Pipeline Template - Worker - Copy Data - Azure SQL.jsonc" for the details of the worker pipeline. 43 | 44 | 45 | ---------------------------------------------------------------------------------------------------------------------------- 46 | * Linked Service Template - Azure Key Vault.jsonc * 47 | ---------------------------------------------------------------------------------------------------------------------------- 48 | 49 | This ARM template will deploy a Linked Service for Azure Key Vault, which you will need if you want to store the 50 | connection strings (or passwords) of other data sources in it. 51 | 52 | 53 | The following resources have to exist before you attempt to deploy the template: 54 | 55 | ** Data Factory 56 | ** Integration Runtime 57 | 58 | 59 | Find the following text and replace with your own values before deployment: 60 | 61 | @[DataFactory] - The name of your Data Factory resource in Azure. 62 | 63 | @[LinkedService] - The name of the Linked Service you are attempting to create. If it already exists, it 64 | may be replaced. 65 | 66 | @[Url] - URL to your Key Vault. It can be found in the overview section of the resource in the 67 | Azure portal. 68 | 69 | 70 | 71 | 72 | ---------------------------------------------------------------------------------------------------------------------------- 73 | * Linked Service Template - Azure SQL Database.jsonc * 74 | ---------------------------------------------------------------------------------------------------------------------------- 75 | 76 | This ARM template will deploy a Linked Service for an Azure SQL Database. The Linked Service can be used for the target 77 | and/or metadata databases. 78 | 79 | By default this template will use a connection string stored in Azure Key Vault, and will use the managed identity of the 80 | Data Factory to authenticate against the data source. A few other options have been provided in the ARM template and have 81 | been commented out. Replace the relevant section with your choice of authentication method before deployment, or skip this 82 | step and create the Linked Service manually. 83 | 84 | 85 | The following resources have to exist before you attempt to deploy the template: 86 | 87 | ** Data Factory 88 | ** Integration Runtime 89 | ** Azure Key Vault Linked Service (if you use the default authentication option) 90 | 91 | 92 | Find the following text and replace with your own values before deployment: 93 | 94 | @[DataFactory] - The name of your Data Factory resource in Azure. 95 | 96 | @[LinkedService] - The name of the Linked Service you are attempting to create. If it already exists, it 97 | may be replaced. 98 | 99 | @[IntegrationRuntime] - Name of the Integration Runtime (IR) that will be used by the Linked Service. 100 | 101 | @[KeyVault] - The name of the Linked Service in ADF that points to your Azure Key Vault, where connection 102 | details are stored. 103 | 104 | @[KeyVaultSecret] - Name of the secret in Azure Key Vault that contains the connection string (or password, 105 | if that authentication option is used). 106 | 107 | @[ActiveDirectoryTenantId] - Unique identifier (GUID) of the Active Directory (or Entra) tenant, if the Service Principal 108 | option is used for authentication. 109 | 110 | @[ServicePrincipalId] - Unique identifier (GUID) of the Service Principal, if that option is used for 111 | authentication. 112 | 113 | @[ServicePrincipalSecret] - Name of the Azure Key Vault secret that contains the Service Principal key, if that 114 | option is used for authentication. 115 | 116 | @[ConnectionString] - Connection string to the data source, if that option is used. 117 | 118 | 119 | 120 | 121 | ---------------------------------------------------------------------------------------------------------------------------- 122 | * Linked Service Template - Source - Azure SQL Database.jsonc * 123 | ---------------------------------------------------------------------------------------------------------------------------- 124 | 125 | This ARM template will deploy a Linked Service for an Azure SQL Database and can be deployed if you are planning to use 126 | Azure SQL as a source. What makes it different from the other Azure SQL Linked Service is that the connection string is 127 | parameterized and assumed to be stored in a Key Vault secret. 128 | 129 | In order for this framework to function, you will need Linked Services for all sources and destinations. 130 | 131 | 132 | The following resources have to exist before you attempt to deploy the template: 133 | 134 | ** Data Factory 135 | ** Integration Runtime 136 | ** Azure Key Vault Linked Service (if you use the default authentication option) 137 | 138 | 139 | Find the following text and replace with your own values before deployment: 140 | 141 | @[DataFactory] - The name of your Data Factory resource in Azure. 142 | 143 | @[LinkedService] - The name of the Linked Service you are attempting to create. If it already exists, it 144 | may be replaced. 145 | 146 | @[IntegrationRuntime] - Name of the Integration Runtime (IR) that will be used by the Linked Service. 147 | 148 | @[KeyVault] - The name of the Linked Service in ADF that points to your Key Vault, where connection 149 | details are stored. 150 | 151 | @[ActiveDirectoryTenantId] - Unique identifier (GUID) of the Active Directory (or Entra) tenant, if the Service Principal 152 | option is used for authentication. 153 | 154 | @[ServicePrincipalId] - Unique identifier (GUID) of the Service Principal, if that option is used for 155 | authentication. 156 | 157 | @[ServicePrincipalSecret] - Name of the Azure Key Vault secret that contains the Service Principal key, if that 158 | option is used for authentication. 159 | 160 | @[ConnectionString] - Connection string to the data source, if that option is used. 161 | 162 | 163 | 164 | 165 | ---------------------------------------------------------------------------------------------------------------------------- 166 | * Dataset Template - Azure SQL Database.jsonc * 167 | ---------------------------------------------------------------------------------------------------------------------------- 168 | 169 | This ARM template will deploy a generic Dataset associated with an Azure SQL Database, that can be used to execute stored 170 | procedures or queries. If you have separate Azure SQL Linked Services for both your metadata and target databases in Azure 171 | SQL, then you will need to deploy a Dataset for each. 172 | 173 | 174 | The following resources have to exist before you attempt to deploy the template: 175 | 176 | ** Data Factory 177 | ** Integration Runtime 178 | ** Linked Service 179 | 180 | 181 | Find the following text and replace with your own values before deployment: 182 | 183 | @[DataFactory] - The name of your Data Factory resource in Azure. 184 | 185 | @[Dataset] - The name of the Dataset you are attempting to create. If it already exists, it 186 | may be replaced. 187 | 188 | @[Folder] - The folder in which you want this Dataset to be placed. The folder will be created 189 | if it doesn't exist. 190 | 191 | @[LinkedService] - The name of the Linked Service this Dataset should be associated with. 192 | 193 | 194 | 195 | 196 | ---------------------------------------------------------------------------------------------------------------------------- 197 | * Dataset Template - Azure SQL Table.jsonc * 198 | ---------------------------------------------------------------------------------------------------------------------------- 199 | 200 | This ARM template will deploy a Dataset associated with an Azure SQL table you want to use as a target. It contains parameters 201 | for the schema and table, and typically used as sink (target) in Copy activities. 202 | 203 | 204 | The following resources have to exist before you attempt to deploy the template: 205 | 206 | ** Data Factory 207 | ** Integration Runtime 208 | ** Linked Service 209 | 210 | 211 | Before deploying the template, you will need to find the following text and replace with your own values: 212 | 213 | @[DataFactory] - The name of your Data Factory resource in Azure. 214 | 215 | @[Dataset] - The name of the Dataset you are attempting to create. If it already exists, it 216 | may be replaced. 217 | 218 | @[Folder] - The folder in which you want this Dataset to be placed. The folder will be created 219 | if it doesn't exist. 220 | 221 | @[LinkedService] - The name of the Linked Service this Dataset should be associated with. 222 | 223 | 224 | 225 | 226 | ---------------------------------------------------------------------------------------------------------------------------- 227 | * Dataset Template - Source - Azure SQL Database.jsonc * 228 | ---------------------------------------------------------------------------------------------------------------------------- 229 | 230 | This ARM template will deploy a Dataset associated with an Azure SQL Database, and can be deployed if you are planning to use 231 | Azure SQL as a source. What makes it different from the other Azure SQL Database Dataset is that it accounts for the fact that 232 | the connection string of the associated Linked Service will be parameterized (and assumed to be stored in a Key Vault secret). 233 | 234 | 235 | The following resources have to exist before you attempt to deploy the template: 236 | 237 | ** Data Factory 238 | ** Integration Runtime 239 | ** Linked Service 240 | 241 | 242 | Before deploying the template, you will need to find the following text and replace with your own values: 243 | 244 | @[DataFactory] - The name of your Data Factory resource in Azure. 245 | 246 | @[Dataset] - The name of the Dataset you are attempting to create. If it already exists, it 247 | may be replaced. 248 | 249 | @[Folder] - The folder in which you want this Dataset to be placed. The folder will be created 250 | if it doesn't exist. 251 | 252 | @[LinkedService] - The name of the Linked Service this Dataset should be associated with. 253 | 254 | 255 | 256 | 257 | ---------------------------------------------------------------------------------------------------------------------------- 258 | * Pipeline Template - Worker - Copy Data - Azure SQL.jsonc * 259 | ---------------------------------------------------------------------------------------------------------------------------- 260 | 261 | This ARM template will deploy a worker pipeline that extracts data from an Azure SQL database as source, and populate a 262 | staging table also hosted in Azure SQL. 263 | 264 | The naming convention for the pipeline will by default be "Copy Data - ", and a folder named "Workers" will 265 | be created if it doesn't exist. 266 | 267 | 268 | The following resources have to exist before you attempt to deploy the template: 269 | 270 | ** Data Factory 271 | ** Linked Services for all sources (including the metadata database) and targets 272 | ** Datasets for all sources (including the metadata database) and targets 273 | 274 | 275 | Before deploying the template, you will need to find the following text and replace with your own values: 276 | 277 | @[DataFactory] - The name of your Data Factory resource in Azure. 278 | 279 | @[SourceSystem] - Name of the source system, which will be used in the name of the deployed pipeline. 280 | 281 | @[SourceDataset] - Name of the Dataset created for the source system. 282 | 283 | @[AdminDataset] - The name of the Dataset that points to the database where the metadata is stored. This 284 | Dataset will be used to execute stored procedures to extract the required metadata. 285 | 286 | @[StagingLinkedService] - The name of the Linked Service associated with the staging database (or target). 287 | 288 | @[StagingTableDataset] - Name of the Dataset created for staging tables. This Dataset should have parameters for 289 | the schema & table, and will be used as the Copy activity's sink. 290 | 291 | 292 | -------------------------------------------------------------------------------- /2024/building-a-framework-for-orchestration-in-azure-data-factory/Metadata/GetExecutionSequences.sql: -------------------------------------------------------------------------------- 1 | /************************************************************************************ 2 | Created By : Martin Schoombee 3 | Company : 28twelve consulting 4 | Date : 7/17/2023 5 | Summary : This script will be called by an Azure Data Factory pipeline, and 6 | return a unique list of execution sequences that will be used to 7 | facilitate parallel or serial execution of ETL tasks. 8 | 9 | Note that only execution sequences of "active" processes and tasks 10 | will be returned. 11 | 12 | Parameters: 13 | @Environment - Beta/Production 14 | @ProcessName - Name of the process to execute, if you want to 15 | execute an entire process. Use 'All' (or omit) to 16 | execute all processes or process a single task. 17 | @TaskName - Name of the task to execute, if a single task 18 | should be executed. Use 'All' (or omit) if an 19 | entire process should be executed. 20 | 21 | 22 | ************************************************************************************* 23 | -- CHANGE CONTROL -- 24 | ************************************************************************************* 25 | DevOps Reference : 26 | CO Number : 27 | Changed By : 28 | Date : 29 | Details : 30 | 31 | *************************************************************************************/ 32 | create or alter procedure [ETL].[GetExecutionSequences] 33 | @Environment varchar(20) = 'Development' 34 | , @ProcessName varchar(50) = 'All' 35 | , @TaskName varchar(50) = 'All' 36 | as 37 | 38 | if @ProcessName = 'All' and @TaskName = 'All' 39 | 40 | throw 50000, 'Cannot initiate all processes and tasks at the same time', 1; 41 | 42 | else 43 | ---------------------------------------------------------------------------------- 44 | -- get unique list of execution sequences to execute 45 | 46 | select distinct 47 | convert 48 | ( 49 | int 50 | , (prc.[ProcessExecutionSequence] * 10000) + tsk.[TaskExecutionSequence] 51 | ) as [ExecutionSequence] 52 | 53 | from [ETL].[Process] prc 54 | inner join [ETL].[Task] tsk on tsk.[ProcessKey] = prc.[ProcessKey] 55 | 56 | where prc.[Environment] = @Environment 57 | and prc.[ExecuteProcess] = convert(bit, 1) 58 | and tsk.[ExecuteTask] = convert(bit, 1) 59 | and 60 | ( 61 | @ProcessName = 'All' 62 | or prc.[ProcessName] = @ProcessName 63 | ) 64 | and 65 | ( 66 | @TaskName = 'All' 67 | or tsk.[TaskName] = @TaskName 68 | ) 69 | 70 | order by [ExecutionSequence] 71 | ; -------------------------------------------------------------------------------- /2024/building-a-framework-for-orchestration-in-azure-data-factory/Metadata/GetTaskQueries.sql: -------------------------------------------------------------------------------- 1 | /************************************************************************************ 2 | Created By : Martin Schoombee 3 | Company : 28twelve consulting 4 | Date : 4/13/2024 5 | Summary : This procedure returns all the queries of a task. 6 | 7 | Placeholders for dynamic values are denoted by @[...] and logic has to 8 | be added for each parameter that needs to be replaced. 9 | 10 | 11 | ************************************************************************************* 12 | -- CHANGE CONTROL -- 13 | ************************************************************************************* 14 | DevOps Reference : 15 | Changed By : 16 | Date : 17 | Details : 18 | 19 | *************************************************************************************/ 20 | create or alter procedure [ETL].[GetTaskQueries] 21 | @TaskKey int = -9 22 | , @TaskName varchar(50) = 'Unknown' 23 | as 24 | 25 | ---------------------------------------------------------------------------------------------------- 26 | -- get the task key if the name was given as parameter 27 | if @TaskKey = -9 28 | begin 29 | set @TaskKey = (select [TaskKey] from [ETL].[Task] where [TaskName] = @TaskName) 30 | end 31 | 32 | 33 | ---------------------------------------------------------------------------------------------------- 34 | -- return all task queries 35 | select qry.[TaskQueryKey] 36 | , qry.[SourceKeyVaultSecret] 37 | -- example of dynamic value injection 38 | , replace 39 | ( 40 | qry.[SourceQuery] 41 | , '@[LoadStartDate]' 42 | , convert(varchar(200), isnull(LoadStartDate.[QueryText], '')) 43 | ) as SourceQuery 44 | , tsk.[TargetSchema] 45 | , tsk.[TargetTable] 46 | , tsk.[ColumnMapping] 47 | 48 | from [ETL].[Task] tsk 49 | inner join [ETL].[TaskQuery] qry on qry.[TaskKey] = tsk.[TaskKey] 50 | -- return parameter values to inject in dynamically into source query 51 | outer apply ( 52 | select top 1 53 | 'dateadd(day, -' + prm.[ParameterValue] + ', convert(date, getdate()))' as [QueryText] 54 | 55 | from [ETL].[Parameter] prm 56 | 57 | where prm.[ParameterName] = 'DaysToLoad' 58 | ) LoadStartDate 59 | 60 | where qry.[TaskKey] = @TaskKey 61 | and qry.[ExecuteQuery] = convert(bit, 1) 62 | 63 | order by qry.[TaskQueryKey] 64 | ; -------------------------------------------------------------------------------- /2024/building-a-framework-for-orchestration-in-azure-data-factory/Metadata/GetTasksToExecute.sql: -------------------------------------------------------------------------------- 1 | /************************************************************************************ 2 | Created By : Martin Schoombee 3 | Company : 28twelve consulting 4 | Date : 7/17/2023 5 | Summary : This script will be called by an Azure Data Factory pipeline, and the 6 | results used to execute all tasks with the same sequence number in 7 | parallel. 8 | 9 | Note that only "active" (ExecuteTask = 1) processes and tasks with be 10 | returned in the results. 11 | 12 | Parameters: 13 | @Environment - Beta/Production/etc. 14 | @ProcessName - Name of the process to execute, if you want to 15 | execute an entire process. Use 'All' (default) 16 | to execute all processes, or to process a 17 | single task. 18 | @TaskName - Name of the task to execute, if a single 19 | task should be executed. Use 'All' (or omit) 20 | if an entire process should be executed. 21 | @ExecutionSequence - Sequence number of the tasks to execute. Use 22 | -9 (or omit) to return all tasks regardless 23 | of the sequence number. 24 | 25 | 26 | ************************************************************************************* 27 | -- CHANGE CONTROL -- 28 | ************************************************************************************* 29 | DevOps Reference : 30 | CO Number : 31 | Changed By : 32 | Date : 33 | Details : 34 | 35 | *************************************************************************************/ 36 | create or alter procedure [ETL].[GetTasksToExecute] 37 | @Environment varchar(20) = 'Development' 38 | , @ProcessName varchar(50) = 'All' 39 | , @TaskName varchar(50) = 'All' 40 | , @ExecutionSequence int = -9 41 | as 42 | 43 | if @ProcessName = 'All' and @TaskName = 'All' 44 | 45 | throw 50000, 'Cannot initiate all processes and tasks at the same time', 1; 46 | 47 | else 48 | ---------------------------------------------------------------------------------- 49 | -- get all processes and tasks to execute 50 | 51 | select distinct -- in case there are duplicates, we don't want to execute a process/task twice 52 | prc.[ProcessKey] 53 | , prc.[Environment] 54 | , prc.[ProcessName] 55 | , tsk.[TaskKey] 56 | , tsk.[TaskName] 57 | , tsk.[TargetSchema] 58 | , tsk.[TargetTable] 59 | , tsk.[ColumnMapping] 60 | , tsk.[DataFactoryPipeline] 61 | , ((prc.[ProcessExecutionSequence] * 10000) + tsk.[TaskExecutionSequence]) as [ExecutionSequence] 62 | 63 | from [ETL].[Process] prc 64 | inner join [ETL].[Task] tsk on tsk.[ProcessKey] = prc.[ProcessKey] 65 | 66 | where prc.[Environment] = @Environment 67 | and prc.[ExecuteProcess] = convert(bit, 1) 68 | and tsk.[ExecuteTask] = convert(bit, 1) 69 | and 70 | ( 71 | @ProcessName = 'All' 72 | or prc.[ProcessName] = @ProcessName 73 | ) 74 | and 75 | ( 76 | @TaskName = 'All' 77 | or tsk.[TaskName] = @TaskName 78 | ) 79 | and 80 | ( 81 | @ExecutionSequence = -9 82 | or ((prc.[ProcessExecutionSequence] * 10000) + tsk.[TaskExecutionSequence]) = @ExecutionSequence 83 | ) 84 | 85 | order by [ExecutionSequence] 86 | , prc.[ProcessName] 87 | , tsk.[TaskName] 88 | ; -------------------------------------------------------------------------------- /2024/building-a-framework-for-orchestration-in-azure-data-factory/Metadata/Metadata Tables.sql: -------------------------------------------------------------------------------- 1 | -- etl.process 2 | CREATE TABLE [ETL].[Process]( 3 | [ProcessKey] [int] IDENTITY(1,1) NOT NULL, 4 | [Environment] [varchar](20) NOT NULL, 5 | [ProcessName] [varchar](50) NOT NULL, 6 | [ProcessDescription] [varchar](200) NOT NULL DEFAULT ('Unknown'), 7 | [ProcessExecutionSequence] [smallint] NOT NULL DEFAULT (CONVERT([smallint],(1))), 8 | [ExecuteProcess] [bit] NOT NULL DEFAULT (CONVERT([bit],(0))), 9 | [InsertDate] [datetime] NOT NULL DEFAULT (getdate()), 10 | [UpdateDate] [datetime] NULL, 11 | CONSTRAINT [PK_Process] PRIMARY KEY CLUSTERED ([ProcessKey]) 12 | ) ON [PRIMARY] 13 | 14 | 15 | -- etl.task 16 | CREATE TABLE [ETL].[Task]( 17 | [TaskKey] [int] IDENTITY(1,1) NOT NULL, 18 | [ProcessKey] [int] NOT NULL, 19 | [TaskName] [varchar](50) NOT NULL, 20 | [TaskDescription] [varchar](200) NOT NULL DEFAULT ('Unknown'), 21 | [TargetSchema] [varchar](50) NOT NULL DEFAULT ('Unknown'), 22 | [TargetTable] [varchar](50) NOT NULL DEFAULT ('Unknown'), 23 | [ColumnMapping] [varchar](MAX) NOT NULL DEFAULT ('{}'), 24 | [DataFactoryPipeline] [varchar](200) NOT NULL DEFAULT ('Unknown'), 25 | [TaskExecutionSequence] [smallint] NOT NULL DEFAULT (CONVERT([smallint],(1))), 26 | [ExecuteTask] [bit] NOT NULL DEFAULT (CONVERT([bit],(0))), 27 | [InsertDate] [datetime] NOT NULL DEFAULT (getdate()), 28 | [UpdateDate] [datetime] NULL, 29 | CONSTRAINT [PK_Task] PRIMARY KEY CLUSTERED ([TaskKey]), 30 | CONSTRAINT [FK_Task_Process] FOREIGN KEY ([ProcessKey]) REFERENCES [ETL].[Process]([ProcessKey]) 31 | ) ON [PRIMARY] 32 | 33 | 34 | -- etl.taskquery 35 | CREATE TABLE [ETL].[TaskQuery]( 36 | [TaskQueryKey] [int] IDENTITY(1,1) NOT NULL, 37 | [TaskKey] [int] NOT NULL, 38 | [SourceKeyVaultSecret] [varchar](200) NOT NULL DEFAULT ('Unknown'), 39 | [SourceQuery] [varchar](MAX) NOT NULL DEFAULT ('Unknown'), 40 | [ExecuteQuery] [bit] NOT NULL DEFAULT (CONVERT([bit],(0))), 41 | [InsertDate] [datetime] NOT NULL DEFAULT (getdate()), 42 | [UpdateDate] [datetime] NULL, 43 | CONSTRAINT [PK_TaskQuery] PRIMARY KEY CLUSTERED ([TaskQueryKey]), 44 | CONSTRAINT [FK_TaskQuery_Task] FOREIGN KEY ([TaskKey]) REFERENCES [ETL].[Task]([TaskKey]) 45 | ) ON [PRIMARY] 46 | 47 | 48 | -- etl.parameter table 49 | CREATE TABLE [ETL].[Parameter]( 50 | [ParameterKey] [int] IDENTITY(1,1) NOT NULL, 51 | [ParameterName] [varchar](50) NOT NULL, 52 | [ParameterDescription] [varchar](500) NOT NULL DEFAULT ('Unknown'), 53 | [ParameterValue] [varchar](200) NOT NULL, 54 | [InsertDate] [datetime] NOT NULL DEFAULT (getdate()), 55 | [UpdateDate] [datetime] NULL, 56 | CONSTRAINT [PK_Parameter] PRIMARY KEY CLUSTERED ([ParameterKey]) 57 | ) ON [PRIMARY] -------------------------------------------------------------------------------- /readme: -------------------------------------------------------------------------------- 1 | ---------------------------------------------------------------------------------------------------------------------------- 2 | ** 2024 Blog Posts ** 3 | ---------------------------------------------------------------------------------------------------------------------------- 4 | Building a framework for orchestration in Azure Data Factory: Recap 5 | https://martinschoombee.com/2024/05/28/building-a-framework-for-orchestration-in-azure-data-factory-recap/ 6 | 7 | Building a framework for orchestration in Azure Data Factory: Controllers 8 | https://martinschoombee.com/2024/05/21/building-a-framework-for-orchestration-in-azure-data-factory-controllers/ 9 | 10 | Building a framework for orchestration in Azure Data Factory: Orchestrators 11 | https://martinschoombee.com/2024/05/14/building-a-framework-for-orchestration-in-azure-data-factory-orchestrators/ 12 | 13 | Building a framework for orchestration in Azure Data Factory: Workers 14 | https://martinschoombee.com/2024/05/07/building-a-framework-for-orchestration-in-azure-data-factory-workers/ 15 | 16 | Building a framework for orchestration in Azure Data Factory: Framework components 17 | https://martinschoombee.com/2024/04/23/building-a-framework-for-orchestration-in-azure-data-factory-framework-components/ 18 | 19 | Building a framework for orchestration in Azure Data Factory: Metadata 20 | https://martinschoombee.com/2024/04/16/building-a-framework-for-orchestration-in-azure-data-factory-metadata/ 21 | 22 | Building a framework for orchestration in Azure Data Factory: Orchestration 23 | https://martinschoombee.com/2024/04/09/building-a-framework-for-orchestration-in-azure-data-factory-orchestration/ 24 | 25 | Building a framework for orchestration in Azure Data Factory: A series 26 | https://martinschoombee.com/2024/04/02/building-a-framework-for-orchestration-in-azure-data-factory-a-series/ 27 | 28 | 29 | ---------------------------------------------------------------------------------------------------------------------------- 30 | ** 2022 Blog Posts ** 31 | ---------------------------------------------------------------------------------------------------------------------------- 32 | Working with OAuth 2.0 APIs in Azure Data Factory: Refreshing tokens 33 | https://martinschoombee.com/2022/03/08/working-with-oauth-2-0-apis-in-azure-data-factory-refreshing-tokens/ 34 | 35 | Working with OAuth 2.0 APIs in Azure Data Factory: The ADF linked service and dataset 36 | https://martinschoombee.com/2022/03/01/working-with-oauth-2-0-apis-in-azure-data-factory-the-adf-linked-service-and-dataset/ 37 | 38 | Working with OAuth 2.0 APIs in Azure Data Factory: Using Postman to get tokens and test API requests 39 | https://martinschoombee.com/2022/02/22/working-with-oauth-2-0-apis-in-azure-data-factory-using-postman-to-get-tokens-and-test-api-requests/ 40 | 41 | Working with OAuth 2.0 APIs in Azure Data Factory: The authorization flow 42 | https://martinschoombee.com/2022/02/15/working-with-oauth-2-0-apis-in-azure-data-factory-the-authorization-flow/ 43 | 44 | Working with OAuth 2.0 APIs in Azure Data Factory: A series 45 | https://martinschoombee.com/2022/02/15/working-with-oauth-2-0-apis-in-azure-data-factory-a-series/ 46 | 47 | Creating development environments in Azure, the easy way 48 | https://martinschoombee.com/2022/02/08/creating-development-environments-in-azure-the-easy-way/ 49 | 50 | 51 | ---------------------------------------------------------------------------------------------------------------------------- 52 | ** 2021 Blog Posts ** 53 | ---------------------------------------------------------------------------------------------------------------------------- 54 | ETL foundations: Staging your data 55 | https://martinschoombee.com/2021/03/30/etl-foundations-staging-your-data/ 56 | 57 | Owning my own business: Lessons learned from year three 58 | https://martinschoombee.com/2021/03/16/owning-my-own-business-lessons-learned-from-year-three/ 59 | 60 | Should I move to Power BI Premium Per User? 61 | https://martinschoombee.com/2021/03/09/should-i-move-to-power-bi-premium-per-user/ 62 | 63 | Azure AD: Assign administrator roles with PowerShell 64 | https://martinschoombee.com/2021/03/02/azure-ad-assign-administrator-roles-with-powershell/ 65 | 66 | Using calculation groups with role-playing dimensions 67 | https://martinschoombee.com/2021/02/09/using-calculation-groups-with-role-playing-dimensions/ 68 | 69 | Fixing incorrect drill-through results in Analysis Services - Part 2 70 | https://martinschoombee.com/2021/02/02/fixing-incorrect-drill-through-results-in-analysis-services-part-2/ 71 | 72 | Fixing incorrect drill-through results in Analysis Services - Part 1 73 | https://martinschoombee.com/2021/01/26/fixing-incorrect-drill-through-results-in-analysis-services-part-1/ 74 | 75 | 76 | ---------------------------------------------------------------------------------------------------------------------------- 77 | ** 2020 Blog Posts ** 78 | ---------------------------------------------------------------------------------------------------------------------------- 79 | The value of durable keys in type-2 dimensions 80 | https://martinschoombee.com/2020/12/01/the-value-of-durable-keys-in-type-2-dimensions/ 81 | 82 | Take your life back, one hour at a time 83 | https://martinschoombee.com/2020/11/17/take-your-life-back-one-hour-at-a-time/ 84 | 85 | Automating Power BI deployments: Assign workspace permissions 86 | https://martinschoombee.com/2020/11/10/automating-power-bi-deployments-assign-workspace-permissions/ 87 | 88 | Automating Power BI deployments: Trigger a refresh 89 | https://martinschoombee.com/2020/11/03/automating-power-bi-deployments-trigger-a-refresh/ 90 | 91 | Automating Power BI deployments: Update refresh schedule 92 | https://martinschoombee.com/2020/10/27/automating-power-bi-deployments-update-refresh-schedule/ 93 | 94 | Automating Power BI deployments: Change data source credentials 95 | https://martinschoombee.com/2020/10/20/automating-power-bi-deployments-change-data-source-credentials/ 96 | 97 | Automating Power BI deployments: Update report parameters 98 | https://martinschoombee.com/2020/10/13/automating-power-bi-deployments-update-report-parameters/ 99 | 100 | Automating Power BI deployments: Deploying reports 101 | https://martinschoombee.com/2020/10/06/automating-power-bi-deployments-deploying-reports/ 102 | 103 | Automating Power BI deployments: Creating a workspace 104 | https://martinschoombee.com/2020/09/29/automating-power-bi-deployments-creating-a-workspace/ 105 | 106 | Automating Power BI deployments: Connecting to the service 107 | https://martinschoombee.com/2020/09/22/automating-power-bi-deployments-connecting-to-the-service/ 108 | 109 | Automating Power BI deployments: PowerShell basics 110 | https://martinschoombee.com/2020/09/15/automating-power-bi-deployments-powershell-basics/ 111 | 112 | Automating Power BI deployments: A series 113 | https://martinschoombee.com/2020/09/15/automating-power-bi-deployments-a-series/ 114 | 115 | Restore a Power BI workspace 116 | https://martinschoombee.com/2020/08/12/restore-a-power-bi-workspace/ 117 | 118 | Power BI development best practices: Naming conventions 119 | https://martinschoombee.com/2020/07/29/power-bi-development-best-practices-naming-conventions/ 120 | 121 | Change password expiration policies for service accounts in O365 122 | https://martinschoombee.com/2020/07/22/change-password-expiration-policies-for-service-accounts-in-o365/ 123 | 124 | A user-friendly way to deal with role-playing dimensions in Power BI 125 | https://martinschoombee.com/2020/06/24/a-user-friendly-way-to-deal-with-role-playing-dimensions-in-power-bi/ 126 | --------------------------------------------------------------------------------