├── .gitignore ├── ADFv2 ├── BackupAzureSQLDW.json ├── PauseAzureSQLDW.json ├── PauseAzureSQLDWIfNoQueriesRunning.json ├── ResumeAzureSQLDW.json ├── ds_sqldw_generic.json └── lsDW.json ├── AzureAutomation ├── CreateRestorePoint.ps1 ├── PauseAzureSQLDW.ps1 ├── ResumeAzureSQLDW.ps1 └── RotateKeys.ps1 ├── CLI └── resumeDW.bat ├── LICENSE ├── README.md ├── SQL └── scale to new DWU and loop until it completes.sql ├── automating-azure-sql-dw.sln └── images ├── ADFMSI.png └── THR2192 - Automating Azure SQL DW.pptx /.gitignore: -------------------------------------------------------------------------------- 1 | ## Ignore Visual Studio temporary files, build results, and 2 | ## files generated by popular Visual Studio add-ons. 3 | ## 4 | ## Get latest from https://github.com/github/gitignore/blob/master/VisualStudio.gitignore 5 | 6 | # User-specific files 7 | *.suo 8 | *.user 9 | *.userosscache 10 | *.sln.docstates 11 | 12 | # User-specific files (MonoDevelop/Xamarin Studio) 13 | *.userprefs 14 | 15 | # Build results 16 | [Dd]ebug/ 17 | [Dd]ebugPublic/ 18 | [Rr]elease/ 19 | [Rr]eleases/ 20 | x64/ 21 | x86/ 22 | bld/ 23 | [Bb]in/ 24 | [Oo]bj/ 25 | [Ll]og/ 26 | 27 | # Visual Studio 2015/2017 cache/options directory 28 | .vs/ 29 | # Uncomment if you have tasks that create the project's static files in wwwroot 30 | #wwwroot/ 31 | 32 | # Visual Studio 2017 auto generated files 33 | Generated\ Files/ 34 | 35 | # MSTest test Results 36 | [Tt]est[Rr]esult*/ 37 | [Bb]uild[Ll]og.* 38 | 39 | # NUNIT 40 | *.VisualState.xml 41 | TestResult.xml 42 | 43 | # Build Results of an ATL Project 44 | [Dd]ebugPS/ 45 | [Rr]eleasePS/ 46 | dlldata.c 47 | 48 | # Benchmark Results 49 | BenchmarkDotNet.Artifacts/ 50 | 51 | # .NET Core 52 | project.lock.json 53 | project.fragment.lock.json 54 | artifacts/ 55 | **/Properties/launchSettings.json 56 | 57 | # StyleCop 58 | StyleCopReport.xml 59 | 60 | # Files built by Visual Studio 61 | *_i.c 62 | *_p.c 63 | *_i.h 64 | *.ilk 65 | *.meta 66 | *.obj 67 | *.iobj 68 | *.pch 69 | *.pdb 70 | *.ipdb 71 | *.pgc 72 | *.pgd 73 | *.rsp 74 | *.sbr 75 | *.tlb 76 | *.tli 77 | *.tlh 78 | *.tmp 79 | *.tmp_proj 80 | *.log 81 | *.vspscc 82 | *.vssscc 83 | .builds 84 | *.pidb 85 | *.svclog 86 | *.scc 87 | 88 | # Chutzpah Test files 89 | _Chutzpah* 90 | 91 | # Visual C++ cache files 92 | ipch/ 93 | *.aps 94 | *.ncb 95 | *.opendb 96 | *.opensdf 97 | *.sdf 98 | *.cachefile 99 | *.VC.db 100 | *.VC.VC.opendb 101 | 102 | # Visual Studio profiler 103 | *.psess 104 | *.vsp 105 | *.vspx 106 | *.sap 107 | 108 | # Visual Studio Trace Files 109 | *.e2e 110 | 111 | # TFS 2012 Local Workspace 112 | $tf/ 113 | 114 | # Guidance Automation Toolkit 115 | *.gpState 116 | 117 | # ReSharper is a .NET coding add-in 118 | _ReSharper*/ 119 | *.[Rr]e[Ss]harper 120 | *.DotSettings.user 121 | 122 | # JustCode is a .NET coding add-in 123 | .JustCode 124 | 125 | # TeamCity is a build add-in 126 | _TeamCity* 127 | 128 | # DotCover is a Code Coverage Tool 129 | *.dotCover 130 | 131 | # AxoCover is a Code Coverage Tool 132 | .axoCover/* 133 | !.axoCover/settings.json 134 | 135 | # Visual Studio code coverage results 136 | *.coverage 137 | *.coveragexml 138 | 139 | # NCrunch 140 | _NCrunch_* 141 | .*crunch*.local.xml 142 | nCrunchTemp_* 143 | 144 | # MightyMoose 145 | *.mm.* 146 | AutoTest.Net/ 147 | 148 | # Web workbench (sass) 149 | .sass-cache/ 150 | 151 | # Installshield output folder 152 | [Ee]xpress/ 153 | 154 | # DocProject is a documentation generator add-in 155 | DocProject/buildhelp/ 156 | DocProject/Help/*.HxT 157 | DocProject/Help/*.HxC 158 | DocProject/Help/*.hhc 159 | DocProject/Help/*.hhk 160 | DocProject/Help/*.hhp 161 | DocProject/Help/Html2 162 | DocProject/Help/html 163 | 164 | # Click-Once directory 165 | publish/ 166 | 167 | # Publish Web Output 168 | *.[Pp]ublish.xml 169 | *.azurePubxml 170 | # Note: Comment the next line if you want to checkin your web deploy settings, 171 | # but database connection strings (with potential passwords) will be unencrypted 172 | *.pubxml 173 | *.publishproj 174 | 175 | # Microsoft Azure Web App publish settings. Comment the next line if you want to 176 | # checkin your Azure Web App publish settings, but sensitive information contained 177 | # in these scripts will be unencrypted 178 | PublishScripts/ 179 | 180 | # NuGet Packages 181 | *.nupkg 182 | # The packages folder can be ignored because of Package Restore 183 | **/[Pp]ackages/* 184 | # except build/, which is used as an MSBuild target. 185 | !**/[Pp]ackages/build/ 186 | # Uncomment if necessary however generally it will be regenerated when needed 187 | #!**/[Pp]ackages/repositories.config 188 | # NuGet v3's project.json files produces more ignorable files 189 | *.nuget.props 190 | *.nuget.targets 191 | 192 | # Microsoft Azure Build Output 193 | csx/ 194 | *.build.csdef 195 | 196 | # Microsoft Azure Emulator 197 | ecf/ 198 | rcf/ 199 | 200 | # Windows Store app package directories and files 201 | AppPackages/ 202 | BundleArtifacts/ 203 | Package.StoreAssociation.xml 204 | _pkginfo.txt 205 | *.appx 206 | 207 | # Visual Studio cache files 208 | # files ending in .cache can be ignored 209 | *.[Cc]ache 210 | # but keep track of directories ending in .cache 211 | !*.[Cc]ache/ 212 | 213 | # Others 214 | ClientBin/ 215 | ~$* 216 | *~ 217 | *.dbmdl 218 | *.dbproj.schemaview 219 | *.jfm 220 | *.pfx 221 | *.publishsettings 222 | orleans.codegen.cs 223 | 224 | # Including strong name files can present a security risk 225 | # (https://github.com/github/gitignore/pull/2483#issue-259490424) 226 | #*.snk 227 | 228 | # Since there are multiple workflows, uncomment next line to ignore bower_components 229 | # (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) 230 | #bower_components/ 231 | 232 | # RIA/Silverlight projects 233 | Generated_Code/ 234 | 235 | # Backup & report files from converting an old project file 236 | # to a newer Visual Studio version. Backup files are not needed, 237 | # because we have git ;-) 238 | _UpgradeReport_Files/ 239 | Backup*/ 240 | UpgradeLog*.XML 241 | UpgradeLog*.htm 242 | ServiceFabricBackup/ 243 | *.rptproj.bak 244 | 245 | # SQL Server files 246 | *.mdf 247 | *.ldf 248 | *.ndf 249 | 250 | # Business Intelligence projects 251 | *.rdl.data 252 | *.bim.layout 253 | *.bim_*.settings 254 | *.rptproj.rsuser 255 | 256 | # Microsoft Fakes 257 | FakesAssemblies/ 258 | 259 | # GhostDoc plugin setting file 260 | *.GhostDoc.xml 261 | 262 | # Node.js Tools for Visual Studio 263 | .ntvs_analysis.dat 264 | node_modules/ 265 | 266 | # Visual Studio 6 build log 267 | *.plg 268 | 269 | # Visual Studio 6 workspace options file 270 | *.opt 271 | 272 | # Visual Studio 6 auto-generated workspace file (contains which files were open etc.) 273 | *.vbw 274 | 275 | # Visual Studio LightSwitch build output 276 | **/*.HTMLClient/GeneratedArtifacts 277 | **/*.DesktopClient/GeneratedArtifacts 278 | **/*.DesktopClient/ModelManifest.xml 279 | **/*.Server/GeneratedArtifacts 280 | **/*.Server/ModelManifest.xml 281 | _Pvt_Extensions 282 | 283 | # Paket dependency manager 284 | .paket/paket.exe 285 | paket-files/ 286 | 287 | # FAKE - F# Make 288 | .fake/ 289 | 290 | # JetBrains Rider 291 | .idea/ 292 | *.sln.iml 293 | 294 | # CodeRush 295 | .cr/ 296 | 297 | # Python Tools for Visual Studio (PTVS) 298 | __pycache__/ 299 | *.pyc 300 | 301 | # Cake - Uncomment if you are using it 302 | # tools/** 303 | # !tools/packages.config 304 | 305 | # Tabs Studio 306 | *.tss 307 | 308 | # Telerik's JustMock configuration file 309 | *.jmconfig 310 | 311 | # BizTalk build output 312 | *.btp.cs 313 | *.btm.cs 314 | *.odx.cs 315 | *.xsd.cs 316 | 317 | # OpenCover UI analysis results 318 | OpenCover/ 319 | 320 | # Azure Stream Analytics local run output 321 | ASALocalRun/ 322 | 323 | # MSBuild Binary and Structured Log 324 | *.binlog 325 | 326 | # NVidia Nsight GPU debugger configuration file 327 | *.nvuser 328 | 329 | # MFractors (Xamarin productivity tool) working folder 330 | .mfractor/ 331 | -------------------------------------------------------------------------------- /ADFv2/BackupAzureSQLDW.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "BackupAzureSQLDW", 3 | "properties": { 4 | "activities": [ 5 | { 6 | "name": "BackupDW", 7 | "type": "WebActivity", 8 | "policy": { 9 | "timeout": "7.00:00:00", 10 | "retry": 0, 11 | "retryIntervalInSeconds": 30, 12 | "secureOutput": false 13 | }, 14 | "typeProperties": { 15 | "url": { 16 | "value": "@concat('https://management.azure.com/subscriptions/',pipeline().parameters.SubscriptionID,'/resourceGroups/',pipeline().parameters.ResourceGroup,'/providers/Microsoft.Sql/servers/',pipeline().parameters.Server,'/databases/',pipeline().parameters.DW,'/restorePoints?api-version=2017-03-01-preview')", 17 | "type": "Expression" 18 | }, 19 | "method": "POST", 20 | "body": { 21 | "value": "@json(concat('{\"restorePointLabel\":\"',string(pipeline().TriggerTime),'\"}'))", 22 | "type": "Expression" 23 | }, 24 | "authentication": { 25 | "type": "MSI", 26 | "resource": "https://management.core.windows.net/" 27 | } 28 | } 29 | } 30 | ], 31 | "parameters": { 32 | "SubscriptionID": { 33 | "type": "String", 34 | "defaultValue": "" 35 | }, 36 | "ResourceGroup": { 37 | "type": "String", 38 | "defaultValue": "" 39 | }, 40 | "Server": { 41 | "type": "String", 42 | "defaultValue": "" 43 | }, 44 | "DW": { 45 | "type": "String", 46 | "defaultValue": "" 47 | } 48 | } 49 | }, 50 | "type": "Microsoft.DataFactory/factories/pipelines" 51 | } -------------------------------------------------------------------------------- /ADFv2/PauseAzureSQLDW.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "PauseAzureSQLDW", 3 | "properties": { 4 | "activities": [ 5 | { 6 | "name": "IfSqlDwNotPaused", 7 | "type": "IfCondition", 8 | "dependsOn": [ 9 | { 10 | "activity": "CheckIfSqlDwPaused", 11 | "dependencyConditions": [ 12 | "Succeeded" 13 | ] 14 | } 15 | ], 16 | "typeProperties": { 17 | "expression": { 18 | "value": "@equals('Online',string(activity('CheckIfSqlDwPaused').output.properties.status))", 19 | "type": "Expression" 20 | }, 21 | "ifTrueActivities": [ 22 | { 23 | "name": "PauseSqlDw", 24 | "type": "WebActivity", 25 | "policy": { 26 | "timeout": "7.00:00:00", 27 | "retry": 0, 28 | "retryIntervalInSeconds": 30, 29 | "secureOutput": false 30 | }, 31 | "typeProperties": { 32 | "url": { 33 | "value": "@concat('https://management.azure.com/subscriptions/',pipeline().parameters.SubscriptionID,'/resourceGroups/',pipeline().parameters.ResourceGroup,'/providers/Microsoft.Sql/servers/',pipeline().parameters.Server,'/databases/',pipeline().parameters.DW,'/pause?api-version=2014-04-01-preview')", 34 | "type": "Expression" 35 | }, 36 | "method": "POST", 37 | "body": { 38 | "value": "@string('')", 39 | "type": "Expression" 40 | }, 41 | "authentication": { 42 | "type": "MSI", 43 | "resource": "https://management.core.windows.net/" 44 | } 45 | } 46 | } 47 | ] 48 | } 49 | }, 50 | { 51 | "name": "CheckIfSqlDwPaused", 52 | "type": "WebActivity", 53 | "policy": { 54 | "timeout": "7.00:00:00", 55 | "retry": 0, 56 | "retryIntervalInSeconds": 30, 57 | "secureOutput": false 58 | }, 59 | "typeProperties": { 60 | "url": { 61 | "value": "@concat('https://management.azure.com/subscriptions/',pipeline().parameters.SubscriptionID,'/resourceGroups/',pipeline().parameters.ResourceGroup,'/providers/Microsoft.Sql/servers/',pipeline().parameters.Server,'/databases/',pipeline().parameters.DW,'?api-version=2014-04-01')", 62 | "type": "Expression" 63 | }, 64 | "method": "GET", 65 | "authentication": { 66 | "type": "MSI", 67 | "resource": "https://management.core.windows.net/" 68 | } 69 | } 70 | }, 71 | { 72 | "name": "UntilSqlDwPaused", 73 | "type": "Until", 74 | "dependsOn": [ 75 | { 76 | "activity": "IfSqlDwNotPaused", 77 | "dependencyConditions": [ 78 | "Succeeded" 79 | ] 80 | } 81 | ], 82 | "typeProperties": { 83 | "expression": { 84 | "value": "@equals('Paused',string(activity('CheckSqlDwStatus').output.properties.status))", 85 | "type": "Expression" 86 | }, 87 | "activities": [ 88 | { 89 | "name": "CheckSqlDwStatus", 90 | "type": "WebActivity", 91 | "policy": { 92 | "timeout": "7.00:00:00", 93 | "retry": 0, 94 | "retryIntervalInSeconds": 30, 95 | "secureOutput": false 96 | }, 97 | "typeProperties": { 98 | "url": { 99 | "value": "@concat('https://management.azure.com/subscriptions/',pipeline().parameters.SubscriptionID,'/resourceGroups/',pipeline().parameters.ResourceGroup,'/providers/Microsoft.Sql/servers/',pipeline().parameters.Server,'/databases/',pipeline().parameters.DW,'?api-version=2014-04-01')", 100 | "type": "Expression" 101 | }, 102 | "method": "GET", 103 | "authentication": { 104 | "type": "MSI", 105 | "resource": "https://management.core.windows.net/" 106 | } 107 | } 108 | } 109 | ], 110 | "timeout": "7.00:00:00" 111 | } 112 | } 113 | ], 114 | "parameters": { 115 | "SubscriptionID": { 116 | "type": "String", 117 | "defaultValue": "" 118 | }, 119 | "ResourceGroup": { 120 | "type": "String", 121 | "defaultValue": "" 122 | }, 123 | "Server": { 124 | "type": "String", 125 | "defaultValue": "" 126 | }, 127 | "DW": { 128 | "type": "String", 129 | "defaultValue": "" 130 | } 131 | } 132 | }, 133 | "type": "Microsoft.DataFactory/factories/pipelines" 134 | } -------------------------------------------------------------------------------- /ADFv2/PauseAzureSQLDWIfNoQueriesRunning.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "PauseAzureSQLDWIfNoQueriesRunning", 3 | "properties": { 4 | "activities": [ 5 | { 6 | "name": "IfSqlDwNotPaused", 7 | "type": "IfCondition", 8 | "dependsOn": [ 9 | { 10 | "activity": "CheckIfSqlDwPaused", 11 | "dependencyConditions": [ 12 | "Succeeded" 13 | ] 14 | } 15 | ], 16 | "userProperties": [], 17 | "typeProperties": { 18 | "expression": { 19 | "value": "@equals('Online',string(activity('CheckIfSqlDwPaused').output.properties.status))", 20 | "type": "Expression" 21 | }, 22 | "ifTrueActivities": [ 23 | { 24 | "name": "CheckRunningQueriesCount", 25 | "type": "Lookup", 26 | "dependsOn": [], 27 | "policy": { 28 | "timeout": "7.00:00:00", 29 | "retry": 0, 30 | "retryIntervalInSeconds": 30, 31 | "secureOutput": false, 32 | "secureInput": false 33 | }, 34 | "userProperties": [], 35 | "typeProperties": { 36 | "source": { 37 | "type": "SqlDWSource", 38 | "sqlReaderQuery": "select count(*) as RunningQueryCount\nFROM sys.dm_pdw_exec_requests r \nwhere r.[status] = 'Running' \nand session_id<>SESSION_ID()", 39 | "queryTimeout": "02:00:00", 40 | "partitionOption": "None" 41 | }, 42 | "dataset": { 43 | "referenceName": "ds_sqldw_generic", 44 | "type": "DatasetReference", 45 | "parameters": { 46 | } 47 | } 48 | } 49 | } 50 | ] 51 | } 52 | }, 53 | { 54 | "name": "CheckIfSqlDwPaused", 55 | "type": "WebActivity", 56 | "dependsOn": [], 57 | "policy": { 58 | "timeout": "7.00:00:00", 59 | "retry": 0, 60 | "retryIntervalInSeconds": 30, 61 | "secureOutput": false 62 | }, 63 | "userProperties": [], 64 | "typeProperties": { 65 | "url": { 66 | "value": "@concat('https://management.azure.com/subscriptions/',pipeline().parameters.SubscriptionID,'/resourceGroups/',pipeline().parameters.ResourceGroup,'/providers/Microsoft.Sql/servers/',pipeline().parameters.Server,'/databases/',pipeline().parameters.DW,'?api-version=2014-04-01')", 67 | "type": "Expression" 68 | }, 69 | "connectVia": { 70 | "referenceName": "AutoResolveIntegrationRuntime", 71 | "type": "IntegrationRuntimeReference" 72 | }, 73 | "method": "GET", 74 | "authentication": { 75 | "type": "MSI", 76 | "resource": "https://management.core.windows.net/" 77 | } 78 | } 79 | }, 80 | { 81 | "name": "IfQueriesNotRunning", 82 | "type": "IfCondition", 83 | "dependsOn": [ 84 | { 85 | "activity": "IfSqlDwNotPaused", 86 | "dependencyConditions": [ 87 | "Succeeded" 88 | ] 89 | } 90 | ], 91 | "userProperties": [], 92 | "typeProperties": { 93 | "expression": { 94 | "value": "@equals(activity('CheckRunningQueriesCount').output.firstRow.RunningQueryCount,0)", 95 | "type": "Expression" 96 | }, 97 | "ifTrueActivities": [ 98 | { 99 | "name": "PauseSqlDw", 100 | "type": "WebActivity", 101 | "dependsOn": [], 102 | "policy": { 103 | "timeout": "7.00:00:00", 104 | "retry": 0, 105 | "retryIntervalInSeconds": 30, 106 | "secureOutput": false 107 | }, 108 | "userProperties": [], 109 | "typeProperties": { 110 | "url": { 111 | "value": "@concat('https://management.azure.com/subscriptions/',pipeline().parameters.SubscriptionID,'/resourceGroups/',pipeline().parameters.ResourceGroup,'/providers/Microsoft.Sql/servers/',pipeline().parameters.Server,'/databases/',pipeline().parameters.DW,'/pause?api-version=2014-04-01-preview')", 112 | "type": "Expression" 113 | }, 114 | "connectVia": { 115 | "referenceName": "AutoResolveIntegrationRuntime", 116 | "type": "IntegrationRuntimeReference" 117 | }, 118 | "method": "POST", 119 | "body": { 120 | "value": "@string('')", 121 | "type": "Expression" 122 | }, 123 | "authentication": { 124 | "type": "MSI", 125 | "resource": "https://management.core.windows.net/" 126 | } 127 | } 128 | } 129 | ] 130 | } 131 | }, 132 | { 133 | "name": "UntilSqlDwPaused", 134 | "type": "Until", 135 | "dependsOn": [ 136 | { 137 | "activity": "IfQueriesNotRunning", 138 | "dependencyConditions": [ 139 | "Succeeded" 140 | ] 141 | } 142 | ], 143 | "userProperties": [], 144 | "typeProperties": { 145 | "expression": { 146 | "value": "@or(equals('Paused',string(activity('CheckSqlDwStatus').output.properties.status)),greater(activity('CheckRunningQueriesCount').output.firstRow.RunningQueryCount,0))", 147 | "type": "Expression" 148 | }, 149 | "activities": [ 150 | { 151 | "name": "CheckSqlDwStatus", 152 | "type": "WebActivity", 153 | "dependsOn": [], 154 | "policy": { 155 | "timeout": "7.00:00:00", 156 | "retry": 0, 157 | "retryIntervalInSeconds": 30, 158 | "secureOutput": false 159 | }, 160 | "userProperties": [], 161 | "typeProperties": { 162 | "url": { 163 | "value": "@concat('https://management.azure.com/subscriptions/',pipeline().parameters.SubscriptionID,'/resourceGroups/',pipeline().parameters.ResourceGroup,'/providers/Microsoft.Sql/servers/',pipeline().parameters.Server,'/databases/',pipeline().parameters.DW,'?api-version=2014-04-01')", 164 | "type": "Expression" 165 | }, 166 | "method": "GET", 167 | "authentication": { 168 | "type": "MSI", 169 | "resource": "https://management.core.windows.net/" 170 | } 171 | } 172 | } 173 | ], 174 | "timeout": "7.00:00:00" 175 | } 176 | } 177 | ], 178 | "parameters": { 179 | "SubscriptionID": { 180 | "type": "String", 181 | "defaultValue": "" 182 | }, 183 | "ResourceGroup": { 184 | "type": "String", 185 | "defaultValue": "" 186 | }, 187 | "Server": { 188 | "type": "String", 189 | "defaultValue": "" 190 | }, 191 | "DW": { 192 | "type": "String", 193 | "defaultValue": "" 194 | } 195 | }, 196 | "annotations": [] 197 | }, 198 | "type": "Microsoft.DataFactory/factories/pipelines" 199 | } -------------------------------------------------------------------------------- /ADFv2/ResumeAzureSQLDW.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "ResumeAzureSQLDW", 3 | "properties": { 4 | "activities": [ 5 | { 6 | "name": "IfSqlDwPaused", 7 | "type": "IfCondition", 8 | "dependsOn": [ 9 | { 10 | "activity": "CheckIfSqlDwPaused", 11 | "dependencyConditions": [ 12 | "Succeeded" 13 | ] 14 | } 15 | ], 16 | "typeProperties": { 17 | "expression": { 18 | "value": "@not(equals('Online',string(activity('CheckIfSqlDwPaused').output.properties.status)))", 19 | "type": "Expression" 20 | }, 21 | "ifTrueActivities": [ 22 | { 23 | "name": "ResumeSqlDw", 24 | "type": "WebActivity", 25 | "policy": { 26 | "timeout": "7.00:00:00", 27 | "retry": 0, 28 | "retryIntervalInSeconds": 30, 29 | "secureOutput": false 30 | }, 31 | "typeProperties": { 32 | "url": { 33 | "value": "@concat('https://management.azure.com/subscriptions/',pipeline().parameters.SubscriptionID,'/resourceGroups/',pipeline().parameters.ResourceGroup,'/providers/Microsoft.Sql/servers/',pipeline().parameters.Server,'/databases/',pipeline().parameters.DW,'/resume?api-version=2014-04-01-preview')", 34 | "type": "Expression" 35 | }, 36 | "method": "POST", 37 | "body": { 38 | "value": "@string('')", 39 | "type": "Expression" 40 | }, 41 | "authentication": { 42 | "type": "MSI", 43 | "resource": "https://management.core.windows.net/" 44 | } 45 | } 46 | } 47 | ] 48 | } 49 | }, 50 | { 51 | "name": "CheckIfSqlDwPaused", 52 | "type": "WebActivity", 53 | "policy": { 54 | "timeout": "7.00:00:00", 55 | "retry": 0, 56 | "retryIntervalInSeconds": 30, 57 | "secureOutput": false 58 | }, 59 | "typeProperties": { 60 | "url": { 61 | "value": "@concat('https://management.azure.com/subscriptions/',pipeline().parameters.SubscriptionID,'/resourceGroups/',pipeline().parameters.ResourceGroup,'/providers/Microsoft.Sql/servers/',pipeline().parameters.Server,'/databases/',pipeline().parameters.DW,'?api-version=2014-04-01')", 62 | "type": "Expression" 63 | }, 64 | "method": "GET", 65 | "authentication": { 66 | "type": "MSI", 67 | "resource": "https://management.core.windows.net/" 68 | } 69 | } 70 | }, 71 | { 72 | "name": "UntilSqlDwResumed", 73 | "type": "Until", 74 | "dependsOn": [ 75 | { 76 | "activity": "IfSqlDwPaused", 77 | "dependencyConditions": [ 78 | "Succeeded" 79 | ] 80 | } 81 | ], 82 | "typeProperties": { 83 | "expression": { 84 | "value": "@equals('Online',string(activity('CheckSqlDwStatus').output.properties.status))", 85 | "type": "Expression" 86 | }, 87 | "activities": [ 88 | { 89 | "name": "CheckSqlDwStatus", 90 | "type": "WebActivity", 91 | "policy": { 92 | "timeout": "7.00:00:00", 93 | "retry": 0, 94 | "retryIntervalInSeconds": 30, 95 | "secureOutput": false 96 | }, 97 | "typeProperties": { 98 | "url": { 99 | "value": "@concat('https://management.azure.com/subscriptions/',pipeline().parameters.SubscriptionID,'/resourceGroups/',pipeline().parameters.ResourceGroup,'/providers/Microsoft.Sql/servers/',pipeline().parameters.Server,'/databases/',pipeline().parameters.DW,'?api-version=2014-04-01')", 100 | "type": "Expression" 101 | }, 102 | "method": "GET", 103 | "authentication": { 104 | "type": "MSI", 105 | "resource": "https://management.core.windows.net/" 106 | } 107 | } 108 | } 109 | ], 110 | "timeout": "7.00:00:00" 111 | } 112 | } 113 | ], 114 | "parameters": { 115 | "SubscriptionID": { 116 | "type": "String", 117 | "defaultValue": "" 118 | }, 119 | "ResourceGroup": { 120 | "type": "String", 121 | "defaultValue": "" 122 | }, 123 | "Server": { 124 | "type": "String", 125 | "defaultValue": "" 126 | }, 127 | "DW": { 128 | "type": "String", 129 | "defaultValue": "" 130 | } 131 | } 132 | }, 133 | "type": "Microsoft.DataFactory/factories/pipelines" 134 | } -------------------------------------------------------------------------------- /ADFv2/ds_sqldw_generic.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "ds_sqldw_generic", 3 | "properties": { 4 | "linkedServiceName": { 5 | "referenceName": "lsDW", 6 | "type": "LinkedServiceReference" 7 | }, 8 | "annotations": [], 9 | "type": "AzureSqlDWTable", 10 | "schema": [] 11 | } 12 | } -------------------------------------------------------------------------------- /ADFv2/lsDW.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "lsDW", 3 | "type": "Microsoft.Synapse/workspaces/linkedservices", 4 | "properties": { 5 | "parameters": { 6 | "databaseName": { 7 | "type": "string", 8 | "defaultValue": "YourDW" 9 | }, 10 | "serverName": { 11 | "type": "string", 12 | "defaultValue": "yourservergoeshere.database.windows.net" 13 | } 14 | }, 15 | "annotations": [], 16 | "type": "AzureSqlDW", 17 | "typeProperties": { 18 | "connectionString": "Integrated Security=False;Encrypt=True;Connection Timeout=30;Data Source=@{linkedService().serverName};Initial Catalog=@{linkedService().databaseName}" 19 | }, 20 | "connectVia": { 21 | "referenceName": "AutoResolveIntegrationRuntime", 22 | "type": "IntegrationRuntimeReference" 23 | } 24 | } 25 | } -------------------------------------------------------------------------------- /AzureAutomation/CreateRestorePoint.ps1: -------------------------------------------------------------------------------- 1 | param( 2 | 3 | [Parameter(Mandatory=$true)] 4 | [string] $SqlDwServerName, 5 | 6 | [Parameter(Mandatory=$true)] 7 | [string] $SqlDwDatabaseName, 8 | 9 | [Parameter(Mandatory=$true)] 10 | [string] $SqlDwResourceGroupName 11 | 12 | ) 13 | 14 | 15 | 16 | try 17 | { 18 | "Logging in to Azure..." 19 | Connect-AzAccount -Identity 20 | } 21 | catch { 22 | Write-Error -Message $_.Exception 23 | throw $_.Exception 24 | } 25 | 26 | $ErrorActionPreference = "Stop"; 27 | 28 | $Date=Get-Date 29 | $Label = $SqlDwDatabaseName + "_Backup_" + $Date 30 | 31 | New-AzSqlDatabaseRestorePoint -ResourceGroupName $SqlDwResourceGroupName -ServerName $SqlDwServerName -DatabaseName $SqlDwDatabaseName -RestorePointLabel $Label 32 | 33 | "Successfully created a User Defined Restore Point $Label" 34 | 35 | 36 | -------------------------------------------------------------------------------- /AzureAutomation/PauseAzureSQLDW.ps1: -------------------------------------------------------------------------------- 1 | param( 2 | 3 | [Parameter(Mandatory=$true)] 4 | [string] $SqlDwServerName, 5 | 6 | [Parameter(Mandatory=$true)] 7 | [string] $SqlDwDatabaseName, 8 | 9 | [Parameter(Mandatory=$true)] 10 | [string] $SqlDwResourceGroupName 11 | 12 | ) 13 | 14 | 15 | try 16 | { 17 | "Logging in to Azure..." 18 | Connect-AzAccount -Identity 19 | } 20 | catch { 21 | Write-Error -Message $_.Exception 22 | throw $_.Exception 23 | } 24 | 25 | $ErrorActionPreference = "Stop"; 26 | 27 | # Get old status 28 | $OldDbSetting = Get-AzSqlDatabase -DatabaseName $SqlDwDatabaseName -ServerName $SqlDwServerName -ResourceGroupName $SqlDwResourceGroupName 29 | $OldStatus = $OldDbSetting.Status 30 | 31 | 32 | 33 | if($OldStatus -eq "Paused") 34 | { 35 | 36 | Write-Output "Database $($SqlDwDatabaseName) already in Offline state." 37 | 38 | } 39 | else 40 | { 41 | 42 | $null = Suspend-AzSqlDatabase -DatabaseName $SqlDwDatabaseName -ServerName $SqlDwServerName -ResourceGroupName $SqlDwResourceGroupName 43 | Write-Output "Paused $($SqlDwDatabaseName) database of $($SqlDwResourceGroupName) resource group." 44 | 45 | } 46 | 47 | -------------------------------------------------------------------------------- /AzureAutomation/ResumeAzureSQLDW.ps1: -------------------------------------------------------------------------------- 1 | param( 2 | 3 | [Parameter(Mandatory=$true)] 4 | [string] $SqlDwServerName, 5 | 6 | [Parameter(Mandatory=$true)] 7 | [string] $SqlDwDatabaseName, 8 | 9 | [Parameter(Mandatory=$true)] 10 | [string] $SqlDwResourceGroupName 11 | 12 | ) 13 | 14 | 15 | 16 | try 17 | { 18 | "Logging in to Azure..." 19 | Connect-AzAccount -Identity 20 | } 21 | catch { 22 | Write-Error -Message $_.Exception 23 | throw $_.Exception 24 | } 25 | 26 | $ErrorActionPreference = "Stop"; 27 | 28 | # Get old status 29 | $OldDbSetting = Get-AzSqlDatabase -DatabaseName $SqlDwDatabaseName -ServerName $SqlDwServerName -ResourceGroupName $SqlDwResourceGroupName 30 | $OldStatus = $OldDbSetting.Status 31 | 32 | 33 | 34 | if($OldStatus -eq "Online") 35 | { 36 | Write-Output "Database $($SqlDwDatabaseName) already in online state." 37 | } 38 | else 39 | { 40 | 41 | $null = Resume-AzSqlDatabase -DatabaseName $SqlDwDatabaseName -ServerName $SqlDwServerName -ResourceGroupName $SqlDwResourceGroupName 42 | Write-Output "Resumed $($SqlDwDatabaseName) database of $($SqlDwResourceGroupName) resource group." 43 | } 44 | 45 | -------------------------------------------------------------------------------- /AzureAutomation/RotateKeys.ps1: -------------------------------------------------------------------------------- 1 | param( 2 | [Parameter(Mandatory=$true)] 3 | [string] $StorageAccountName, 4 | 5 | [Parameter(Mandatory=$true)] 6 | [string] $StorageAccountResourceGroupName, 7 | 8 | [Parameter(Mandatory=$true)] 9 | [string] $VaultName, 10 | 11 | [Parameter(Mandatory=$true)] 12 | [string] $SqlDwServerName, 13 | 14 | [Parameter(Mandatory=$true)] 15 | [string] $SqlDwDatabaseName, 16 | 17 | [Parameter(Mandatory=$true)] 18 | [string] $SqlDwResourceGroupName, 19 | 20 | [Parameter(Mandatory=$true)] 21 | [string] $AzureASServer, 22 | 23 | [Parameter(Mandatory=$true)] 24 | [string] $AzureASResourceGroupName 25 | ) 26 | 27 | 28 | #hardcoded variables that don't change from dev to prod 29 | $StorageConnectionStringSecret = "storage-connection-string" 30 | 31 | $SqlDwAdminUsername = "" 32 | $SqlDwAdminPasswordSecret = "dw-admin-password" 33 | 34 | $SqlDwAdfConnectionStringSecret = "dw-AdfLoader-connectionstring" 35 | $SqlDwAdfLoaderUsername = "" 36 | 37 | $SqlDwCubeReadOnlyUsername = "" 38 | $SqlDwCubeReadOnlyPasswordSecret = "dw-CubeReadOnly-password" 39 | 40 | $SqlDwDatabaseScopedCredentialName = "credStorage" 41 | 42 | 43 | $connectionName = "AzureRunAsConnection" 44 | try 45 | { 46 | # Get the connection "AzureRunAsConnection " 47 | $servicePrincipalConnection=Get-AutomationConnection -Name $connectionName 48 | 49 | "Logging in to Azure..." 50 | $login = Connect-AzureRmAccount ` 51 | -ServicePrincipal ` 52 | -TenantId $servicePrincipalConnection.TenantId ` 53 | -ApplicationId $servicePrincipalConnection.ApplicationId ` 54 | -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint 55 | "Login complete." 56 | } 57 | catch { 58 | if (!$servicePrincipalConnection) 59 | { 60 | $ErrorMessage = "Connection $connectionName not found." 61 | throw $ErrorMessage 62 | } else{ 63 | Write-Error -Message $_.Exception 64 | throw $_.Exception 65 | } 66 | } 67 | 68 | $ErrorActionPreference = "Stop"; 69 | 70 | "Rotating keys for the following systems:" 71 | "Azure SQL DW server $SqlDwServerName database $SqlDwDatabaseName" 72 | "Azure Blob Storage account $StorageAccountName" 73 | "Azure Key Vault $VaultName" 74 | 75 | Set-AzureRmKeyVaultAccessPolicy -VaultName $VaultName -ServicePrincipalName $servicePrincipalConnection.ApplicationId -PermissionsToSecrets Get,Set; 76 | "Successfully ensured Azure Automation has Azure Key Vault access" 77 | 78 | 79 | $status = Get-AzureRmSqlDatabase –ResourceGroupName $SqlDwResourceGroupName –ServerName $SqlDwServerName -DatabaseName $SqlDwDatabaseName | Select Status 80 | "Azure SQL DW state: " + $status.Status; 81 | 82 | # Check the status 83 | if($status.Status -eq "Paused") 84 | { 85 | $sqldw = Resume-AzureRmSqlDatabase –ResourceGroupName $SqlDwResourceGroupName –ServerName $SqlDwServerName –DatabaseName $SqlDwDatabaseName 86 | $mustPauseAzureSQLDW = $true; 87 | "Successfully Resumed SQL DW" 88 | } 89 | else 90 | { 91 | $mustPauseAzureSQLDW = $false; 92 | } 93 | 94 | 95 | $asServer = Get-AzureRmAnalysisServicesServer -ResourceGroupName $AzureASResourceGroupName -Name $AzureASServer 96 | [string]$AzureAsServerFullName = $asServer.ServerFullName; 97 | 98 | "Current Azure AS status: $($asServer.State)" 99 | 100 | if ($asServer.State -ne "Succeeded") 101 | { 102 | $null = ($asServer | Resume-AzureRmAnalysisServicesServer -Verbose) 103 | "Successfully Resumed Azure AS" 104 | $mustPauseAzureAS = $true; 105 | } 106 | else 107 | { 108 | $mustPauseAzureAS = $false; 109 | } 110 | 111 | 112 | $ipinfo = Invoke-RestMethod http://ipinfo.io/json 113 | 114 | if ($asServer.FirewallConfig -ne $null) 115 | { 116 | for ($i = 0; $i -lt $asServer.FirewallConfig.FirewallRules.Count; $i++) 117 | { 118 | $rule = $asServer.FirewallConfig.FirewallRules[$i]; 119 | if ($rule.FirewallRuleName -eq "AzureAutomation") 120 | { 121 | $asServer.FirewallConfig.FirewallRules.Remove($rule); 122 | $i--; 123 | } 124 | } 125 | 126 | #backup the firewall rules 127 | $rulesBackup = $asServer.FirewallConfig.FirewallRules.ToArray() 128 | 129 | #add a new AzureAutomation firewall rule 130 | $newRule = New-AzureRmAnalysisServicesFirewallRule -FirewallRuleName "AzureAutomation" -RangeStart $ipinfo.ip -RangeEnd $ipinfo.ip 131 | $asServer.FirewallConfig.FirewallRules.Add($newRule) 132 | Set-AzureRmAnalysisServicesServer -ResourceGroupName $AzureASResourceGroupName -Name $AzureASServer -FirewallConfig $asServer.FirewallConfig 133 | 134 | "Updated Azure AS firewall to allow current Azure Automation Public IP: " + $ipinfo.ip 135 | } 136 | else 137 | { 138 | "Azure AS Firewall is off" 139 | } 140 | 141 | 142 | #if the Azure SQL firewall does NOT allow connections from Azure 143 | if (-Not (Get-AzureRmSqlServerFirewallRule -ResourceGroupName $SqlDwResourceGroupName -ServerName $SqlDwServerName | Where-Object { $_.StartIpAddress -eq "0.0.0.0" })) 144 | { 145 | if (Get-AzureRmSqlServerFirewallRule -ResourceGroupName $SqlDwResourceGroupName -ServerName $SqlDwServerName -FirewallRuleName "AzureAutomation" -ErrorAction Ignore) 146 | { 147 | Set-AzureRmSqlServerFirewallRule -FirewallRuleName "AzureAutomation" -StartIpAddress $ipinfo.ip -EndIpAddress $ipinfo.ip -ResourceGroupName $SqlDwResourceGroupName -ServerName $SqlDwServerName; 148 | } 149 | else 150 | { 151 | New-AzureRmSqlServerFirewallRule -FirewallRuleName "AzureAutomation" -StartIpAddress $ipinfo.ip -EndIpAddress $ipinfo.ip -ResourceGroupName $SqlDwResourceGroupName -ServerName $SqlDwServerName; 152 | } 153 | "Updated Azure SQL DW firewall to allow current Azure Automation Public IP: " + $ipinfo.ip 154 | } 155 | else 156 | { 157 | "Azure SQL DW Firewall allows Azure to connect" 158 | } 159 | 160 | 161 | 162 | function InstallAndLoadTOM { 163 | $null = Register-PackageSource -Name nuget.org -Location http://www.nuget.org/api/v2 -Force -Trusted -ProviderName NuGet; 164 | $install = Install-Package Microsoft.AnalysisServices.retail.amd64 -ProviderName NuGet; 165 | if ($install.Payload.Directories -ne $null) 166 | { 167 | $dllFolder = $install.Payload.Directories[0].Location + "\" + $install.Payload.Directories[0].Name + "\lib\net45\" 168 | Add-Type -Path ($dllFolder + "Microsoft.AnalysisServices.Core.dll") 169 | Add-Type -Path ($dllFolder + "Microsoft.AnalysisServices.Tabular.Json.dll") 170 | Add-Type -Path ($dllFolder + "Microsoft.AnalysisServices.Tabular.dll") 171 | $amoAzureASServer = New-Object -TypeName Microsoft.AnalysisServices.Tabular.Server 172 | "Loaded Tabular Object Model assemblies" 173 | } 174 | } 175 | 176 | 177 | #passing in $keyIndex=0 changes key1 178 | #passing in $keyIndex=1 changes key2 179 | function RotateStorageAccountKey([int]$keyIndex) 180 | { 181 | [string]$keyName = "key" + ($keyIndex+1) 182 | 183 | #rotate the key in blob storage 184 | $SAKeys = New-AzureRmStorageAccountKey -ResourceGroupName $StorageAccountResourceGroupName -Name $StorageAccountName -KeyName $keyName -Verbose 185 | $storageAccountKey = $SAKeys.Keys[$keyIndex].Value; 186 | $storageConnectionString = "DefaultEndpointsProtocol=https;AccountName=$StorageAccountName;AccountKey=" + $storageAccountKey; 187 | 188 | "Successfully rotated blob storage $keyName" 189 | 190 | $secretvalue = ConvertTo-SecureString $storageConnectionString -AsPlainText -Force 191 | $secret = Set-AzureKeyVaultSecret -VaultName $VaultName -Name $StorageConnectionStringSecret -SecretValue $secretvalue 192 | 193 | "Successfully updated Azure Key Vault secret $StorageConnectionStringSecret" 194 | 195 | $secret = Get-AzureKeyVaultSecret -VaultName $VaultName -Name $SqlDwAdminPasswordSecret 196 | $SqlDwAdminPassword = $secret.SecretValueText 197 | 198 | $conn = New-Object System.Data.SqlClient.SqlConnection 199 | $conn.ConnectionString = 'Data Source=' + $SqlDwServerName + '.database.windows.net;Initial Catalog=' + $SqlDwDatabaseName + ';Integrated Security=False;User ID=' + $SqlDwAdminUsername + ';Password=' + $SqlDwAdminPassword + ';Connect Timeout=60;Encrypt=False;TrustServerCertificate=False;' 200 | $conn.Open() 201 | 202 | $cmd = New-Object System.Data.SqlClient.SqlCommand 203 | $cmd.Connection = $conn 204 | $cmd.CommandText = "ALTER DATABASE SCOPED CREDENTIAL $SqlDwDatabaseScopedCredentialName WITH IDENTITY = 'blob', SECRET = '$storageAccountKey';" 205 | $queryOutput = $cmd.ExecuteNonQuery(); 206 | 207 | "Successfully updated Azure SQL DW database scoped credential $SqlDwDatabaseScopedCredentialName with storage account key" 208 | } 209 | 210 | 211 | #rotate key 2 (index 1) then key 1 (index 0) 212 | RotateStorageAccountKey(1) 213 | RotateStorageAccountKey(0) 214 | 215 | 216 | function GetNewPassword 217 | { 218 | return (([char[]]((New-Guid).ToString() + '*%#!/ABCDEF')) | sort {Get-Random}) -join '' 219 | } 220 | 221 | 222 | $secret = Get-AzureKeyVaultSecret -VaultName $VaultName -Name $SqlDwAdminPasswordSecret 223 | $SqlDwAdminPassword = $secret.SecretValueText 224 | 225 | $conn = New-Object System.Data.SqlClient.SqlConnection 226 | $conn.ConnectionString = 'Data Source=' + $SqlDwServerName + '.database.windows.net;Initial Catalog=master;Integrated Security=False;User ID=' + $SqlDwAdminUsername + ';Password=' + $SqlDwAdminPassword + ';Connect Timeout=60;Encrypt=False;TrustServerCertificate=False;' 227 | $conn.Open() 228 | 229 | #generate a new SQL DW AdfLoader password 230 | $NewSqlDwAdfLoaderPassword = GetNewPassword; 231 | 232 | $cmd = New-Object System.Data.SqlClient.SqlCommand 233 | $cmd.Connection = $conn 234 | $cmd.CommandText = "ALTER LOGIN $SqlDwAdfLoaderUsername WITH PASSWORD = '$NewSqlDwAdfLoaderPassword';" 235 | $queryOutput = $cmd.ExecuteNonQuery(); 236 | "Successfully updated password for $SqlDwAdfLoaderUsername login" 237 | 238 | $NewSqlDwAdfConnectionString = 'Server=tcp:' + $SqlDwServerName + '.database.windows.net,1433;Database=' + $SqlDwDatabaseName + ';User ID=' + $SqlDwAdfLoaderUsername + ';Password=' + $NewSqlDwAdfLoaderPassword + ';Trusted_Connection=False;Encrypt=True;Connection Timeout=60' 239 | 240 | $secretvalue = ConvertTo-SecureString $NewSqlDwAdfConnectionString -AsPlainText -Force 241 | $secret = Set-AzureKeyVaultSecret -VaultName $VaultName -Name $SqlDwAdfConnectionStringSecret -SecretValue $secretvalue 242 | "Successfully updated Azure Key Vault secret $SqlDwAdfConnectionStringSecret" 243 | 244 | 245 | #generate a new SQL DW CubeReadOnly password 246 | $NewSqlDwCubeReadOnlyPassword = GetNewPassword; 247 | 248 | $cmd = New-Object System.Data.SqlClient.SqlCommand 249 | $cmd.Connection = $conn 250 | $cmd.CommandText = "ALTER LOGIN $SqlDwCubeReadOnlyUsername WITH PASSWORD = '$NewSqlDwCubeReadOnlyPassword';" 251 | $queryOutput = $cmd.ExecuteNonQuery(); 252 | "Successfully updated password for $SqlDwCubeReadOnlyUsername login" 253 | 254 | $secretvalue = ConvertTo-SecureString $NewSqlDwCubeReadOnlyPassword -AsPlainText -Force 255 | $secret = Set-AzureKeyVaultSecret -VaultName $VaultName -Name $SqlDwCubeReadOnlyPasswordSecret -SecretValue $secretvalue 256 | "Successfully updated Azure Key Vault secret $SqlDwCubeReadOnlyPasswordSecret" 257 | 258 | InstallAndLoadTOM 259 | 260 | 261 | $amoAzureASServer = New-Object -TypeName Microsoft.AnalysisServices.Tabular.Server 262 | $amoAzureASServer.Connect("Data Source=$AzureAsServerFullName;User ID=app:" + $servicePrincipalConnection.ApplicationId + "@" + $servicePrincipalConnection.TenantId + ";Provider=MSOLAP;Persist Security Info=True;Impersonation Level=Impersonate;Password=cert:" + $servicePrincipalConnection.CertificateThumbprint) 263 | if ($amoAzureASServer.Databases.Count -eq 0) 264 | { 265 | "Warning: No Azure AS databases found! Ensure app:" + $servicePrincipalConnection.ApplicationId + "@" + $servicePrincipalConnection.TenantId + " has Analysis Server Admin access" 266 | } 267 | $amoAzureASServer.Databases | ForEach-Object { 268 | $db = $_; 269 | "Finding data sources in Azure AS database " + $db.Name 270 | $madeChange = $false; 271 | $db.Model.DataSources | ForEach-Object { 272 | $ds = $_; 273 | if ($ds.GetType().Name -eq "ProviderDataSource") 274 | { 275 | if ($ds.Provider -eq "System.Data.SqlClient") 276 | { 277 | $ds.ImpersonationMode = [Microsoft.AnalysisServices.Tabular.ImpersonationMode]::ImpersonateServiceAccount; #developers may have had to deploy with the ImpersonateAccount setting because they aren't server admins 278 | $ds.Account = ""; 279 | $ds.Password = ""; 280 | $ds.ConnectionString = "x"; 281 | $connStr = "Data Source=tcp:$SqlDwServerName.database.windows.net,1433;Initial Catalog=$SqlDwDatabaseName;User ID=$SqlDwCubeReadOnlyUsername;Password=$NewSqlDwCubeReadOnlyPassword;Persist Security Info=true;Encrypt=true;TrustServerCertificate=false;Packet Size=32767;" 282 | $tmsl = [Microsoft.AnalysisServices.Tabular.JsonScripter]::ScriptAlter($ds); 283 | $tmsl = $tmsl.Replace("********",$connStr); 284 | #would normally do $db.Model.SaveChanges() but was getting an error so used this as a workaround 285 | $results = $amoAzureASServer.Execute($tmsl); 286 | 287 | foreach ($message in $results.Messages) 288 | { 289 | if ($message.GetType().FullName -eq "Microsoft.AnalysisServices.XmlaError") 290 | { 291 | throw $message.Description 292 | } 293 | if ($message.GetType().FullName -eq "Microsoft.AnalysisServices.XmlaWarning") 294 | { 295 | "Warning $($message.Description)" 296 | } 297 | } 298 | "Updated connection string in data source: " + $ds.Name 299 | } 300 | elseif ($ds.Provider -eq "") 301 | { 302 | $ds.ImpersonationMode = [Microsoft.AnalysisServices.Tabular.ImpersonationMode]::ImpersonateServiceAccount; #developers may have had to deploy with the ImpersonateAccount setting because they aren't server admins 303 | $ds.Account = ""; 304 | $ds.Password = ""; 305 | $ds.ConnectionString = "x"; 306 | $connStr = "Provider=SQLNCLI11;Data Source=tcp:$SqlDwServerName.database.windows.net,1433;Initial Catalog=$SqlDwDatabaseName;User ID=$SqlDwCubeReadOnlyUsername;Password=$NewSqlDwCubeReadOnlyPassword;Persist Security Info=true;Packet Size=32767;" 307 | $tmsl = [Microsoft.AnalysisServices.Tabular.JsonScripter]::ScriptAlter($ds); 308 | $tmsl = $tmsl.Replace("********",$connStr); 309 | #would normally do $db.Model.SaveChanges() but was getting an error so used this as a workaround 310 | $results = $amoAzureASServer.Execute($tmsl); 311 | 312 | foreach ($message in $results.Messages) 313 | { 314 | if ($message.GetType().FullName -eq "Microsoft.AnalysisServices.XmlaError") 315 | { 316 | throw $message.Description 317 | } 318 | if ($message.GetType().FullName -eq "Microsoft.AnalysisServices.XmlaWarning") 319 | { 320 | "Warning $($message.Description)" 321 | } 322 | } 323 | "Updated connection string in data source: " + $ds.Name 324 | } 325 | else 326 | { 327 | "Unsupported data source: " + $ds.Name 328 | } 329 | } 330 | else 331 | { 332 | $ds.Credential.AuthenticationKind = "UsernamePassword"; 333 | $ds.Credential.Username = $SqlDwCubeReadOnlyUsername; 334 | $ds.Credential.Password = "x"; 335 | $ds.Credential.EncryptConnection = $true; 336 | $tmsl = [Microsoft.AnalysisServices.Tabular.JsonScripter]::ScriptAlter($ds); 337 | $tmsl = $tmsl.Replace("********",$NewSqlDwCubeReadOnlyPassword); 338 | 339 | #would normally do $db.Model.SaveChanges() but was getting an error so used this as a workaround 340 | $results = $amoAzureASServer.Execute($tmsl); 341 | 342 | foreach ($message in $results.Messages) 343 | { 344 | if ($message.GetType().FullName -eq "Microsoft.AnalysisServices.XmlaError") 345 | { 346 | throw $message.Description 347 | } 348 | if ($message.GetType().FullName -eq "Microsoft.AnalysisServices.XmlaWarning") 349 | { 350 | "Warning $($message.Description)" 351 | } 352 | } 353 | 354 | "Updated credentials in modern data source: " + $ds.Name 355 | } 356 | } 357 | } 358 | $null = $amoAzureASServer.Disconnect(); 359 | 360 | 361 | #generate a new SQL DW admin password 362 | $NewSqlDwAdminPassword = GetNewPassword; 363 | 364 | $cmd.CommandText = "ALTER LOGIN $SqlDwAdminUsername WITH PASSWORD = '$NewSqlDwAdminPassword';" 365 | $queryOutput = $cmd.ExecuteNonQuery(); 366 | "Successfully updated password for $SqlDwAdminUsername login" 367 | 368 | $secretvalue = ConvertTo-SecureString $NewSqlDwAdminPassword -AsPlainText -Force 369 | $secret = Set-AzureKeyVaultSecret -VaultName $VaultName -Name $SqlDwAdminPasswordSecret -SecretValue $secretvalue 370 | "Successfully updated Azure Key Vault secret $SqlDwAdminPasswordSecret" 371 | 372 | 373 | if (Get-AzureRmSqlServerFirewallRule -ResourceGroupName $SqlDwResourceGroupName -ServerName $SqlDwServerName -FirewallRuleName "AzureAutomation" -ErrorAction Ignore) 374 | { 375 | Remove-AzureRmSqlServerFirewallRule -FirewallRuleName "AzureAutomation" -ResourceGroupName $SqlDwResourceGroupName -ServerName $SqlDwServerName; 376 | "Reset Azure SQL DW firewall rules" 377 | } 378 | 379 | 380 | if($mustPauseAzureSQLDW) 381 | { 382 | $null = Suspend-AzureRmSqlDatabase –ResourceGroupName $SqlDwResourceGroupName –ServerName $SqlDwServerName –DatabaseName $SqlDwDatabaseName 383 | "Successfully Paused SQL DW" 384 | } 385 | 386 | 387 | 388 | if ($asServer.FirewallConfig -ne $null) 389 | { 390 | #reset firewall to the state it was in before this script started 391 | $asServer.FirewallConfig.FirewallRules.Clear() 392 | $asServer.FirewallConfig.FirewallRules.AddRange($rulesBackup) 393 | Set-AzureRmAnalysisServicesServer -ResourceGroupName $AzureASResourceGroupName -Name $AzureASServer -FirewallConfig $asServer.FirewallConfig 394 | "Reset Azure AS firewall rules" 395 | } 396 | 397 | 398 | if ($mustPauseAzureAS) 399 | { 400 | $null = ($asServer | Suspend-AzureRmAnalysisServicesServer -Verbose) 401 | "Successfully Paused Azure AS" 402 | } 403 | 404 | "Successfully completed rotating keys" -------------------------------------------------------------------------------- /CLI/resumeDW.bat: -------------------------------------------------------------------------------- 1 | REM from within an Azure VM with the Managed Service Identity (MSI) and the MSI having permissions to the Azure SQL Server, use az login --identity 2 | REM otherwise run "az login -h" for other options 3 | REM fill in the parameters below 4 | call az login --identity 5 | call az account set -s 6 | call az sql dw resume --resource-group --server --name 7 | 8 | REM occasionally the resume command completes slightly before the DW is unpaused so loop until status says Online 9 | :checkOnline 10 | call az sql dw show --resource-group --server --name | findstr /C:"\\\"status\\\": \\\"Online\\\"" 11 | if not %errorlevel%==0 goto :checkOnline 12 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 furmangg 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Automating Azure Synapse Analytics (formerly Azure SQL DW) - Code Samples 2 | My Ignite 2018 presentation entitled [Automating Azure SQL Data Warehouse](https://myignite.techcommunity.microsoft.com/sessions/66195?source=sessions) (download [slides](https://github.com/furmangg/automating-azure-sql-dw/raw/master/images/THR2192%20-%20Automating%20Azure%20SQL%20DW.pptx)) on September 26, 2018 included demos of various ways to automate Azure SQL DW. These code samples are included here. 3 | 4 | ##### Screencast Video Recording: 5 | 6 | Screencast Video Recording 7 | 8 | 9 | ### [ADFv2](https://github.com/furmangg/automating-azure-sql-dw/tree/master/ADFv2) 10 | 11 | These ADF pipelines will likely not work against a SQL Pool running in an Azure Synapse Analytics workspace (preview) but do work against "Azure Synapse Analytics (formerly Azure SQL DW)". Stay tuned for subsequent updates which support Synapse workspaces. 12 | 13 | #### BackupAzureSQLDW 14 | 15 | The [ADFv2/BackupAzureSQLDW.json](https://raw.githubusercontent.com/furmangg/automating-azure-sql-dw/master/ADFv2/BackupAzureSQLDW.json) file contains an Azure Data Factory v2 pipeline which triggers a backup of your Azure SQL DW. In proper terms, this pipeline creates a [user-defined restore point](https://docs.microsoft.com/en-us/azure/sql-data-warehouse/backup-and-restore#user-defined-restore-points) in your DW. If you DW is paused through the day except for a few hours during the loads then it may not be online long enough to get an automatic restore point created. Furthermore, automatic restore points may happen in the middle of a load making them useless for a restore. Triggering a user-defined restore point ensures you backup the DW at a consistent point before or after a load. 16 | 17 | Set the following parameters upon execution of the pipeline: 18 | * **SubscriptionID** - The GUID identifier for the subscription the Azure SQL DW is running from. To get this ID, go to the Subscriptions tab of the Azure Portal. 19 | * **ResourceGroup** - The name of the resource group where the Azure SQL DW lives. 20 | * **Server** - The name of your Azure SQL DW server. This is not the full _yourdwserver.database.windows.net_ server name. This is just the initial _yourdwserver_ section. 21 | * **DW** - The name of the DW database. 22 | 23 | This pipeline executes the command under your ADF Managed Service Identity (MSI). Thus that MSI must be granted proper permissions. If your data factory is named "gregadfv2" then go to the Access Control (IAM) tab in the Azure SQL Server's blade, click Add, choose Role=Contributor, then search for your data factory's name, select the MSI that is returned from the search and click Save: 24 | 25 | ![Assigning MSI permissions](images/ADFMSI.png) 26 | 27 | 28 | #### PauseAzureSQLDW 29 | 30 | The [ADFv2/PauseAzureSQLDW.json](https://raw.githubusercontent.com/furmangg/automating-azure-sql-dw/master/ADFv2/PauseAzureSQLDW.json) file contains an Azure Data Factory v2 pipeline which pauses your DW and loops until the pause is complete. This pipeline immediately pauses your DW without checking whether any queries or loads are running. 31 | 32 | Set the following parameters upon execution of the pipeline: 33 | * **SubscriptionID** - The GUID identifier for the subscription the Azure SQL DW is running from. To get this ID, go to the Subscriptions tab of the Azure Portal. 34 | * **ResourceGroup** - The name of the resource group where the Azure SQL DW lives. 35 | * **Server** - The name of your Azure SQL DW server. This is not the full _yourdwserver.database.windows.net_ server name. This is just the initial _yourdwserver_ section. 36 | * **DW** - The name of the DW database. 37 | 38 | This pipeline executes the command under your ADF Managed Service Identity (MSI). Thus that MSI must be granted proper permissions as explained in the instructions for BackupAzureSQLDW above. 39 | 40 | 41 | #### ResumeAzureSQLDW 42 | 43 | The [ADFv2/ResumeAzureSQLDW.json](https://raw.githubusercontent.com/furmangg/automating-azure-sql-dw/master/ADFv2/ResumeAzureSQLDW.json) file contains an Azure Data Factory v2 pipeline which resumes (unpauses) your DW and loops until the DW is online. If the DW is already online then it does nothing. 44 | 45 | Set the following parameters upon execution of the pipeline: 46 | * **SubscriptionID** - The GUID identifier for the subscription the Azure SQL DW is running from. To get this ID, go to the Subscriptions tab of the Azure Portal. 47 | * **ResourceGroup** - The name of the resource group where the Azure SQL DW lives. 48 | * **Server** - The name of your Azure SQL DW server. This is not the full _yourdwserver.database.windows.net_ server name. This is just the initial _yourdwserver_ section. 49 | * **DW** - The name of the DW database. 50 | 51 | This pipeline executes the command under your ADF Managed Service Identity (MSI). Thus that MSI must be granted proper permissions as explained in the instructions for BackupAzureSQLDW above. 52 | 53 | 54 | #### PauseAzureSQLDWIfNoQueriesRunning 55 | 56 | The [ADFv2/PauseAzureSQLDWIfNoQueriesRunning.json](https://raw.githubusercontent.com/furmangg/automating-azure-sql-dw/master/ADFv2/PauseAzureSQLDWIfNoQueriesRunning.json) file contains an Azure Data Factory v2 pipeline which pauses your DW and loops until the pause is complete. This pipeline only pauses your DW if no queries or loads are running. 57 | 58 | Set the following parameters upon execution of the pipeline: 59 | * **SubscriptionID** - The GUID identifier for the subscription the Azure SQL DW is running from. To get this ID, go to the Subscriptions tab of the Azure Portal. 60 | * **ResourceGroup** - The name of the resource group where the Azure SQL DW lives. 61 | * **Server** - The name of your Azure SQL DW server. This is not the full _yourdwserver.database.windows.net_ server name. This is just the initial _yourdwserver_ section. 62 | * **DW** - The name of the DW database. 63 | 64 | This pipeline executes the command under your ADF Managed Service Identity (MSI). Thus that MSI must be granted proper permissions as explained in the instructions for BackupAzureSQLDW above. The MSI must also have db_owner permission in the DW. 65 | 66 | This pipeline depends on the [lsDW](https://raw.githubusercontent.com/furmangg/automating-azure-sql-dw/master/ADFv2/lsDW.json) linked service and the [ds_sqldw_generic](https://raw.githubusercontent.com/furmangg/automating-azure-sql-dw/master/ADFv2/ds_sqldw_generic.json) dataset to enable the Lookup activity to check whether queries are running. 67 | 68 |

69 | 70 | 71 | 72 | ### [CLI](https://github.com/furmangg/automating-azure-sql-dw/tree/master/CLI) 73 | 74 | This CLI script will likely not work against a SQL Pool running in an Azure Synapse Analytics workspace (preview) but do work against "Azure Synapse Analytics (formerly Azure SQL DW)". Stay tuned for subsequent updates which support Synapse workspaces. 75 | 76 | #### resumeDW.bat 77 | 78 | The [CLI/resumeDW.bat](https://raw.githubusercontent.com/furmangg/automating-azure-sql-dw/master/CLI/resumeDW.bat) file is a batch script which calls the [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) to resume (unpause) your DW and loop until the DW is online. If the DW is already online then it does nothing. 79 | 80 | The script as written is designed to run from within an Azure VM where the Managed Service Identity (MSI) has been [enabled](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/qs-configure-portal-windows-vm) and [granted](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/howto-assign-access-portal) Contributor permissions to the Azure SQL Server. 81 | 82 |

83 | 84 | ### [Azure Automation](https://github.com/furmangg/automating-azure-sql-dw/tree/master/AzureAutomation) 85 | 86 | This Azure Automation runbook will likely not work against a SQL Pool running in an Azure Synapse Analytics workspace (preview) but do work against "Azure Synapse Analytics (formerly Azure SQL DW)". Stay tuned for subsequent updates which support Synapse workspaces. 87 | 88 | #### RotateKeys.ps1 89 | 90 | The [AzureAutomation/RotateKeys.ps1](https://raw.githubusercontent.com/furmangg/automating-azure-sql-dw/master/AzureAutomation/RotateKeys.ps1) script is designed to keep your Azure SQL DW environment as secure as possible. In case an employee leaves who knows the SQL DW admin password or the blob storage key, key rotation will reset these sensitive passwords and secrets. This script performs the following tasks: 91 | 92 | 1. Authenticates with the Azure Automation RunAs account 93 | 2. Ensures that RunAs account has Azure Key Vault access 94 | 3. Starts the DW if it is paused 95 | 4. Starts your Azure Analysis Services if it was paused 96 | 5. Looks up the current public IP address of the current Azure Automation instance. It then temporarily opens the Azure SQL DW and Azure AS firewall to that IP so that the Azure Automation runbook is able to perform the necessary operations. 97 | 6. Rotates the Azure storage account key and then updates this in Azure Key Vault and in the Polybase database scoped credential in Azure SQL DW. 98 | 7. Connects to Azure SQL DW with the admin account and changes the password of the admin, AdfLoader and CubeReadOnly SQL accounts 99 | 8. Updates Azure Key Vault with the connection string Azure Data Factory uses 100 | 9. Updates Azure Key Vault with the CubeReadOnly password so that cube developers can lookup the latest password. 101 | 10. Updates Azure Key Vault with the Azure SQL DW admin SQL password so that key rotation next week can look it up. 102 | 11. Connect to Azure Analysis Services and loop through all the databases fixing the connection strings in the data sources. (Currently it assumes all data sources connect to Azure SQL DW, so this code may need to be tweaked for your environment.) 103 | 12. Resets the Azure SQL DW and Azure Analysis Services firewall to remove the temporary Azure Automation IP address. 104 | 13. Pauses Azure SQL DW if Azure Automation started it during key rotation. 105 | 14. Pauses Azure Analysis Services if Azure Automation started it during key rotation. 106 | 107 | Deploying this solution requires a number of steps: 108 | 109 | 1. In your Azure Automation account, import the AzureRM.profile, AzureRM.AnalysisServices, AzureRM.KeyVault, and PackageManagement. 110 | 1. Ensure the RunAs account is created and is a Contributor on the subscription (or at least the resource groups for Azure Key Vault, Azure SQL DW, Azure Analysis Services and Azure Blob Storage.) 111 | 1. Create an Azure Key Vault and grant your ADF Managed Service Identity (MSI) permission to get secrets. (Azure Automation will grant its RunAs account permission to Azure Key Vault secrets.) Grant cube developers or other developers permissions to get secrets. 112 | 1. Create a new secret called "dw-admin-password" and populate it with the SQL DW admin password. 113 | 1. Create a new secret called "dw-AdfLoader-connectionstring" and populate it with a placeholder like the letter "x". The runbook will set this after key rotation runs the first time. 114 | 1. Create a new secret called "dw-CubeReadOnly-password" and populate it with a placeholder like the letter "x". The runbook will set this after key rotation. 115 | 1. Create a new secret called "storage-connection-string" and populate it with a placeholder like the letter "x". The runbook will set this after key rotation. 116 | 1. Create a new SQL DW login called AdfLoader with db_owner permissions and then update the <YourSqlDwLoginUsedInAdf> token in the PowerShell script with its name. 117 | 1. Create a new SQL DW login called CubeReadOnly with db_datareader permissions and then update the <YourSqlDwLoginForAzureAS> token in the PowerShell script with its name. 118 | 1. Update the <YourSqlDwAdminUsername> token in the PowerShell script with the name of your SQL DW admin account. 119 | 1. Create a new database scoped credential in Azure SQL DW called "credStorage" pointing at the Azure Blob Storage account. Create any additional Polybase objects like external data sources that reference that credential. 120 | 1. Connect to Azure Analysis Services in SSMS and right click on the server node and choose Properties and on the Security tab. Click Add to add a new admin user. Search for the name of your Azure Automation Account in order to find the RunAs account and add this identity. 121 | 1. Deploy your Azure Analysis Services model using the CubeReadOnly account. 122 | 1. Create a new lsAzureKeyVault linked service in Azure Data Factory pointing to Azure Key Vault. 123 | 1. Create a new linked service in Azure Data Factory pointing to Azure SQL DW but have it get the connection string from the "dw-AdfLoader-connectionstring" secret in lsAzureKeyVault. 124 | 1. Create a new linked service in Azure Data Factory pointing to Azure Blob Storage but have it get the connection string from the "storage-connection-string" secret in lsAzureKeyVault. 125 | 1. In the Azure Automation pane for the RotateKeys runbook click the Schedule button, setup a schedule such as every Sunday morning, then set the parameters as follows. 126 | 127 | The runbook has the following parameters: 128 | * **StorageAccountName** - The name of your storage account which will have its keys rotated, the new key updated in Azure SQL DW in the database scoped credential, and the key updated in the Azure Key Vault secret which Azure Automation uses. 129 | * **StorageAccountResourceGroupName** - The name of the resource group where your storage account lives. 130 | * **VaultName** - The name of your Azure Key Vault. 131 | * **SqlDwServerName** - The name of your Azure SQL DW server. This is not the full _yourdwserver.database.windows.net_ server name. This is just the initial _yourdwserver_ section. 132 | * **SqlDwDatabaseName** - The name of the DW database. 133 | * **SqlDwResourceGroupName** - The name of the resource group where your DW lives. 134 | * **AzureASServer** - The name of your Azure Analysis Services. This is not the full asazure:// URI. This is just the final section saying the name of your server. 135 | * **AzureASResourceGroupName** - The name of the resource group where your Azure Analysis Services lives. 136 | 137 |

138 | 139 | 140 | ### [SQL](https://github.com/furmangg/automating-azure-sql-dw/tree/master/SQL) 141 | 142 | #### scale to new DWU and loop until it completes.sql 143 | 144 | The [SQL/scale to new DWU and loop until it completes.bat](https://raw.githubusercontent.com/furmangg/automating-azure-sql-dw/master/SQL/scale%20to%20new%20DWU%20and%20loop%20until%20it%20completes.sql) file SQL script you can run against the master database on the server containing your Azure SQL DW. It assumes there's only one DW on that server and it assumes that the user running it is a server admin or has been granted dbmanager role (as mentioned at the top of the script in comments) 145 | 146 |

147 | 148 | 149 | ### Questions or Issues 150 | 151 | Use the [Issues](https://github.com/furmangg/automating-azure-sql-dw/issues) tab to report bugs or post questions. Better yet, fix the problem yourself and propose changes... 152 | 153 | 154 | ### Proposing Changes 155 | 156 | Enhancements to code or documentation are welcome. Create a pull request. 157 | 158 |

159 | -------------------------------------------------------------------------------- /SQL/scale to new DWU and loop until it completes.sql: -------------------------------------------------------------------------------- 1 | --author: Greg Galloway 2 | --from: https://github.com/furmangg/automating-azure-sql-dw 3 | 4 | --to allow a user to scale the DW using ALTER DATABASE then you need this permission in master 5 | --ALTER ROLE dbmanager ADD MEMBER [UserNameHere]; 6 | --run this against master 7 | declare @DWU varchar(20) = 'DW200c'; 8 | 9 | --either set this to specific value or just assume there's only one DW on this server 10 | declare @db varchar(255); 11 | SELECT @db = db.[name] 12 | FROM 13 | sys.database_service_objectives ds 14 | JOIN sys.databases db ON ds.database_id = db.database_id 15 | where ds.edition = 'DataWarehouse'; 16 | 17 | declare @sql varchar(8000) = 'ALTER DATABASE ' + @db 18 | + ' MODIFY (SERVICE_OBJECTIVE = ''' + @DWU + ''')'; 19 | print 'starting executing ALTER DATABASE for ' + @db + ': ' + convert(varchar,getdate(),120); 20 | exec(@sql) 21 | print 'done executing ALTER DATABASE: ' + convert(varchar,getdate(),120); 22 | 23 | waitfor delay '00:00:02'; 24 | 25 | --PENDING = operation is waiting for resource or quota availability. 26 | --IN_PROGRESS = operation has started and is in progress. 27 | --COMPLETED = operation completed successfully. 28 | --FAILED = operation failed. See the error_desc column for details. 29 | --CANCELLED = operation stopped at the request of the user. 30 | while ( 31 | select top 1 state_desc 32 | from sys.dm_operation_status 33 | WHERE resource_type_desc = 'Database' 34 | AND major_resource_id = @db 35 | AND operation = 'ALTER DATABASE' 36 | order by start_time desc 37 | ) not in ('COMPLETED','FAILED','CANCELLED') 38 | begin 39 | print 'waiting for scaling to complete: ' + convert(varchar,getdate(),120); 40 | waitfor delay '00:00:30'; 41 | end 42 | 43 | declare @status nvarchar(120); 44 | declare @error_desc nvarchar(2048); 45 | select top 1 @status = state_desc, @error_desc = error_desc 46 | from sys.dm_operation_status 47 | WHERE resource_type_desc = 'Database' 48 | AND major_resource_id = @db 49 | AND operation = 'ALTER DATABASE' 50 | order by start_time desc; 51 | 52 | if @status in ('FAILED','CANCELLED') 53 | begin 54 | declare @errmsg nvarchar(2048) = 'Scaling did NOT succeed. Status=' + @status + '; Error=' + @error_desc; 55 | THROW 51000, @errmsg, 20; 56 | end 57 | 58 | print 'scaling complete!'; 59 | 60 | --wait! need to wait another minute as a subsequent statement will often fail 61 | waitfor delay '00:01:00'; 62 | -------------------------------------------------------------------------------- /automating-azure-sql-dw.sln: -------------------------------------------------------------------------------- 1 |  2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio 14 4 | VisualStudioVersion = 14.0.25420.1 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{E24C65DC-7377-472B-9ABA-BC803B73C61A}") = "C:", ".", "{CBE05ABA-F2CE-4573-8B28-C8E2350121F2}" 7 | ProjectSection(WebsiteProperties) = preProject 8 | TargetFrameworkMoniker = ".NETFramework,Version%3Dv4.0" 9 | Debug.AspNetCompiler.VirtualPath = "/localhost_55670" 10 | Debug.AspNetCompiler.PhysicalPath = "." 11 | Debug.AspNetCompiler.TargetPath = "PrecompiledWeb\localhost_55670\" 12 | Debug.AspNetCompiler.Updateable = "true" 13 | Debug.AspNetCompiler.ForceOverwrite = "true" 14 | Debug.AspNetCompiler.FixedNames = "false" 15 | Debug.AspNetCompiler.Debug = "True" 16 | Release.AspNetCompiler.VirtualPath = "/localhost_55670" 17 | Release.AspNetCompiler.PhysicalPath = "." 18 | Release.AspNetCompiler.TargetPath = "PrecompiledWeb\localhost_55670\" 19 | Release.AspNetCompiler.Updateable = "true" 20 | Release.AspNetCompiler.ForceOverwrite = "true" 21 | Release.AspNetCompiler.FixedNames = "false" 22 | Release.AspNetCompiler.Debug = "False" 23 | VWDPort = "55670" 24 | SlnRelativePath = "." 25 | StartServerOnDebug = "false" 26 | EndProjectSection 27 | EndProject 28 | Global 29 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 30 | Debug|Any CPU = Debug|Any CPU 31 | EndGlobalSection 32 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 33 | {CBE05ABA-F2CE-4573-8B28-C8E2350121F2}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 34 | EndGlobalSection 35 | GlobalSection(SolutionProperties) = preSolution 36 | HideSolutionNode = FALSE 37 | EndGlobalSection 38 | EndGlobal 39 | -------------------------------------------------------------------------------- /images/ADFMSI.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/furmangg/automating-azure-sql-dw/abfda68f1de82cfffaf065f0265266169280de65/images/ADFMSI.png -------------------------------------------------------------------------------- /images/THR2192 - Automating Azure SQL DW.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/furmangg/automating-azure-sql-dw/abfda68f1de82cfffaf065f0265266169280de65/images/THR2192 - Automating Azure SQL DW.pptx --------------------------------------------------------------------------------