├── .github ├── CODE_OF_CONDUCT.md ├── ISSUE_TEMPLATE.md └── PULL_REQUEST_TEMPLATE.md ├── CHANGELOG.md ├── CONTRIBUTING.md ├── LICENSE.md ├── README.md ├── architecture-diagram.png └── deploy ├── main.bicep └── modules ├── acr-role-assignment.bicep ├── acr.bicep ├── app.bicep ├── application-insights.bicep ├── data-factory-pipeline.bicep ├── data-factory.bicep ├── vm.bicep └── vnet.bicep /.github/CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Microsoft Open Source Code of Conduct 2 | 3 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 4 | 5 | Resources: 6 | 7 | - [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/) 8 | - [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) 9 | - Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns 10 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | 4 | > Please provide us with the following information: 5 | > --------------------------------------------------------------- 6 | 7 | ### This issue is for a: (mark with an `x`) 8 | ``` 9 | - [ ] bug report -> please search issues before submitting 10 | - [ ] feature request 11 | - [ ] documentation issue or request 12 | - [ ] regression (a behavior that used to work and stopped in a new release) 13 | ``` 14 | 15 | ### Minimal steps to reproduce 16 | > 17 | 18 | ### Any log messages given by the failure 19 | > 20 | 21 | ### Expected/desired behavior 22 | > 23 | 24 | ### OS and Version? 25 | > Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?) 26 | 27 | ### Versions 28 | > 29 | 30 | ### Mention any other details that might be useful 31 | 32 | > --------------------------------------------------------------- 33 | > Thanks! We'll be in touch soon. 34 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | ## Purpose 2 | 3 | * ... 4 | 5 | ## Does this introduce a breaking change? 6 | 7 | ``` 8 | [ ] Yes 9 | [ ] No 10 | ``` 11 | 12 | ## Pull Request Type 13 | What kind of change does this Pull Request introduce? 14 | 15 | 16 | ``` 17 | [ ] Bugfix 18 | [ ] Feature 19 | [ ] Code style update (formatting, local variables) 20 | [ ] Refactoring (no functional changes, no api changes) 21 | [ ] Documentation content changes 22 | [ ] Other... Please describe: 23 | ``` 24 | 25 | ## How to Test 26 | * Get the code 27 | 28 | ``` 29 | git clone [repo-address] 30 | cd [repo-name] 31 | git checkout [branch-name] 32 | npm install 33 | ``` 34 | 35 | * Test the code 36 | 37 | ``` 38 | ``` 39 | 40 | ## What to Check 41 | Verify that the following are valid 42 | * ... 43 | 44 | ## Other Information 45 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | ## [project-title] Changelog 2 | 3 | 4 | # x.y.z (yyyy-mm-dd) 5 | 6 | *Features* 7 | * ... 8 | 9 | *Bug Fixes* 10 | * ... 11 | 12 | *Breaking Changes* 13 | * ... 14 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to [project-title] 2 | 3 | This project welcomes contributions and suggestions. Most contributions require you to agree to a 4 | Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us 5 | the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com. 6 | 7 | When you submit a pull request, a CLA bot will automatically determine whether you need to provide 8 | a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions 9 | provided by the bot. You will only need to do this once across all repos using our CLA. 10 | 11 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 12 | For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or 13 | contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. 14 | 15 | - [Code of Conduct](#coc) 16 | - [Issues and Bugs](#issue) 17 | - [Feature Requests](#feature) 18 | - [Submission Guidelines](#submit) 19 | 20 | ## Code of Conduct 21 | Help us keep this project open and inclusive. Please read and follow our [Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 22 | 23 | ## Found an Issue? 24 | If you find a bug in the source code or a mistake in the documentation, you can help us by 25 | [submitting an issue](#submit-issue) to the GitHub Repository. Even better, you can 26 | [submit a Pull Request](#submit-pr) with a fix. 27 | 28 | ## Want a Feature? 29 | You can *request* a new feature by [submitting an issue](#submit-issue) to the GitHub 30 | Repository. If you would like to *implement* a new feature, please submit an issue with 31 | a proposal for your work first, to be sure that we can use it. 32 | 33 | * **Small Features** can be crafted and directly [submitted as a Pull Request](#submit-pr). 34 | 35 | ## Submission Guidelines 36 | 37 | ### Submitting an Issue 38 | Before you submit an issue, search the archive, maybe your question was already answered. 39 | 40 | If your issue appears to be a bug, and hasn't been reported, open a new issue. 41 | Help us to maximize the effort we can spend fixing issues and adding new 42 | features, by not reporting duplicate issues. Providing the following information will increase the 43 | chances of your issue being dealt with quickly: 44 | 45 | * **Overview of the Issue** - if an error is being thrown a non-minified stack trace helps 46 | * **Version** - what version is affected (e.g. 0.1.2) 47 | * **Motivation for or Use Case** - explain what are you trying to do and why the current behavior is a bug for you 48 | * **Browsers and Operating System** - is this a problem with all browsers? 49 | * **Reproduce the Error** - provide a live example or a unambiguous set of steps 50 | * **Related Issues** - has a similar issue been reported before? 51 | * **Suggest a Fix** - if you can't fix the bug yourself, perhaps you can point to what might be 52 | causing the problem (line of code or commit) 53 | 54 | You can file new issues by providing the above information at the corresponding repository's issues link: https://github.com/[organization-name]/[repository-name]/issues/new]. 55 | 56 | ### Submitting a Pull Request (PR) 57 | Before you submit your Pull Request (PR) consider the following guidelines: 58 | 59 | * Search the repository (https://github.com/[organization-name]/[repository-name]/pulls) for an open or closed PR 60 | that relates to your submission. You don't want to duplicate effort. 61 | 62 | * Make your changes in a new git fork: 63 | 64 | * Commit your changes using a descriptive commit message 65 | * Push your fork to GitHub: 66 | * In GitHub, create a pull request 67 | * If we suggest changes then: 68 | * Make the required updates. 69 | * Rebase your fork and force push to your GitHub repository (this will update your Pull Request): 70 | 71 | ```shell 72 | git rebase master -i 73 | git push -f 74 | ``` 75 | 76 | That's it! Thank you for your contribution! 77 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) Microsoft Corporation. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Azure Data Factory self-hosted integration runtime on App Service 2 | 3 | This sample illustrates how to host an [Azure Data Factory self-hosted integration runtime](https://docs.microsoft.com/azure/data-factory/concepts-integration-runtime) in Azure App Service. 4 | 5 | By using this approach, you can gain the benefits of using a self-hosted integration runtime while avoiding having to manage virtual machines or other infrastructure. 6 | 7 | ## Approach and architecture 8 | 9 | This sample runs the self-hosted integration in a Windows container on App Service. Azure Data Factory [supports running a self-hosted integration runtime on Windows containers](https://docs.microsoft.com/azure/data-factory/how-to-run-self-hosted-integration-runtime-in-windows-container), and [they provide a GitHub repository](https://github.com/Azure/Azure-Data-Factory-Integration-Runtime-in-Windows-Container) with a Dockerfile and associated scripts. Azure Container Registry builds the Dockerfile by using [ACR tasks](https://docs.microsoft.com/azure/container-registry/container-registry-tasks-overview). 10 | 11 | The App Service container app uses [VNet integration](https://docs.microsoft.com/azure/app-service/overview-vnet-integration) to connect to a virtual network. This means that the self-hosted integration runtime can [connect to Data Factory by using a private endpoint](https://docs.microsoft.com/azure/data-factory/data-factory-private-link), and it can also access servers and other resources that are accessible thorugh the virtual network. 12 | 13 | To illustrate the end-to-end flow, the sample deploys an example Data Factory pipeline that connects to a web server on a virtual machine by using a private IP address. 14 | 15 | ### Architecture diagram 16 | 17 | ![Architecture diagram](architecture-diagram.png) 18 | 19 | These are the data flows used by the solution: 20 | 21 | 1. When the app starts, App Service pulls the container image from the container registry. 22 | - The app uses a managed identity to pull the container image from the container registry. However, App Service requires that the `DOCKER_REGISTRY_SERVER_*` settings are in the app's settings. 23 | - [App Service doesn't support](https://azure.github.io/AppService/2021/07/03/Linux-container-from-ACR-with-private-endpoint.html#:~:text=Windows%20containers%20do%20not%20support%20pulling%20images%20over%20virtual%20network%20integration) pulling a Windows container image through a VNet-integrated container registry. 24 | 1. After the container is started, the self-hosted integration runtime loads. It connects to the data factory by using a private endpoint. 25 | 1. When the data factory's pipeline runs, the self-hosted integration runtime accesses the web server on the virtual machine. 26 | 27 | ## Deploy and test the sample 28 | 29 | The entire deployment is defined as a Bicep file, with a series of modules to deploy each part of the solution. 30 | 31 | To run the deployment, first create a resource group, such as by using the following Azure CLI command: 32 | 33 | ```azurecli 34 | az group create \ 35 | --name SHIR \ 36 | --location australiaeast 37 | ``` 38 | 39 | Next, initiate the deployment of the Bicep file. The only mandatory parameter is `vmAdminPassword`, which must be set to a value that conforms to the password naming rules for virtual machines. The following Azure CLI command initiates the deployment: 40 | 41 | ```azurecli 42 | az deployment group create \ 43 | --resource-group SHIR \ 44 | --template-file deploy/main.bicep \ 45 | --parameters 'vmAdminPassword=' ['irNodeExpirationTime='] 46 | ``` 47 | 48 | where the optional parameter `irNodeExpirationTime` specifies the time in seconds when the offline nodes expire after App Service stops or restarts. The expired nodes will be removed automatically during next restarting. The minimum expiration time, as well as the default value, is 600 seconds (10 minutes). 49 | 50 | The deployment takes approximately 30-45 minutes to complete. The majority of this time is the step to build the container image within Azure Container Registry. 51 | 52 | After the deployment completes, wait about another 10-15 minutes for App Service to deploy the container image. You can monitor the progress of this step by using the Deployment Center page on the App Service app resource in the Azure portal. 53 | 54 | To test the deployment when it's completed: 55 | 56 | 1. Open the Azure portal, navigate to the resource group (named *SHIR* by default), and open the data factory. 57 | 1. Select **Open Azure Data Factory Studio**. A separate page opens up in your browser. 58 | 1. On the left navigation bar, select **Manage**, and then select **Integration runtimes**. Look at the **self-hosted-runtime** item. The status should show as *Available*. If it says *Unavailable*, it probably means the container is still starting. Wait a few minutes and refresh the list. 59 | 1. On the left navigation bar, select **Author**. 60 | 1. Under **Pipelines**, select **sample-pipeline**. The pipeline opens. There's a single task named *GetWebContent*. 61 | 1. On the toolbar, select **Debug** to start the pipeline running. The run appears in the bottom pane. Wait a few moments and select the refresh button on the bottom pane's toolbar. 62 | 1. The task shows the *Succeeded* status. This means the self-hosted integration runtime successfully connected to the web server on the virtual machine and accessed its data. 63 | 64 | ## Note 65 | 66 | In this sample, we use App Service to host the container because the other serverless/PaaS container hosting options in Azure don't support VNet integration with Windows containers (at least at the time of writing, June 2022). 67 | -------------------------------------------------------------------------------- /architecture-diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure-Samples/azure-data-factory-runtime-app-service/16344762be12ad70d7ed0fd6f55c83d2efae3927/architecture-diagram.png -------------------------------------------------------------------------------- /deploy/main.bicep: -------------------------------------------------------------------------------- 1 | @description('The location into which the Azure resources should be deployed.') 2 | param location string = resourceGroup().location 3 | 4 | @description('The name of the container registry to create. This must be globally unique.') 5 | param containerRegistryName string = 'shir${uniqueString(resourceGroup().id)}' 6 | 7 | @description('The name of the virtual network to create.') 8 | param vnetName string = 'shirdemo' 9 | 10 | @description('The name of the data factory to create. This must be globally unique.') 11 | param dataFactoryName string = 'shirdemo${uniqueString(resourceGroup().id)}' 12 | 13 | @description('The name of the App Service application to create. This must be globally unique.') 14 | param appName string = 'app-${uniqueString(resourceGroup().id)}' 15 | 16 | @description('The SKU of the App Service plan to run the self-hosted integration runtime container.') 17 | param appServicePlanSku object = { 18 | name: 'P2v3' 19 | capacity: 1 20 | } 21 | 22 | @description('The port for nodes remote access.') 23 | param irNodeRemoteAccessPort int = 8060 24 | 25 | @description('The expiration time of the offline nodes in seconds. The value should not be less than 600.') 26 | param irNodeExpirationTime int = 600 27 | 28 | @description('The name of the SKU to use when creating the virtual machine.') 29 | param vmSize string = 'Standard_DS1_v2' 30 | 31 | @description('The type of disk and storage account to use for the virtual machine\'s OS disk.') 32 | param vmOSDiskStorageAccountType string = 'StandardSSD_LRS' 33 | 34 | @description('The administrator username to use for the virtual machine.') 35 | param vmAdminUsername string = 'shirdemoadmin' 36 | 37 | @description('The administrator password to use for the virtual machine.') 38 | @secure() 39 | param vmAdminPassword string 40 | 41 | // Deploy the container registry and build the container image. 42 | module acr 'modules/acr.bicep' = { 43 | name: 'acr' 44 | params: { 45 | name: containerRegistryName 46 | location: location 47 | } 48 | } 49 | 50 | // Deploy a virtual network with the subnets required for this solution. 51 | module vnet 'modules/vnet.bicep' = { 52 | name: 'vnet' 53 | params: { 54 | name: vnetName 55 | location: location 56 | } 57 | } 58 | 59 | // Deploy a virtual machine with a private web server. 60 | var vmImageReference = { 61 | publisher: 'MicrosoftWindowsServer' 62 | offer: 'WindowsServer' 63 | sku: '2019-Datacenter' 64 | version: 'latest' 65 | } 66 | 67 | module vm 'modules/vm.bicep' = { 68 | name: 'vm' 69 | params: { 70 | location: location 71 | subnetResourceId: vnet.outputs.vmSubnetResourceId 72 | vmSize: vmSize 73 | vmImageReference: vmImageReference 74 | vmOSDiskStorageAccountType: vmOSDiskStorageAccountType 75 | vmAdminUsername: vmAdminUsername 76 | vmAdminPassword: vmAdminPassword 77 | } 78 | } 79 | 80 | // Deploy the data factory. 81 | module adf 'modules/data-factory.bicep' = { 82 | name: 'adf' 83 | params: { 84 | dataFactoryName: dataFactoryName 85 | location: location 86 | virtualNetworkName: vnet.outputs.virtualNetworkName 87 | dataFactorySubnetResourceId: vnet.outputs.dataFactorySubnetResourceId 88 | } 89 | } 90 | 91 | // Deploy a Data Factory pipeline to connect to the private web server on the VM. 92 | module dataFactoryPipeline 'modules/data-factory-pipeline.bicep' = { 93 | name: 'adf-pipeline' 94 | params: { 95 | dataFactoryName: adf.outputs.dataFactoryName 96 | integrationRuntimeName: adf.outputs.integrationRuntimeName 97 | virtualMachinePrivateIPAddress: vm.outputs.virtualMachinePrivateIPAddress 98 | } 99 | } 100 | 101 | // Deploy Application Insights, which the App Service app uses. 102 | module applicationInsights 'modules/application-insights.bicep' = { 103 | name: 'application-insights' 104 | params: { 105 | location: location 106 | } 107 | } 108 | 109 | // Deploy the App Service app resources and deploy the container image from the container registry. 110 | module app 'modules/app.bicep' = { 111 | name: 'app' 112 | params: { 113 | location: location 114 | appName: appName 115 | appOutboundSubnetResourceId: vnet.outputs.appOutboundSubnetResourceId 116 | applicationInsightsInstrumentationKey: applicationInsights.outputs.instrumentationKey 117 | applicationInsightsConnectionString: applicationInsights.outputs.connectionString 118 | containerRegistryName: acr.outputs.containerRegistryName 119 | containerImageName: acr.outputs.containerImageName 120 | containerImageTag: acr.outputs.containerImageTag 121 | dataFactoryName: adf.outputs.dataFactoryName 122 | dataFactoryIntegrationRuntimeName: adf.outputs.integrationRuntimeName 123 | irNodeRemoteAccessPort: irNodeRemoteAccessPort 124 | irNodeExpirationTime: irNodeExpirationTime 125 | appServicePlanSku: appServicePlanSku 126 | } 127 | } 128 | -------------------------------------------------------------------------------- /deploy/modules/acr-role-assignment.bicep: -------------------------------------------------------------------------------- 1 | @description('The name of the container registry that the principal should be granted access to.') 2 | param containerRegistryName string 3 | 4 | @description('The Azure AD object ID to grant AcrPull access to.') 5 | param principalId string 6 | 7 | @description('This is the built-in AcrPull role. See https://docs.microsoft.com/azure/role-based-access-control/built-in-roles#acrpull') 8 | resource acrPullRoleDefinition 'Microsoft.Authorization/roleDefinitions@2018-01-01-preview' existing = { 9 | scope: subscription() 10 | name: '7f951dda-4ed3-4680-a7ca-43fe172d538d' 11 | } 12 | 13 | resource containerRegistry 'Microsoft.ContainerRegistry/registries@2021-06-01-preview' existing = { 14 | name: containerRegistryName 15 | } 16 | 17 | resource roleAssignment 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = { 18 | scope: containerRegistry 19 | name: guid(containerRegistry.id, principalId, acrPullRoleDefinition.id) 20 | properties: { 21 | roleDefinitionId: acrPullRoleDefinition.id 22 | principalId: principalId 23 | principalType: 'ServicePrincipal' 24 | } 25 | } 26 | -------------------------------------------------------------------------------- /deploy/modules/acr.bicep: -------------------------------------------------------------------------------- 1 | @description('The name of the container registry to create. This must be globally unique.') 2 | param name string = 'shir${uniqueString(resourceGroup().id)}' 3 | 4 | @description('The name of the SKU to use when creating the container registry.') 5 | param skuName string = 'Standard' 6 | 7 | @description('The location into which the Azure resources should be deployed.') 8 | param location string = resourceGroup().location 9 | 10 | resource containerRegistry 'Microsoft.ContainerRegistry/registries@2021-06-01-preview' = { 11 | name: name 12 | location: location 13 | sku: { 14 | name: skuName 15 | } 16 | properties: { 17 | adminUserEnabled: true 18 | } 19 | } 20 | 21 | // Build the container image. 22 | var containerImageName = 'adf/shir' 23 | var containerImageTag = 'v3' 24 | var dockerfileSourceGitRepository = 'https://github.com/Azure/Azure-Data-Factory-Integration-Runtime-in-Windows-Container.git' 25 | resource buildTask 'Microsoft.ContainerRegistry/registries/taskRuns@2019-06-01-preview' = { 26 | parent: containerRegistry 27 | name: 'buildTask' 28 | properties: { 29 | runRequest: { 30 | type: 'DockerBuildRequest' 31 | dockerFilePath: 'Dockerfile' 32 | sourceLocation: dockerfileSourceGitRepository 33 | imageNames: [ 34 | '${containerImageName}:${containerImageTag}' 35 | ] 36 | platform: { 37 | os: 'Windows' 38 | architecture: 'x86' 39 | } 40 | } 41 | } 42 | } 43 | 44 | output containerRegistryName string = containerRegistry.name 45 | output containerImageName string = containerImageName 46 | output containerImageTag string = containerImageTag 47 | -------------------------------------------------------------------------------- /deploy/modules/app.bicep: -------------------------------------------------------------------------------- 1 | @description('The name of the App Service plan to create.') 2 | param appServicePlanName string = 'shir-plan' 3 | 4 | @description('The location into which the Azure resources should be deployed.') 5 | param location string 6 | 7 | @description('The name of the App Service application to create. This must be globally unique.') 8 | param appName string 9 | 10 | @description('The resource ID of the subnet to use for outbound connections.') 11 | param appOutboundSubnetResourceId string 12 | 13 | @description('The name of the SKU to use when creating the virtual machine.') 14 | param appServicePlanSku object 15 | 16 | @description('The instrumentation key of the Application Insights resource to send telemetry to.') 17 | param applicationInsightsInstrumentationKey string 18 | 19 | @description('The connection string for the Application Insights resource to send telemetry to.') 20 | param applicationInsightsConnectionString string 21 | 22 | @description('The name of the container registry that has the container image.') 23 | param containerRegistryName string 24 | 25 | @description('The repository name of the container image within the registry.') 26 | param containerImageName string 27 | 28 | @description('The tag of the container image within the registry.') 29 | param containerImageTag string 30 | 31 | @description('The name of the data factory that the self-hosted integration runtime should connect to.') 32 | param dataFactoryName string 33 | 34 | @description('The name of the self-hosted integration runtime that the container should connect to.') 35 | param dataFactoryIntegrationRuntimeName string 36 | 37 | @description('The name of the node to create for the self-hosted integration runtime.') 38 | param dataFactoryIntegrationRuntimeNodeName string = 'AppServiceContainer' 39 | 40 | @description('The port for nodes remote access.') 41 | param irNodeRemoteAccessPort int 42 | 43 | @description('The expiration time of offline nodes in seconds. The value should not be less than 600.') 44 | param irNodeExpirationTime int 45 | 46 | var managedIdentityName = 'SampleApp' 47 | 48 | resource containerRegistry 'Microsoft.ContainerRegistry/registries@2021-06-01-preview' existing = { 49 | name: containerRegistryName 50 | } 51 | 52 | var containerRegistryHostName = '${containerRegistryName}.azurecr.io' 53 | var appWindowsFxVersion = 'DOCKER|${containerRegistryHostName}/${containerImageName}:${containerImageTag}' 54 | 55 | resource dataFactory 'Microsoft.DataFactory/factories@2018-06-01' existing = { 56 | name: dataFactoryName 57 | 58 | resource integrationRuntime 'integrationRuntimes' existing = { 59 | name: dataFactoryIntegrationRuntimeName 60 | } 61 | } 62 | 63 | var dataFactoryAuthKey = dataFactory::integrationRuntime.listAuthKeys().authKey1 64 | 65 | resource appServicePlan 'Microsoft.Web/serverfarms@2021-03-01' = { 66 | name: appServicePlanName 67 | location: location 68 | sku: appServicePlanSku 69 | properties: { 70 | hyperV: true 71 | } 72 | } 73 | 74 | resource managedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2018-11-30' = { 75 | name: managedIdentityName 76 | location: location 77 | } 78 | 79 | module appAcrRoleAssignment 'acr-role-assignment.bicep' = { 80 | name: 'app-acr-role-assignment' 81 | params: { 82 | containerRegistryName: containerRegistryName 83 | principalId: managedIdentity.properties.principalId 84 | } 85 | } 86 | 87 | resource app 'Microsoft.Web/sites@2021-03-01' = { 88 | name: appName 89 | location: location 90 | identity: { 91 | type: 'UserAssigned' 92 | userAssignedIdentities: { 93 | '${managedIdentity.id}': {} 94 | } 95 | } 96 | properties: { 97 | serverFarmId: appServicePlan.id 98 | virtualNetworkSubnetId: appOutboundSubnetResourceId 99 | siteConfig: { 100 | alwaysOn: true 101 | ftpsState: 'Disabled' 102 | appSettings: [ 103 | { 104 | name: 'APPINSIGHTS_INSTRUMENTATIONKEY' 105 | value: applicationInsightsInstrumentationKey 106 | } 107 | { 108 | name: 'APPLICATIONINSIGHTS_CONNECTION_STRING' 109 | value: applicationInsightsConnectionString 110 | } 111 | { 112 | name: 'ApplicationInsightsAgent_EXTENSION_VERSION' 113 | value: '~2' 114 | } 115 | { 116 | name: 'XDT_MicrosoftApplicationInsights_Mode' 117 | value: 'Recommended' 118 | } 119 | { 120 | name: 'XDT_MicrosoftApplicationInsights_Java' 121 | value: '1' 122 | } 123 | { 124 | // The URL for the container registry. 125 | name: 'DOCKER_REGISTRY_SERVER_URL' 126 | value: 'https://${containerRegistryHostName}' 127 | } 128 | { 129 | // The username to use when accessing the container registry. 130 | name: 'DOCKER_REGISTRY_SERVER_USERNAME' 131 | value: containerRegistry.listCredentials().username 132 | } 133 | { 134 | // The password to use when accessing the container registry. 135 | name: 'DOCKER_REGISTRY_SERVER_PASSWORD' 136 | value: containerRegistry.listCredentials().passwords[0].value 137 | } 138 | { 139 | // Disable persistent storage because we don't need it. 140 | name: 'WEBSITES_ENABLE_APP_SERVICE_STORAGE' 141 | value: 'false' 142 | } 143 | { 144 | // Disable container availability checks, because this runs as a background process. 145 | name: 'CONTAINER_AVAILABILITY_CHECK_MODE' 146 | value: 'Off' 147 | } 148 | { 149 | // The authentication key to use when connecting to Azure Data Factory. 150 | name: 'AUTH_KEY' 151 | value: dataFactoryAuthKey 152 | } 153 | { 154 | // The name of the node to create for the self-hosted integration runtime. 155 | name: 'NODE_NAME' 156 | value: dataFactoryIntegrationRuntimeNodeName 157 | } 158 | { 159 | name: 'ENABLE_HA' 160 | value: 'true' 161 | } 162 | { 163 | name: 'HA_PORT' 164 | value: '${irNodeRemoteAccessPort}' 165 | } 166 | { 167 | name: 'ENABLE_AE' 168 | value: 'true' 169 | } 170 | { 171 | name: 'AE_TIME' 172 | value: '${irNodeExpirationTime}' 173 | } 174 | ] 175 | // Use a user-assigned managed identity to connect to the container registry. (We still need the DOCKER_REGISTRY_SERVER_* settings in the appSettings array, though.) 176 | acrUseManagedIdentityCreds: true 177 | acrUserManagedIdentityID: managedIdentity.properties.clientId 178 | 179 | // The container image to deploy. 180 | windowsFxVersion: appWindowsFxVersion 181 | } 182 | } 183 | dependsOn: [ 184 | appAcrRoleAssignment // Wait for the role assignment so the app can pull from the container registry. 185 | ] 186 | } 187 | -------------------------------------------------------------------------------- /deploy/modules/application-insights.bicep: -------------------------------------------------------------------------------- 1 | @description('The name of the Application Insights resource to create.') 2 | param name string = 'shir-app-insights' 3 | 4 | @description('The location into which the Azure resources should be deployed.') 5 | param location string 6 | 7 | resource applicationInsights 'Microsoft.Insights/components@2020-02-02' = { 8 | name: name 9 | location: location 10 | kind: 'web' 11 | properties: { 12 | Application_Type: 'web' 13 | } 14 | } 15 | 16 | output instrumentationKey string = applicationInsights.properties.InstrumentationKey 17 | 18 | output connectionString string = applicationInsights.properties.ConnectionString 19 | -------------------------------------------------------------------------------- /deploy/modules/data-factory-pipeline.bicep: -------------------------------------------------------------------------------- 1 | @description('The name of the data factory that this pipeline should be added to.') 2 | param dataFactoryName string 3 | 4 | @description('The resource name of the self-hosted integration runtime that should be used to run this pipeline\'s activities.') 5 | param integrationRuntimeName string 6 | 7 | @description('The private IP address of the virtual machine that contains the private web server, which the pipeline will access.') 8 | param virtualMachinePrivateIPAddress string 9 | 10 | var pipelineName = 'sample-pipeline' 11 | 12 | resource dataFactory 'Microsoft.DataFactory/factories@2018-06-01' existing = { 13 | name: dataFactoryName 14 | 15 | resource integrationRuntime 'integrationRuntimes' existing = { 16 | name: integrationRuntimeName 17 | } 18 | } 19 | 20 | resource pipeline 'Microsoft.DataFactory/factories/pipelines@2018-06-01' = { 21 | parent: dataFactory 22 | name: pipelineName 23 | properties: { 24 | activities: [ 25 | { 26 | name: 'GetWebContent' 27 | type: 'WebActivity' 28 | typeProperties: { 29 | url: 'http://${virtualMachinePrivateIPAddress}/' 30 | connectVia: { 31 | referenceName: dataFactory::integrationRuntime.name 32 | type: 'IntegrationRuntimeReference' 33 | } 34 | method: 'GET' 35 | disableCertValidation: true 36 | } 37 | } 38 | ] 39 | } 40 | } 41 | -------------------------------------------------------------------------------- /deploy/modules/data-factory.bicep: -------------------------------------------------------------------------------- 1 | @description('The name of the data factory to create. This must be globally unique.') 2 | param dataFactoryName string 3 | 4 | @description('The location into which the Azure resources should be deployed.') 5 | param location string 6 | 7 | @description('The name of the virtual network.') 8 | param virtualNetworkName string 9 | 10 | @description('The resource ID of the subnet to use for the data factory\'s private endpoint.') 11 | param dataFactorySubnetResourceId string 12 | 13 | var integrationRuntimeName = 'self-hosted-runtime' 14 | var privateEndpointName = 'self-hosted-runtime-private-endpoint' 15 | var privateEndpointNicName = 'self-hosted-runtime-private-endpoint-nic' 16 | var privateDnsZoneName = 'privatelink.datafactory.azure.net' 17 | 18 | resource dataFactory 'Microsoft.DataFactory/factories@2018-06-01' = { 19 | name: dataFactoryName 20 | location: location 21 | properties: { 22 | publicNetworkAccess: 'Disabled' 23 | } 24 | } 25 | 26 | resource integrationRuntime 'Microsoft.DataFactory/factories/integrationRuntimes@2018-06-01' = { 27 | parent: dataFactory 28 | name: integrationRuntimeName 29 | properties: { 30 | type: 'SelfHosted' 31 | } 32 | } 33 | 34 | resource privateEndpoint 'Microsoft.Network/privateEndpoints@2021-05-01' = { 35 | name: privateEndpointName 36 | location: location 37 | properties: { 38 | subnet: { 39 | id: dataFactorySubnetResourceId 40 | } 41 | customNetworkInterfaceName: privateEndpointNicName 42 | privateLinkServiceConnections: [ 43 | { 44 | name: privateEndpointName 45 | properties: { 46 | privateLinkServiceId: dataFactory.id 47 | groupIds: [ 48 | 'dataFactory' 49 | ] 50 | } 51 | } 52 | ] 53 | } 54 | 55 | resource privateDnsZoneGroup 'privateDnsZoneGroups' = { 56 | name: 'default' 57 | properties: { 58 | privateDnsZoneConfigs: [ 59 | { 60 | name: 'privatelink-datafactory-azure-net' 61 | properties: { 62 | privateDnsZoneId: privateDnsZone.id 63 | } 64 | } 65 | ] 66 | } 67 | } 68 | } 69 | 70 | resource virtualNetwork 'Microsoft.Network/virtualNetworks@2021-08-01' existing = { 71 | name: virtualNetworkName 72 | } 73 | 74 | resource privateDnsZone 'Microsoft.Network/privateDnsZones@2020-06-01' = { 75 | name: privateDnsZoneName 76 | location: 'global' 77 | } 78 | 79 | resource privateDnsZoneLinkToVNet 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2018-09-01' = { 80 | parent: privateDnsZone 81 | name: 'link_to_${toLower(virtualNetwork.name)}' 82 | location: 'global' 83 | properties: { 84 | registrationEnabled: false 85 | virtualNetwork: { 86 | id: virtualNetwork.id 87 | } 88 | } 89 | } 90 | 91 | output dataFactoryName string = dataFactory.name 92 | 93 | output integrationRuntimeName string = integrationRuntimeName 94 | -------------------------------------------------------------------------------- /deploy/modules/vm.bicep: -------------------------------------------------------------------------------- 1 | @description('The location into which the Azure resources should be deployed.') 2 | param location string 3 | 4 | @description('The resource ID of the virtual network subnet that the VM should be deployed into.') 5 | param subnetResourceId string 6 | 7 | @description('The name of the SKU to use when creating the virtual machine.') 8 | param vmSize string 9 | 10 | @description('The details of the image to deploy on the virtual machine.') 11 | param vmImageReference object 12 | 13 | @description('The type of disk and storage account to use for the virtual machine\'s OS disk.') 14 | param vmOSDiskStorageAccountType string 15 | 16 | @description('The administrator username to use for the virtual machine.') 17 | param vmAdminUsername string 18 | 19 | @description('The administrator password to use for the virtual machine.') 20 | @secure() 21 | param vmAdminPassword string 22 | 23 | var vmName = 'MyVM' 24 | var vmNicName = 'MyVM-NIC' 25 | var vmOSDiskName = 'MyVM-OSDisk' 26 | 27 | resource vmNic 'Microsoft.Network/networkInterfaces@2021-08-01' = { 28 | name: vmNicName 29 | location: location 30 | properties: { 31 | ipConfigurations: [ 32 | { 33 | name: 'ipConfig' 34 | properties: { 35 | privateIPAllocationMethod: 'Dynamic' 36 | subnet: { 37 | id: subnetResourceId 38 | } 39 | } 40 | } 41 | ] 42 | } 43 | } 44 | 45 | resource vm 'Microsoft.Compute/virtualMachines@2022-03-01' = { 46 | name: vmName 47 | location: location 48 | properties: { 49 | hardwareProfile: { 50 | vmSize: vmSize 51 | } 52 | osProfile: { 53 | computerName: vmName 54 | adminUsername: vmAdminUsername 55 | adminPassword: vmAdminPassword 56 | } 57 | storageProfile: { 58 | imageReference: vmImageReference 59 | osDisk: { 60 | name: vmOSDiskName 61 | caching: 'ReadWrite' 62 | createOption: 'FromImage' 63 | managedDisk: { 64 | storageAccountType: vmOSDiskStorageAccountType 65 | } 66 | diskSizeGB: 128 67 | } 68 | } 69 | networkProfile: { 70 | networkInterfaces: [ 71 | { 72 | id: vmNic.id 73 | } 74 | ] 75 | } 76 | } 77 | 78 | resource installCustomScriptExtension 'extensions' = { 79 | name: 'InstallCustomScript' 80 | location: location 81 | properties: { 82 | publisher: 'Microsoft.Compute' 83 | type: 'CustomScriptExtension' 84 | typeHandlerVersion: '1.0' 85 | autoUpgradeMinorVersion: true 86 | protectedSettings: { 87 | commandToExecute: 'powershell -ExecutionPolicy Unrestricted Install-WindowsFeature -Name Web-Server -Restart' 88 | } 89 | } 90 | } 91 | } 92 | 93 | output virtualMachinePrivateIPAddress string = vmNic.properties.ipConfigurations[0].properties.privateIPAddress 94 | -------------------------------------------------------------------------------- /deploy/modules/vnet.bicep: -------------------------------------------------------------------------------- 1 | @description('The name of the virtual network to create.') 2 | param name string 3 | 4 | @description('The location into which the Azure resources should be deployed.') 5 | param location string 6 | 7 | @description('The address prefix of the entire virtual network.') 8 | param vnetAddressPrefix string = '10.0.0.0/16' 9 | 10 | @description('The address prefix of the subnet for outbound communication from the application.') 11 | param appOutboundSubnetAddressPrefix string = '10.0.0.0/24' 12 | 13 | @description('The address prefix of the subnet for the data factory\'s private endpoint.') 14 | param dataFactorySubnetAddressPrefix string = '10.0.1.0/24' 15 | 16 | @description('The address prefix of the subnet for the virtual machine with the private web server.') 17 | param vmSubnetAddressPrefix string = '10.0.2.0/24' 18 | 19 | var appOutboundSubnetName = 'app-outbound' 20 | var dataFactorySubnetName = 'data-factory' 21 | var vmSubnetName = 'vm' 22 | 23 | resource virtualNetwork 'Microsoft.Network/virtualNetworks@2021-08-01' = { 24 | name: name 25 | location: location 26 | properties: { 27 | addressSpace: { 28 | addressPrefixes: [ 29 | vnetAddressPrefix 30 | ] 31 | } 32 | subnets: [ 33 | { 34 | name: appOutboundSubnetName 35 | properties: { 36 | addressPrefix: appOutboundSubnetAddressPrefix 37 | delegations: [ 38 | { 39 | name: 'delegation' 40 | properties: { 41 | serviceName: 'Microsoft.Web/serverfarms' 42 | } 43 | } 44 | ] 45 | serviceEndpoints: [ 46 | { 47 | service: 'Microsoft.Storage' 48 | } 49 | ] 50 | } 51 | } 52 | { 53 | name: dataFactorySubnetName 54 | properties: { 55 | addressPrefix: dataFactorySubnetAddressPrefix 56 | privateEndpointNetworkPolicies: 'Enabled' 57 | } 58 | } 59 | { 60 | name: vmSubnetName 61 | properties: { 62 | addressPrefix: vmSubnetAddressPrefix 63 | } 64 | } 65 | ] 66 | } 67 | 68 | resource appOutboundSubnet 'subnets' existing = { 69 | name: appOutboundSubnetName 70 | } 71 | 72 | resource dataFactorySubnet 'subnets' existing = { 73 | name: dataFactorySubnetName 74 | } 75 | 76 | resource vmSubnet 'subnets' existing = { 77 | name: vmSubnetName 78 | } 79 | } 80 | 81 | output virtualNetworkName string = virtualNetwork.name 82 | 83 | output appOutboundSubnetResourceId string = virtualNetwork::appOutboundSubnet.id 84 | 85 | output dataFactorySubnetResourceId string = virtualNetwork::dataFactorySubnet.id 86 | 87 | output vmSubnetResourceId string = virtualNetwork::vmSubnet.id 88 | --------------------------------------------------------------------------------