├── .gitignore ├── CODE_OF_CONDUCT.md ├── LICENSE ├── README.md ├── SECURITY.md ├── assets └── privateaml_architecture.png ├── samples ├── runner.ipynb └── train.py ├── setup.md └── terraform ├── .terraform.lock.hcl ├── acr ├── acr.tf ├── output.tf └── variables.tf ├── admin-jumpbox ├── output.tf ├── variables.tf └── vm.tf ├── aml ├── aml.tf ├── output.tf └── variables.tf ├── azure-monitor ├── ampls.json ├── azure-monitor.tf ├── output.tf └── variables.tf ├── bastion ├── bastion.tf └── variables.tf ├── firewall ├── data.tf ├── firewall.tf ├── locals.tf ├── output.tf └── variables.tf ├── keyvault ├── keyvault.tf ├── output.tf └── variables.tf ├── locals.tf ├── main.tf ├── network ├── locals.tf ├── network.tf ├── network_security_groups.tf ├── output.tf └── variables.tf ├── outputs.tf ├── routetable ├── routetable.tf └── variables.tf ├── storage ├── output.tf ├── storage.tf └── variables.tf ├── terraform.tfvars.tmpl └── variables.tf /.gitignore: -------------------------------------------------------------------------------- 1 | # Local .terraform directories 2 | **/.terraform/* 3 | 4 | # .tfstate files 5 | *.tfstate 6 | *.tfstate.* 7 | 8 | # .tfvars files 9 | *.tfvars -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Microsoft Open Source Code of Conduct 2 | 3 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 4 | 5 | Resources: 6 | 7 | - [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/) 8 | - [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) 9 | - Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns 10 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) Microsoft Corporation. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Private Azure ML deployment 2 | 3 | ## Project Status 4 | 5 | There is a need across industries to enable researchers, analysts, and developers to work with sensitive data sets, while still leveraging benefits coming from the managed public cloud solutions. Setting up an environment that enforces a secure boundary around resources, prevents the exfiltration of sensitive data and enables information governance controls to be enforced, can be a time consuming task. 6 | 7 | This project provides Infrastructure as Code scripts for a quick and easy deployment of such an environment, based around privately enclosed Azure Machine Learning, with all the required services pre-connected. Applying the Terraform script with default parameters, will configure a fully functioning private AML environment, that can then be either used as such or connected to internal network via VPN. 8 | 9 | While the scripts provided in this repository will get you a quick-start with secure AML environment, for a full production scale Trusted Research Environment you should consider using the [AzureTRE](https://github.com/microsoft/AzureTRE/), that this repository is a spin-off. AzureTRE is a solution accelerator aiming to be a great starting point for a customized Trusted Research Environment, allowing users to customize and deploy fully isolated research environments, one type being AzureML. 10 | 11 | This project does not have a dedicated team of maintainers but relies on you and the community to maintain and enhance the solution. No guarantees can be offered as to response times on issues, feature requests, or to the long term road map for the project. 12 | 13 | It is important before deployment of the solution that the [Support Policy](SUPPORT.md) is read and understood. 14 | 15 | ## Architecture 16 | 17 | ![Architecture overview](/assets/privateaml_architecture.png) 18 | 19 | ## Getting started 20 | 21 | Pre-requirements: 22 | 23 | * [Azure CLI (az) installed](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) 24 | * [Terraform installed](https://learn.hashicorp.com/tutorials/terraform/install-cli?in=terraform/azure-get-started) 25 | * [Git installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) 26 | 27 | Commands in this guide are Linux commands, but you can use PowerShell as well, or any other command line tool. 28 | 29 | To provision AML resources: 30 | 31 | 1. Clone this repository 32 | 33 | ```git clone https://github.com/microsoft/privateAML``` 34 | 1. Switch to terraform directory: 35 | 36 | ```cd privateAML/terraform/``` 37 | 1. Copy parameter template file terraform.tfvars.tmpl to terraform.tfvars: 38 | 39 | ```cp terraform.tfvars.tmpl terraform.tfvars``` 40 | 1. Modify the parameters in terraform.tfvars and save the file (you can skip this step if you are OK with defaults): 41 | 42 | ```nano terraform.tfvars``` 43 | 44 | | Parameter | Description | 45 | | --- | --- | 46 | | Name | A 4 character name you can give to the resources. It will be combined with 4 random numbers to make names globally unique. If you would like to adjust the naming pattern, just adjust [locals.tf](./terraform/locals.tf) file where final name is generated. Default is "test". | 47 | | Location | Azure region/location where resources should be deployed, e.g. westus, eastus, northeurope, etc. Default is "westeurope" | 48 | | VNET_address_space | A network IP range that will be used to create the VNET and subnets used by the service. Default is "10.1.0.0/22." | 49 | 50 | 1. Initialize terraform: 51 | 52 | ```terraform init``` 53 | 1. Deploy resources: 54 | 55 | ```terraform apply -auto-approve``` 56 | 1. Once the resources are deployed you can fetch user name and password for the Jumpbox (marked as sensitive so blocked from outputting after terraform apply) by taking json output from terraform: 57 | 58 | ```terraform output -json``` 59 | 60 | ## Route option 1 - accessing the environment through Jumpbox 61 | 62 | The Jumpbox provides an option for a single user to access the secure AML environment. This option is suitable for quick testing and proofs of concept. It is however not suitable for a multi-user deployment, and you should consider connecting the VNET to your on-premises network with VPN (see Route option 2 below). 63 | 64 | 1. Connect to the Jumpbox by navigating to Azure portal, select the VM created, press Connect and choose Bastion. Enter the user name and password retrieved from the previous step. From the Jumpbox you are able to access all of the resources residing on private network (e.g. ml.azure.com) 65 | 1. (Optional): If your organization requires device enrollment before accessing corporate resources (i.e. if you see an error "You can't get there from here." or "This device does not meet your organization's compliance requirements"), enroll the Jumpbox to Azure AD by following the steps in Edge: open Edge and click "Sign in to sync data", select "Work or school account", and then press OK on "Allow my organization to manage my device". It takes a few minutes for the policies to be applied, device scanned and confirmed as secure to access corporate resources. You will know that the process is complete, once you are able to access the [https://ml.azure.com](https://ml.azure.com). 66 | 67 | ## Route option 2 - accessing the environment over VPN connection 68 | 69 | For a scalable, multi-user and a user-friendly secure connection to the provisioned AML resources, establish a VPN connection between your on-premises network and VNET that has been provisioned. 70 | 71 | 1. Follow the steps in this guide: [https://docs.microsoft.com/en-us/azure/vpn-gateway/tutorial-site-to-site-portal](https://docs.microsoft.com/en-us/azure/vpn-gateway/tutorial-site-to-site-portal) to create a Hub VNET, VPN Gateway and establish a connection to the on-premises network. 72 | 1. After completing steps above, peer the AML VNET (spoke) to the Hub VNET created in the previous step. To do this, from the Azure Portal, click on the AML VNET, select Peerings - Add then select the Hub VNET created in the previous step. 73 | 1. Test the connection from the on-premises, by opening your browser and navigating to [https://ml.azure.com](https://ml.azure.com) 74 | 75 | ## Using the environment 76 | 77 | 1. Navigate to the Azure ML at [https://ml.azure.com](https://ml.azure.com) 78 | 1. Select the ML workspace that was created (e.g. "ml-test-1234") 79 | 1. Switch to "Compute" tab and click on New to create a new Compute Instance. 80 | 1. Click on Advanced Settings and select the Virtual Network, Subnet, and enable "No public IP". Press Create to create the Compute Instance. 81 | 1. Switch over to "Compute clusters" tab, and click on New. 82 | 1. Click Next, enter the "Compute name" and then select the Virtual Network, Subnet, and enable "No public IP". Enable the "Assign a managed identity" option with "System assigned": this will allow compute cluster to connect to the ACR. 83 | 1. Once the compute instance is created, navigate to Notebooks and create a new training and runner notebooks. You can find the sample notebooks in the [./samples](./samples) folder. 84 | 1. Open the runner notebook and execute the cells. 85 | 86 | ## Background 87 | 88 | While AzureML can be easily deployed with a few clicks as a quick public research platform, the default setup doesn't meet the security needs of many organizations. Things like controlled data ingestion, network isolation and prevention of exfiltration of sensitive data are left to the system admins to configure. 89 | 90 | Since this is a rather common scenario, and the task of creating a secure ML environment is time consuming, we saw a need for a script style accelerator that can be easily customized and reused. 91 | 92 | ## Network isolation details 93 | 94 | * All provisioned components have either a private IP or a private endpoint connection, and reside on same VNET 95 | * All inbound traffic is blocked 96 | * Outbound traffic is limited and only access to the following URLs are allowed: 97 | 98 | | URL | Reason | 99 | | --- | --- | 100 | | ml.azure.com | AML Portal | 101 | | viennaglobal.azurecr.io | Required by AML | 102 | | *openml.org | Required by AML | 103 | | enterpriseregistration.windows.net | Required for device enrollment to AAD | 104 | | 169.254.169.254 | Required for device enrollment to AAD | 105 | | login.microsoftonline.com | Required for device enrollment to AAD | 106 | | pas.windows.net | Required for device enrollment to AAD | 107 | | *manage-beta.microsoft.com | Required for device enrollment to AAD | 108 | | *manage.microsoft.com | Required for device enrollment to AAD | 109 | | login.windows.net | Required for device enrollment to AAD | 110 | | *.azureedge.net | Required for device enrollment to AAD | 111 | | go.microsoft.com | Required for device enrollment to AAD | 112 | | msft.sts.microsoft.com | Required for device enrollment to AAD | 113 | | *github.com | Access to github | 114 | | *githubassets.com | Access to github | 115 | | git-scm.com | Access to github | 116 | | *githubusercontent.com | Access to github | 117 | | *core.windows.net | Access to Azure External Storage | 118 | | aka.ms | Access to Microsoft short links | 119 | | *powershellgallery.com | Access to PowerShellGallery | 120 | | management.azure.com | Access to the Azure management plane | 121 | | graph.microsoft.com | Access to the Microsoft Graph | 122 | | graph.windows.net | Access to the Microsoft Graph | 123 | | aadcdn.msftauth.net | Required for Azure auth | 124 | | azure.archive.ubuntu.com | Azure packages for Ubuntu | 125 | | repo.anaconda.com | Conda repository | 126 | | conda.anaconda.org | Conda repository | 127 | | binstar-cio-packages-prod.s3.amazonaws.com | Anaconda packages stored on Amazon S3 | 128 | 129 | 130 | * A network security group is setup on Shared Subnet, with following security rules: 131 | 132 | | Direction | Action | Port | Source | Destination | 133 | | --- | --- | --- | --- | --- | 134 | | Inbound | Allow | 29877,29876 | BatchNodeManagement | VirtualNetwork | 135 | | Inbound | Allow | 44224 | Internet | VirtualNetwork | 136 | | Inbound | Allow | Any | AzureLoadBalancer | VirtualNetwork | 137 | | Inbound | Deny | Any | Any | Any | 138 | | Outbound | Allow | 445 | VirtualNetwork | Storage | 139 | | Outbound | Allow | Any | VirtualNetwork | Storage | 140 | | Outbound | Allow | Any | Any | Internet | 141 | 142 | ## Contributing 143 | 144 | This project welcomes contributions and suggestions. Most contributions require you to agree to a 145 | Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us 146 | the rights to use your contribution. For details, visit [Contributor License Agreement](https://cla.opensource.microsoft.com). 147 | 148 | When you submit a pull request, a CLA bot will automatically determine whether you need to provide 149 | a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions 150 | provided by the bot. You will only need to do this once across all repos using our CLA. 151 | 152 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 153 | For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or 154 | contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. 155 | 156 | ## Trademarks 157 | 158 | This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). 159 | Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. 160 | Any use of third-party trademarks or logos are subject to those third-party's policies. 161 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | ## Security 4 | 5 | Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/). 6 | 7 | If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://docs.microsoft.com/en-us/previous-versions/tn-archive/cc751383(v=technet.10)), please report it to us as described below. 8 | 9 | ## Reporting Security Issues 10 | 11 | **Please do not report security vulnerabilities through public GitHub issues.** 12 | 13 | Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://msrc.microsoft.com/create-report). 14 | 15 | If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://www.microsoft.com/en-us/msrc/pgp-key-msrc). 16 | 17 | You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://www.microsoft.com/msrc). 18 | 19 | Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue: 20 | 21 | * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.) 22 | * Full paths of source file(s) related to the manifestation of the issue 23 | * The location of the affected source code (tag/branch/commit or direct URL) 24 | * Any special configuration required to reproduce the issue 25 | * Step-by-step instructions to reproduce the issue 26 | * Proof-of-concept or exploit code (if possible) 27 | * Impact of the issue, including how an attacker might exploit the issue 28 | 29 | This information will help us triage your report more quickly. 30 | 31 | If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://microsoft.com/msrc/bounty) page for more details about our active programs. 32 | 33 | ## Preferred Languages 34 | 35 | We prefer all communications to be in English. 36 | 37 | ## Policy 38 | 39 | Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://www.microsoft.com/en-us/msrc/cvd). 40 | 41 | -------------------------------------------------------------------------------- /assets/privateaml_architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/privateAML/dd53700351697c22961313db1bfafa5069808073/assets/privateaml_architecture.png -------------------------------------------------------------------------------- /samples/runner.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 3, 6 | "metadata": { 7 | "gather": { 8 | "logged": 1637311613475 9 | } 10 | }, 11 | "outputs": [], 12 | "source": [ 13 | "from azureml.core import Workspace\n", 14 | "from azureml.core import Environment\n", 15 | "\n", 16 | "ws = Workspace.from_config()\n", 17 | "test_env = Environment(\"test_env\")" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 4, 23 | "metadata": { 24 | "gather": { 25 | "logged": 1637311616230 26 | }, 27 | "jupyter": { 28 | "outputs_hidden": false, 29 | "source_hidden": false 30 | }, 31 | "nteract": { 32 | "transient": { 33 | "deleting": false 34 | } 35 | } 36 | }, 37 | "outputs": [], 38 | "source": [ 39 | "# Build a customer docker image by creating a dockerfile here:\n", 40 | "dockerfile = r\"\"\"\n", 41 | "FROM mcr.microsoft.com/azureml/openmpi3.1.2-ubuntu18.04:20210615.v1\n", 42 | "RUN echo \"Hello from custom container!\"\n", 43 | "\"\"\"\n", 44 | "\n", 45 | "# Set the base image to None, because the image is defined by Dockerfile.\n", 46 | "test_env.docker.base_image = None\n", 47 | "test_env.docker.base_dockerfile = dockerfile" 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": 5, 53 | "metadata": { 54 | "gather": { 55 | "logged": 1637311618827 56 | }, 57 | "jupyter": { 58 | "outputs_hidden": false, 59 | "source_hidden": false 60 | }, 61 | "nteract": { 62 | "transient": { 63 | "deleting": false 64 | } 65 | } 66 | }, 67 | "outputs": [], 68 | "source": [ 69 | "from azureml.core.compute import ComputeTarget, AmlCompute\n", 70 | "from azureml.core.compute_target import ComputeTargetException\n", 71 | "\n", 72 | "# Enter the name of your cluster.\n", 73 | "cluster_name = \"cluster\"\n", 74 | "\n", 75 | "compute_target = ComputeTarget(workspace=ws, name=cluster_name)" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": 8, 81 | "metadata": { 82 | "gather": { 83 | "logged": 1637311667736 84 | }, 85 | "jupyter": { 86 | "outputs_hidden": false, 87 | "source_hidden": false 88 | }, 89 | "nteract": { 90 | "transient": { 91 | "deleting": false 92 | } 93 | } 94 | }, 95 | "outputs": [], 96 | "source": [ 97 | "from azureml.core import ScriptRunConfig\n", 98 | "\n", 99 | "src = ScriptRunConfig(source_directory='.',\n", 100 | " script='train.py',\n", 101 | " compute_target=compute_target,\n", 102 | " environment=test_env)" 103 | ] 104 | }, 105 | { 106 | "cell_type": "code", 107 | "execution_count": 9, 108 | "metadata": { 109 | "jupyter": { 110 | "outputs_hidden": false, 111 | "source_hidden": false 112 | }, 113 | "nteract": { 114 | "transient": { 115 | "deleting": false 116 | } 117 | } 118 | }, 119 | "outputs": [], 120 | "source": [ 121 | "from azureml.core import Experiment\n", 122 | "\n", 123 | "run = Experiment(ws,'Demo').submit(src)\n", 124 | "run.wait_for_completion(show_output=True)" 125 | ] 126 | } 127 | ], 128 | "metadata": { 129 | "kernel_info": { 130 | "name": "python3-azureml" 131 | }, 132 | "kernelspec": { 133 | "display_name": "Python 3.6 - AzureML", 134 | "language": "python", 135 | "name": "python3-azureml" 136 | }, 137 | "language_info": { 138 | "codemirror_mode": { 139 | "name": "ipython", 140 | "version": 3 141 | }, 142 | "file_extension": ".py", 143 | "mimetype": "text/x-python", 144 | "name": "python", 145 | "nbconvert_exporter": "python", 146 | "pygments_lexer": "ipython3", 147 | "version": "3.6.9" 148 | }, 149 | "nteract": { 150 | "version": "nteract-front-end@1.0.0" 151 | } 152 | }, 153 | "nbformat": 4, 154 | "nbformat_minor": 0 155 | } 156 | -------------------------------------------------------------------------------- /samples/train.py: -------------------------------------------------------------------------------- 1 | #Enter your training script here -------------------------------------------------------------------------------- /setup.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/privateAML/dd53700351697c22961313db1bfafa5069808073/setup.md -------------------------------------------------------------------------------- /terraform/.terraform.lock.hcl: -------------------------------------------------------------------------------- 1 | # This file is maintained automatically by "terraform init". 2 | # Manual edits may be lost in future updates. 3 | 4 | provider "registry.terraform.io/hashicorp/azurerm" { 5 | version = "3.9.0" 6 | constraints = "3.9.0" 7 | hashes = [ 8 | "h1:wWzuH8s+OrS2OH4dZdryUbsb6apY/s3uOFJSIcS9S34=", 9 | "zh:23393a7546297332ee28cf122df16afc5087709c441c7df2fd0df09f6f8a8bf5", 10 | "zh:30922c40c84922a49137e36a2bb24b0af1c6ce1da4d2497d18546585287a8aee", 11 | "zh:31b29b2ebdf6259b1231517ec00739c6432fbabe9c9ca1d29323b70e00843936", 12 | "zh:49b97e3143e4af063e5775e168715609a2a8b8eb110d925a3a7f5446b3adc50c", 13 | "zh:57a92431556c6d65990c6b5852eca598155c5720055ddc4b8b0ce85c3de9d845", 14 | "zh:664fa16b2e2c6b05454038c35e5d579c0c34d91886092dcc2c516fc1e45c46a9", 15 | "zh:84fa1464160233b11efb53df47aaa696d8f4d9e93cac7707942272e5544ab609", 16 | "zh:a8b308f50c3ac353ddbdbc0b7aba0276790313e71b061e65e46cc1c9e4bb99e7", 17 | "zh:be50de973e8e8bd759719295108c2fbaa7aa5529fdeba9609d74623a969b97f6", 18 | "zh:be80d6e93e97c486c5adb24e9a8b485e681226941772824bb9a1a6a2767c28ad", 19 | "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c", 20 | "zh:f9f226852dcf06a79bf81df8702f91b67b6b3111b9f33f93f643cf6359ec09bb", 21 | ] 22 | } 23 | 24 | provider "registry.terraform.io/hashicorp/local" { 25 | version = "2.2.3" 26 | hashes = [ 27 | "h1:aWp5iSUxBGgPv1UnV5yag9Pb0N+U1I0sZb38AXBFO8A=", 28 | "zh:04f0978bb3e052707b8e82e46780c371ac1c66b689b4a23bbc2f58865ab7d5c0", 29 | "zh:6484f1b3e9e3771eb7cc8e8bab8b35f939a55d550b3f4fb2ab141a24269ee6aa", 30 | "zh:78a56d59a013cb0f7eb1c92815d6eb5cf07f8b5f0ae20b96d049e73db915b238", 31 | "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", 32 | "zh:8aa9950f4c4db37239bcb62e19910c49e47043f6c8587e5b0396619923657797", 33 | "zh:996beea85f9084a725ff0e6473a4594deb5266727c5f56e9c1c7c62ded6addbb", 34 | "zh:9a7ef7a21f48fabfd145b2e2a4240ca57517ad155017e86a30860d7c0c109de3", 35 | "zh:a63e70ac052aa25120113bcddd50c1f3cfe61f681a93a50cea5595a4b2cc3e1c", 36 | "zh:a6e8d46f94108e049ad85dbed60354236dc0b9b5ec8eabe01c4580280a43d3b8", 37 | "zh:bb112ce7efbfcfa0e65ed97fa245ef348e0fd5bfa5a7e4ab2091a9bd469f0a9e", 38 | "zh:d7bec0da5c094c6955efed100f3fe22fca8866859f87c025be1760feb174d6d9", 39 | "zh:fb9f271b72094d07cef8154cd3d50e9aa818a0ea39130bc193132ad7b23076fd", 40 | ] 41 | } 42 | 43 | provider "registry.terraform.io/hashicorp/random" { 44 | version = "3.3.1" 45 | hashes = [ 46 | "h1:0gchydbWoBlDXzMdpFztkkCxzAbqQ/BNxb2XwYcTJiw=", 47 | "zh:0af603dc14d0f7ec900b885f7e46226e5743323ce6fc25437738aae20906a799", 48 | "zh:0de2d8f185b006c8928c18e7374ba0ca1df5bbc8c0dc492fb1e539c3184b7472", 49 | "zh:118600e801bf73003ad2c57106564a5abdfe3b0e660b05b595e6884a009f32bd", 50 | "zh:4d7ff20cc1344040911b197741a364c20a51d31ea6c746ce77b0454ad96b9733", 51 | "zh:58b6443bdf638864bf32e580b60e1c811e0b38060d2dcc3a438ae4d83360300d", 52 | "zh:6b4418698a62a39dc9f6ac82f0a48bc115a4ab409435ba918511837f02817f44", 53 | "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", 54 | "zh:a1a43952a4c4cb6970b3cc4bef8c5da74ced36bd4f451e9850074b464ab4ed21", 55 | "zh:c88095d4ca8cd30ccded57d39e55963f35044ffafebd56a2dc8568730edaf51a", 56 | "zh:c955d31c41f4a13c1dd4290afa84c762282998f9afc110970dd288cc6fac5847", 57 | "zh:d80e9b1d0f45e8377a20fe6280722ef6a49eb63d0aa0c71a2fd5a0817ec60458", 58 | "zh:da26a6f89d595e0416c99f4ad288f23b52490ab44882af8c8d9a8ef56c938c70", 59 | ] 60 | } 61 | -------------------------------------------------------------------------------- /terraform/acr/acr.tf: -------------------------------------------------------------------------------- 1 | resource "azurerm_container_registry" "acr" { 2 | name = "acr${var.name}" 3 | location = var.location 4 | resource_group_name = var.resource_group_name 5 | sku = "Premium" 6 | admin_enabled = false 7 | 8 | public_network_access_enabled = false 9 | 10 | network_rule_set { 11 | default_action = "Allow" 12 | ip_rule = [ 13 | { 14 | action = "Allow" 15 | ip_range = "13.69.64.88/29" 16 | }, 17 | { 18 | action = "Allow" 19 | ip_range = "13.69.106.80/29" 20 | }, 21 | { 22 | action = "Allow" 23 | ip_range = "13.69.110.0/24" 24 | }, 25 | { 26 | action = "Allow" 27 | ip_range = "13.69.110.0/24" 28 | }, 29 | { 30 | action = "Allow" 31 | ip_range = "13.69.112.192/26" 32 | }, 33 | { 34 | action = "Allow" 35 | ip_range = "20.50.200.0/24" 36 | }, 37 | { 38 | action = "Allow" 39 | ip_range = "20.61.97.128/25" 40 | }, 41 | { 42 | action = "Allow" 43 | ip_range = "52.178.18.0/23" 44 | }, 45 | { 46 | action = "Allow" 47 | ip_range = "52.178.20.0/24" 48 | }, 49 | { 50 | action = "Allow" 51 | ip_range = "52.236.186.80/29" 52 | }, 53 | { 54 | action = "Allow" 55 | ip_range = "52.236.191.0/24" 56 | } 57 | ] 58 | } 59 | 60 | lifecycle { ignore_changes = [tags] } 61 | } 62 | 63 | resource "azurerm_private_dns_zone" "azurecr" { 64 | name = "privatelink.azurecr.io" 65 | resource_group_name = var.resource_group_name 66 | 67 | lifecycle { ignore_changes = [tags] } 68 | } 69 | 70 | resource "azurerm_private_dns_zone_virtual_network_link" "acrlink" { 71 | name = "acrcorelink" 72 | resource_group_name = var.resource_group_name 73 | private_dns_zone_name = azurerm_private_dns_zone.azurecr.name 74 | virtual_network_id = var.core_vnet 75 | 76 | lifecycle { ignore_changes = [tags] } 77 | } 78 | 79 | resource "azurerm_private_endpoint" "acrpe" { 80 | name = "acrpe-${azurerm_container_registry.acr.name}" 81 | location = var.location 82 | resource_group_name = var.resource_group_name 83 | subnet_id = var.shared_subnet 84 | 85 | lifecycle { ignore_changes = [tags] } 86 | 87 | private_dns_zone_group { 88 | name = "private-dns-zone-group" 89 | private_dns_zone_ids = [azurerm_private_dns_zone.azurecr.id] 90 | } 91 | 92 | private_service_connection { 93 | name = "acrpesc-${azurerm_container_registry.acr.name}" 94 | private_connection_resource_id = azurerm_container_registry.acr.id 95 | is_manual_connection = false 96 | subresource_names = ["registry"] 97 | } 98 | } 99 | 100 | resource "azurerm_monitor_diagnostic_setting" "acrdiagnostic" { 101 | name = "diagnostics-acr-${var.name}" 102 | target_resource_id = azurerm_container_registry.acr.id 103 | log_analytics_workspace_id = var.log_analytics_workspace_id 104 | 105 | log { 106 | category = "ContainerRegistryRepositoryEvents" 107 | enabled = true 108 | 109 | retention_policy { 110 | enabled = true 111 | } 112 | } 113 | 114 | log { 115 | category = "ContainerRegistryLoginEvents" 116 | enabled = true 117 | 118 | retention_policy { 119 | enabled = true 120 | } 121 | } 122 | 123 | metric { 124 | category = "AllMetrics" 125 | 126 | retention_policy { 127 | enabled = true 128 | } 129 | } 130 | } 131 | 132 | resource "azurerm_monitor_diagnostic_setting" "acrpediagnostic" { 133 | name = "diagnostics-acr-pe-${var.name}" 134 | target_resource_id = azurerm_private_endpoint.acrpe.network_interface[0].id 135 | log_analytics_workspace_id = var.log_analytics_workspace_id 136 | 137 | metric { 138 | category = "AllMetrics" 139 | 140 | retention_policy { 141 | enabled = true 142 | } 143 | } 144 | } 145 | -------------------------------------------------------------------------------- /terraform/acr/output.tf: -------------------------------------------------------------------------------- 1 | output "name" { 2 | value = azurerm_container_registry.acr.name 3 | } 4 | 5 | output "id" { 6 | value = azurerm_container_registry.acr.id 7 | } -------------------------------------------------------------------------------- /terraform/acr/variables.tf: -------------------------------------------------------------------------------- 1 | variable "name" {} 2 | variable "location" {} 3 | variable "resource_group_name" {} 4 | variable "shared_subnet" {} 5 | variable "core_vnet" {} 6 | variable "log_analytics_workspace_id" {} -------------------------------------------------------------------------------- /terraform/admin-jumpbox/output.tf: -------------------------------------------------------------------------------- 1 | output "jumpbox_user" { 2 | value = random_string.username.result 3 | } 4 | output "jumpbox_password" { 5 | value = random_password.password.result 6 | } -------------------------------------------------------------------------------- /terraform/admin-jumpbox/variables.tf: -------------------------------------------------------------------------------- 1 | variable "name" {} 2 | variable "location" {} 3 | variable "resource_group_name" {} 4 | variable "shared_subnet" {} 5 | variable "key_vault_id" {} 6 | variable "log_analytics_workspace_id" {} -------------------------------------------------------------------------------- /terraform/admin-jumpbox/vm.tf: -------------------------------------------------------------------------------- 1 | data "azurerm_subscription" "current" {} 2 | 3 | resource "azurerm_network_interface" "jumpbox_nic" { 4 | name = "nic-vm-${var.name}" 5 | resource_group_name = var.resource_group_name 6 | location = var.location 7 | 8 | ip_configuration { 9 | name = "internalIPConfig" 10 | subnet_id = var.shared_subnet 11 | private_ip_address_allocation = "Dynamic" 12 | } 13 | } 14 | 15 | resource "azurerm_monitor_diagnostic_setting" "jumpbox_nicdiagnostic" { 16 | name = "diagnostics-jumpbox-nic-${var.name}" 17 | target_resource_id = azurerm_network_interface.jumpbox_nic.id 18 | log_analytics_workspace_id = var.log_analytics_workspace_id 19 | 20 | metric { 21 | category = "AllMetrics" 22 | 23 | retention_policy { 24 | enabled = true 25 | } 26 | } 27 | } 28 | 29 | resource "random_string" "username" { 30 | length = 4 31 | upper = true 32 | lower = true 33 | number = true 34 | min_numeric = 1 35 | min_lower = 1 36 | special = false 37 | } 38 | 39 | resource "random_password" "password" { 40 | length = 16 41 | lower = true 42 | min_lower = 1 43 | upper = true 44 | min_upper = 1 45 | number = true 46 | min_numeric = 1 47 | special = true 48 | min_special = 1 49 | override_special = "_%@" 50 | } 51 | 52 | resource "azurerm_virtual_machine" "jumpbox" { 53 | name = "vm-${var.name}" 54 | resource_group_name = var.resource_group_name 55 | location = var.location 56 | network_interface_ids = [azurerm_network_interface.jumpbox_nic.id] 57 | vm_size = "Standard_DS1_v2" 58 | 59 | delete_os_disk_on_termination = true 60 | 61 | delete_data_disks_on_termination = true 62 | 63 | storage_image_reference { 64 | publisher = "MicrosoftWindowsDesktop" 65 | offer = "windows-10" 66 | sku = "20h2-pro-g2" 67 | version = "latest" 68 | } 69 | storage_os_disk { 70 | name = "vm-dsk-${var.name}" 71 | caching = "ReadWrite" 72 | create_option = "FromImage" 73 | managed_disk_type = "Standard_LRS" 74 | } 75 | os_profile { 76 | computer_name = "vm-${var.name}" 77 | admin_username = random_string.username.result 78 | admin_password = random_password.password.result 79 | } 80 | 81 | os_profile_windows_config { 82 | } 83 | 84 | tags = { 85 | environment = "staging" 86 | } 87 | } 88 | 89 | resource "azurerm_monitor_diagnostic_setting" "jumpbox_diagnostic" { 90 | name = "diagnostics-jumpbox-${var.name}" 91 | target_resource_id = azurerm_virtual_machine.jumpbox.id 92 | log_analytics_workspace_id = var.log_analytics_workspace_id 93 | 94 | metric { 95 | category = "AllMetrics" 96 | 97 | retention_policy { 98 | enabled = true 99 | } 100 | } 101 | } 102 | -------------------------------------------------------------------------------- /terraform/aml/aml.tf: -------------------------------------------------------------------------------- 1 | resource "azurerm_machine_learning_workspace" "ml" { 2 | name = "ml-${var.name}" 3 | location = var.location 4 | resource_group_name = var.resource_group_name 5 | application_insights_id = var.application_insights_id 6 | key_vault_id = var.key_vault_id 7 | storage_account_id = var.storage_account_id 8 | container_registry_id = var.container_registry_id 9 | identity { 10 | type = "SystemAssigned" 11 | } 12 | 13 | lifecycle { ignore_changes = [tags] } 14 | } 15 | 16 | resource "azurerm_private_dns_zone" "azureml" { 17 | name = "privatelink.api.azureml.ms" 18 | resource_group_name = var.resource_group_name 19 | 20 | lifecycle { ignore_changes = [tags] } 21 | } 22 | 23 | resource "azurerm_private_dns_zone_virtual_network_link" "azuremllink" { 24 | name = "azuremllink" 25 | resource_group_name = var.resource_group_name 26 | private_dns_zone_name = azurerm_private_dns_zone.azureml.name 27 | virtual_network_id = var.core_vnet 28 | 29 | lifecycle { ignore_changes = [tags] } 30 | } 31 | 32 | resource "azurerm_private_dns_zone" "azuremlcert" { 33 | name = "privatelink.cert.api.azureml.ms" 34 | resource_group_name = var.resource_group_name 35 | 36 | lifecycle { ignore_changes = [tags] } 37 | } 38 | 39 | resource "azurerm_private_dns_zone_virtual_network_link" "azuremlcertlink" { 40 | name = "azuremlcertlink" 41 | resource_group_name = var.resource_group_name 42 | private_dns_zone_name = azurerm_private_dns_zone.azuremlcert.name 43 | virtual_network_id = var.core_vnet 44 | 45 | lifecycle { ignore_changes = [tags] } 46 | } 47 | 48 | resource "azurerm_private_dns_zone" "notebooks" { 49 | name = "privatelink.notebooks.azure.net" 50 | resource_group_name = var.resource_group_name 51 | 52 | lifecycle { ignore_changes = [tags] } 53 | } 54 | 55 | resource "azurerm_private_dns_zone_virtual_network_link" "notebookslink" { 56 | name = "notebookslink" 57 | resource_group_name = var.resource_group_name 58 | private_dns_zone_name = azurerm_private_dns_zone.notebooks.name 59 | virtual_network_id = var.core_vnet 60 | 61 | lifecycle { ignore_changes = [tags] } 62 | } 63 | 64 | resource "azurerm_private_endpoint" "mlpe" { 65 | name = "mlpe-${azurerm_machine_learning_workspace.ml.name}" 66 | location = var.location 67 | resource_group_name = var.resource_group_name 68 | subnet_id = var.shared_subnet 69 | 70 | lifecycle { ignore_changes = [tags] } 71 | 72 | private_dns_zone_group { 73 | name = "private-dns-zone-group" 74 | private_dns_zone_ids = [azurerm_private_dns_zone.azureml.id, azurerm_private_dns_zone.notebooks.id, azurerm_private_dns_zone.azuremlcert.id] 75 | } 76 | 77 | private_service_connection { 78 | name = "mlpesc-${azurerm_machine_learning_workspace.ml.name}" 79 | private_connection_resource_id = azurerm_machine_learning_workspace.ml.id 80 | is_manual_connection = false 81 | subresource_names = ["amlworkspace"] 82 | } 83 | } 84 | 85 | resource "azurerm_monitor_diagnostic_setting" "mlpediagnostic" { 86 | name = "diagnostics-ml-pe-${var.name}" 87 | target_resource_id = azurerm_private_endpoint.mlpe.network_interface[0].id 88 | log_analytics_workspace_id = var.log_analytics_workspace_id 89 | 90 | metric { 91 | category = "AllMetrics" 92 | 93 | retention_policy { 94 | enabled = true 95 | } 96 | } 97 | } -------------------------------------------------------------------------------- /terraform/aml/output.tf: -------------------------------------------------------------------------------- 1 | output "azureml_workspace_name" { 2 | value = azurerm_machine_learning_workspace.ml.name 3 | } 4 | -------------------------------------------------------------------------------- /terraform/aml/variables.tf: -------------------------------------------------------------------------------- 1 | variable "name" {} 2 | variable "location" {} 3 | variable "resource_group_name" {} 4 | variable "shared_subnet" {} 5 | variable "application_insights_id" {} 6 | variable "key_vault_id" {} 7 | variable "storage_account_id" {} 8 | variable "container_registry_id" {} 9 | variable "core_vnet" {} 10 | variable "log_analytics_workspace_id" {} -------------------------------------------------------------------------------- /terraform/azure-monitor/ampls.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", 3 | "contentVersion": "1.0.0.0", 4 | "parameters": { 5 | "private_link_scope_name": { 6 | "type": "String" 7 | }, 8 | "workspace_name": { 9 | "type": "String" 10 | }, 11 | "app_insights_name": { 12 | "type": "String" 13 | } 14 | }, 15 | "variables": {}, 16 | "resources": [ 17 | { 18 | "type": "microsoft.insights/privatelinkscopes", 19 | "apiVersion": "2021-07-01-preview", 20 | "name": "[parameters('private_link_scope_name')]", 21 | "location": "global", 22 | "properties": { 23 | "accessModeSettings": { 24 | "queryAccessMode":"Open", 25 | "ingestionAccessMode":"Open" 26 | } 27 | } 28 | }, 29 | { 30 | "type": "microsoft.insights/privatelinkscopes/scopedresources", 31 | "apiVersion": "2019-10-17-preview", 32 | "name": "[concat(parameters('private_link_scope_name'), '/', concat(parameters('workspace_name'), '-connection'))]", 33 | "dependsOn": [ 34 | "[resourceId('microsoft.insights/privatelinkscopes', parameters('private_link_scope_name'))]" 35 | ], 36 | "properties": { 37 | "linkedResourceId": "[resourceId('microsoft.operationalinsights/workspaces', parameters('workspace_name'))]" 38 | } 39 | }, 40 | { 41 | "type": "microsoft.insights/privatelinkscopes/scopedresources", 42 | "apiVersion": "2019-10-17-preview", 43 | "name": "[concat(parameters('private_link_scope_name'), '/', concat(parameters('app_insights_name'), '-connection'))]", 44 | "dependsOn": [ 45 | "[resourceId('microsoft.insights/privatelinkscopes', parameters('private_link_scope_name'))]" 46 | ], 47 | "properties": { 48 | "linkedResourceId": "[resourceId('microsoft.insights/components', parameters('app_insights_name'))]" 49 | } 50 | } 51 | ], 52 | "outputs": { 53 | "resourceId": { 54 | "type": "String", 55 | "value": "[resourceId('microsoft.insights/privatelinkscopes', parameters('private_link_scope_name'))]" 56 | } 57 | } 58 | } 59 | -------------------------------------------------------------------------------- /terraform/azure-monitor/azure-monitor.tf: -------------------------------------------------------------------------------- 1 | resource "azurerm_log_analytics_workspace" "core" { 2 | name = "log-${var.name}" 3 | resource_group_name = var.resource_group_name 4 | location = var.location 5 | retention_in_days = 30 6 | sku = "PerGB2018" 7 | 8 | lifecycle { ignore_changes = [tags] } 9 | } 10 | 11 | resource "azurerm_application_insights" "core" { 12 | name = "appi-${var.name}" 13 | resource_group_name = var.resource_group_name 14 | location = var.location 15 | workspace_id = azurerm_log_analytics_workspace.core.id 16 | application_type = "web" 17 | 18 | lifecycle { ignore_changes = [tags] } 19 | } 20 | 21 | data "local_file" "ampls_arm_template" { 22 | filename = "${path.module}/ampls.json" 23 | } 24 | 25 | resource "azurerm_resource_group_template_deployment" "ampls_core" { 26 | name = "ampls-${var.name}" 27 | resource_group_name = var.resource_group_name 28 | deployment_mode = "Incremental" 29 | template_content = data.local_file.ampls_arm_template.content 30 | 31 | parameters_content = jsonencode({ 32 | "private_link_scope_name" = { 33 | value = "ampls-${var.name}" 34 | } 35 | "workspace_name" = { 36 | value = azurerm_log_analytics_workspace.core.name 37 | } 38 | "app_insights_name" = { 39 | value = azurerm_application_insights.core.name 40 | } 41 | }) 42 | } 43 | -------------------------------------------------------------------------------- /terraform/azure-monitor/output.tf: -------------------------------------------------------------------------------- 1 | output "app_insights_instrumentation_key" { 2 | value = azurerm_application_insights.core.instrumentation_key 3 | } 4 | 5 | output "app_insights_connection_string" { 6 | value = azurerm_application_insights.core.connection_string 7 | } 8 | 9 | output "app_insights_id" { 10 | value = azurerm_application_insights.core.id 11 | } 12 | 13 | output "log_analytics_workspace_id" { 14 | value = azurerm_log_analytics_workspace.core.id 15 | } 16 | 17 | output "log_analytics_workspace_name" { 18 | value = azurerm_log_analytics_workspace.core.name 19 | } 20 | -------------------------------------------------------------------------------- /terraform/azure-monitor/variables.tf: -------------------------------------------------------------------------------- 1 | variable "name" {} 2 | variable "location" {} 3 | variable "resource_group_name" {} 4 | -------------------------------------------------------------------------------- /terraform/bastion/bastion.tf: -------------------------------------------------------------------------------- 1 | resource "azurerm_public_ip" "bastion" { 2 | name = "pip-bas-${var.name}" 3 | resource_group_name = var.resource_group_name 4 | location = var.location 5 | allocation_method = "Static" 6 | sku = "Standard" 7 | 8 | lifecycle { ignore_changes = [tags] } 9 | } 10 | 11 | resource "azurerm_bastion_host" "bastion" { 12 | name = "bas-${var.name}" 13 | resource_group_name = var.resource_group_name 14 | location = var.location 15 | 16 | ip_configuration { 17 | name = "configuration" 18 | subnet_id = var.bastion_subnet 19 | public_ip_address_id = azurerm_public_ip.bastion.id 20 | } 21 | 22 | lifecycle { ignore_changes = [tags] } 23 | } 24 | 25 | resource "azurerm_monitor_diagnostic_setting" "pipbastiondiagnostic" { 26 | name = "diagnostics-pip-bastion-${var.name}" 27 | target_resource_id = azurerm_public_ip.bastion.id 28 | log_analytics_workspace_id = var.log_analytics_workspace_id 29 | 30 | log { 31 | category = "DDoSProtectionNotifications" 32 | enabled = true 33 | 34 | retention_policy { 35 | enabled = true 36 | days = 365 37 | } 38 | } 39 | 40 | log { 41 | category = "DDoSMitigationReports" 42 | enabled = true 43 | 44 | retention_policy { 45 | enabled = true 46 | days = 365 47 | } 48 | } 49 | 50 | log { 51 | category = "DDoSMitigationFlowLogs" 52 | enabled = true 53 | 54 | retention_policy { 55 | enabled = true 56 | days = 365 57 | } 58 | } 59 | 60 | metric { 61 | category = "AllMetrics" 62 | 63 | retention_policy { 64 | enabled = true 65 | days = 365 66 | } 67 | } 68 | } 69 | -------------------------------------------------------------------------------- /terraform/bastion/variables.tf: -------------------------------------------------------------------------------- 1 | variable "name" {} 2 | variable "location" {} 3 | variable "resource_group_name" {} 4 | variable "bastion_subnet" {} 5 | variable "log_analytics_workspace_id" {} -------------------------------------------------------------------------------- /terraform/firewall/data.tf: -------------------------------------------------------------------------------- 1 | data "azurerm_subnet" "firewall" { 2 | name = "AzureFirewallSubnet" 3 | virtual_network_name = "vnet-${var.name}" 4 | 5 | resource_group_name = var.resource_group_name 6 | } 7 | 8 | data "azurerm_subnet" "shared" { 9 | name = "SharedSubnet" 10 | virtual_network_name = "vnet-${var.name}" 11 | 12 | resource_group_name = var.resource_group_name 13 | } 14 | -------------------------------------------------------------------------------- /terraform/firewall/firewall.tf: -------------------------------------------------------------------------------- 1 | resource "azurerm_public_ip" "fwpip" { 2 | name = "pip-fw-${var.name}" 3 | resource_group_name = var.resource_group_name 4 | location = var.location 5 | allocation_method = "Static" 6 | sku = "Standard" 7 | 8 | lifecycle { ignore_changes = [tags] } 9 | } 10 | 11 | resource "azurerm_monitor_diagnostic_setting" "pipfwipdiagnostic" { 12 | name = "diagnostics-pip-fwip-${var.name}" 13 | target_resource_id = azurerm_public_ip.fwpip.id 14 | log_analytics_workspace_id = var.log_analytics_workspace_id 15 | 16 | log { 17 | category = "DDoSProtectionNotifications" 18 | enabled = true 19 | 20 | retention_policy { 21 | enabled = true 22 | days = 365 23 | } 24 | } 25 | 26 | log { 27 | category = "DDoSMitigationReports" 28 | enabled = true 29 | 30 | retention_policy { 31 | enabled = true 32 | days = 365 33 | } 34 | } 35 | 36 | log { 37 | category = "DDoSMitigationFlowLogs" 38 | enabled = true 39 | 40 | retention_policy { 41 | enabled = true 42 | days = 365 43 | } 44 | } 45 | 46 | metric { 47 | category = "AllMetrics" 48 | 49 | retention_policy { 50 | enabled = true 51 | days = 365 52 | } 53 | } 54 | } 55 | 56 | resource "azurerm_firewall" "fw" { 57 | depends_on = [azurerm_public_ip.fwpip] 58 | name = "fw-${var.name}" 59 | sku_name = "AZFW_VNet" 60 | sku_tier = "Standard" 61 | resource_group_name = var.resource_group_name 62 | location = var.location 63 | ip_configuration { 64 | name = "fw-ip-configuration" 65 | subnet_id = data.azurerm_subnet.firewall.id 66 | public_ip_address_id = azurerm_public_ip.fwpip.id 67 | } 68 | 69 | lifecycle { ignore_changes = [tags] } 70 | } 71 | 72 | resource "azurerm_monitor_diagnostic_setting" "firewall" { 73 | name = "diagnostics-firewall-${var.name}" 74 | target_resource_id = azurerm_firewall.fw.id 75 | log_analytics_workspace_id = var.log_analytics_workspace_id 76 | log_analytics_destination_type = "AzureDiagnostics" 77 | log { 78 | category = "AzureFirewallApplicationRule" 79 | enabled = true 80 | 81 | 82 | retention_policy { 83 | enabled = false 84 | days = 0 85 | } 86 | } 87 | 88 | log { 89 | 90 | category = "AzureFirewallNetworkRule" 91 | enabled = true 92 | 93 | retention_policy { 94 | enabled = false 95 | days = 0 96 | } 97 | } 98 | log { 99 | 100 | category = "AzureFirewallDnsProxy" 101 | enabled = true 102 | 103 | retention_policy { 104 | enabled = false 105 | days = 0 106 | } 107 | } 108 | log { 109 | 110 | category = "AzureFirewallNetworkRule" 111 | enabled = true 112 | 113 | retention_policy { 114 | enabled = false 115 | days = 0 116 | } 117 | } 118 | 119 | metric { 120 | category = "AllMetrics" 121 | enabled = true 122 | 123 | retention_policy { 124 | enabled = false 125 | days = 0 126 | } 127 | } 128 | 129 | } 130 | 131 | resource "azurerm_firewall_application_rule_collection" "shared_subnet" { 132 | name = "arc-shared_subnet" 133 | azure_firewall_name = azurerm_firewall.fw.name 134 | resource_group_name = azurerm_firewall.fw.resource_group_name 135 | priority = 100 136 | action = "Allow" 137 | 138 | rule { 139 | name = "admin-resources" 140 | source_addresses = data.azurerm_subnet.shared.address_prefixes 141 | target_fqdns = local.allowed_general_urls 142 | 143 | protocol { 144 | port = "443" 145 | type = "Https" 146 | } 147 | 148 | protocol { 149 | port = "80" 150 | type = "Http" 151 | } 152 | 153 | 154 | } 155 | 156 | rule { 157 | name = "allowMLrelated" 158 | source_addresses = data.azurerm_subnet.shared.address_prefixes 159 | target_fqdns = local.allowed_aml_urls 160 | 161 | protocol { 162 | port = "443" 163 | type = "Https" 164 | } 165 | 166 | protocol { 167 | port = "80" 168 | type = "Http" 169 | } 170 | } 171 | 172 | rule { 173 | name = "allowADrelated" 174 | source_addresses = data.azurerm_subnet.shared.address_prefixes 175 | target_fqdns = local.allowed_ad_urls 176 | 177 | protocol { 178 | port = "443" 179 | type = "Https" 180 | } 181 | 182 | protocol { 183 | port = "80" 184 | type = "Http" 185 | } 186 | } 187 | 188 | } 189 | 190 | resource "azurerm_firewall_network_rule_collection" "general" { 191 | name = "general" 192 | azure_firewall_name = azurerm_firewall.fw.name 193 | resource_group_name = azurerm_firewall.fw.resource_group_name 194 | priority = 100 195 | action = "Allow" 196 | 197 | rule { 198 | name = "time" 199 | 200 | protocols = [ 201 | "UDP" 202 | ] 203 | 204 | destination_addresses = [ 205 | "*" 206 | ] 207 | 208 | destination_ports = [ 209 | "123" 210 | ] 211 | source_addresses = [ 212 | "*" 213 | ] 214 | } 215 | 216 | depends_on = [ 217 | azurerm_firewall_application_rule_collection.shared_subnet 218 | ] 219 | } 220 | 221 | resource "azurerm_firewall_network_rule_collection" "shared_nrc" { 222 | name = "shared" 223 | azure_firewall_name = azurerm_firewall.fw.name 224 | resource_group_name = azurerm_firewall.fw.resource_group_name 225 | priority = 101 226 | action = "Allow" 227 | 228 | rule { 229 | name = "allowStorage" 230 | 231 | source_addresses = data.azurerm_subnet.shared.address_prefixes 232 | 233 | 234 | destination_ports = [ 235 | "*" 236 | ] 237 | 238 | destination_addresses = local.allowed_service_tags 239 | 240 | protocols = [ 241 | "TCP" 242 | ] 243 | } 244 | 245 | depends_on = [ 246 | azurerm_firewall_network_rule_collection.general 247 | ] 248 | } 249 | -------------------------------------------------------------------------------- /terraform/firewall/locals.tf: -------------------------------------------------------------------------------- 1 | data "azurerm_resource_group" "core" { 2 | name = var.resource_group_name 3 | } 4 | 5 | data "azurerm_network_service_tags" "storage" { 6 | location = data.azurerm_resource_group.core.location 7 | service = "Storage" 8 | location_filter = data.azurerm_resource_group.core.location 9 | } 10 | 11 | locals { 12 | allowed_aml_urls = ["ml.azure.com", "viennaglobal.azurecr.io", "*openml.org"] 13 | allowed_ad_urls = ["enterpriseregistration.windows.net", "169.254.169.254", "login.microsoftonline.com", "pas.windows.net", "*manage-beta.microsoft.com", "*manage.microsoft.com", "login.windows.net","msft.sts.microsoft.com"] 14 | allowed_general_urls = ["binstar-cio-packages-prod.s3.amazonaws.com", "conda.anaconda.org", "repo.anaconda.com", "azure.archive.ubuntu.com", "go.microsoft.com", "*.azureedge.net", "*github.com", "*githubassets.com", "*powershellgallery.com", "git-scm.com", "*githubusercontent.com", "*core.windows.net", "aka.ms", "management.azure.com", "graph.microsoft.com", "login.microsoftonline.com", "aadcdn.msftauth.net", "graph.windows.net"] 15 | allowed_service_tags = [data.azurerm_network_service_tags.storage.id, "AzureContainerRegistry"] 16 | } 17 | -------------------------------------------------------------------------------- /terraform/firewall/output.tf: -------------------------------------------------------------------------------- 1 | output "firewall_private_ip_address" { 2 | value = azurerm_firewall.fw.ip_configuration.0.private_ip_address 3 | } 4 | 5 | output "firewall_public_ip" { 6 | value = azurerm_public_ip.fwpip.ip_address 7 | } 8 | -------------------------------------------------------------------------------- /terraform/firewall/variables.tf: -------------------------------------------------------------------------------- 1 | variable "name" {} 2 | variable "location" {} 3 | variable "resource_group_name" {} 4 | variable "log_analytics_workspace_id" {} 5 | -------------------------------------------------------------------------------- /terraform/keyvault/keyvault.tf: -------------------------------------------------------------------------------- 1 | data "azurerm_client_config" "deployer" {} 2 | 3 | resource "azurerm_key_vault" "kv" { 4 | name = "kv-${var.name}" 5 | tenant_id = var.tenant_id 6 | location = var.location 7 | resource_group_name = var.resource_group_name 8 | sku_name = "standard" 9 | purge_protection_enabled = true 10 | 11 | lifecycle { ignore_changes = [tags] } 12 | 13 | network_acls { 14 | bypass = "AzureServices" 15 | default_action = "Deny" 16 | } 17 | } 18 | 19 | resource "azurerm_key_vault_access_policy" "deployer" { 20 | key_vault_id = azurerm_key_vault.kv.id 21 | tenant_id = data.azurerm_client_config.deployer.tenant_id 22 | object_id = data.azurerm_client_config.deployer.object_id 23 | 24 | key_permissions = ["Get", "List", "Update", "Create", "Import", "Delete"] 25 | secret_permissions = ["Get", "List", "Set", "Delete", "Purge"] 26 | certificate_permissions = ["Get", "List", "Update", "Create", "Import", "Delete", "Purge"] 27 | storage_permissions = ["Get", "List", "Update", "Delete"] 28 | } 29 | 30 | resource "azurerm_private_dns_zone" "vaultcore" { 31 | name = "privatelink.vaultcore.azure.net" 32 | resource_group_name = var.resource_group_name 33 | 34 | lifecycle { ignore_changes = [tags] } 35 | } 36 | 37 | resource "azurerm_private_dns_zone_virtual_network_link" "vaultcore" { 38 | name = "vaultcorelink" 39 | resource_group_name = var.resource_group_name 40 | private_dns_zone_name = azurerm_private_dns_zone.vaultcore.name 41 | virtual_network_id = var.core_vnet 42 | 43 | lifecycle { ignore_changes = [tags] } 44 | } 45 | 46 | resource "azurerm_private_endpoint" "kvpe" { 47 | name = "pe-kv-${var.name}" 48 | location = var.location 49 | resource_group_name = var.resource_group_name 50 | subnet_id = var.shared_subnet 51 | 52 | lifecycle { ignore_changes = [tags] } 53 | 54 | private_dns_zone_group { 55 | name = "private-dns-zone-group" 56 | private_dns_zone_ids = [azurerm_private_dns_zone.vaultcore.id] 57 | } 58 | 59 | private_service_connection { 60 | name = "psc-kv-${var.name}" 61 | private_connection_resource_id = azurerm_key_vault.kv.id 62 | is_manual_connection = false 63 | subresource_names = ["Vault"] 64 | } 65 | } 66 | 67 | resource "azurerm_monitor_diagnostic_setting" "kvpediagnostic" { 68 | name = "diagnostics-kvpe-${var.name}" 69 | target_resource_id = azurerm_private_endpoint.kvpe.network_interface[0].id 70 | log_analytics_workspace_id = var.log_analytics_workspace_id 71 | 72 | metric { 73 | category = "AllMetrics" 74 | 75 | retention_policy { 76 | enabled = true 77 | } 78 | } 79 | } 80 | 81 | resource "azurerm_monitor_diagnostic_setting" "kvdiagnostic" { 82 | name = "diagnostics-kv-${var.name}" 83 | target_resource_id = azurerm_key_vault.kv.id 84 | log_analytics_workspace_id = var.log_analytics_workspace_id 85 | 86 | log { 87 | category = "AuditEvent" 88 | enabled = true 89 | 90 | retention_policy { 91 | enabled = true 92 | days = 365 93 | } 94 | } 95 | 96 | metric { 97 | category = "AllMetrics" 98 | 99 | retention_policy { 100 | enabled = true 101 | days = 365 102 | } 103 | } 104 | } 105 | -------------------------------------------------------------------------------- /terraform/keyvault/output.tf: -------------------------------------------------------------------------------- 1 | output "key_vault_id" { 2 | value = azurerm_key_vault.kv.id 3 | } 4 | 5 | output "keyvault_name" { 6 | value = azurerm_key_vault.kv.name 7 | } -------------------------------------------------------------------------------- /terraform/keyvault/variables.tf: -------------------------------------------------------------------------------- 1 | variable "name" {} 2 | variable "location" {} 3 | variable "resource_group_name" {} 4 | variable "core_vnet" {} 5 | variable "shared_subnet" {} 6 | variable "tenant_id" {} 7 | variable "log_analytics_workspace_id" {} 8 | -------------------------------------------------------------------------------- /terraform/locals.tf: -------------------------------------------------------------------------------- 1 | data "azurerm_subscription" "current" {} 2 | 3 | data "azurerm_client_config" "current" {} 4 | 5 | # Random unique id 6 | resource "random_string" "unique_id" { 7 | length = 4 8 | min_numeric = 4 9 | } 10 | 11 | locals { 12 | name = lower(replace("${var.name}${random_string.unique_id.result}", "-", "")) 13 | } -------------------------------------------------------------------------------- /terraform/main.tf: -------------------------------------------------------------------------------- 1 | # Azure Provider source and version being used 2 | terraform { 3 | required_providers { 4 | azurerm = { 5 | source = "hashicorp/azurerm" 6 | version = "=3.9.0" 7 | } 8 | } 9 | 10 | } 11 | 12 | provider "azurerm" { 13 | features { 14 | key_vault { 15 | purge_soft_delete_on_destroy = false 16 | recover_soft_deleted_key_vaults = true 17 | } 18 | } 19 | } 20 | 21 | resource "azurerm_resource_group" "core" { 22 | location = var.location 23 | name = "rg-${local.name}" 24 | tags = { 25 | project = "Private Azure ML" 26 | name = local.name 27 | source = "https://github.com/microsoft/privateAML/" 28 | } 29 | 30 | lifecycle { ignore_changes = [tags] } 31 | } 32 | 33 | module "azure_monitor" { 34 | source = "./azure-monitor" 35 | name = local.name 36 | location = var.location 37 | resource_group_name = azurerm_resource_group.core.name 38 | } 39 | 40 | module "network" { 41 | source = "./network" 42 | name = local.name 43 | location = var.location 44 | resource_group_name = azurerm_resource_group.core.name 45 | vnet_address_space = var.vnet_address_space 46 | log_analytics_workspace_id = module.azure_monitor.log_analytics_workspace_id 47 | } 48 | 49 | module "storage" { 50 | source = "./storage" 51 | name = local.name 52 | location = var.location 53 | resource_group_name = azurerm_resource_group.core.name 54 | shared_subnet = module.network.shared_subnet_id 55 | core_vnet = module.network.core_vnet_id 56 | log_analytics_workspace_id = module.azure_monitor.log_analytics_workspace_id 57 | 58 | depends_on = [ 59 | module.network 60 | ] 61 | } 62 | 63 | module "keyvault" { 64 | source = "./keyvault" 65 | name = local.name 66 | location = var.location 67 | resource_group_name = azurerm_resource_group.core.name 68 | shared_subnet = module.network.shared_subnet_id 69 | core_vnet = module.network.core_vnet_id 70 | tenant_id = data.azurerm_client_config.current.tenant_id 71 | log_analytics_workspace_id = module.azure_monitor.log_analytics_workspace_id 72 | } 73 | 74 | module "firewall" { 75 | source = "./firewall" 76 | name = local.name 77 | location = var.location 78 | resource_group_name = azurerm_resource_group.core.name 79 | log_analytics_workspace_id = module.azure_monitor.log_analytics_workspace_id 80 | 81 | depends_on = [ 82 | module.network 83 | ] 84 | } 85 | 86 | module "routetable" { 87 | source = "./routetable" 88 | name = local.name 89 | location = var.location 90 | resource_group_name = azurerm_resource_group.core.name 91 | shared_subnet = module.network.shared_subnet_id 92 | firewall_private_ip_address = module.firewall.firewall_private_ip_address 93 | } 94 | 95 | module "bastion" { 96 | source = "./bastion" 97 | name = local.name 98 | location = var.location 99 | resource_group_name = azurerm_resource_group.core.name 100 | bastion_subnet = module.network.bastion_subnet_id 101 | log_analytics_workspace_id = module.azure_monitor.log_analytics_workspace_id 102 | } 103 | 104 | module "acr" { 105 | source = "./acr" 106 | name = local.name 107 | location = var.location 108 | resource_group_name = azurerm_resource_group.core.name 109 | shared_subnet = module.network.shared_subnet_id 110 | core_vnet = module.network.core_vnet_id 111 | log_analytics_workspace_id = module.azure_monitor.log_analytics_workspace_id 112 | } 113 | 114 | module "aml" { 115 | source = "./aml" 116 | name = local.name 117 | location = var.location 118 | resource_group_name = azurerm_resource_group.core.name 119 | shared_subnet = module.network.shared_subnet_id 120 | application_insights_id = module.azure_monitor.app_insights_id 121 | key_vault_id = module.keyvault.key_vault_id 122 | storage_account_id = module.storage.storage_account_id 123 | container_registry_id = module.acr.id 124 | core_vnet = module.network.core_vnet_id 125 | log_analytics_workspace_id = module.azure_monitor.log_analytics_workspace_id 126 | } 127 | 128 | module "jumpbox" { 129 | source = "./admin-jumpbox" 130 | name = local.name 131 | location = var.location 132 | resource_group_name = azurerm_resource_group.core.name 133 | shared_subnet = module.network.shared_subnet_id 134 | key_vault_id = module.keyvault.key_vault_id 135 | log_analytics_workspace_id = module.azure_monitor.log_analytics_workspace_id 136 | depends_on = [ 137 | module.keyvault 138 | ] 139 | } 140 | -------------------------------------------------------------------------------- /terraform/network/locals.tf: -------------------------------------------------------------------------------- 1 | locals { 2 | core_services_vnet_subnets = cidrsubnets(var.vnet_address_space,6, 2, 4, 3) 3 | firewall_subnet_address_space = local.core_services_vnet_subnets[1] 4 | bastion_subnet_address_prefix = local.core_services_vnet_subnets[2] 5 | shared_services_subnet_address_prefix = local.core_services_vnet_subnets[3] 6 | } 7 | -------------------------------------------------------------------------------- /terraform/network/network.tf: -------------------------------------------------------------------------------- 1 | resource "azurerm_virtual_network" "core" { 2 | name = "vnet-${var.name}" 3 | location = var.location 4 | resource_group_name = var.resource_group_name 5 | address_space = [var.vnet_address_space] 6 | 7 | lifecycle { ignore_changes = [tags] } 8 | } 9 | 10 | resource "azurerm_subnet" "bastion" { 11 | name = "AzureBastionSubnet" 12 | virtual_network_name = azurerm_virtual_network.core.name 13 | resource_group_name = var.resource_group_name 14 | address_prefixes = [local.bastion_subnet_address_prefix] 15 | } 16 | 17 | resource "azurerm_subnet" "azure_firewall" { 18 | name = "AzureFirewallSubnet" 19 | virtual_network_name = azurerm_virtual_network.core.name 20 | resource_group_name = var.resource_group_name 21 | address_prefixes = [local.firewall_subnet_address_space] 22 | } 23 | 24 | resource "azurerm_subnet" "shared" { 25 | name = "SharedSubnet" 26 | virtual_network_name = azurerm_virtual_network.core.name 27 | resource_group_name = var.resource_group_name 28 | address_prefixes = [local.shared_services_subnet_address_prefix] 29 | # notice that private endpoints do not adhere to NSG rules 30 | enforce_private_link_endpoint_network_policies = true 31 | enforce_private_link_service_network_policies = true 32 | } 33 | 34 | resource "azurerm_monitor_diagnostic_setting" "networkcorediagnostic" { 35 | name = "diagnostics-net-core-${var.name}" 36 | target_resource_id = azurerm_virtual_network.core.id 37 | log_analytics_workspace_id = var.log_analytics_workspace_id 38 | 39 | log { 40 | category = "VMProtectionAlerts" 41 | enabled = true 42 | 43 | retention_policy { 44 | enabled = true 45 | } 46 | } 47 | 48 | metric { 49 | category = "AllMetrics" 50 | 51 | retention_policy { 52 | enabled = true 53 | } 54 | } 55 | } 56 | -------------------------------------------------------------------------------- /terraform/network/network_security_groups.tf: -------------------------------------------------------------------------------- 1 | # Network security group for Azure Bastion subnet 2 | # See https://docs.microsoft.com/azure/bastion/bastion-nsg 3 | resource "azurerm_network_security_group" "bastion" { 4 | name = "nsg-bastion-subnet" 5 | location = var.location 6 | resource_group_name = var.resource_group_name 7 | 8 | security_rule { 9 | name = "AllowInboundInternet" 10 | priority = 4000 11 | direction = "Inbound" 12 | access = "Allow" 13 | protocol = "Tcp" 14 | source_port_range = "*" 15 | destination_port_range = "443" 16 | source_address_prefix = "Internet" 17 | destination_address_prefix = "*" 18 | } 19 | 20 | security_rule { 21 | name = "AllowInboundGatewayManager" 22 | priority = 4001 23 | direction = "Inbound" 24 | access = "Allow" 25 | protocol = "Tcp" 26 | source_port_range = "*" 27 | destination_port_range = "443" 28 | source_address_prefix = "GatewayManager" 29 | destination_address_prefix = "*" 30 | } 31 | 32 | security_rule { 33 | name = "AllowInboundAzureLoadBalancer" 34 | priority = 4002 35 | direction = "Inbound" 36 | access = "Allow" 37 | protocol = "Tcp" 38 | source_port_range = "*" 39 | destination_port_range = "443" 40 | source_address_prefix = "AzureLoadBalancer" 41 | destination_address_prefix = "*" 42 | } 43 | 44 | security_rule { 45 | name = "AllowInboundHostCommunication" 46 | priority = 4003 47 | direction = "Inbound" 48 | access = "Allow" 49 | protocol = "*" 50 | source_port_range = "*" 51 | destination_port_ranges = ["5701", "8080"] 52 | source_address_prefix = "VirtualNetwork" 53 | destination_address_prefix = "VirtualNetwork" 54 | } 55 | 56 | security_rule { 57 | name = "AllowOutboundSshRdp" 58 | priority = 4020 59 | direction = "Outbound" 60 | access = "Allow" 61 | protocol = "*" 62 | source_port_range = "*" 63 | destination_port_ranges = ["22", "3389"] 64 | source_address_prefix = "*" 65 | destination_address_prefix = "VirtualNetwork" 66 | } 67 | 68 | security_rule { 69 | name = "AllowOutboundAzureCloud" 70 | priority = 4021 71 | direction = "Outbound" 72 | access = "Allow" 73 | protocol = "Tcp" 74 | source_port_range = "*" 75 | destination_port_range = "443" 76 | source_address_prefix = "*" 77 | destination_address_prefix = "AzureCloud" 78 | } 79 | 80 | security_rule { 81 | name = "AllowOutboundHostCommunication" 82 | priority = 4022 83 | direction = "Outbound" 84 | access = "Allow" 85 | protocol = "*" 86 | source_port_range = "*" 87 | destination_port_ranges = ["5701", "8080"] 88 | source_address_prefix = "VirtualNetwork" 89 | destination_address_prefix = "VirtualNetwork" 90 | } 91 | 92 | security_rule { 93 | name = "AllowOutboundGetSessionInformation" 94 | priority = 4023 95 | direction = "Outbound" 96 | access = "Allow" 97 | protocol = "*" 98 | source_port_range = "*" 99 | destination_port_range = "80" 100 | source_address_prefix = "*" 101 | destination_address_prefix = "Internet" 102 | } 103 | } 104 | 105 | resource "azurerm_subnet_network_security_group_association" "bastion" { 106 | subnet_id = azurerm_subnet.bastion.id 107 | network_security_group_id = azurerm_network_security_group.bastion.id 108 | } 109 | 110 | resource "azurerm_network_security_group" "shared_rules" { 111 | name = "nsg-shared" 112 | location = var.location 113 | resource_group_name = var.resource_group_name 114 | } 115 | 116 | resource "azurerm_subnet_network_security_group_association" "shared" { 117 | subnet_id = azurerm_subnet.shared.id 118 | network_security_group_id = azurerm_network_security_group.shared_rules.id 119 | } 120 | 121 | resource "azurerm_network_security_rule" "allow-batch-inbound" { 122 | access = "Allow" 123 | destination_port_ranges = ["29876", "29877"] 124 | destination_address_prefix = "VirtualNetwork" 125 | source_address_prefix = "BatchNodeManagement" 126 | direction = "Inbound" 127 | name = "allow-Batch-inbound" 128 | network_security_group_name = azurerm_network_security_group.shared_rules.name 129 | priority = 200 130 | protocol = "Tcp" 131 | resource_group_name = var.resource_group_name 132 | source_port_range = "*" 133 | } 134 | 135 | resource "azurerm_network_security_rule" "allow-aml-inbound" { 136 | access = "Allow" 137 | destination_port_ranges = ["44224"] 138 | destination_address_prefix = "VirtualNetwork" 139 | source_address_prefix = "Internet" 140 | direction = "Inbound" 141 | name = "allow-AzureML-inbound" 142 | network_security_group_name = azurerm_network_security_group.shared_rules.name 143 | priority = 201 144 | protocol = "Tcp" 145 | resource_group_name = var.resource_group_name 146 | source_port_range = "*" 147 | } 148 | 149 | resource "azurerm_network_security_rule" "allow-Outbound_Storage_445" { 150 | access = "Allow" 151 | destination_port_range = "445" 152 | destination_address_prefix = "Storage" 153 | source_address_prefix = "VirtualNetwork" 154 | direction = "Outbound" 155 | name = "allow-Outbound-Storage-445" 156 | network_security_group_name = azurerm_network_security_group.shared_rules.name 157 | priority = 202 158 | protocol = "Tcp" 159 | resource_group_name = var.resource_group_name 160 | source_port_range = "*" 161 | } 162 | -------------------------------------------------------------------------------- /terraform/network/output.tf: -------------------------------------------------------------------------------- 1 | output "core_vnet_id" { 2 | value = azurerm_virtual_network.core.id 3 | } 4 | 5 | output "bastion_subnet_id" { 6 | value = azurerm_subnet.bastion.id 7 | } 8 | 9 | output "azure_firewall_subnet_id" { 10 | value = azurerm_subnet.azure_firewall.id 11 | } 12 | 13 | output "shared_subnet_id" { 14 | value = azurerm_subnet.shared.id 15 | } 16 | -------------------------------------------------------------------------------- /terraform/network/variables.tf: -------------------------------------------------------------------------------- 1 | variable "name" {} 2 | variable "location" {} 3 | variable "resource_group_name" {} 4 | variable "vnet_address_space" {} 5 | variable "log_analytics_workspace_id" {} -------------------------------------------------------------------------------- /terraform/outputs.tf: -------------------------------------------------------------------------------- 1 | output "core_resource_group_name" { 2 | value = azurerm_resource_group.core.name 3 | } 4 | 5 | output "log_analytics_name" { 6 | value = module.azure_monitor.log_analytics_workspace_name 7 | } 8 | 9 | output "keyvault_name" { 10 | value = module.keyvault.keyvault_name 11 | } 12 | 13 | output "jumpbox_user" { 14 | value = module.jumpbox.jumpbox_user 15 | } 16 | 17 | output "jumpbox_pass" { 18 | value = module.jumpbox.jumpbox_password 19 | sensitive = true 20 | } -------------------------------------------------------------------------------- /terraform/routetable/routetable.tf: -------------------------------------------------------------------------------- 1 | resource "azurerm_route_table" "rt" { 2 | name = "rt-${var.name}" 3 | resource_group_name = var.resource_group_name 4 | location = var.location 5 | disable_bgp_route_propagation = false 6 | 7 | lifecycle { ignore_changes = [tags] } 8 | 9 | route { 10 | name = "DefaultRoute" 11 | address_prefix = "0.0.0.0/0" 12 | next_hop_type = "VirtualAppliance" 13 | next_hop_in_ip_address = var.firewall_private_ip_address 14 | } 15 | } 16 | 17 | resource "azurerm_subnet_route_table_association" "rt_shared_subnet_association" { 18 | subnet_id = var.shared_subnet 19 | route_table_id = azurerm_route_table.rt.id 20 | } 21 | -------------------------------------------------------------------------------- /terraform/routetable/variables.tf: -------------------------------------------------------------------------------- 1 | variable "name" {} 2 | variable "location" {} 3 | variable "resource_group_name" {} 4 | variable "firewall_private_ip_address" {} 5 | variable "shared_subnet" {} 6 | -------------------------------------------------------------------------------- /terraform/storage/output.tf: -------------------------------------------------------------------------------- 1 | output "storage_account_name" { 2 | value = azurerm_storage_account.stg.name 3 | } 4 | 5 | output "storage_account_id" { 6 | value = azurerm_storage_account.stg.id 7 | } 8 | 9 | output "storage_account_access_key" { 10 | value = azurerm_storage_account.stg.primary_access_key 11 | } 12 | -------------------------------------------------------------------------------- /terraform/storage/storage.tf: -------------------------------------------------------------------------------- 1 | resource "azurerm_storage_account" "stg" { 2 | name = lower(replace("stg-${var.name}", "-", "")) 3 | resource_group_name = var.resource_group_name 4 | location = var.location 5 | account_tier = "Standard" 6 | account_replication_type = "LRS" 7 | network_rules { 8 | default_action = "Deny" 9 | } 10 | 11 | lifecycle { ignore_changes = [tags] } 12 | } 13 | 14 | resource "azurerm_private_dns_zone" "filecore" { 15 | name = "privatelink.file.core.windows.net" 16 | resource_group_name = var.resource_group_name 17 | 18 | lifecycle { ignore_changes = [tags] } 19 | } 20 | 21 | resource "azurerm_private_dns_zone_virtual_network_link" "filecorelink" { 22 | name = "filecorelink" 23 | resource_group_name = var.resource_group_name 24 | private_dns_zone_name = azurerm_private_dns_zone.filecore.name 25 | virtual_network_id = var.core_vnet 26 | 27 | lifecycle { ignore_changes = [tags] } 28 | } 29 | 30 | resource "azurerm_private_dns_zone" "blobcore" { 31 | name = "privatelink.blob.core.windows.net" 32 | resource_group_name = var.resource_group_name 33 | 34 | lifecycle { ignore_changes = [tags] } 35 | } 36 | 37 | resource "azurerm_private_dns_zone_virtual_network_link" "blobcore" { 38 | name = "blobcorelink" 39 | resource_group_name = var.resource_group_name 40 | private_dns_zone_name = azurerm_private_dns_zone.blobcore.name 41 | virtual_network_id = var.core_vnet 42 | 43 | lifecycle { ignore_changes = [tags] } 44 | } 45 | 46 | resource "azurerm_private_endpoint" "blobpe" { 47 | name = "pe-blob-${var.name}" 48 | location = var.location 49 | resource_group_name = var.resource_group_name 50 | subnet_id = var.shared_subnet 51 | 52 | lifecycle { ignore_changes = [tags] } 53 | 54 | private_dns_zone_group { 55 | name = "private-dns-zone-group-blobcore" 56 | private_dns_zone_ids = [azurerm_private_dns_zone.blobcore.id] 57 | } 58 | 59 | private_service_connection { 60 | name = "psc-stg-${var.name}" 61 | private_connection_resource_id = azurerm_storage_account.stg.id 62 | is_manual_connection = false 63 | subresource_names = ["Blob"] 64 | } 65 | } 66 | 67 | resource "azurerm_monitor_diagnostic_setting" "blobpeiagnostic" { 68 | name = "diagnostics-blobpe-${var.name}" 69 | target_resource_id = azurerm_private_endpoint.blobpe.network_interface[0].id 70 | log_analytics_workspace_id = var.log_analytics_workspace_id 71 | 72 | metric { 73 | category = "AllMetrics" 74 | 75 | retention_policy { 76 | enabled = true 77 | } 78 | } 79 | } 80 | 81 | resource "azurerm_private_endpoint" "filepe" { 82 | name = "pe-file-${var.name}" 83 | location = var.location 84 | resource_group_name = var.resource_group_name 85 | subnet_id = var.shared_subnet 86 | 87 | lifecycle { ignore_changes = [tags] } 88 | 89 | private_dns_zone_group { 90 | name = "private-dns-zone-group-filecore" 91 | private_dns_zone_ids = [azurerm_private_dns_zone.filecore.id] 92 | } 93 | 94 | private_service_connection { 95 | name = "psc-filestg-${var.name}" 96 | private_connection_resource_id = azurerm_storage_account.stg.id 97 | is_manual_connection = false 98 | subresource_names = ["file"] 99 | } 100 | } 101 | 102 | resource "azurerm_monitor_diagnostic_setting" "filepeiagnostic" { 103 | name = "diagnostics-filepe-${var.name}" 104 | target_resource_id = azurerm_private_endpoint.filepe.network_interface[0].id 105 | log_analytics_workspace_id = var.log_analytics_workspace_id 106 | 107 | metric { 108 | category = "AllMetrics" 109 | 110 | retention_policy { 111 | enabled = true 112 | } 113 | } 114 | } 115 | -------------------------------------------------------------------------------- /terraform/storage/variables.tf: -------------------------------------------------------------------------------- 1 | variable "name" {} 2 | variable "location" {} 3 | variable "resource_group_name" {} 4 | variable "core_vnet" {} 5 | variable "shared_subnet" {} 6 | variable "log_analytics_workspace_id" {} -------------------------------------------------------------------------------- /terraform/terraform.tfvars.tmpl: -------------------------------------------------------------------------------- 1 | name = "test" 2 | location = "westeurope" 3 | vnet_address_space = "10.1.0.0/22" -------------------------------------------------------------------------------- /terraform/variables.tf: -------------------------------------------------------------------------------- 1 | variable "name" { 2 | type = string 3 | description = "A 4 character name identifier, such as 'test'" 4 | validation { 5 | condition = length(var.name) < 5 6 | error_message = "The id value must be max 4 chars." 7 | } 8 | } 9 | 10 | variable "location" { 11 | type = string 12 | description = "Azure region for deployment of services, such as 'westeurope'" 13 | } 14 | 15 | variable "vnet_address_space" { 16 | type = string 17 | description = "VNET Address Space, such as '10.0.0.0/22'" 18 | } 19 | --------------------------------------------------------------------------------