├── .artifactignore ├── .gitattributes ├── .gitignore ├── Brewfile ├── LICENSE ├── README.md ├── kusto ├── denied-outbound-http-traffic.kql └── denied-outbound-traffic.kql ├── manifests ├── aks-helloworld-one.yaml ├── aks-helloworld-two.yaml ├── aspnetapp.yaml ├── hello-world-ingress.yaml └── internal-vote.yaml ├── pipelines └── azure-aks-ci.yml ├── scripts ├── cleanup_peerings.ps1 ├── create_nsg_assignment.ps1 ├── deploy.ps1 ├── deploy_app.ps1 ├── functions.ps1 ├── kube_config.ps1 ├── kube_tunnel.ps1 └── wait_for_app_gw.ps1 ├── terraform ├── .terraform-version ├── .terraform.lock.hcl ├── aks.auto.tfvars.sample ├── backend.tf.sample ├── main.tf ├── modules.tf ├── modules │ ├── aks-network │ │ ├── egress.tf │ │ ├── ingress.tf │ │ ├── main.tf │ │ ├── outputs.tf │ │ └── variables.tf │ ├── aks │ │ ├── main.tf │ │ ├── monitoring.tf │ │ ├── outputs.tf │ │ └── variables.tf │ ├── bastion │ │ ├── main.tf │ │ └── variables.tf │ ├── kubernetes │ │ └── main.tf │ ├── network │ │ ├── egress.tf │ │ ├── firewall.tf │ │ ├── main.tf │ │ ├── outputs.tf │ │ └── variables.tf │ └── service-principal │ │ ├── main.tf │ │ ├── outputs.tf │ │ └── variables.tf ├── outputs.tf ├── provider.tf └── variables.tf └── visuals ├── aspnetapp.png ├── diagram-blog.png ├── diagram.png ├── diagram.vsdx └── votingapp.png /.artifactignore: -------------------------------------------------------------------------------- 1 | *.tfstate -------------------------------------------------------------------------------- /.gitattributes: -------------------------------------------------------------------------------- 1 | # Auto detect text files and perform LF normalization 2 | * text=auto 3 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Local .kube directories 2 | **/.kube/* 3 | 4 | # Local .terraform directories 5 | **/.terraform/* 6 | 7 | # .tfstate files 8 | *.tfstate 9 | *.tfstate.* 10 | 11 | # Crash log files 12 | crash.log 13 | 14 | # Ignore any .tfvars files that are generated automatically for each Terraform run. Most 15 | # .tfvars files are managed as part of configuration and so should be included in 16 | # version control. 17 | # 18 | # example.tfvars 19 | 20 | # Ignore override files as they are usually used to override resources locally and so 21 | # are not checked in 22 | override.tf 23 | override.tf.json 24 | *_override.tf 25 | *_override.tf.json 26 | *.auto.tfvars 27 | 28 | # Include override files you do wish to add to version control using negated pattern 29 | # 30 | # !example_override.tf 31 | 32 | # Ignore files with sensitive information 33 | *.auto.tfvars 34 | *.tfvars 35 | backend.tf 36 | 37 | # Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan 38 | *.tfplan 39 | 40 | # Ignore temporary files 41 | ~*.* 42 | tmp*.* -------------------------------------------------------------------------------- /Brewfile: -------------------------------------------------------------------------------- 1 | brew "azure-cli" 2 | brew "azure/kubelogin/kubelogin" 3 | brew "kubernetes-cli" 4 | brew "kubernetes-helm" 5 | #brew "hashicorp/tap/terraform" 6 | brew "tfenv" 7 | 8 | cask "powershell" -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Eric van Wijk 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Network Isolated AKS 2 | 3 | [![Build Status](https://dev.azure.com/ericvan/PipelineSamples/_apis/build/status/workloads/azure-aks-ci?branchName=main)](https://dev.azure.com/ericvan/PipelineSamples/_build/latest?definitionId=155&branchName=main) 4 | 5 | This repo lets you provision a network isolated Azure Kubernetes Service, customizing egress, ingress with both Internal Load Balancer and Application Gateway, and the Kubernetes API Server (AKS management nodes) connected via Private Link. It uses Terraform as that can create all the Azure AD, Azure and Kubernetes resources required. 6 | 7 | ## Description 8 | ![alt text](visuals/diagram.png "Network view") 9 | 10 | When you create an Azure Kubernetes Service (AKS) in the Azure Portal, or with the Azure CLI, by default it will be open in the sense of traffic (both application & management) using public IP addresses. This is a challenge in Enterprise, especially in regulated industries. Effort is needed to embed services in Virtual Networks, and in the case of AKS there are quite a few moving parts to consider. 11 | 12 | To constrain connectivity to/from an AKS cluster, the following available measures are implemented in this repo: 13 | 14 | 1. The Kubernetes API server is the entry point for Kubernetes management operations, and is hosted as a multi-tenant PaaS service by Microsoft. The API server can be projected in the Virtual Network using Private Link ([article](https://docs.microsoft.com/en-us/azure/aks/private-clusters)) 15 | 1. Instead of an external Load Balancer (with a public IP address), use an internal Load Balancer ([article](https://docs.microsoft.com/en-us/azure/aks/internal-lb)) 16 | limit-egress-traffic#restrict-egress-traffic-using-azure-firewall)) 17 | 1. Application Gateway can be used to manage ingress traffic. There are multiple ways to set this up, by far the simplest is to use the AKS add on. This lets AKS create the Application Gateway and maintain it's configuration ([article](https://docs.microsoft.com/en-us/azure/application-gateway/tutorial-ingress-controller-add-on-existing)) 18 | 1. Use [user defined routes](https://docs.microsoft.com/en-us/azure/aks/egress-outboundtype) and an Azure Firewall to manage egress, instead of breaking out to the Internet directly ([article](https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#restrict-egress-traffic-using-azure-firewall)) 19 | 1. Connect Azure Container Registry directly to the Virtual Network using a [Private Endpoint](https://docs.microsoft.com/en-us/azure/private-link/private-endpoint-overview). 20 | 1. To prevent yourself from being boxed in, CI/CD should be able to access the cluster. See [connectivity](#connectivity) below. 21 | 22 | Note 2. and 3. are overlapping, you only need one of both. 23 | 24 | ### AKS Networking modes 25 | AKS supports two networking 'modes'. These modes control the IP address allocation of the agent nodes in the cluster. In short: 26 | - [kubenet](https://docs.microsoft.com/en-us/azure/aks/configure-kubenet) creates IP addresses in a different address space, and uses NAT (network address translation) to expose the agents. This is where the Kubernetes term 'external IP' comes from, this is a private IP address known to the rest of the network. 27 | - [Azure CNI](https://docs.microsoft.com/en-us/azure/aks/configure-azure-cni) uses the same address space for agents as the rest of the virtual network. 28 | See [comparison](https://docs.microsoft.com/en-us/azure/aks/concepts-network#compare-network-models) 29 | 30 | I won't go into detail of these modes, as the network mode is __irrelevant__ for the isolation measures you need to take. Choosing one over the other does not make a major difference for network isolation. This deployment has been [tested](https://dev.azure.com/ericvan/VDC/_build/latest?definitionId=85&branchName=main) with Azure CNI. 31 | 32 | ## Pre-requisites 33 | ### Tools 34 | - To get started you need [Git](https://git-scm.com/), [Terraform](https://www.terraform.io/downloads.html) (to get that you can use [tfenv](https://github.com/tfutils/tfenv) on Linux & macOS, [Homebrew](https://github.com/hashicorp/homebrew-tap) on macOS or [chocolatey](https://chocolatey.org/packages/terraform) on Windows) 35 | - Some scripts require [PowerShell](https://github.com/PowerShell/PowerShell#get-powershell) 36 | - Application deployment requires [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) 37 | 38 | If you're on macOS, you can run `brew bundle` in the repo root to get the required tools, as there is a `Brewfile`. 39 | 40 | ### Connectivity 41 | As this provisions an isolated AKS, how will you be able to access the AKS cluster once deployed? If you set the `peer_network_id` Terraform variable to a network where you're running Terraform from (or you are connected to e.g. using VPN), this project will set up the peering and Private DNS link required to look up the Kubernetes API Server and access cluster nodes. Without this you can only perform partial deployment, you won't be able to deploy applications. 42 | Example connectivity scenarios: 43 | - The [pipeline](pipelines/azure-aks-ci.yml) in this repository uses a [Scale Set Agent pool](https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/scale-set-agents?view=azure-devops) deployed into a virtual network by [geekzter/azure-pipeline-agents](https://github.com/geekzter/azure-pipeline-agents) 44 | - You can use [geekzter/azure-devenv](https://github.com/geekzter/azure-devenv) to create a fully prepped development VM in a VNet, and deploy from there 45 | 46 | ## Provisioning 47 | 1. Clone repository: 48 | `git clone https://github.com/geekzter/azure-aks.git` 49 | 50 | 1. Change to the [`terraform`](./terraform) directrory 51 | `cd azure-aks/terraform` 52 | 53 | 1. Login to Azure with Azure CLI: 54 | `az login` 55 | 56 | 1. This also authenticates the Terraform [azuread](https://registry.terraform.io/providers/hashicorp/azuread/latest/docs/guides/azure_cli) and [azurerm](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/azure_cli) providers. Optionally, you can select the subscription to target: 57 | `az account set --subscription 00000000-0000-0000-0000-000000000000` 58 | `ARM_SUBSCRIPTION_ID=$(az account show --query id -o tsv)` (bash, zsh) 59 | `$env:ARM_SUBSCRIPTION_ID=$(az account show --query id -o tsv)` (pwsh) 60 | 61 | 1. You can then provision resources by first initializing Terraform: 62 | `terraform init` 63 | 64 | 1. And then running: 65 | `terraform apply` 66 | 67 | 1. Demo applications are deployed with the following script: 68 | `scripts/deploy_app.ps1` 69 | This script will output the url's used by the demo applications. One application is exposed via Application Gateway and is publically accessible, the other over the internal Load Balancer. 70 | 71 | Once deployed the applications will look like this: 72 | 73 | Internal Load Balancer: Voting App |Application Gateway: ASP.NET App 74 | :----------------:|:-----------------: 75 | ![](visuals/votingapp.png)|![](visuals/aspnetapp.png) 76 | 77 | 78 | ### Clean Up 79 | When you want to destroy resources, run: 80 | `terraform destroy` 81 | 82 | 83 | ## Resources 84 | - [Azure Kubernetes Service checklist](https://www.the-aks-checklist.com/) 85 | - [Baseline architecture for an Azure Kubernetes Service (AKS) cluster](https://aka.ms/architecture/aks-baseline) 86 | - [Enterprise-Scale Construction Set for Azure Kubernetes Services using Terraform](https://github.com/Azure/caf-terraform-landingzones-starter/tree/starter/enterprise_scale/construction_sets/aks/online/aks_secure_baseline) 87 | -------------------------------------------------------------------------------- /kusto/denied-outbound-http-traffic.kql: -------------------------------------------------------------------------------- 1 | // Taken from https://docs.microsoft.com/en-us/azure/firewall/log-analytics-samples 2 | AzureDiagnostics 3 | | where Category == "AzureFirewallApplicationRule" 4 | | parse msg_s with Protocol " request from " SourceIP ":" SourcePortInt:int " " TempDetails 5 | | parse TempDetails with "was " Action1 ". Reason: " Rule1 6 | | parse TempDetails with "to " FQDN ":" TargetPortInt:int ". Action: " Action2 "." * 7 | | parse TempDetails with * ". Rule Collection: " RuleCollection2a ". Rule:" Rule2a 8 | | parse TempDetails with * "Deny." RuleCollection2b ". Proceeding with" Rule2b 9 | | extend TargetPort = tostring(TargetPortInt) 10 | | extend Action1 = case(Action1 == "Deny","Deny","Unknown Action") 11 | | extend Action = case(Action2 == "",Action1,Action2),Rule = case(Rule2a == "", case(Rule1 == "",case(Rule2b == "","N/A", Rule2b),Rule1),Rule2a), 12 | RuleCollection = case(RuleCollection2b == "",case(RuleCollection2a == "","No rule matched",RuleCollection2a), RuleCollection2b),FQDN = case(FQDN == "", "N/A", FQDN),TargetPort = case(TargetPort == "", "N/A", TargetPort) 13 | | project TimeGenerated, SourceIP, FQDN, TargetPort, Action ,RuleCollection, Rule 14 | | order by TimeGenerated desc 15 | | where Action == "Deny" -------------------------------------------------------------------------------- /kusto/denied-outbound-traffic.kql: -------------------------------------------------------------------------------- 1 | AzureDiagnostics 2 | | where Category == "AzureFirewallNetworkRule" 3 | | parse msg_s with Protocol " request from " SourceIP ":" SourcePortInt:int " to " TargetIP ":" TargetPortInt:int * 4 | | parse msg_s with * ". Action: " Action1a 5 | | parse msg_s with * " was " Action1b " to " NatDestination 6 | | parse msg_s with Protocol2 " request from " SourceIP2 " to " TargetIP2 ". Action: " Action2 7 | | extend SourcePort = tostring(SourcePortInt),Port = tostring(TargetPortInt) 8 | | extend Action = case(Action1a == "", case(Action1b == "",Action2,Action1b), Action1a),Protocol = case(Protocol == "", Protocol2, Protocol),SourceIP = case(SourceIP == "", SourceIP2, SourceIP),TargetIP = case(TargetIP == "", TargetIP2, TargetIP),SourcePort = case(SourcePort == "", "N/A", SourcePort),Port = case(Port == "", "N/A", Port),NatDestination = case(NatDestination == "", "N/A", NatDestination) 9 | | where Action == "Deny" 10 | | order by TimeGenerated desc 11 | | project TimeGenerated, Protocol, SourceIP,TargetIP,Port -------------------------------------------------------------------------------- /manifests/aks-helloworld-one.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: Deployment 3 | metadata: 4 | name: aks-helloworld-one 5 | spec: 6 | replicas: 1 7 | selector: 8 | matchLabels: 9 | app: aks-helloworld-one 10 | template: 11 | metadata: 12 | labels: 13 | app: aks-helloworld-one 14 | spec: 15 | containers: 16 | - name: aks-helloworld-one 17 | image: mcr.microsoft.com/azuredocs/aks-helloworld:v1 18 | ports: 19 | - containerPort: 80 20 | env: 21 | - name: TITLE 22 | value: "Welcome to Azure Kubernetes Service (AKS)" 23 | --- 24 | apiVersion: v1 25 | kind: Service 26 | metadata: 27 | name: aks-helloworld-one 28 | spec: 29 | type: ClusterIP 30 | ports: 31 | - port: 80 32 | selector: 33 | app: aks-helloworld-one -------------------------------------------------------------------------------- /manifests/aks-helloworld-two.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: Deployment 3 | metadata: 4 | name: aks-helloworld-two 5 | spec: 6 | replicas: 1 7 | selector: 8 | matchLabels: 9 | app: aks-helloworld-two 10 | template: 11 | metadata: 12 | labels: 13 | app: aks-helloworld-two 14 | spec: 15 | containers: 16 | - name: aks-helloworld-two 17 | image: mcr.microsoft.com/azuredocs/aks-helloworld:v1 18 | ports: 19 | - containerPort: 80 20 | env: 21 | - name: TITLE 22 | value: "AKS Ingress Demo" 23 | --- 24 | apiVersion: v1 25 | kind: Service 26 | metadata: 27 | name: aks-helloworld-two 28 | spec: 29 | type: ClusterIP 30 | ports: 31 | - port: 80 32 | selector: 33 | app: aks-helloworld-two -------------------------------------------------------------------------------- /manifests/aspnetapp.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Pod 3 | metadata: 4 | name: aspnetapp 5 | labels: 6 | app: aspnetapp 7 | spec: 8 | containers: 9 | - image: "mcr.microsoft.com/dotnet/samples:aspnetapp" 10 | name: aspnetapp-image 11 | ports: 12 | - containerPort: 80 13 | protocol: TCP 14 | 15 | --- 16 | 17 | apiVersion: v1 18 | kind: Service 19 | metadata: 20 | name: aspnetapp 21 | spec: 22 | selector: 23 | app: aspnetapp 24 | ports: 25 | - protocol: TCP 26 | port: 80 27 | targetPort: 80 28 | 29 | --- 30 | 31 | apiVersion: networking.k8s.io/v1 32 | kind: Ingress 33 | metadata: 34 | name: aspnetapp 35 | annotations: 36 | kubernetes.io/ingress.class: azure/application-gateway 37 | spec: 38 | rules: 39 | - http: 40 | paths: 41 | - path: / 42 | backend: 43 | service: 44 | name: aspnetapp 45 | port: 46 | number: 80 47 | pathType: Exact 48 | -------------------------------------------------------------------------------- /manifests/hello-world-ingress.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: networking.k8s.io/v1beta1 2 | kind: Ingress 3 | metadata: 4 | name: hello-world-ingress 5 | namespace: ingress-basic 6 | annotations: 7 | kubernetes.io/ingress.class: nginx 8 | nginx.ingress.kubernetes.io/ssl-redirect: "false" 9 | nginx.ingress.kubernetes.io/use-regex: "true" 10 | nginx.ingress.kubernetes.io/rewrite-target: /$1 11 | spec: 12 | rules: 13 | - http: 14 | paths: 15 | - backend: 16 | serviceName: aks-helloworld-one 17 | servicePort: 80 18 | path: /hello-world-one(/|$)(.*) 19 | - backend: 20 | serviceName: aks-helloworld-two 21 | servicePort: 80 22 | path: /hello-world-two(/|$)(.*) 23 | - backend: 24 | serviceName: aks-helloworld-one 25 | servicePort: 80 26 | path: /(.*) 27 | --- 28 | apiVersion: networking.k8s.io/v1beta1 29 | kind: Ingress 30 | metadata: 31 | name: hello-world-ingress-static 32 | namespace: ingress-basic 33 | annotations: 34 | kubernetes.io/ingress.class: nginx 35 | nginx.ingress.kubernetes.io/ssl-redirect: "false" 36 | nginx.ingress.kubernetes.io/rewrite-target: /static/$2 37 | spec: 38 | rules: 39 | - http: 40 | paths: 41 | - backend: 42 | serviceName: aks-helloworld-one 43 | servicePort: 80 44 | path: /static(/|$)(.*) -------------------------------------------------------------------------------- /manifests/internal-vote.yaml: -------------------------------------------------------------------------------- 1 | # Merged: 2 | # https://docs.microsoft.com/en-us/azure/aks/internal-lb#create-an-internal-load-balancer 3 | # https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough-rm-template#run-the-application 4 | apiVersion: apps/v1 5 | kind: Deployment 6 | metadata: 7 | name: azure-vote-back 8 | spec: 9 | replicas: 1 10 | selector: 11 | matchLabels: 12 | app: azure-vote-back 13 | template: 14 | metadata: 15 | labels: 16 | app: azure-vote-back 17 | spec: 18 | nodeSelector: 19 | "kubernetes.io/os": linux 20 | containers: 21 | - name: azure-vote-back 22 | image: mcr.microsoft.com/oss/bitnami/redis:6.0.8 23 | env: 24 | - name: ALLOW_EMPTY_PASSWORD 25 | value: "yes" 26 | resources: 27 | requests: 28 | cpu: 100m 29 | memory: 128Mi 30 | limits: 31 | cpu: 250m 32 | memory: 256Mi 33 | ports: 34 | - containerPort: 6379 35 | name: redis 36 | --- 37 | apiVersion: v1 38 | kind: Service 39 | metadata: 40 | name: azure-vote-back 41 | spec: 42 | ports: 43 | - port: 6379 44 | selector: 45 | app: azure-vote-back 46 | --- 47 | apiVersion: apps/v1 48 | kind: Deployment 49 | metadata: 50 | name: azure-vote-front 51 | spec: 52 | replicas: 1 53 | selector: 54 | matchLabels: 55 | app: azure-vote-front 56 | template: 57 | metadata: 58 | labels: 59 | app: azure-vote-front 60 | spec: 61 | nodeSelector: 62 | "kubernetes.io/os": linux 63 | containers: 64 | - name: azure-vote-front 65 | image: mcr.microsoft.com/azuredocs/azure-vote-front:v1 66 | resources: 67 | requests: 68 | cpu: 100m 69 | memory: 128Mi 70 | limits: 71 | cpu: 250m 72 | memory: 256Mi 73 | ports: 74 | - containerPort: 80 75 | env: 76 | - name: REDIS 77 | value: "azure-vote-back" 78 | --- 79 | apiVersion: v1 80 | kind: Service 81 | metadata: 82 | name: azure-vote-front 83 | annotations: 84 | service.beta.kubernetes.io/azure-load-balancer-internal: "true" 85 | spec: 86 | type: LoadBalancer 87 | ports: 88 | - port: 80 89 | selector: 90 | app: azure-vote-front -------------------------------------------------------------------------------- /pipelines/azure-aks-ci.yml: -------------------------------------------------------------------------------- 1 | parameters: 2 | - name: workspace 3 | displayName: Pipeline Environment / Terraform Workspace 4 | type: string 5 | default: ci 6 | values: 7 | - ci 8 | - ci1 9 | - ci2 10 | - ci3 11 | - cd 12 | - cd1 13 | - cd2 14 | - cd3 15 | - name: clear 16 | displayName: Clear state 17 | type: boolean 18 | default: false 19 | - name: deploy 20 | displayName: Deploy applications 21 | type: boolean 22 | default: true 23 | - name: destroy 24 | displayName: Destroy infrastructure 25 | type: string 26 | default: Always 27 | values: 28 | - Always 29 | - Never 30 | - 'On failure' 31 | - 'On success' 32 | - name: testReentrance 33 | displayName: Test Terraform re-entrance (apply twice) 34 | type: boolean 35 | default: true 36 | - name: unpinTerraform 37 | displayName: Unpin Terraform version 38 | type: boolean 39 | default: false 40 | - name: unpinTerraformProviders 41 | displayName: Unpin Terraform provider versions 42 | type: string 43 | default: No 44 | values: 45 | - No 46 | - Yes 47 | - Strategy 48 | - name: useDefaultK8sVersion 49 | displayName: Use default Kubernetes version 50 | type: boolean 51 | default: true 52 | 53 | name: $(Date:yyyyMMdd)$(Rev:.r)-$(Build.DefinitionVersion)-$(SourceBranchName)-${{ parameters.workspace }}-$(Build.BuildId) 54 | 55 | trigger: none 56 | 57 | pr: 58 | branches: 59 | include: 60 | - '*' 61 | paths: 62 | exclude: 63 | - '.devcontainer/**' 64 | - 'visuals/**' 65 | - '*.md' 66 | 67 | schedules: 68 | - cron: '0 1 * * Mon,Wed,Fri' 69 | displayName: 'Bi-Nightly build (UTC)' 70 | # Run if there are no changes 71 | always: 'true' 72 | branches: 73 | include: 74 | - main 75 | 76 | resources: 77 | repositories: 78 | - repository: azure-identity-scripts 79 | type: github 80 | endpoint: github.com # Service Connection name 81 | name: geekzter/azure-identity-scripts 82 | 83 | variables: 84 | - group: 'aks-ci' 85 | - name: 'jobTimeOutMinutes' 86 | value: 180 87 | - name: 'repository' 88 | value: 'azure-aks' 89 | - name: AZURE_CORE_ONLY_SHOW_ERRORS 90 | value: 'true' 91 | - name: AZURE_EXTENSION_USE_DYNAMIC_INSTALL 92 | value: 'yes_without_prompt' 93 | - name: 'TF_IN_AUTOMATION' 94 | value: 'true' 95 | - name: 'TF_INPUT' 96 | value: 0 97 | - name: 'TF_WORKSPACE' 98 | value: ${{ parameters.workspace }} 99 | - name: 'identityScriptDirectory' 100 | value: '$(Build.SourcesDirectory)/azure-identity-scripts/scripts/azure-devops' 101 | - name: 'manifestDirectory' 102 | value: '$(Build.SourcesDirectory)/azure-aks/manifests' 103 | - name: 'scriptDirectory' 104 | value: '$(Build.SourcesDirectory)/azure-aks/scripts' 105 | - name: 'terraformDirectory' 106 | value: '$(Build.SourcesDirectory)/azure-aks/terraform' 107 | - name: 'TF_VAR_run_id' 108 | value: '$(Build.BuildId)' 109 | 110 | jobs: 111 | - job: 'Provision' 112 | ${{ if and(eq(parameters.destroy, 'Always'),parameters.deploy) }}: 113 | displayName: 'Provision (${{ parameters.workspace }}), Deploy, Test & Destroy' 114 | ${{ if and(eq(parameters.destroy, 'Never'),parameters.deploy) }}: 115 | displayName: 'Provision (${{ parameters.workspace }}), Deploy & Test' 116 | ${{ if and(eq(parameters.destroy, 'On failure'),parameters.deploy) }}: 117 | displayName: 'Provision (${{ parameters.workspace }}), Deploy, Test & Destroy (${{ lower(parameters.destroy) }})' 118 | ${{ if and(eq(parameters.destroy, 'On success'),parameters.deploy) }}: 119 | displayName: 'Provision (${{ parameters.workspace }}), Deploy, Test & Destroy (${{ lower(parameters.destroy) }})' 120 | ${{ if and(eq(parameters.destroy, 'Always'),not(parameters.deploy)) }}: 121 | displayName: 'Provision (${{ parameters.workspace }}) & Destroy' 122 | ${{ if and(eq(parameters.destroy, 'Never'),not(parameters.deploy)) }}: 123 | displayName: 'Provision (${{ parameters.workspace }})' 124 | ${{ if and(eq(parameters.destroy, 'On failure'),not(parameters.deploy)) }}: 125 | displayName: 'Provision (${{ parameters.workspace }}) & Destroy (${{ lower(parameters.destroy) }})' 126 | ${{ if and(eq(parameters.destroy, 'On success'),not(parameters.deploy)) }}: 127 | displayName: 'Provision (${{ parameters.workspace }}) & Destroy (${{ lower(parameters.destroy) }})' 128 | condition: succeeded() 129 | timeoutInMinutes: $[ variables['jobTimeOutMinutes'] ] 130 | 131 | pool: 132 | name: '$(pool)' 133 | vmImage: $(vmImage) 134 | 135 | ${{ if or(eq(parameters.unpinTerraformProviders, 'Strategy'),not(eq(variables['Build.Reason'], 'Manual'))) }}: 136 | strategy: 137 | matrix: 138 | pinTerraformProviders: 139 | randomSeed: $(Build.BuildId)0 140 | resourceGroup: '$(TF_VAR_resource_prefix)-${{ parameters.workspace }}a-$(Build.BuildId)' 141 | substituteAlternateVariables: false 142 | terraformArtifactName: 'terraformPrimary' 143 | TF_VAR_resource_suffix: '$(Build.BuildId)' 144 | TF_WORKSPACE: '${{ parameters.workspace }}a' 145 | unpinTerraformProviders: ${{ lower(eq(parameters.unpinTerraformProviders, 'Yes')) }} 146 | unpinTerraformProviders: 147 | randomSeed: $(Build.BuildId)5 148 | resourceGroup: '$(TF_VAR_resource_prefix)-${{ parameters.workspace }}b-$(Build.BuildId)' 149 | substituteAlternateVariables: true 150 | terraformArtifactName: 'terraformAlternate' 151 | TF_VAR_resource_suffix: '$(Build.BuildId)' 152 | TF_WORKSPACE: '${{ parameters.workspace }}b' 153 | unpinTerraformProviders: ${{ lower(or(eq(parameters.unpinTerraformProviders, 'Yes'),eq(parameters.unpinTerraformProviders, 'Strategy'),ne(variables['Build.Reason'], 'Manual'))) }} 154 | maxParallel: 2 155 | 156 | variables: 157 | ${{ if not(or(eq(parameters.unpinTerraformProviders, 'Strategy'),not(eq(variables['Build.Reason'], 'Manual')))) }}: 158 | # Not runnig as strategy 159 | terraformArtifactName: 'terraformPrimary' 160 | ${{ if parameters.clear }}: # Don't reset suffix if we want to keep existing resources 161 | TF_VAR_resource_suffix: '$(Build.BuildId)' 162 | TF_WORKSPACE: '${{ parameters.workspace }}' 163 | unpinTerraformProviders: ${{ eq(parameters.unpinTerraformProviders, 'Yes') }} 164 | randomSeed: $(Build.BuildId) 165 | resourceGroup: '$(TF_VAR_resource_prefix)-$(TF_WORKSPACE)-$(TF_VAR_resource_suffix)' 166 | 167 | workspace: 168 | clean: all 169 | 170 | steps: 171 | - checkout: self 172 | - checkout: azure-identity-scripts 173 | 174 | - ${{ if parameters.clear }}: 175 | - task: AzureCLI@2 176 | name: clear 177 | displayName: 'Remove conflicting resources from previous runs' 178 | inputs: 179 | azureSubscription: '$(subscriptionConnection)' 180 | scriptType: pscore 181 | scriptLocation: inlineScript 182 | inlineScript: | 183 | # Remove VNet peerings 184 | if ($env:PIPELINE_DEMO_AGENT_VIRTUAL_NETWORK_ID) { 185 | $peeringIDs = $(az network vnet show --ids $env:PIPELINE_DEMO_AGENT_VIRTUAL_NETWORK_ID --query "virtualNetworkPeerings[?starts_with(name,'k8s-${env:TF_WORKSPACE}-')].id" -o tsv 2>$null) 186 | if ($peeringIDs) { 187 | Write-Host "Removing virtual network peerings `"${peeringIDs}`"..." 188 | &{ # az writes information to stderr 189 | $ErrorActionPreference = 'SilentlyContinue' 190 | az resource delete --ids $peeringIDs 2>&1 191 | } 192 | } else { 193 | Write-Host "No virtual network peerings to remove" 194 | } 195 | } 196 | useGlobalConfig: true 197 | failOnStandardError: true 198 | workingDirectory: '$(terraformDirectory)' 199 | 200 | - ${{ if not(parameters.unpinTerraform) }}: 201 | - pwsh: | 202 | $terraformVersion = (Get-Content .terraform-version) 203 | Write-Host "##vso[task.setvariable variable=version;isOutput=true]${terraformVersion}" 204 | Copy-Item backend.tf.sample backend.tf 205 | name: terraformConfig 206 | displayName: 'Prepare Terraform config' 207 | workingDirectory: '$(terraformDirectory)' 208 | 209 | - ${{ if parameters.unpinTerraform }}: 210 | - pwsh: | 211 | (Get-Content ./provider.tf) -replace "required_version *= `" *(~>|=) +",'required_version = ">= ' | Out-File provider.tf 212 | Get-Content ./provider.tf 213 | Write-Host "##vso[task.setvariable variable=version;isOutput=true]latest" 214 | Copy-Item backend.tf.sample backend.tf 215 | name: terraformConfig 216 | displayName: 'Prepare Terraform config (latest version)' 217 | workingDirectory: '$(terraformDirectory)' 218 | 219 | - task: TerraformInstaller@1 220 | displayName: 'Install terraform' 221 | inputs: 222 | terraformVersion: '$(terraformConfig.version)' 223 | 224 | - ${{ if or(eq(parameters.unpinTerraformProviders, 'Yes'),eq(parameters.unpinTerraformProviders, 'Strategy'),not(eq(variables['Build.Reason'], 'Manual'))) }}: 225 | # Unpin version e.g. "= 2.56" -> "~> 2.56" 226 | - pwsh: | 227 | (Get-Content ./provider.tf) -replace " = `" *= +",' = "~> ' | Out-File provider.tf 228 | Get-Content ./provider.tf 229 | if (Test-Path .terraform.lock.hcl) { 230 | Remove-Item .terraform.lock.hcl -Force 231 | } 232 | displayName: 'Unpin Terraform provider versions' 233 | # condition required as '- ${{ if ' template expression is not evaluated when using a strategy 234 | condition: and(succeeded(), eq(variables['unpinTerraformProviders'],'true')) 235 | workingDirectory: '$(terraformDirectory)' 236 | 237 | - task: TerraformCLI@1 238 | displayName: 'Terraform init' 239 | inputs: 240 | command: 'init' 241 | workingDirectory: '$(terraformDirectory)' 242 | backendType: 'azurerm' 243 | backendServiceArm: '$(subscriptionConnection)' 244 | backendAzureRmResourceGroupName: '$(TF_STATE_RESOURCE_GROUP_NAME)' 245 | backendAzureRmStorageAccountName: '$(TF_STATE_STORAGE_ACCOUNT_NAME)' 246 | backendAzureRmContainerName: '$(TF_STATE_CONTAINER_NAME)' 247 | backendAzureRmKey: 'terraform.tfstate' 248 | allowTelemetryCollection: true 249 | 250 | - publish: $(terraformDirectory) 251 | displayName: 'Publish Terraform workspace' 252 | artifact: $(terraformArtifactName) 253 | 254 | - ${{ if parameters.clear }}: 255 | - task: AzureCLI@2 256 | name: cleanup 257 | displayName: 'Clear Terraform state' 258 | inputs: 259 | azureSubscription: '$(subscriptionConnection)' 260 | scriptType: pscore 261 | scriptLocation: inlineScript 262 | inlineScript: | 263 | $(identityScriptDirectory)/set_terraform_azurerm_vars.ps1 264 | 265 | $terraformState = (terraform state pull | ConvertFrom-Json) 266 | if ($terraformState.resources) { 267 | Write-Host "Clearing Terraform state in workspace ${env:TF_WORKSPACE}..." 268 | $terraformState.outputs = New-Object PSObject # Empty output 269 | $terraformState.resources = @() # No resources 270 | $terraformState.serial++ 271 | $terraformState | ConvertTo-Json | terraform state push - 272 | } else { 273 | Write-Host "No resources in Terraform state in workspace ${env:TF_WORKSPACE}..." 274 | } 275 | terraform state pull 276 | addSpnToEnvironment: true 277 | useGlobalConfig: true 278 | failOnStandardError: true 279 | workingDirectory: '$(terraformDirectory)' 280 | 281 | - ${{ if not(parameters.clear) }}: 282 | - task: AzureCLI@2 283 | displayName: 'Restore kubeconfig' 284 | inputs: 285 | azureSubscription: '$(subscriptionConnection)' 286 | scriptType: pscore 287 | scriptLocation: inlineScript 288 | inlineScript: | 289 | if ((${env:system.debug} -eq "true") -or ($env:system_debug -eq "true") -or ($env:SYSTEM_DEBUG -eq "true")) { 290 | $DebugPreference = "Continue" 291 | $InformationPreference = "Continue" 292 | $VerbosePreference = "Continue" 293 | Set-PSDebug -Trace 2 294 | } 295 | ./kube_config.ps1 296 | addSpnToEnvironment: true 297 | useGlobalConfig: true 298 | failOnStandardError: true 299 | workingDirectory: '$(scriptDirectory)' 300 | 301 | - task: AzureCLI@2 302 | displayName: 'Prepare Terraform variables' 303 | inputs: 304 | azureSubscription: '$(subscriptionConnection)' 305 | scriptType: pscore 306 | scriptLocation: inlineScript 307 | inlineScript: | 308 | if ($${{ parameters.useDefaultK8sVersion }} -and !($env:TF_VAR_kubernetes_version -or $env:TF_VAR_KUBERNETES_VERSION)) { 309 | # Use Azure CLI default (Terraform may use a different one) 310 | az aks get-versions -l $(TF_VAR_location) --query "orchestrators[?default].orchestratorVersion" -o tsv | Set-Item env:TF_VAR_kubernetes_version 311 | } 312 | 313 | # Use pipeline agent VNet as network to peer from 314 | $env:TF_VAR_peer_network_id ??= $env:PIPELINE_DEMO_AGENT_VIRTUAL_NETWORK_ID 315 | 316 | if ($env:TF_VAR_ADDRESS_SPACE) { 317 | Remove-Item env:TF_VAR_ADDRESS_SPACE 2>$null # Override below 318 | } 319 | if ($${{ not(parameters.clear) }} -and (!((terraform output resource_suffix 2>&1) -match "Warning"))) { 320 | # Keep random values of previous runs 321 | $env:TF_VAR_address_space = "$(terraform output -raw address_space 2>$null)" 322 | $env:TF_VAR_RESOURCE_SUFFIX = $null 323 | $env:TF_VAR_resource_suffix = "$(terraform output -raw resource_suffix 2>$null)" 324 | } 325 | # Set random CIDR (to reduce the risk of clashing VNet peerings with agent VNet) 326 | $env:TF_VAR_address_space ??= "$([IPAddress]::Parse([String] (167772160 + (65536*(Get-Random -Minimum 0 -Maximum 255 -SetSeed $(randomSeed))))) | Select-Object -ExpandProperty IPAddressToString)/16" 327 | 328 | # List environment variables 329 | Get-ChildItem -Path Env: -Recurse -Include ARM_*,AZURE_*,PIPELINE_*,TF_* | Sort-Object -Property Name 330 | 331 | # Convert uppercased Terraform environment variables to .auto.tfvars file 332 | foreach ($tfvar in $(Get-ChildItem -Path Env: -Recurse -Include TF_VAR_*)) { 333 | $terraformVariableName = $tfvar.Name.Substring(7).ToLowerInvariant() 334 | $terraformVariableValue = $tfVar.Value 335 | 336 | if ($terraformVariableValue -imatch "^\W*(true|false|\[[^\]]*\]|\{[^\}]*\})\W*$") { 337 | # Boolean or List, write as-is 338 | Write-Output "${terraformVariableName} = ${terraformVariableValue}" | Out-File ci.auto.tfvars -Append -Force 339 | } else { 340 | Write-Output "${terraformVariableName} = `"${terraformVariableValue}`"" | Out-File ci.auto.tfvars -Append -Force 341 | } 342 | } 343 | Write-Host "Contents of ci.auto.tfvars:" 344 | Get-Content ci.auto.tfvars 345 | 346 | useGlobalConfig: true 347 | failOnStandardError: true 348 | workingDirectory: '$(terraformDirectory)' 349 | 350 | - task: AzureCLI@2 351 | displayName: 'Terraform plan & apply' 352 | name: terraformApply 353 | inputs: 354 | azureSubscription: '$(subscriptionConnection)' 355 | scriptType: pscore 356 | scriptLocation: inlineScript 357 | inlineScript: | 358 | $(identityScriptDirectory)/set_terraform_azurerm_vars.ps1 359 | ./deploy.ps1 -apply -force 360 | addSpnToEnvironment: true 361 | useGlobalConfig: true 362 | failOnStandardError: true 363 | retryCountOnTaskFailure: 3 364 | workingDirectory: '$(scriptDirectory)' 365 | 366 | - ${{ if parameters.deploy }}: 367 | - task: KubectlInstaller@0 368 | displayName: 'Install kubectl' 369 | condition: and(succeeded(), not(eq(coalesce(variables['terraformApply.aks_name'],'null'),'null')), eq(variables['terraformApply.peered_network'],'true')) 370 | inputs: 371 | kubectlVersion: 'latest' 372 | 373 | - ${{ if or(parameters.deploy, parameters.testReentrance) }}: 374 | - task: AzureCLI@2 375 | displayName: 'Wait for Application Gateway' 376 | condition: and(succeeded(), not(eq(coalesce(variables['terraformApply.application_gateway_id'],'null'),'null'))) 377 | inputs: 378 | azureSubscription: '$(subscriptionConnection)' 379 | scriptType: pscore 380 | scriptLocation: scriptPath 381 | scriptPath: $(scriptDirectory)/wait_for_app_gw.ps1 382 | arguments: 383 | -AksName $(terraformApply.aks_name) ` 384 | -ApplicationGatewayName "$(terraformApply.application_gateway_id)".Split("/")[-1] ` 385 | -ResourceGroupName $(terraformApply.resource_group) 386 | useGlobalConfig: true 387 | failOnStandardError: true 388 | powerShellIgnoreLASTEXITCODE: false 389 | workingDirectory: '$(scriptDirectory)' 390 | 391 | - ${{ if parameters.deploy }}: 392 | - task: AzureCLI@2 393 | condition: and(succeeded(), not(eq(coalesce(variables['terraformApply.aks_name'],'null'),'null')), eq(variables['terraformApply.peered_network'],'true')) 394 | displayName: 'Deploy & Test applications' 395 | inputs: 396 | azureSubscription: '$(subscriptionConnection)' 397 | scriptType: pscore 398 | scriptLocation: inlineScript 399 | inlineScript: | 400 | # Diagnostics 401 | if ((${env:system.debug} -eq "true") -or ($env:system_debug -eq "true") -or ($env:SYSTEM_DEBUG -eq "true")) { 402 | $DebugPreference = "Continue" 403 | $InformationPreference = "Continue" 404 | $VerbosePreference = "Continue" 405 | Set-PSDebug -Trace 2 406 | } 407 | 408 | $(identityScriptDirectory)/set_terraform_azurerm_vars.ps1 409 | ./kube_config.ps1 410 | ./deploy_app.ps1 411 | addSpnToEnvironment: true 412 | useGlobalConfig: true 413 | failOnStandardError: true 414 | powerShellIgnoreLASTEXITCODE: false 415 | workingDirectory: '$(scriptDirectory)' 416 | env: 417 | KUBECONFIG: variables['KUBE_CONFIG_PATH'] 418 | 419 | - ${{ if parameters.testReentrance }}: 420 | - task: AzureCLI@2 421 | displayName: 'Terraform plan & apply (re-entrance test)' 422 | inputs: 423 | azureSubscription: '$(subscriptionConnection)' 424 | scriptType: pscore 425 | scriptLocation: inlineScript 426 | inlineScript: | 427 | $(identityScriptDirectory)/set_terraform_azurerm_vars.ps1 428 | ./deploy.ps1 -apply -force 429 | addSpnToEnvironment: true 430 | useGlobalConfig: true 431 | failOnStandardError: true 432 | retryCountOnTaskFailure: 3 433 | workingDirectory: '$(scriptDirectory)' 434 | 435 | - ${{ if not(eq(parameters.destroy, 'Never')) }}: 436 | - pwsh: | 437 | Write-Host "Indicating success for job '$(Agent.JobName)'" 438 | Write-Host "##vso[task.setvariable variable=result;isOutput=true]success" 439 | name: provisioningResult 440 | displayName: 'Indicate provisioning success' 441 | condition: succeeded() 442 | 443 | - ${{ if not(eq(parameters.destroy, 'Never')) }}: 444 | - task: TerraformCLI@1 445 | displayName: 'Terraform destroy (${{ lower(parameters.destroy) }})' 446 | ${{ if eq(parameters.destroy, 'Always') }}: 447 | condition: succeededOrFailed() 448 | ${{ if eq(parameters.destroy, 'On failure') }}: 449 | condition: failed() 450 | ${{ if eq(parameters.destroy, 'On success') }}: 451 | condition: succeeded() 452 | continueOnError: true # Treat failure as warning during destroy, we will clean up anyway 453 | inputs: 454 | command: 'destroy' 455 | workingDirectory: '$(terraformDirectory)' 456 | environmentServiceName: '$(subscriptionConnection)' 457 | runAzLogin: true 458 | allowTelemetryCollection: true 459 | retryCountOnTaskFailure: 3 460 | env: 461 | KUBECONFIG: variables['KUBE_CONFIG_PATH'] 462 | 463 | - ${{ if ne(parameters.destroy, 'Never') }}: 464 | - task: AzureCLI@2 465 | name: teardown 466 | displayName: 'Tear down remaining resources' 467 | ${{ if eq(parameters.destroy, 'Always') }}: 468 | condition: or(always(),canceled()) 469 | ${{ if eq(parameters.destroy, 'On failure') }}: 470 | condition: not(eq(variables['provisioningResult.result'],'success')) 471 | ${{ if eq(parameters.destroy, 'On success') }}: 472 | condition: eq(variables['provisioningResult.result'],'success') 473 | inputs: 474 | azureSubscription: '$(subscriptionConnection)' 475 | scriptType: pscore 476 | scriptLocation: inlineScript 477 | inlineScript: | 478 | $ErrorActionPreference = "Continue" # Continue to remove resources if remove by resoyrce group fails 479 | # Build JMESPath expression 480 | $tagQuery = "[?tags.repository == '$(repository)' && tags.workspace == '${env:TF_WORKSPACE}' && tags.runid == '$(Build.BuildId)' && properties.provisioningState != 'Deleting'].id" 481 | Write-Host "Removing resources identified by `"$tagQuery`"..." 482 | 483 | # Remove resource groups 484 | $resourceGroupIDs = $(az group list --query "${tagQuery}" -o tsv) 485 | if ($resourceGroupIDs) { 486 | Write-Host "Removing resource group(s) `"${resourceGroupIDs}`"..." 487 | &{ # az writes information to stderr 488 | $ErrorActionPreference = 'SilentlyContinue' 489 | az resource delete --ids $resourceGroupIDs 2>&1 490 | } 491 | } else { 492 | Write-Host "No resource groups to remove" 493 | } 494 | 495 | # Remove (remaining) resources 496 | $resourceIDs = $(az resource list --query "${tagQuery}" -o tsv) 497 | if ($resourceIDs) { 498 | Write-Host "Removing resources `"${resourceIDs}`"..." 499 | &{ # az writes information to stderr 500 | $ErrorActionPreference = 'SilentlyContinue' 501 | az resource delete --ids $resourceIDs 2>&1 502 | } 503 | } else { 504 | Write-Host "No resources to remove" 505 | } 506 | 507 | # Remove VNet peerings 508 | ./cleanup_peerings.ps1 509 | addSpnToEnvironment: true 510 | useGlobalConfig: true 511 | failOnStandardError: true 512 | workingDirectory: '$(scriptDirectory)' 513 | 514 | - ${{ if ne(parameters.destroy, 'Never') }}: 515 | - task: AzureCLI@2 516 | name: cleanup 517 | displayName: 'Clean up Terraform state' 518 | ${{ if eq(parameters.destroy, 'Always') }}: 519 | condition: or(always(),canceled()) 520 | ${{ if eq(parameters.destroy, 'On failure') }}: 521 | condition: not(eq(variables['provisioningResult.result'],'success')) 522 | ${{ if eq(parameters.destroy, 'On success') }}: 523 | condition: eq(variables['provisioningResult.result'],'success') 524 | inputs: 525 | azureSubscription: '$(subscriptionConnection)' 526 | scriptType: pscore 527 | scriptLocation: inlineScript 528 | inlineScript: | 529 | $(identityScriptDirectory)/set_terraform_azurerm_vars.ps1 530 | 531 | $terraformState = (terraform state pull | ConvertFrom-Json) 532 | if ($terraformState.resources) { 533 | Write-Host "Clearing Terraform state in workspace ${env:TF_WORKSPACE}..." 534 | $terraformState.outputs = New-Object PSObject # Empty output 535 | $terraformState.resources = @() # No resources 536 | $terraformState.serial++ 537 | $terraformState | ConvertTo-Json | terraform state push - 538 | } else { 539 | Write-Host "No resources in Terraform state in workspace ${env:TF_WORKSPACE}..." 540 | } 541 | terraform state pull 542 | addSpnToEnvironment: true 543 | useGlobalConfig: true 544 | failOnStandardError: true 545 | workingDirectory: '$(terraformDirectory)' 546 | -------------------------------------------------------------------------------- /scripts/cleanup_peerings.ps1: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env pwsh 2 | <# 3 | .SYNOPSIS 4 | Clean up VNet peerings 5 | #> 6 | #Requires -Version 7.2 7 | param ( 8 | [parameter(Mandatory=$false)] 9 | [ValidateNotNullOrEmpty()] 10 | [string] 11 | $AgentNetworkId=$env:PIPELINE_DEMO_AGENT_VIRTUAL_NETWORK_ID, 12 | 13 | [parameter(Mandatory=$false)] 14 | [string[]] 15 | $Workspace=$env:TF_WORKSPACE ?? "default" 16 | ) 17 | 18 | # az network vnet peering list doesn't support '--ids', peal of elements manually 19 | $agentNetworkName = ($AgentNetworkId -split "/")[-1] 20 | if (!$agentNetworkName) { 21 | Write-Error "AgentNetworkId is not a valid virtual network resource id: '${AgentNetworkId}'" 22 | exit 23 | } 24 | $agentResourceGroupName = ($AgentNetworkId -split "/")[4] 25 | $agentSubscriptionId = ($AgentNetworkId -split "/")[2] 26 | 27 | $jmesPathQuery = "ends_with(name,'from-peer') " 28 | if ($Workspace) { 29 | $jmesPathQuery += " && (" 30 | $firstWorkspace = $true 31 | foreach ($individualWorkspace in $Workspace) { 32 | if (!$firstWorkspace) { 33 | $jmesPathQuery += " || " 34 | } 35 | $jmesPathQuery += " contains(name, 'aks-${individualWorkspace}-') " 36 | $firstWorkspace = $false 37 | } 38 | $jmesPathQuery += ") " 39 | } 40 | Write-Debug "jmesPathQuery: $jmesPathQuery" 41 | 42 | az network vnet peering list -g $agentResourceGroupName ` 43 | --vnet-name $agentNetworkName ` 44 | --subscription $agentSubscriptionId ` 45 | --query "[?${jmesPathQuery}].name" ` 46 | -o tsv ` 47 | | Set-Variable peeringNames 48 | Write-Debug "peerings: $peeringNames" 49 | 50 | if ($peeringNames) { 51 | foreach ($peeringName in $peeringNames) { 52 | Write-Verbose "az network vnet peering delete -g $agentResourceGroupName --vnet-name $agentNetworkName -n $peeringName --subscription $agentSubscriptionId" 53 | az network vnet peering delete -g $agentResourceGroupName ` 54 | --vnet-name $agentNetworkName ` 55 | -n $peeringName ` 56 | --subscription $agentSubscriptionId 57 | } 58 | } else { 59 | Write-Host "No virtual network peerings to remove for workspace '${Workspace}' in network '${AgentNetworkId}'" 60 | } 61 | -------------------------------------------------------------------------------- /scripts/create_nsg_assignment.ps1: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env pwsh 2 | 3 | param ( 4 | [parameter(Mandatory=$true)][string]$SubnetId, 5 | [parameter(Mandatory=$true)][string]$NsgId, 6 | [parameter(Mandatory=$false)][int]$MaxTries=10, 7 | [parameter(Mandatory=$false)][int]$WaitSeconds=10 8 | ) 9 | 10 | az network vnet subnet show --ids ${SubnetId} --query networkSecurityGroup.id -o tsv | Set-Variable existingNsgId 11 | 12 | $tries = 0 13 | while (($NsgId -ine $existingNsgId) -and ($tries -le $MaxTries)) { 14 | Start-Sleep -Seconds $WaitSeconds 15 | $tries++ 16 | az network vnet subnet update --ids ${SubnetId} --nsg ${NsgId} --query networkSecurityGroup.id -o tsv 2>&1 | Set-Variable existingNsgId 17 | } 18 | 19 | if ($tries -gt $MaxTries) { 20 | Write-Error "Failed to update subnet security group after $MaxTries tries" 21 | exit 1 22 | } -------------------------------------------------------------------------------- /scripts/deploy.ps1: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env pwsh 2 | 3 | <# 4 | .SYNOPSIS 5 | Deploys Azure resources using Terraform 6 | 7 | .DESCRIPTION 8 | This script is a wrapper around Terraform. It is provided for convenience only, as it works around some limitations in the demo. 9 | E.g. terraform might need resources to be started before executing, and resources may not be accessible from the current locastion (IP address). 10 | 11 | .EXAMPLE 12 | ./deploy.ps1 -apply 13 | #> 14 | #Requires -Version 7.2 15 | 16 | ### Arguments 17 | param ( 18 | [parameter(Mandatory=$false,HelpMessage="Initialize Terraform backend, modules & provider")][switch]$Init=$false, 19 | [parameter(Mandatory=$false,HelpMessage="Perform Terraform plan stage")][switch]$Plan=$false, 20 | [parameter(Mandatory=$false,HelpMessage="Perform Terraform validate stage")][switch]$Validate=$false, 21 | [parameter(Mandatory=$false,HelpMessage="Perform Terraform apply stage (implies plan)")][switch]$Apply=$false, 22 | [parameter(Mandatory=$false,HelpMessage="Perform Terraform destroy stage")][switch]$Destroy=$false, 23 | [parameter(Mandatory=$false,HelpMessage="Show Terraform output variables")][switch]$Output=$false, 24 | [parameter(Mandatory=$false,HelpMessage="Don't show prompts unless something get's deleted that should not be")][switch]$Force=$false, 25 | [parameter(Mandatory=$false,HelpMessage="Initialize Terraform backend, upgrade modules & provider")][switch]$Upgrade=$false, 26 | [parameter(Mandatory=$false,HelpMessage="Don't try to set up a Terraform backend if it does not exist")][switch]$NoBackend=$false 27 | ) 28 | 29 | ### Internal Functions 30 | . (Join-Path $PSScriptRoot functions.ps1) 31 | 32 | ### Validation 33 | if (!(Get-Command terraform -ErrorAction SilentlyContinue)) { 34 | $tfMissingMessage = "Terraform not found" 35 | if ($IsWindows) { 36 | $tfMissingMessage += "`nInstall Terraform e.g. from Chocolatey (https://chocolatey.org/packages/terraform) 'choco install terraform'" 37 | } else { 38 | $tfMissingMessage += "`nInstall Terraform e.g. using tfenv (https://github.com/tfutils/tfenv)" 39 | } 40 | throw $tfMissingMessage 41 | } 42 | 43 | Write-Information $MyInvocation.line 44 | $script:ErrorActionPreference = "Stop" 45 | 46 | $workspace = Get-TerraformWorkspace 47 | $planFile = "${workspace}.tfplan".ToLower() 48 | $varsFile = "${workspace}.tfvars".ToLower() 49 | $pipeline = ![string]::IsNullOrEmpty($env:AGENT_VERSION) 50 | $inAutomation = ($env:TF_IN_AUTOMATION -ieq "true") 51 | if (($workspace -ieq "prod") -and $Force) { 52 | $Force = $false 53 | Write-Warning "Ignoring -Force in workspace '${workspace}'" 54 | } 55 | 56 | try { 57 | $tfdirectory = (Get-TerraformDirectory) 58 | Push-Location $tfdirectory 59 | AzLogin -DisplayMessages 60 | # Print version info 61 | terraform -version 62 | 63 | if ($Init -or $Upgrade) { 64 | if (!$NoBackend) { 65 | $backendFile = (Join-Path $tfdirectory backend.tf) 66 | $backendTemplate = "${backendFile}.sample" 67 | $newBackend = (!(Test-Path $backendFile)) 68 | $tfbackendArgs = "" 69 | if ($newBackend) { 70 | if (!$env:TF_STATE_backend_storage_account -or !$env:TF_STATE_backend_storage_container) { 71 | Write-Warning "Environment variables TF_STATE_backend_storage_account and TF_STATE_backend_storage_container must be set when creating a new backend from $backendTemplate" 72 | $fail = $true 73 | } 74 | if (!($env:TF_STATE_backend_resource_group -or $env:ARM_ACCESS_KEY -or $env:ARM_SAS_TOKEN)) { 75 | Write-Warning "Environment variables ARM_ACCESS_KEY or ARM_SAS_TOKEN or TF_STATE_backend_resource_group (with $identity granted 'Storage Blob Data Contributor' role) must be set when creating a new backend from $backendTemplate" 76 | $fail = $true 77 | } 78 | if ($fail) { 79 | Write-Warning "This script assumes Terraform backend exists at ${backendFile}, but it does not exist" 80 | Write-Host "You can copy ${backendTemplate} -> ${backendFile} and configure a storage account manually" 81 | Write-Host "See documentation at https://www.terraform.io/docs/backends/types/azurerm.html" 82 | exit 83 | } 84 | 85 | # Terraform azurerm backend does not exist, create one 86 | Write-Host "Creating '$backendFile'" 87 | Copy-Item -Path $backendTemplate -Destination $backendFile 88 | 89 | $tfbackendArgs += " -reconfigure" 90 | } 91 | 92 | if ($env:TF_STATE_backend_resource_group) { 93 | $tfbackendArgs += " -backend-config=`"resource_group_name=${env:TF_STATE_backend_resource_group}`"" 94 | } 95 | if ($env:TF_STATE_backend_storage_account) { 96 | $tfbackendArgs += " -backend-config=`"storage_account_name=${env:TF_STATE_backend_storage_account}`"" 97 | } 98 | if ($env:TF_STATE_backend_storage_container) { 99 | $tfbackendArgs += " -backend-config=`"container_name=${env:TF_STATE_backend_storage_container}`"" 100 | } 101 | } 102 | 103 | $initCmd = "terraform init $tfbackendArgs" 104 | if ($Upgrade) { 105 | $initCmd += " -upgrade" 106 | } 107 | Invoke "$initCmd" 108 | } 109 | 110 | if ($Validate) { 111 | Invoke "terraform validate" 112 | } 113 | 114 | # Prepare common arguments 115 | if ($Force) { 116 | $forceArgs = "-auto-approve" 117 | } 118 | 119 | if (!(Get-ChildItem Env:TF_VAR_* -Exclude TF_STATE_backend_*) -and (Test-Path $varsFile)) { 120 | # Load variables from file, if it exists and environment variables have not been set 121 | $varArgs = " -var-file='$varsFile'" 122 | } 123 | 124 | if ($Plan -or $Apply) { 125 | # Get VNet info from environment (geekzter/azure-devenv) 126 | $env:TF_VAR_peer_network_has_gateway ??= $env:GEEKZTER_AGENT_VIRTUAL_NETWORK_HAS_GATEWAY 127 | $env:TF_VAR_peer_network_id ??= $env:GEEKZTER_AGENT_VIRTUAL_NETWORK_ID 128 | 129 | # Create plan 130 | Invoke "terraform plan $varArgs -out='$planFile'" 131 | } 132 | 133 | if ($Apply) { 134 | Write-Verbose "Converting $planFile into JSON so we can perform some inspection..." 135 | $planJSON = (terraform show -json $planFile) 136 | 137 | if (!$inAutomation) { 138 | if (!$Force) { 139 | # Prompt to continue 140 | $defaultChoice = 0 141 | $choices = @( 142 | [System.Management.Automation.Host.ChoiceDescription]::new("&Continue", "Deploy infrastructure") 143 | [System.Management.Automation.Host.ChoiceDescription]::new("&Exit", "Abort infrastructure deployment") 144 | ) 145 | $decision = $Host.UI.PromptForChoice("Continue", "Do you wish to proceed executing Terraform plan $planFile in workspace $workspace?", $choices, $defaultChoice) 146 | 147 | if ($decision -eq 0) { 148 | Write-Host "$($choices[$decision].HelpMessage)" 149 | } else { 150 | Write-Host "$($PSStyle.Formatting.Warning)$($choices[$decision].HelpMessage)$($PSStyle.Reset)" 151 | exit 152 | } 153 | } 154 | } 155 | 156 | Invoke "terraform apply $forceArgs '$planFile'" 157 | } 158 | 159 | if ($Output) { 160 | Invoke "terraform output" 161 | } 162 | 163 | if (($Apply -or $Output) -and $pipeline) { 164 | # Export Terraform output as Pipeline output variables for subsequent tasks 165 | Set-PipelineVariablesFromTerraform 166 | } 167 | 168 | if ($Destroy) { 169 | Invoke "terraform destroy $varArgs" # $forceArgs" 170 | } 171 | } finally { 172 | Pop-Location 173 | } -------------------------------------------------------------------------------- /scripts/deploy_app.ps1: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env pwsh 2 | param ( 3 | [parameter(Mandatory=$false)][bool]$Deploy=$true, 4 | [parameter(Mandatory=$false)][bool]$Test=$true 5 | ) 6 | 7 | . (Join-Path $PSScriptRoot functions.ps1) 8 | $manifestsDirectory = (Join-Path (Split-Path $PSScriptRoot -Parent) manifests) 9 | 10 | Get-Tools 11 | 12 | try { 13 | ChangeTo-TerraformDirectory 14 | $aksName = (Get-TerraformOutput aks_name) 15 | #$resourceGroup = (Get-TerraformOutput resource_group) 16 | 17 | #az aks get-credentials --name $aksName --resource-group $resourceGroup -a --overwrite-existing 18 | 19 | $null = Prepare-KubeConfig -Workspace $(terraform workspace show) 20 | kubectl config use-context $aksName 2>&1 21 | 22 | # ILB Demo: https://docs.microsoft.com/en-us/azure/aks/internal-lb 23 | if ($Deploy) { 24 | Write-Host "`nDeploying Voting App..." 25 | kubectl apply -f (Join-Path $manifestsDirectory internal-vote.yaml) 2>&1 26 | if ($DebugPreference -ine "SilentlyContinue") { 27 | kubectl get service azure-vote-front 2>&1 28 | } 29 | } 30 | $ilbIPAddress = Get-LoadBalancerIPAddress -KubernetesService azure-vote-front 31 | 32 | # AGIC Demo: https://docs.microsoft.com/en-us/azure/application-gateway/tutorial-ingress-controller-add-on-existing#deploy-a-sample-application-using-agic 33 | $agicFQDN = (Get-TerraformOutput application_gateway_fqdn) 34 | if ($Deploy) { 35 | if ($agicFQDN) { 36 | Write-Host "`nDeploying ASP.NET App..." 37 | kubectl apply -f (Join-Path $manifestsDirectory aspnetapp.yaml) 2>&1 38 | if ($DebugPreference -ine "SilentlyContinue") { 39 | kubectl describe ingress aspnetapp 2>&1 40 | kubectl get ingress 2>&1 41 | } 42 | } else { 43 | Write-Warning "`nNo Application Gateway found. ASP.NET App will not be deployed." 44 | } 45 | } 46 | 47 | # Test after deployment, this should be faster 48 | if ($Test) { 49 | if ($ilbIPAddress) { 50 | $ilbUrl = "http://${ilbIPAddress}/" 51 | Test-App $ilbUrl 52 | } else { 53 | Write-Warning "Internal Load Balancer not found" 54 | } 55 | if ($agicFQDN) { 56 | $agicUrl = "http://${agicFQDN}/" 57 | Test-App $agicUrl 58 | } else { 59 | Write-Warning "Application Gateway Ingress Controller not found" 60 | } 61 | } 62 | } finally { 63 | Pop-Location 64 | } -------------------------------------------------------------------------------- /scripts/functions.ps1: -------------------------------------------------------------------------------- 1 | function AzLogin ( 2 | [parameter(Mandatory=$false)][switch]$DisplayMessages=$false 3 | ) { 4 | # Are we logged into the wrong tenant? 5 | Invoke-Command -ScriptBlock { 6 | $Private:ErrorActionPreference = "Continue" 7 | if ($env:ARM_TENANT_ID) { 8 | $script:loggedInTenantId = $(az account show --query tenantId -o tsv 2>$null) 9 | } 10 | } 11 | if ($loggedInTenantId -and ($loggedInTenantId -ine $env:ARM_TENANT_ID)) { 12 | Write-Warning "Logged into tenant $loggedInTenantId instead of $env:ARM_TENANT_ID (`$env:ARM_TENANT_ID), logging off az session" 13 | az logout -o none 14 | } 15 | 16 | # Are we logged in? 17 | $account = $null 18 | az account show 2>$null | ConvertFrom-Json | Set-Variable account 19 | # Set Azure CLI context 20 | if (-not $account) { 21 | if ($env:CODESPACES -ieq "true") { 22 | $azLoginSwitches = "--use-device-code" 23 | } 24 | if ($env:ARM_TENANT_ID) { 25 | az login -t $env:ARM_TENANT_ID -o none $($azLoginSwitches) 26 | } else { 27 | az login -o none $($azLoginSwitches) 28 | } 29 | } 30 | 31 | if ($DisplayMessages) { 32 | if ($env:ARM_SUBSCRIPTION_ID -or ($(az account list --query "length([])" -o tsv) -eq 1)) { 33 | Write-Host "Using subscription '$(az account show --query "name" -o tsv)'" 34 | } else { 35 | if ($env:TF_IN_AUTOMATION -ine "true") { 36 | # Active subscription may not be the desired one, prompt the user to select one 37 | $subscriptions = (az account list --query "sort_by([].{id:id, name:name},&name)" -o json | ConvertFrom-Json) 38 | $index = 0 39 | $subscriptions | Format-Table -Property @{name="index";expression={$script:index;$script:index+=1}}, id, name 40 | Write-Host "Set `$env:ARM_SUBSCRIPTION_ID to the id of the subscription you want to use to prevent this prompt" -NoNewline 41 | 42 | do { 43 | Write-Host "`nEnter the index # of the subscription you want Terraform to use: " -ForegroundColor Cyan -NoNewline 44 | $occurrence = Read-Host 45 | } while (($occurrence -notmatch "^\d+$") -or ($occurrence -lt 1) -or ($occurrence -gt $subscriptions.Length)) 46 | $env:ARM_SUBSCRIPTION_ID = $subscriptions[$occurrence-1].id 47 | 48 | Write-Host "Using subscription '$($subscriptions[$occurrence-1].name)'" -ForegroundColor Yellow 49 | Start-Sleep -Seconds 1 50 | } else { 51 | Write-Host "Using subscription '$(az account show --query "name" -o tsv)', set `$env:ARM_SUBSCRIPTION_ID if you want to use another one" 52 | } 53 | } 54 | } 55 | 56 | if ($env:ARM_SUBSCRIPTION_ID) { 57 | az account set -s $env:ARM_SUBSCRIPTION_ID -o none 58 | } 59 | 60 | # Populate Terraform azurerm variables where possible 61 | if ($userType -ine "user") { 62 | # Pass on pipeline service principal credentials to Terraform 63 | $env:ARM_CLIENT_ID ??= $env:servicePrincipalId 64 | $env:ARM_CLIENT_SECRET ??= $env:servicePrincipalKey 65 | $env:ARM_TENANT_ID ??= $env:tenantId 66 | # Get from Azure CLI context 67 | $env:ARM_TENANT_ID ??= $(az account show --query tenantId -o tsv) 68 | $env:ARM_SUBSCRIPTION_ID ??= $(az account show --query id -o tsv) 69 | } 70 | # Variables for Terraform azurerm Storage backend 71 | if (!$env:ARM_ACCESS_KEY -and !$env:ARM_SAS_TOKEN) { 72 | if ($env:TF_VAR_backend_storage_account -and $env:TF_VAR_backend_storage_container) { 73 | $env:ARM_SAS_TOKEN=$(az storage container generate-sas -n $env:TF_VAR_backend_storage_container --as-user --auth-mode login --account-name $env:TF_VAR_backend_storage_account --permissions acdlrw --expiry (Get-Date).AddDays(7).ToString("yyyy-MM-dd") -o tsv) 74 | } 75 | } 76 | } 77 | 78 | function ChangeTo-TerraformDirectory() { 79 | Push-Location (Get-TerraformDirectory) 80 | } 81 | 82 | function Get-Tools() { 83 | if (!(Get-Command az -ErrorAction SilentlyContinue)) { 84 | Write-Warning "Azure CLI not found" 85 | exit 86 | } 87 | if (!(Get-Command kubectl -ErrorAction SilentlyContinue)) { 88 | Write-Information "kubectl not found, using Azure CLI to get it..." 89 | az aks install-cli 90 | } 91 | } 92 | 93 | function Get-LoadBalancerIPAddress( 94 | [parameter(Mandatory=$true)][string]$KubernetesService, 95 | [parameter(Mandatory=$false)][int]$MaxTests=600 96 | ) { 97 | $ilb = (kubectl get service azure-vote-front -o=jsonpath='{.status.loadBalancer}' | ConvertFrom-Json) 98 | if (!$ilb) { 99 | Write-Warning "Could not find ILB for service $KubernetesService" 100 | exit 101 | } 102 | $tests = 0 103 | while ((!$ilb.ingress.ip) -and ($tests -le $MaxTests)) { 104 | $tests++ 105 | Start-Sleep 1 106 | $ilb = (kubectl get service azure-vote-front -o=jsonpath='{.status.loadBalancer}' | ConvertFrom-Json) 107 | } 108 | if (!$ilb.ingress.ip) { 109 | Write-Warning "Could not obtain ILB external IP address for service $KubernetesService" 110 | exit 111 | } 112 | 113 | Write-Verbose "Get-LoadBalancerIPAddress: $($ilb.ingress.ip)" 114 | return $ilb.ingress.ip 115 | } 116 | 117 | function Get-TerraformDirectory() { 118 | return (Join-Path (Split-Path -parent -Path $MyInvocation.PSScriptRoot) "terraform") 119 | } 120 | 121 | function Get-TerraformOutput ( 122 | [parameter(Mandatory=$true)][string]$OutputVariable 123 | ) { 124 | Invoke-Command -ScriptBlock { 125 | $Private:ErrorActionPreference = "SilentlyContinue" 126 | Write-Verbose "terraform output ${OutputVariable}: evaluating..." 127 | $result = $(terraform output -raw $OutputVariable 2>$null) 128 | if ($result -match "\[\d+m") { 129 | # Terraform warning, return null for missing output 130 | Write-Verbose "terraform output ${OutputVariable}: `$null (${result})" 131 | return $null 132 | } else { 133 | Write-Verbose "terraform output ${OutputVariable}: ${result}" 134 | return $result 135 | } 136 | } 137 | } 138 | 139 | function Get-TerraformWorkspace () { 140 | Push-Location (Get-TerraformDirectory) 141 | try { 142 | return $(terraform workspace show) 143 | } finally { 144 | Pop-Location 145 | } 146 | } 147 | 148 | function Invoke ( 149 | [string]$cmd 150 | ) { 151 | Write-Host "`n$cmd" -ForegroundColor Green 152 | Invoke-Expression $cmd 153 | $exitCode = $LASTEXITCODE 154 | if ($exitCode -ne 0) { 155 | Write-Warning "'$cmd' exited with status $exitCode" 156 | exit $exitCode 157 | } 158 | } 159 | 160 | function Prepare-KubeConfig ( 161 | [parameter(Mandatory=$true)][string]$Workspace 162 | ) { 163 | $kubeConfig = (Get-TerraformOutput kube_config) 164 | 165 | if ($kubeConfig) { 166 | # Make sure the local file exists, terraform apply may have run on another host 167 | $kubeConfigMoniker = ($Workspace -eq "default") ? "" : $Workspace 168 | $kubeConfigDirectory = (Join-Path (Split-Path $PSScriptRoot -Parent) .kube) 169 | New-Item -ItemType Directory -Force -Path $kubeConfigDirectory | Out-Null 170 | $kubeConfigFile = (Join-Path $kubeConfigDirectory "${kubeConfigMoniker}config") 171 | $kubeConfig | Set-Content -Path $kubeConfigFile 172 | $env:KUBECONFIG = $kubeConfigFile 173 | Write-Host "Prepared ${kubeConfigFile}" 174 | return $kubeConfigFile 175 | } 176 | } 177 | 178 | function Set-PipelineVariablesFromTerraform () { 179 | $json = terraform output -json | ConvertFrom-Json -AsHashtable 180 | foreach ($outputVariable in $json.keys) { 181 | $value = $json[$outputVariable].value 182 | if ($value) { 183 | # Write variable output in the format a Pipeline can understand 184 | # https://github.com/Microsoft/azure-pipelines-agent/blob/master/docs/preview/outputvariable.md 185 | Write-Host "##vso[task.setvariable variable=${outputVariable};isOutput=true]${value}" 186 | } 187 | } 188 | } 189 | 190 | function Start-Agents () { 191 | ChangeTo-TerraformDirectory 192 | 193 | $nodeResourceGroup = (Get-TerraformOutput node_resource_group) 194 | if ($nodeResourceGroup) { 195 | $location = $(az group show -g $nodeResourceGroup --query location -o tsv) 196 | $vmssNames = $(az resource list -l $location -g $nodeResourceGroup --resource-type "Microsoft.Compute/virtualMachineScaleSets" --query "[].name" -o tsv) 197 | foreach ($vmssName in $vmssNames) { 198 | Write-Host "Starting nodes in scale set ${vmssName}..." 199 | az vmss start -n $vmssName -g $nodeResourceGroup 200 | } 201 | } else { 202 | Write-Host "No terraform output for node_resource_group" 203 | } 204 | 205 | Pop-Location 206 | } 207 | 208 | function Test-App ( 209 | [parameter(Mandatory=$true)][string]$AppUrl, 210 | [parameter(Mandatory=$false)][int]$MaxTests=600 211 | ) { 212 | $test = 0 213 | Write-Host "Testing $AppUrl (max $MaxTests times)" -NoNewLine 214 | while (!$responseOK -and ($test -lt $MaxTests)) { 215 | try { 216 | $test++ 217 | Write-Host "." -NoNewLine 218 | $homePageResponse = Invoke-WebRequest -UseBasicParsing -Uri $AppUrl 219 | if ($homePageResponse.StatusCode -lt 400) { 220 | $responseOK = $true 221 | } else { 222 | $responseOK = $false 223 | } 224 | } 225 | catch { 226 | $responseOK = $false 227 | if ($test -ge $MaxTests) { 228 | throw 229 | } else { 230 | Start-Sleep -Milliseconds 1000 231 | } 232 | } 233 | } 234 | Write-Host "✓" # Force NewLine 235 | Write-Information "Request to $AppUrl completed with HTTP Status Code $($homePageResponse.StatusCode)" 236 | } -------------------------------------------------------------------------------- /scripts/kube_config.ps1: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env pwsh 2 | . (Join-Path $PSScriptRoot functions.ps1) 3 | 4 | try { 5 | ChangeTo-TerraformDirectory 6 | if (Prepare-KubeConfig -Workspace $(terraform workspace show)) { 7 | kubectl config use-context (Get-TerraformOutput aks_name) 8 | kubectl config view 9 | kubectl cluster-info 10 | kubectl get nodes 11 | } else { 12 | Write-Warning "Terraform did not provision K8s yet" >2&1 13 | } 14 | } finally { 15 | Pop-Location 16 | } 17 | -------------------------------------------------------------------------------- /scripts/kube_tunnel.ps1: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env pwsh 2 | . (Join-Path $PSScriptRoot functions.ps1) 3 | 4 | try { 5 | ChangeTo-TerraformDirectory 6 | $null = Prepare-KubeConfig -Workspace $(terraform workspace show) 7 | 8 | kubectl config use-context (Get-TerraformOutput aks_name) 9 | 10 | # Open SSH tunnel for local portal on 127.0.0.1:8001 11 | Get-Job | Where-Object {$_.Command.Contains("kubectl proxy")} | Stop-Job 12 | kubectl proxy & 13 | #az aks browse --resource-group (Get-TerraformOutput resource_group) --name (Get-TerraformOutput aks_name) 14 | 15 | # Wait for agent nodes to have started 16 | Start-Agents 17 | 18 | Write-Host "Open Kubernetes Dashboard at http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy" 19 | } finally { 20 | Pop-Location 21 | } 22 | 23 | 24 | -------------------------------------------------------------------------------- /scripts/wait_for_app_gw.ps1: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env pwsh 2 | param ( 3 | [parameter(Mandatory=$true)][string]$AksName, 4 | [parameter(Mandatory=$false)][string]$ApplicationGatewayName="applicationgateway", 5 | [parameter(Mandatory=$true)][string]$ResourceGroupName 6 | ) 7 | . (Join-Path $PSScriptRoot functions.ps1) 8 | 9 | function Wait-ApplicationGateway( 10 | [parameter(Mandatory=$true)][string]$AksName, 11 | [parameter(Mandatory=$true)][string]$ApplicationGatewayName, 12 | [parameter(Mandatory=$true)][string]$ResourceGroupName 13 | ) { 14 | $intervalSeconds = 10 15 | $timeoutSeconds = 300 16 | 17 | Write-Host "Waiting for AKS $AksName to finish provisioning..." 18 | az aks wait -g $ResourceGroupName -n $AksName --created --interval $intervalSeconds --timeout $timeoutSeconds 19 | az aks wait -g $ResourceGroupName -n $AksName --updated --interval $intervalSeconds --timeout $timeoutSeconds 20 | 21 | $nodeResourceGroupName = $(az aks show -n $AksName -g $ResourceGroupName --query nodeResourceGroup -o tsv) 22 | 23 | Write-Host "Waiting for Application Gateway ${ApplicationGatewayName} to finish updating..." 24 | az network application-gateway wait -g $nodeResourceGroupName -n $ApplicationGatewayName --created --updated --interval $intervalSeconds --timeout $timeoutSeconds 25 | } 26 | 27 | Wait-ApplicationGateway -AksName $AksName -ResourceGroupName $ResourceGroupName -ApplicationGatewayName $ApplicationGatewayName 28 | -------------------------------------------------------------------------------- /terraform/.terraform-version: -------------------------------------------------------------------------------- 1 | 1.3.6 -------------------------------------------------------------------------------- /terraform/.terraform.lock.hcl: -------------------------------------------------------------------------------- 1 | # This file is maintained automatically by "terraform init". 2 | # Manual edits may be lost in future updates. 3 | 4 | provider "registry.terraform.io/hashicorp/azuread" { 5 | version = "2.45.0" 6 | constraints = "~> 2.12" 7 | hashes = [ 8 | "h1:/uvs5iEiakqbl4PEGzNob8Rqbnw7YaeXfjnTMfJJK2w=", 9 | "zh:08d80f6ab1d8bcb02976a5fde0b108fc008093a7a6f1b5d083d5cf1e01dc0b86", 10 | "zh:1c3e89cf19118fc07d7b04257251fc9897e722c16e0a0df7b07fcd261f8c12e7", 11 | "zh:2571a343d8e1af699c1e1d815a5654ad98665427c6f3912b41557ff775a73c75", 12 | "zh:2a20afb34eaca73c2206250abbb40997498a1a9aca34e8d6524370fbf6278f82", 13 | "zh:3044cb26396399f0082ee06a8f788c6260ab12e752f64c0d7921e7d4e39c14b8", 14 | "zh:311fcfdee359129748caaae5b204113a3e3b7e1fc00704c21300461114eb2075", 15 | "zh:a3af689267be6f7ebfd9726aa2bfd4faabb9efbcab9da24949ae5fec87d18d23", 16 | "zh:adbe97fd77ef94c5f1fc8b10e7dc87048035dc39ff7ea81042350fc311034850", 17 | "zh:c6ce4f348596515d1748c047b51462cb40802b57316b13f8fb4760ef1c5e775c", 18 | "zh:d0b8854b5097ffd6f6d6a14544f74745b23ba949472351e585c57402f5ca28ae", 19 | "zh:ebdf01510f10e468aaa6c8f5fd9fc50b7232acbbe5fcbc50e8bea2de5d964d3e", 20 | "zh:f3a55da36dcfd0bf34f785ce3c47dbe8538408c1aaf729604a6ef841fc8ce6f4", 21 | ] 22 | } 23 | 24 | provider "registry.terraform.io/hashicorp/azurerm" { 25 | version = "3.79.0" 26 | constraints = "~> 3.24" 27 | hashes = [ 28 | "h1:GhVT9cr1ro88hTU7DI6NybuVQ1SanbQLvBUzBxp9UZs=", 29 | "zh:0cd62eff55944be5bee31b376b410f07232227490b902af8f4785021edeb707f", 30 | "zh:168128566331d18b89565205ed78a6a64c3f55a2555956f7e4c15773de56905c", 31 | "zh:63068b268ae4080fe3e33f75c174e83ed2355b5812ec62a29e5f7c7e71399ab9", 32 | "zh:6e88c32eafc7c01d9564bca18f2e47a7f54f2fec1b64700f3f7a6f927757a034", 33 | "zh:8f1f40fc00bc22eb5ea4fa6a4b4815d2a44a2a7ba086cadf2a37366f8fa65c88", 34 | "zh:96e6309019a0367bb77bec52cd0bcbd049ac943e7a28ed0b7635b8e9ed5776d3", 35 | "zh:ba4840eb4da0df74adfe9bf59ff7e63d4a38c1ae0028c93c06285d766fc06f0f", 36 | "zh:dd49b4cc241251077dbdbae5137f03f1d66873408b0b43d3ff5f98fa254ffca4", 37 | "zh:e0a99adb8b1c1b951e2b19c677fb4c1ba78350b829f0ff73aa141ec1bc1ecd8b", 38 | "zh:ee523758d5b17fd04fa869b7f6ad92f1b321eaaa9e1508609dcca1577dee3c44", 39 | "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c", 40 | "zh:fbc79c9a3cc63e6f3f154cdca23b8ccb6de86495c0b9ae605bea50072c514032", 41 | ] 42 | } 43 | 44 | provider "registry.terraform.io/hashicorp/external" { 45 | version = "2.3.1" 46 | constraints = "~> 2.1" 47 | hashes = [ 48 | "h1:gznGscVJ0USxy4CdihpjRKPsKvyGr/zqPvBoFLJTQDc=", 49 | "zh:001e2886dc81fc98cf17cf34c0d53cb2dae1e869464792576e11b0f34ee92f54", 50 | "zh:2eeac58dd75b1abdf91945ac4284c9ccb2bfb17fa9bdb5f5d408148ff553b3ee", 51 | "zh:2fc39079ba61411a737df2908942e6970cb67ed2f4fb19090cd44ce2082903dd", 52 | "zh:472a71c624952cff7aa98a7b967f6c7bb53153dbd2b8f356ceb286e6743bb4e2", 53 | "zh:4cff06d31272aac8bc35e9b7faec42cf4554cbcbae1092eaab6ab7f643c215d9", 54 | "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", 55 | "zh:7ed16ccd2049fa089616b98c0bd57219f407958f318f3c697843e2397ddf70df", 56 | "zh:842696362c92bf2645eb85c739410fd51376be6c488733efae44f4ce688da50e", 57 | "zh:8985129f2eccfd7f1841ce06f3bf2bbede6352ec9e9f926fbaa6b1a05313b326", 58 | "zh:a5f0602d8ec991a5411ef42f872aa90f6347e93886ce67905c53cfea37278e05", 59 | "zh:bf4ab82cbe5256dcef16949973bf6aa1a98c2c73a98d6a44ee7bc40809d002b8", 60 | "zh:e70770be62aa70198fa899526d671643ff99eecf265bf1a50e798fc3480bd417", 61 | ] 62 | } 63 | 64 | provider "registry.terraform.io/hashicorp/helm" { 65 | version = "2.11.0" 66 | constraints = "~> 2.0" 67 | hashes = [ 68 | "h1:AOp9vXIM4uT1c/PVwsWTPiLVGlO2SSYrfiirV5rjCMQ=", 69 | "zh:013857c88f3e19a4b162344e21dc51891c4ac8b600da8391f7fb2b6d234961e1", 70 | "zh:044fffa233a93cdcf8384afbe9e1ab6c9d0b5b176cbae56ff465eb9611302975", 71 | "zh:208b7cdd4fa3a1b25ae817dc00a9198ef98be0ddc3a577b5b72bc0f006afb997", 72 | "zh:3e8b33f56cfe387277572a92037a1ca1cbe4e3aa6b5c19a8c2431193b07f7865", 73 | "zh:7dd663d5619bd71676899b05b19d36f585189fdabc6b0b03c23579524a8fd9bf", 74 | "zh:ae5329cb3e5bf0b86b02e823aac3ef3bd0d4b1618ff013cd0076dca0be8322e4", 75 | "zh:ba6201695b55d51bedacdb017cb8d03d7a8ada51d0168ac44fef3fa791a85ab4", 76 | "zh:c61285c8b1ba10f50cf94c9dcf98f2f3b720f14906a18be71b9b422279b5d806", 77 | "zh:d522d388246f38b9f329c511ec579b516d212670b954f9dab64efb27e51862af", 78 | "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c", 79 | "zh:f92546e26b670da61437ae2cbd038427c9374ce5f7a78df52193397da90bd997", 80 | "zh:f9ad1407e5c0d5e3474094491025bf100828e8c1a01acdf9591d7dd1eb59f961", 81 | ] 82 | } 83 | 84 | provider "registry.terraform.io/hashicorp/http" { 85 | version = "3.4.0" 86 | constraints = "~> 3.1" 87 | hashes = [ 88 | "h1:m0d6+9xK/9TJSE9Z6nM4IwHXZgod4/jkdsf7CZSpUvo=", 89 | "zh:56712497a87bc4e91bbaf1a5a2be4b3f9cfa2384baeb20fc9fad0aff8f063914", 90 | "zh:6661355e1090ebacab16a40ede35b029caffc279d67da73a000b6eecf0b58eba", 91 | "zh:67b92d343e808b92d7e6c3bbcb9b9d5475fecfed0836963f7feb9d9908bd4c4f", 92 | "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", 93 | "zh:86ebb9be9b685c96dbb5c024b55d87526d57a4b127796d6046344f8294d3f28e", 94 | "zh:902be7cfca4308cba3e1e7ba6fc292629dfd150eb9a9f054a854fa1532b0ceba", 95 | "zh:9ba26e0215cd53b21fe26a0a98c007de1348b7d13a75ae3cfaf7729e0f2c50bb", 96 | "zh:a195c941e1f1526147134c257ff549bea4c89c953685acd3d48d9de7a38f39dc", 97 | "zh:a7967b3d2a8c3e7e1dc9ae381ca753268f9fce756466fe2fc9e414ca2d85a92e", 98 | "zh:bde56542e9a093434d96bea21c341285737c6d38fea2f05e12ba7b333f3e9c05", 99 | "zh:c0306f76903024c497fd01f9fd9bace5854c263e87a97bc2e89dcc96d35ca3cc", 100 | "zh:f9335a6c336171e85f8e3e99c3d31758811a19aeb21fa8c9013d427e155ae2a9", 101 | ] 102 | } 103 | 104 | provider "registry.terraform.io/hashicorp/kubernetes" { 105 | version = "2.23.0" 106 | constraints = "~> 2.0" 107 | hashes = [ 108 | "h1:arTzD0XG/DswGCAx9JEttkSKe9RyyFW9W7UWcXF13dU=", 109 | "zh:10488a12525ed674359585f83e3ee5e74818b5c98e033798351678b21b2f7d89", 110 | "zh:1102ba5ca1a595f880e67102bbf999cc8b60203272a078a5b1e896d173f3f34b", 111 | "zh:1347cf958ed3f3f80b3c7b3e23ddda3d6c6573a81847a8ee92b7df231c238bf6", 112 | "zh:2cb18e9f5156bc1b1ee6bc580a709f7c2737d142722948f4a6c3c8efe757fa8d", 113 | "zh:5506aa6f28dcca2a265ccf8e34478b5ec2cb43b867fe6d93b0158f01590fdadd", 114 | "zh:6217a20686b631b1dcb448ee4bc795747ebc61b56fbe97a1ad51f375ebb0d996", 115 | "zh:8accf916c00579c22806cb771e8909b349ffb7eb29d9c5468d0a3f3166c7a84a", 116 | "zh:9379b0b54a0fa030b19c7b9356708ec8489e194c3b5e978df2d31368563308e5", 117 | "zh:aa99c580890691036c2931841e88e7ee80d59ae52289c8c2c28ea0ac23e31520", 118 | "zh:c57376d169875990ac68664d227fb69cd0037b92d0eba6921d757c3fd1879080", 119 | "zh:e6068e3f94f6943b5586557b73f109debe19d1a75ca9273a681d22d1ce066579", 120 | "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c", 121 | ] 122 | } 123 | 124 | provider "registry.terraform.io/hashicorp/local" { 125 | version = "2.4.0" 126 | constraints = "~> 2.1" 127 | hashes = [ 128 | "h1:ZUEYUmm2t4vxwzxy1BvN1wL6SDWrDxfH7pxtzX8c6d0=", 129 | "zh:53604cd29cb92538668fe09565c739358dc53ca56f9f11312b9d7de81e48fab9", 130 | "zh:66a46e9c508716a1c98efbf793092f03d50049fa4a83cd6b2251e9a06aca2acf", 131 | "zh:70a6f6a852dd83768d0778ce9817d81d4b3f073fab8fa570bff92dcb0824f732", 132 | "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", 133 | "zh:82a803f2f484c8b766e2e9c32343e9c89b91997b9f8d2697f9f3837f62926b35", 134 | "zh:9708a4e40d6cc4b8afd1352e5186e6e1502f6ae599867c120967aebe9d90ed04", 135 | "zh:973f65ce0d67c585f4ec250c1e634c9b22d9c4288b484ee2a871d7fa1e317406", 136 | "zh:c8fa0f98f9316e4cfef082aa9b785ba16e36ff754d6aba8b456dab9500e671c6", 137 | "zh:cfa5342a5f5188b20db246c73ac823918c189468e1382cb3c48a9c0c08fc5bf7", 138 | "zh:e0e2b477c7e899c63b06b38cd8684a893d834d6d0b5e9b033cedc06dd7ffe9e2", 139 | "zh:f62d7d05ea1ee566f732505200ab38d94315a4add27947a60afa29860822d3fc", 140 | "zh:fa7ce69dde358e172bd719014ad637634bbdabc49363104f4fca759b4b73f2ce", 141 | ] 142 | } 143 | 144 | provider "registry.terraform.io/hashicorp/null" { 145 | version = "3.2.1" 146 | constraints = "~> 3.1" 147 | hashes = [ 148 | "h1:ydA0/SNRVB1o95btfshvYsmxA+jZFRZcvKzZSB+4S1M=", 149 | "zh:58ed64389620cc7b82f01332e27723856422820cfd302e304b5f6c3436fb9840", 150 | "zh:62a5cc82c3b2ddef7ef3a6f2fedb7b9b3deff4ab7b414938b08e51d6e8be87cb", 151 | "zh:63cff4de03af983175a7e37e52d4bd89d990be256b16b5c7f919aff5ad485aa5", 152 | "zh:74cb22c6700e48486b7cabefa10b33b801dfcab56f1a6ac9b6624531f3d36ea3", 153 | "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", 154 | "zh:79e553aff77f1cfa9012a2218b8238dd672ea5e1b2924775ac9ac24d2a75c238", 155 | "zh:a1e06ddda0b5ac48f7e7c7d59e1ab5a4073bbcf876c73c0299e4610ed53859dc", 156 | "zh:c37a97090f1a82222925d45d84483b2aa702ef7ab66532af6cbcfb567818b970", 157 | "zh:e4453fbebf90c53ca3323a92e7ca0f9961427d2f0ce0d2b65523cc04d5d999c2", 158 | "zh:e80a746921946d8b6761e77305b752ad188da60688cfd2059322875d363be5f5", 159 | "zh:fbdb892d9822ed0e4cb60f2fedbdbb556e4da0d88d3b942ae963ed6ff091e48f", 160 | "zh:fca01a623d90d0cad0843102f9b8b9fe0d3ff8244593bd817f126582b52dd694", 161 | ] 162 | } 163 | 164 | provider "registry.terraform.io/hashicorp/random" { 165 | version = "3.5.1" 166 | constraints = "~> 3.1" 167 | hashes = [ 168 | "h1:IL9mSatmwov+e0+++YX2V6uel+dV6bn+fC/cnGDK3Ck=", 169 | "zh:04e3fbd610cb52c1017d282531364b9c53ef72b6bc533acb2a90671957324a64", 170 | "zh:119197103301ebaf7efb91df8f0b6e0dd31e6ff943d231af35ee1831c599188d", 171 | "zh:4d2b219d09abf3b1bb4df93d399ed156cadd61f44ad3baf5cf2954df2fba0831", 172 | "zh:6130bdde527587bbe2dcaa7150363e96dbc5250ea20154176d82bc69df5d4ce3", 173 | "zh:6cc326cd4000f724d3086ee05587e7710f032f94fc9af35e96a386a1c6f2214f", 174 | "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", 175 | "zh:b6d88e1d28cf2dfa24e9fdcc3efc77adcdc1c3c3b5c7ce503a423efbdd6de57b", 176 | "zh:ba74c592622ecbcef9dc2a4d81ed321c4e44cddf7da799faa324da9bf52a22b2", 177 | "zh:c7c5cde98fe4ef1143bd1b3ec5dc04baf0d4cc3ca2c5c7d40d17c0e9b2076865", 178 | "zh:dac4bad52c940cd0dfc27893507c1e92393846b024c5a9db159a93c534a3da03", 179 | "zh:de8febe2a2acd9ac454b844a4106ed295ae9520ef54dc8ed2faf29f12716b602", 180 | "zh:eab0d0495e7e711cca367f7d4df6e322e6c562fc52151ec931176115b83ed014", 181 | ] 182 | } 183 | 184 | provider "registry.terraform.io/hashicorp/time" { 185 | version = "0.9.1" 186 | constraints = "~> 0.7" 187 | hashes = [ 188 | "h1:VxyoYYOCaJGDmLz4TruZQTSfQhvwEcMxvcKclWdnpbs=", 189 | "zh:00a1476ecf18c735cc08e27bfa835c33f8ac8fa6fa746b01cd3bcbad8ca84f7f", 190 | "zh:3007f8fc4a4f8614c43e8ef1d4b0c773a5de1dcac50e701d8abc9fdc8fcb6bf5", 191 | "zh:5f79d0730fdec8cb148b277de3f00485eff3e9cf1ff47fb715b1c969e5bbd9d4", 192 | "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", 193 | "zh:8c8094689a2bed4bb597d24a418bbbf846e15507f08be447d0a5acea67c2265a", 194 | "zh:a6d9206e95d5681229429b406bc7a9ba4b2d9b67470bda7df88fa161508ace57", 195 | "zh:aa299ec058f23ebe68976c7581017de50da6204883950de228ed9246f309e7f1", 196 | "zh:b129f00f45fba1991db0aa954a6ba48d90f64a738629119bfb8e9a844b66e80b", 197 | "zh:ef6cecf5f50cda971c1b215847938ced4cb4a30a18095509c068643b14030b00", 198 | "zh:f1f46a4f6c65886d2dd27b66d92632232adc64f92145bf8403fe64d5ffa5caea", 199 | "zh:f79d6155cda7d559c60d74883a24879a01c4d5f6fd7e8d1e3250f3cd215fb904", 200 | "zh:fd59fa73074805c3575f08cd627eef7acda14ab6dac2c135a66e7a38d262201c", 201 | ] 202 | } 203 | -------------------------------------------------------------------------------- /terraform/aks.auto.tfvars.sample: -------------------------------------------------------------------------------- 1 | # deploy_aks = true 2 | # deploy_bastion = true 3 | # configure_kubernetes = false 4 | 5 | # Network configuration automatically set in geekzter/azure-devenv 6 | # peer_network_id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myRG/providers/Microsoft.Network/virtualNetworks/myVNet" 7 | # peer_network_has_gateway = true -------------------------------------------------------------------------------- /terraform/backend.tf.sample: -------------------------------------------------------------------------------- 1 | terraform { 2 | backend "azurerm" { 3 | resource_group_name = "Automation" 4 | storage_account_name = "terraformstate" 5 | container_name = "aks" 6 | key = "terraform.tfstate" 7 | } 8 | } -------------------------------------------------------------------------------- /terraform/main.tf: -------------------------------------------------------------------------------- 1 | resource random_string password { 2 | length = 12 3 | upper = true 4 | lower = true 5 | numeric = true 6 | special = true 7 | # override_special = "!@#$%&*()-_=+[]{}<>:?" # default 8 | # Avoid characters that may cause shell scripts to break 9 | override_special = "." 10 | } 11 | 12 | resource random_string suffix { 13 | length = 4 14 | upper = false 15 | lower = true 16 | numeric = false 17 | special = false 18 | } 19 | 20 | locals { 21 | aks_name = "${var.resource_prefix}-${terraform.workspace}-${local.suffix}" 22 | owner = var.application_owner != "" ? var.application_owner : data.azuread_client_config.current.object_id 23 | kube_config_relative_path = var.kube_config_path != "" ? var.kube_config_path : "../.kube/${local.workspace_moniker}config" 24 | kube_config_absolute_path = var.kube_config_path != "" ? var.kube_config_path : "${path.root}/../.kube/${local.workspace_moniker}config" 25 | 26 | # Making sure all character classes are represented, as random does not guarantee that 27 | password = ".Az9${random_string.password.result}" 28 | suffix = var.resource_suffix != "" ? lower(var.resource_suffix) : random_string.suffix.result 29 | environment = var.resource_environment != "" ? lower(var.resource_environment) : terraform.workspace 30 | workspace_moniker = terraform.workspace == "default" ? "" : terraform.workspace 31 | } 32 | 33 | data azuread_client_config current {} 34 | data azurerm_client_config current {} 35 | data azurerm_subscription primary {} 36 | 37 | data http localpublicip { 38 | # Get public IP address of the machine running this terraform template 39 | url = "https://ipinfo.io/ip" 40 | } 41 | 42 | resource azurerm_resource_group rg { 43 | name = "${lower(var.resource_prefix)}-${lower(local.environment)}-${lower(local.suffix)}" 44 | location = var.location 45 | 46 | tags = { 47 | application = var.application_name 48 | environment = local.environment 49 | github-repo = "https://github.com/geekzter/azure-aks" 50 | owner = local.owner 51 | provisioner = "terraform" 52 | provisioner-client-id = data.azurerm_client_config.current.client_id 53 | provisioner-object-id = data.azuread_client_config.current.object_id 54 | repository = "azure-aks" 55 | runid = var.run_id 56 | shutdown = "true" 57 | suffix = local.suffix 58 | workspace = terraform.workspace 59 | } 60 | } 61 | 62 | resource azurerm_container_registry acr { 63 | name = "${lower(replace(var.resource_prefix,"/\\W/",""))}${terraform.workspace}reg${local.suffix}" 64 | resource_group_name = azurerm_resource_group.rg.name 65 | location = var.location 66 | sku = "Premium" 67 | admin_enabled = true 68 | 69 | tags = azurerm_resource_group.rg.tags 70 | } 71 | resource azurerm_monitor_diagnostic_setting acr { 72 | name = "${azurerm_container_registry.acr.name}-logs" 73 | target_resource_id = azurerm_container_registry.acr.id 74 | log_analytics_workspace_id = azurerm_log_analytics_workspace.log_analytics.id 75 | 76 | enabled_log { 77 | category = "ContainerRegistryRepositoryEvents" 78 | } 79 | enabled_log { 80 | category = "ContainerRegistryLoginEvents" 81 | } 82 | 83 | metric { 84 | category = "AllMetrics" 85 | } 86 | } 87 | 88 | resource azurerm_log_analytics_workspace log_analytics { 89 | name = "${azurerm_resource_group.rg.name}-logs" 90 | # Doesn't deploy in all regions e.g. South India 91 | location = var.workspace_location 92 | resource_group_name = azurerm_resource_group.rg.name 93 | sku = "Standalone" 94 | retention_in_days = 90 95 | 96 | tags = azurerm_resource_group.rg.tags 97 | } 98 | resource azurerm_log_analytics_saved_search query { 99 | name = each.value 100 | log_analytics_workspace_id = azurerm_log_analytics_workspace.log_analytics.id 101 | 102 | category = "Favorites" 103 | display_name = replace(replace(each.value,"-"," "),".kql","") 104 | query = file("${path.root}/../kusto/${each.value}") 105 | 106 | for_each = toset([ 107 | "denied-outbound-http-traffic.kql", 108 | "denied-outbound-traffic.kql", 109 | ]) 110 | } -------------------------------------------------------------------------------- /terraform/modules.tf: -------------------------------------------------------------------------------- 1 | # Provision base network infrastructure 2 | module network { 3 | source = "./modules/network" 4 | resource_group_name = azurerm_resource_group.rg.name 5 | address_space = var.address_space 6 | location = var.location 7 | log_analytics_workspace_id = azurerm_log_analytics_workspace.log_analytics.id 8 | nsg_reassign_wait_minutes = var.deploy_application_gateway ? var.nsg_reassign_wait_minutes : 0 9 | peer_network_has_gateway = var.peer_network_has_gateway 10 | peer_network_id = var.peer_network_id 11 | tags = azurerm_resource_group.rg.tags 12 | } 13 | 14 | module bastion { 15 | source = "./modules/bastion" 16 | resource_group_name = azurerm_resource_group.rg.name 17 | location = var.location 18 | log_analytics_workspace_id = azurerm_log_analytics_workspace.log_analytics.id 19 | subnet_id = module.network.bastion_subnet_id 20 | tags = azurerm_resource_group.rg.tags 21 | 22 | count = var.deploy_bastion ? 1 : 0 23 | depends_on = [module.network] 24 | } 25 | 26 | # Provision base Kubernetes infrastructure provided by Azure 27 | module aks { 28 | source = "./modules/aks" 29 | name = local.aks_name 30 | 31 | admin_username = "aksadmin" 32 | application_gateway_subnet_id= module.network.application_gateway_subnet_id 33 | client_object_id = data.azuread_client_config.current.object_id 34 | configure_access_control = var.configure_access_control 35 | deploy_application_gateway = var.deploy_application_gateway 36 | dns_prefix = var.resource_prefix 37 | dns_host_suffix = var.dns_host_suffix 38 | location = var.location 39 | kube_config_path = local.kube_config_absolute_path 40 | kubernetes_version = var.kubernetes_version 41 | log_analytics_workspace_id = azurerm_log_analytics_workspace.log_analytics.id 42 | node_size = var.node_size 43 | node_subnet_id = module.network.nodes_subnet_id 44 | private_cluster_enabled = var.private_cluster_enabled 45 | resource_group_id = azurerm_resource_group.rg.id 46 | ssh_public_key_file = var.ssh_public_key_file 47 | tags = azurerm_resource_group.rg.tags 48 | 49 | count = var.deploy_aks ? 1 : 0 50 | depends_on = [module.network] 51 | } 52 | 53 | # Provision AKS network infrastructure (allowing dependencies on AKS) 54 | module aks_network { 55 | source = "./modules/aks-network" 56 | resource_group_name = azurerm_resource_group.rg.name 57 | 58 | admin_ip_group_id = module.network.admin_ip_group_id 59 | aks_id = module.aks.0.aks_id 60 | container_registry_id = azurerm_container_registry.acr.id 61 | firewall_id = module.network.firewall_id 62 | location = var.location 63 | nodes_ip_group_id = module.network.nodes_ip_group_id 64 | nodes_subnet_id = module.network.nodes_subnet_id 65 | paas_subnet_id = module.network.paas_subnet_id 66 | peer_network_id = var.peer_network_id 67 | private_cluster_enabled = var.private_cluster_enabled 68 | resource_group_id = azurerm_resource_group.rg.id 69 | tags = azurerm_resource_group.rg.tags 70 | virtual_network_id = module.network.virtual_network_id 71 | 72 | count = var.deploy_aks ? 1 : 0 73 | depends_on = [module.aks] 74 | } 75 | 76 | # Confugure Kubernetes 77 | module k8s { 78 | source = "./modules/kubernetes" 79 | 80 | count = var.deploy_aks && var.configure_kubernetes && var.peer_network_id != "" ? 1 : 0 81 | depends_on = [module.aks,module.aks_network] 82 | } -------------------------------------------------------------------------------- /terraform/modules/aks-network/egress.tf: -------------------------------------------------------------------------------- 1 | 2 | # https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#azure-global-required-network-rules 3 | # Rules that have a dependency on AKS being created first 4 | resource azurerm_firewall_network_rule_collection iag_net_outbound_rules { 5 | name = "${data.azurerm_firewall.gateway.name}-aks-network-rules" 6 | azure_firewall_name = data.azurerm_firewall.gateway.name 7 | resource_group_name = data.azurerm_firewall.gateway.resource_group_name 8 | priority = 1002 9 | action = "Allow" 10 | 11 | rule { 12 | name = "AllowOutboundAKSAPIServer1" 13 | source_ip_groups = [var.nodes_ip_group_id] 14 | destination_ports = ["1194"] 15 | destination_ip_groups = [azurerm_ip_group.api_server.id] 16 | # destination_addresses = [ 17 | # "AzureCloud.${data.azurerm_firewall.gateway.location}", 18 | # ] 19 | protocols = ["UDP"] 20 | } 21 | 22 | rule { 23 | name = "AllowOutboundAKSAPIServer2" 24 | source_ip_groups = [var.nodes_ip_group_id] 25 | destination_ports = ["9000"] 26 | destination_ip_groups = [azurerm_ip_group.api_server.id] 27 | # destination_addresses = [ 28 | # "AzureCloud.${data.azurerm_firewall.gateway.location}", 29 | # ] 30 | protocols = ["TCP"] 31 | } 32 | 33 | rule { 34 | name = "AllowOutboundAKSAPIServerHTTPS" 35 | source_ip_groups = [var.nodes_ip_group_id] 36 | destination_ports = ["443"] 37 | destination_ip_groups = [azurerm_ip_group.api_server.id] 38 | # destination_addresses = [ 39 | # "AzureCloud.${data.azurerm_firewall.gateway.location}", 40 | # ] 41 | protocols = ["TCP"] 42 | } 43 | 44 | rule { 45 | name = "AllowOutboundAKSAzureMonitor" 46 | source_ip_groups = [var.nodes_ip_group_id] 47 | destination_ports = ["443"] 48 | destination_ip_groups = [azurerm_ip_group.api_server.id] 49 | destination_addresses = [ 50 | "AzureMonitor", 51 | ] 52 | protocols = ["TCP"] 53 | } 54 | 55 | rule { 56 | name = "AllowOutboundAKSAzureDevSpaces" 57 | source_ip_groups = [var.nodes_ip_group_id] 58 | destination_ports = ["443"] 59 | destination_ip_groups = [azurerm_ip_group.api_server.id] 60 | destination_addresses = [ 61 | "AzureDevSpaces", 62 | ] 63 | protocols = ["TCP"] 64 | } 65 | } 66 | 67 | # https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#azure-global-required-fqdn--application-rules 68 | resource azurerm_firewall_application_rule_collection aks_app_rules { 69 | name = "${data.azurerm_firewall.gateway.name}-aks-app-rules" 70 | azure_firewall_name = data.azurerm_firewall.gateway.name 71 | resource_group_name = data.azurerm_firewall.gateway.resource_group_name 72 | priority = 2002 73 | action = "Allow" 74 | 75 | # https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#azure-global-required-fqdn--application-rules 76 | rule { 77 | name = "Allow outbound traffic" 78 | 79 | source_ip_groups = [var.nodes_ip_group_id] 80 | target_fqdns = [ 81 | "*.hcp.${data.azurerm_kubernetes_cluster.aks.location}.azmk8s.io", 82 | ] 83 | 84 | protocol { 85 | port = "443" 86 | type = "Https" 87 | } 88 | } 89 | } -------------------------------------------------------------------------------- /terraform/modules/aks-network/ingress.tf: -------------------------------------------------------------------------------- 1 | # Azure Internal Load Balancer provisioned by application (manifest) 2 | # resource kubernetes_service internal_load_balancer { 3 | # metadata { 4 | # annotations = { 5 | # "service.beta.kubernetes.io/azure-load-balancer-internal" = "true" 6 | # } 7 | # name = "azure-all-front" 8 | # } 9 | # spec { 10 | # selector = { 11 | # app = "azure-all-front" 12 | # } 13 | # session_affinity = "ClientIP" 14 | # port { 15 | # port = 80 16 | # } 17 | 18 | # type = "LoadBalancer" 19 | # } 20 | 21 | # depends_on = [ 22 | # azurerm_firewall_network_rule_collection.iag_net_outbound_rules, 23 | # azurerm_firewall_application_rule_collection.aks_app_rules, 24 | # azurerm_private_dns_zone_virtual_network_link.api_server_domain, 25 | # ] 26 | 27 | # count = var.peer_network_id != "" ? 1 : 0 28 | # } 29 | 30 | # Application Ingress controller is created as AKS add on -------------------------------------------------------------------------------- /terraform/modules/aks-network/main.tf: -------------------------------------------------------------------------------- 1 | data azurerm_kubernetes_cluster aks { 2 | name = element(split("/",var.aks_id),length(split("/",var.aks_id))-1) 3 | resource_group_name = element(split("/",var.aks_id),length(split("/",var.aks_id))-5) 4 | } 5 | 6 | data azurerm_firewall gateway { 7 | name = element(split("/",var.firewall_id),length(split("/",var.firewall_id))-1) 8 | resource_group_name = element(split("/",var.firewall_id),length(split("/",var.firewall_id))-5) 9 | } 10 | 11 | data azurerm_public_ip firewall_pip { 12 | name = element(split("/",data.azurerm_firewall.gateway.ip_configuration.0.public_ip_address_id),length(split("/",data.azurerm_firewall.gateway.ip_configuration.0.public_ip_address_id))-1) 13 | resource_group_name = element(split("/",data.azurerm_firewall.gateway.ip_configuration.0.public_ip_address_id),length(split("/",data.azurerm_firewall.gateway.ip_configuration.0.public_ip_address_id))-5) 14 | } 15 | 16 | data azurerm_subnet nodes_subnet { 17 | name = element(split("/",var.nodes_subnet_id),length(split("/",var.nodes_subnet_id))-1) 18 | virtual_network_name = element(split("/",var.nodes_subnet_id),length(split("/",var.nodes_subnet_id))-3) 19 | resource_group_name = element(split("/",var.nodes_subnet_id),length(split("/",var.nodes_subnet_id))-7) 20 | } 21 | 22 | locals { 23 | api_server_domain = join(".",slice(split(".",local.api_server_host),1,length(split(".",local.api_server_host)))) 24 | api_server_host = regex("^(?:(?P[^:/?#]+):)?(?://(?P[^:/?#]*))?", data.azurerm_kubernetes_cluster.aks.kube_admin_config.0.host).host 25 | peer_network_name = element(split("/",var.peer_network_id),length(split("/",var.peer_network_id))-1) 26 | } 27 | 28 | resource azurerm_ip_group api_server { 29 | name = "${var.resource_group_name}-ipgroup-apiserver" 30 | location = var.location 31 | resource_group_name = var.resource_group_name 32 | cidrs = data.azurerm_kubernetes_cluster.aks.api_server_authorized_ip_ranges 33 | 34 | tags = var.tags 35 | } 36 | 37 | resource azurerm_private_dns_zone acr { 38 | name = "privatelink.azurecr.io" 39 | resource_group_name = var.resource_group_name 40 | } 41 | resource azurerm_private_dns_zone_virtual_network_link acr { 42 | name = "${var.resource_group_name}-registry-dns-link" 43 | resource_group_name = var.resource_group_name 44 | private_dns_zone_name = azurerm_private_dns_zone.acr.name 45 | virtual_network_id = var.virtual_network_id 46 | } 47 | resource azurerm_private_endpoint acr_endpoint { 48 | name = "${var.resource_group_name}-registry-endpoint" 49 | location = var.location 50 | resource_group_name = var.resource_group_name 51 | 52 | subnet_id = var.paas_subnet_id 53 | 54 | private_dns_zone_group { 55 | name = azurerm_private_dns_zone.acr.name 56 | private_dns_zone_ids = [azurerm_private_dns_zone.acr.id] 57 | } 58 | 59 | private_service_connection { 60 | is_manual_connection = false 61 | name = "${var.resource_group_name}-registry-endpoint-connection" 62 | private_connection_resource_id = var.container_registry_id 63 | subresource_names = ["registry"] 64 | } 65 | 66 | tags = var.tags 67 | } 68 | 69 | # Set up name resolution for peered network 70 | data azurerm_private_dns_zone api_server_domain { 71 | name = local.api_server_domain 72 | resource_group_name = data.azurerm_kubernetes_cluster.aks.node_resource_group 73 | 74 | count = var.private_cluster_enabled ? 1 : 0 75 | } 76 | resource azurerm_private_dns_zone_virtual_network_link api_server_domain { 77 | name = "${local.peer_network_name}-zone-link" 78 | resource_group_name = data.azurerm_kubernetes_cluster.aks.node_resource_group 79 | private_dns_zone_name = data.azurerm_private_dns_zone.api_server_domain.0.name 80 | virtual_network_id = var.peer_network_id 81 | 82 | tags = var.tags 83 | 84 | count = var.peer_network_id != "" && var.private_cluster_enabled ? 1 : 0 85 | } 86 | -------------------------------------------------------------------------------- /terraform/modules/aks-network/outputs.tf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekzter/azure-aks/5de39f09d032c14eb1a695e02c7d3306afc493f0/terraform/modules/aks-network/outputs.tf -------------------------------------------------------------------------------- /terraform/modules/aks-network/variables.tf: -------------------------------------------------------------------------------- 1 | variable admin_ip_group_id {} 2 | variable aks_id {} 3 | variable container_registry_id {} 4 | variable firewall_id {} 5 | variable location {} 6 | variable nodes_ip_group_id {} 7 | variable nodes_subnet_id {} 8 | variable paas_subnet_id {} 9 | variable peer_network_id {} 10 | variable private_cluster_enabled { 11 | type = bool 12 | } 13 | variable resource_group_id {} 14 | variable resource_group_name {} 15 | variable tags { 16 | type = map 17 | } 18 | variable virtual_network_id {} -------------------------------------------------------------------------------- /terraform/modules/aks/main.tf: -------------------------------------------------------------------------------- 1 | locals { 2 | kubernetes_version = var.kubernetes_version != null && var.kubernetes_version != "" ? var.kubernetes_version : data.azurerm_kubernetes_service_versions.current.latest_version 3 | resource_group_name = element(split("/",var.resource_group_id),length(split("/",var.resource_group_id))-1) 4 | } 5 | 6 | data azurerm_subscription primary {} 7 | 8 | data azurerm_subnet nodes_subnet { 9 | name = element(split("/",var.node_subnet_id),length(split("/",var.node_subnet_id))-1) 10 | virtual_network_name = element(split("/",var.node_subnet_id),length(split("/",var.node_subnet_id))-3) 11 | resource_group_name = element(split("/",var.node_subnet_id),length(split("/",var.node_subnet_id))-7) 12 | } 13 | 14 | resource azurerm_user_assigned_identity aks_identity { 15 | name = "${var.name}-identity" 16 | location = var.location 17 | resource_group_name = local.resource_group_name 18 | } 19 | 20 | # AKS needs permission to make changes for kubelet networking mode 21 | resource azurerm_role_assignment spn_network_permission { 22 | scope = var.resource_group_id 23 | role_definition_name = "Network Contributor" 24 | principal_id = azurerm_user_assigned_identity.aks_identity.principal_id 25 | 26 | count = var.configure_access_control ? 1 : 0 27 | } 28 | 29 | # AKS needs permission for BYO DNS 30 | resource azurerm_role_assignment spn_dns_permission { 31 | scope = var.resource_group_id 32 | role_definition_name = "Private DNS Zone Contributor" 33 | principal_id = azurerm_user_assigned_identity.aks_identity.principal_id 34 | 35 | count = var.configure_access_control ? 1 : 0 36 | } 37 | 38 | # Requires Terraform owner access to resource group, in order to be able to perform access management 39 | resource azurerm_role_assignment spn_permission { 40 | scope = var.resource_group_id 41 | role_definition_name = "Virtual Machine Contributor" 42 | principal_id = azurerm_user_assigned_identity.aks_identity.principal_id 43 | 44 | count = var.configure_access_control ? 1 : 0 45 | } 46 | 47 | # Grant Terraform user Cluster Admin role 48 | resource azurerm_role_assignment terraform_cluster_permission { 49 | scope = var.resource_group_id 50 | role_definition_name = "Azure Kubernetes Service Cluster Admin Role" 51 | principal_id = var.client_object_id 52 | 53 | count = var.configure_access_control ? 1 : 0 54 | } 55 | 56 | data azurerm_kubernetes_service_versions current { 57 | location = var.location 58 | include_preview = false 59 | } 60 | 61 | resource azurerm_kubernetes_cluster aks { 62 | name = var.name 63 | location = var.location 64 | resource_group_name = local.resource_group_name 65 | dns_prefix = var.dns_prefix 66 | 67 | # Triggers resource to be recreated 68 | kubernetes_version = local.kubernetes_version 69 | 70 | automatic_channel_upgrade = "stable" 71 | 72 | azure_active_directory_role_based_access_control { 73 | admin_group_object_ids = [var.client_object_id] 74 | azure_rbac_enabled = true 75 | managed = true 76 | } 77 | 78 | azure_policy_enabled = true 79 | 80 | default_node_pool { 81 | enable_auto_scaling = true 82 | enable_host_encryption = false # Requires 'Microsoft.Compute/EncryptionAtHost' feature 83 | enable_node_public_ip = false 84 | min_count = 3 85 | max_count = 10 86 | name = "default" 87 | node_count = 3 88 | tags = var.tags 89 | # https://docs.microsoft.com/en-us/azure/virtual-machines/disk-encryption#supported-vm-sizes 90 | vm_size = var.node_size 91 | vnet_subnet_id = var.node_subnet_id 92 | } 93 | 94 | http_application_routing_enabled = true 95 | 96 | identity { 97 | type = "UserAssigned" 98 | identity_ids = [azurerm_user_assigned_identity.aks_identity.id] 99 | } 100 | 101 | dynamic "ingress_application_gateway" { 102 | for_each = range(var.deploy_application_gateway ? 1 : 0) 103 | content { 104 | gateway_name = "applicationgateway" 105 | subnet_id = var.application_gateway_subnet_id 106 | } 107 | } 108 | 109 | # local_account_disabled = true # Will become default in 1.24 110 | 111 | network_profile { 112 | network_plugin = "azure" 113 | network_policy = "azure" 114 | outbound_type = "userDefinedRouting" 115 | } 116 | 117 | oms_agent { 118 | log_analytics_workspace_id = var.log_analytics_workspace_id 119 | } 120 | 121 | private_cluster_enabled = var.private_cluster_enabled 122 | private_dns_zone_id = "System" 123 | #private_cluster_public_fqdn_enabled = true 124 | 125 | role_based_access_control_enabled = true 126 | 127 | lifecycle { 128 | ignore_changes = [ 129 | default_node_pool.0.node_count # Ignore changes made by autoscaling 130 | ] 131 | } 132 | 133 | tags = var.tags 134 | 135 | depends_on = [ 136 | azurerm_role_assignment.spn_permission, 137 | azurerm_role_assignment.spn_dns_permission, 138 | azurerm_role_assignment.spn_network_permission, 139 | ] 140 | } 141 | 142 | data azurerm_private_endpoint_connection api_server_endpoint { 143 | name = "kube-apiserver" 144 | resource_group_name = azurerm_kubernetes_cluster.aks.node_resource_group 145 | 146 | count = var.private_cluster_enabled ? 1 : 0 147 | } 148 | 149 | data azurerm_application_gateway app_gw { 150 | name = split("/",azurerm_kubernetes_cluster.aks.ingress_application_gateway[0].effective_gateway_id)[8] 151 | resource_group_name = azurerm_kubernetes_cluster.aks.node_resource_group 152 | 153 | count = var.deploy_application_gateway ? 1 : 0 154 | } 155 | resource random_string application_gateway_domain_label { 156 | length = min(16,63-length(var.dns_host_suffix)) 157 | upper = false 158 | lower = true 159 | numeric = false 160 | special = false 161 | } 162 | 163 | locals { 164 | application_gateway_domain_label = "${random_string.application_gateway_domain_label.result}${var.dns_host_suffix}" 165 | } 166 | 167 | resource null_resource application_gateway_domain_label { 168 | provisioner local-exec { 169 | command = "az network public-ip update --dns-name ${local.application_gateway_domain_label} -n ${data.azurerm_application_gateway.app_gw.0.name}-appgwpip -g ${azurerm_kubernetes_cluster.aks.node_resource_group} --subscription ${data.azurerm_subscription.primary.subscription_id} --query 'dnsSettings'" 170 | } 171 | 172 | count = var.deploy_application_gateway ? 1 : 0 173 | depends_on = [random_string.application_gateway_domain_label] 174 | } 175 | 176 | data azurerm_public_ip application_gateway_public_ip { 177 | name = "${data.azurerm_application_gateway.app_gw.0.name}-appgwpip" 178 | resource_group_name = azurerm_kubernetes_cluster.aks.node_resource_group 179 | 180 | count = var.deploy_application_gateway ? 1 : 0 181 | depends_on = [null_resource.application_gateway_domain_label] 182 | } 183 | 184 | data azurerm_resources scale_sets { 185 | resource_group_name = azurerm_kubernetes_cluster.aks.node_resource_group 186 | type = "Microsoft.Compute/virtualMachineScaleSets" 187 | 188 | required_tags = azurerm_kubernetes_cluster.aks.tags 189 | } 190 | 191 | # Export kube_config for kubectl 192 | resource local_file kube_config { 193 | filename = var.kube_config_path 194 | content = azurerm_kubernetes_cluster.aks.kube_admin_config_raw 195 | } 196 | -------------------------------------------------------------------------------- /terraform/modules/aks/monitoring.tf: -------------------------------------------------------------------------------- 1 | 2 | data azurerm_log_analytics_workspace log_analytics { 3 | name = element(split("/",var.log_analytics_workspace_id),length(split("/",var.log_analytics_workspace_id))-1) 4 | resource_group_name = element(split("/",var.log_analytics_workspace_id),length(split("/",var.log_analytics_workspace_id))-5) 5 | } 6 | 7 | resource azurerm_log_analytics_solution log_analytics_solution { 8 | solution_name = "ContainerInsights" 9 | location = var.location 10 | resource_group_name = data.azurerm_log_analytics_workspace.log_analytics.resource_group_name 11 | workspace_resource_id = var.log_analytics_workspace_id 12 | workspace_name = data.azurerm_log_analytics_workspace.log_analytics.name 13 | 14 | plan { 15 | publisher = "Microsoft" 16 | product = "OMSGallery/ContainerInsights" 17 | } 18 | } 19 | 20 | resource azurerm_monitor_diagnostic_setting aks { 21 | name = "${azurerm_kubernetes_cluster.aks.name}-logs" 22 | target_resource_id = azurerm_kubernetes_cluster.aks.id 23 | log_analytics_workspace_id = var.log_analytics_workspace_id 24 | 25 | enabled_log { 26 | category = "kube-apiserver" 27 | } 28 | enabled_log { 29 | category = "kube-audit" 30 | } 31 | enabled_log { 32 | category = "kube-audit-admin" 33 | } 34 | enabled_log { 35 | category = "kube-controller-manager" 36 | } 37 | enabled_log { 38 | category = "kube-scheduler" 39 | } 40 | enabled_log { 41 | category = "cluster-autoscaler" 42 | } 43 | enabled_log { 44 | category = "guard" 45 | } 46 | 47 | metric { 48 | category = "AllMetrics" 49 | } 50 | } 51 | 52 | resource azurerm_monitor_diagnostic_setting application_gateway_logs { 53 | name = "${split("/",azurerm_kubernetes_cluster.aks.ingress_application_gateway[0].effective_gateway_id)[8]}-logs" 54 | target_resource_id = azurerm_kubernetes_cluster.aks.ingress_application_gateway[0].effective_gateway_id 55 | log_analytics_workspace_id = var.log_analytics_workspace_id 56 | 57 | enabled_log { 58 | category = "ApplicationGatewayAccessLog" 59 | } 60 | enabled_log { 61 | category = "ApplicationGatewayPerformanceLog" 62 | } 63 | enabled_log { 64 | category = "ApplicationGatewayFirewallLog" 65 | } 66 | 67 | metric { 68 | category = "AllMetrics" 69 | } 70 | 71 | count = var.deploy_application_gateway ? 1 : 0 72 | } 73 | 74 | resource azurerm_monitor_diagnostic_setting scale_set { 75 | name = "${split("/",data.azurerm_resources.scale_sets.resources[0].id)[8]}-logs" 76 | target_resource_id = data.azurerm_resources.scale_sets.resources[0].id 77 | log_analytics_workspace_id = var.log_analytics_workspace_id 78 | 79 | metric { 80 | category = "AllMetrics" 81 | } 82 | 83 | lifecycle { 84 | ignore_changes = [ 85 | # New values are not known after plan stage, but won't change 86 | name, 87 | target_resource_id 88 | ] 89 | } 90 | } -------------------------------------------------------------------------------- /terraform/modules/aks/outputs.tf: -------------------------------------------------------------------------------- 1 | output aks_id { 2 | value = azurerm_kubernetes_cluster.aks.id 3 | } 4 | 5 | output application_gateway_id { 6 | value = var.deploy_application_gateway ? azurerm_kubernetes_cluster.aks.ingress_application_gateway[0].effective_gateway_id : null 7 | } 8 | 9 | output application_gateway_fqdn { 10 | value = var.deploy_application_gateway ? data.azurerm_public_ip.application_gateway_public_ip.0.fqdn : null 11 | } 12 | 13 | output application_gateway_public_ip { 14 | value = var.deploy_application_gateway ? data.azurerm_public_ip.application_gateway_public_ip.0.ip_address : null 15 | } 16 | 17 | output kube_config { 18 | value = azurerm_kubernetes_cluster.aks.kube_admin_config_raw 19 | } 20 | 21 | output kubernetes_api_server_ip_address { 22 | value = var.private_cluster_enabled ? data.azurerm_private_endpoint_connection.api_server_endpoint[0].private_service_connection.0.private_ip_address : null 23 | } 24 | output kubernetes_client_certificate { 25 | value = azurerm_kubernetes_cluster.aks.kube_admin_config.0.client_certificate 26 | } 27 | output kubernetes_client_key { 28 | value = azurerm_kubernetes_cluster.aks.kube_admin_config.0.client_key 29 | } 30 | output kubernetes_cluster_ca_certificate { 31 | value = azurerm_kubernetes_cluster.aks.kube_admin_config.0.cluster_ca_certificate 32 | } 33 | output kubernetes_host { 34 | sensitive = true 35 | value = azurerm_kubernetes_cluster.aks.kube_admin_config.0.host 36 | } 37 | 38 | output kubernetes_version { 39 | value = azurerm_kubernetes_cluster.aks.kubernetes_version 40 | } 41 | 42 | output node_pool_scale_set_id { 43 | value = data.azurerm_resources.scale_sets.resources[0].id 44 | } 45 | 46 | output node_resource_group { 47 | value = azurerm_kubernetes_cluster.aks.node_resource_group 48 | } -------------------------------------------------------------------------------- /terraform/modules/aks/variables.tf: -------------------------------------------------------------------------------- 1 | variable admin_username {} 2 | variable application_gateway_subnet_id {} 3 | variable client_object_id {} 4 | variable configure_access_control { 5 | type = bool 6 | } 7 | variable deploy_application_gateway { 8 | type = bool 9 | } 10 | variable dns_prefix {} 11 | variable dns_host_suffix {} 12 | variable kube_config_path {} 13 | variable kubernetes_version {} 14 | variable location {} 15 | variable log_analytics_workspace_id {} 16 | variable name {} 17 | variable node_size {} 18 | variable node_subnet_id {} 19 | variable private_cluster_enabled { 20 | type = bool 21 | } 22 | variable resource_group_id {} 23 | variable ssh_public_key_file {} 24 | variable tags { 25 | type = map 26 | } -------------------------------------------------------------------------------- /terraform/modules/bastion/main.tf: -------------------------------------------------------------------------------- 1 | resource azurerm_public_ip bastion_pip { 2 | name = "${var.resource_group_name}-bastion-pip" 3 | location = var.location 4 | resource_group_name = var.resource_group_name 5 | allocation_method = "Static" 6 | sku = "Standard" # Zone redundant 7 | 8 | tags = var.tags 9 | } 10 | resource azurerm_monitor_diagnostic_setting bastion_pip { 11 | name = "${azurerm_public_ip.bastion_pip.name}-logs" 12 | target_resource_id = azurerm_public_ip.bastion_pip.id 13 | log_analytics_workspace_id = var.log_analytics_workspace_id 14 | 15 | enabled_log { 16 | category = "DDoSProtectionNotifications" 17 | } 18 | enabled_log { 19 | category = "DDoSMitigationFlowLogs" 20 | } 21 | enabled_log { 22 | category = "DDoSMitigationReports" 23 | } 24 | 25 | metric { 26 | category = "AllMetrics" 27 | } 28 | } 29 | 30 | # https://docs.microsoft.com/en-us/azure/bastion/bastion-nsg 31 | resource azurerm_network_security_group bastion_nsg { 32 | name = "${var.resource_group_name}-bastion-nsg" 33 | location = var.location 34 | resource_group_name = var.resource_group_name 35 | 36 | tags = var.tags 37 | } 38 | resource azurerm_network_security_rule https_inbound { 39 | name = "AllowHttpsInbound" 40 | priority = 220 41 | direction = "Inbound" 42 | access = "Allow" 43 | protocol = "Tcp" 44 | source_port_range = "*" 45 | destination_port_range = "443" 46 | source_address_prefix = "Internet" 47 | destination_address_prefix = "*" 48 | resource_group_name = var.resource_group_name 49 | network_security_group_name = azurerm_network_security_group.bastion_nsg.name 50 | } 51 | resource azurerm_network_security_rule gateway_manager_inbound { 52 | name = "AllowGatewayManagerInbound" 53 | priority = 230 54 | direction = "Inbound" 55 | access = "Allow" 56 | protocol = "Tcp" 57 | source_port_range = "*" 58 | destination_port_range = "443" 59 | source_address_prefix = "GatewayManager" 60 | destination_address_prefix = "*" 61 | resource_group_name = var.resource_group_name 62 | network_security_group_name = azurerm_network_security_group.bastion_nsg.name 63 | } 64 | resource azurerm_network_security_rule load_balancer_inbound { 65 | name = "AllowLoadBalancerInbound" 66 | priority = 240 67 | direction = "Inbound" 68 | access = "Allow" 69 | protocol = "Tcp" 70 | source_port_range = "*" 71 | destination_port_range = "443" 72 | source_address_prefix = "AzureLoadBalancer" 73 | destination_address_prefix = "*" 74 | resource_group_name = var.resource_group_name 75 | network_security_group_name = azurerm_network_security_group.bastion_nsg.name 76 | } 77 | resource azurerm_network_security_rule bastion_host_communication_inbound { 78 | name = "AllowBastionHostCommunication" 79 | priority = 250 80 | direction = "Inbound" 81 | access = "Allow" 82 | protocol = "*" 83 | source_port_range = "*" 84 | destination_port_ranges = ["5701","8080"] 85 | source_address_prefix = "VirtualNetwork" 86 | destination_address_prefix = "VirtualNetwork" 87 | resource_group_name = var.resource_group_name 88 | network_security_group_name = azurerm_network_security_group.bastion_nsg.name 89 | } 90 | resource azurerm_network_security_rule ras_outbound { 91 | name = "AllowSshRdpOutbound" 92 | priority = 200 93 | direction = "Outbound" 94 | access = "Allow" 95 | protocol = "*" 96 | source_port_range = "*" 97 | destination_port_ranges = ["22","3389"] 98 | source_address_prefix = "*" 99 | destination_address_prefix = "VirtualNetwork" 100 | resource_group_name = var.resource_group_name 101 | network_security_group_name = azurerm_network_security_group.bastion_nsg.name 102 | } 103 | resource azurerm_network_security_rule azure_outbound { 104 | name = "AllowAzureCloudOutbound" 105 | priority = 210 106 | direction = "Outbound" 107 | access = "Allow" 108 | protocol = "Tcp" 109 | source_port_range = "*" 110 | destination_port_range = "443" 111 | source_address_prefix = "*" 112 | destination_address_prefix = "AzureCloud" 113 | resource_group_name = var.resource_group_name 114 | network_security_group_name = azurerm_network_security_group.bastion_nsg.name 115 | } 116 | resource azurerm_network_security_rule bastion_host_communication_oubound { 117 | name = "AllowBastionCommunication" 118 | priority = 220 119 | direction = "Outbound" 120 | access = "Allow" 121 | protocol = "*" 122 | source_port_range = "*" 123 | destination_port_ranges = ["5701","8080"] 124 | source_address_prefix = "VirtualNetwork" 125 | destination_address_prefix = "VirtualNetwork" 126 | resource_group_name = var.resource_group_name 127 | network_security_group_name = azurerm_network_security_group.bastion_nsg.name 128 | } 129 | resource azurerm_network_security_rule get_session_oubound { 130 | name = "AllowGetSessionInformation" 131 | priority = 230 132 | direction = "Outbound" 133 | access = "Allow" 134 | protocol = "*" 135 | source_port_range = "*" 136 | destination_port_range = "80" 137 | source_address_prefix = "*" 138 | destination_address_prefix = "Internet" 139 | resource_group_name = var.resource_group_name 140 | network_security_group_name = azurerm_network_security_group.bastion_nsg.name 141 | } 142 | resource azurerm_subnet_network_security_group_association bastion_nsg { 143 | subnet_id = var.subnet_id 144 | network_security_group_id = azurerm_network_security_group.bastion_nsg.id 145 | 146 | depends_on = [ 147 | azurerm_network_security_rule.https_inbound, 148 | azurerm_network_security_rule.gateway_manager_inbound, 149 | azurerm_network_security_rule.load_balancer_inbound, 150 | azurerm_network_security_rule.bastion_host_communication_inbound, 151 | azurerm_network_security_rule.ras_outbound, 152 | azurerm_network_security_rule.azure_outbound, 153 | azurerm_network_security_rule.bastion_host_communication_oubound, 154 | azurerm_network_security_rule.get_session_oubound, 155 | ] 156 | } 157 | 158 | resource azurerm_bastion_host managed_bastion { 159 | name = "${var.resource_group_name}-bastion" 160 | location = var.location 161 | resource_group_name = var.resource_group_name 162 | 163 | ip_configuration { 164 | name = "bastion-ipconfig" 165 | subnet_id = var.subnet_id 166 | public_ip_address_id = azurerm_public_ip.bastion_pip.id 167 | } 168 | 169 | tags = var.tags 170 | 171 | depends_on = [azurerm_subnet_network_security_group_association.bastion_nsg] 172 | } 173 | 174 | resource azurerm_monitor_diagnostic_setting bastion_logs { 175 | name = "${azurerm_bastion_host.managed_bastion.name}-logs" 176 | target_resource_id = azurerm_bastion_host.managed_bastion.id 177 | log_analytics_workspace_id = var.log_analytics_workspace_id 178 | 179 | enabled_log { 180 | category = "BastionAuditLogs" 181 | } 182 | } -------------------------------------------------------------------------------- /terraform/modules/bastion/variables.tf: -------------------------------------------------------------------------------- 1 | variable location {} 2 | variable log_analytics_workspace_id {} 3 | variable resource_group_name {} 4 | variable subnet_id {} 5 | variable tags { 6 | type = map 7 | } -------------------------------------------------------------------------------- /terraform/modules/kubernetes/main.tf: -------------------------------------------------------------------------------- 1 | # # https://weaveworks.github.io/kured/ 2 | # resource helm_release kured { 3 | # name = "kured-release" 4 | # repository = "https://weaveworks.github.io/kured/" 5 | # chart = "kured" 6 | # } -------------------------------------------------------------------------------- /terraform/modules/network/egress.tf: -------------------------------------------------------------------------------- 1 | 2 | # https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#azure-global-required-network-rules 3 | # Only rules that have no dependency on AKS being created first 4 | resource azurerm_firewall_network_rule_collection iag_net_outbound_rules { 5 | name = "${azurerm_firewall.gateway.name}-network-rules" 6 | azure_firewall_name = azurerm_firewall.gateway.name 7 | resource_group_name = azurerm_firewall.gateway.resource_group_name 8 | priority = 1001 9 | action = "Allow" 10 | 11 | # rule { 12 | # name = "AllowOutboundAKSAPIServer1" 13 | # source_ip_groups = [azurerm_ip_group.nodes.id] 14 | # destination_ports = ["1194"] 15 | # destination_addresses = [ 16 | # "AzureCloud.${var.location}", 17 | # ] 18 | # protocols = ["UDP"] 19 | # } 20 | # rule { 21 | # name = "AllowOutboundAKSAPIServer2" 22 | # source_ip_groups = [azurerm_ip_group.nodes.id] 23 | # destination_ports = ["9000"] 24 | # destination_addresses = [ 25 | # "AzureCloud.${var.location}", 26 | # ] 27 | # protocols = ["TCP"] 28 | # } 29 | # rule { 30 | # name = "AllowOutboundAKSAPIServerHTTPS" 31 | # source_ip_groups = [azurerm_ip_group.nodes.id] 32 | # destination_ports = ["443"] 33 | # destination_addresses = [ 34 | # "AzureCloud.${var.location}", 35 | # ] 36 | # protocols = ["TCP"] 37 | # } 38 | 39 | rule { 40 | name = "AllowUbuntuNTP" 41 | source_ip_groups = [azurerm_ip_group.nodes.id] 42 | destination_ports = ["123"] 43 | destination_fqdns = [ 44 | "ntp.ubuntu.com", 45 | ] 46 | protocols = ["UDP"] 47 | } 48 | rule { 49 | name = "AllowNTP" 50 | source_ip_groups = [azurerm_ip_group.nodes.id] 51 | destination_ports = ["123"] 52 | destination_fqdns = [ 53 | "pool.ntp.org", 54 | ] 55 | protocols = ["UDP"] 56 | } 57 | } 58 | 59 | # https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#azure-global-required-fqdn--application-rules 60 | resource azurerm_firewall_application_rule_collection aks_app_rules { 61 | name = "${azurerm_firewall.gateway.name}-app-rules" 62 | azure_firewall_name = azurerm_firewall.gateway.name 63 | resource_group_name = azurerm_firewall.gateway.resource_group_name 64 | priority = 2001 65 | action = "Allow" 66 | 67 | # https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#azure-global-required-fqdn--application-rules 68 | rule { 69 | name = "Allow outbound traffic" 70 | 71 | source_ip_groups = [azurerm_ip_group.nodes.id] 72 | target_fqdns = [ 73 | "*.hcp.${var.location}.azmk8s.io", 74 | "mcr.microsoft.com", 75 | "*.data.mcr.microsoft.com", 76 | "management.azure.com", 77 | "login.microsoftonline.com", 78 | "packages.microsoft.com", 79 | "acs-mirror.azureedge.net", 80 | ] 81 | 82 | protocol { 83 | port = "443" 84 | type = "Https" 85 | } 86 | } 87 | 88 | # https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#optional-recommended-fqdn--application-rules-for-aks-clusters 89 | rule { 90 | name = "Allow outbound AKS optional traffic (recommended)" 91 | 92 | source_ip_groups = [azurerm_ip_group.nodes.id] 93 | target_fqdns = [ 94 | "security.ubuntu.com", 95 | "azure.archive.ubuntu.com", 96 | "changelogs.ubuntu.com", 97 | ] 98 | 99 | protocol { 100 | port = "80" 101 | type = "Http" 102 | } 103 | } 104 | 105 | # https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#gpu-enabled-aks-clusters 106 | rule { 107 | name = "Allow outbound AKS optional traffic (GPU enabled nodes)" 108 | 109 | source_ip_groups = [azurerm_ip_group.nodes.id] 110 | target_fqdns = [ 111 | "nvidia.github.io", 112 | "*.download.nvidia.com", 113 | "apt.dockerproject.org", 114 | ] 115 | 116 | protocol { 117 | port = "443" 118 | type = "Https" 119 | } 120 | } 121 | 122 | # https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#gpu-enabled-aks-clusters 123 | rule { 124 | name = "Allow outbound AKS optional traffic (Windows enabled nodes)" 125 | 126 | source_ip_groups = [azurerm_ip_group.nodes.id] 127 | target_fqdns = [ 128 | "onegetcdn.azureedge.net", 129 | "go.microsoft.com", 130 | ] 131 | 132 | protocol { 133 | port = "443" 134 | type = "Https" 135 | } 136 | } 137 | 138 | # https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#gpu-enabled-aks-clusters 139 | rule { 140 | name = "Allow outbound AKS optional traffic (Windows enabled nodes, port 80)" 141 | 142 | source_ip_groups = [azurerm_ip_group.nodes.id] 143 | target_fqdns = [ 144 | "*.mp.microsoft.com", 145 | "www.msftconnecttest.com", 146 | "ctldl.windowsupdate.com", 147 | ] 148 | 149 | protocol { 150 | port = "443" 151 | type = "Https" 152 | } 153 | } 154 | 155 | # https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#azure-monitor-for-containers 156 | rule { 157 | name = "Allow outbound AKS Azure Monitor traffic" 158 | 159 | source_ip_groups = [azurerm_ip_group.nodes.id] 160 | target_fqdns = [ 161 | "dc.services.visualstudio.com", 162 | "*.ods.opinsights.azure.com", 163 | "*.oms.opinsights.azure.com", 164 | "*.monitoring.azure.com", 165 | ] 166 | 167 | protocol { 168 | port = "443" 169 | type = "Https" 170 | } 171 | } 172 | 173 | # https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#required-network-rules-1 174 | rule { 175 | name = "Allow DevSpaces" 176 | 177 | source_ip_groups = [azurerm_ip_group.nodes.id] 178 | fqdn_tags = ["AzureDevSpaces"] 179 | } 180 | 181 | 182 | # https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#azure-dev-spaces 183 | rule { 184 | name = "Allow outbound AKS Dev Spaces" 185 | 186 | source_ip_groups = [azurerm_ip_group.nodes.id] 187 | target_fqdns = [ 188 | "cloudflare.docker.com", 189 | "gcr.io", 190 | "storage.googleapis.com", 191 | ] 192 | 193 | protocol { 194 | port = "443" 195 | type = "Https" 196 | } 197 | } 198 | 199 | # https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#azure-policy 200 | rule { 201 | name = "Allow outbound AKS Azure Policy" 202 | 203 | source_ip_groups = [azurerm_ip_group.nodes.id] 204 | target_fqdns = [ 205 | "data.policy.core.windows.net", 206 | "store.policy.core.windows.net", 207 | "gov-prod-policy-data.trafficmanager.net", 208 | "raw.githubusercontent.com", 209 | "dc.services.visualstudio.com", 210 | ] 211 | 212 | protocol { 213 | port = "443" 214 | type = "Https" 215 | } 216 | } 217 | 218 | # https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#restrict-egress-traffic-using-azure-firewall 219 | # Not required for private AKS, which uses a Private Endpoint 220 | # rule { 221 | # name = "Allow outbound AKS" 222 | 223 | # source_ip_groups = [azurerm_ip_group.nodes.id] 224 | # fqdn_tags = ["AzureKubernetesService"] 225 | # } 226 | 227 | # Traffic required, but not documented 228 | rule { 229 | name = "Allow misc container management traffic" 230 | 231 | source_ip_groups = [azurerm_ip_group.nodes.id] 232 | target_fqdns = [ 233 | "api.snapcraft.io", 234 | "auth.docker.io", 235 | "github.com", 236 | "ifconfig.co", 237 | "motd.ubuntu.com", 238 | "production.cloudflare.docker.com", 239 | "registry-1.docker.io", 240 | ] 241 | 242 | protocol { 243 | port = "443" 244 | type = "Https" 245 | } 246 | } 247 | 248 | } -------------------------------------------------------------------------------- /terraform/modules/network/firewall.tf: -------------------------------------------------------------------------------- 1 | resource random_string firewall_domain_name_label { 2 | length = 16 3 | upper = false 4 | lower = true 5 | numeric = false 6 | special = false 7 | } 8 | 9 | resource azurerm_ip_group admin { 10 | name = "${var.resource_group_name}-ipgroup-admin" 11 | location = var.location 12 | resource_group_name = var.resource_group_name 13 | cidrs = local.admin_cidrs 14 | 15 | tags = var.tags 16 | } 17 | 18 | resource azurerm_ip_group nodes { 19 | name = "${var.resource_group_name}-ipgroup-nodes" 20 | location = var.location 21 | resource_group_name = var.resource_group_name 22 | cidrs = azurerm_subnet.nodes_subnet.address_prefixes 23 | 24 | tags = var.tags 25 | } 26 | 27 | # https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#restrict-egress-traffic-using-azure-firewall 28 | # We recommend having a minimum of 20 Frontend IPs on the Azure Firewall for production scenarios to avoid incurring in SNAT port exhaustion issues. 29 | resource azurerm_public_ip firewall_pip { 30 | name = "${var.resource_group_name}-iag-pip" 31 | location = var.location 32 | resource_group_name = var.resource_group_name 33 | allocation_method = "Static" 34 | sku = "Standard" # Zone redundant 35 | domain_name_label = random_string.firewall_domain_name_label.result 36 | 37 | tags = var.tags 38 | } 39 | resource azurerm_monitor_diagnostic_setting firewall_pip { 40 | name = "${azurerm_public_ip.firewall_pip.name}-logs" 41 | target_resource_id = azurerm_public_ip.firewall_pip.id 42 | log_analytics_workspace_id = var.log_analytics_workspace_id 43 | 44 | enabled_log { 45 | category = "DDoSProtectionNotifications" 46 | } 47 | enabled_log { 48 | category = "DDoSMitigationFlowLogs" 49 | } 50 | enabled_log { 51 | category = "DDoSMitigationReports" 52 | } 53 | 54 | metric { 55 | category = "AllMetrics" 56 | } 57 | } 58 | 59 | resource azurerm_firewall gateway { 60 | name = "${var.resource_group_name}-iag" 61 | location = var.location 62 | resource_group_name = var.resource_group_name 63 | sku_name = "AZFW_VNet" 64 | sku_tier = "Standard" 65 | 66 | dns_servers = var.dns_servers 67 | 68 | ip_configuration { 69 | name = "firewall_ipconfig" 70 | subnet_id = azurerm_subnet.firewall_subnet.id 71 | public_ip_address_id = azurerm_public_ip.firewall_pip.id 72 | } 73 | 74 | tags = var.tags 75 | } 76 | 77 | resource azurerm_monitor_diagnostic_setting firewall_logs { 78 | name = "${azurerm_firewall.gateway.name}-logs" 79 | target_resource_id = azurerm_firewall.gateway.id 80 | log_analytics_workspace_id = var.log_analytics_workspace_id 81 | 82 | enabled_log { 83 | category = "AzureFirewallApplicationRule" 84 | } 85 | 86 | enabled_log { 87 | category = "AzureFirewallNetworkRule" 88 | } 89 | 90 | metric { 91 | category = "AllMetrics" 92 | } 93 | } -------------------------------------------------------------------------------- /terraform/modules/network/main.tf: -------------------------------------------------------------------------------- 1 | data http local_public_ip { 2 | # Get public IP address of the machine running this terraform template 3 | url = "https://ipinfo.io/ip" 4 | } 5 | 6 | data http local_public_prefix { 7 | # Get public IP prefix of the machine running this terraform template 8 | url = "https://stat.ripe.net/data/network-info/data.json?resource=${chomp(data.http.local_public_ip.response_body)}" 9 | } 10 | 11 | locals { 12 | admin_cidrs = [ 13 | cidrsubnet("${chomp(data.http.local_public_ip.response_body)}/30",0,0), # /32 not allowed in network_rules 14 | jsondecode(chomp(data.http.local_public_prefix.response_body)).data.prefix 15 | ] 16 | bastion_cidr = cidrsubnet(azurerm_virtual_network.network.address_space[0],3,2) # /26, assuming network is /23 17 | firewall_cidr = cidrsubnet(azurerm_virtual_network.network.address_space[0],3,0) 18 | nodes_cidr = cidrsubnet(azurerm_virtual_network.network.address_space[0],1,1) # /24, assuming network is /23 19 | paas_cidr = cidrsubnet(azurerm_virtual_network.network.address_space[0],3,3) # /24, assuming network is /23 20 | waf_cidr = cidrsubnet(azurerm_virtual_network.network.address_space[0],3,1) # /26, assuming network is /23 21 | } 22 | 23 | resource azurerm_virtual_network network { 24 | name = "${var.resource_group_name}-network" 25 | location = var.location 26 | resource_group_name = var.resource_group_name 27 | address_space = [var.address_space] 28 | dns_servers = var.dns_servers 29 | 30 | tags = var.tags 31 | } 32 | resource azurerm_monitor_diagnostic_setting network { 33 | name = "${azurerm_virtual_network.network.name}-logs" 34 | target_resource_id = azurerm_virtual_network.network.id 35 | log_analytics_workspace_id = var.log_analytics_workspace_id 36 | 37 | enabled_log { 38 | category = "VMProtectionAlerts" 39 | } 40 | 41 | metric { 42 | category = "AllMetrics" 43 | } 44 | } 45 | resource azurerm_subnet firewall_subnet { 46 | name = "AzureFirewallSubnet" 47 | virtual_network_name = azurerm_virtual_network.network.name 48 | resource_group_name = var.resource_group_name 49 | address_prefixes = [local.firewall_cidr] 50 | } 51 | 52 | resource azurerm_subnet waf_subnet { 53 | name = "ApplicationGatewaySubnet" 54 | virtual_network_name = azurerm_virtual_network.network.name 55 | resource_group_name = var.resource_group_name 56 | address_prefixes = [local.waf_cidr] 57 | 58 | # Reduce the likelihood of race conditions 59 | depends_on = [ 60 | azurerm_network_security_rule.allow_application_gateway_management, 61 | azurerm_network_security_rule.allow_azure_loadbalancer, 62 | azurerm_network_security_rule.allow_http 63 | ] 64 | } 65 | resource azurerm_network_security_group waf_nsg { 66 | name = "${azurerm_virtual_network.network.name}-waf-nsg" 67 | location = var.location 68 | resource_group_name = azurerm_virtual_network.network.resource_group_name 69 | 70 | tags = var.tags 71 | } 72 | # https://docs.microsoft.com/en-us/azure/application-gateway/configuration-infrastructure#network-security-groups 73 | resource azurerm_network_security_rule allow_application_gateway_management { 74 | name = "AllowAppGWManagementInbound" 75 | priority = 201 76 | direction = "Inbound" 77 | access = "Allow" 78 | protocol = "Tcp" 79 | source_port_range = "*" 80 | destination_port_range = "65200-65535" # Unblocks ApplicationGatewaySubnetInboundTrafficBlockedByNetworkSecurityGroup 81 | source_address_prefix = "GatewayManager" 82 | destination_address_prefix = "*" 83 | resource_group_name = azurerm_network_security_group.waf_nsg.resource_group_name 84 | network_security_group_name = azurerm_network_security_group.waf_nsg.name 85 | } 86 | resource azurerm_network_security_rule allow_azure_loadbalancer { 87 | name = "AllowAzureLoadBalancerInbound" 88 | priority = 202 89 | direction = "Inbound" 90 | access = "Allow" 91 | protocol = "Tcp" 92 | source_port_range = "*" 93 | destination_port_range = "*" 94 | source_address_prefix = "AzureLoadBalancer" 95 | destination_address_prefix = "*" 96 | resource_group_name = azurerm_network_security_group.waf_nsg.resource_group_name 97 | network_security_group_name = azurerm_network_security_group.waf_nsg.name 98 | } 99 | resource azurerm_network_security_rule allow_http { 100 | name = "AllowHttpInbound" 101 | priority = 203 102 | direction = "Inbound" 103 | access = "Allow" 104 | protocol = "Tcp" 105 | source_port_range = "*" 106 | destination_port_ranges = ["80","443"] 107 | source_address_prefix = "Internet" 108 | destination_address_prefixes = [local.waf_cidr] 109 | resource_group_name = azurerm_network_security_group.waf_nsg.resource_group_name 110 | network_security_group_name = azurerm_network_security_group.waf_nsg.name 111 | } 112 | resource azurerm_subnet_network_security_group_association waf_subnet { 113 | subnet_id = azurerm_subnet.waf_subnet.id 114 | network_security_group_id = azurerm_network_security_group.waf_nsg.id 115 | } 116 | resource time_sleep waf_nsg_wait_time { 117 | create_duration = "${var.nsg_reassign_wait_minutes}m" 118 | depends_on = [azurerm_subnet_network_security_group_association.waf_subnet] 119 | 120 | count = var.nsg_reassign_wait_minutes == 0 ? 0 : 1 121 | } 122 | data azurerm_subnet waf_subnet { 123 | name = azurerm_subnet.waf_subnet.name 124 | resource_group_name = azurerm_subnet.waf_subnet.resource_group_name 125 | virtual_network_name = azurerm_subnet.waf_subnet.virtual_network_name 126 | 127 | depends_on = [ 128 | time_sleep.waf_nsg_wait_time 129 | ] 130 | 131 | count = var.nsg_reassign_wait_minutes == 0 ? 0 : 1 132 | } 133 | # Address race condition where policy assigned NSG before we can assign our own 134 | # Let's wait for any updates to happen, then overwrite with our own 135 | resource null_resource waf_nsg_association { 136 | triggers = { 137 | nsg = coalesce(data.azurerm_subnet.waf_subnet.0.network_security_group_id,azurerm_network_security_group.waf_nsg.id) 138 | } 139 | 140 | provisioner local-exec { 141 | command = "${path.root}/../scripts/create_nsg_assignment.ps1 -SubnetId ${azurerm_subnet.waf_subnet.id} -NsgId ${azurerm_network_security_group.waf_nsg.id}" 142 | interpreter = ["pwsh","-nop","-command"] 143 | } 144 | 145 | count = var.nsg_reassign_wait_minutes == 0 ? 0 : 1 146 | } 147 | 148 | resource azurerm_subnet bastion_subnet { 149 | name = "AzureBastionSubnet" 150 | virtual_network_name = azurerm_virtual_network.network.name 151 | resource_group_name = var.resource_group_name 152 | address_prefixes = [local.bastion_cidr] 153 | 154 | # # Reduce the likelihood of race conditions 155 | # depends_on = [ 156 | # azurerm_network_security_rule.https_inbound, 157 | # azurerm_network_security_rule.gateway_manager_inbound, 158 | # azurerm_network_security_rule.load_balancer_inbound, 159 | # azurerm_network_security_rule.bastion_host_communication_inbound, 160 | # azurerm_network_security_rule.ras_outbound, 161 | # azurerm_network_security_rule.azure_outbound, 162 | # azurerm_network_security_rule.bastion_host_communication_oubound, 163 | # azurerm_network_security_rule.get_session_oubound, 164 | # ] 165 | } 166 | # # https://docs.microsoft.com/en-us/azure/bastion/bastion-nsg 167 | # resource azurerm_network_security_group bastion_nsg { 168 | # name = "${azurerm_virtual_network.network.name}-bastion-nsg" 169 | # location = var.location 170 | # resource_group_name = azurerm_virtual_network.network.resource_group_name 171 | 172 | # tags = var.tags 173 | # } 174 | # resource azurerm_network_security_rule https_inbound { 175 | # name = "AllowHttpsInbound" 176 | # priority = 220 177 | # direction = "Inbound" 178 | # access = "Allow" 179 | # protocol = "Tcp" 180 | # source_port_range = "*" 181 | # destination_port_range = "443" 182 | # source_address_prefix = "Internet" 183 | # destination_address_prefix = "*" 184 | # resource_group_name = azurerm_network_security_group.bastion_nsg.resource_group_name 185 | # network_security_group_name = azurerm_network_security_group.bastion_nsg.name 186 | # } 187 | # resource azurerm_network_security_rule gateway_manager_inbound { 188 | # name = "AllowGatewayManagerInbound" 189 | # priority = 230 190 | # direction = "Inbound" 191 | # access = "Allow" 192 | # protocol = "Tcp" 193 | # source_port_range = "*" 194 | # destination_port_range = "443" 195 | # source_address_prefix = "GatewayManager" 196 | # destination_address_prefix = "*" 197 | # resource_group_name = azurerm_network_security_group.bastion_nsg.resource_group_name 198 | # network_security_group_name = azurerm_network_security_group.bastion_nsg.name 199 | # } 200 | # resource azurerm_network_security_rule load_balancer_inbound { 201 | # name = "AllowLoadBalancerInbound" 202 | # priority = 240 203 | # direction = "Inbound" 204 | # access = "Allow" 205 | # protocol = "Tcp" 206 | # source_port_range = "*" 207 | # destination_port_range = "443" 208 | # source_address_prefix = "AzureLoadBalancer" 209 | # destination_address_prefix = "*" 210 | # resource_group_name = azurerm_network_security_group.bastion_nsg.resource_group_name 211 | # network_security_group_name = azurerm_network_security_group.bastion_nsg.name 212 | # } 213 | # resource azurerm_network_security_rule bastion_host_communication_inbound { 214 | # name = "AllowBastionHostCommunication" 215 | # priority = 250 216 | # direction = "Inbound" 217 | # access = "Allow" 218 | # protocol = "*" 219 | # source_port_range = "*" 220 | # destination_port_ranges = ["5701","8080"] 221 | # source_address_prefix = "VirtualNetwork" 222 | # destination_address_prefix = "VirtualNetwork" 223 | # resource_group_name = azurerm_network_security_group.bastion_nsg.resource_group_name 224 | # network_security_group_name = azurerm_network_security_group.bastion_nsg.name 225 | # } 226 | # resource azurerm_network_security_rule ras_outbound { 227 | # name = "AllowSshRdpOutbound" 228 | # priority = 200 229 | # direction = "Outbound" 230 | # access = "Allow" 231 | # protocol = "*" 232 | # source_port_range = "*" 233 | # destination_port_ranges = ["22","3389"] 234 | # source_address_prefix = "*" 235 | # destination_address_prefix = "VirtualNetwork" 236 | # resource_group_name = azurerm_network_security_group.bastion_nsg.resource_group_name 237 | # network_security_group_name = azurerm_network_security_group.bastion_nsg.name 238 | # } 239 | # resource azurerm_network_security_rule azure_outbound { 240 | # name = "AllowAzureCloudOutbound" 241 | # priority = 210 242 | # direction = "Outbound" 243 | # access = "Allow" 244 | # protocol = "Tcp" 245 | # source_port_range = "*" 246 | # destination_port_range = "443" 247 | # source_address_prefix = "*" 248 | # destination_address_prefix = "AzureCloud" 249 | # resource_group_name = azurerm_network_security_group.bastion_nsg.resource_group_name 250 | # network_security_group_name = azurerm_network_security_group.bastion_nsg.name 251 | # } 252 | # resource azurerm_network_security_rule bastion_host_communication_oubound { 253 | # name = "AllowBastionCommunication" 254 | # priority = 220 255 | # direction = "Outbound" 256 | # access = "Allow" 257 | # protocol = "*" 258 | # source_port_range = "*" 259 | # destination_port_ranges = ["5701","8080"] 260 | # source_address_prefix = "VirtualNetwork" 261 | # destination_address_prefix = "VirtualNetwork" 262 | # resource_group_name = azurerm_network_security_group.bastion_nsg.resource_group_name 263 | # network_security_group_name = azurerm_network_security_group.bastion_nsg.name 264 | # } 265 | # resource azurerm_network_security_rule get_session_oubound { 266 | # name = "AllowGetSessionInformation" 267 | # priority = 230 268 | # direction = "Outbound" 269 | # access = "Allow" 270 | # protocol = "*" 271 | # source_port_range = "*" 272 | # destination_port_range = "80" 273 | # source_address_prefix = "*" 274 | # destination_address_prefix = "Internet" 275 | # resource_group_name = azurerm_network_security_group.bastion_nsg.resource_group_name 276 | # network_security_group_name = azurerm_network_security_group.bastion_nsg.name 277 | # } 278 | # resource azurerm_subnet_network_security_group_association bastion_nsg { 279 | # subnet_id = azurerm_subnet.bastion_subnet.id 280 | # network_security_group_id = azurerm_network_security_group.bastion_nsg.id 281 | 282 | # depends_on = [ 283 | # azurerm_network_security_rule.https_inbound, 284 | # azurerm_network_security_rule.gateway_manager_inbound, 285 | # azurerm_network_security_rule.load_balancer_inbound, 286 | # azurerm_network_security_rule.bastion_host_communication_inbound, 287 | # azurerm_network_security_rule.ras_outbound, 288 | # azurerm_network_security_rule.azure_outbound, 289 | # azurerm_network_security_rule.bastion_host_communication_oubound, 290 | # azurerm_network_security_rule.get_session_oubound, 291 | # ] 292 | # } 293 | 294 | resource azurerm_subnet paas_subnet { 295 | name = "PrivateEndpoints" 296 | virtual_network_name = azurerm_virtual_network.network.name 297 | resource_group_name = var.resource_group_name 298 | address_prefixes = [local.paas_cidr] 299 | private_endpoint_network_policies_enabled = true 300 | } 301 | resource azurerm_subnet nodes_subnet { 302 | name = "KubernetesClusterNodes" 303 | virtual_network_name = azurerm_virtual_network.network.name 304 | resource_group_name = var.resource_group_name 305 | address_prefixes = [local.nodes_cidr] 306 | private_endpoint_network_policies_enabled = true 307 | } 308 | 309 | resource azurerm_route_table user_defined_routes { 310 | name = "${var.resource_group_name}-routes" 311 | location = var.location 312 | resource_group_name = var.resource_group_name 313 | 314 | route { 315 | name = "VnetLocal" 316 | address_prefix = "10.0.0.0/8" 317 | next_hop_type = "VnetLocal" 318 | } 319 | route { 320 | name = "InternetViaFW" 321 | address_prefix = "0.0.0.0/0" 322 | next_hop_type = "VirtualAppliance" 323 | next_hop_in_ip_address = azurerm_firewall.gateway.ip_configuration.0.private_ip_address 324 | } 325 | 326 | # AKS (in kubelet network mode) may add routes Terraform is not aware off 327 | lifecycle { 328 | ignore_changes = [ 329 | route 330 | ] 331 | } 332 | 333 | tags = var.tags 334 | } 335 | 336 | resource azurerm_subnet_route_table_association user_defined_routes { 337 | subnet_id = azurerm_subnet.nodes_subnet.id 338 | route_table_id = azurerm_route_table.user_defined_routes.id 339 | } 340 | 341 | data azurerm_virtual_network peered_network { 342 | name = element(split("/",var.peer_network_id),length(split("/",var.peer_network_id))-1) 343 | resource_group_name = element(split("/",var.peer_network_id),length(split("/",var.peer_network_id))-5) 344 | 345 | count = var.peer_network_id == "" ? 0 : 1 346 | } 347 | 348 | resource azurerm_virtual_network_peering network_to_peer { 349 | name = "${azurerm_virtual_network.network.name}-to-peer" 350 | resource_group_name = azurerm_virtual_network.network.resource_group_name 351 | virtual_network_name = azurerm_virtual_network.network.name 352 | remote_virtual_network_id = data.azurerm_virtual_network.peered_network.0.id 353 | 354 | allow_forwarded_traffic = true 355 | allow_gateway_transit = false 356 | allow_virtual_network_access = true 357 | use_remote_gateways = var.peer_network_has_gateway 358 | 359 | count = var.peer_network_id == "" ? 0 : 1 360 | 361 | depends_on = [azurerm_virtual_network_peering.peer_to_network] 362 | } 363 | 364 | resource azurerm_virtual_network_peering peer_to_network { 365 | name = "${azurerm_virtual_network.network.name}-from-peer" 366 | resource_group_name = data.azurerm_virtual_network.peered_network.0.resource_group_name 367 | virtual_network_name = data.azurerm_virtual_network.peered_network.0.name 368 | remote_virtual_network_id = azurerm_virtual_network.network.id 369 | 370 | allow_forwarded_traffic = true 371 | allow_gateway_transit = var.peer_network_has_gateway 372 | allow_virtual_network_access = true 373 | use_remote_gateways = false 374 | 375 | count = var.peer_network_id == "" ? 0 : 1 376 | } -------------------------------------------------------------------------------- /terraform/modules/network/outputs.tf: -------------------------------------------------------------------------------- 1 | output admin_ip_group_id { 2 | value = azurerm_ip_group.admin.id 3 | } 4 | 5 | output application_gateway_subnet_id { 6 | value = azurerm_subnet.waf_subnet.id 7 | } 8 | 9 | output bastion_subnet_id { 10 | value = azurerm_subnet.bastion_subnet.id 11 | } 12 | 13 | output firewall_fqdn { 14 | value = azurerm_public_ip.firewall_pip.fqdn 15 | } 16 | output firewall_id { 17 | value = azurerm_firewall.gateway.id 18 | } 19 | output firewall_subnet_id { 20 | value = azurerm_subnet.firewall_subnet.id 21 | } 22 | 23 | output nodes_ip_group_id { 24 | value = azurerm_ip_group.nodes.id 25 | } 26 | output nodes_subnet_id { 27 | value = azurerm_subnet.nodes_subnet.id 28 | } 29 | 30 | output paas_subnet_id { 31 | value = azurerm_subnet.paas_subnet.id 32 | } 33 | 34 | output virtual_network_id { 35 | value = azurerm_virtual_network.network.id 36 | } -------------------------------------------------------------------------------- /terraform/modules/network/variables.tf: -------------------------------------------------------------------------------- 1 | variable address_space {} 2 | variable dns_servers { 3 | type = list 4 | default = ["168.63.129.16"] 5 | } 6 | variable location {} 7 | variable log_analytics_workspace_id {} 8 | variable nsg_reassign_wait_minutes { 9 | type = number 10 | } 11 | variable peer_network_id {} 12 | variable peer_network_has_gateway { 13 | type = bool 14 | } 15 | variable resource_group_name {} 16 | variable tags { 17 | type = map 18 | } -------------------------------------------------------------------------------- /terraform/modules/service-principal/main.tf: -------------------------------------------------------------------------------- 1 | data azuread_client_config current {} 2 | 3 | resource azuread_application app { 4 | display_name = var.name 5 | owners = [data.azuread_client_config.current.object_id] 6 | sign_in_audience = "AzureADMyOrg" 7 | 8 | web { 9 | homepage_url = "https://${var.name}" 10 | implicit_grant { 11 | access_token_issuance_enabled = false 12 | } 13 | redirect_uris = ["http://${var.name}/replyignored"] 14 | } 15 | } 16 | 17 | resource azuread_service_principal spn { 18 | application_id = azuread_application.app.application_id 19 | owners = [data.azuread_client_config.current.object_id] 20 | } 21 | 22 | resource time_rotating secret_expiration { 23 | rotation_years = 1 24 | } 25 | 26 | resource azuread_service_principal_password spnsecret { 27 | rotate_when_changed = { 28 | rotation = timeadd(time_rotating.secret_expiration.id, "8760h") # One year from now 29 | } 30 | 31 | service_principal_id = azuread_service_principal.spn.id 32 | } -------------------------------------------------------------------------------- /terraform/modules/service-principal/outputs.tf: -------------------------------------------------------------------------------- 1 | output application_id { 2 | value = azuread_application.app.application_id 3 | } 4 | output object_id { 5 | value = azuread_service_principal.spn.object_id 6 | } 7 | output principal_id { 8 | value = azuread_service_principal.spn.id 9 | } 10 | output secret { 11 | value = azuread_service_principal_password.spnsecret.value 12 | } -------------------------------------------------------------------------------- /terraform/modules/service-principal/variables.tf: -------------------------------------------------------------------------------- 1 | variable name {} -------------------------------------------------------------------------------- /terraform/outputs.tf: -------------------------------------------------------------------------------- 1 | output address_space { 2 | value = var.address_space 3 | } 4 | 5 | output application_gateway_fqdn { 6 | value = var.deploy_aks ? module.aks.0.application_gateway_fqdn : null 7 | } 8 | output application_gateway_id { 9 | value = var.deploy_aks ? module.aks.0.application_gateway_id : null 10 | } 11 | output application_gateway_public_ip { 12 | value = var.deploy_aks ? module.aks.0.application_gateway_public_ip : null 13 | } 14 | 15 | output aks_name { 16 | value = var.deploy_aks ? local.aks_name : null 17 | } 18 | 19 | output connectivity_message { 20 | value = var.peer_network_id == "" ? "No peering configured. You will NOT be able to deploy applications from this host." : null 21 | } 22 | 23 | output firewall_fqdn { 24 | value = module.network.firewall_fqdn 25 | } 26 | 27 | output kube_config { 28 | sensitive = true 29 | value = var.deploy_aks ? module.aks.0.kube_config : null 30 | } 31 | 32 | output kube_config_base64 { 33 | sensitive = true 34 | value = var.deploy_aks ? base64encode(module.aks.0.kube_config) : null 35 | } 36 | output kube_config_path { 37 | # Return machine independent relative path 38 | value = local.kube_config_relative_path 39 | } 40 | 41 | output kubernetes_api_server_ip_address { 42 | value = var.deploy_aks ? module.aks.0.kubernetes_api_server_ip_address : null 43 | } 44 | 45 | output kubernetes_client_certificate { 46 | sensitive = true 47 | value = var.deploy_aks ? chomp(module.aks.0.kubernetes_client_certificate) : null 48 | } 49 | 50 | output kubernetes_host { 51 | sensitive = true 52 | value = var.deploy_aks ? module.aks.0.kubernetes_host : null 53 | } 54 | 55 | output kubernetes_version { 56 | value = var.deploy_aks ? module.aks.0.kubernetes_version : null 57 | } 58 | 59 | output node_resource_group { 60 | value = var.deploy_aks ? module.aks.0.node_resource_group : null 61 | } 62 | 63 | output peered_network { 64 | value = var.peer_network_id != "" ? true : false 65 | } 66 | 67 | output resource_group { 68 | value = azurerm_resource_group.rg.name 69 | } 70 | output resource_suffix { 71 | value = local.suffix 72 | } -------------------------------------------------------------------------------- /terraform/provider.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_providers { 3 | azuread = "~> 2.12" 4 | azurerm = "~> 3.24" 5 | external = "~> 2.1" 6 | helm = "~> 2.0" 7 | http = "~> 3.1" 8 | kubernetes = "~> 2.0" 9 | local = "~> 2.1" 10 | null = "~> 3.1" 11 | random = "~> 3.1" 12 | time = "~> 0.7" 13 | } 14 | required_version = "~> 1.0" 15 | } 16 | 17 | # Microsoft Azure Resource Manager Provider 18 | # This provider block uses the following environment variables: 19 | # ARM_SUBSCRIPTION_ID, ARM_CLIENT_ID, ARM_CLIENT_SECRET and ARM_TENANT_ID 20 | provider azurerm { 21 | features { 22 | resource_group { 23 | prevent_deletion_if_contains_resources = false 24 | } 25 | virtual_machine { 26 | # Don't do this in production 27 | delete_os_disk_on_deletion = true 28 | } 29 | } 30 | 31 | subscription_id = var.subscription_id != null && var.subscription_id != "" ? var.subscription_id : data.azurerm_subscription.default.subscription_id 32 | tenant_id = var.tenant_id != null && var.tenant_id != "" ? var.tenant_id : data.azurerm_subscription.default.tenant_id 33 | } 34 | 35 | provider azurerm { 36 | alias = "default" 37 | features {} 38 | } 39 | data azurerm_subscription default { 40 | provider = azurerm.default 41 | } 42 | 43 | # Use AKS to prepare Helm provider 44 | provider helm { 45 | kubernetes { 46 | config_path = var.deploy_aks ? local.kube_config_absolute_path : "" 47 | host = var.deploy_aks ? module.aks.0.kubernetes_host : "" 48 | client_certificate = var.deploy_aks ? base64decode(module.aks.0.kubernetes_client_certificate) : "" 49 | client_key = var.deploy_aks ? base64decode(module.aks.0.kubernetes_client_key) : "" 50 | cluster_ca_certificate = var.deploy_aks ? base64decode(module.aks.0.kubernetes_cluster_ca_certificate) : "" 51 | } 52 | } 53 | 54 | # Use AKS to prepare Kubernetes provider 55 | provider kubernetes { 56 | config_path = var.deploy_aks ? local.kube_config_absolute_path : "" 57 | host = var.deploy_aks ? module.aks.0.kubernetes_host : "" 58 | client_certificate = var.deploy_aks ? base64decode(module.aks.0.kubernetes_client_certificate) : "" 59 | client_key = var.deploy_aks ? base64decode(module.aks.0.kubernetes_client_key) : "" 60 | cluster_ca_certificate = var.deploy_aks ? base64decode(module.aks.0.kubernetes_cluster_ca_certificate) : "" 61 | } -------------------------------------------------------------------------------- /terraform/variables.tf: -------------------------------------------------------------------------------- 1 | variable address_space { 2 | default = "10.32.100.0/23" 3 | } 4 | 5 | variable application_name { 6 | description = "Value of 'application' resource tag" 7 | default = "Kubernetes" 8 | } 9 | 10 | variable application_owner { 11 | description = "Value of 'owner' resource tag" 12 | default = "" # Empty string takes objectId of current user 13 | } 14 | 15 | variable configure_access_control { 16 | description = "Assumes the Terraform user is an owner of the subscription. Turning this off reduces functionality somewhat" 17 | default = true 18 | type = bool 19 | } 20 | 21 | variable configure_kubernetes { 22 | type = bool 23 | default = false 24 | description = "Whether to configure Kubernetes using the Terraform Kubernetes provider" 25 | } 26 | 27 | variable deploy_aks { 28 | type = bool 29 | default = true 30 | description = "Whether to deploy AKS & Kubernetes. False will deploy network infrastructure only." 31 | } 32 | 33 | variable deploy_bastion { 34 | type = bool 35 | default = false 36 | description = "Whether to deploy managed bastion" 37 | } 38 | 39 | # Turn this off if you can't open required ports (65200-65535, ApplicationGatewaySubnetInboundTrafficBlockedByNetworkSecurityGroup) 40 | # https://docs.microsoft.com/en-us/azure/application-gateway/configuration-infrastructure#network-security-groups 41 | variable deploy_application_gateway { 42 | type = bool 43 | default = true 44 | description = "Whether to deploy Application Gateway" 45 | } 46 | 47 | variable dns_host_suffix { 48 | default = "mycicd" 49 | } 50 | 51 | variable kube_config_path { 52 | description = "Path to the kube config file (e.g. .kube/config)" 53 | default = "" 54 | } 55 | 56 | variable kubernetes_version { 57 | default = "" 58 | } 59 | 60 | variable location { 61 | description = "The location/region where the resources will be created." 62 | default = "westeurope" 63 | } 64 | 65 | variable node_size { 66 | default = "Standard_D2s_v3" 67 | } 68 | 69 | variable nsg_reassign_wait_minutes { 70 | type = number 71 | default = 0 72 | } 73 | 74 | variable peer_network_has_gateway { 75 | type = bool 76 | default = false 77 | } 78 | 79 | variable peer_network_id { 80 | description = "Virtual network to be peered with. This is usefull to run Terraform from and be able to access a private API server." 81 | default = "" 82 | } 83 | 84 | variable private_cluster_enabled { 85 | type = bool 86 | default = true 87 | } 88 | 89 | variable resource_prefix { 90 | description = "The prefix to put in front of resource names created" 91 | default = "k8s" 92 | } 93 | variable resource_suffix { 94 | description = "The suffix to put at the of resource names created" 95 | default = "" # Empty string triggers a random suffix 96 | } 97 | 98 | variable resource_environment { 99 | description = "The logical environment (tier) resource will be deployed in" 100 | default = "" # Empty string defaults to workspace name 101 | } 102 | 103 | variable run_id { 104 | description = "The ID that identifies the pipeline / workflow that invoked Terraform" 105 | default = "" 106 | } 107 | 108 | variable ssh_public_key_file { 109 | type = string 110 | default = "~/.ssh/id_rsa.pub" 111 | } 112 | 113 | variable subscription_id { 114 | description = "Configure subscription_id independent from ARM_SUBSCRIPTION_ID" 115 | default = null 116 | } 117 | variable tenant_id { 118 | description = "Configure tenant_id independent from ARM_TENANT_ID" 119 | default = null 120 | } 121 | 122 | variable workspace_location { 123 | description = "The location/region where the monitoring workspaces will be created." 124 | default = "westeurope" 125 | } 126 | -------------------------------------------------------------------------------- /visuals/aspnetapp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekzter/azure-aks/5de39f09d032c14eb1a695e02c7d3306afc493f0/visuals/aspnetapp.png -------------------------------------------------------------------------------- /visuals/diagram-blog.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekzter/azure-aks/5de39f09d032c14eb1a695e02c7d3306afc493f0/visuals/diagram-blog.png -------------------------------------------------------------------------------- /visuals/diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekzter/azure-aks/5de39f09d032c14eb1a695e02c7d3306afc493f0/visuals/diagram.png -------------------------------------------------------------------------------- /visuals/diagram.vsdx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekzter/azure-aks/5de39f09d032c14eb1a695e02c7d3306afc493f0/visuals/diagram.vsdx -------------------------------------------------------------------------------- /visuals/votingapp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekzter/azure-aks/5de39f09d032c14eb1a695e02c7d3306afc493f0/visuals/votingapp.png --------------------------------------------------------------------------------