├── .gitignore ├── LICENSE ├── README.md ├── bootstrap-rke-cluster.png ├── infra-live └── dev │ └── app-environment │ └── terragrunt.hcl ├── infra-modules └── app-environment │ ├── bastion-asg │ ├── autoscaling_group.tf │ ├── bastion_userdata.sh │ ├── iam.tf │ └── vars.tf │ ├── cluster-asg │ ├── autoscaling_group.tf │ ├── iam.tf │ ├── master_userdata.sh │ └── vars.tf │ ├── cluster-worker-asg │ ├── autoscaling_group.tf │ ├── iam.tf │ ├── vars.tf │ └── worker_userdata.sh │ ├── main.tf │ ├── variables.tf │ └── vpc │ ├── control_plane_sg.tf │ ├── data_plane_sg.tf │ ├── nat_gateway.tf │ ├── nlb.tf │ ├── output.tf │ ├── public_sg.tf │ ├── vars.tf │ └── vpc.tf ├── openSUSE-Leap-AWS.png ├── rke-aws-k8s.png ├── rke-cluster-config └── cluster.yml └── terragrunt.hcl /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | .idea 3 | *.iml 4 | **/.terragrunt-cache/* 5 | 6 | # Local .terraform directories 7 | **/.terraform/* 8 | 9 | # Terragrunt generated files 10 | backend.tf 11 | 12 | # .tfstate files 13 | *.tfstate 14 | *.tfstate.* 15 | .idea/* 16 | 17 | # .tfvars files 18 | sensitive.tfvars 19 | 20 | # Crash log files 21 | crash.log 22 | 23 | # Ignore override files as they are usually used to override resources locally and so 24 | # are not checked in 25 | override.tf 26 | override.tf.json 27 | *_override.tf 28 | *_override.tf.json 29 | 30 | # Include override files you do wish to add to version control using negated pattern 31 | # 32 | # !example_override.tf 33 | 34 | # Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan 35 | # example: *tfplan* 36 | 37 | *.pem 38 | 39 | *config.txt -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2021 Lukonde Mwila 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Bootstrap RKE Kubernetes Cluster in AWS Environment 2 | This repository contains source code to bootstrap an RKE Kubernetes cluster in AWS with a single command, provided you have met all the prerequisites and followed the initial configuration steps. 3 | 4 | ![RKE AWS K8S](./rke-aws-k8s.png) 5 | 6 | ## Architecture Overview 7 | The diagram below depicts the overall architecture for the network VPC and the Auto Scaling Groups that are used for the RKE Kubernetes cluster. 8 | 9 | ![Architecture Overview](./bootstrap-rke-cluster.png) 10 | 11 | ## Prerequisites 12 | * AWS account 13 | * AWS profile configured with CLI on local machine 14 | * [Terraform](https://www.terraform.io/downloads.html) 15 | * [Terragrunt](https://terragrunt.gruntwork.io/docs/getting-started/install/) 16 | * Understanding of how Rancher Kubernetes Engine (RKE) works. 17 | 18 | ## Project Structure 19 | ``` 20 | ├── README.md 21 | ├── infra-live 22 | | └── dev 23 | ├── infra-modules 24 | | └── app-environment 25 | ├── rke-cluster-config 26 | | └── cluster.yml 27 | ├── sensitive.tfvars (not committed to Git repo) 28 | └── terragrunt.hcl 29 | ``` 30 | 31 | ## Initial Configuration 32 | Before running the provisioning command, make sure to follow these initial steps. 33 | 34 | *This project uses Terragrunt which is a wrapper for Terraform to make Terraform code more [DRY](https://terragrunt.gruntwork.io/docs/features/keep-your-terraform-code-dry/) and easy to reuse for multiple environments. Terragrunt commands are similar to Terraform commands (i.e. instead of `terraform apply` run `terragrunt apply`).* 35 | 36 | ### RKE Cluster Configuration 37 | Create an S3 bucket that will be used to store the cluster configuration for your RKE K8s cluster. 38 | 39 | ```aws s3api create-bucket --bucket --region --create-bucket-configuration LocationConstraint=``` 40 | 41 | An example of the cluster config file can be found in the `rke-cluster-config` directory. Make sure you *DO NOT* populate the nodes property, this will be handled automatically by the scripts in the userdata during the infrastructure provisioning process. Once you have finalized your cluster.yml file, upload it to the S3 bucket you just created. 42 | 43 | ``` 44 | aws s3 cp ./cluster.yml s3://your-rke-cluster-config-bucket 45 | ``` 46 | 47 | ### SSH Key for Cluster Nodes 48 | Create an SSH key that will be used to establish connections with the node(s) in your cluster. You can do this from the AWS Console in the EC2 section. Upon creation, the *.pem* file will be automatically downloaded. You can upload this file to `your-rke-cluster-config-bucket`. 49 | 50 | ``` 51 | aws s3 cp ./ec2-ssh-key.pem s3://your-rke-cluster-config-bucket 52 | ``` 53 | 54 | *Take note of the name you give to this SSH Key because you will need to provide it as a variable value in the sensitive.tfvars file later.* 55 | 56 | ### Subscribe to openSUSE Leap in AWS Marketplace 57 | The nodes in this project are provision with openSUSE Leap 15.2. In order to use this OS, you need to subscribe to it [here](https://aws.amazon.com/marketplace/pp/prodview-wn2xje27ui45o). You will need to be logged into the AWS account that you will use to provision this infrastructure. It is available for the *free tier* in AWS. 58 | 59 | ![openSUSE Leap in AWS Marketplace](./openSUSE-Leap-AWS.png) 60 | 61 | ### Remote Backend State Configuration 62 | To configure remote backend state for your infrastructure, create an S3 bucket and DynamoDB table before running *terragrunt init*. You can populate the parent terragrunt configuration file (terragrunt.hcl) in the root directory with your respective values for the bucket name (your-terraform-state-bucket) and table name (your-terraform-lock-table). In this same config file, you can update the region and aws profile name. 63 | 64 | #### Create S3 Bucket for State Backend 65 | ```aws s3api create-bucket --bucket --region --create-bucket-configuration LocationConstraint=``` 66 | 67 | #### Create DynamoDB table for State Locking 68 | ```aws dynamodb create-table --table-name --attribute-definitions AttributeName=LockID,AttributeType=S --key-schema AttributeName=LockID,KeyType=HASH --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1``` 69 | 70 | ### Create `sensitive.tfvars` file 71 | Your child terragrunt conig files (`infra-life/dev/app-environment/terragrunt.hcl`) is set up to read any sensitive variable values from a file called `sensitive.tfvars` at the root directory. This file is not committed to the Git repo. The contents of the file can be as follows: 72 | ``` 73 | profile="k8s-deployer" 74 | region="eu-west-1" 75 | cluster_name="my-rke-cluster" 76 | key_name="my-ec2-ssh-key" 77 | ``` 78 | 79 | ## Provision Infrastructure & RKE Cluster 80 | Once you've completed all the above steps in the initial configuration, redirect to the `infra-live/dev/app-environment` directory and run the following commands: 81 | 82 | ``` 83 | terragrunt init 84 | terragrunt plan 85 | terragrunt apply -auto-approve 86 | ``` 87 | 88 | To create the infrastructure for separate environments, create separate folders in `infra-live` such as `infra-live/test/app-environment` and `infra-live/prod/app-environment` with the same terragrunt.tf file located in `infra-live/dev/app-environment`. 89 | 90 | Modify the Terragrunt config file input block for other environments to the relevant name: 91 | ### Dev 92 | ``` 93 | inputs = { 94 | environment = "dev" 95 | } 96 | ``` 97 | ### Test 98 | ``` 99 | inputs = { 100 | environment = "test" 101 | } 102 | ``` 103 | ### Prod 104 | ``` 105 | inputs = { 106 | environment = "prod" 107 | } 108 | ``` 109 | 110 | *Remember to update the secret name when creating the infrastructure for additional environments.* 111 | 112 | To destroy the infrastructure run the `terrargunt destroy` command from the same location. 113 | 114 | ## How Does It All Work? 115 | ### Automated Steps 116 | 1. VPC is provisioned by Terraform. 117 | 2. Control plane node(s) is provisioned by Terraform. The userdata script prepares the node(s) (i.e. software installation and node config steps), downloads the cluster.yml file from your bucket and updates it with the node details for each node in the control plane, and then uploads it back to the bucket. 118 | 3. Worker node(s) is provisioned by Terraform. The userdata script prepares the nodes similar to control plane nodes, downloads the cluster.yml file from your bucket and updates it with the node details for each node in the worker node group, and then uploads it back to the bucket. 119 | 4. Bastion host(s) is provisioned by Terraform. The userdata script installs the AWS CLI, RKE CLI tool and kubectl. The cluster.yml file and SSH key for the nodes are downloaded to the bastion host. The RKE provisioning command is run referencing the cluster.yml file and the SSH key locally stored on the machine. 120 | 5. Once the RKE cluster has been provisioned, the cluster config details are saved in Secrets Manager. 121 | ### Manual Steps 122 | 1. SSH into your bastion host using the relevant key. 123 | 2. Copy the RKE cluster config details to the bastion hosts kube config directory from Secrets Manager 124 | ``` 125 | mkdir -p $HOME/.kube 126 | 127 | KUBE_CONFIG=$(sudo aws secretsmanager get-secret-value --secret-id rkekubeconfig --version-stage AWSCURRENT --region eu-west-1 | jq -r .SecretString) 128 | 129 | sudo tee $HOME/.kube/config<