├── .github └── FUNDING.yml ├── README.0.11.md ├── README.md ├── errors.md ├── lambda ├── .gitignore ├── lambda.tf ├── pip.sh └── source │ ├── .gitignore │ ├── README.md │ ├── main.py │ ├── requirements.txt │ └── setup.cfg └── s3-backend ├── .gitignore ├── README.md ├── main.tf └── outputs.tf /.github/FUNDING.yml: -------------------------------------------------------------------------------- 1 | github: [ozbillwang] 2 | custom: ["https://www.buymeacoffee.com/ozbillwang", "https://github.com/sponsors/ozbillwang"] 3 | -------------------------------------------------------------------------------- /README.0.11.md: -------------------------------------------------------------------------------- 1 | # terraform-best-practices 2 | 3 | Terraform Best Practices for AWS users. 4 | 5 | 6 | 7 | 8 | - [Run terraform command with var-file](#run-terraform-command-with-var-file) 9 | - [Manage s3 backend for tfstate files](#manage-s3-backend-for-tfstate-files) 10 | - [Notes](#notes) 11 | - [Manage multiple Terraform modules and environments easily with Terragrunt](#manage-multiple-terraform-modules-and-environments-easily-with-terragrunt) 12 | - [Retrieve state meta data from a remote backend](#retrieve-state-meta-data-from-a-remote-backend) 13 | - [Turn on debug when you need to do troubleshooting.](#turn-on-debug-when-you-need-to-do-troubleshooting) 14 | - [Use shared modules](#use-shared-modules) 15 | - [Notes](#notes-1) 16 | - [Isolate environment](#isolate-environment) 17 | - [Use terraform import to include as many resources as you can](#use-terraform-import-to-include-as-many-resources-as-you-can) 18 | - [Avoid hard coding the resources](#avoid-hard-coding-the-resources) 19 | - [Format terraform code](#format-terraform-code) 20 | - [Enable version control on terraform state files bucket](#enable-version-control-on-terraform-state-files-bucket) 21 | - [Generate README for each module with input and output variables](#generate-readme-for-each-module-with-input-and-output-variables) 22 | - [Update terraform version](#update-terraform-version) 23 | - [Run terraform from docker container](#run-terraform-from-docker-container) 24 | - [Troubleshooting with messy output - (Decommissioned)](#troubleshooting-with-messy-output---decommissioned) 25 | - [Run test](#run-test) 26 | - [Quick start](#quick-start) 27 | - [Run test within docker container](#run-test-within-docker-container) 28 | - [Minimum AWS permissions necessary for a Terraform run](#minimum-aws-permissions-necessary-for-a-terraform-run) 29 | - [Tips to deal with lambda functions](#tips-to-deal-with-lambda-functions) 30 | - [Explanation](#explanation) 31 | - [Notes](#notes-2) 32 | - [Usage of variable "self"](#usage-of-variable-self) 33 | - [One more use case](#one-more-use-case) 34 | - [Use pre-installed Terraform plugins](#use-pre-installed-terraform-plugins) 35 | - [Useful documents you should read](#useful-documents-you-should-read) 36 | 37 | 38 | 39 | ## Run terraform command with var-file 40 | 41 | ``` 42 | $ cat config/dev.tfvars 43 | 44 | name = "dev-stack" 45 | s3_terraform_bucket = "dev-stack-terraform" 46 | tag_team_name = "hello-world" 47 | 48 | $ terraform plan -var-file=config/dev.tfvars 49 | ``` 50 | 51 | With `var-file`, you can easily manage environment (dev/stag/uat/prod) variables. 52 | 53 | With `var-file`, you avoid running terraform with long list of key-value pairs ( `-var foo=bar` ) 54 | 55 | ## Manage s3 backend for tfstate files 56 | 57 | Terraform doesn't support [Interpolated variables in terraform backend config](https://github.com/hashicorp/terraform/pull/12067), normally you write a separate script to define s3 backend bucket name for different environments, but I recommend to hard code it directly as below. 58 | 59 | Add below code in terraform configuration files. 60 | 61 | ``` 62 | $ cat main.tf 63 | 64 | terraform { 65 | required_version = "~> 0.10" 66 | 67 | backend "s3" { 68 | encrypt = true 69 | } 70 | } 71 | ``` 72 | 73 | Define backend variables for particular environment 74 | 75 | ``` 76 | $ cat config/backend-dev.conf 77 | bucket = "-terraform-development" 78 | key = "development/service-1.tfstate" 79 | encrypt = true 80 | region = "ap-southeast-2" 81 | kms_key_id = "alias/terraform" 82 | dynamodb_table = "terraform-lock" 83 | ``` 84 | 85 | ### Notes 86 | 87 | - bucket - s3 bucket name, has to be globally unique. 88 | - key - Set some meaningful names for different services and applications, such as vpc.tfstate, application_name.tfstate, etc 89 | - dynamodb_table - optional when you want to enable [State Locking](https://www.terraform.io/docs/state/locking.html) 90 | 91 | After you set `config/backend-dev.conf` and `config/dev.tfvars` properly (for each environment). You can easily run terraform as below: 92 | 93 | ```bash 94 | env=dev 95 | terraform get -update=true 96 | terraform init -backend-config=config/backend-${env}.conf 97 | terraform plan -var-file=config/${env}.tfvars 98 | terraform apply -var-file=config/${env}.tfvars 99 | ``` 100 | 101 | ## Manage multiple Terraform modules and environments easily with Terragrunt 102 | 103 | Terragrunt is a thin wrapper for Terraform that provides extra tools for working with multiple Terraform modules. https://www.gruntwork.io 104 | 105 | Sample for reference: https://github.com/gruntwork-io/terragrunt-infrastructure-live-example 106 | 107 | Its README is too long, if you need a quick start, follow below steps: 108 | 109 | ```bash 110 | # Install terraform and terragrunt 111 | # Make sure you are in right aws account 112 | $ aws s3 ls 113 | # use terragrunt to deploy 114 | $ git clone https://github.com/gruntwork-io/terragrunt-infrastructure-live-example.git 115 | $ cd terragrunt-infrastructure-live-example 116 | # for example, you want to deploy mysql in stage non-prod at region us-east-1 117 | $ cd non-prod/us-east-1/stage/mysql 118 | $ terragrunt plan 119 | # Confirm everything works 120 | $ terragrunt apply 121 | ``` 122 | 123 | So if you followed the setting in terragrunt properly, you don't need to care about the backend state files and variable file path in different environments, even more, you can run `terragrunt plan-all` to plan all modules together. 124 | 125 | ## Retrieve state meta data from a remote backend 126 | 127 | Normally we have several layers to manage terraform resources, such as network, database, and application layers. After you create the basic network resources, such as vpc, security group, subnets, nat gateway in vpc stack. Your database layer and applications layer should always refer the resource from vpc layer directly via `terraform_remote_state` data source. 128 | 129 | ```terraform 130 | data "terraform_remote_state" "vpc" { 131 | backend = "s3" 132 | config{ 133 | bucket = "${var.s3_terraform_bucket}" 134 | key = "${var.environment}/vpc.tfstate" 135 | region="${var.aws_region}" 136 | } 137 | } 138 | 139 | # Retrieves the vpc_id and subnet_ids directly from remote backend state files. 140 | resource "aws_xx_xxxx" "main" { 141 | # ... 142 | subnet_ids = "${split(",", data.terraform_remote_state.vpc.data_subnets)}" 143 | vpc_id = "${data.terraform_remote_state.vpc.vpc_id}" 144 | } 145 | ``` 146 | 147 | ## Turn on debug when you need to do troubleshooting. 148 | 149 | ```bash 150 | TF_LOG=DEBUG terraform 151 | 152 | # or if you run with terragrunt 153 | TF_LOG=DEBUG terragrunt 154 | ``` 155 | 156 | ## Use shared modules 157 | 158 | Manage terraform resource with shared modules, this will save a lot of coding time. No need to re-invent the wheel! 159 | 160 | You can start from below links: 161 | 162 | [terraform module usage](https://www.terraform.io/docs/modules/usage.html) 163 | 164 | [Terraform Module Registry](https://registry.terraform.io/) 165 | 166 | [Terraform aws modules](https://github.com/terraform-aws-modules) 167 | 168 | ### Notes 169 | 170 | terraform modules don't support `count` parameter currently. You can follow up this ticket for updates: https://github.com/hashicorp/terraform/issues/953 171 | 172 | ## Isolate environment 173 | 174 | Sometimes, developers like to create a security group and share it to all non-prod (dev/qa) environments. Don't do that, create resources with different name for each environment and each resource. 175 | 176 | ```terraform 177 | variable "application" { 178 | description = "application name" 179 | default = "" 180 | } 181 | 182 | variable "environment" { 183 | description = "environment name" 184 | } 185 | 186 | locals { 187 | name_prefix = "${var.application}-${var.environment}" 188 | } 189 | 190 | resource "" { 191 | name = "${local.name_prefix}-" 192 | # ... 193 | } 194 | ``` 195 | 196 | With that, you will easily define the resource with a meaningful and unique name, and you can build more of the same application stack for different developers without change a lot. For example, you update the environment to dev, staging, uat, prod, etc. 197 | 198 | > Tips: some aws resource names have length limits, such as less than 24 characters, so when you define variables of application and environment name, use short name. 199 | 200 | ## Use terraform import to include as many resources as you can 201 | 202 | Sometimes developers manually created resources. You need to mark these resource and use `terraform import` to include them in codes. 203 | 204 | [terraform import](https://www.terraform.io/docs/import/usage.html) 205 | 206 | ## Avoid hard coding the resources 207 | 208 | A sample: 209 | 210 | ``` 211 | account_number="123456789012" 212 | account_alias="mycompany" 213 | region="us-east-2" 214 | ``` 215 | 216 | The current aws account id, account alias and current region can be input directly via [data sources](https://www.terraform.io/docs/providers/aws/). 217 | 218 | ```terraform 219 | # The attribute `${data.aws_caller_identity.current.account_id}` will be current account number. 220 | data "aws_caller_identity" "current" {} 221 | 222 | # The attribue `${data.aws_iam_account_alias.current.account_alias}` will be current account alias 223 | data "aws_iam_account_alias" "current" {} 224 | 225 | # The attribute `${data.aws_region.current.name}` will be current region 226 | data "aws_region" "current" {} 227 | 228 | # Set as [local values](https://www.terraform.io/docs/configuration/locals.html) 229 | locals { 230 | account_id = "${data.aws_caller_identity.current.account_id}" 231 | account_alias = "${data.aws_iam_account_alias.current.account_alias}" 232 | region = "${data.aws_region.current.name}" 233 | } 234 | ``` 235 | 236 | ## Format terraform code 237 | 238 | Always run `terraform fmt` to format terraform configuration files and make them neat. 239 | 240 | I used below code in Travis CI pipeline (you can re-use it in any pipelines) to validate and format check the codes before you can merge it to master branch. 241 | 242 | - find . -type f -name "*.tf" -exec dirname {} \;|sort -u | while read m; do (terraform validate -check-variables=false "$m" && echo "√ $m") || exit 1 ; done 243 | - terraform fmt -check=true -write=false -diff=true 244 | 245 | ## Enable version control on terraform state files bucket 246 | 247 | Always set backend to s3 and enable version control on this bucket. 248 | 249 | If you'd like to manage terraform state bucket as well, I recommend using this repostory I wrote [tf_aws_tfstate_bucket](https://github.com/BWITS/tf_aws_tfstate_bucket) to create the bucket and replicate to other regions automatically. 250 | 251 | ## Generate README for each module with input and output variables 252 | 253 | You needn't manually manage `USAGE` about input variables and outputs. [terraform-docs](https://github.com/segmentio/terraform-docs) can do this job automatically. 254 | 255 | ```bash 256 | $ brew install terraform-docs 257 | $ cd terraform/modules/vpc 258 | $ terraform-docs md . > README.md 259 | ``` 260 | 261 | For details on how to run `terraform-docs`, check this repository: https://github.com/segmentio/terraform-docs 262 | 263 | There is a simple sample for you to start [tf_aws_acme](https://github.com/BWITS/tf_aws_acme), the README is generatd by `terraform-docs` 264 | 265 | ## Update terraform version 266 | 267 | Hashicorp doesn't have a good qa/build/release process for their software and does not follow semantic versioning rules. 268 | 269 | For example, `terraform init` isn't compatible between 0.9 and 0.8. Now they are going to split providers and use "init" to install providers as plugin in coming version 0.10 270 | 271 | So recommend to keep updating to latest terraform version 272 | 273 | ## Run terraform from docker container 274 | 275 | Terraform releases official docker containers that you can easily control which version you can run. 276 | 277 | Recommend to run terraform docker container, when you set your build job in CI/CD pipeline. 278 | 279 | ```bash 280 | TERRAFORM_IMAGE=hashicorp/terraform:0.9.8 281 | TERRAFORM_CMD="docker run -ti --rm -w /app -v ${HOME}/.aws:/root/.aws -v ${HOME}/.ssh:/root/.ssh -v `pwd`:/app -w /app ${TERRAFORM_IMAGE}" 282 | ${TERRAFORM_CMD} init 283 | ${TERRAFORM_CMD} plan 284 | ``` 285 | 286 | ## Troubleshooting with messy output - (Decommissioned) 287 | 288 | > (Decommissioned) after terraform v0.12.x, we needn't run with `terraform-landscape` any more. The new terraform output looks nice already. 289 | 290 | Sometime, you applied the changes several times, the plan output always prompts there are some changes, essepecially in iam and s3 policy. It is hard to troubleshooting the problem with messy json output in one line. 291 | 292 | With the tool [terraform-landscape](https://github.com/coinbase/terraform-landscape), it improves Terraform plan output to be easier for reading, you can easily find out where is the problem. For details, please go through the project at https://github.com/coinbase/terraform-landscape 293 | 294 | # Install terraform_landscape 295 | gem install terraform_landscape 296 | # On MacOS, you can install with brew 297 | brew install terraform_landscape 298 | 299 | terraform plan -var-file=${env}/${env}.tfvars -input=false -out=plan -lock=false |tee report 300 | landscape < report 301 | 302 | # run terraform-landscape container as command 303 | alias landscape="docker run -i --rm -v $(pwd):/apps alpine/landscape:0.1.18" 304 | landscape --help 305 | terraform plan |tee report 306 | landscape - < report 307 | # Or 308 | terraform plan | landscape - 309 | 310 | Another quick way to handle the messy output is to run below command, if you manually copy the part of messy output to a file 311 | 312 | cat output.txt | grep -Ev '"([^"]*)" => "\1"' 313 | 314 | ## Run test 315 | 316 | Recommend to add [awspec](https://github.com/k1LoW/awspec) tests through [kitchen](https://kitchen.ci/) and [kitchen-terraform](https://newcontext-oss.github.io/kitchen-terraform/). 317 | 318 | ### Quick start 319 | 320 | Reference: repo [terraform-aws-modules/terraform-aws-eks](https://github.com/terraform-aws-modules/terraform-aws-eks#testing) 321 | 322 | ### Run test within docker container 323 | 324 | Reference: [README for terraform awspec container](https://github.com/alpine-docker/bundle-terraform-awspec) 325 | 326 | ## Minimum AWS permissions necessary for a Terraform run 327 | 328 | There will be no answer for this. But with below iam policy you can easily get started. 329 | 330 | ```json 331 | { 332 | "Version": "2012-10-17", 333 | "Statement": [ 334 | { 335 | "Sid": "AllowSpecifics", 336 | "Action": [ 337 | "lambda:*", 338 | "apigateway:*", 339 | "ec2:*", 340 | "rds:*", 341 | "s3:*", 342 | "sns:*", 343 | "states:*", 344 | "ssm:*", 345 | "sqs:*", 346 | "iam:*", 347 | "elasticloadbalancing:*", 348 | "autoscaling:*", 349 | "cloudwatch:*", 350 | "cloudfront:*", 351 | "route53:*", 352 | "ecr:*", 353 | "logs:*", 354 | "ecs:*", 355 | "application-autoscaling:*", 356 | "logs:*", 357 | "events:*", 358 | "elasticache:*", 359 | "es:*", 360 | "kms:*", 361 | "dynamodb:*" 362 | ], 363 | "Effect": "Allow", 364 | "Resource": "*" 365 | }, 366 | { 367 | "Sid": "DenySpecifics", 368 | "Action": [ 369 | "iam:*User*", 370 | "iam:*Login*", 371 | "iam:*Group*", 372 | "iam:*Provider*", 373 | "aws-portal:*", 374 | "budgets:*", 375 | "config:*", 376 | "directconnect:*", 377 | "aws-marketplace:*", 378 | "aws-marketplace-management:*", 379 | "ec2:*ReservedInstances*" 380 | ], 381 | "Effect": "Deny", 382 | "Resource": "*" 383 | } 384 | ] 385 | } 386 | ``` 387 | 388 | Depending on your company or project requirement, you can easily update the resources in `Allow` section which terraform commands should have, and add deny policies in `Deny` section if some of permissions are not required. 389 | 390 | ## Tips to deal with lambda functions 391 | 392 | Headache to save python packages from `pip install` into source codes and generate lambda zip file manually? Here is full codes with solution. 393 | 394 | The folder [lambda](./lambda) includes all codes, here is the explanation. 395 | 396 | ``` 397 | $ tree 398 | . 399 | ├── lambda.tf # terraform HCL to deal with lambda 400 | ├── pip.sh # script to install python packages with pip. 401 | └── source 402 | ├── .gitignore # Ignore all other files 403 | ├── main.py # Lambda function, replace with yours 404 | ├── requirements.txt # python package list, replace with yours. 405 | └── setup.cfg # Useful for mac users who installed python using Homebrew 406 | ``` 407 | 408 | Replace `main.py` and `requirements.txt` with your applications. 409 | 410 | ### Explanation 411 | 412 | After you run `terraform apply`, it will: 413 | 414 | 1. install all pip packages into source folder 415 | 2. zip the source folder to `source.zip` 416 | 3. deploy lambda function with `source.zip` 417 | 4. because of `source/.gitignore`, it will ignore all new installed pip packages in git source codes. 418 | 419 | This solution is reference from the comments in [Ability to zip AWS Lambda function on the fly](https://github.com/hashicorp/terraform/issues/8344#issuecomment-345807204)) 420 | 421 | You should be fine to do the same for lambda functions using nodejs (`npm install`) or other languages with this tip. 422 | 423 | ### Notes 424 | 425 | You need have python/pip installed when run terraform commands, if you run in terraform container, make sure you install python/pip in it. 426 | 427 | ## Usage of variable "self" 428 | 429 | Quote from terraform documents: 430 | 431 | > Attributes of your own resource 432 | 433 | > The syntax is self.ATTRIBUTE. For example ${self.private_ip} will interpolate that resource's private IP address. 434 | 435 | > Note: The self.ATTRIBUTE syntax is only allowed and valid within provisioners. 436 | 437 | ### One more use case 438 | 439 | ```terraform 440 | resource "aws_ecr_repository" "jenkins" { 441 | name = "${var.image_name}" 442 | provisioner "local-exec" { 443 | command = "./deploy-image.sh ${self.repository_url} ${var.jenkins_image_name}" 444 | } 445 | } 446 | 447 | variable "jenkins_image_name" { 448 | default = "mycompany/jenkins" 449 | description = "Jenkins image name." 450 | } 451 | ``` 452 | 453 | You can easily define ecr image url (`.dkr.ecr..amazonaws.com/`) with ${self.repository_url} 454 | 455 | Any attributes in this resource can be self referenced by this way. 456 | 457 | Reference: https://github.com/shuaibiyy/terraform-ecs-jenkins/blob/master/docker/main.tf 458 | 459 | ## Use pre-installed Terraform plugins 460 | 461 | There is a way to use pre-installed Terraform plugins instead of downloading them with `terraform init`, the accepted answer below gives the detail: 462 | 463 | [Use pre-installed Terraform plugins instead of downloading them with terraform init](https://stackoverflow.com/questions/50944395/use-pre-installed-terraform-plugins-instead-of-downloading-them-with-terraform-i?rq=1) 464 | 465 | ## Useful documents you should read 466 | 467 | [terraform tips & tricks: loops, if-statements, and gotchas](https://blog.gruntwork.io/terraform-tips-tricks-loops-if-statements-and-gotchas-f739bbae55f9) 468 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Terraform Best Practices 🌐 2 | 3 | Terraform Best Practices for AWS users. 4 | 5 | 6 | 7 | 8 | - [Run terraform command with var-file](#run-terraform-command-with-var-file) 9 | - [Enable version control on terraform state files bucket](#enable-version-control-on-terraform-state-files-bucket) 10 | - [Manage S3 backend for tfstate files](#manage-s3-backend-for-tfstate-files) 11 | - [Notes on S3](#notes-on-s3) 12 | - [Manage multiple Terraform modules and environments easily with Terragrunt](#manage-multiple-terraform-modules-and-environments-easily-with-terragrunt) 13 | - [layers](#layers) 14 | - [Retrieve state meta data from a remote backend](#retrieve-state-meta-data-from-a-remote-backend) 15 | - [When troubleshooting, remember to enable debugging](#when-troubleshooting-remember-to-enable-debugging) 16 | - [re-use terraform modules to save your coding time](#re-use-terraform-modules-to-save-your-coding-time) 17 | - [Environment Isolation](#environment-isolation) 18 | - [Use terraform import to include as many resources as you can](#use-terraform-import-to-include-as-many-resources-as-you-can) 19 | - [Avoid hard coding the resources](#avoid-hard-coding-the-resources) 20 | - [Validate and format terraform code](#validate-and-format-terraform-code) 21 | - [Generate README for each module with input and output variables](#generate-readme-for-each-module-with-input-and-output-variables) 22 | - [Update terraform version](#update-terraform-version) 23 | - [Efficient Workspace Management with workspace sub-command](#efficient-workspace-management-with-workspace-sub-command) 24 | - [Terraform version manager](#terraform-version-manager) 25 | - [Run terraform in docker container](#run-terraform-in-docker-container) 26 | - [Run test](#run-test) 27 | - [Minimum AWS permissions necessary for a Terraform run](#minimum-aws-permissions-necessary-for-a-terraform-run) 28 | - [Usage of variable "self"](#usage-of-variable-self) 29 | - [One more use case](#one-more-use-case) 30 | - [Use pre-installed Terraform plugins](#use-pre-installed-terraform-plugins) 31 | - [Tips to upgrade to terraform 0.12](#tips-to-upgrade-to-terraform-012) 32 | - [Tips to upgrade to terraform 0.13+](#tips-to-upgrade-to-terraform-013) 33 | - [Contributing](#contributing) 34 | - [Useful terraform modules](#useful-terraform-modules) 35 | 36 | 37 | 38 | > The README for terraform version 0.11 and less has been renamed to [README.0.11.md](README.0.11.md) 39 | 40 | ## Run terraform command with var-file 41 | 42 | ```bash 43 | $ cat config/dev.tfvars 44 | 45 | name = "dev-stack" 46 | s3_terraform_bucket = "dev-stack-terraform" 47 | tag_team_name = "hello-world" 48 | 49 | $ terraform plan -var-file=config/dev.tfvars 50 | ``` 51 | 52 | With `var-file`, you can easily manage environment (dev/stag/uat/prod) variables. 53 | 54 | With `var-file`, you avoid running terraform with long list of key-value pairs ( `-var foo=bar` ) 55 | 56 | ## Enable version control on terraform state files bucket 57 | 58 | Always set backend to s3 and enable version control on this bucket. 59 | 60 | [s3-backend](s3-backend) to create s3 bucket and dynamodb table to use as terraform backend. 61 | 62 | ## Manage S3 backend for tfstate files 63 | 64 | Terraform doesn't support [Interpolated variables in terraform backend config](https://github.com/hashicorp/terraform/pull/12067), normally you write a separate script to define s3 backend bucket name for different environments, but I recommend to hard code it directly as below. This way is called [partial configuration](https://www.terraform.io/docs/backends/config.html#partial-configuration). 65 | 66 | Add below code in terraform configuration files. 67 | 68 | ```bash 69 | $ cat main.tf 70 | 71 | terraform { 72 | backend "s3" { 73 | encrypt = true 74 | } 75 | } 76 | ``` 77 | 78 | Define backend variables for particular environment 79 | 80 | ```bash 81 | $ cat config/backend-dev.conf 82 | bucket = "-terraform-states" 83 | key = "development/service-name.tfstate" 84 | encrypt = true 85 | region = "ap-southeast-2" 86 | #dynamodb_table = "terraform-lock" 87 | ``` 88 | 89 | ### Notes on S3 90 | 91 | - `bucket` - existing s3 bucket name. Tips: The s3 bucket has to be globally unique, normally I put account id in its name. 92 | - `key` - Set some meaningful names for different services and applications, such as vpc.tfstate, .tfstate, etc 93 | - `dynamodb_table` - optional when you want to enable [State Locking](https://www.terraform.io/docs/state/locking.html) 94 | 95 | After you set `config/backend-dev.conf` and `config/dev.tfvars` properly (for each environment). You can easily run terraform as below: 96 | 97 | ```bash 98 | env=dev 99 | terraform get -update=true 100 | terraform init -reconfigure -backend-config=config/backend-${env}.conf 101 | terraform fmt 102 | terraform validate 103 | terraform plan -var-file=config/${env}.tfvars -out='planfile' 104 | 105 | # if above dry-run is fine, run below command to apply the change. 106 | # terraform apply 'planfile' 107 | ``` 108 | 109 | If you encountered any unexpected issues, delete the cache folder, and try again. 110 | 111 | ```bash 112 | rm -rf .terraform 113 | ``` 114 | 115 | ## Manage multiple Terraform modules and environments easily with Terragrunt 116 | 117 | Terragrunt is a thin wrapper for Terraform that provides extra tools for working with multiple Terraform modules. 118 | 119 | Sample for reference: 120 | 121 | Its README is too long, if you need a quick start, follow below steps: 122 | 123 | ```bash 124 | # Install terraform and terragrunt 125 | # Make sure you are in right aws account 126 | $ aws s3 ls 127 | # use terragrunt to deploy 128 | $ git clone https://github.com/gruntwork-io/terragrunt-infrastructure-live-example.git 129 | $ cd terragrunt-infrastructure-live-example 130 | # for example, you want to deploy mysql in stage non-prod at region us-east-1 131 | $ cd non-prod/us-east-1/stage/mysql 132 | $ terragrunt plan 133 | # Confirm everything works 134 | $ terragrunt apply 135 | ``` 136 | 137 | So if you followed the setting in terragrunt properly, you don't need to care about the backend state files and variable file path in different environments, even more, you can run `terragrunt plan-all` to plan all modules together. 138 | 139 | ## layers 140 | 141 | Avoid consolidating everything into a single Terraform module, such as combining network creation with application services. The best practice is to separate them into different modules: 142 | 143 | * VPC with subnets, gateways, etc 144 | * App-1 and related resources 145 | * App-2 and related resources 146 | * Database-1 147 | 148 | ## Retrieve state meta data from a remote backend 149 | 150 | Typically, we have several layers for managing Terraform resources, including the network, database, and application layers. Once you've created the essential network resources, like VPC, security groups, subnets, and NAT gateways in the VPC stack, your database and application layers should always reference these resources directly using data source [terraform_remote_state](https://developer.hashicorp.com/terraform/language/state/remote-state-data) . 151 | 152 | > Note: Starting from Terraform version 0.12 and beyond, you must include additional outputs to reference the attributes; otherwise, you will receive an error message [Unsupported attribute](https://github.com/hashicorp/terraform/issues/21442) 153 | 154 | ```terraform 155 | data "terraform_remote_state" "this" { 156 | backend = "s3" 157 | config = { 158 | bucket = var.s3_terraform_bucket 159 | key = "${var.environment}/vpc.tfstate" 160 | region = var.aws_region 161 | } 162 | } 163 | 164 | # Retrieves the vpc_id and subnet_ids directly from remote state files from backend s3 bucket. 165 | resource "aws_xx_xxxx" "this" { 166 | # ... 167 | subnet_ids = split(",", data.terraform_remote_state.vpc.outputs.data_subnets) 168 | vpc_id = data.terraform_remote_state.this.outputs.vpc_id 169 | } 170 | ``` 171 | 172 | ## When troubleshooting, remember to enable debugging 173 | 174 | ```bash 175 | TF_LOG=DEBUG terraform 176 | 177 | # or if you run with terragrunt 178 | TF_LOG=DEBUG terragrunt 179 | ``` 180 | 181 | ## re-use terraform modules to save your coding time 182 | 183 | Compare to AWS Cloudformation template (CFN), managing Terraform resources with shared modules is one of the best features in Terraform. This approach saves a significant amount of coding time, eliminating the need to reinvent the wheel! 184 | 185 | You can start from below links: 186 | 187 | - [Terraform module usage](https://www.terraform.io/docs/modules/usage.html) 188 | - [Terraform Module Registry](https://registry.terraform.io/) 189 | - [Terraform aws modules](https://github.com/terraform-aws-modules) 190 | 191 | ## Environment Isolation 192 | 193 | At times, developers may consider creating a security group and sharing it across all non-production (dev/staging/qa) environments. However, it's advisable not to do so. Instead, create distinct resources with unique names for each environment and for each resource 194 | 195 | ```terraform 196 | variable "application" { 197 | description = "application name" 198 | default = "" 199 | } 200 | 201 | variable "environment" { 202 | description = "environment name" 203 | default = "" 204 | } 205 | 206 | locals { 207 | name_prefix = "${var.application}-${var.environment}" 208 | } 209 | 210 | resource "" "this" { 211 | name = "${local.name_prefix}-" 212 | # ... 213 | } 214 | ``` 215 | 216 | By doing so, you can effortlessly define resources with meaningful and distinct names, and you can replicate the same application stack with minimal changes. For instance, you can update the environment to accommodate development, staging, user acceptance testing (UAT), production, and more." 217 | 218 | > Tip: Keep in mind that certain resources have name length restrictions, often less than 24 characters. When defining variables for application and environment names, opt for short names, ideally between 3 to 4 letters. 219 | 220 | ## Use terraform import to include as many resources as you can 221 | 222 | Utilize [terraform import](https://www.terraform.io/docs/import/usage.html) to incorporate as many resources as possible into your Terraform configuration. Occasionally, developers may already manually create resources, and it's essential to identify these resources and bring them into your codebase using the `terraform` import command. 223 | 224 | ## Avoid hard coding the resources 225 | 226 | A sample: 227 | 228 | ``` 229 | account_number="123456789012" 230 | account_alias="mycompany" 231 | region="us-east-2" 232 | ``` 233 | 234 | The current aws account id, account alias and current region can be generated by [data sources](https://www.terraform.io/docs/providers/aws/). 235 | 236 | ```terraform 237 | # The attribute `${data.aws_caller_identity.this.account_id}` will be current account number. 238 | data "aws_caller_identity" "this" {} 239 | 240 | # The attribue `${data.aws_iam_account_alias.this.account_alias}` will be current account alias 241 | data "aws_iam_account_alias" "this" {} 242 | 243 | # The attribute `${data.aws_region.this.name}` will be current region 244 | data "aws_region" "this" {} 245 | 246 | # Set as [local values](https://www.terraform.io/docs/configuration/locals.html) 247 | locals { 248 | account_id = data.aws_caller_identity.this.account_id 249 | account_alias = data.aws_iam_account_alias.this.account_alias 250 | region = data.aws_region.this.name 251 | } 252 | ``` 253 | Now, you are fine to reference them with varaibles: 254 | 255 | * local.account_id 256 | * local.account_alias 257 | * local.region 258 | 259 | ## Validate and format terraform code 260 | 261 | Always run `terraform fmt` to format terraform configuration files and make them neat before commit the codes. 262 | 263 | I used below code in pipeline to validate the codes before you can merge it to master branch. 264 | 265 | ```yml 266 | script: 267 | - terraform init -reconfigure 268 | - terraform validate 269 | ``` 270 | 271 | One more check [tflint](https://github.com/wata727/tflint) you can add 272 | 273 | ```yml 274 | - find . -type f -name "*.tf" -exec dirname {} \;|sort -u |while read line; do pushd $line; docker run --rm -v $(pwd):/data -t wata727/tflint; popd; done 275 | ``` 276 | 277 | ## Generate README for each module with input and output variables 278 | 279 | You don't have to manually handle the documentation for input and output variables. A tool called [terraform-docs])https://github.com/terraform-docs/terraform-docs) can automate this task for you 280 | 281 | Sample command with `docker run` (so you don't have to install it directly) 282 | 283 | ```bash 284 | docker run --rm -v $(pwd):/data cytopia/terraform-docs terraform-docs md . > README.md 285 | ``` 286 | 287 | For details on how to run `terraform-docs`, check this repository: 288 | 289 | There is a simple sample for you to start [tf_aws_acme](https://github.com/BWITS/tf_aws_acme), the README is generated by `terraform-docs` 290 | 291 | ## Update terraform version 292 | 293 | It's advisable to stay updated with the latest Terraform versions. 294 | 295 | ## Efficient Workspace Management with workspace sub-command 296 | The `terraform workspace select -or-create` command simplifies workspace management by either selecting an existing workspace or creating a new one if it doesn’t exist. Use it like this: 297 | 298 | ``` 299 | terraform workspace select -or-create 300 | ``` 301 | This ensures you're always working in the correct workspace. After running the command, verify your active workspace with: 302 | 303 | ``` 304 | terraform workspace show 305 | ``` 306 | 307 | This command helps keep your Terraform environment organized and prevents accidental changes in the wrong environment. 308 | 309 | ## Terraform version manager 310 | 311 | You can manage multiple terraform versions with [tfenv](https://github.com/tfutils/tfenv) 312 | 313 | sample commands for mac users. 314 | 315 | ```bash 316 | # install tfenv 317 | $ brew install tfenv 318 | 319 | # install several terraform binary with different versions 320 | $ tfenv install 1.1.9 321 | $ tfenv install 1.2.1 322 | $ tfenv install 0.12.11 323 | 324 | # list terraform versions managed by tfenv 325 | $ terraform list 326 | 327 | # set the default terraform version 328 | $ terraform use 1.2.1 329 | $ terraform version 330 | ``` 331 | 332 | ## Run terraform in docker container 333 | 334 | Terraform releases official docker containers that you can easily control which version you can run. 335 | 336 | Recommend to run terraform docker container, when you set your build job in CI/CD pipeline. 337 | 338 | ```bash 339 | # (1) must mount the local folder to /apps in container. 340 | # (2) must mount the aws credentials and ssh config folder in container. 341 | $ TERRAFORM_IMAGE=hashicorp/terraform:0.12.3 342 | $ TERRAFORM_CMD="docker run -ti --rm -w /app -v ${HOME}/.aws:/root/.aws -v ${HOME}/.ssh:/root/.ssh -v `pwd`:/app -w /app ${TERRAFORM_IMAGE}" 343 | ${TERRAFORM_CMD} init 344 | ${TERRAFORM_CMD} plan 345 | ``` 346 | 347 | Or with `terragrunt` by image [alpine/terragrunt](https://hub.docker.com/r/alpine/terragrunt) 348 | 349 | ```bash 350 | # (1) must mount the local folder to /apps in container. 351 | # (2) must mount the aws credentials and ssh config folder in container. 352 | $ docker run -ti --rm -v $HOME/.aws:/root/.aws -v ${HOME}/.ssh:/root/.ssh -v `pwd`:/apps alpine/terragrunt:0.12.3 bash 353 | # cd to terragrunt configuration directory, if required. 354 | $ terragrunt plan-all 355 | $ terragrunt apply-all 356 | ``` 357 | 358 | ## Run test 359 | 360 | Way 1: Recommend to add [awspec](https://github.com/k1LoW/awspec) tests through [kitchen](https://kitchen.ci/) and [kitchen-terraform](https://newcontext-oss.github.io/kitchen-terraform/). 361 | Run test within docker container, you can take reference: [README for terraform awspec container](https://github.com/alpine-docker/bundle-terraform-awspec) 362 | 363 | Way 2: [terratest](https://terratest.gruntwork.io/) 364 | 365 | Way 3: [terraform test](https://developer.hashicorp.com/terraform/language/tests) This testing framework is available in Terraform **v1.6.0** and later. 366 | 367 | ## Minimum AWS permissions necessary for a Terraform run 368 | 369 | There will be no answer for this. But with below iam policy you can easily get started. 370 | 371 | ```json 372 | { 373 | "Version": "2012-10-17", 374 | "Statement": [ 375 | { 376 | "Sid": "AllowSpecifics", 377 | "Action": [ 378 | "lambda:*", 379 | "apigateway:*", 380 | "ec2:*", 381 | "rds:*", 382 | "s3:*", 383 | "sns:*", 384 | "states:*", 385 | "ssm:*", 386 | "sqs:*", 387 | "iam:*", 388 | "elasticloadbalancing:*", 389 | "autoscaling:*", 390 | "cloudwatch:*", 391 | "cloudfront:*", 392 | "route53:*", 393 | "ecr:*", 394 | "logs:*", 395 | "ecs:*", 396 | "application-autoscaling:*", 397 | "logs:*", 398 | "events:*", 399 | "elasticache:*", 400 | "es:*", 401 | "kms:*", 402 | "dynamodb:*" 403 | ], 404 | "Effect": "Allow", 405 | "Resource": "*" 406 | }, 407 | { 408 | "Sid": "DenySpecifics", 409 | "Action": [ 410 | "iam:*User*", 411 | "iam:*Login*", 412 | "iam:*Group*", 413 | "iam:*Provider*", 414 | "aws-portal:*", 415 | "budgets:*", 416 | "config:*", 417 | "directconnect:*", 418 | "aws-marketplace:*", 419 | "aws-marketplace-management:*", 420 | "ec2:*ReservedInstances*" 421 | ], 422 | "Effect": "Deny", 423 | "Resource": "*" 424 | } 425 | ] 426 | } 427 | ``` 428 | 429 | Depending on your company's or project's requirements, you can easily modify the resources in the **Allow** section to specify which Terraform commands should have access, and you can add deny policies in the **Deny** section to restrict permissions that are not needed. 430 | 431 | ## Usage of variable "self" 432 | 433 | Quote from terraform documents: 434 | 435 | ```log 436 | Attributes of your own resource 437 | 438 | The syntax is self.ATTRIBUTE. For example \${self.private_ip} will interpolate that resource's private IP address. 439 | 440 | Note: The self.ATTRIBUTE syntax is only allowed and valid within provisioners. 441 | ``` 442 | 443 | ### One more use case 444 | 445 | ```terraform 446 | resource "aws_ecr_repository" "this" { 447 | name = var.image_name 448 | provisioner "local-exec" { 449 | command = "./deploy-image.sh ${self.repository_url} ${var.jenkins_image_name}" 450 | } 451 | } 452 | 453 | variable "jenkins_image_name" { 454 | default = "mycompany/jenkins" 455 | description = "Jenkins image name." 456 | } 457 | ``` 458 | 459 | You can easily define ecr image url (`.dkr.ecr..amazonaws.com/`) with \${self.repository_url} 460 | 461 | Any attributes in this resource can be self referenced by this way. 462 | 463 | Reference: 464 | 465 | ## Use pre-installed Terraform plugins 466 | 467 | There is a way to use pre-installed Terraform plugins instead of downloading them with `terraform init`, the accepted answer below gives the detail: 468 | 469 | [Use pre-installed Terraform plugins instead of downloading them with terraform init](https://stackoverflow.com/questions/50944395/use-pre-installed-terraform-plugins-instead-of-downloading-them-with-terraform-i?rq=1) 470 | 471 | ## Tips to upgrade to terraform 0.12 472 | 473 | ``` 474 | terraform 0.12upgrade 475 | ``` 476 | 477 | If you have any codes older than 0.12, please go through official documents first, 478 | 479 | - [terraform Input Variables](https://www.terraform.io/docs/configuration/variables.html), a lot of new features you have to know. 480 | - [Upgrading to Terraform v0.12](https://www.terraform.io/upgrade-guides/0-12.html) 481 | - [terraform command 0.12upgrade](https://www.terraform.io/docs/commands/0.12upgrade.html) 482 | - [Announcing Terraform 0.12](https://www.hashicorp.com/blog/announcing-terraform-0-12) 483 | 484 | Then here are extra tips for you. 485 | 486 | - Upgrade to terraform 0.11 first, if you have any. 487 | - Upgrade terraform moudles to 0.12 first, because terraform 0.12 can't work with 0.11 modules. 488 | - Define `type` for each variable, otherwise you will get weird error messages. 489 | 490 | ## Tips to upgrade to terraform 0.13+ 491 | 492 | In fact the command `terraform 0.13upgrade` in terraform v0.13.3 (the latest version currently) doesn't work to convert older versions less than v0.11 493 | 494 | So you have to download terraform 0.12 version to do the upgrade. But from hashicorp terraform website, there is only v0.13.x for downloading now. 495 | 496 | Here is a simple way if you can run with docker 497 | 498 | ``` 499 | # cd to the terraform tf files folder, run below commands 500 | 501 | # do the upgrade within terraform 0.12 container 502 | $ docker run -ti --rm -v $(pwd):/apps -w /apps --entrypoint=sh hashicorp/terraform:0.12.29 503 | /apps # terraform init 504 | /apps # terraform 0.12upgrade -yes 505 | /apps # exit 506 | 507 | # double check with 0.13upgrade 508 | $ terraform 0.13upgrade -yes 509 | $ 510 | ``` 511 | 512 | # Contributing 513 | 514 | - Update [README.md](README.md) 515 | - install [doctoc](https://github.com/thlorenz/doctoc) 516 | 517 | ``` 518 | npm install -g doctoc 519 | ``` 520 | 521 | - update README 522 | 523 | ``` 524 | doctoc --github README.md 525 | ``` 526 | 527 | - commit the update and raise pull request for reviewing. 528 | 529 | # Useful terraform modules 530 | 531 | 1. terraform aws ami helper 532 | 533 | usage module to easly find some useful AWS ami id, here is a sample to get latest amazon linux 2 ami id 534 | 535 | ```terraform 536 | module "helper" { 537 | source = "recarnot/ami-helper/aws" 538 | os = module.helper.AMAZON_LINUX_2 539 | } 540 | 541 | output "id" { 542 | value = module.helper.ami_id 543 | } 544 | 545 | # reference: [recarnot/terraform-aws-ami-helper](https://github.com/recarnot/terraform-aws-ami-helper) 546 | ``` 547 | -------------------------------------------------------------------------------- /errors.md: -------------------------------------------------------------------------------- 1 | Not sure if this is useful, but let me put here first. 2 | 3 | ### Error #1: Terraform destroy fails "variable ... is nil, but no error was reported" 4 | 5 | Fix: https://github.com/hashicorp/terraform/issues/18197#issuecomment-439015313 6 | -------------------------------------------------------------------------------- /lambda/.gitignore: -------------------------------------------------------------------------------- 1 | .terraform 2 | source.zip 3 | terraform.tfstate* 4 | -------------------------------------------------------------------------------- /lambda/lambda.tf: -------------------------------------------------------------------------------- 1 | # codes for pip install and zip packaging 2 | resource "null_resource" "pip" { 3 | triggers { 4 | main = "${base64sha256(file("${path.module}/source/main.py"))}" 5 | requirements = "${base64sha256(file("${path.module}/source/requirements.txt"))}" 6 | execute = "${base64sha256(file("${path.module}/pip.sh"))}" 7 | } 8 | 9 | provisioner "local-exec" { 10 | command = "${path.module}/pip.sh ${path.module}/source" 11 | } 12 | } 13 | 14 | data "archive_file" "source" { 15 | type = "zip" 16 | source_dir = "${path.module}/source" 17 | output_path = "${path.module}/source.zip" 18 | 19 | depends_on = ["null_resource.pip"] 20 | } 21 | 22 | # codes for lambda functions 23 | resource "aws_iam_role" "lambda" { 24 | name = "iam_for_lambda" 25 | 26 | assume_role_policy = <-terraform-states 7 | 8 | # usage 9 | 10 | ``` 11 | # make sure you are on the right aws account 12 | pip install awscli 13 | aws s3 ls 14 | 15 | # If you don't set default region in your aws configuration, and you want to create the resources in region "us-east-1" 16 | export AWS_DEFAULT_REGION=us-east-1 17 | export AWS_REGION=us-east-1 18 | 19 | # Dry-run 20 | terraform init 21 | terraform plan 22 | 23 | # apply the change 24 | terraform apply 25 | ``` 26 | -------------------------------------------------------------------------------- /s3-backend/main.tf: -------------------------------------------------------------------------------- 1 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2 | # CREATE AN S3 BUCKET AND DYNAMODB TABLE TO USE AS A TERRAFORM BACKEND 3 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4 | 5 | # ---------------------------------------------------------------------------------------------------------------------- 6 | # REQUIRE A SPECIFIC TERRAFORM VERSION OR HIGHER 7 | # This module has been updated with 0.12 syntax, which means it is no longer compatible with any versions below 0.12. 8 | # This module is forked from https://github.com/gruntwork-io/intro-to-terraform/tree/master/s3-backend 9 | # ---------------------------------------------------------------------------------------------------------------------- 10 | 11 | terraform { 12 | required_version = ">= 0.12" 13 | } 14 | 15 | # ------------------------------------------------------------------------------ 16 | # CONFIGURE OUR AWS CONNECTION 17 | # ------------------------------------------------------------------------------ 18 | 19 | provider "aws" {} 20 | 21 | # ------------------------------------------------------------------------------ 22 | # CREATE THE S3 BUCKET 23 | # ------------------------------------------------------------------------------ 24 | 25 | data "aws_caller_identity" "current" {} 26 | 27 | locals { 28 | account_id = data.aws_caller_identity.current.account_id 29 | } 30 | 31 | resource "aws_s3_bucket" "terraform_state" { 32 | # With account id, this S3 bucket names can be *globally* unique. 33 | bucket = "${local.account_id}-terraform-states" 34 | 35 | # Enable versioning so we can see the full revision history of our 36 | # state files 37 | versioning { 38 | enabled = true 39 | } 40 | 41 | # Enable server-side encryption by default 42 | server_side_encryption_configuration { 43 | rule { 44 | apply_server_side_encryption_by_default { 45 | sse_algorithm = "AES256" 46 | } 47 | } 48 | } 49 | } 50 | 51 | resource "aws_s3_bucket_public_access_block" "terraform_state" { 52 | bucket = aws_s3_bucket.terraform_state.id 53 | 54 | block_public_acls = true 55 | block_public_policy = true 56 | ignore_public_acls = true 57 | restrict_public_buckets = true 58 | } 59 | 60 | # ------------------------------------------------------------------------------ 61 | # CREATE THE DYNAMODB TABLE 62 | # ------------------------------------------------------------------------------ 63 | 64 | resource "aws_dynamodb_table" "terraform_lock" { 65 | name = "terraform-lock" 66 | billing_mode = "PAY_PER_REQUEST" 67 | hash_key = "LockID" 68 | 69 | attribute { 70 | name = "LockID" 71 | type = "S" 72 | } 73 | } 74 | -------------------------------------------------------------------------------- /s3-backend/outputs.tf: -------------------------------------------------------------------------------- 1 | output "s3_bucket_name" { 2 | value = aws_s3_bucket.terraform_state.id 3 | description = "The NAME of the S3 bucket" 4 | } 5 | 6 | output "s3_bucket_arn" { 7 | value = aws_s3_bucket.terraform_state.arn 8 | description = "The ARN of the S3 bucket" 9 | } 10 | 11 | output "s3_bucket_region" { 12 | value = aws_s3_bucket.terraform_state.region 13 | description = "The REGION of the S3 bucket" 14 | } 15 | 16 | output "dynamodb_table_name" { 17 | value = aws_dynamodb_table.terraform_lock.name 18 | description = "The ARN of the DynamoDB table" 19 | } 20 | 21 | output "dynamodb_table_arn" { 22 | value = aws_dynamodb_table.terraform_lock.arn 23 | description = "The ARN of the DynamoDB table" 24 | } 25 | --------------------------------------------------------------------------------