├── README.md
└── dogs.gif


/README.md:
--------------------------------------------------------------------------------
  1 | ## Introduction
  2 | 
  3 | This article was originally published during [SysAdvent 2016](http://sysadvent.blogspot.com/2016/12/day-14-terraform-deployment-strategy.html).
  4 | 
  5 | [HashiCorp's](https://www.hashicorp.com/) infrastructure management tool, [Terraform](https://www.terraform.io/), is no doubt very flexible and powerful.  The question is, how do we write Terraform code and construct our infrastructure in a reproducible fashion that makes sense? How can we keep code [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself), segment state, and reduce the risk of making changes to our service/stack/infrastructure?
  6 | 
  7 | This post describes a design pattern to help answer the previous questions. This article is divided into two sections, with the first section describing and defining the design pattern with a _Deployment Example_. The second part uses a [multi-repository GitHub organization](https://github.com/TerraformDesignPattern) to create a _Real World Example_ of the design pattern.
  8 | 
  9 | This post assumes you understand or are familiar with AWS and basic Terraform concepts such as <a href="https://www.terraform.io/docs/commands/index.html">CLI Commands</a>, <a href="https://www.terraform.io/docs/configuration/providers.html">Providers</a>, <a href="https://www.terraform.io/docs/providers/aws/index.html">AWS Provider</a>, <a href="https://www.terraform.io/docs/state/index.html">Remote State</a>, <a href="https://www.terraform.io/docs/providers/terraform/d/remote_state.html">Remote State Data Sources</a>, and <a href="https://www.terraform.io/docs/modules/index.html">Modules</a>.
 10 | 
 11 | ### Modules
 12 | 
 13 | Essentially, any directory with one or more `.tf` files can be used as or considered a Terraform module. I am going to be creating a couple of _module types_ and giving them names for reference. The first module type constructs all of the resources a service will need to be operational such as EC2 instances, S3 bucket, etc. The remaining module types will instantiate the aforementioned module type.
 14 | 
 15 | The first module type is a _service module._ A service module can be thought of as a reuseable library that deploys a single service's infrastructure. _Service modules_ are the brains and contain the logic to create Terraform resources. They are the _"how"_ we build our infrastructure. 
 16 | 
 17 | The other module types are _environment modules._ We will run our Terraform commands within this module type. The environment modules all live within a single repository, as compared to service modules, which live in individual repositories or alongside the code of the service they build infrastructure for. This is _"where"_ our infrastructure will be built.
 18 | 
 19 | ## Deployment Example
 20 | 
 21 | I am going to start by describing how we would deploy a service and then deconstruct the concepts as we move through the deployment.
 22 | 
 23 | ### Running Terraform
 24 | 
 25 | As mentioned earlier, the [environments repository](https://github.com/TerraformDesignPattern/environments) is where we actually run the Terraform command to instantiate a service module. I've written a [Bash wrapper](https://github.com/TerraformDesignPattern/environments/blob/master/templates/service-remote.sh) to manage the service's remote state configuration and ensure we always have to latest modules. 
 26 | 
 27 | So instead of running `terraform apply` we will run `./remote.sh apply` 
 28 | 
 29 | The `apply` argument will set and get our remote state configuration, run a `terraform get -update` and then run `terraform apply`
 30 | 
 31 | ### Environment Module Example
 32 | 
 33 | #### Directory Structure
 34 | 
 35 | The environment module's respository contains a strict directory hierarchy:
 36 | 
 37 | ```
 38 | production-account (aws-account)
 39 | |__ us-east-1 (aws-region)
 40 |     |__ production-us-east-1-vpc (vpc)
 41 |         |__ production (environment)
 42 |             |__ ssh-bastion (service) 
 43 |                |__ remote.sh <~~~~~ YOU ARE HERE
 44 | ```
 45 | 
 46 | ##### Dynamic State Management
 47 | 
 48 | The directory structure is one of the cornerstones of this design as it enables us to dynamically generate the S3 key for a service's state file. Within the remote.sh script (shown below) we parse the directory structure and then set/get our remote state. 
 49 | 
 50 | A symlink of `/templates/service-remote.sh` to `remote.sh` is created in each service folder. 
 51 | 
 52 | ```
 53 | #!/bin/bash -e
 54 | # Must be run in the service's directory.
 55 | 
 56 | help_message() {
 57 |   echo -e "Usage: $0 [apply|destroy|plan|refresh|show]\n"
 58 |   echo -e "The following arguments are supported:"
 59 |   echo -e "\tapply   \t Refresh the Terraform remote state, perform a \"terraform get -update\", and issue a \"terraform apply\""
 60 |   echo -e "\tdestroy \t Refresh the Terraform remote state and destroy the Terraform stack"
 61 |   echo -e "\tplan    \t Refresh the Terraform remote state, perform a \"terraform get -update\", and issues a \"terraform plan\""
 62 |   echo -e "\trefresh \t Refresh the Terraform remote state"
 63 |   echo -e "\tshow    \t Refresh and show the Terraform remote state"
 64 |   exit 1
 65 | }
 66 | 
 67 | apply() {
 68 |   plan
 69 |   echo -e "\n\n***** Running \"terraform apply\" *****"
 70 |   terraform apply
 71 | }
 72 | 
 73 | destroy() {
 74 |   plan
 75 |   echo -e "\n\n***** Running \"terraform destroy\" *****"
 76 |   terraform destroy
 77 | }
 78 | 
 79 | plan() {
 80 |   refresh
 81 |   terraform get -update
 82 |   echo -e  "\n\n***** Running \"terraform plan\" *****"
 83 |   terraform plan
 84 | }
 85 | 
 86 | refresh() {
 87 | 
 88 |   account=$(pwd | awk -F "/" '{print $(NF-4)}')
 89 |   region=$(pwd | awk -F "/" '{print $(NF-3)}')
 90 |   vpc=$(pwd | awk -F "/" '{print $(NF-2)}')
 91 |   environment=$(pwd | awk -F "/" '{print $(NF-1)}')
 92 |   service=$(pwd | awk -F "/" '{print $NF}')
 93 | 
 94 |   echo -e "\n\n***** Refreshing State *****"
 95 | 
 96 |   terraform remote config -backend=s3 \
 97 |                           -backend-config="bucket=${account}" \
 98 |                           -backend-config="key=${region}/${vpc}/${environment}/${service}/terraform.tfstate" \
 99 |                           -backend-config="region=us-east-1"
100 | }
101 | 
102 | show() {
103 |   refresh
104 |   echo -e "\n\n***** Running \"terraform show\"  *****"
105 |   terraform show
106 | }
107 | 
108 | ## Begin script ##
109 | if [ "$#" -ne 1 ] || [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
110 |   help_message
111 | fi
112 | 
113 | ACTION="$1"
114 | 
115 | case $ACTION in
116 |   apply|destroy|plan|refresh|show)
117 |     $ACTION
118 |     ;;
119 |   ****)
120 |     echo "That is not a vaild choice."
121 |     help_message
122 |     ;;
123 | esac
124 | ```
125 | 
126 | ### Service Module Instantiation
127 | 
128 | In addition to remote.sh, the environment's service directory contains a [main.tf](https://github.com/TerraformDesignPattern/environments/blob/master/sysadvent-production/us-east-1/production-us-east-1-vpc/production/ssh-bastion/main.tf) file.
129 | 
130 | ```
131 | module "environment" {
132 |   source = "../"
133 | }
134 | 
135 | module "bastion" {
136 |   source = "git@github.com:TerraformDesignPattern/bastionhost.git"
137 | 
138 |   aws_account      = "${module.environment.aws_account}"
139 |   aws_region       = "${module.environment.aws_region}"
140 |   environment_name = "${module.environment.environment_name}"
141 |   hostname         = "${var.hostname}"
142 |   image_id         = "${var.image_id}"
143 |   vpc_name         = "${module.environment.vpc_name}"
144 | }
145 | ```
146 | 
147 | We call two modules in `main.tf`. The first is an _environment module_, which we'll talk about in a moment, and the second is an _environment service module_. 
148 | 
149 | ### Environment Service Module
150 | 
151 | _Environment service module?_ 
152 | 
153 | Everytime I've introduced this term, I've seen this... 
154 | 
155 | ![alt tag](https://raw.githubusercontent.com/TerraformDesignPattern/SysAdvent2016/master/dogs.gif)
156 | 
157 | An environment service module, or _ESM_ for short, is just a way to specify, in conversation, that we are talking about the code that actually instantiates a service module.
158 | 
159 | If you look at the ESM declaration in `main.tf` above, you'll see it is using the output from the _enviroment module_ to define variables that will be passed into the _service module_. If we take a step back to review our directory structure we see the service we are deploying sits within the production environment's directory:
160 | 
161 | `sysadvent-production/us-east-1/production-us-east-1-vpc/production/ssh-bastion`
162 | 
163 | Within the production environment's directory is an [outputs.tf](https://github.com/TerraformDesignPattern/environments/blob/master/sysadvent-production/us-east-1/production-us-east-1-vpc/production/outputs.tf) file.
164 | 
165 | ```
166 | output "aws_account" {
167 |   value = "production"
168 | }
169 | 
170 | output "aws_region" {
171 |   value = "us-east-1"
172 | }
173 | 
174 | output "environment_name" {
175 |   value = "production"
176 | }
177 | 
178 | output "vpc_name" {
179 |   value = "production-us-east-1-vpc"
180 | }
181 | ```
182 | 
183 | We are able to create an entire service, regardless of resources, with a very generic ESM and just four values from our environment module. We are using our organization's defined and somewhat colloquial terms to create our infrastructure. We don't need to remember ARNs, ID's or other allusive information. We don't need to remember naming conventions either as the service module will take care of this for us.
184 | 
185 | ### Service Module Example 
186 | 
187 | So far we've established a repeatable ways to run our Terraform command and guarantee that our state is managed properly and consistently. We've also instantiated a _service module_ from within an _environment service module_. We are now going to dive into the components of a service module.
188 | 
189 | A service module will be reused throughout your infrastrcuture so it must be generic and parameterized. The module will create all Terraform provider resources required by the service.
190 | 
191 | In my experience, I've found splitting each resource type into its own file improves readability. Below is the list of Terraform files from our [example bastion host service module repository](https://github.com/TerraformDesignPattern/bastionhost):
192 | 
193 | ```
194 | bastionhost
195 | |-- data.tf
196 | |-- ec2.tf
197 | |-- LICENSE
198 | |-- outputs.tf
199 | |-- providers.tf
200 | |-- route53.tf
201 | |-- security_groups.tf
202 | `-- variables.tf
203 | ```
204 | 
205 | The contents of most of these files will look pretty generic to the average Terraform user. The power of this pattern lies within the [data.tf](https://github.com/TerraformDesignPattern/bastionhost/blob/master/data.tf) as it allows the simplistic instantiation.
206 | 
207 | ```
208 | // Account Remote State
209 | data "terraform_remote_state" "account" {
210 |   backend = "s3"
211 | 
212 |   config {
213 |     bucket = "${var.aws_account}"
214 |     key    = "terraform.tfstate"
215 |     region = "us-east-1"
216 |   }
217 | }
218 | 
219 | // VPC Remote State
220 | data "terraform_remote_state" "vpc" {
221 |   backend = "s3"
222 | 
223 |   config {
224 |     bucket = "${var.aws_account}"
225 |     key    = "${var.aws_region}/${var.vpc_name}/terraform.tfstate"
226 |     region = "us-east-1"
227 |   }
228 | }
229 | ```
230 | 
231 | Sooooo. What populates the state file for the VPC data resources? Enter the  _VPC Service Module_ 
232 | 
233 | >There is no cattle.  
234 | There are no layers.  
235 | There is no spoon.  
236 | 
237 | Everything is just a compartmentalized service. The module that creates your VPC is a separate "service" that lives in its own repository.
238 | 
239 | We create our account resources (DNS zones, SSL Certs, Users) within an account service module, our vpc resources from a VPC module within a vpc service module and our services (Application Services, RDS, Web Services) within an Environment Service Module.
240 | 
241 | We use a [Bash wrapper](https://github.com/TerraformDesignPattern/environments/tree/master/templates) to publish the state of resources in a consistent fashion.
242 | 
243 | Lastly, we abstract the complexity of infrastructure configuration management by querying Terraform state files based on a strict S3 key structure.
244 | 
245 | ## Real World Example
246 | 
247 | Follow along with the example by pulling the [TerraformDesignPattern/environments repository](https://github.com/TerraformDesignPattern/environments). The configuration of each module within the environment repository will consist of roughly the same steps: 
248 | 
249 | 1. Create the required files. Usually `main.tf` and `variables.tf` or simply an `outputs.tf` file.
250 | 2. Populate `variables.tf`/`outputs.tf` with your desired values.
251 | 3. Create a symlink to a specific `remote.sh` (`account-remote.sh`, `service-remote.sh` or `vpc-remote.sh`) from within the appropriate directory.
252 |   * For example, to create the `remote.sh` wrapper for your account service module, issue the following from within your `environments/$ACCOUNT` directory: `ln -s ../templates/account-remote.sh remote.sh`
253 | 4. Run `./remote.sh apply`
254 | 
255 | ### Prerequisites
256 | 
257 | * __Domain Name__: I went over to [Namecheap.com](https://www.namecheap.com/) and grabbed the `sysadvent.host` domain name for $.88.  
258 | * __State File S3 Bucket__: Create the S3 bucket to store your state files in. This should be the name of the [account folder](https://github.com/TerraformDesignPattern/environments/tree/master/sysadvent-production) within your [environments](https://github.com/TerraformDesignPattern/environments) repository. For this example I created the `sysadvent-production` S3 bucket.  
259 | * __SSH Public Key__: As of the writing of this post, the `aws_key_pair` Terraform resource does not currently support creating a public key, only importing one.
260 | * __SSL ARN__: [AWS Certificate Manager offers free SSL/TLS certificates](http://docs.aws.amazon.com/acm/latest/userguide/gs-acm-request.html).
261 | 
262 | ### Getting Started
263 | 
264 | This _Real World Example_ assumes you are provisioning a new AWS account and domain. For those working in a brown field, the following section provides a quick example of how to build a scaffolding that can be used to deploy the design pattern.
265 | 
266 | #### State Scaffolding or Fake It 'Til You Make It (With Terraform)
267 | 
268 | Within the [environments/sysadvent-production account directory](https://github.com/TerraformDesignPattern/environments/tree/master/sysadvent-production) is an [s3.tf file](https://github.com/TerraformDesignPattern/environments/blob/master/sysadvent-production/s3.tf) that creates a `dummy_object` in S3:
269 | 
270 | ```
271 | resource "aws_s3_bucket_object" "object" {
272 |   bucket = "${var.aws_account}"
273 |   key    = "dummy_object"
274 |   source = "outputs.tf"
275 |   etag   = "${md5(file("outputs.tf"))}"
276 | }
277 | ```
278 | 
279 | A new object will be uploaded when `outputs.tf` is changed. This change updates the remote state file and thus any outputs that have been added to `outputs.tf` will be added to the remote state file as well. To use a resource (IAM Role, VPC ID, or Zone ID) that was not created with Terraform, simply add the desired data to the account, vpc, or ESM's `outputs.tf` file. Since not all resources can be imported via data resources, this enables us to migrate in small, iterable phases. 
280 | 
281 | In the example below, the AWS Account ID will be added to the account's state file via this mechanism. The `outputs.tf` file defines the account ID via the `aws_account` variable:
282 | 
283 | ```
284 | output "aws_account" {
285 |   value = "${var.aws_account}"
286 | }
287 | ```
288 | 
289 | ### Stage One: Create The Account Service Module (ASM)
290 | 
291 | ```
292 | Working Directory: environments/sysadvent-production
293 | ```
294 | 
295 | As per the name, this is were account wide resources are created such as DNS Zones or Cloudtrail Logs. The `sysadvent-production` ASM will create the following:
296 | 
297 | * Cloudwatch Log Stream for the account's Cloudtrail
298 | * Import a public key 
299 | * Route53 Zone
300 | * The "scaffolding" S3 _dummy_object_ from `outputs.tf` to publish:
301 |   * AWS Account ID
302 |   * Domain Name
303 |   * SSL ARN
304 |   
305 | #### Populate Variables
306 | 
307 | Populate the variables in the account's [variables.tf](https://github.com/TerraformDesignPattern/environments/blob/master/sysadvent-production/variables.tf) file:
308 | 
309 | ```
310 | aws_account - name of the account level folder
311 | aws_account_id - your AWS provided account ID
312 | domain_name - your choosen domain name
313 | key_pair_name - what you want to name the key you are going to import
314 | ssl_arn - ARN of the SSL certificate you created, free, with Amazon's Certificate Manager
315 | public_key - the actual public key you want to import as your key pair
316 | ```
317 | 
318 | #### Execute!
319 | 
320 | Once you have created a state file S3 bucket and populated the `variables.tf` file with your desired values run `./remote.sh apply`.
321 | 
322 | ### Stage Two: Create The VPC Service Module (VSM)
323 | 
324 | ```
325 | Working Directory: environments/sysadvent-production/us-east-1/production-us-east-1-vpc
326 | ```
327 | 
328 | The [TerraformDesignPattern/vpc](https://github.com/TerraformDesignPattern/vpc) module creates the following resources:
329 | 
330 | * VPC
331 | * Enable VPC Flow Logs
332 | * Cloudwatch Log Stream for the VPC's Flow Logs
333 | * Flow Log IAM Policies
334 | * An Internet Gateway
335 | * Three private subnets 
336 | * Three public subnets with nat gateways and elastic IP addresses
337 | * Routes, route tables, and associations 
338 | 
339 | #### Populate Variables
340 | 
341 | Populate the following variables in the VSM's [variables.tf](https://github.com/TerraformDesignPattern/environments/blob/master/sysadvent-production/us-east-1/production-us-east-1-vpc/variables.tf) file:
342 | 
343 | ```
344 | availability_zones
345 | aws_region
346 | private_subnets
347 | public_subnets
348 | vpc_cidr
349 | vpc_name
350 | ```
351 | 
352 | #### Execute!
353 | 
354 | Once you have populated the `variables.tf` file with your desired values, create the resources by running `./remote.sh apply`.
355 | 
356 | ### Stage Three: Create The Environment Module
357 | 
358 | ```
359 | Working Directory: environments/sysadvent-production/us-east-1/production-us-east-1-vpc/production
360 | ```
361 | 
362 | The _environment module_ stores the minimal amount of information required to pass to an _environment service module_. As mentioned previously, this module consists of a single [outputs.tf](https://github.com/TerraformDesignPattern/environments/blob/master/sysadvent-production/us-east-1/production-us-east-1-vpc/production/outputs.tf) file which requires you to configure the following: 
363 | 
364 | ```
365 | aws_account
366 | aws_region
367 | environment_name
368 | vpc_name
369 | ```
370 | 
371 | ### Stage Four: Create An Environment Service Module (ESM)
372 | 
373 | ```
374 | Working Directory: environments/sysadvent-production/us-east-1/production-us-east-1-vpc/production/elk
375 | ```
376 | 
377 | Congratulations, you've made it to the point where this wall of text really pays off. 
378 | 
379 | Create the [main.tf](https://github.com/TerraformDesignPattern/environments/blob/master/sysadvent-production/us-east-1/production-us-east-1-vpc/production/elk/main.tf) file. The ELK's ESM calls an _environment module_, an _ami_image_id module_ , and the _ELK service module_. The _environment module_ supplies environment specific data such as the AWS account, region, environment name and VPC name. This module data is, in turn, passed to the _ami_image_id module_. The _ami_image_id module_ will return AMI ID's based on the enviroment's region. 
380 | 
381 | The ELK ESM will create the following resources:
382 | 
383 | * Three instance ELK stack within the previously created private subnet.
384 | * The Elasticsearch instances will be clustered via the EC2 discovery plugin.
385 | * A public facing ELB to access Kibana.
386 | * A Route53 DNS entry pointing to the ELB.
387 | 
388 | #### Execute!
389 | 
390 | Once you created the `main.tf` file, create the resources by running `./remote.sh apply`.
391 | 
392 | ## Appendix
393 | 
394 | ### Links
395 | 
396 | * [Terraforn Design Pattern Github Organization](https://github.com/TerraformDesignPattern)
397 |   * [Bastion Host Service Module](https://github.com/TerraformDesignPattern/bastionhost)
398 |   * [Cloud Trail Module](https://github.com/TerraformDesignPattern/cloudtrail)
399 |   * [Environments Module](https://github.com/TerraformDesignPattern/environments)
400 |   * [ELK Service Module](https://github.com/TerraformDesignPattern/elk)
401 |   * [Packer Service Module](https://github.com/TerraformDesignPattern/packer)  
402 |   * [VPC Module](https://github.com/TerraformDesignPattern/vpc)
403 | * [Terraform Style Guide](https://github.com/jonbrouse/terraform-style-guide/blob/master/README.md)
404 | 
405 | ### Gotchas
406 | 
407 | During the development of this pattern I stumbled across a couple _gotchas_. I wanted to share these with you but didn't think they were necessarily pertinent to an introductory article.
408 | 
409 | #### Global Service Resources
410 | 
411 | ##### Example: IAM Roles
412 | 
413 | We want to keep the creation of IAM roles in a compartmentalized module but IAM roles are  global thus you can only create them once. Using _count_ and a lookup based on our region, we can tell Terraform to only create the IAM role in a single region.
414 | 
415 | Example `iam.tf` file:
416 | 
417 | ```
418 | ...
419 | count = "${lookup(var.create_iam_role, var.aws_region, 0)}"
420 | ...
421 | ```
422 | 
423 | Example `variables.tf` file:
424 | 
425 | ```
426 | ...
427 | variable "create_iam_role" {
428 |   default = {
429 |     "us-east-1" = 1
430 |   }
431 | }
432 | ...
433 | ```
434 | 
435 | #### Referencing an ASM or VSM From an ESM
436 | 
437 | The ASM and VSM create account and VPC resources, respectively. If you were to reference an ASM or VSM a la an environment module within an ESM, you'll essentially be attempting to recreate the resources originally created by the ASM and VSM. 
438 | 


--------------------------------------------------------------------------------
/dogs.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/TerraformDesignPattern/SysAdvent2016/831bd442797cf620050ffb44c1844f0c5bc78e88/dogs.gif


--------------------------------------------------------------------------------