├── .gitignore ├── CHANGELOG.md ├── LICENSE ├── Makefile ├── README-vault.md ├── README.md ├── Vagrantfile ├── app-deploy-key.pem.yaml.sample ├── cluster-manager.sh ├── envs.sh.sample ├── graph-examples ├── admiral.png ├── efs.png ├── elb-ci.png ├── etcd.png ├── iam.png ├── rds.png ├── route53.png ├── s3.png ├── vpc.png └── worker.png ├── modules ├── asg-with-elb │ ├── autoscaling-group-with-elb.tf │ └── variables.tf ├── cloudtrail │ ├── main.tf │ └── variables.tf ├── cluster │ ├── cluster.tf │ └── variables.tf ├── efs-target │ └── efs-target.tf └── subnet │ └── subnet.tf ├── resources ├── certs │ ├── Makefile │ ├── README.md │ ├── etcd-client.cnf │ ├── etcd.cnf │ ├── rootCA.cnf │ └── site.cnf ├── cloud-config │ ├── admiral.yaml.tmpl │ ├── common-files.yaml.tmpl │ ├── dockerhub.yaml │ ├── etcd.yaml.tmpl │ ├── files-vault.yaml │ ├── files.yaml │ ├── s3-cloudconfig-bootstrap.sh │ ├── systemd-units-flannel.yaml │ ├── systemd-units.yaml │ ├── vault.yaml.tmpl │ └── worker.yaml.tmpl ├── makefiles │ ├── admiral.mk │ ├── cloudtrail.mk │ ├── efs.mk │ ├── elb-ci.mk │ ├── etcd.mk │ ├── iam.mk │ ├── init.mk │ ├── rds.mk │ ├── route53.mk │ ├── s3.mk │ ├── vault.mk │ ├── vpc.mk │ └── worker.mk ├── policies │ ├── admiral_policy.json │ ├── assume_role_policy.json │ ├── deployment_policy.json │ ├── dockerhub_policy.json │ ├── etcd_policy.json │ ├── vault_policy.json │ └── worker_policy.json └── terraforms │ ├── admiral │ └── admiral.tf │ ├── cloudtrail │ └── cloudtrail.tf │ ├── efs │ └── efs.tf │ ├── elb-ci │ └── ci.tf │ ├── elb-dockerhub │ └── dockerhub.tf │ ├── etcd │ └── etcd.tf │ ├── iam │ └── iam.tf │ ├── rds │ ├── mysql.tf │ ├── postgres.tf │ └── security-group.tf │ ├── route53 │ └── route53.tf │ ├── s3 │ └── s3.tf │ ├── vault │ ├── elb.tf │ ├── variables.tf │ └── vault.tf │ ├── vpc │ ├── vpc-subnet-admiral.tf │ ├── vpc-subnet-elb.tf │ ├── vpc-subnet-etcd.tf │ ├── vpc-subnet-rds.tf │ ├── vpc-subnet-vault.tf │ ├── vpc-subnet-worker.tf │ └── vpc.tf │ └── worker │ └── worker.tf └── scripts ├── aws-keypair.sh ├── cloudtrail-admin.sh ├── gen-provider.sh ├── gen-rds-password.sh ├── gen-tf-vars.sh ├── get-ami.sh ├── get-dns-name.sh ├── get-ec2-public-id.sh ├── get-vpc-id.sh ├── session-lock.sh ├── setup_vault.sh ├── substitute-AWS-ACCOUNT.sh ├── substitute-CLUSTER-NAME.sh └── tf-apply-confirm.sh /.gitignore: -------------------------------------------------------------------------------- 1 | build*/* 2 | secrets 3 | hushhush 4 | *.pem 5 | *.key 6 | .DS_Store 7 | .git 8 | .terraform 9 | .vagrant 10 | coreos-cluster 11 | envs.sh 12 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | ## v0.2.1 2 | 3 | FEATURES: 4 | 5 | * Added AWS Cloudtrail module 6 | * Added Vault cluster with dedicated etcd storage backend 7 | 8 | ## v0.2.0 9 | 10 | DESIGN CHANGE: 11 | 12 | In previous implementation, we use one build directory for all resources, so there is only one __terraform.tfstate__ file to maintain. 13 | 14 | For more complicated cases, ongoing modifications to the infrastructure, e.g., a small change such as a security group rule, could take long time because all resources need to be refreshed and checked for consistency. 15 | 16 | In this release, each resource has its owner __terraform/\__ and coresponding __build/\__ directory, so it can 17 | be invidually managed, for example, `make ` target will only plan and appply changes for that resource, without checking vpc dependency, if you are sure there is no vpc changes. This will speed up operations. 18 | 19 | This release is backwards incompatible because of the structure change. You will need to migrate to this structure. 20 | 21 | FEATURES: 22 | 23 | * **envs.sh:** the file is used to override default environment variable values in **Makefile**. See [envs.sh.sample](https://github.com/xuwang/aws-terraform/blob/master/envs.sh.sample). 24 | * **lock, unlock:** these targets can be used in a team workflow to make sure only the person who owns the lock can alter the infrastructure. An pair of AWS key is used to facilitate the lock. 25 | * **session start, session end:** same as _lock_, _unlock_, these targets force git pull and git push to keep repository in-sync if you use git to maintain terraform status and code. 26 | * **etcd cluster:** the default cluster contains 1 etcd node, 1 worker node, t2.miro instance type. Change these in __terraform/etcd/etcd.tf__ or __terraform/worker/worker.tf__ for different configurations. The etcd cluster is in autoscaling group and can self-discover IP changes. 27 | * **graph** _make graph_ target will generate dependency graph in png format, under build/ directory. See [graph-examples](https://github.com/xuwang/aws-terraform/tree/master/graph-examples) for examples. 28 | * **two stage bootstraping:** all instances use the same [_user-data_](https://github.com/xuwang/aws-terraform/blob/master/resources/cloud-config/s3-cloudconfig-bootstrap.sh) file, a bootstrap script that will download the instance's specific 29 | cloud-config file from their corresponding S3 bucket, then CoreOS will run cloud-config using the downloaed cloud-config yaml file. This means that you rarely need to tear down and rebuild machine if the only change is in the cloud-config.yaml: reboot the instance will pick up the change. 30 | * **applicaiton bootstraping:** a git-sync timer unit is provisioned by cloud-config to download application relocated code, such as post-boot provisionning, account 31 | creation, file system mount, docker units files etc. The content of the applicaiton repo is cloned under/var/lib/apps location. The timer runs every minute, to pick up new changes. A [default app repo](https://github.com/dockerage/coreos-cluster-apps) is provided, and you can change it in __envs.sh__ to use your own repoistory. You can also configure a private key for git-sync to use for the sync. 32 | * **route53, iam server certificate**: these are optional resources. If you define APP_DOMAIN in envs.sh, the domain name will be used as a default route53 zone and a self-signed star server certificate will be generated and can used as default elb certitificate. 33 | * **default VPC**: If you change AWS region, you need to go through __terraform/vpc__ directory to make sure availablity zones are set correctly for the region,otherwise, the build will fail. 34 | * **technical details**: See [Technical notes](https://github.com/xuwang/aws-terraform#technical-notes). 35 | 36 | ## v0.1.0 37 | 38 | Initial release. 39 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2015 Xu Wang 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | ################### 2 | ## Customization ## 3 | ################### 4 | # Change here or use environment variables, e.g. export AWS_PROFILE=. 5 | 6 | # Default SHELL for make for consistency on different platforms 7 | SHELL := /bin/bash 8 | # Load environments 9 | include envs.sh 10 | 11 | # Default AWS profile and cluster name. Please choose cluster name carefully. It will used as prefix in many AWS resources to be created. 12 | AWS_PROFILE ?= NODEFAULT 13 | CLUSTER_NAME ?= NODEFAULT 14 | # Application repository. Automatically synced to /var/lib/apps every minute 15 | APP_REPOSITORY ?= https://github.com/dockerage/coreos-cluster-apps 16 | APP_REPOSITORY_DEPLOYKEY ?= '' 17 | 18 | # Domain: default domain for Route53 zone and a self-signed *.domain cert for default ELBs. 19 | APP_DOMAIN ?= 'example.com' 20 | 21 | # For get-ami.sh 22 | COREOS_UPDATE_CHANNEL ?= stable 23 | AWS_REGION ?= us-west-2 24 | VM_TYPE ?= hvm 25 | 26 | # All resources used in destroy_all, in the order of dependencies. 27 | # It doesn't hurt if a resource in the list is not created, but if it does, add 28 | # it to the list to make sure cleanup is done properly. 29 | ALL_RESOURCES := vault admiral worker etcd iam s3 elb-ci elb-gitlab elb_dockerhub efs rds route53 cloudtrail vpc 30 | 31 | # To prevent you from mistakenly using a wrong account (and end up destroying live environment), 32 | # a list of allowed AWS account IDs should be defined: 33 | #ALLOWED_ACCOUNT_IDS := "123456789012","012345678901" 34 | AWS_ACCOUNT := $(shell aws --profile ${AWS_PROFILE} iam get-user | jq -r ".User.Arn" | grep -Eo '[[:digit:]]{12}') 35 | AWS_USER := $(shell aws --profile ${AWS_PROFILE} iam get-user | jq -r ".User.UserName") 36 | ALLOWED_ACCOUNT_IDS := "$(AWS_ACCOUNT)" 37 | 38 | # Working Directories and files 39 | ROOT_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST)))) 40 | SCRIPTS := $(ROOT_DIR)scripts 41 | MODULES := $(ROOT_DIR)modules 42 | RESOURCES := $(ROOT_DIR)resources 43 | TF_RESOURCES := $(ROOT_DIR)resources/terraforms 44 | BUILD := $(ROOT_DIR)build-$(CLUSTER_NAME) 45 | BUILD_SUBDIRS = $(shell [ -d $(BUILD) ] && cd $(BUILD) && ls -d */ | tr '/' ' ') 46 | CONFIG := $(BUILD)/cloud-config 47 | CERTS := $(BUILD)/certs 48 | SITE_CERT := $(CERTS)/site.pem 49 | POLICIES := $(BUILD)/policies 50 | AMI_VAR := ami.tf 51 | 52 | # LOCKKEY to prevent multiple terraform runs. The private key for the lock will be put in $HOME/.aws/{LOCK_KEYNAME}.pem 53 | # which is used to valide if you own the lock. 54 | LOCK_KEYNAME := $(CLUSTER_NAME)-tfstate-lock 55 | 56 | # Default to confirm when applying Terraform changes to the infrastrucutre 57 | CONFIRM_TF_APPLY ?= YES 58 | 59 | # Terraform files 60 | TF_PROVIDER := provider.tf 61 | TF_DESTROY_PLAN_OUT := destroy.tfplan 62 | TF_APPLY_PLAN := apply.tfplan 63 | TF_STATE := terraform.tfstate 64 | 65 | # Terraform commands 66 | # Note: for production, set -refresh=true to be safe 67 | TF_APPLY := terraform apply -refresh=true 68 | # Note: for production, remove --force to confirm destroy. 69 | TF_DESTROY := terraform destroy -force 70 | TF_DESTROY_PLAN := terraform plan -destroy -refresh=true 71 | TF_GET := terraform get -update 72 | TF_GRAPH := terraform graph -module-depth=0 73 | TF_PLAN := terraform plan -module-depth=1 -refresh=true 74 | TF_SHOW := terraform show -module-depth=1 75 | TF_REFRESH := terraform refresh 76 | TF_TAINT := terraform taint -allow-missing 77 | TF_OUTPUT := terraform output 78 | 79 | # Git clone/pull command 80 | GIT_SSH_COMMAND := ssh -i /root/.ssh/git-sync-rsa.pem -o 'StrictHostKeyChecking no' 81 | 82 | # Comman separated list of cidr blocks to allow ssh; default to $(curl -s http://ipinfo.io/ip)/32 83 | # TF_VAR_allow_ssh_cidr := "$(shell curl -s http://ipinfo.io/ip)/32" 84 | TF_VAR_timestamp := $(shell date +%Y-%m-%d-%H%M) 85 | TF_VAR_iamuser := $(AWS_USER) 86 | 87 | ########################## 88 | ## End of customization ## 89 | ########################## 90 | 91 | export 92 | 93 | all: worker admiral 94 | 95 | help: 96 | @echo "Usage: make plan_ | | plan_destroy_ | destroy_" 97 | @echo "Or make show_ | graph" 98 | @echo "Or make plan_destroy_all | destroy_all" 99 | @echo "Available resources: cloudtrail vault vpc s3 route53 iam efs elb etcd worker admiral rds" 100 | @echo "For example: make plan_worker # to show what resources are planned for worker" 101 | 102 | lock: 103 | $(SCRIPTS)/session-lock.sh -l $(LOCK_KEYNAME) 104 | 105 | unlock: 106 | $(SCRIPTS)/session-lock.sh -u $(LOCK_KEYNAME) 107 | 108 | session_start: lock 109 | $(MAKE) pull_tf_state 110 | 111 | session_end: 112 | @if ! git diff-index --name-status --exit-code HEAD -- ; then \ 113 | echo "You have unpublished changes:"; exit 1 ; \ 114 | fi 115 | $(MAKE) push_tf_state 116 | $(SCRIPTS)/session-lock.sh -u $(LOCK_KEYNAME) && rm session_start 117 | 118 | plan_destroy_all: 119 | @echo $(BUILD_SUBDIRS) 120 | @rm -rf /tmp/$(CLUSTER_NAME); mkdir -p /tmp/$(CLUSTER_NAME) 121 | @$(foreach resource,$(BUILD_SUBDIRS),cd $(BUILD)/$(resource) && $(TF_DESTROY_PLAN) -out /tmp/$(CLUSTER_NAME)/$(resource)-destroy.plan 2> /tmp/destroy.err;) 122 | 123 | confirm: 124 | @echo "CONTINUE? [Y/N]: "; read ANSWER; \ 125 | if [ ! "$$ANSWER" = "Y" ]; then \ 126 | echo "Exiting." ; exit 1 ; \ 127 | fi 128 | 129 | destroy_all: | plan_destroy_all 130 | @for i in /tmp/$(CLUSTER_NAME)/*.plan; do $(TF_SHOW) $$i; done | grep -- - 131 | @$(eval total=$(shell for i in /tmp/$(CLUSTER_NAME)/*.plan; do $(TF_SHOW) $$i; done | grep -- - | wc -l)) 132 | @echo "" 133 | @echo "Will destroy $$total resources" 134 | @$(MAKE) confirm 135 | @for i in $(ALL_RESOURCES); do \ 136 | if [ -d $(BUILD)/$$i ]; then \ 137 | $(MAKE) "destroy_$$i"; \ 138 | fi ; \ 139 | done 140 | #rm -rf $(BUILD) 141 | 142 | destroy: 143 | @echo "Usage: make destroy_ | make plan_destroy_all | make destroy_all" 144 | @echo "For example: make destroy_worker" 145 | @echo "Node: destroy may fail because of outstanding dependences" 146 | 147 | graph: | $(BUILD) 148 | @mkdir -p $(BUILD)/graph 149 | @for i in $(ALL_RESOURCES); do \ 150 | if [ -d $(BUILD)/$$i ]; then \ 151 | cd $(BUILD)/$$i ; \ 152 | $(TF_GRAPH) | dot -Tpng > $(BUILD)/graph/$$i.png ; \ 153 | fi ; \ 154 | done 155 | 156 | plan: 157 | @echo "plan_" 158 | show_all: 159 | @$(foreach resource,$(BUILD_SUBDIRS),$(TF_SHOW) $(BUILD)/$(resource)/terraform.tfstate 2> /dev/null; ) 160 | 161 | # TODO: Push/Pull terraform states from a tf state repo 162 | # For team work, you need to commit terraform to a remote location, such as git repo, S3 163 | # Should implement a locking method to prevent alter infrastructure at the same time. 164 | pull_tf_state: 165 | @mkdir -p $(BUILD) 166 | @echo pull terraform state from ... 167 | #git pull --rebase 168 | 169 | push_tf_state: 170 | @echo push terraform state to .... 171 | #git push 172 | 173 | # Load all resouces makefile 174 | include resources/makefiles/*.mk 175 | 176 | .PHONY: all confirm destroy destroy_all graph lock unlock plan_destroy_all help pull_tf_state push_tf_state 177 | .NOTPARALLEL: 178 | 179 | -------------------------------------------------------------------------------- /README-vault.md: -------------------------------------------------------------------------------- 1 | 2 | This repo contains a vault cluster runs with etcd backend. By default, the vault and etcd are running 3 | on the same instance. 3 instances are created. 4 | 5 | With this setup, you can afford to lose one machine and still keep etcd healthy, so if needed, only reboot machines one at a time and 6 | check etcd health before reboot another. 7 | 8 | ## Get ips of the vault servers 9 | 10 | You can get ips of the vault servers by: 11 | 12 | ``` 13 | $ make get_vault_ips 14 | ``` 15 | 16 | The ssh access is open to your own machine's IP from where you built the vault. 17 | 18 | ## Initialize and unseal the vault 19 | 20 | You need to run Vault initialization on one of the vault servers. The master key and 5 unsealing keys will be stored in etcd K/V store. 21 | You need to run unseal process on all vault servers after its reboot. 22 | 23 | * Copy scripts/setup_vault.sh to all vault servers: 24 | 25 | ``` 26 | $ scp scripts/setup_vault.sh core@:/tmp 27 | ``` 28 | 29 | * Initialize and unseal the vault servers 30 | 31 | Run this on all vault servers - note it is okay to run setup_vault.multiple times. It will skip initialization if the vault is already 32 | initialized. 33 | ``` 34 | $ ssh core@ 35 | $ cd /tmp 36 | $ ./setup_vault.sh 37 | ``` 38 | 39 | ## Validate vault setup 40 | 41 | This doesn't need vault authentication 42 | 43 | ``` 44 | $ vault status 45 | core@ip-10-10-6-72-vault /tmp $vault status 46 | Sealed: false 47 | Key Shares: 5 48 | Key Threshold: 3 49 | Unseal Progress: 0 50 | Version: 0.6.4 51 | Cluster Name: vault-cluster-4932646e 52 | Cluster ID: b47b8de3-b609-e730-d2f2-c894d1c77ec8 53 | ``` 54 | Authenticate to vault and check mounts: 55 | 56 | ``` 57 | $ vault auth $(etcdctl get /service/vault/root-token) 58 | Successfully authenticated! You are now logged in. 59 | token: xxxxxx-ea73-1063-02bf-070f2ab60123 60 | token_duration: 0 61 | token_policies: [root] 62 | 63 | $ core@ip-10-10-6-72-vault /etc/profile.d $vault mounts 64 | Path Type Default TTL Max TTL Description 65 | cubbyhole/ cubbyhole n/a n/a per-token private secret storage 66 | secret/ generic system system generic secret storage 67 | sys/ system n/a n/a system endpoints used for control, policy and debugging 68 | ``` 69 | ## Validate vault elb health 70 | 71 | You can check on AWS EC2/Loadbalancer console, or run: 72 | 73 | ``` 74 | $ aws elb --profile describe-instance-health --load-balancer-name=vault --region 75 | ``` 76 | -------------------------------------------------------------------------------- /Vagrantfile: -------------------------------------------------------------------------------- 1 | # -*- mode: ruby -*- 2 | # vi: set ft=ruby : 3 | 4 | Vagrant.configure("2") do |config| 5 | config.vm.box = "ubuntu/trusty64" 6 | config.vm.define vm_name = "my-aws-terraform-box" 7 | 8 | 9 | # app ports 10 | # config.vm.network :forwarded_port, guest: 8080, host: 80, auto_correct: true 11 | # config.vm.network :forwarded_port, guest: 8443, host: 443, auto_correct: true 12 | # Create a private network, which allows host-only access to the machine 13 | # using a specific IP. 14 | config.vm.network :private_network, ip: "192.168.33.10" 15 | 16 | # Create a public network, which generally matched to bridged network. 17 | # Bridged networks make the machine appear as another physical device on 18 | # your network. 19 | # config.vm.network :public_network 20 | 21 | # Share an additional folder to the guest VM. The first argument is 22 | # the path on the host to the actual folder. The second argument is 23 | # the path on the guest to mount the folder. And the optional third 24 | # argument is a set of non-required options. 25 | config.vm.synced_folder ".", "/home/vagrant/aws-terraform" 26 | config.vm.synced_folder "~/.aws", "/home/vagrant/.aws" 27 | config.vm.provision "file", source: "~/.gitconfig", destination: ".gitconfig" 28 | 29 | # Get rid of stdin: not tty error 30 | config.ssh.shell = "bash -c 'BASH_ENV=/etc/profile exec bash'" 31 | 32 | # Provider-specific configuration so you can fine-tune various 33 | # backing providers for Vagrant. These expose provider-specific options. 34 | # Example for VirtualBox: 35 | # 36 | config.vm.provider :virtualbox do |vb| 37 | vb.customize [ 38 | "modifyvm", :id, 39 | "--memory", "2048" 40 | ] 41 | end 42 | config.vm.provision "shell", inline: $install_tools 43 | end 44 | 45 | $install_tools = <