├── .gitignore ├── Readme.md ├── consul ├── client.json ├── server.json ├── systemd-consul-client.service └── systemd-consul-server.service ├── jobs ├── echo-job.nomad ├── golang-redis-pg.nomad ├── hdfs.nomad ├── raw.nomad └── redis-job.nomad ├── nomad ├── client.hcl ├── server.hcl ├── systemd-nomad-client.service └── systemd-nomad-server.service ├── packer ├── packer.json ├── preseed.cfg └── provision.sh ├── terraform ├── main.tf ├── provision.sh └── terraform-provider-virtualbox └── vault ├── server.hcl └── systemd-vault-server.service /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | .vscode 3 | packer_cache 4 | output-virtualbox-iso 5 | output-virtualbox-iso-bak 6 | *.box 7 | terraform/example 8 | *tfstate* 9 | terraform/.terraform 10 | -------------------------------------------------------------------------------- /Readme.md: -------------------------------------------------------------------------------- 1 | # Local HashiCorp Stack 2 | 3 | ## Introduction 4 | 5 | This projects lets you run a 3 Server + 3 Client Nomad/Consul cluster in 6 Virtualbox VMs on OS X using Packer & Terraform 6 | 7 | ## Contents 8 | 9 | - [Motivation](#motivation) 10 | - [Prerequisites](#prerequisites) 11 | - [Build](#build) 12 | - [Deploy](#deploy) 13 | - [Jobs](#jobs) 14 | - [UI](#ui) 15 | - [HDFS](#hdfs) 16 | - [Spark](#spark) 17 | - [Vault](#vault) 18 | - [Attributions](#attributions) 19 | 20 | ## Motivation 21 | 22 | HashiCorp tools enable you to build/maintain multi-datacenter systems with ease. However, you usually don't have datacenters to play with. This project builds VirtualBox VMs that you can run Terraform against to play with Nomad, Consul, etc. 23 | 24 | The workflow is: 25 | - Build ISOs (Packer) 26 | - Deploy VMs to your local machine (Terraform + 3rd Party Provider) 27 | - Play with Nomad, Consul, etc. 28 | 29 | (Packer is used directly instead of Vagrant so the pipeline is the same when you build & deploy against hypervisors and clouds) 30 | 31 | ## Prerequisites 32 | 33 | - OS X 34 | - [Homebrew](https://brew.sh/) 35 | - `brew install packer terraform nomad` 36 | - `brew cask install virtualbox` 37 | 38 | ## Build 39 | 40 | ```bash 41 | cd packer 42 | packer build -on-error=abort -force packer.json 43 | cd output-virtualbox-iso 44 | tar -zcvf ubuntu-16.04-docker.box *.ovf *.vmdk 45 | cd ../.. 46 | ``` 47 | 48 | ## Deploy 49 | 50 | ```bash 51 | cd terraform 52 | # Remove any cached golden images before redeploying 53 | rm -rf ~/.terraform/virtualbox/gold/ubuntu-16.04-docker 54 | terraform init 55 | terraform apply 56 | cd .. 57 | ``` 58 | 59 | You can ssh onto a host by running: 60 | 61 | ```bash 62 | ssh -o 'IdentitiesOnly yes' packer@192.168.0.118 63 | # password: packer 64 | ``` 65 | 66 | ## Jobs 67 | 68 | Take the IP Address of the server deployment and run Nomad jobs: 69 | 70 | ```bash 71 | cd jobs 72 | nomad run -address http://192.168.0.118:4646 redis-job.nomad 73 | nomad run -address http://192.168.0.118:4646 echo-job.nomad 74 | nomad run -address http://192.168.0.118:4646 golang-redis-pg.nomad 75 | nomad run -address http://192.168.0.118:4646 raw.nomad 76 | cd .. 77 | ``` 78 | 79 | You can view the logs of an `allocation`: 80 | 81 | ```bash 82 | nomad logs -address http://192.168.0.118:4646 bf90d9cb 83 | ``` 84 | 85 | At a later time, you can stop the nomad jobs (but first look at [the UI](#ui)): 86 | 87 | ```bash 88 | cd jobs 89 | nomad stop -address http://192.168.0.118:4646 Echo-Job 90 | nomad stop -address http://192.168.0.118:4646 Redis-Job 91 | nomad stop -address http://192.168.0.118:4646 Golang-Redis-PG 92 | nomad stop -address http://192.168.0.118:4646 view_files 93 | cd .. 94 | ``` 95 | 96 | ## UI 97 | 98 | Using the IP Address of the server deployment, you can: 99 | 100 | - view the Nomad UI at: [http://192.168.0.118:4646/ui](http://192.168.0.118:4646/ui) 101 | - view the Consul UI at: [http://192.168.0.118:8500/ui](http://192.168.0.118:8500/ui) 102 | 103 | ## HDFS 104 | 105 | You can deploy HDFS by running: 106 | 107 | ```bash 108 | cd jobs 109 | nomad run -address http://192.168.0.118:4646 hdfs.nomad 110 | cd .. 111 | ``` 112 | 113 | (Give it a minute to download the docker image..) 114 | 115 | Then you can view the UI at: [http://192.168.0.118:50070/](http://192.168.0.118:50070/) 116 | 117 | ## Spark 118 | 119 | SSH into a server node then start PySpark: 120 | 121 | ```bash 122 | pyspark \ 123 | --master nomad \ 124 | --conf spark.executor.instances=2 \ 125 | --conf spark.nomad.datacenters=dc-1 \ 126 | --conf spark.nomad.sparkDistribution=local:///usr/local/bin/spark 127 | ``` 128 | 129 | Then run some PySpark commands: 130 | 131 | ```python 132 | df = spark.read.json("/usr/local/bin/spark/examples/src/main/resources/people.json") 133 | df.show() 134 | df.printSchema() 135 | df.createOrReplaceTempView("people") 136 | sqlDF = spark.sql("SELECT * FROM people") 137 | sqlDF.show() 138 | ``` 139 | 140 | ## Vault 141 | 142 | Init the Vault system and go through the process for 1 of the Vault servers 143 | 144 | ```bash 145 | vault init -address http://192.168.0.118:8200 146 | vault unseal -address http://192.168.0.118:8200 147 | vault auth -address=http://192.168.0.118:8200 66344296-222d-5be6-e052-15679209e0e7 148 | vault write -address=http://192.168.0.118:8200 secret/names name=ryan 149 | vault read -address=http://192.168.0.118:8200 secret/names 150 | ``` 151 | 152 | Then unseal the other Vault servers for HA 153 | 154 | ```bash 155 | vault unseal -address http://192.168.0.125:8200 156 | vault unseal -address http://192.168.0.161:8200 157 | ``` 158 | 159 | Then check Consul to see the health checks show that all the vault servers are unlocked 160 | 161 | ## Attributions 162 | 163 | - [https://github.com/geerlingguy/packer-ubuntu-1604](https://github.com/geerlingguy/packer-ubuntu-1604) 164 | - [https://github.com/ccll/terraform-provider-virtualbox](https://github.com/ccll/terraform-provider-virtualbox) 165 | - [https://github.com/hashicorp/nomad/tree/master/terraform](https://github.com/hashicorp/nomad/tree/master/terraform) 166 | -------------------------------------------------------------------------------- /consul/client.json: -------------------------------------------------------------------------------- 1 | { 2 | "log_level": "DEBUG", 3 | "data_dir": "/consul/data", 4 | "datacenter": "dc-1", 5 | "server": false, 6 | "retry_join": ["server-1", "server-2", "server-3"], 7 | "ports": { 8 | "dns": 53 9 | } 10 | } 11 | -------------------------------------------------------------------------------- /consul/server.json: -------------------------------------------------------------------------------- 1 | { 2 | "log_level": "DEBUG", 3 | "data_dir": "/consul/data", 4 | "datacenter": "dc-1", 5 | "server": true, 6 | "ui": true, 7 | "bootstrap_expect": 3, 8 | "addresses": { 9 | "http": "0.0.0.0" 10 | }, 11 | "retry_join": ["server-1", "server-2", "server-3"], 12 | "ports": { 13 | "dns": 53 14 | } 15 | } 16 | -------------------------------------------------------------------------------- /consul/systemd-consul-client.service: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=Consul 3 | Documentation=https://consul.io/docs/ 4 | 5 | [Service] 6 | Restart=on-failure 7 | ExecStartPre=/bin/bash -c "/bin/systemctl set-environment CONSUL_ADDR=$(ifconfig eth0 | grep 'inet addr' | cut -d ':' -f 2 | cut -d ' ' -f 1)" 8 | ExecStart=/bin/consul agent -config-file /consul/client.json -bind ${CONSUL_ADDR} 9 | ExecReload=/bin/kill -HUP $MAINPID 10 | LimitNOFILE=65536 11 | KillSignal=SIGTERM 12 | User=root 13 | Group=root 14 | 15 | [Install] 16 | WantedBy=multi-user.target 17 | -------------------------------------------------------------------------------- /consul/systemd-consul-server.service: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=Consul 3 | Documentation=https://consul.io/docs/ 4 | 5 | [Service] 6 | Restart=on-failure 7 | ExecStartPre=/bin/bash -c "/bin/systemctl set-environment CONSUL_ADDR=$(ifconfig eth0 | grep 'inet addr' | cut -d ':' -f 2 | cut -d ' ' -f 1)" 8 | ExecStart=/bin/consul agent -config-file /consul/server.json -bind ${CONSUL_ADDR} 9 | ExecReload=/bin/kill -HUP $MAINPID 10 | LimitNOFILE=65536 11 | KillSignal=SIGTERM 12 | User=root 13 | Group=root 14 | 15 | [Install] 16 | WantedBy=multi-user.target 17 | -------------------------------------------------------------------------------- /jobs/echo-job.nomad: -------------------------------------------------------------------------------- 1 | job "Echo-Job" { 2 | datacenters = ["dc-1"] 3 | 4 | group "Echo-Group" { 5 | task "Echo-Task" { 6 | driver = "docker" 7 | 8 | config { 9 | image = "hashicorp/http-echo" 10 | privileged = true 11 | network_mode = "host" 12 | 13 | args = [ 14 | "-listen", ":5678", 15 | "-text", "hello world", 16 | ] 17 | } 18 | 19 | service { 20 | name = "EchoServer" 21 | port = "echoPort" 22 | check { 23 | type = "http" 24 | port = "echoPort" 25 | path = "/" 26 | interval = "15s" 27 | timeout = "10s" 28 | } 29 | } 30 | 31 | resources { 32 | network { 33 | port "echoPort" { 34 | static = "5678" 35 | } 36 | } 37 | } 38 | } 39 | } 40 | } 41 | -------------------------------------------------------------------------------- /jobs/golang-redis-pg.nomad: -------------------------------------------------------------------------------- 1 | job "Golang-Redis-PG" { 2 | datacenters = ["dc-1"] 3 | 4 | group "Golang-Group" { 5 | count = 1 6 | 7 | task "Golang-Task" { 8 | driver = "docker" 9 | 10 | env { 11 | GLP_LISTEN_PORT = 9998 12 | GLP_REDIS_HOST = "RedisServer.service.dc-1.consul" 13 | GLP_REDIS_PORT = 6379 14 | GLP_PG_HOST = "PostgresServer.service.dc-1.consul" 15 | GLP_PG_PORT = 5432 16 | GLP_PG_USER = "postgres" 17 | GLP_PG_PASS = "password" 18 | GLP_PG_DB = "postgres" 19 | } 20 | 21 | config { 22 | image = "rms1000watt/golang-redis-pg:latest" 23 | network_mode = "host" 24 | dns_servers = ["127.0.0.1"] 25 | } 26 | 27 | service { 28 | name = "GolangServer" 29 | port = "golangPort" 30 | check { 31 | type = "tcp" 32 | port = "golangPort" 33 | interval = "15s" 34 | timeout = "10s" 35 | } 36 | } 37 | 38 | resources { 39 | network { 40 | port "golangPort" { 41 | static = "9998" 42 | } 43 | } 44 | } 45 | } 46 | } 47 | 48 | group "Redis-Group" { 49 | task "Redis-Task" { 50 | driver = "docker" 51 | 52 | config { 53 | image = "redis:4.0.6-alpine" 54 | network_mode = "host" 55 | } 56 | 57 | service { 58 | name = "RedisServer" 59 | port = "redisPort" 60 | check { 61 | type = "tcp" 62 | port = "redisPort" 63 | interval = "15s" 64 | timeout = "10s" 65 | } 66 | } 67 | 68 | resources { 69 | network { 70 | port "redisPort" { 71 | static = "6379" 72 | } 73 | } 74 | } 75 | } 76 | } 77 | 78 | group "Postgres-Group" { 79 | task "Postgres-Task" { 80 | driver = "docker" 81 | 82 | env { 83 | POSTGRES_USER = "postgres" 84 | POSTGRES_PASSWORD = "password" 85 | } 86 | 87 | config { 88 | image = "postgres:9.6.6-alpine" 89 | privileged = true 90 | network_mode = "host" 91 | volumes = [ "local/postgres:/var/lib/postgresql/data" ] 92 | } 93 | 94 | service { 95 | name = "PostgresServer" 96 | port = "postgresPort" 97 | check { 98 | type = "tcp" 99 | port = "postgresPort" 100 | interval = "15s" 101 | timeout = "10s" 102 | } 103 | } 104 | 105 | resources { 106 | memory = 512 107 | network { 108 | port "postgresPort" { 109 | static = "5432" 110 | } 111 | } 112 | } 113 | } 114 | } 115 | } -------------------------------------------------------------------------------- /jobs/hdfs.nomad: -------------------------------------------------------------------------------- 1 | job "hdfs" { 2 | 3 | datacenters = [ "dc-1" ] 4 | 5 | group "NameNode" { 6 | 7 | constraint { 8 | operator = "distinct_hosts" 9 | value = "true" 10 | } 11 | 12 | task "NameNode" { 13 | 14 | driver = "docker" 15 | 16 | config { 17 | image = "rcgenova/hadoop-2.7.3" 18 | command = "bash" 19 | args = [ "-c", "hdfs namenode -format && exec hdfs namenode -D fs.defaultFS=hdfs://${NOMAD_ADDR_ipc}/ -D dfs.permissions.enabled=false" ] 20 | network_mode = "host" 21 | port_map { 22 | ipc = 8020 23 | ui = 50070 24 | } 25 | dns_servers = ["127.0.0.1"] 26 | } 27 | 28 | resources { 29 | memory = 256 30 | network { 31 | port "ipc" { 32 | static = "8020" 33 | } 34 | port "ui" { 35 | static = "50070" 36 | } 37 | } 38 | } 39 | 40 | service { 41 | name = "hdfs" 42 | port = "ipc" 43 | } 44 | } 45 | } 46 | 47 | group "DataNode" { 48 | 49 | count = 2 50 | 51 | constraint { 52 | operator = "distinct_hosts" 53 | value = "true" 54 | } 55 | 56 | task "DataNode" { 57 | 58 | driver = "docker" 59 | 60 | config { 61 | network_mode = "host" 62 | image = "rcgenova/hadoop-2.7.3" 63 | args = [ "hdfs", "datanode" 64 | , "-D", "fs.defaultFS=hdfs://hdfs.service.consul/" 65 | , "-D", "dfs.permissions.enabled=false" 66 | ] 67 | port_map { 68 | data = 50010 69 | ipc = 50020 70 | ui = 50075 71 | } 72 | dns_servers = ["127.0.0.1"] 73 | } 74 | 75 | resources { 76 | memory = 256 77 | network { 78 | port "data" { 79 | static = "50010" 80 | } 81 | port "ipc" { 82 | static = "50020" 83 | } 84 | port "ui" { 85 | static = "50075" 86 | } 87 | } 88 | } 89 | 90 | } 91 | } 92 | 93 | } 94 | -------------------------------------------------------------------------------- /jobs/raw.nomad: -------------------------------------------------------------------------------- 1 | job "view_files" { 2 | type = "system" 3 | datacenters = ["dc-1"] 4 | 5 | group "view_files"{ 6 | restart { 7 | mode = "fail" 8 | } 9 | 10 | task "view_files" { 11 | driver = "raw_exec" 12 | 13 | config { 14 | command = "ls" 15 | args = ["-la"] 16 | } 17 | } 18 | } 19 | } -------------------------------------------------------------------------------- /jobs/redis-job.nomad: -------------------------------------------------------------------------------- 1 | job "Redis-Job" { 2 | datacenters = ["dc-1"] 3 | 4 | group "Redis-Group" { 5 | task "Redis-Task" { 6 | driver = "docker" 7 | 8 | config { 9 | image = "redis" 10 | privileged = true 11 | network_mode = "host" 12 | } 13 | 14 | service { 15 | name = "RedisServer" 16 | port = "redisPort" 17 | check { 18 | type = "tcp" 19 | port = "redisPort" 20 | interval = "15s" 21 | timeout = "10s" 22 | } 23 | } 24 | 25 | resources { 26 | network { 27 | port "redisPort" { 28 | static = "6379" 29 | } 30 | } 31 | } 32 | } 33 | } 34 | } 35 | -------------------------------------------------------------------------------- /nomad/client.hcl: -------------------------------------------------------------------------------- 1 | log_level = "DEBUG" 2 | data_dir = "/nomad/data" 3 | datacenter = "dc-1" 4 | 5 | client { 6 | enabled = true 7 | options { 8 | "driver.raw_exec.enable" = "true" 9 | "docker.privileged.enabled" = "true" 10 | } 11 | } 12 | 13 | ports { 14 | http = 4656 15 | } 16 | -------------------------------------------------------------------------------- /nomad/server.hcl: -------------------------------------------------------------------------------- 1 | log_level = "DEBUG" 2 | data_dir = "/nomad/data" 3 | datacenter = "dc-1" 4 | 5 | server { 6 | enabled = true 7 | bootstrap_expect = 3 8 | 9 | retry_join = ["server-1", "server-2", "server-3"] 10 | } 11 | 12 | ports { 13 | http = 4646 14 | rpc = 4647 15 | serf = 4648 16 | } -------------------------------------------------------------------------------- /nomad/systemd-nomad-client.service: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=Nomad 3 | Documentation=https://nomadproject.io/docs/ 4 | 5 | [Service] 6 | Restart=on-failure 7 | ExecStart=/bin/nomad agent -config /nomad/client.hcl 8 | ExecReload=/bin/kill -HUP $MAINPID 9 | LimitNOFILE=65536 10 | KillSignal=SIGTERM 11 | User=root 12 | Group=root 13 | 14 | [Install] 15 | WantedBy=multi-user.target 16 | -------------------------------------------------------------------------------- /nomad/systemd-nomad-server.service: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=Nomad 3 | Documentation=https://nomadproject.io/docs/ 4 | 5 | [Service] 6 | Restart=on-failure 7 | ExecStart=/bin/nomad agent -config /nomad/server.hcl 8 | ExecReload=/bin/kill -HUP $MAINPID 9 | LimitNOFILE=65536 10 | KillSignal=SIGTERM 11 | User=root 12 | Group=root 13 | 14 | [Install] 15 | WantedBy=multi-user.target 16 | -------------------------------------------------------------------------------- /packer/packer.json: -------------------------------------------------------------------------------- 1 | { 2 | "variables": { 3 | "ubuntu_vm_name": "ubuntu-16.04-docker", 4 | "cpu": "2", 5 | "ram": "1024", 6 | "disk": "40960", 7 | "iso_url": "http://releases.ubuntu.com/16.04/ubuntu-16.04.6-server-amd64.iso", 8 | "iso_checksum": "ac8a79a86a905ebdc3ef3f5dd16b7360", 9 | "iso_checksum_type": "md5", 10 | "user": "packer", 11 | "pass": "packer" 12 | }, 13 | "builders": [ 14 | { 15 | "type": "virtualbox-iso", 16 | "guest_os_type": "Ubuntu_64", 17 | "vm_name": "{{user `ubuntu_vm_name`}}", 18 | "disk_size": "{{user `disk`}}", 19 | "headless": false, 20 | "http_directory": ".", 21 | "ssh_wait_timeout": "30m", 22 | "iso_url": "{{user `iso_url`}}", 23 | "iso_checksum": "{{user `iso_checksum`}}", 24 | "iso_checksum_type": "{{user `iso_checksum_type`}}", 25 | "ssh_username": "{{user `user`}}", 26 | "ssh_password": "{{user `pass`}}", 27 | "vboxmanage": [ 28 | ["modifyvm", "{{.Name}}", "--memory", "{{user `ram`}}"], 29 | ["modifyvm", "{{.Name}}", "--cpus", "{{user `cpu`}}"] 30 | ], 31 | "shutdown_command": "echo 'packer' | sudo -S shutdown -P now", 32 | "boot_command": [ 33 | "", 34 | "", 35 | "", 36 | "", 37 | "/install/vmlinuz", 38 | " auto", 39 | " console-setup/ask_detect=false", 40 | " console-setup/layoutcode=us", 41 | " console-setup/modelcode=pc105", 42 | " debconf/frontend=noninteractive", 43 | " debian-installer=en_US", 44 | " fb=false", 45 | " initrd=/install/initrd.gz", 46 | " kbd-chooser/method=us", 47 | " keyboard-configuration/layout=USA", 48 | " keyboard-configuration/variant=USA", 49 | " locale=en_US", 50 | " netcfg/get_domain=vm", 51 | " netcfg/get_hostname=ubuntu", 52 | " grub-installer/bootdev=/dev/sda", 53 | " noapic", 54 | " preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/preseed.cfg", 55 | " -- ", 56 | "" 57 | ] 58 | } 59 | ], 60 | "provisioners": [ 61 | { 62 | "type": "shell", 63 | "execute_command": "echo 'packer' | sudo -S -E sh {{.Path}}", 64 | "scripts": [ 65 | "./provision.sh" 66 | ] 67 | } 68 | ] 69 | } 70 | -------------------------------------------------------------------------------- /packer/preseed.cfg: -------------------------------------------------------------------------------- 1 | choose-mirror-bin mirror/http/proxy string 2 | d-i base-installer/kernel/override-image string linux-server 3 | d-i clock-setup/utc boolean true 4 | d-i clock-setup/utc-auto boolean true 5 | d-i finish-install/reboot_in_progress note 6 | d-i grub-installer/only_debian boolean true 7 | d-i grub-installer/with_other_os boolean true 8 | d-i partman-auto-lvm/guided_size string max 9 | d-i partman-auto/choose_recipe select atomic 10 | d-i partman-auto/method string lvm 11 | d-i partman-lvm/confirm boolean true 12 | d-i partman-lvm/confirm boolean true 13 | d-i partman-lvm/confirm_nooverwrite boolean true 14 | d-i partman-lvm/device_remove_lvm boolean true 15 | d-i partman/choose_partition select finish 16 | d-i partman/confirm boolean true 17 | d-i partman/confirm_nooverwrite boolean true 18 | d-i partman/confirm_write_new_label boolean true 19 | d-i pkgsel/include string openssh-server cryptsetup build-essential libssl-dev libreadline-dev zlib1g-dev linux-source dkms nfs-common 20 | d-i pkgsel/install-language-support boolean false 21 | d-i pkgsel/update-policy select none 22 | d-i pkgsel/upgrade select full-upgrade 23 | d-i time/zone string UTC 24 | tasksel tasksel/first multiselect standard, ubuntu-server 25 | 26 | d-i console-setup/ask_detect boolean false 27 | d-i keyboard-configuration/layoutcode string us 28 | d-i keyboard-configuration/modelcode string pc105 29 | d-i debian-installer/locale string en_US 30 | 31 | # Create packer user account. 32 | d-i passwd/user-fullname string packer 33 | d-i passwd/username string packer 34 | d-i passwd/user-password password packer 35 | d-i passwd/user-password-again password packer 36 | d-i user-setup/allow-password-weak boolean true 37 | d-i user-setup/encrypt-home boolean false 38 | d-i passwd/user-default-groups packer sudo 39 | d-i passwd/user-uid string 900 40 | -------------------------------------------------------------------------------- /packer/provision.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # Vars 4 | DOCKER_VERSION=17.06.2~ce-0~ubuntu 5 | DOCKER_COMPOSE_VERSION=1.16.1 6 | NOMAD_VERSION=0.7.0 7 | CONSUL_VERSION=1.0.0 8 | VAULT_VERSION=0.9.1 9 | HADOOP_VERSION=2.7.4 10 | 11 | # Passwordless sudo 12 | echo "packer ALL=NOPASSWD:ALL" > /etc/sudoers.d/packer 13 | 14 | # Update grub to use eth0, eth1 naming convention 15 | sed -i 's|GRUB_CMDLINE_LINUX=""|GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0"|g' /etc/default/grub 16 | update-grub 17 | 18 | # Add eth0, eth1 network interfaces 19 | echo "auto eth0" >> /etc/network/interfaces 20 | echo "iface eth0 inet dhcp" >> /etc/network/interfaces 21 | 22 | echo "auto eth1" >> /etc/network/interfaces 23 | echo "iface eth1 inet dhcp" >> /etc/network/interfaces 24 | 25 | # Install dependencies 26 | apt-get -y install software-properties-common 27 | apt-get update 28 | apt-get install -y curl wget unzip tree redis-tools jq 29 | 30 | # Install VirtualBox Guest Tools 31 | apt-get -y install virtualbox-guest-dkms 32 | 33 | # Numpy (for Spark) 34 | apt-get install -y python-setuptools 35 | easy_install pip 36 | pip install numpy 37 | 38 | # Consul 39 | if [ ! -f /bin/consul ]; then 40 | curl -L -o consul_${CONSUL_VERSION}_linux_amd64.zip https://releases.hashicorp.com/consul/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip 41 | unzip -d /bin consul_${CONSUL_VERSION}_linux_amd64.zip 42 | rm consul_${CONSUL_VERSION}_linux_amd64.zip 43 | chmod +x /bin/consul 44 | fi 45 | 46 | # Nomad 47 | if [ ! -f /bin/nomad ]; then 48 | curl -L -o nomad_${NOMAD_VERSION}_linux_amd64.zip https://releases.hashicorp.com/nomad/${NOMAD_VERSION}/nomad_${NOMAD_VERSION}_linux_amd64.zip 49 | unzip -d /bin nomad_${NOMAD_VERSION}_linux_amd64.zip 50 | rm nomad_${NOMAD_VERSION}_linux_amd64.zip 51 | chmod +x /bin/nomad 52 | fi 53 | 54 | # Vault 55 | if [ ! -f /bin/vault ]; then 56 | curl -L -o vault_${VAULT_VERSION}_linux_amd64.zip https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip 57 | unzip -d /bin vault_${VAULT_VERSION}_linux_amd64.zip 58 | rm vault_${VAULT_VERSION}_linux_amd64.zip 59 | chmod +x /bin/vault 60 | fi 61 | 62 | # Install Docker 63 | if [ ! -f /usr/bin/docker ]; then 64 | curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - 65 | add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" 66 | apt-get update 67 | apt-get -y install docker-ce=$DOCKER_VERSION 68 | gpasswd -a packer docker 69 | newgrp docker 70 | fi 71 | 72 | # Install Docker Compose 73 | if [ ! -f /usr/local/bin/docker-compose ]; then 74 | curl -L https://github.com/docker/compose/releases/download/$DOCKER_COMPOSE_VERSION/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose 75 | chmod +x /usr/local/bin/docker-compose 76 | fi 77 | 78 | # Java 79 | sudo add-apt-repository -y ppa:openjdk-r/ppa 80 | sudo apt-get update 81 | sudo apt-get install -y openjdk-8-jdk 82 | JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::") 83 | 84 | # Spark with Nomad scheduling 85 | if [ ! -f /usr/local/bin/spark ]; then 86 | sudo wget -P /ops/examples/spark https://s3.amazonaws.com/nomad-spark/spark-2.2.0-bin-nomad-0.7.0.tgz 87 | sudo tar -xvf /ops/examples/spark/spark-2.2.0-bin-nomad-0.7.0.tgz --directory /ops/examples/spark 88 | sudo mv /ops/examples/spark/spark-2.2.0-bin-nomad-0.7.0 /usr/local/bin/spark 89 | sudo chown -R root:root /usr/local/bin/spark 90 | fi 91 | 92 | # Hadoop (to enable the HDFS CLI) 93 | wget -O - http://apache.mirror.iphh.net/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz | sudo tar xz -C /usr/local/ 94 | 95 | # Update ~/.bashrc for Spark 96 | echo "export PATH=$PATH:/usr/local/bin/spark/bin:/usr/local/hadoop-$HADOOP_VERSION/bin" | sudo tee --append /home/packer/.bashrc 97 | echo "export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre" | sudo tee --append /home/packer/.bashrc 98 | -------------------------------------------------------------------------------- /terraform/main.tf: -------------------------------------------------------------------------------- 1 | resource "virtualbox_vm" "server" { 2 | count = 3 3 | name = "${format("server-%02d", count.index+1)}" 4 | image = "../packer/output-virtualbox-iso/ubuntu-16.04-docker.box" 5 | cpus = 1 6 | memory = "512 mb" 7 | network_adapter { 8 | type = "bridged" 9 | host_interface = "en0" 10 | } 11 | } 12 | 13 | resource "virtualbox_vm" "client" { 14 | count = 3 15 | name = "${format("client-%02d", count.index+1)}" 16 | image = "../packer/output-virtualbox-iso/ubuntu-16.04-docker.box" 17 | cpus = 1 18 | memory = "2048 mb" 19 | network_adapter { 20 | type = "bridged" 21 | host_interface = "en0" 22 | } 23 | } 24 | 25 | resource "null_resource" "server-provisioner" { 26 | # Run server-provisioner if the mac address changes (if machine is deleted not just turned off) 27 | triggers { 28 | mac_addresses = "${join(",", virtualbox_vm.server.*.network_adapter.0.mac_address)}" 29 | } 30 | 31 | count = "${virtualbox_vm.server.count}" 32 | connection { 33 | type = "ssh" 34 | host = "${element(virtualbox_vm.server.*.network_adapter.0.ipv4_address, count.index)}" 35 | user = "packer" 36 | password = "packer" 37 | timeout = "1m" 38 | } 39 | 40 | provisioner "remote-exec" { 41 | inline = [ 42 | "sudo mkdir /nomad", 43 | "sudo mkdir /consul", 44 | "sudo mkdir /vault", 45 | "sudo chmod 777 /nomad", 46 | "sudo chmod 777 /consul", 47 | "sudo chmod 777 /vault", 48 | ] 49 | } 50 | 51 | provisioner "file" { 52 | source = "../nomad/server.hcl" 53 | destination = "/nomad/server.hcl" 54 | } 55 | 56 | provisioner "file" { 57 | source = "../nomad/systemd-nomad-server.service" 58 | destination = "/nomad/systemd-nomad-server.service" 59 | } 60 | 61 | provisioner "file" { 62 | source = "../consul/server.json" 63 | destination = "/consul/server.json" 64 | } 65 | 66 | provisioner "file" { 67 | source = "../consul/systemd-consul-server.service" 68 | destination = "/consul/systemd-consul-server.service" 69 | } 70 | 71 | provisioner "file" { 72 | source = "../vault/server.hcl" 73 | destination = "/vault/server.hcl" 74 | } 75 | 76 | provisioner "file" { 77 | source = "../vault/systemd-vault-server.service" 78 | destination = "/vault/systemd-vault-server.service" 79 | } 80 | 81 | provisioner "file" { 82 | source = "provision.sh" 83 | destination = "/home/packer/provision.sh" 84 | } 85 | 86 | provisioner "remote-exec" { 87 | inline = [ 88 | "echo '${element(virtualbox_vm.server.*.network_adapter.0.ipv4_address, 1)} server-1' | sudo tee -a /etc/hosts", 89 | "echo '${element(virtualbox_vm.server.*.network_adapter.0.ipv4_address, 2)} server-2' | sudo tee -a /etc/hosts", 90 | "echo '${element(virtualbox_vm.server.*.network_adapter.0.ipv4_address, 3)} server-3' | sudo tee -a /etc/hosts", 91 | "sudo mv /nomad/systemd-nomad-server.service /etc/systemd/system/nomad.service", 92 | "sudo mv /consul/systemd-consul-server.service /etc/systemd/system/consul.service", 93 | "sudo mv /vault/systemd-vault-server.service /etc/systemd/system/vault.service", 94 | "chmod a+x /home/packer/provision.sh", 95 | "sudo /home/packer/provision.sh", 96 | ] 97 | } 98 | } 99 | 100 | resource "null_resource" "client-provisioner" { 101 | # Run client-provisioner if the mac address changes (if machine is deleted not just turned off) 102 | triggers { 103 | mac_addresses = "${join(",", virtualbox_vm.client.*.network_adapter.0.mac_address)}" 104 | } 105 | 106 | count = "${virtualbox_vm.client.count}" 107 | connection { 108 | type = "ssh" 109 | host = "${element(virtualbox_vm.client.*.network_adapter.0.ipv4_address, count.index)}" 110 | user = "packer" 111 | password = "packer" 112 | timeout = "1m" 113 | } 114 | 115 | provisioner "remote-exec" { 116 | inline = [ 117 | "sudo mkdir /nomad", 118 | "sudo mkdir /consul", 119 | "sudo chmod 777 /nomad", 120 | "sudo chmod 777 /consul", 121 | ] 122 | } 123 | 124 | provisioner "file" { 125 | source = "../nomad/client.hcl" 126 | destination = "/nomad/client.hcl" 127 | } 128 | 129 | provisioner "file" { 130 | source = "../nomad/systemd-nomad-client.service" 131 | destination = "/nomad/systemd-nomad-client.service" 132 | } 133 | 134 | provisioner "file" { 135 | source = "../consul/client.json" 136 | destination = "/consul/client.json" 137 | } 138 | 139 | provisioner "file" { 140 | source = "../consul/systemd-consul-client.service" 141 | destination = "/consul/systemd-consul-client.service" 142 | } 143 | 144 | provisioner "file" { 145 | source = "provision.sh" 146 | destination = "/home/packer/provision.sh" 147 | } 148 | 149 | provisioner "remote-exec" { 150 | inline = [ 151 | "echo '${element(virtualbox_vm.server.*.network_adapter.0.ipv4_address, 1)} server-1' | sudo tee -a /etc/hosts", 152 | "echo '${element(virtualbox_vm.server.*.network_adapter.0.ipv4_address, 2)} server-2' | sudo tee -a /etc/hosts", 153 | "echo '${element(virtualbox_vm.server.*.network_adapter.0.ipv4_address, 3)} server-3' | sudo tee -a /etc/hosts", 154 | "sudo mv /nomad/systemd-nomad-client.service /etc/systemd/system/nomad.service", 155 | "sudo mv /consul/systemd-consul-client.service /etc/systemd/system/consul.service", 156 | "chmod a+x /home/packer/provision.sh", 157 | "sudo /home/packer/provision.sh", 158 | ] 159 | } 160 | } 161 | 162 | output "Servers" { 163 | value = ["${virtualbox_vm.server.*.network_adapter.0.ipv4_address}"] 164 | } 165 | 166 | output "Clients" { 167 | value = ["${virtualbox_vm.client.*.network_adapter.0.ipv4_address}"] 168 | } 169 | -------------------------------------------------------------------------------- /terraform/provision.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # Update hostname 4 | NEW_HOSTNAME=$(ifconfig eth0 | grep 'inet addr' | cut -d ':' -f 2 | cut -d ' ' -f 1 | sed -e 's|\.|-|g' | awk '{printf "ip-%s", $0}') 5 | hostname $NEW_HOSTNAME 6 | sed -i 's|ubuntu|'"$NEW_HOSTNAME"'|g' /etc/hostname 7 | sed -i 's|ubuntu|'"$NEW_HOSTNAME"'|g' /etc/hosts 8 | systemctl restart systemd-logind.service 9 | hostnamectl set-hostname $NEW_HOSTNAME 10 | 11 | # Update resolv.conf 12 | echo "nameserver 127.0.0.1" | tee -a /etc/resolvconf/resolv.conf.d/head 13 | resolvconf -u 14 | 15 | # Start Consul 16 | systemctl enable consul.service 17 | systemctl start consul 18 | 19 | # Start Nomad 20 | systemctl enable nomad.service 21 | systemctl start nomad 22 | 23 | # Start Vault 24 | systemctl enable vault.service 25 | systemctl start vault 26 | 27 | # TODO: Remove this and provision at Packer time 28 | # Update ~/.bashrc for Spark 29 | if ! grep -q "/usr/local/bin/spark/bin" ~/.bashrc; then 30 | echo "export PATH=$PATH:/usr/local/bin/spark/bin:/usr/local/hadoop-2.7.4/bin" | sudo tee -a ~/.bashrc 31 | fi 32 | 33 | if ! grep -q "java-8-openjdk-amd64" ~/.bashrc; then 34 | echo "export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre" | sudo tee -a ~/.bashrc 35 | fi 36 | 37 | # Update ~/.bashrc for Spark 38 | if ! grep -q "NOMAD_ADDR" ~/.bashrc; then 39 | echo "export NOMAD_ADDR=http://server-1:4646" | sudo tee -a ~/.bashrc 40 | fi 41 | -------------------------------------------------------------------------------- /terraform/terraform-provider-virtualbox: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rms1000watt/local-hashicorp-stack/fbd9853c088e7340330cef46371a2335fa3f9553/terraform/terraform-provider-virtualbox -------------------------------------------------------------------------------- /vault/server.hcl: -------------------------------------------------------------------------------- 1 | storage "consul" { 2 | address = "127.0.0.1:8500" 3 | path = "vault" 4 | } 5 | 6 | listener "tcp" { 7 | address = "0.0.0.0:8200" 8 | tls_disable = 1 9 | } 10 | -------------------------------------------------------------------------------- /vault/systemd-vault-server.service: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=Vault 3 | Documentation=https://vaultproject.io/docs/ 4 | 5 | [Service] 6 | Restart=on-failure 7 | ExecStart=/bin/vault server -config /vault/server.hcl 8 | ExecReload=/bin/kill -HUP $MAINPID 9 | LimitNOFILE=65536 10 | KillSignal=SIGTERM 11 | User=root 12 | Group=root 13 | 14 | [Install] 15 | WantedBy=multi-user.target 16 | --------------------------------------------------------------------------------