├── .gitignore ├── LICENSE.txt ├── README.md ├── packer-fdb ├── packer.json └── upload │ ├── policy-rc.d │ └── ubuntu.sh ├── packer-tester ├── packer.json └── upload │ └── ubuntu.sh └── terraform ├── .gitignore ├── init-fdb.sh ├── init-tester.sh ├── main.tf ├── makefile ├── outputs.tf └── variables.tf /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | .idea 3 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 BitGN and Contributors 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # About 2 | 3 | The purpose of this project is to make it easy to run various load 4 | tests on systems with FoundationDB in the cloud. 5 | 6 | We can achieve that by: 7 | 8 | 1. "Prebaking" cloud-specific VM images for the FoundationDB cluster 9 | nodes along with the tester nodes and machines for capturing the 10 | telemetry. 11 | 2. Using these images to quickly create FoundationDB clusters with the 12 | specific configuration. 13 | 14 | The first step is handled by the [Packer](https://www.packer.io), the 15 | second - by the [Terraform](https://www.terraform.io). 16 | 17 | We want to start with a fixed topology: 18 | 19 | - foundationDB nodes in the same network, forming a cluster; 20 | - tester machines connected to the FDB. 21 | 22 | Number and VM type for the node and load tester machines could be 23 | changed within the configuration. 24 | 25 | 26 | # Creating AMIs 27 | 28 | First you need to create packer images for FDB and tester 29 | machines. 30 | 31 | Note, that in this setup we actually create a temporary EC2 instance 32 | in the cloud, install all the dependencies there, make a snapshot 33 | (AMI) and kill the original machine. With these snapshots we could 34 | then quickly launch a dozen of EC2 instances, connecting them into a 35 | cluster. 36 | 37 | You can create a FoundationDB AMI like this (make sure to fill in your 38 | AWS credentials): 39 | 40 | ``` 41 | $ export AWS_ACCESS_KEY="" 42 | $ export AWS_SECRET_KEY="" 43 | $ cd packer-fdb 44 | $ packer build --only=aws packer.json 45 | ``` 46 | 47 | Tester AMI can be created exactly like this but you enter 48 | `packer-tester` folder instead. 49 | 50 | ## Creating Docker 51 | 52 | Should you want to create docker images instead of AMIs, you can do so 53 | via: 54 | 55 | ``` 56 | $ cd packer-fdb 57 | $ packer build --only=docker packer.json 58 | ``` 59 | 60 | # Deploying Clusters 61 | 62 | ## Setup Terraform 63 | 64 | Before deploying pre-baked images into AWS you need to configure your 65 | working copy first. Install Terraform, then go to the `terraform` 66 | folder of this repository and execute: 67 | 68 | ``` 69 | $ terraform init 70 | ``` 71 | 72 | Then, you would need to create a file `.secret.aws.tfvars`, filling it 73 | with your AWS credentials: 74 | 75 | ``` 76 | aws_access_key = "" 77 | aws_secret_key = "" 78 | aws_account_id = "" 79 | ``` 80 | 81 | Afterwards you would also need to create a new ssh key called 82 | `terraform` and place it into your `~/.ssh/` folder. Terraform will 83 | install it into all the new machines, making it possible for you to 84 | connect to them via ssh. 85 | 86 | You can do that with: 87 | 88 | ``` 89 | $ ssh-keygen -t rsa -b 4096 -C "terraform" -f "$HOME/.ssh/terraform" 90 | ``` 91 | 92 | ## Deploy a cluster 93 | 94 | In order to deploy a cluster you would need to execute the following 95 | in the `terraform` folder: 96 | 97 | ``` 98 | # prepare terraform plan, using a separate file with the credentials 99 | $ terraform plan -var-file=.secret.aws.tfvars -out my.plan 100 | # carry out the plan 101 | $ terraform apply "my.plan" 102 | ``` 103 | 104 | The process should take 3-4 minutes and print out in the end something 105 | like this: 106 | 107 | ``` 108 | aws_instance.fdb[0]: Creation complete after 3m32s (ID: i-0b90a62a90636c1ad) 109 | 110 | Apply complete! Resources: 12 added, 0 changed, 0 destroyed. 111 | 112 | Outputs: 113 | 114 | fdb_address = [ 115 | fdb-01.amazonaws.com, 116 | fdb-02.amazonaws.com, 117 | fdb-03.amazonaws.com 118 | ] 119 | tester_address = [ 120 | tester-01.amazonaws.com 121 | ] 122 | ``` 123 | 124 | Note that the machine names would be different each time (and much 125 | longer). This is just a sample output. 126 | 127 | Congratulations, you now have a FoundationDB cluster running in 128 | AWS. You can test it by connecting to the test machine with your 129 | terraform key: 130 | 131 | ``` 132 | $ ssh -i ~/.ssh/terraform ubuntu@tester-01.amazonaws.com 133 | ``` 134 | 135 | On your first connection, the ssh might ask you about accepting the 136 | new fingerprint. This happens because we have a brand new server 137 | running. Just type in 'yes'. 138 | 139 | Once connected to the test machine, you could verify that the client 140 | tools are installed and the cluster is responding: 141 | 142 | ``` 143 | $ fdbcli 144 | Using cluster file `/etc/foundationdb/fdb.cluster'. 145 | 146 | The database is available. 147 | 148 | Welcome to the fdbcli. For help, type `help'. 149 | fdb> status details 150 | 151 | Using cluster file `/etc/foundationdb/fdb.cluster'. 152 | 153 | Configuration: 154 | Redundancy mode - double 155 | Storage engine - ssd-2 156 | Coordinators - 3 157 | 158 | ... 159 | ``` 160 | 161 | Congratulations, FDB cluster is up and running! 162 | 163 | # Destroy the cluster 164 | 165 | Keeping AWS instances running costs money. So generally it is advised 166 | to destroy all the resources after the experiment. 167 | 168 | Terraform makes it easy: 169 | 170 | ``` 171 | $ terraform destroy -var-file=.secret.aws.tfvars 172 | 173 | .... 174 | 175 | Plan: 0 to add, 0 to change, 12 to destroy. 176 | 177 | Do you really want to destroy? 178 | Terraform will destroy all your managed infrastructure, as shown above. 179 | There is no undo. Only 'yes' will be accepted to confirm. 180 | 181 | Enter a value: yes 182 | 183 | .... 184 | 185 | Destroy complete! Resources: 12 destroyed. 186 | 187 | ``` 188 | 189 | 190 | # Modify the cluster 191 | 192 | You can tune the cluster configuration by editing `variables.tf` file 193 | to your liking. Ideally, you would do that before creating a new 194 | cluster. 195 | 196 | The most important options there are: 197 | 198 | ``` 199 | variable "aws_fdb_size" { 200 | default = "t2.medium" 201 | description = "machine type to run FoundationDB servers" 202 | 203 | } 204 | # using only 1 machine will conflict with the default cluster config 205 | # 'configure new memory double' 206 | variable "aws_fdb_count" { 207 | default = 3 208 | description = "how many machines do we want in our cluster. Minimum 2" 209 | } 210 | 211 | variable "aws_tester_size" { 212 | default = "m4.xlarge" 213 | description = "instance type for launching tester machines" 214 | } 215 | ``` 216 | 217 | 218 | # Plans 219 | 220 | I plan to improve this repository a bit later by: 221 | 222 | 1. Introducing a load tester tool (pre-installed to the tester image) 223 | with common benchmarks. 224 | 2. Adding a few scripts to visualize the results. 225 | 3. Adding more Terraform configurations tuned for better performance. 226 | 227 | # Contact 228 | 229 | If you have any questions, please don't hesitate to get in touch by 230 | sending an email to rinat at abdullin.com. 231 | -------------------------------------------------------------------------------- /packer-fdb/packer.json: -------------------------------------------------------------------------------- 1 | { 2 | "variables": { 3 | "aws_access_key": "{{env `AWS_ACCESS_KEY`}}", 4 | "aws_secret_key": "{{env `AWS_SECRET_KEY`}}" 5 | }, 6 | "builders": [ 7 | { 8 | "name": "aws", 9 | "type": "amazon-ebs", 10 | "access_key": "{{user `aws_access_key`}}", 11 | "secret_key": "{{user `aws_secret_key`}}", 12 | "region": "eu-central-1", 13 | "source_ami_filter": { 14 | "filters": { 15 | "virtualization-type": "hvm", 16 | "name": "ubuntu/images/*ubuntu-xenial-16.04-amd64-server-*", 17 | "root-device-type": "ebs" 18 | }, 19 | "owners": ["099720109477"], 20 | "most_recent": true 21 | }, 22 | "instance_type": "t2.small", 23 | "ssh_username": "ubuntu", 24 | "ami_name": "bitgn-fdb", 25 | "force_deregister": true, 26 | 27 | "ami_description": "Ubuntu 16:04 with FoundationDB server", 28 | "tags": { 29 | "OS_Version" : "Ubuntu", 30 | "Release": "16.04", 31 | "Component": "fdb" 32 | } 33 | }, 34 | { 35 | "name": "test", 36 | "type": "docker", 37 | "image": "ubuntu:16.04", 38 | "discard": true 39 | }, 40 | { 41 | "name": "docker", 42 | "type": "docker", 43 | "image": "ubuntu:16.04", 44 | "commit": true 45 | } 46 | ], 47 | "provisioners": [ 48 | { 49 | "type": "file", 50 | "source": "upload/", 51 | "destination": "/tmp/" 52 | }, 53 | { 54 | "type": "shell", 55 | "inline": [ 56 | "sleep 30", 57 | "cd /tmp", 58 | "./ubuntu.sh" 59 | ] 60 | } 61 | ] 62 | } 63 | -------------------------------------------------------------------------------- /packer-fdb/upload/policy-rc.d: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | # docker can be crazy and prevent service starting. 4 | # we are fine with that but need to make things work 5 | 6 | exit 0 -------------------------------------------------------------------------------- /packer-fdb/upload/ubuntu.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e 3 | set -x 4 | 5 | # from http://unix.stackexchange.com/a/28793 6 | # if we aren't root - elevate. This is useful for AMI 7 | if [ $EUID != 0 ]; then 8 | sudo "$0" "$@" 9 | exit $? 10 | fi 11 | 12 | export DEBIAN_FRONTEND=noninteractive 13 | 14 | # set timezone to UTC 15 | dpkg-reconfigure tzdata 16 | 17 | # https://groups.google.com/forum/#!msg/foundationdb-user/BtJf-1Mlx4I/fxXZClLpnOUJ 18 | # sources: https://github.com/ripple/docker-fdb-server/blob/master/Dockerfile 19 | # https://hub.docker.com/r/arypurnomoz/fdb-server/~/dockerfile/ 20 | 21 | # linux-aws - https://forums.aws.amazon.com/thread.jspa?messageID=769521&tstart=0 22 | 23 | # need to clean since images could have stale metadata 24 | apt-get clean && apt-get update 25 | apt-get install -y -qq build-essential python linux-aws sysstat iftop htop iotop ne 26 | 27 | # install fdbtop 28 | curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash - 29 | apt-get install -y -qq nodejs 30 | npm install -g fdbtop 31 | 32 | ######### FDB 33 | 34 | cd /tmp 35 | 36 | # download the dependencies 37 | wget https://www.foundationdb.org/downloads/6.0.15/ubuntu/installers/foundationdb-clients_6.0.15-1_amd64.deb 38 | wget https://www.foundationdb.org/downloads/6.0.15/ubuntu/installers/foundationdb-server_6.0.15-1_amd64.deb 39 | 40 | # server depends on the client packages 41 | dpkg -i foundationdb-clients_6.0.15-1_amd64.deb foundationdb-server_6.0.15-1_amd64.deb 42 | # stop the service 43 | service foundationdb stop 44 | 45 | # add default user to foundationdb group 46 | sudo usermod -a -G foundationdb ubuntu 47 | 48 | # ensure correct permissions 49 | chown -R foundationdb:foundationdb /etc/foundationdb 50 | chmod -R ug+w /etc/foundationdb 51 | 52 | ######### Cleanup 53 | 54 | apt-get clean 55 | rm -rf /var/lib/apt/lists/* 56 | rm -rf /tmp/* 57 | -------------------------------------------------------------------------------- /packer-tester/packer.json: -------------------------------------------------------------------------------- 1 | { 2 | "variables": { 3 | "aws_access_key": "{{env `AWS_ACCESS_KEY`}}", 4 | "aws_secret_key": "{{env `AWS_SECRET_KEY`}}" 5 | }, 6 | "builders": [ 7 | { 8 | "name": "aws", 9 | "type": "amazon-ebs", 10 | "access_key": "{{user `aws_access_key`}}", 11 | "secret_key": "{{user `aws_secret_key`}}", 12 | "region": "eu-central-1", 13 | "source_ami_filter": { 14 | "filters": { 15 | "virtualization-type": "hvm", 16 | "name": "ubuntu/images/*ubuntu-xenial-16.04-amd64-server-*", 17 | "root-device-type": "ebs" 18 | }, 19 | "owners": ["099720109477"], 20 | "most_recent": true 21 | }, 22 | "instance_type": "t2.small", 23 | "ssh_username": "ubuntu", 24 | "ami_name": "bitgn-tester", 25 | "ami_description": "Ubuntu with testing tools for FoundationDB", 26 | "force_deregister": true, 27 | "tags": { 28 | "OS_Version" : "Ubuntu", 29 | "Release": "16.04 LTS", 30 | "Component": "tester" 31 | } 32 | }, 33 | { 34 | "name": "docker", 35 | "type": "docker", 36 | "image": "ubuntu:16.04", 37 | "commit": true 38 | }, 39 | { 40 | "name": "test", 41 | "type": "docker", 42 | "image": "ubuntu:16.04", 43 | "discard": true 44 | } 45 | ], 46 | "provisioners": [ 47 | { 48 | "type": "file", 49 | "source": "upload/", 50 | "destination": "/tmp/" 51 | }, 52 | { 53 | "type": "shell", 54 | "inline": [ 55 | "sleep 30", 56 | "cd /tmp", 57 | "./ubuntu.sh" 58 | ] 59 | } 60 | ] 61 | } 62 | -------------------------------------------------------------------------------- /packer-tester/upload/ubuntu.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e 3 | set -x 4 | 5 | # from http://unix.stackexchange.com/a/28793 6 | # if we aren't root - elevate. This is useful for AMI 7 | if [ $EUID != 0 ]; then 8 | sudo "$0" "$@" 9 | exit $? 10 | fi 11 | 12 | export DEBIAN_FRONTEND=noninteractive 13 | 14 | 15 | # set timezone to UTC 16 | dpkg-reconfigure tzdata 17 | 18 | # need to clean since images could have stale metadata 19 | apt-get clean && apt-get update 20 | apt-get -qq -y install build-essential libssl-dev git python curl wget htop mosh screen iftop 21 | 22 | 23 | #cleanup 24 | apt-get clean && rm -rf /var/lib/apt/lists/* 25 | 26 | 27 | ########## Java Runtime 28 | # from here https://github.com/OpenTreeOfLife/germinator/wiki/Debian-upgrade-notes:-jessie-and-openjdk-8 29 | # add jessie backports 30 | # echo "deb http://http.debian.net/debian jessie-backports main" | tee -a /etc/apt/sources.list 31 | # apt-get update 32 | # apt-get -qq -y install openjdk-8-jre-headless && apt-get clean && rm -rf /var/lib/apt/lists/* 33 | 34 | ######### FDB 35 | 36 | cd /tmp 37 | 38 | wget -nv https://www.foundationdb.org/downloads/5.1.7/ubuntu/installers/foundationdb-clients_5.1.7-1_amd64.deb 39 | dpkg -i foundationdb-clients_5.1.7-1_amd64.deb 40 | 41 | # create the folder if it doesn't exist 42 | mkdir -p /etc/foundationdb 43 | 44 | # make the directory and the cluster file writeable 45 | chmod 777 /etc/foundationdb 46 | 47 | # write empty cluster file and setup permissions on it 48 | touch /etc/foundationdb/fdb.cluster 49 | chmod 666 /etc/foundationdb/fdb.cluster 50 | 51 | 52 | # cleanup 53 | rm -rf /var/lib/apt/lists/* 54 | rm -rf /tmp/* 55 | -------------------------------------------------------------------------------- /terraform/.gitignore: -------------------------------------------------------------------------------- 1 | *.backup 2 | *.tfstate 3 | *.plan 4 | .secret.aws.tfvars 5 | .terraform* 6 | -------------------------------------------------------------------------------- /terraform/init-fdb.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e 3 | 4 | VM_TYPE=$1 5 | VM_COUNT=$2 6 | SELF_IP=$3 7 | SEED_IP=$4 8 | FDB_PROCS=$5 9 | 10 | 11 | echo "./init-fdb.sh $@" 12 | 13 | # avoid confusing FoundationDB 14 | service foundationdb stop 15 | 16 | # resolve IP address as host name 17 | echo "$SELF_IP $(hostname)" >> /etc/hosts 18 | 19 | # wipe the data from the image 20 | rm -rf /var/lib/foundationdb/data/4500/ 21 | 22 | # make 1st node the coordinator 23 | echo "Drtu0T4S:i8uQIB9r@$SEED_IP:4500" > /etc/foundationdb/fdb.cluster 24 | 25 | # make sure the cluster file is writeable by everybody 26 | chmod ugo+w /etc/foundationdb/fdb.cluster 27 | 28 | # we already have 1 process at 4500 29 | COUNTER=1 30 | while [ $COUNTER -lt $FDB_PROCS ]; do 31 | let "PORT = COUNTER + 4500" 32 | echo "PORT $PORT" 33 | echo "[fdbserver.$PORT]" >> /etc/foundationdb/foundationdb.conf 34 | let COUNTER=COUNTER+1 35 | done 36 | 37 | 38 | # NVME disks aren't formatted. Mounting them in fstab - no good 39 | # mounting NVME disk: https://stackoverflow.com/questions/45167717/mounting-a-nvme-disk-on-aws-ec2 40 | 41 | 42 | case $VM_TYPE in 43 | "m3.large" | "m3.medium" ) 44 | echo use local instance store 45 | mount /dev/xvdb /var/lib/foundationdb 46 | echo /dev/xvdb /var/lib/foundationdb ext3 defaults,nofail 0 2 >> /etc/fstab 47 | mkdir -p /var/lib/foundationdb/data 48 | chown -R foundationdb:foundationdb /var/lib/foundationdb 49 | ;; 50 | "i3.large" ) 51 | echo SSD optimized 52 | mkfs.ext4 -E nodiscard /dev/nvme0n1 53 | mount /dev/nvme0n1 /var/lib/foundationdb 54 | mkdir -p /var/lib/foundationdb/data 55 | chown -R foundationdb:foundationdb /var/lib/foundationdb 56 | ;; 57 | esac 58 | 59 | 60 | 61 | if [ "$SELF_IP" == "$SEED_IP" ]; then 62 | echo "Seed setup" 63 | service foundationdb start 64 | sleep 60 65 | fdbcli --exec "configure new ssd double" --timeout 60 66 | fdbcli --exec "coordinators auto; status" --timeout 60 67 | else 68 | echo "Follower setup" 69 | # start the service 70 | service foundationdb start 71 | fi -------------------------------------------------------------------------------- /terraform/init-tester.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e 3 | 4 | FDB_TYPE=$1 5 | FDB_COUNT=$2 6 | SELF_IP=$3 7 | SEED_IP=$4 8 | TESTER_TYPE=$5 9 | 10 | 11 | echo "./init-tester.sh $@" 12 | 13 | # resolve IP address as host name 14 | echo "$SELF_IP $(hostname)" >> /etc/hosts 15 | 16 | # make 1st node the coordinator 17 | echo "Drtu0T4S:i8uQIB9r@$SEED_IP:4500" > /etc/foundationdb/fdb.cluster 18 | 19 | # make sure the cluster file is writeable by everybody 20 | chmod ugo+w /etc/foundationdb/fdb.cluster 21 | 22 | # print cluster info for the benchmarking purposes 23 | echo "fdb_type: $FDB_TYPE" >> /etc/cluster 24 | echo "fdb_count: $FDB_COUNT" >> /etc/cluster 25 | echo "tester_type: $TESTER_TYPE" >> /etc/cluster 26 | 27 | # make cluster info readable by anybody 28 | chmod ugo+r /etc/cluster -------------------------------------------------------------------------------- /terraform/main.tf: -------------------------------------------------------------------------------- 1 | provider "aws" { 2 | access_key = "${var.aws_access_key}" 3 | secret_key = "${var.aws_secret_key}" 4 | region = "${var.aws_region}" 5 | } 6 | 7 | 8 | 9 | // Find our latest available AMI for the fdb node 10 | // TODO: switch to a shared and hosted stable image 11 | data "aws_ami" "fdb" { 12 | most_recent = true 13 | 14 | filter { 15 | name = "name" 16 | values = ["bitgn-fdb"] 17 | } 18 | owners = ["self"] 19 | } 20 | 21 | 22 | 23 | // Find our latest available AMI for the fdb testing node 24 | // TODO: switch to a shared and hosted stable image 25 | data "aws_ami" "tester" { 26 | most_recent = true 27 | 28 | filter { 29 | name = "name" 30 | values = ["bitgn-tester"] 31 | } 32 | owners = ["self"] 33 | } 34 | 35 | 36 | # Create a VPC to launch our instances into 37 | resource "aws_vpc" "default" { 38 | cidr_block = "10.0.0.0/16" 39 | # this will solve sudo: unable to resolve host ip-10-0-xx-xx 40 | enable_dns_hostnames = true 41 | } 42 | 43 | 44 | # Create an internet gateway to give our subnet access to the outside world 45 | resource "aws_internet_gateway" "default" { 46 | vpc_id = "${aws_vpc.default.id}" 47 | 48 | } 49 | # Grant the VPC internet access on its main route table 50 | resource "aws_route" "internet_access" { 51 | route_table_id = "${aws_vpc.default.main_route_table_id}" 52 | destination_cidr_block = "0.0.0.0/0" 53 | gateway_id = "${aws_internet_gateway.default.id}" 54 | } 55 | 56 | 57 | # Create a subnet to launch our instances into 58 | resource "aws_subnet" "client" { 59 | vpc_id = "${aws_vpc.default.id}" 60 | cidr_block = "10.0.1.0/24" 61 | map_public_ip_on_launch = true 62 | availability_zone = "${var.aws_availability_zone}" 63 | 64 | tags = { 65 | Name = "Client Subnet" 66 | Project = "TF:bitgn" 67 | } 68 | } 69 | 70 | 71 | # Create a subnet to launch our instances into 72 | resource "aws_subnet" "db" { 73 | vpc_id = "${aws_vpc.default.id}" 74 | cidr_block = "10.0.2.0/24" 75 | map_public_ip_on_launch = true 76 | availability_zone = "${var.aws_availability_zone}" 77 | 78 | tags = { 79 | Name = "FDB Subnet" 80 | Project = "TF:bitgn" 81 | } 82 | } 83 | 84 | 85 | 86 | # security group with only SSH access 87 | resource "aws_security_group" "tester_group" { 88 | name = "tf_tester_group" 89 | description = "Terraform: SSH only" 90 | vpc_id = "${aws_vpc.default.id}" 91 | 92 | # SSH access from anywhere 93 | ingress { 94 | from_port = 22 95 | to_port = 22 96 | protocol = "tcp" 97 | cidr_blocks = ["0.0.0.0/0"] 98 | } 99 | 100 | # Mosh UDP ports 101 | ingress { 102 | from_port = 60000 103 | to_port = 60010 104 | protocol = "udp" 105 | cidr_blocks = ["0.0.0.0/0"] 106 | } 107 | 108 | 109 | # outbound internet access 110 | egress { 111 | from_port = 0 112 | to_port = 0 113 | protocol = "-1" 114 | cidr_blocks = ["0.0.0.0/0"] 115 | } 116 | } 117 | 118 | # security group with only SSH access 119 | resource "aws_security_group" "fdb_group" { 120 | name = "tf_fdb_group" 121 | description = "Terraform: SSH and FDB" 122 | vpc_id = "${aws_vpc.default.id}" 123 | 124 | # SSH access from anywhere 125 | ingress { 126 | from_port = 22 127 | to_port = 22 128 | protocol = "tcp" 129 | cidr_blocks = ["0.0.0.0/0"] 130 | } 131 | 132 | # FDB access from the VPC. We open a port for each process 133 | ingress { 134 | from_port = 4500 135 | to_port = "${4500 + var.fdb_procs_per_machine - 1}" 136 | protocol = "tcp" 137 | cidr_blocks = ["10.0.0.0/16"] 138 | } 139 | # outbound internet access 140 | egress { 141 | from_port = 0 142 | to_port = 0 143 | protocol = "-1" 144 | cidr_blocks = ["0.0.0.0/0"] 145 | } 146 | } 147 | 148 | 149 | resource "aws_key_pair" "auth" { 150 | key_name = "${var.key_name}" 151 | public_key = "${file(var.public_key_path)}" 152 | } 153 | 154 | 155 | resource "aws_instance" "tester" { 156 | # The connection block tells our provisioner how to 157 | # communicate with the resource (instance) 158 | connection { 159 | # The default username for our AMI 160 | user = "ubuntu" 161 | agent = "false" 162 | 163 | private_key = "${file(var.private_key_path)}" 164 | # The connection will use the local SSH agent for authentication. 165 | } 166 | 167 | 168 | instance_type = "${var.aws_tester_size}" 169 | availability_zone = "${var.aws_availability_zone}" 170 | 171 | # Grab AMI id from the data source 172 | ami = "${data.aws_ami.tester.id}" 173 | 174 | # The name of our SSH keypair we created above. 175 | key_name = "${aws_key_pair.auth.id}" 176 | 177 | # Our Security group to allow HTTP and SSH access 178 | vpc_security_group_ids = ["${aws_security_group.tester_group.id}"] 179 | 180 | # We're going to launch into the client subnet. 181 | subnet_id = "${aws_subnet.client.id}" 182 | 183 | tags { 184 | Name = "Tester" 185 | Project = "TF:bitgn" 186 | } 187 | 188 | provisioner "file" { 189 | source = "init-tester.sh" 190 | destination = "/tmp/init-tester.sh" 191 | } 192 | 193 | provisioner "remote-exec" { 194 | inline = [ 195 | "sudo chmod +x /tmp/init-tester.sh", 196 | "sudo /tmp/init-tester.sh ${var.aws_fdb_size} ${var.aws_fdb_count} ${self.private_ip} ${cidrhost(aws_subnet.db.cidr_block, 101)} ${var.aws_tester_size}" 197 | ] 198 | } 199 | } 200 | 201 | 202 | 203 | resource "aws_instance" "fdb" { 204 | # The connection block tells our provisioner how to 205 | # communicate with the resource (instance) 206 | connection { 207 | # The default username for our AMI 208 | user = "ubuntu" 209 | agent = "false" 210 | 211 | private_key = "${file(var.private_key_path)}" 212 | # The connection will use the local SSH agent for authentication. 213 | } 214 | 215 | availability_zone = "${var.aws_availability_zone}" 216 | instance_type = "${var.aws_fdb_size}" 217 | count = "${var.aws_fdb_count}" 218 | # Grab AMI id from the data source 219 | ami = "${data.aws_ami.fdb.id}" 220 | 221 | 222 | # I want a very specific IP address to be assigned. However 223 | # AWS reserves both the first four IP addresses and the last IP address 224 | # in each subnet CIDR block. They're not available for you to use. 225 | private_ip = "${cidrhost(aws_subnet.db.cidr_block, count.index+1+100)}" 226 | 227 | 228 | # The name of our SSH keypair we created above. 229 | key_name = "${aws_key_pair.auth.id}" 230 | 231 | # Our Security group to allow HTTP and SSH access 232 | vpc_security_group_ids = ["${aws_security_group.fdb_group.id}"] 233 | 234 | # We're going to launch into the DB subnet 235 | subnet_id = "${aws_subnet.db.id}" 236 | 237 | tags { 238 | Name = "${format("fdb-%03d", count.index + 1)}" 239 | Project = "TF:bitgn" 240 | } 241 | 242 | provisioner "file" { 243 | source = "init-fdb.sh" 244 | destination = "/tmp/init-fdb.sh" 245 | } 246 | 247 | provisioner "remote-exec" { 248 | inline = [ 249 | "sudo chmod +x /tmp/init-fdb.sh", 250 | "sudo /tmp/init-fdb.sh ${var.aws_fdb_size} ${var.aws_fdb_count} ${self.private_ip} ${cidrhost(aws_subnet.db.cidr_block, 101)} ${var.fdb_procs_per_machine}", 251 | ] 252 | } 253 | } 254 | -------------------------------------------------------------------------------- /terraform/makefile: -------------------------------------------------------------------------------- 1 | # Terraform makefile by Rinat Abdullin 2018-05-22 2 | BOLD=$(shell tput bold) 3 | ERROR=$(shell tput setaf 1) 4 | RESET=$(shell tput sgr0) 5 | 6 | VARS=".secret.aws.tfvars" 7 | CURRENT_FOLDER=$(shell basename "$$(pwd)") 8 | 9 | help: ## Display help by printing target comments 10 | @grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | \ 11 | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}' 12 | 13 | 14 | 15 | set-env: 16 | @if [ ! -f "$(VARS)" ]; then \ 17 | echo "$(BOLD)$(ERROR)Could not find variables file: $(VARS)$(RESET)"; \ 18 | exit 1; \ 19 | fi 20 | 21 | prep: set-env 22 | terraform init 23 | 24 | 25 | plan: prep ## Show what terraform thinks it will do 26 | @terraform plan -var-file="$(VARS)" -lock=false -out my.plan 27 | 28 | apply: plan ## Apply the changes 29 | @terraform apply "my.plan" 30 | @terraform output tester_address > ~/.bitgn_tester 31 | 32 | destroy: prep ## Destroy current setup 33 | @terraform destroy -var-file="$(VARS)" -lock=false 34 | rm ~/.bitgn_tester 35 | -------------------------------------------------------------------------------- /terraform/outputs.tf: -------------------------------------------------------------------------------- 1 | 2 | output "tester_address" { 3 | value = "${aws_instance.tester.*.public_dns}" 4 | } 5 | 6 | output "fdb_address" { 7 | value = "${aws_instance.fdb.*.public_dns}" 8 | } 9 | 10 | -------------------------------------------------------------------------------- /terraform/variables.tf: -------------------------------------------------------------------------------- 1 | variable "public_key_path" { 2 | description = "Path to the SSH public key to be used for authentication." 3 | default = "~/.ssh/terraform.pub" 4 | } 5 | 6 | variable "private_key_path" { 7 | description = "Path to the SSH private key" 8 | default = "~/.ssh/terraform" 9 | } 10 | 11 | variable "key_name" { 12 | default = "terraform" 13 | } 14 | 15 | 16 | variable "aws_access_key" {} 17 | variable "aws_secret_key" {} 18 | 19 | variable "aws_region" { 20 | default = "eu-central-1" 21 | description = "AWS region to launch servers." 22 | } 23 | 24 | variable "aws_availability_zone" { 25 | default = "eu-central-1b" 26 | } 27 | 28 | 29 | // instance store options: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html 30 | 31 | # good options: 32 | # i3.large - will use local NVMe SSD 33 | # m3.large - will use local instance store 34 | 35 | variable "aws_fdb_size" { 36 | default = "m3.large" 37 | description = "machine type to run FoundationDB servers" 38 | } 39 | variable "fdb_procs_per_machine" { 40 | default = 2 41 | description = "number of FDB processes per machine" 42 | } 43 | # using only 1 machine will conflict with the default cluster config 44 | # 'configure new memory double' 45 | variable "aws_fdb_count" { 46 | default = 3 47 | description = "Number of machines in a cluster. Minimum 2" 48 | } 49 | 50 | 51 | # good options 52 | # m3.large 53 | # c5.2xlarge 54 | variable "aws_tester_size" { 55 | default = "m3.large" 56 | description = "instance type for launching tester machines" 57 | } 58 | 59 | --------------------------------------------------------------------------------