├── LICENSE
├── readme.md
├── riak-cluster.json
├── riak-vpc-cluster-with-frontend-appservers.json
├── riak-vpc-cluster.json
└── tools
└── join_riak_cluster.py
/LICENSE:
--------------------------------------------------------------------------------
1 | Copyright (c) 2013, Basho Technologies
2 | All rights reserved.
3 |
4 | Redistribution and use in source and binary forms, with or without
5 | modification, are permitted provided that the following conditions are met:
6 |
7 | 1. Redistributions of source code must retain the above copyright notice, this
8 | list of conditions and the following disclaimer.
9 | 2. Redistributions in binary form must reproduce the above copyright notice,
10 | this list of conditions and the following disclaimer in the documentation
11 | and/or other materials provided with the distribution.
12 |
13 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
14 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
15 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
16 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
17 | ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
18 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
19 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
20 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
21 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
22 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
23 |
24 | The views and conclusions contained in the software and documentation are those
25 | of the authors and should not be interpreted as representing official policies,
26 | either expressed or implied, of Basho Technologies.
27 |
--------------------------------------------------------------------------------
/readme.md:
--------------------------------------------------------------------------------
1 | # Riak AWS CloudFormation Recipes
2 |
3 |
4 | These [Amazon CloudFormation](http://aws.amazon.com/cloudformation/) recipes are designed to help you get started quickly with a Riak cluster. They are not for production use. All of these recipes build clusters based on Amazon Linux and should work in any AWS region.
5 |
6 | The recipes come in 3 flavors: riak cluster, riak vpc cluster, and riak vpc cluster with frontend appservers.
7 |
8 | ## riak cluster
9 |
10 |
11 | This builds a a simple riak cluster. The end result is a fully joined riak cluster. You have the oppurtunity to pass the following options to the cluster at launch time:
12 |
13 | * **RiakClusterSize**
14 | * The number of nodes that you would like in the cluster
15 | * **RiakInstanceType**
16 | * The instance size you'd like to use
17 | * **RingSize**
18 | * The Riak Ring size you want to use (64,128,256)
19 | * **DiskType**
20 | * The type of disk you want on your root volume (ebs or ephemeral)
21 | * **KeyName**
22 | * The ec2 ssh keypair that you want associated with these instances. You will use this to log into your instance. This key must be uploaded prior to launching the cluster.
23 |
24 | Once the stack is completed, use your EC2 console to find an IP address of to ssh to and test drive the cluster.
25 |
26 | The filename for this recipe is **riak-cluster.json** .
27 |
28 | This stack can take around 10 minutes to build. It will build the number of instances specified in RiakClusterSize.
29 |
30 | ## riak vpc cluster
31 |
32 | This recipe builds a Riak cluster inside of a EC2 Virtual Private Cloud (VPC). This means that the cluster is shut off from the outside. To access the cluster, you will first need to ssh into the BastionHost. This recipe has all the same options as riak-cluster plus the following:
33 |
34 |
35 |
36 | * **BastionInstanceType**
37 | * The ec2 instance type of the bastion host
38 |
39 | * **NATInstanceType**
40 | * The ec2 instance type of the NAT device
41 |
42 | * **SSHFrom**
43 | * The IP range which you'd like to allow to SSH to the BastionHost. Must be a valid CIDR range of the form x.x.x.x/x.
44 |
45 |
46 | Once the stack is completed, check the outputs of the stack to find the BastionHost IP address. This is the IP address that you must ssh to first before reaching the Riak nodes.
47 |
48 | This stack can take around 20 minutes to build. It will build the number of instances specified in RiakClusterSize, plus 1 NAT instance and 1 Bastion instance.
49 |
50 |
51 | The filename for this receipe is **riak-vpc-cluster.json** .
52 |
53 |
54 |
55 | ## riak vpc cluster with appservers
56 |
57 | This builds upon the riak-vpc-cluster with the addition of a set of load balanced application servers. The application servers sit behind a public Elastic Load Balancer (ELB) and the make Riak api requests to an internal ELB responsible for the Riak Servers. We are using a demo application called "Riagi" which is a simple image uploading tool to demonstrate the cluster. This recipe has all the same options as riak-cluster-vpc plus the following:
58 |
59 |
60 |
61 | * **FrontendClusterSize**
62 | * The number of ec2 instances that will make up the Frontend Cluster.
63 | * **FrontendInstanceType**
64 | * The ec2 instance type of the Frontend instances.
65 |
66 |
67 | Once the stack is completed, check the outputs of the stack to find the BastionIP address. This is the IP address that you must ssh to first before reaching the Riak nodes. It will build the number of instances specified in RiakClusterSize, FrontendClusterSize, 1 NAT instance, and 1 Bastion instance.
68 |
69 | Also in the outputs you will find the Bastion Host IP and the Website URL to access Riagi.
70 |
71 |
72 | The filename for this receipe is **riak-vpc-cluster-with-frontend-appservers.json** .
73 |
74 |
75 |
--------------------------------------------------------------------------------
/riak-cluster.json:
--------------------------------------------------------------------------------
1 | {
2 | "AWSTemplateFormatVersion": "2010-09-09",
3 | "Description": "Launches a Riak Cluster",
4 | "Mappings": {
5 | "AWSInstanceType2Arch": {
6 | "c1.medium": {
7 | "Arch": "64"
8 | },
9 | "c1.xlarge": {
10 | "Arch": "64"
11 | },
12 | "cc1.4xlarge": {
13 | "Arch": "64Cluster"
14 | },
15 | "cc2.8xlarge": {
16 | "Arch": "64Cluster"
17 | },
18 | "cg1.4xlarge": {
19 | "Arch": "64GPU"
20 | },
21 | "m1.large": {
22 | "Arch": "64"
23 | },
24 | "m1.medium": {
25 | "Arch": "64"
26 | },
27 | "m1.small": {
28 | "Arch": "64"
29 | },
30 | "m1.xlarge": {
31 | "Arch": "64"
32 | },
33 | "m2.2xlarge": {
34 | "Arch": "64"
35 | },
36 | "m2.4xlarge": {
37 | "Arch": "64"
38 | },
39 | "m2.xlarge": {
40 | "Arch": "64"
41 | },
42 | "t1.micro": {
43 | "Arch": "64"
44 | }
45 | },
46 | "ebs": {
47 | "ap-northeast-1": {
48 | "32": "ami-486cd349 ",
49 | "64": "ami-4e6cd34f",
50 | "64Cluster": "NOT_YET_SUPPORTED",
51 | "64GPU": "NOT_YET_SUPPORTED"
52 | },
53 | "ap-southeast-1": {
54 | "32": "ami-a2a7e7f0",
55 | "64": "ami-a6a7e7f4",
56 | "64Cluster": "NOT_YET_SUPPORTED",
57 | "64GPU": "NOT_YET_SUPPORTED"
58 | },
59 | "eu-west-1": {
60 | "32": "ami-937474e7",
61 | "64": "ami-c37474b7",
62 | "64Cluster": "ami-d97474ad",
63 | "64GPU": "ami-1b02026f"
64 | },
65 | "sa-east-1": {
66 | "32": "ami-e209d0ff",
67 | "64": "ami-1e08d103",
68 | "64Cluster": "NOT_YET_SUPPORTED",
69 | "64GPU": "NOT_YET_SUPPORTED"
70 | },
71 | "us-east-1": {
72 | "32": "ami-1a249873",
73 | "64": "ami-1624987f",
74 | "64Cluster": "ami-08249861",
75 | "64GPU": "ami-02f54a6b"
76 | },
77 | "us-west-1": {
78 | "32": "ami-19f9de5c",
79 | "64": "ami-1bf9de5e",
80 | "64Cluster": "NOT_YET_SUPPORTED",
81 | "64GPU": "NOT_YET_SUPPORTED"
82 | },
83 | "us-west-2": {
84 | "32": "ami-2231bf12",
85 | "64": "ami-2a31bf1a",
86 | "64Cluster": "ami-2431bf14",
87 | "64GPU": "NOT_YET_SUPPORTED"
88 | }
89 | },
90 | "ephemeral": {
91 | "ap-northeast-1": {
92 | "32": "ami-586cd359",
93 | "64": "ami-5a6cd35b"
94 | },
95 | "ap-southeast-1": {
96 | "32": "ami-aaa7e7f8",
97 | "64": "ami-a8a7e7fa"
98 | },
99 | "eu-west-1": {
100 | "32": "ami-cf7474bb",
101 | "64": "ami-b57474c1"
102 | },
103 | "sa-east-1": {
104 | "32": " ami-1a08d107",
105 | "64": " ami-1608d10b"
106 | },
107 | "us-east-1": {
108 | "32": "ami-10249879",
109 | "64": "ami-e8249881"
110 | },
111 | "us-west-1": {
112 | "32": "ami-27f9de62",
113 | "64": "ami-21f9de64"
114 | },
115 | "us-west-2": {
116 | "32": "ami-2c31bf1c",
117 | "64": "ami-2e31bf1e"
118 | }
119 | }
120 | },
121 | "Parameters": {
122 | "DiskType": {
123 | "AllowedValues": [
124 | "ephemeral",
125 | "ebs"
126 | ],
127 | "Default": "ephemeral",
128 | "Description": "Type of Disk to use ( ephemeral/ebs )",
129 | "Type": "String"
130 | },
131 | "KeyName": {
132 | "Description": "Name of an existing EC2 KeyPair to enable SSH access to the Riak Cluster",
133 | "Type": "String"
134 | },
135 | "RiakClusterSize": {
136 | "Description": "Number of nodes in the Riak cluster",
137 | "Type": "String"
138 | },
139 | "RiakInstanceType": {
140 | "AllowedValues": [
141 | "m1.small",
142 | "m1.medium",
143 | "m1.large",
144 | "m1.xlarge",
145 | "m2.xlarge",
146 | "m2.2xlarge",
147 | "m2.4xlarge",
148 | "c1.medium",
149 | "c1.xlarge",
150 | "cc1.4xlarge"
151 | ],
152 | "ConstraintDescription": "must be valid instance type. ",
153 | "Default": "m1.large",
154 | "Description": "Type of EC2 instance for Riak",
155 | "Type": "String"
156 | },
157 | "RingSize": {
158 | "AllowedValues": [
159 | "64",
160 | "128",
161 | "256",
162 | "512"
163 | ],
164 | "Default": "64",
165 | "Description": "Size of the Riak Ring",
166 | "Type": "String"
167 | }
168 | },
169 | "Resources": {
170 | "ApplicationWaitCondition": {
171 | "DependsOn": "RiakServerGroup",
172 | "Properties": {
173 | "Handle": {
174 | "Ref": "ApplicationWaitHandle"
175 | },
176 | "Timeout": "4500"
177 | },
178 | "Type": "AWS::CloudFormation::WaitCondition"
179 | },
180 | "ApplicationWaitHandle": {
181 | "Type": "AWS::CloudFormation::WaitConditionHandle"
182 | },
183 | "CFNInitUser": {
184 | "Properties": {
185 | "Path": "/",
186 | "Policies": [
187 | {
188 | "PolicyDocument": {
189 | "Statement": [
190 | {
191 | "Action": [
192 | "cloudformation:DescribeStackResource",
193 | "s3:GetObject"
194 | ],
195 | "Effect": "Allow",
196 | "Resource": "*"
197 | }
198 | ]
199 | },
200 | "PolicyName": "AccessForCFNInit"
201 | }
202 | ]
203 | },
204 | "Type": "AWS::IAM::User"
205 | },
206 | "CFNKeys": {
207 | "Properties": {
208 | "UserName": {
209 | "Ref": "CFNInitUser"
210 | }
211 | },
212 | "Type": "AWS::IAM::AccessKey"
213 | },
214 | "ClusterCommunication1": {
215 | "Properties": {
216 | "FromPort": "-1",
217 | "GroupName": {
218 | "Ref": "InstanceSecurityGroup"
219 | },
220 | "IpProtocol": "icmp",
221 | "SourceSecurityGroupName": {
222 | "Ref": "InstanceSecurityGroup"
223 | },
224 | "ToPort": "-1"
225 | },
226 | "Type": "AWS::EC2::SecurityGroupIngress"
227 | },
228 | "ClusterCommunication2": {
229 | "Properties": {
230 | "FromPort": "1",
231 | "GroupName": {
232 | "Ref": "InstanceSecurityGroup"
233 | },
234 | "IpProtocol": "tcp",
235 | "SourceSecurityGroupName": {
236 | "Ref": "InstanceSecurityGroup"
237 | },
238 | "ToPort": "65356"
239 | },
240 | "Type": "AWS::EC2::SecurityGroupIngress"
241 | },
242 | "ClusterCommunication3": {
243 | "Properties": {
244 | "FromPort": "1",
245 | "GroupName": {
246 | "Ref": "InstanceSecurityGroup"
247 | },
248 | "IpProtocol": "udp",
249 | "SourceSecurityGroupName": {
250 | "Ref": "InstanceSecurityGroup"
251 | },
252 | "ToPort": "65356"
253 | },
254 | "Type": "AWS::EC2::SecurityGroupIngress"
255 | },
256 | "InstanceSecurityGroup": {
257 | "Properties": {
258 | "GroupDescription": "Enable SSH access via port 22",
259 | "SecurityGroupIngress": [
260 | {
261 | "CidrIp": "0.0.0.0/0",
262 | "FromPort": "22",
263 | "IpProtocol": "tcp",
264 | "ToPort": "22"
265 | }
266 | ]
267 | },
268 | "Type": "AWS::EC2::SecurityGroup"
269 | },
270 | "LaunchConfig": {
271 | "Properties": {
272 | "IamInstanceProfile": {
273 | "Ref": "RootInstanceProfile"
274 | },
275 | "ImageId": {
276 | "Fn::FindInMap": [
277 | {
278 | "Ref": "DiskType"
279 | },
280 | {
281 | "Ref": "AWS::Region"
282 | },
283 | {
284 | "Fn::FindInMap": [
285 | "AWSInstanceType2Arch",
286 | {
287 | "Ref": "RiakInstanceType"
288 | },
289 | "Arch"
290 | ]
291 | }
292 | ]
293 | },
294 | "InstanceType": {
295 | "Ref": "RiakInstanceType"
296 | },
297 | "KeyName": {
298 | "Ref": "KeyName"
299 | },
300 | "SecurityGroups": [
301 | {
302 | "Ref": "InstanceSecurityGroup"
303 | }
304 | ],
305 | "UserData": {
306 | "Fn::Base64": {
307 | "Fn::Join": [
308 | "\n",
309 | [
310 | "#!/bin/bash -v",
311 | "exec > >(tee /var/log/cfn-data.log|logger -t user-data -s 2>/dev/console) 2>&1",
312 | "",
313 | "sleep 10",
314 | "",
315 | "function retry {",
316 | " nTrys=0",
317 | " maxTrys=5",
318 | " status=256",
319 | " until [ $status == 0 ] ; do",
320 | " $1",
321 | " status=$?",
322 | " nTrys=$(($nTrys + 1))",
323 | " if [ $nTrys -gt $maxTrys ] ; then",
324 | " echo \"Number of re-trys exceeded. Exit code: $status\"",
325 | " exit $status",
326 | " fi",
327 | " if [ $status != 0 ] ; then",
328 | " echo \"Failed (exit code $status)... retry $nTrys\"",
329 | " sleep 10",
330 | " fi",
331 | " done",
332 | "}",
333 | "",
334 | "yum update -y aws-cfn-bootstrap",
335 | "",
336 | "#for all the stuff that complains about sudo and tty",
337 | "sed -i 's,Defaults requiretty,#Defaults requiretty,g' /etc/sudoers",
338 | "",
339 | "function error_exit",
340 | "{",
341 | {
342 | "Fn::Join": [
343 | "",
344 | [
345 | " /opt/aws/bin/cfn-signal -e 1 -r \"$1\" '",
346 | {
347 | "Ref": "ApplicationWaitHandle"
348 | },
349 | "'"
350 | ]
351 | ]
352 | },
353 | "}",
354 | "yum update -y aws-cfn-bootstrap",
355 | "#this runs the first stage of cfinit",
356 | {
357 | "Fn::Join": [
358 | "",
359 | [
360 | "#/opt/aws/bin/cfn-init -c ascending -v --region ",
361 | {
362 | "Ref": "AWS::Region"
363 | },
364 | " -s ",
365 | {
366 | "Ref": "AWS::StackName"
367 | },
368 | " -r ",
369 | "LaunchConfig",
370 | " --access-key ",
371 | {
372 | "Ref": "CFNKeys"
373 | },
374 | " --secret-key ",
375 | {
376 | "Fn::GetAtt": [
377 | "CFNKeys",
378 | "SecretAccessKey"
379 | ]
380 | },
381 | " || error_exit 'Failed to initialize client using cfn-init'"
382 | ]
383 | ]
384 | },
385 | "",
386 | "",
387 | "",
388 | "",
389 | "yum install -y rubygems git puppet ruby-devel make gcc",
390 | "gem install hiera hiera-puppet",
391 | "mkdir -p /var/lib/hiera",
392 | {
393 | "Fn::Join": [
394 | "",
395 | [
396 | "echo 'ring_creation_size: ",
397 | {
398 | "Ref": "RingSize"
399 | },
400 | "' >> '/var/lib/hiera/common.yaml'"
401 | ]
402 | ]
403 | },
404 | "",
405 | "mkdir -p /tmp/puppet/modules",
406 | "retry \"git clone http://github.com/basho/puppet-riak -b cfn /tmp/puppet/modules/riak\"",
407 | "retry \"git clone https://github.com/KrisBuytaert/puppet-hiera.git /tmp/puppet/modules/puppet-hiera\"",
408 | "retry \"git clone https://github.com/puppetlabs/puppetlabs-stdlib /tmp/puppet/modules/stdlib\"",
409 | "puppet apply --modulepath=/tmp/puppet/modules -e \"include riak\" --trace --debug --verbose",
410 | "",
411 | "echo \"riak soft nofile 65336\" >> /etc/security/limits.conf",
412 | "echo \"riak hard nofile 65336\" >> /etc/security/limits.conf",
413 | "echo \"root soft nofile 65336\" >> /etc/security/limits.conf",
414 | "echo \"root hard nofile 65336\" >> /etc/security/limits.conf",
415 | "",
416 | "echo \"vm.swappiness = 0\" >> /etc/sysctl.conf",
417 | "echo \"net.ipv4.tcp_max_syn_backlog = 40000\" >> /etc/sysctl.conf",
418 | "echo \"net.core.somaxconn=4000\" >> /etc/sysctl.conf",
419 | "echo \"net.ipv4.tcp_timestamps = 0\" >> /etc/sysctl.conf",
420 | "echo \"net.ipv4.tcp_sack = 1\" >> /etc/sysctl.conf",
421 | "echo \"net.ipv4.tcp_window_scaling = 1\" >> /etc/sysctl.conf",
422 | "echo \"net.ipv4.tcp_fin_timeout = 15\" >> /etc/sysctl.conf",
423 | "echo \"net.ipv4.tcp_keepalive_intvl = 30\" >> /etc/sysctl.conf",
424 | "echo \"net.ipv4.tcp_tw_reuse = 1\" >> /etc/sysctl.conf",
425 | "echo \"+P 256000\" >> /etc/riak/vm.args",
426 | "echo \"+swt very_low\" >> /etc/riak/vm.args",
427 | "echo \"+zdbbl 32768\" >> /etc/riak/vm.args",
428 | "sysctl -p",
429 | "",
430 | "service riak stop",
431 | "service riak start",
432 | "sleep 5",
433 | "retry \"git clone http://github.com/basho/cloudformation-riak /tmp/cloudformation-riak\"",
434 | "/tmp/cloudformation-riak/tools/join_riak_cluster.py",
435 | "/usr/sbin/riak ping",
436 | "result_code=$?",
437 | {
438 | "Fn::Join": [
439 | "",
440 | [
441 | "/opt/aws/bin/cfn-signal -e $result_code '",
442 | {
443 | "Ref": "ApplicationWaitHandle"
444 | },
445 | "'"
446 | ]
447 | ]
448 | },
449 | "#turn back on requiretty for sudo",
450 | "sed -i 's,#Defaults requiretty,Defaults requiretty,g' /etc/sudoers"
451 | ]
452 | ]
453 | }
454 | }
455 | },
456 | "Type": "AWS::AutoScaling::LaunchConfiguration"
457 | },
458 | "RiakServerGroup": {
459 | "Properties": {
460 | "AvailabilityZones": {
461 | "Fn::GetAZs": ""
462 | },
463 | "LaunchConfigurationName": {
464 | "Ref": "LaunchConfig"
465 | },
466 | "MaxSize": {
467 | "Ref": "RiakClusterSize"
468 | },
469 | "MinSize": {
470 | "Ref": "RiakClusterSize"
471 | }
472 | },
473 | "Type": "AWS::AutoScaling::AutoScalingGroup"
474 | },
475 | "RolePolicies": {
476 | "Properties": {
477 | "PolicyDocument": {
478 | "Statement": [
479 | {
480 | "Action": [
481 | "ec2:DescribeAvailabilityZones",
482 | "ec2:DescribeInstanceStatus",
483 | "ec2:DescribeInstances",
484 | "ec2:DescribeRegions",
485 | "ec2:DescribeTags"
486 | ],
487 | "Effect": "Allow",
488 | "Resource": "*"
489 | }
490 | ]
491 | },
492 | "PolicyName": "root",
493 | "Roles": [
494 | {
495 | "Ref": "RootRole"
496 | }
497 | ]
498 | },
499 | "Type": "AWS::IAM::Policy"
500 | },
501 | "RootInstanceProfile": {
502 | "Properties": {
503 | "Path": "/",
504 | "Roles": [
505 | {
506 | "Ref": "RootRole"
507 | }
508 | ]
509 | },
510 | "Type": "AWS::IAM::InstanceProfile"
511 | },
512 | "RootRole": {
513 | "Properties": {
514 | "AssumeRolePolicyDocument": {
515 | "Statement": [
516 | {
517 | "Action": [
518 | "sts:AssumeRole"
519 | ],
520 | "Effect": "Allow",
521 | "Principal": {
522 | "Service": [
523 | "ec2.amazonaws.com"
524 | ]
525 | }
526 | }
527 | ]
528 | },
529 | "Path": "/"
530 | },
531 | "Type": "AWS::IAM::Role"
532 | }
533 | }
534 | }
--------------------------------------------------------------------------------
/riak-vpc-cluster-with-frontend-appservers.json:
--------------------------------------------------------------------------------
1 | {
2 | "AWSTemplateFormatVersion": "2010-09-09",
3 | "Description": "Launches a Riak Cluster",
4 | "Mappings": {
5 | "AWSInstanceType2Arch": {
6 | "c1.medium": {
7 | "Arch": "64"
8 | },
9 | "c1.xlarge": {
10 | "Arch": "64"
11 | },
12 | "cc1.4xlarge": {
13 | "Arch": "64Cluster"
14 | },
15 | "cc2.8xlarge": {
16 | "Arch": "64Cluster"
17 | },
18 | "cg1.4xlarge": {
19 | "Arch": "64GPU"
20 | },
21 | "m1.large": {
22 | "Arch": "64"
23 | },
24 | "m1.medium": {
25 | "Arch": "64"
26 | },
27 | "m1.small": {
28 | "Arch": "64"
29 | },
30 | "m1.xlarge": {
31 | "Arch": "64"
32 | },
33 | "m2.2xlarge": {
34 | "Arch": "64"
35 | },
36 | "m2.4xlarge": {
37 | "Arch": "64"
38 | },
39 | "m2.xlarge": {
40 | "Arch": "64"
41 | },
42 | "t1.micro": {
43 | "Arch": "64"
44 | }
45 | },
46 | "AWSNATAMI": {
47 | "ap-northeast-1": {
48 | "AMI": "ami-14d86d15"
49 | },
50 | "ap-southeast-1": {
51 | "AMI": "ami-02eb9350"
52 | },
53 | "eu-west-1": {
54 | "AMI": "ami-0b5b6c7f"
55 | },
56 | "sa-east-1": {
57 | "AMI": "ami-0439e619"
58 | },
59 | "us-east-1": {
60 | "AMI": "ami-c6699baf"
61 | },
62 | "us-west-1": {
63 | "AMI": "ami-3bcc9e7e"
64 | },
65 | "us-west-2": {
66 | "AMI": "ami-52ff7262"
67 | }
68 | },
69 | "SubnetConfig": {
70 | "Private": {
71 | "CIDR": "10.0.1.0/24"
72 | },
73 | "Public": {
74 | "CIDR": "10.0.0.0/24"
75 | },
76 | "VPC": {
77 | "CIDR": "10.0.0.0/16"
78 | }
79 | },
80 | "ebs": {
81 | "ap-northeast-1": {
82 | "32": "ami-486cd349 ",
83 | "64": "ami-4e6cd34f",
84 | "64Cluster": "NOT_YET_SUPPORTED",
85 | "64GPU": "NOT_YET_SUPPORTED"
86 | },
87 | "ap-southeast-1": {
88 | "32": "ami-a2a7e7f0",
89 | "64": "ami-a6a7e7f4",
90 | "64Cluster": "NOT_YET_SUPPORTED",
91 | "64GPU": "NOT_YET_SUPPORTED"
92 | },
93 | "eu-west-1": {
94 | "32": "ami-937474e7",
95 | "64": "ami-c37474b7",
96 | "64Cluster": "ami-d97474ad",
97 | "64GPU": "ami-1b02026f"
98 | },
99 | "sa-east-1": {
100 | "32": "ami-e209d0ff",
101 | "64": "ami-1e08d103",
102 | "64Cluster": "NOT_YET_SUPPORTED",
103 | "64GPU": "NOT_YET_SUPPORTED"
104 | },
105 | "us-east-1": {
106 | "32": "ami-1a249873",
107 | "64": "ami-1624987f",
108 | "64Cluster": "ami-08249861",
109 | "64GPU": "ami-02f54a6b"
110 | },
111 | "us-west-1": {
112 | "32": "ami-19f9de5c",
113 | "64": "ami-1bf9de5e",
114 | "64Cluster": "NOT_YET_SUPPORTED",
115 | "64GPU": "NOT_YET_SUPPORTED"
116 | },
117 | "us-west-2": {
118 | "32": "ami-2231bf12",
119 | "64": "ami-2a31bf1a",
120 | "64Cluster": "ami-2431bf14",
121 | "64GPU": "NOT_YET_SUPPORTED"
122 | }
123 | },
124 | "ephemeral": {
125 | "ap-northeast-1": {
126 | "32": "ami-586cd359",
127 | "64": "ami-5a6cd35b"
128 | },
129 | "ap-southeast-1": {
130 | "32": "ami-aaa7e7f8",
131 | "64": "ami-a8a7e7fa"
132 | },
133 | "eu-west-1": {
134 | "32": "ami-cf7474bb",
135 | "64": "ami-b57474c1"
136 | },
137 | "sa-east-1": {
138 | "32": " ami-1a08d107",
139 | "64": " ami-1608d10b"
140 | },
141 | "us-east-1": {
142 | "32": "ami-10249879",
143 | "64": "ami-e8249881"
144 | },
145 | "us-west-1": {
146 | "32": "ami-27f9de62",
147 | "64": "ami-21f9de64"
148 | },
149 | "us-west-2": {
150 | "32": "ami-2c31bf1c",
151 | "64": "ami-2e31bf1e"
152 | }
153 | }
154 | },
155 | "Outputs": {
156 | "Bastion": {
157 | "Description": "IP Address of the Bastion host",
158 | "Value": {
159 | "Ref": "BastionIPAddress"
160 | }
161 | },
162 | "WebSite": {
163 | "Description": "URL of the website",
164 | "Value": {
165 | "Fn::Join": [
166 | "",
167 | [
168 | "http://",
169 | {
170 | "Fn::GetAtt": [
171 | "PublicElasticLoadBalancer",
172 | "DNSName"
173 | ]
174 | }
175 | ]
176 | ]
177 | }
178 | }
179 | },
180 | "Parameters": {
181 | "BastionInstanceType": {
182 | "AllowedValues": [
183 | "t1.micro",
184 | "m1.small",
185 | "m1.medium",
186 | "m1.large",
187 | "m1.xlarge",
188 | "m2.xlarge",
189 | "m2.2xlarge",
190 | "m2.4xlarge",
191 | "c1.medium",
192 | "c1.xlarge",
193 | "cc1.4xlarge",
194 | "cc2.8xlarge",
195 | "cg1.4xlarge"
196 | ],
197 | "ConstraintDescription": "must be a valid EC2 instance type.",
198 | "Default": "m1.small",
199 | "Description": "Bastion Host EC2 instance type",
200 | "Type": "String"
201 | },
202 | "DiskType": {
203 | "AllowedValues": [
204 | "ephemeral",
205 | "ebs"
206 | ],
207 | "Default": "ephemeral",
208 | "Description": "Type of Disk to use ( ephemeral/ebs )",
209 | "Type": "String"
210 | },
211 | "FrontendClusterSize": {
212 | "Default": "2",
213 | "Description": "Number of EC2 instances to launch for the Frontend cluster.",
214 | "Type": "Number"
215 | },
216 | "FrontendInstanceType": {
217 | "AllowedValues": [
218 | "t1.micro",
219 | "m1.small",
220 | "m1.medium",
221 | "m1.large",
222 | "m1.xlarge",
223 | "m2.xlarge",
224 | "m2.2xlarge",
225 | "m2.4xlarge",
226 | "c1.medium",
227 | "c1.xlarge",
228 | "cc1.4xlarge",
229 | "cc2.8xlarge",
230 | "cg1.4xlarge"
231 | ],
232 | "ConstraintDescription": "must be a valid EC2 instance type.",
233 | "Default": "m1.small",
234 | "Description": "Frontend Server EC2 instance type",
235 | "Type": "String"
236 | },
237 | "KeyName": {
238 | "AllowedPattern": "[-_ a-zA-Z0-9]*",
239 | "ConstraintDescription": "can contain only alphanumeric characters, spaces, dashes and underscores.",
240 | "Description": "Name of an existing EC2 KeyPair to enable SSH access to the instances",
241 | "MaxLength": "64",
242 | "MinLength": "1",
243 | "Type": "String"
244 | },
245 | "NATInstanceType": {
246 | "AllowedValues": [
247 | "t1.micro",
248 | "m1.small",
249 | "m1.medium",
250 | "m1.large",
251 | "m1.xlarge",
252 | "m2.xlarge",
253 | "m2.2xlarge",
254 | "m2.4xlarge",
255 | "c1.medium",
256 | "c1.xlarge",
257 | "cc1.4xlarge",
258 | "cc2.8xlarge",
259 | "cg1.4xlarge"
260 | ],
261 | "ConstraintDescription": "must be a valid EC2 instance type.",
262 | "Default": "m1.small",
263 | "Description": "NET Device EC2 instance type",
264 | "Type": "String"
265 | },
266 | "RiakClusterSize": {
267 | "Default": "3",
268 | "Description": "Number of EC2 instances to launch for Riak Cluster",
269 | "Type": "Number"
270 | },
271 | "RiakInstanceType": {
272 | "AllowedValues": [
273 | "t1.micro",
274 | "m1.small",
275 | "m1.medium",
276 | "m1.large",
277 | "m1.xlarge",
278 | "m2.xlarge",
279 | "m2.2xlarge",
280 | "m2.4xlarge",
281 | "c1.medium",
282 | "c1.xlarge",
283 | "cc1.4xlarge",
284 | "cc2.8xlarge",
285 | "cg1.4xlarge"
286 | ],
287 | "ConstraintDescription": "must be a valid EC2 instance type.",
288 | "Default": "m1.small",
289 | "Description": "Riak Server EC2 instance type",
290 | "Type": "String"
291 | },
292 | "RingSize": {
293 | "AllowedValues": [
294 | "64",
295 | "128",
296 | "256",
297 | "512"
298 | ],
299 | "Default": "64",
300 | "Description": "Size of the Riak Ring",
301 | "Type": "String"
302 | },
303 | "SSHFrom": {
304 | "AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})",
305 | "ConstraintDescription": "must be a valid CIDR range of the form x.x.x.x/x.",
306 | "Default": "0.0.0.0/0",
307 | "Description": "Lockdown SSH access to the bastion host (default can be accessed from anywhere)",
308 | "MaxLength": "18",
309 | "MinLength": "9",
310 | "Type": "String"
311 | }
312 | },
313 | "Resources": {
314 | "BackendFleet": {
315 | "Properties": {
316 | "AvailabilityZones": [
317 | {
318 | "Fn::GetAtt": [
319 | "PrivateSubnet",
320 | "AvailabilityZone"
321 | ]
322 | }
323 | ],
324 | "DesiredCapacity": {
325 | "Ref": "RiakClusterSize"
326 | },
327 | "LaunchConfigurationName": {
328 | "Ref": "BackendLaunchConfig"
329 | },
330 | "LoadBalancerNames": [
331 | {
332 | "Ref": "PrivateElasticLoadBalancer"
333 | }
334 | ],
335 | "MaxSize": "10",
336 | "MinSize": "1",
337 | "Tags": [
338 | {
339 | "Key": "Network",
340 | "PropagateAtLaunch": "true",
341 | "Value": "Private"
342 | }
343 | ],
344 | "VPCZoneIdentifier": [
345 | {
346 | "Ref": "PrivateSubnet"
347 | }
348 | ]
349 | },
350 | "Type": "AWS::AutoScaling::AutoScalingGroup"
351 | },
352 | "BackendLaunchConfig": {
353 | "DependsOn": [
354 | "BastionIPAddress",
355 | "NATIPAddress",
356 | "PrivateSubnetRouteTableAssociation",
357 | "PrivateSubnetNetworkAclAssociation"
358 | ],
359 | "Properties": {
360 | "IamInstanceProfile": {
361 | "Ref": "RootInstanceProfile"
362 | },
363 | "ImageId": {
364 | "Fn::FindInMap": [
365 | {
366 | "Ref": "DiskType"
367 | },
368 | {
369 | "Ref": "AWS::Region"
370 | },
371 | {
372 | "Fn::FindInMap": [
373 | "AWSInstanceType2Arch",
374 | {
375 | "Ref": "RiakInstanceType"
376 | },
377 | "Arch"
378 | ]
379 | }
380 | ]
381 | },
382 | "InstanceType": {
383 | "Ref": "RiakInstanceType"
384 | },
385 | "KeyName": {
386 | "Ref": "KeyName"
387 | },
388 | "SecurityGroups": [
389 | {
390 | "Ref": "BackendSecurityGroup"
391 | }
392 | ],
393 | "UserData": {
394 | "Fn::Base64": {
395 | "Fn::Join": [
396 | "\n",
397 | [
398 | "#!/bin/bash -v",
399 | "exec > >(tee /var/log/cfn-data.log|logger -t user-data -s 2>/dev/console) 2>&1",
400 | "",
401 | "sleep 10",
402 | "",
403 | "function retry {",
404 | " nTrys=0",
405 | " maxTrys=5",
406 | " status=256",
407 | " until [ $status == 0 ] ; do",
408 | " $1",
409 | " status=$?",
410 | " nTrys=$(($nTrys + 1))",
411 | " if [ $nTrys -gt $maxTrys ] ; then",
412 | " echo \"Number of re-trys exceeded. Exit code: $status\"",
413 | " exit $status",
414 | " fi",
415 | " if [ $status != 0 ] ; then",
416 | " echo \"Failed (exit code $status)... retry $nTrys\"",
417 | " sleep 10",
418 | " fi",
419 | " done",
420 | "}",
421 | "",
422 | "yum update -y aws-cfn-bootstrap",
423 | "",
424 | "#for all the stuff that complains about sudo and tty",
425 | "sed -i 's,Defaults requiretty,#Defaults requiretty,g' /etc/sudoers",
426 | "",
427 | "function error_exit",
428 | "{",
429 | {
430 | "Fn::Join": [
431 | "",
432 | [
433 | " /opt/aws/bin/cfn-signal -e 1 -r \"$1\" '",
434 | {
435 | "Ref": "BackendWaitHandle"
436 | },
437 | "'"
438 | ]
439 | ]
440 | },
441 | "}",
442 | "yum update -y aws-cfn-bootstrap",
443 | "#this runs the first stage of cfinit",
444 | {
445 | "Fn::Join": [
446 | "",
447 | [
448 | "#/opt/aws/bin/cfn-init -c ascending -v --region ",
449 | {
450 | "Ref": "AWS::Region"
451 | },
452 | " -s ",
453 | {
454 | "Ref": "AWS::StackName"
455 | },
456 | " -r ",
457 | "BackendLaunchConfig",
458 | " --access-key ",
459 | {
460 | "Ref": "HostKeys"
461 | },
462 | " --secret-key ",
463 | {
464 | "Fn::GetAtt": [
465 | "HostKeys",
466 | "SecretAccessKey"
467 | ]
468 | },
469 | " || error_exit 'Failed to initialize client using cfn-init'"
470 | ]
471 | ]
472 | },
473 | "",
474 | "",
475 | "",
476 | "",
477 | "yum install -y rubygems git puppet ruby-devel make gcc",
478 | "gem install hiera hiera-puppet",
479 | "mkdir -p /var/lib/hiera",
480 | "#enables search",
481 | "echo 'search_enabled: true' >> '/var/lib/hiera/common.yaml'",
482 | "",
483 | {
484 | "Fn::Join": [
485 | "",
486 | [
487 | "echo 'ring_creation_size: ",
488 | {
489 | "Ref": "RingSize"
490 | },
491 | "' >> '/var/lib/hiera/common.yaml'"
492 | ]
493 | ]
494 | },
495 | "",
496 | "mkdir -p /tmp/puppet/modules",
497 | "retry \"git clone http://github.com/basho/puppet-riak -b cfn /tmp/puppet/modules/riak\"",
498 | "retry \"git clone https://github.com/KrisBuytaert/puppet-hiera.git /tmp/puppet/modules/puppet-hiera\"",
499 | "retry \"git clone https://github.com/puppetlabs/puppetlabs-stdlib /tmp/puppet/modules/stdlib\"",
500 | "puppet apply --modulepath=/tmp/puppet/modules -e \"include riak\" --trace --debug --verbose",
501 | "",
502 | "echo \"riak soft nofile 65336\" >> /etc/security/limits.conf",
503 | "echo \"riak hard nofile 65336\" >> /etc/security/limits.conf",
504 | "echo \"root soft nofile 65336\" >> /etc/security/limits.conf",
505 | "echo \"root hard nofile 65336\" >> /etc/security/limits.conf",
506 | "",
507 | "echo \"vm.swappiness = 0\" >> /etc/sysctl.conf",
508 | "echo \"net.ipv4.tcp_max_syn_backlog = 40000\" >> /etc/sysctl.conf",
509 | "echo \"net.core.somaxconn=4000\" >> /etc/sysctl.conf",
510 | "echo \"net.ipv4.tcp_timestamps = 0\" >> /etc/sysctl.conf",
511 | "echo \"net.ipv4.tcp_sack = 1\" >> /etc/sysctl.conf",
512 | "echo \"net.ipv4.tcp_window_scaling = 1\" >> /etc/sysctl.conf",
513 | "echo \"net.ipv4.tcp_fin_timeout = 15\" >> /etc/sysctl.conf",
514 | "echo \"net.ipv4.tcp_keepalive_intvl = 30\" >> /etc/sysctl.conf",
515 | "echo \"net.ipv4.tcp_tw_reuse = 1\" >> /etc/sysctl.conf",
516 | "echo \"+P 256000\" >> /etc/riak/vm.args",
517 | "echo \"+swt very_low\" >> /etc/riak/vm.args",
518 | "echo \"+zdbbl 32768\" >> /etc/riak/vm.args",
519 | "sysctl -p",
520 | "",
521 | "service riak stop",
522 | "service riak start",
523 | "sleep 5",
524 | "retry \"git clone http://github.com/basho/cloudformation-riak /tmp/cloudformation-riak\"",
525 | "/tmp/cloudformation-riak/tools/join_riak_cluster.py",
526 | "search-cmd install riagi-users",
527 | "search-cmd install riagi-image-metadata",
528 | "/usr/sbin/riak ping",
529 | "result_code=$?",
530 | {
531 | "Fn::Join": [
532 | "",
533 | [
534 | "/opt/aws/bin/cfn-signal -e $result_code '",
535 | {
536 | "Ref": "BackendWaitHandle"
537 | },
538 | "'"
539 | ]
540 | ]
541 | },
542 | "#turn back on requiretty for sudo",
543 | "sed -i 's,#Defaults requiretty,Defaults requiretty,g' /etc/sudoers"
544 | ]
545 | ]
546 | }
547 | }
548 | },
549 | "Type": "AWS::AutoScaling::LaunchConfiguration"
550 | },
551 | "BackendSecurityGroup": {
552 | "Properties": {
553 | "GroupDescription": "Allow access from private load balancer and bastion as well as outbound HTTP and HTTPS traffic",
554 | "SecurityGroupEgress": [
555 | {
556 | "CidrIp": "0.0.0.0/0",
557 | "FromPort": "80",
558 | "IpProtocol": "tcp",
559 | "ToPort": "80"
560 | },
561 | {
562 | "CidrIp": "0.0.0.0/0",
563 | "FromPort": "443",
564 | "IpProtocol": "tcp",
565 | "ToPort": "443"
566 | }
567 | ],
568 | "SecurityGroupIngress": [
569 | {
570 | "FromPort": "8087",
571 | "IpProtocol": "tcp",
572 | "SourceSecurityGroupId": {
573 | "Ref": "PrivateLoadBalancerSecurityGroup"
574 | },
575 | "ToPort": "8087"
576 | },
577 | {
578 | "FromPort": "8098",
579 | "IpProtocol": "tcp",
580 | "SourceSecurityGroupId": {
581 | "Ref": "PrivateLoadBalancerSecurityGroup"
582 | },
583 | "ToPort": "8098"
584 | },
585 | {
586 | "FromPort": "22",
587 | "IpProtocol": "tcp",
588 | "SourceSecurityGroupId": {
589 | "Ref": "BastionSecurityGroup"
590 | },
591 | "ToPort": "22"
592 | }
593 | ],
594 | "VpcId": {
595 | "Ref": "VPC"
596 | }
597 | },
598 | "Type": "AWS::EC2::SecurityGroup"
599 | },
600 | "BackendWaitCondition": {
601 | "DependsOn": "BackendFleet",
602 | "Properties": {
603 | "Count": {
604 | "Ref": "RiakClusterSize"
605 | },
606 | "Handle": {
607 | "Ref": "BackendWaitHandle"
608 | },
609 | "Timeout": "600"
610 | },
611 | "Type": "AWS::CloudFormation::WaitCondition"
612 | },
613 | "BackendWaitHandle": {
614 | "Type": "AWS::CloudFormation::WaitConditionHandle"
615 | },
616 | "BastionHost": {
617 | "DependsOn": [
618 | "PublicSubnetRouteTableAssociation",
619 | "PublicSubnetNetworkAclAssociation"
620 | ],
621 | "Properties": {
622 | "ImageId": {
623 | "Fn::FindInMap": [
624 | "ephemeral",
625 | {
626 | "Ref": "AWS::Region"
627 | },
628 | {
629 | "Fn::FindInMap": [
630 | "AWSInstanceType2Arch",
631 | {
632 | "Ref": "BastionInstanceType"
633 | },
634 | "Arch"
635 | ]
636 | }
637 | ]
638 | },
639 | "InstanceType": {
640 | "Ref": "BastionInstanceType"
641 | },
642 | "KeyName": {
643 | "Ref": "KeyName"
644 | },
645 | "SecurityGroupIds": [
646 | {
647 | "Ref": "BastionSecurityGroup"
648 | }
649 | ],
650 | "SubnetId": {
651 | "Ref": "PublicSubnet"
652 | }
653 | },
654 | "Type": "AWS::EC2::Instance"
655 | },
656 | "BastionIPAddress": {
657 | "DependsOn": "GatewayToInternet",
658 | "Properties": {
659 | "Domain": "vpc",
660 | "InstanceId": {
661 | "Ref": "BastionHost"
662 | }
663 | },
664 | "Type": "AWS::EC2::EIP"
665 | },
666 | "BastionSecurityGroup": {
667 | "Properties": {
668 | "GroupDescription": "Enable access to the Bastion host",
669 | "SecurityGroupEgress": [
670 | {
671 | "CidrIp": {
672 | "Fn::FindInMap": [
673 | "SubnetConfig",
674 | "Private",
675 | "CIDR"
676 | ]
677 | },
678 | "FromPort": "22",
679 | "IpProtocol": "tcp",
680 | "ToPort": "22"
681 | }
682 | ],
683 | "SecurityGroupIngress": [
684 | {
685 | "CidrIp": {
686 | "Ref": "SSHFrom"
687 | },
688 | "FromPort": "22",
689 | "IpProtocol": "tcp",
690 | "ToPort": "22"
691 | }
692 | ],
693 | "VpcId": {
694 | "Ref": "VPC"
695 | }
696 | },
697 | "Type": "AWS::EC2::SecurityGroup"
698 | },
699 | "CFNInitUser": {
700 | "Properties": {
701 | "Path": "/",
702 | "Policies": [
703 | {
704 | "PolicyDocument": {
705 | "Statement": [
706 | {
707 | "Action": [
708 | "cloudformation:DescribeStackResource",
709 | "s3:GetObject"
710 | ],
711 | "Effect": "Allow",
712 | "Resource": "*"
713 | }
714 | ]
715 | },
716 | "PolicyName": "AccessForCFNInit"
717 | }
718 | ]
719 | },
720 | "Type": "AWS::IAM::User"
721 | },
722 | "ClusterCommunication1": {
723 | "Properties": {
724 | "FromPort": "-1",
725 | "GroupId": {
726 | "Ref": "BackendSecurityGroup"
727 | },
728 | "IpProtocol": "icmp",
729 | "SourceSecurityGroupId": {
730 | "Ref": "BackendSecurityGroup"
731 | },
732 | "ToPort": "-1"
733 | },
734 | "Type": "AWS::EC2::SecurityGroupIngress"
735 | },
736 | "ClusterCommunication2": {
737 | "Properties": {
738 | "FromPort": "1",
739 | "GroupId": {
740 | "Ref": "BackendSecurityGroup"
741 | },
742 | "IpProtocol": "tcp",
743 | "SourceSecurityGroupId": {
744 | "Ref": "BackendSecurityGroup"
745 | },
746 | "ToPort": "65356"
747 | },
748 | "Type": "AWS::EC2::SecurityGroupIngress"
749 | },
750 | "ClusterCommunication3": {
751 | "Properties": {
752 | "FromPort": "1",
753 | "GroupId": {
754 | "Ref": "BackendSecurityGroup"
755 | },
756 | "IpProtocol": "udp",
757 | "SourceSecurityGroupId": {
758 | "Ref": "BackendSecurityGroup"
759 | },
760 | "ToPort": "65356"
761 | },
762 | "Type": "AWS::EC2::SecurityGroupIngress"
763 | },
764 | "ClusterCommunication4": {
765 | "Properties": {
766 | "FromPort": "-1",
767 | "GroupId": {
768 | "Ref": "BackendSecurityGroup"
769 | },
770 | "IpProtocol": "icmp",
771 | "SourceSecurityGroupId": {
772 | "Ref": "BackendSecurityGroup"
773 | },
774 | "ToPort": "-1"
775 | },
776 | "Type": "AWS::EC2::SecurityGroupEgress"
777 | },
778 | "ClusterCommunication5": {
779 | "Properties": {
780 | "FromPort": "1",
781 | "GroupId": {
782 | "Ref": "BackendSecurityGroup"
783 | },
784 | "IpProtocol": "tcp",
785 | "SourceSecurityGroupId": {
786 | "Ref": "BackendSecurityGroup"
787 | },
788 | "ToPort": "65356"
789 | },
790 | "Type": "AWS::EC2::SecurityGroupEgress"
791 | },
792 | "ClusterCommunication6": {
793 | "Properties": {
794 | "FromPort": "1",
795 | "GroupId": {
796 | "Ref": "BackendSecurityGroup"
797 | },
798 | "IpProtocol": "udp",
799 | "SourceSecurityGroupId": {
800 | "Ref": "BackendSecurityGroup"
801 | },
802 | "ToPort": "65356"
803 | },
804 | "Type": "AWS::EC2::SecurityGroupEgress"
805 | },
806 | "FrontendClustertoBackendClusterCommunication1": {
807 | "Properties": {
808 | "DestinationSecurityGroupId": {
809 | "Ref": "PrivateLoadBalancerSecurityGroup"
810 | },
811 | "FromPort": "8098",
812 | "GroupId": {
813 | "Ref": "FrontendSecurityGroup"
814 | },
815 | "IpProtocol": "tcp",
816 | "ToPort": "8098"
817 | },
818 | "Type": "AWS::EC2::SecurityGroupEgress"
819 | },
820 | "FrontendClustertoBackendClusterCommunication2": {
821 | "Properties": {
822 | "DestinationSecurityGroupId": {
823 | "Ref": "PrivateLoadBalancerSecurityGroup"
824 | },
825 | "FromPort": "8087",
826 | "GroupId": {
827 | "Ref": "FrontendSecurityGroup"
828 | },
829 | "IpProtocol": "tcp",
830 | "ToPort": "8087"
831 | },
832 | "Type": "AWS::EC2::SecurityGroupEgress"
833 | },
834 | "FrontendFleet": {
835 | "Properties": {
836 | "AvailabilityZones": [
837 | {
838 | "Fn::GetAtt": [
839 | "PrivateSubnet",
840 | "AvailabilityZone"
841 | ]
842 | }
843 | ],
844 | "DesiredCapacity": {
845 | "Ref": "FrontendClusterSize"
846 | },
847 | "LaunchConfigurationName": {
848 | "Ref": "FrontendServerLaunchConfig"
849 | },
850 | "LoadBalancerNames": [
851 | {
852 | "Ref": "PublicElasticLoadBalancer"
853 | }
854 | ],
855 | "MaxSize": "10",
856 | "MinSize": "1",
857 | "Tags": [
858 | {
859 | "Key": "Network",
860 | "PropagateAtLaunch": "true",
861 | "Value": "Public"
862 | }
863 | ],
864 | "VPCZoneIdentifier": [
865 | {
866 | "Ref": "PrivateSubnet"
867 | }
868 | ]
869 | },
870 | "Type": "AWS::AutoScaling::AutoScalingGroup"
871 | },
872 | "FrontendSecurityGroup": {
873 | "Properties": {
874 | "GroupDescription": "Allow access from load balancer and bastion as well as outbound HTTP and HTTPS traffic",
875 | "SecurityGroupEgress": [
876 | {
877 | "CidrIp": "0.0.0.0/0",
878 | "FromPort": "80",
879 | "IpProtocol": "tcp",
880 | "ToPort": "80"
881 | },
882 | {
883 | "CidrIp": "0.0.0.0/0",
884 | "FromPort": "443",
885 | "IpProtocol": "tcp",
886 | "ToPort": "443"
887 | }
888 | ],
889 | "SecurityGroupIngress": [
890 | {
891 | "FromPort": "80",
892 | "IpProtocol": "tcp",
893 | "SourceSecurityGroupId": {
894 | "Ref": "PublicLoadBalancerSecurityGroup"
895 | },
896 | "ToPort": "80"
897 | },
898 | {
899 | "FromPort": "443",
900 | "IpProtocol": "tcp",
901 | "SourceSecurityGroupId": {
902 | "Ref": "PublicLoadBalancerSecurityGroup"
903 | },
904 | "ToPort": "443"
905 | },
906 | {
907 | "FromPort": "22",
908 | "IpProtocol": "tcp",
909 | "SourceSecurityGroupId": {
910 | "Ref": "BastionSecurityGroup"
911 | },
912 | "ToPort": "22"
913 | }
914 | ],
915 | "VpcId": {
916 | "Ref": "VPC"
917 | }
918 | },
919 | "Type": "AWS::EC2::SecurityGroup"
920 | },
921 | "FrontendServerLaunchConfig": {
922 | "DependsOn": "BastionHost",
923 | "Properties": {
924 | "ImageId": {
925 | "Fn::FindInMap": [
926 | "ephemeral",
927 | {
928 | "Ref": "AWS::Region"
929 | },
930 | {
931 | "Fn::FindInMap": [
932 | "AWSInstanceType2Arch",
933 | {
934 | "Ref": "FrontendInstanceType"
935 | },
936 | "Arch"
937 | ]
938 | }
939 | ]
940 | },
941 | "InstanceType": {
942 | "Ref": "FrontendInstanceType"
943 | },
944 | "KeyName": {
945 | "Ref": "KeyName"
946 | },
947 | "SecurityGroups": [
948 | {
949 | "Ref": "FrontendSecurityGroup"
950 | }
951 | ],
952 | "UserData": {
953 | "Fn::Base64": {
954 | "Fn::Join": [
955 | "\n",
956 | [
957 | "#!/bin/bash -v",
958 | "exec > >(tee /var/log/cfn-data.log|logger -t user-data -s 2>/dev/console) 2>&1",
959 | "",
960 | "sleep 10",
961 | "",
962 | "function retry {",
963 | " nTrys=0",
964 | " maxTrys=5",
965 | " status=256",
966 | " until [ $status == 0 ] ; do",
967 | " $1",
968 | " status=$?",
969 | " nTrys=$(($nTrys + 1))",
970 | " if [ $nTrys -gt $maxTrys ] ; then",
971 | " echo \"Number of re-trys exceeded. Exit code: $status\"",
972 | " exit $status",
973 | " fi",
974 | " if [ $status != 0 ] ; then",
975 | " echo \"Failed (exit code $status)... retry $nTrys\"",
976 | " sleep 10",
977 | " fi",
978 | " done",
979 | "}",
980 | "",
981 | "yum update -y aws-cfn-bootstrap",
982 | "",
983 | "#for all the stuff that complains about sudo and tty",
984 | "sed -i 's,Defaults requiretty,#Defaults requiretty,g' /etc/sudoers",
985 | "",
986 | "function error_exit",
987 | "{",
988 | {
989 | "Fn::Join": [
990 | "",
991 | [
992 | " /opt/aws/bin/cfn-signal -e 1 -r \"$1\" '",
993 | {
994 | "Ref": "FrontendWaitHandle"
995 | },
996 | "'"
997 | ]
998 | ]
999 | },
1000 | "}",
1001 | "yum update -y aws-cfn-bootstrap",
1002 | "#this runs the first stage of cfinit",
1003 | {
1004 | "Fn::Join": [
1005 | "",
1006 | [
1007 | "#/opt/aws/bin/cfn-init -c ascending -v --region ",
1008 | {
1009 | "Ref": "AWS::Region"
1010 | },
1011 | " -s ",
1012 | {
1013 | "Ref": "AWS::StackName"
1014 | },
1015 | " -r ",
1016 | "FrontendLaunchConfig",
1017 | " --access-key ",
1018 | {
1019 | "Ref": "HostKeys"
1020 | },
1021 | " --secret-key ",
1022 | {
1023 | "Fn::GetAtt": [
1024 | "HostKeys",
1025 | "SecretAccessKey"
1026 | ]
1027 | },
1028 | " || error_exit 'Failed to initialize client using cfn-init'"
1029 | ]
1030 | ]
1031 | },
1032 | "",
1033 | "",
1034 | "",
1035 | "",
1036 | "yum install -y rubygems git puppet ruby-devel make gcc",
1037 | "gem install hiera hiera-puppet",
1038 | "mkdir -p /var/lib/hiera",
1039 | "RIAGI_HOME=/usr/local/riagi",
1040 | "",
1041 | "git clone https://github.com/jsmartin/riagi $RIAGI_HOME",
1042 | "cd $RIAGI_HOME",
1043 | "cp django_riak.py /usr/lib/python2.6/site-packages/",
1044 | "",
1045 | "",
1046 | "yum -y install Django14 --enablerepo=epel",
1047 | "",
1048 | "yum -y install python-imaging python-pip protobuf-c protobuf-compiler httpd mod_wsgi --enablerepo=epel",
1049 | "pip-python install riak==1.5",
1050 | "pip-python install shortuuid",
1051 | "",
1052 | "cat << EOF >> /etc/httpd/conf/httpd.conf",
1053 | "Alias /static/ $RIAGI_HOME/riagi/static/",
1054 | "Alias /media/ $RIAGI_HOME/riagi/media/",
1055 | "",
1056 | "WSGIScriptAlias / $RIAGI_HOME/riagi/wsgi.py",
1057 | "WSGIPythonPath $RIAGI_HOME",
1058 | "",
1059 | "",
1060 | "",
1061 | "Order deny,allow",
1062 | "Allow from all",
1063 | "",
1064 | "",
1065 | "",
1066 | "",
1067 | "Order deny,allow",
1068 | "Allow from all",
1069 | "",
1070 | "",
1071 | "",
1072 | "Order deny,allow",
1073 | "Allow from all",
1074 | "",
1075 | "EOF",
1076 | "",
1077 | "mkdir /etc/riagi",
1078 | "cat << EOF > /etc/riagi/riak_lb.yaml",
1079 | "loadbalancer:",
1080 | {
1081 | "Fn::Join": [
1082 | "",
1083 | [
1084 | " hostname: ",
1085 | {
1086 | "Fn::GetAtt": [
1087 | "PrivateElasticLoadBalancer",
1088 | "DNSName"
1089 | ]
1090 | },
1091 | ""
1092 | ]
1093 | ]
1094 | },
1095 | " port: 8098",
1096 | "media:",
1097 | " url:",
1098 | " root:",
1099 | "static:",
1100 | " url: /static/",
1101 | " root: $RIAGI_HOME/riagi/static",
1102 | "EOF",
1103 | "",
1104 | "/sbin/service httpd start",
1105 | "",
1106 | "curl --max-time 120 --retry 3 --retry-max-time 120 localhost|grep basho",
1107 | "result_code=$?",
1108 | {
1109 | "Fn::Join": [
1110 | "",
1111 | [
1112 | "/opt/aws/bin/cfn-signal -e $result_code '",
1113 | {
1114 | "Ref": "FrontendWaitHandle"
1115 | },
1116 | "'"
1117 | ]
1118 | ]
1119 | },
1120 | "#turn back on requiretty for sudo",
1121 | "sed -i 's,#Defaults requiretty,Defaults requiretty,g' /etc/sudoers"
1122 | ]
1123 | ]
1124 | }
1125 | }
1126 | },
1127 | "Type": "AWS::AutoScaling::LaunchConfiguration"
1128 | },
1129 | "FrontendWaitCondition": {
1130 | "DependsOn": "FrontendFleet",
1131 | "Properties": {
1132 | "Count": {
1133 | "Ref": "FrontendClusterSize"
1134 | },
1135 | "Handle": {
1136 | "Ref": "FrontendWaitHandle"
1137 | },
1138 | "Timeout": "300"
1139 | },
1140 | "Type": "AWS::CloudFormation::WaitCondition"
1141 | },
1142 | "FrontendWaitHandle": {
1143 | "Type": "AWS::CloudFormation::WaitConditionHandle"
1144 | },
1145 | "GatewayToInternet": {
1146 | "Properties": {
1147 | "InternetGatewayId": {
1148 | "Ref": "InternetGateway"
1149 | },
1150 | "VpcId": {
1151 | "Ref": "VPC"
1152 | }
1153 | },
1154 | "Type": "AWS::EC2::VPCGatewayAttachment"
1155 | },
1156 | "HostKeys": {
1157 | "Properties": {
1158 | "UserName": {
1159 | "Ref": "CFNInitUser"
1160 | }
1161 | },
1162 | "Type": "AWS::IAM::AccessKey"
1163 | },
1164 | "InboundEmphemeralPublicNetworkAclEntry": {
1165 | "Properties": {
1166 | "CidrBlock": "0.0.0.0/0",
1167 | "Egress": "false",
1168 | "NetworkAclId": {
1169 | "Ref": "PublicNetworkAcl"
1170 | },
1171 | "PortRange": {
1172 | "From": "1024",
1173 | "To": "65535"
1174 | },
1175 | "Protocol": "6",
1176 | "RuleAction": "allow",
1177 | "RuleNumber": "103"
1178 | },
1179 | "Type": "AWS::EC2::NetworkAclEntry"
1180 | },
1181 | "InboundHTTPPublicNetworkAclEntry": {
1182 | "Properties": {
1183 | "CidrBlock": "0.0.0.0/0",
1184 | "Egress": "false",
1185 | "NetworkAclId": {
1186 | "Ref": "PublicNetworkAcl"
1187 | },
1188 | "PortRange": {
1189 | "From": "80",
1190 | "To": "80"
1191 | },
1192 | "Protocol": "6",
1193 | "RuleAction": "allow",
1194 | "RuleNumber": "100"
1195 | },
1196 | "Type": "AWS::EC2::NetworkAclEntry"
1197 | },
1198 | "InboundHTTPSPublicNetworkAclEntry": {
1199 | "Properties": {
1200 | "CidrBlock": "0.0.0.0/0",
1201 | "Egress": "false",
1202 | "NetworkAclId": {
1203 | "Ref": "PublicNetworkAcl"
1204 | },
1205 | "PortRange": {
1206 | "From": "443",
1207 | "To": "443"
1208 | },
1209 | "Protocol": "6",
1210 | "RuleAction": "allow",
1211 | "RuleNumber": "101"
1212 | },
1213 | "Type": "AWS::EC2::NetworkAclEntry"
1214 | },
1215 | "InboundPrivateNetworkAclEntry": {
1216 | "Properties": {
1217 | "CidrBlock": "0.0.0.0/0",
1218 | "Egress": "false",
1219 | "NetworkAclId": {
1220 | "Ref": "PrivateNetworkAcl"
1221 | },
1222 | "PortRange": {
1223 | "From": "0",
1224 | "To": "65535"
1225 | },
1226 | "Protocol": "6",
1227 | "RuleAction": "allow",
1228 | "RuleNumber": "100"
1229 | },
1230 | "Type": "AWS::EC2::NetworkAclEntry"
1231 | },
1232 | "InboundSSHPublicNetworkAclEntry": {
1233 | "Properties": {
1234 | "CidrBlock": {
1235 | "Ref": "SSHFrom"
1236 | },
1237 | "Egress": "false",
1238 | "NetworkAclId": {
1239 | "Ref": "PublicNetworkAcl"
1240 | },
1241 | "PortRange": {
1242 | "From": "22",
1243 | "To": "22"
1244 | },
1245 | "Protocol": "6",
1246 | "RuleAction": "allow",
1247 | "RuleNumber": "102"
1248 | },
1249 | "Type": "AWS::EC2::NetworkAclEntry"
1250 | },
1251 | "InternetGateway": {
1252 | "Properties": {
1253 | "Tags": [
1254 | {
1255 | "Key": "Application",
1256 | "Value": {
1257 | "Ref": "AWS::StackName"
1258 | }
1259 | },
1260 | {
1261 | "Key": "Network",
1262 | "Value": "Public"
1263 | }
1264 | ]
1265 | },
1266 | "Type": "AWS::EC2::InternetGateway"
1267 | },
1268 | "NATDevice": {
1269 | "Properties": {
1270 | "ImageId": {
1271 | "Fn::FindInMap": [
1272 | "AWSNATAMI",
1273 | {
1274 | "Ref": "AWS::Region"
1275 | },
1276 | "AMI"
1277 | ]
1278 | },
1279 | "InstanceType": {
1280 | "Ref": "NATInstanceType"
1281 | },
1282 | "KeyName": {
1283 | "Ref": "KeyName"
1284 | },
1285 | "SecurityGroupIds": [
1286 | {
1287 | "Ref": "NATSecurityGroup"
1288 | }
1289 | ],
1290 | "SourceDestCheck": "false",
1291 | "SubnetId": {
1292 | "Ref": "PublicSubnet"
1293 | }
1294 | },
1295 | "Type": "AWS::EC2::Instance"
1296 | },
1297 | "NATIPAddress": {
1298 | "DependsOn": "GatewayToInternet",
1299 | "Properties": {
1300 | "Domain": "vpc",
1301 | "InstanceId": {
1302 | "Ref": "NATDevice"
1303 | }
1304 | },
1305 | "Type": "AWS::EC2::EIP"
1306 | },
1307 | "NATSecurityGroup": {
1308 | "Properties": {
1309 | "GroupDescription": "Enable internal access to the NAT device",
1310 | "SecurityGroupEgress": [
1311 | {
1312 | "CidrIp": "0.0.0.0/0",
1313 | "FromPort": "80",
1314 | "IpProtocol": "tcp",
1315 | "ToPort": "80"
1316 | },
1317 | {
1318 | "CidrIp": "0.0.0.0/0",
1319 | "FromPort": "443",
1320 | "IpProtocol": "tcp",
1321 | "ToPort": "443"
1322 | }
1323 | ],
1324 | "SecurityGroupIngress": [
1325 | {
1326 | "CidrIp": "0.0.0.0/0",
1327 | "FromPort": "80",
1328 | "IpProtocol": "tcp",
1329 | "ToPort": "80"
1330 | },
1331 | {
1332 | "CidrIp": "0.0.0.0/0",
1333 | "FromPort": "443",
1334 | "IpProtocol": "tcp",
1335 | "ToPort": "443"
1336 | },
1337 | {
1338 | "CidrIp": {
1339 | "Ref": "SSHFrom"
1340 | },
1341 | "FromPort": "22",
1342 | "IpProtocol": "tcp",
1343 | "ToPort": "22"
1344 | }
1345 | ],
1346 | "VpcId": {
1347 | "Ref": "VPC"
1348 | }
1349 | },
1350 | "Type": "AWS::EC2::SecurityGroup"
1351 | },
1352 | "OutBoundPrivateNetworkAclEntry": {
1353 | "Properties": {
1354 | "CidrBlock": "0.0.0.0/0",
1355 | "Egress": "true",
1356 | "NetworkAclId": {
1357 | "Ref": "PrivateNetworkAcl"
1358 | },
1359 | "PortRange": {
1360 | "From": "0",
1361 | "To": "65535"
1362 | },
1363 | "Protocol": "6",
1364 | "RuleAction": "allow",
1365 | "RuleNumber": "100"
1366 | },
1367 | "Type": "AWS::EC2::NetworkAclEntry"
1368 | },
1369 | "OutboundPublicNetworkAclEntry": {
1370 | "Properties": {
1371 | "CidrBlock": "0.0.0.0/0",
1372 | "Egress": "true",
1373 | "NetworkAclId": {
1374 | "Ref": "PublicNetworkAcl"
1375 | },
1376 | "PortRange": {
1377 | "From": "0",
1378 | "To": "65535"
1379 | },
1380 | "Protocol": "6",
1381 | "RuleAction": "allow",
1382 | "RuleNumber": "100"
1383 | },
1384 | "Type": "AWS::EC2::NetworkAclEntry"
1385 | },
1386 | "PrivateElasticLoadBalancer": {
1387 | "Properties": {
1388 | "HealthCheck": {
1389 | "HealthyThreshold": "3",
1390 | "Interval": "90",
1391 | "Target": "HTTP:8098/",
1392 | "Timeout": "60",
1393 | "UnhealthyThreshold": "5"
1394 | },
1395 | "Listeners": [
1396 | {
1397 | "InstancePort": "8098",
1398 | "LoadBalancerPort": "8098",
1399 | "Protocol": "HTTP"
1400 | },
1401 | {
1402 | "InstancePort": "8087",
1403 | "LoadBalancerPort": "8087",
1404 | "Protocol": "TCP"
1405 | }
1406 | ],
1407 | "Scheme": "internal",
1408 | "SecurityGroups": [
1409 | {
1410 | "Ref": "PrivateLoadBalancerSecurityGroup"
1411 | }
1412 | ],
1413 | "Subnets": [
1414 | {
1415 | "Ref": "PrivateSubnet"
1416 | }
1417 | ]
1418 | },
1419 | "Type": "AWS::ElasticLoadBalancing::LoadBalancer"
1420 | },
1421 | "PrivateLoadBalancerSecurityGroup": {
1422 | "Properties": {
1423 | "GroupDescription": "Private ELB Security Group with access on from the Frontend Fleet only",
1424 | "SecurityGroupIngress": [
1425 | {
1426 | "FromPort": "8087",
1427 | "IpProtocol": "tcp",
1428 | "SourceSecurityGroupId": {
1429 | "Ref": "FrontendSecurityGroup"
1430 | },
1431 | "ToPort": "8087"
1432 | },
1433 | {
1434 | "FromPort": "8098",
1435 | "IpProtocol": "tcp",
1436 | "SourceSecurityGroupId": {
1437 | "Ref": "FrontendSecurityGroup"
1438 | },
1439 | "ToPort": "8098"
1440 | }
1441 | ],
1442 | "VpcId": {
1443 | "Ref": "VPC"
1444 | }
1445 | },
1446 | "Type": "AWS::EC2::SecurityGroup"
1447 | },
1448 | "PrivateNetworkAcl": {
1449 | "Properties": {
1450 | "Tags": [
1451 | {
1452 | "Key": "Application",
1453 | "Value": {
1454 | "Ref": "AWS::StackName"
1455 | }
1456 | },
1457 | {
1458 | "Key": "Network",
1459 | "Value": "Private"
1460 | }
1461 | ],
1462 | "VpcId": {
1463 | "Ref": "VPC"
1464 | }
1465 | },
1466 | "Type": "AWS::EC2::NetworkAcl"
1467 | },
1468 | "PrivateRoute": {
1469 | "Properties": {
1470 | "DestinationCidrBlock": "0.0.0.0/0",
1471 | "InstanceId": {
1472 | "Ref": "NATDevice"
1473 | },
1474 | "RouteTableId": {
1475 | "Ref": "PrivateRouteTable"
1476 | }
1477 | },
1478 | "Type": "AWS::EC2::Route"
1479 | },
1480 | "PrivateRouteTable": {
1481 | "Properties": {
1482 | "Tags": [
1483 | {
1484 | "Key": "Application",
1485 | "Value": {
1486 | "Ref": "AWS::StackName"
1487 | }
1488 | },
1489 | {
1490 | "Key": "Network",
1491 | "Value": "Private"
1492 | }
1493 | ],
1494 | "VpcId": {
1495 | "Ref": "VPC"
1496 | }
1497 | },
1498 | "Type": "AWS::EC2::RouteTable"
1499 | },
1500 | "PrivateSubnet": {
1501 | "Properties": {
1502 | "CidrBlock": {
1503 | "Fn::FindInMap": [
1504 | "SubnetConfig",
1505 | "Private",
1506 | "CIDR"
1507 | ]
1508 | },
1509 | "Tags": [
1510 | {
1511 | "Key": "Application",
1512 | "Value": {
1513 | "Ref": "AWS::StackName"
1514 | }
1515 | },
1516 | {
1517 | "Key": "Network",
1518 | "Value": "Private"
1519 | }
1520 | ],
1521 | "VpcId": {
1522 | "Ref": "VPC"
1523 | }
1524 | },
1525 | "Type": "AWS::EC2::Subnet"
1526 | },
1527 | "PrivateSubnetNetworkAclAssociation": {
1528 | "Properties": {
1529 | "NetworkAclId": {
1530 | "Ref": "PrivateNetworkAcl"
1531 | },
1532 | "SubnetId": {
1533 | "Ref": "PrivateSubnet"
1534 | }
1535 | },
1536 | "Type": "AWS::EC2::SubnetNetworkAclAssociation"
1537 | },
1538 | "PrivateSubnetRouteTableAssociation": {
1539 | "Properties": {
1540 | "RouteTableId": {
1541 | "Ref": "PrivateRouteTable"
1542 | },
1543 | "SubnetId": {
1544 | "Ref": "PrivateSubnet"
1545 | }
1546 | },
1547 | "Type": "AWS::EC2::SubnetRouteTableAssociation"
1548 | },
1549 | "PublicElasticLoadBalancer": {
1550 | "Properties": {
1551 | "HealthCheck": {
1552 | "HealthyThreshold": "3",
1553 | "Interval": "90",
1554 | "Target": "HTTP:80/",
1555 | "Timeout": "60",
1556 | "UnhealthyThreshold": "5"
1557 | },
1558 | "Listeners": [
1559 | {
1560 | "InstancePort": "80",
1561 | "LoadBalancerPort": "80",
1562 | "Protocol": "HTTP"
1563 | }
1564 | ],
1565 | "SecurityGroups": [
1566 | {
1567 | "Ref": "PublicLoadBalancerSecurityGroup"
1568 | }
1569 | ],
1570 | "Subnets": [
1571 | {
1572 | "Ref": "PublicSubnet"
1573 | }
1574 | ]
1575 | },
1576 | "Type": "AWS::ElasticLoadBalancing::LoadBalancer"
1577 | },
1578 | "PublicLoadBalancerSecurityGroup": {
1579 | "Properties": {
1580 | "GroupDescription": "Public ELB Security Group with HTTP access on port 80 from the internet",
1581 | "SecurityGroupEgress": [
1582 | {
1583 | "CidrIp": "0.0.0.0/0",
1584 | "FromPort": "80",
1585 | "IpProtocol": "tcp",
1586 | "ToPort": "80"
1587 | }
1588 | ],
1589 | "SecurityGroupIngress": [
1590 | {
1591 | "CidrIp": "0.0.0.0/0",
1592 | "FromPort": "80",
1593 | "IpProtocol": "tcp",
1594 | "ToPort": "80"
1595 | },
1596 | {
1597 | "CidrIp": "0.0.0.0/0",
1598 | "FromPort": "443",
1599 | "IpProtocol": "tcp",
1600 | "ToPort": "443"
1601 | }
1602 | ],
1603 | "VpcId": {
1604 | "Ref": "VPC"
1605 | }
1606 | },
1607 | "Type": "AWS::EC2::SecurityGroup"
1608 | },
1609 | "PublicNetworkAcl": {
1610 | "Properties": {
1611 | "Tags": [
1612 | {
1613 | "Key": "Application",
1614 | "Value": {
1615 | "Ref": "AWS::StackName"
1616 | }
1617 | },
1618 | {
1619 | "Key": "Network",
1620 | "Value": "Public"
1621 | }
1622 | ],
1623 | "VpcId": {
1624 | "Ref": "VPC"
1625 | }
1626 | },
1627 | "Type": "AWS::EC2::NetworkAcl"
1628 | },
1629 | "PublicRoute": {
1630 | "Properties": {
1631 | "DestinationCidrBlock": "0.0.0.0/0",
1632 | "GatewayId": {
1633 | "Ref": "InternetGateway"
1634 | },
1635 | "RouteTableId": {
1636 | "Ref": "PublicRouteTable"
1637 | }
1638 | },
1639 | "Type": "AWS::EC2::Route"
1640 | },
1641 | "PublicRouteTable": {
1642 | "Properties": {
1643 | "Tags": [
1644 | {
1645 | "Key": "Application",
1646 | "Value": {
1647 | "Ref": "AWS::StackName"
1648 | }
1649 | },
1650 | {
1651 | "Key": "Network",
1652 | "Value": "Public"
1653 | }
1654 | ],
1655 | "VpcId": {
1656 | "Ref": "VPC"
1657 | }
1658 | },
1659 | "Type": "AWS::EC2::RouteTable"
1660 | },
1661 | "PublicSubnet": {
1662 | "Properties": {
1663 | "CidrBlock": {
1664 | "Fn::FindInMap": [
1665 | "SubnetConfig",
1666 | "Public",
1667 | "CIDR"
1668 | ]
1669 | },
1670 | "Tags": [
1671 | {
1672 | "Key": "Application",
1673 | "Value": {
1674 | "Ref": "AWS::StackName"
1675 | }
1676 | },
1677 | {
1678 | "Key": "Network",
1679 | "Value": "Public"
1680 | }
1681 | ],
1682 | "VpcId": {
1683 | "Ref": "VPC"
1684 | }
1685 | },
1686 | "Type": "AWS::EC2::Subnet"
1687 | },
1688 | "PublicSubnetNetworkAclAssociation": {
1689 | "Properties": {
1690 | "NetworkAclId": {
1691 | "Ref": "PublicNetworkAcl"
1692 | },
1693 | "SubnetId": {
1694 | "Ref": "PublicSubnet"
1695 | }
1696 | },
1697 | "Type": "AWS::EC2::SubnetNetworkAclAssociation"
1698 | },
1699 | "PublicSubnetRouteTableAssociation": {
1700 | "Properties": {
1701 | "RouteTableId": {
1702 | "Ref": "PublicRouteTable"
1703 | },
1704 | "SubnetId": {
1705 | "Ref": "PublicSubnet"
1706 | }
1707 | },
1708 | "Type": "AWS::EC2::SubnetRouteTableAssociation"
1709 | },
1710 | "RolePolicies": {
1711 | "Properties": {
1712 | "PolicyDocument": {
1713 | "Statement": [
1714 | {
1715 | "Action": [
1716 | "ec2:DescribeAvailabilityZones",
1717 | "ec2:DescribeInstanceStatus",
1718 | "ec2:DescribeInstances",
1719 | "ec2:DescribeRegions",
1720 | "ec2:DescribeTags"
1721 | ],
1722 | "Effect": "Allow",
1723 | "Resource": "*"
1724 | }
1725 | ]
1726 | },
1727 | "PolicyName": "root",
1728 | "Roles": [
1729 | {
1730 | "Ref": "RootRole"
1731 | }
1732 | ]
1733 | },
1734 | "Type": "AWS::IAM::Policy"
1735 | },
1736 | "RootInstanceProfile": {
1737 | "Properties": {
1738 | "Path": "/",
1739 | "Roles": [
1740 | {
1741 | "Ref": "RootRole"
1742 | }
1743 | ]
1744 | },
1745 | "Type": "AWS::IAM::InstanceProfile"
1746 | },
1747 | "RootRole": {
1748 | "Properties": {
1749 | "AssumeRolePolicyDocument": {
1750 | "Statement": [
1751 | {
1752 | "Action": [
1753 | "sts:AssumeRole"
1754 | ],
1755 | "Effect": "Allow",
1756 | "Principal": {
1757 | "Service": [
1758 | "ec2.amazonaws.com"
1759 | ]
1760 | }
1761 | }
1762 | ]
1763 | },
1764 | "Path": "/"
1765 | },
1766 | "Type": "AWS::IAM::Role"
1767 | },
1768 | "VPC": {
1769 | "Properties": {
1770 | "CidrBlock": {
1771 | "Fn::FindInMap": [
1772 | "SubnetConfig",
1773 | "VPC",
1774 | "CIDR"
1775 | ]
1776 | },
1777 | "Tags": [
1778 | {
1779 | "Key": "Application",
1780 | "Value": {
1781 | "Ref": "AWS::StackName"
1782 | }
1783 | },
1784 | {
1785 | "Key": "Network",
1786 | "Value": "Public"
1787 | }
1788 | ]
1789 | },
1790 | "Type": "AWS::EC2::VPC"
1791 | }
1792 | }
1793 | }
--------------------------------------------------------------------------------
/riak-vpc-cluster.json:
--------------------------------------------------------------------------------
1 | {
2 | "AWSTemplateFormatVersion": "2010-09-09",
3 | "Description": "Launches a Riak Cluster",
4 | "Mappings": {
5 | "AWSInstanceType2Arch": {
6 | "c1.medium": {
7 | "Arch": "64"
8 | },
9 | "c1.xlarge": {
10 | "Arch": "64"
11 | },
12 | "cc1.4xlarge": {
13 | "Arch": "64Cluster"
14 | },
15 | "cc2.8xlarge": {
16 | "Arch": "64Cluster"
17 | },
18 | "cg1.4xlarge": {
19 | "Arch": "64GPU"
20 | },
21 | "m1.large": {
22 | "Arch": "64"
23 | },
24 | "m1.medium": {
25 | "Arch": "64"
26 | },
27 | "m1.small": {
28 | "Arch": "64"
29 | },
30 | "m1.xlarge": {
31 | "Arch": "64"
32 | },
33 | "m2.2xlarge": {
34 | "Arch": "64"
35 | },
36 | "m2.4xlarge": {
37 | "Arch": "64"
38 | },
39 | "m2.xlarge": {
40 | "Arch": "64"
41 | },
42 | "t1.micro": {
43 | "Arch": "64"
44 | }
45 | },
46 | "AWSNATAMI": {
47 | "ap-northeast-1": {
48 | "AMI": "ami-14d86d15"
49 | },
50 | "ap-southeast-1": {
51 | "AMI": "ami-02eb9350"
52 | },
53 | "eu-west-1": {
54 | "AMI": "ami-0b5b6c7f"
55 | },
56 | "sa-east-1": {
57 | "AMI": "ami-0439e619"
58 | },
59 | "us-east-1": {
60 | "AMI": "ami-c6699baf"
61 | },
62 | "us-west-1": {
63 | "AMI": "ami-3bcc9e7e"
64 | },
65 | "us-west-2": {
66 | "AMI": "ami-52ff7262"
67 | }
68 | },
69 | "SubnetConfig": {
70 | "Private": {
71 | "CIDR": "10.0.1.0/24"
72 | },
73 | "Public": {
74 | "CIDR": "10.0.0.0/24"
75 | },
76 | "VPC": {
77 | "CIDR": "10.0.0.0/16"
78 | }
79 | },
80 | "ebs": {
81 | "ap-northeast-1": {
82 | "32": "ami-486cd349 ",
83 | "64": "ami-4e6cd34f",
84 | "64Cluster": "NOT_YET_SUPPORTED",
85 | "64GPU": "NOT_YET_SUPPORTED"
86 | },
87 | "ap-southeast-1": {
88 | "32": "ami-a2a7e7f0",
89 | "64": "ami-a6a7e7f4",
90 | "64Cluster": "NOT_YET_SUPPORTED",
91 | "64GPU": "NOT_YET_SUPPORTED"
92 | },
93 | "eu-west-1": {
94 | "32": "ami-937474e7",
95 | "64": "ami-c37474b7",
96 | "64Cluster": "ami-d97474ad",
97 | "64GPU": "ami-1b02026f"
98 | },
99 | "sa-east-1": {
100 | "32": "ami-e209d0ff",
101 | "64": "ami-1e08d103",
102 | "64Cluster": "NOT_YET_SUPPORTED",
103 | "64GPU": "NOT_YET_SUPPORTED"
104 | },
105 | "us-east-1": {
106 | "32": "ami-1a249873",
107 | "64": "ami-1624987f",
108 | "64Cluster": "ami-08249861",
109 | "64GPU": "ami-02f54a6b"
110 | },
111 | "us-west-1": {
112 | "32": "ami-19f9de5c",
113 | "64": "ami-1bf9de5e",
114 | "64Cluster": "NOT_YET_SUPPORTED",
115 | "64GPU": "NOT_YET_SUPPORTED"
116 | },
117 | "us-west-2": {
118 | "32": "ami-2231bf12",
119 | "64": "ami-2a31bf1a",
120 | "64Cluster": "ami-2431bf14",
121 | "64GPU": "NOT_YET_SUPPORTED"
122 | }
123 | },
124 | "ephemeral": {
125 | "ap-northeast-1": {
126 | "32": "ami-586cd359",
127 | "64": "ami-5a6cd35b"
128 | },
129 | "ap-southeast-1": {
130 | "32": "ami-aaa7e7f8",
131 | "64": "ami-a8a7e7fa"
132 | },
133 | "eu-west-1": {
134 | "32": "ami-cf7474bb",
135 | "64": "ami-b57474c1"
136 | },
137 | "sa-east-1": {
138 | "32": " ami-1a08d107",
139 | "64": " ami-1608d10b"
140 | },
141 | "us-east-1": {
142 | "32": "ami-10249879",
143 | "64": "ami-e8249881"
144 | },
145 | "us-west-1": {
146 | "32": "ami-27f9de62",
147 | "64": "ami-21f9de64"
148 | },
149 | "us-west-2": {
150 | "32": "ami-2c31bf1c",
151 | "64": "ami-2e31bf1e"
152 | }
153 | }
154 | },
155 | "Outputs": {
156 | "Bastion": {
157 | "Description": "IP Address of the Bastion host",
158 | "Value": {
159 | "Ref": "BastionIPAddress"
160 | }
161 | }
162 | },
163 | "Parameters": {
164 | "BastionInstanceType": {
165 | "AllowedValues": [
166 | "t1.micro",
167 | "m1.small",
168 | "m1.medium",
169 | "m1.large",
170 | "m1.xlarge",
171 | "m2.xlarge",
172 | "m2.2xlarge",
173 | "m2.4xlarge",
174 | "c1.medium",
175 | "c1.xlarge",
176 | "cc1.4xlarge",
177 | "cc2.8xlarge",
178 | "cg1.4xlarge"
179 | ],
180 | "ConstraintDescription": "must be a valid EC2 instance type.",
181 | "Default": "m1.small",
182 | "Description": "Bastion Host EC2 instance type",
183 | "Type": "String"
184 | },
185 | "DiskType": {
186 | "AllowedValues": [
187 | "ephemeral",
188 | "ebs"
189 | ],
190 | "Default": "ephemeral",
191 | "Description": "Type of Disk to use ( ephemeral/ebs )",
192 | "Type": "String"
193 | },
194 | "KeyName": {
195 | "AllowedPattern": "[-_ a-zA-Z0-9]*",
196 | "ConstraintDescription": "can contain only alphanumeric characters, spaces, dashes and underscores.",
197 | "Description": "Name of an existing EC2 KeyPair to enable SSH access to the instances",
198 | "MaxLength": "64",
199 | "MinLength": "1",
200 | "Type": "String"
201 | },
202 | "NATInstanceType": {
203 | "AllowedValues": [
204 | "t1.micro",
205 | "m1.small",
206 | "m1.medium",
207 | "m1.large",
208 | "m1.xlarge",
209 | "m2.xlarge",
210 | "m2.2xlarge",
211 | "m2.4xlarge",
212 | "c1.medium",
213 | "c1.xlarge",
214 | "cc1.4xlarge",
215 | "cc2.8xlarge",
216 | "cg1.4xlarge"
217 | ],
218 | "ConstraintDescription": "must be a valid EC2 instance type.",
219 | "Default": "m1.small",
220 | "Description": "NET Device EC2 instance type",
221 | "Type": "String"
222 | },
223 | "RiakClusterSize": {
224 | "Default": "3",
225 | "Description": "Number of EC2 instances to launch for Riak Cluster",
226 | "Type": "Number"
227 | },
228 | "RiakInstanceType": {
229 | "AllowedValues": [
230 | "t1.micro",
231 | "m1.small",
232 | "m1.medium",
233 | "m1.large",
234 | "m1.xlarge",
235 | "m2.xlarge",
236 | "m2.2xlarge",
237 | "m2.4xlarge",
238 | "c1.medium",
239 | "c1.xlarge",
240 | "cc1.4xlarge",
241 | "cc2.8xlarge",
242 | "cg1.4xlarge"
243 | ],
244 | "ConstraintDescription": "must be a valid EC2 instance type.",
245 | "Default": "m1.small",
246 | "Description": "Riak Server EC2 instance type",
247 | "Type": "String"
248 | },
249 | "RingSize": {
250 | "AllowedValues": [
251 | "64",
252 | "128",
253 | "256",
254 | "512"
255 | ],
256 | "Default": "64",
257 | "Description": "Size of the Riak Ring",
258 | "Type": "String"
259 | },
260 | "SSHFrom": {
261 | "AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})",
262 | "ConstraintDescription": "must be a valid CIDR range of the form x.x.x.x/x.",
263 | "Default": "0.0.0.0/0",
264 | "Description": "Lockdown SSH access to the bastion host (default can be accessed from anywhere)",
265 | "MaxLength": "18",
266 | "MinLength": "9",
267 | "Type": "String"
268 | }
269 | },
270 | "Resources": {
271 | "BackendFleet": {
272 | "Properties": {
273 | "AvailabilityZones": [
274 | {
275 | "Fn::GetAtt": [
276 | "PrivateSubnet",
277 | "AvailabilityZone"
278 | ]
279 | }
280 | ],
281 | "DesiredCapacity": {
282 | "Ref": "RiakClusterSize"
283 | },
284 | "LaunchConfigurationName": {
285 | "Ref": "BackendLaunchConfig"
286 | },
287 | "LoadBalancerNames": [
288 | {
289 | "Ref": "PrivateElasticLoadBalancer"
290 | }
291 | ],
292 | "MaxSize": "10",
293 | "MinSize": "1",
294 | "Tags": [
295 | {
296 | "Key": "Network",
297 | "PropagateAtLaunch": "true",
298 | "Value": "Private"
299 | }
300 | ],
301 | "VPCZoneIdentifier": [
302 | {
303 | "Ref": "PrivateSubnet"
304 | }
305 | ]
306 | },
307 | "Type": "AWS::AutoScaling::AutoScalingGroup"
308 | },
309 | "BackendLaunchConfig": {
310 | "DependsOn": [
311 | "BastionIPAddress",
312 | "NATIPAddress",
313 | "PrivateSubnetRouteTableAssociation",
314 | "PrivateSubnetNetworkAclAssociation"
315 | ],
316 | "Properties": {
317 | "IamInstanceProfile": {
318 | "Ref": "RootInstanceProfile"
319 | },
320 | "ImageId": {
321 | "Fn::FindInMap": [
322 | {
323 | "Ref": "DiskType"
324 | },
325 | {
326 | "Ref": "AWS::Region"
327 | },
328 | {
329 | "Fn::FindInMap": [
330 | "AWSInstanceType2Arch",
331 | {
332 | "Ref": "RiakInstanceType"
333 | },
334 | "Arch"
335 | ]
336 | }
337 | ]
338 | },
339 | "InstanceType": {
340 | "Ref": "RiakInstanceType"
341 | },
342 | "KeyName": {
343 | "Ref": "KeyName"
344 | },
345 | "SecurityGroups": [
346 | {
347 | "Ref": "BackendSecurityGroup"
348 | }
349 | ],
350 | "UserData": {
351 | "Fn::Base64": {
352 | "Fn::Join": [
353 | "\n",
354 | [
355 | "#!/bin/bash -v",
356 | "exec > >(tee /var/log/cfn-data.log|logger -t user-data -s 2>/dev/console) 2>&1",
357 | "",
358 | "sleep 10",
359 | "",
360 | "function retry {",
361 | " nTrys=0",
362 | " maxTrys=5",
363 | " status=256",
364 | " until [ $status == 0 ] ; do",
365 | " $1",
366 | " status=$?",
367 | " nTrys=$(($nTrys + 1))",
368 | " if [ $nTrys -gt $maxTrys ] ; then",
369 | " echo \"Number of re-trys exceeded. Exit code: $status\"",
370 | " exit $status",
371 | " fi",
372 | " if [ $status != 0 ] ; then",
373 | " echo \"Failed (exit code $status)... retry $nTrys\"",
374 | " sleep 10",
375 | " fi",
376 | " done",
377 | "}",
378 | "",
379 | "yum update -y aws-cfn-bootstrap",
380 | "",
381 | "#for all the stuff that complains about sudo and tty",
382 | "sed -i 's,Defaults requiretty,#Defaults requiretty,g' /etc/sudoers",
383 | "",
384 | "function error_exit",
385 | "{",
386 | {
387 | "Fn::Join": [
388 | "",
389 | [
390 | " /opt/aws/bin/cfn-signal -e 1 -r \"$1\" '",
391 | {
392 | "Ref": "BackendWaitHandle"
393 | },
394 | "'"
395 | ]
396 | ]
397 | },
398 | "}",
399 | "yum update -y aws-cfn-bootstrap",
400 | "#this runs the first stage of cfinit",
401 | {
402 | "Fn::Join": [
403 | "",
404 | [
405 | "#/opt/aws/bin/cfn-init -c ascending -v --region ",
406 | {
407 | "Ref": "AWS::Region"
408 | },
409 | " -s ",
410 | {
411 | "Ref": "AWS::StackName"
412 | },
413 | " -r ",
414 | "BackendLaunchConfig",
415 | " --access-key ",
416 | {
417 | "Ref": "HostKeys"
418 | },
419 | " --secret-key ",
420 | {
421 | "Fn::GetAtt": [
422 | "HostKeys",
423 | "SecretAccessKey"
424 | ]
425 | },
426 | " || error_exit 'Failed to initialize client using cfn-init'"
427 | ]
428 | ]
429 | },
430 | "",
431 | "",
432 | "",
433 | "",
434 | "yum install -y rubygems git puppet ruby-devel make gcc",
435 | "gem install hiera hiera-puppet",
436 | "mkdir -p /var/lib/hiera",
437 | "#enables search",
438 | "echo 'search_enabled: true' >> '/var/lib/hiera/common.yaml'",
439 | "",
440 | {
441 | "Fn::Join": [
442 | "",
443 | [
444 | "echo 'ring_creation_size: ",
445 | {
446 | "Ref": "RingSize"
447 | },
448 | "' >> '/var/lib/hiera/common.yaml'"
449 | ]
450 | ]
451 | },
452 | "",
453 | "mkdir -p /tmp/puppet/modules",
454 | "retry \"git clone http://github.com/basho/puppet-riak -b cfn /tmp/puppet/modules/riak\"",
455 | "retry \"git clone https://github.com/KrisBuytaert/puppet-hiera.git /tmp/puppet/modules/puppet-hiera\"",
456 | "retry \"git clone https://github.com/puppetlabs/puppetlabs-stdlib /tmp/puppet/modules/stdlib\"",
457 | "puppet apply --modulepath=/tmp/puppet/modules -e \"include riak\" --trace --debug --verbose",
458 | "",
459 | "echo \"riak soft nofile 65336\" >> /etc/security/limits.conf",
460 | "echo \"riak hard nofile 65336\" >> /etc/security/limits.conf",
461 | "echo \"root soft nofile 65336\" >> /etc/security/limits.conf",
462 | "echo \"root hard nofile 65336\" >> /etc/security/limits.conf",
463 | "",
464 | "echo \"vm.swappiness = 0\" >> /etc/sysctl.conf",
465 | "echo \"net.ipv4.tcp_max_syn_backlog = 40000\" >> /etc/sysctl.conf",
466 | "echo \"net.core.somaxconn=4000\" >> /etc/sysctl.conf",
467 | "echo \"net.ipv4.tcp_timestamps = 0\" >> /etc/sysctl.conf",
468 | "echo \"net.ipv4.tcp_sack = 1\" >> /etc/sysctl.conf",
469 | "echo \"net.ipv4.tcp_window_scaling = 1\" >> /etc/sysctl.conf",
470 | "echo \"net.ipv4.tcp_fin_timeout = 15\" >> /etc/sysctl.conf",
471 | "echo \"net.ipv4.tcp_keepalive_intvl = 30\" >> /etc/sysctl.conf",
472 | "echo \"net.ipv4.tcp_tw_reuse = 1\" >> /etc/sysctl.conf",
473 | "echo \"+P 256000\" >> /etc/riak/vm.args",
474 | "echo \"+swt very_low\" >> /etc/riak/vm.args",
475 | "echo \"+zdbbl 32768\" >> /etc/riak/vm.args",
476 | "sysctl -p",
477 | "",
478 | "service riak stop",
479 | "service riak start",
480 | "sleep 5",
481 | "retry \"git clone http://github.com/basho/cloudformation-riak /tmp/cloudformation-riak\"",
482 | "/tmp/cloudformation-riak/tools/join_riak_cluster.py",
483 | "search-cmd install riagi-users",
484 | "search-cmd install riagi-image-metadata",
485 | "/usr/sbin/riak ping",
486 | "result_code=$?",
487 | {
488 | "Fn::Join": [
489 | "",
490 | [
491 | "/opt/aws/bin/cfn-signal -e $result_code '",
492 | {
493 | "Ref": "BackendWaitHandle"
494 | },
495 | "'"
496 | ]
497 | ]
498 | },
499 | "#turn back on requiretty for sudo",
500 | "sed -i 's,#Defaults requiretty,Defaults requiretty,g' /etc/sudoers"
501 | ]
502 | ]
503 | }
504 | }
505 | },
506 | "Type": "AWS::AutoScaling::LaunchConfiguration"
507 | },
508 | "BackendSecurityGroup": {
509 | "Properties": {
510 | "GroupDescription": "Allow access from private load balancer and bastion as well as outbound HTTP and HTTPS traffic",
511 | "SecurityGroupEgress": [
512 | {
513 | "CidrIp": "0.0.0.0/0",
514 | "FromPort": "80",
515 | "IpProtocol": "tcp",
516 | "ToPort": "80"
517 | },
518 | {
519 | "CidrIp": "0.0.0.0/0",
520 | "FromPort": "443",
521 | "IpProtocol": "tcp",
522 | "ToPort": "443"
523 | }
524 | ],
525 | "SecurityGroupIngress": [
526 | {
527 | "FromPort": "8087",
528 | "IpProtocol": "tcp",
529 | "SourceSecurityGroupId": {
530 | "Ref": "PrivateLoadBalancerSecurityGroup"
531 | },
532 | "ToPort": "8087"
533 | },
534 | {
535 | "FromPort": "8098",
536 | "IpProtocol": "tcp",
537 | "SourceSecurityGroupId": {
538 | "Ref": "PrivateLoadBalancerSecurityGroup"
539 | },
540 | "ToPort": "8098"
541 | },
542 | {
543 | "FromPort": "22",
544 | "IpProtocol": "tcp",
545 | "SourceSecurityGroupId": {
546 | "Ref": "BastionSecurityGroup"
547 | },
548 | "ToPort": "22"
549 | }
550 | ],
551 | "VpcId": {
552 | "Ref": "VPC"
553 | }
554 | },
555 | "Type": "AWS::EC2::SecurityGroup"
556 | },
557 | "BackendWaitCondition": {
558 | "DependsOn": "BackendFleet",
559 | "Properties": {
560 | "Count": {
561 | "Ref": "RiakClusterSize"
562 | },
563 | "Handle": {
564 | "Ref": "BackendWaitHandle"
565 | },
566 | "Timeout": "600"
567 | },
568 | "Type": "AWS::CloudFormation::WaitCondition"
569 | },
570 | "BackendWaitHandle": {
571 | "Type": "AWS::CloudFormation::WaitConditionHandle"
572 | },
573 | "BastionHost": {
574 | "DependsOn": [
575 | "PublicSubnetRouteTableAssociation",
576 | "PublicSubnetNetworkAclAssociation"
577 | ],
578 | "Properties": {
579 | "ImageId": {
580 | "Fn::FindInMap": [
581 | "ephemeral",
582 | {
583 | "Ref": "AWS::Region"
584 | },
585 | {
586 | "Fn::FindInMap": [
587 | "AWSInstanceType2Arch",
588 | {
589 | "Ref": "BastionInstanceType"
590 | },
591 | "Arch"
592 | ]
593 | }
594 | ]
595 | },
596 | "InstanceType": {
597 | "Ref": "BastionInstanceType"
598 | },
599 | "KeyName": {
600 | "Ref": "KeyName"
601 | },
602 | "SecurityGroupIds": [
603 | {
604 | "Ref": "BastionSecurityGroup"
605 | }
606 | ],
607 | "SubnetId": {
608 | "Ref": "PublicSubnet"
609 | }
610 | },
611 | "Type": "AWS::EC2::Instance"
612 | },
613 | "BastionIPAddress": {
614 | "DependsOn": "GatewayToInternet",
615 | "Properties": {
616 | "Domain": "vpc",
617 | "InstanceId": {
618 | "Ref": "BastionHost"
619 | }
620 | },
621 | "Type": "AWS::EC2::EIP"
622 | },
623 | "BastionSecurityGroup": {
624 | "Properties": {
625 | "GroupDescription": "Enable access to the Bastion host",
626 | "SecurityGroupEgress": [
627 | {
628 | "CidrIp": {
629 | "Fn::FindInMap": [
630 | "SubnetConfig",
631 | "Private",
632 | "CIDR"
633 | ]
634 | },
635 | "FromPort": "22",
636 | "IpProtocol": "tcp",
637 | "ToPort": "22"
638 | }
639 | ],
640 | "SecurityGroupIngress": [
641 | {
642 | "CidrIp": {
643 | "Ref": "SSHFrom"
644 | },
645 | "FromPort": "22",
646 | "IpProtocol": "tcp",
647 | "ToPort": "22"
648 | }
649 | ],
650 | "VpcId": {
651 | "Ref": "VPC"
652 | }
653 | },
654 | "Type": "AWS::EC2::SecurityGroup"
655 | },
656 | "CFNInitUser": {
657 | "Properties": {
658 | "Path": "/",
659 | "Policies": [
660 | {
661 | "PolicyDocument": {
662 | "Statement": [
663 | {
664 | "Action": [
665 | "cloudformation:DescribeStackResource",
666 | "s3:GetObject"
667 | ],
668 | "Effect": "Allow",
669 | "Resource": "*"
670 | }
671 | ]
672 | },
673 | "PolicyName": "AccessForCFNInit"
674 | }
675 | ]
676 | },
677 | "Type": "AWS::IAM::User"
678 | },
679 | "ClusterCommunication1": {
680 | "Properties": {
681 | "FromPort": "-1",
682 | "GroupId": {
683 | "Ref": "BackendSecurityGroup"
684 | },
685 | "IpProtocol": "icmp",
686 | "SourceSecurityGroupId": {
687 | "Ref": "BackendSecurityGroup"
688 | },
689 | "ToPort": "-1"
690 | },
691 | "Type": "AWS::EC2::SecurityGroupIngress"
692 | },
693 | "ClusterCommunication2": {
694 | "Properties": {
695 | "FromPort": "1",
696 | "GroupId": {
697 | "Ref": "BackendSecurityGroup"
698 | },
699 | "IpProtocol": "tcp",
700 | "SourceSecurityGroupId": {
701 | "Ref": "BackendSecurityGroup"
702 | },
703 | "ToPort": "65356"
704 | },
705 | "Type": "AWS::EC2::SecurityGroupIngress"
706 | },
707 | "ClusterCommunication3": {
708 | "Properties": {
709 | "FromPort": "1",
710 | "GroupId": {
711 | "Ref": "BackendSecurityGroup"
712 | },
713 | "IpProtocol": "udp",
714 | "SourceSecurityGroupId": {
715 | "Ref": "BackendSecurityGroup"
716 | },
717 | "ToPort": "65356"
718 | },
719 | "Type": "AWS::EC2::SecurityGroupIngress"
720 | },
721 | "ClusterCommunication4": {
722 | "Properties": {
723 | "FromPort": "-1",
724 | "GroupId": {
725 | "Ref": "BackendSecurityGroup"
726 | },
727 | "IpProtocol": "icmp",
728 | "SourceSecurityGroupId": {
729 | "Ref": "BackendSecurityGroup"
730 | },
731 | "ToPort": "-1"
732 | },
733 | "Type": "AWS::EC2::SecurityGroupEgress"
734 | },
735 | "ClusterCommunication5": {
736 | "Properties": {
737 | "FromPort": "1",
738 | "GroupId": {
739 | "Ref": "BackendSecurityGroup"
740 | },
741 | "IpProtocol": "tcp",
742 | "SourceSecurityGroupId": {
743 | "Ref": "BackendSecurityGroup"
744 | },
745 | "ToPort": "65356"
746 | },
747 | "Type": "AWS::EC2::SecurityGroupEgress"
748 | },
749 | "ClusterCommunication6": {
750 | "Properties": {
751 | "FromPort": "1",
752 | "GroupId": {
753 | "Ref": "BackendSecurityGroup"
754 | },
755 | "IpProtocol": "udp",
756 | "SourceSecurityGroupId": {
757 | "Ref": "BackendSecurityGroup"
758 | },
759 | "ToPort": "65356"
760 | },
761 | "Type": "AWS::EC2::SecurityGroupEgress"
762 | },
763 | "GatewayToInternet": {
764 | "Properties": {
765 | "InternetGatewayId": {
766 | "Ref": "InternetGateway"
767 | },
768 | "VpcId": {
769 | "Ref": "VPC"
770 | }
771 | },
772 | "Type": "AWS::EC2::VPCGatewayAttachment"
773 | },
774 | "HostKeys": {
775 | "Properties": {
776 | "UserName": {
777 | "Ref": "CFNInitUser"
778 | }
779 | },
780 | "Type": "AWS::IAM::AccessKey"
781 | },
782 | "InboundEmphemeralPublicNetworkAclEntry": {
783 | "Properties": {
784 | "CidrBlock": "0.0.0.0/0",
785 | "Egress": "false",
786 | "NetworkAclId": {
787 | "Ref": "PublicNetworkAcl"
788 | },
789 | "PortRange": {
790 | "From": "1024",
791 | "To": "65535"
792 | },
793 | "Protocol": "6",
794 | "RuleAction": "allow",
795 | "RuleNumber": "103"
796 | },
797 | "Type": "AWS::EC2::NetworkAclEntry"
798 | },
799 | "InboundHTTPPublicNetworkAclEntry": {
800 | "Properties": {
801 | "CidrBlock": "0.0.0.0/0",
802 | "Egress": "false",
803 | "NetworkAclId": {
804 | "Ref": "PublicNetworkAcl"
805 | },
806 | "PortRange": {
807 | "From": "80",
808 | "To": "80"
809 | },
810 | "Protocol": "6",
811 | "RuleAction": "allow",
812 | "RuleNumber": "100"
813 | },
814 | "Type": "AWS::EC2::NetworkAclEntry"
815 | },
816 | "InboundHTTPSPublicNetworkAclEntry": {
817 | "Properties": {
818 | "CidrBlock": "0.0.0.0/0",
819 | "Egress": "false",
820 | "NetworkAclId": {
821 | "Ref": "PublicNetworkAcl"
822 | },
823 | "PortRange": {
824 | "From": "443",
825 | "To": "443"
826 | },
827 | "Protocol": "6",
828 | "RuleAction": "allow",
829 | "RuleNumber": "101"
830 | },
831 | "Type": "AWS::EC2::NetworkAclEntry"
832 | },
833 | "InboundPrivateNetworkAclEntry": {
834 | "Properties": {
835 | "CidrBlock": "0.0.0.0/0",
836 | "Egress": "false",
837 | "NetworkAclId": {
838 | "Ref": "PrivateNetworkAcl"
839 | },
840 | "PortRange": {
841 | "From": "0",
842 | "To": "65535"
843 | },
844 | "Protocol": "6",
845 | "RuleAction": "allow",
846 | "RuleNumber": "100"
847 | },
848 | "Type": "AWS::EC2::NetworkAclEntry"
849 | },
850 | "InboundSSHPublicNetworkAclEntry": {
851 | "Properties": {
852 | "CidrBlock": {
853 | "Ref": "SSHFrom"
854 | },
855 | "Egress": "false",
856 | "NetworkAclId": {
857 | "Ref": "PublicNetworkAcl"
858 | },
859 | "PortRange": {
860 | "From": "22",
861 | "To": "22"
862 | },
863 | "Protocol": "6",
864 | "RuleAction": "allow",
865 | "RuleNumber": "102"
866 | },
867 | "Type": "AWS::EC2::NetworkAclEntry"
868 | },
869 | "InternetGateway": {
870 | "Properties": {
871 | "Tags": [
872 | {
873 | "Key": "Application",
874 | "Value": {
875 | "Ref": "AWS::StackName"
876 | }
877 | },
878 | {
879 | "Key": "Network",
880 | "Value": "Public"
881 | }
882 | ]
883 | },
884 | "Type": "AWS::EC2::InternetGateway"
885 | },
886 | "NATDevice": {
887 | "Properties": {
888 | "ImageId": {
889 | "Fn::FindInMap": [
890 | "AWSNATAMI",
891 | {
892 | "Ref": "AWS::Region"
893 | },
894 | "AMI"
895 | ]
896 | },
897 | "InstanceType": {
898 | "Ref": "NATInstanceType"
899 | },
900 | "KeyName": {
901 | "Ref": "KeyName"
902 | },
903 | "SecurityGroupIds": [
904 | {
905 | "Ref": "NATSecurityGroup"
906 | }
907 | ],
908 | "SourceDestCheck": "false",
909 | "SubnetId": {
910 | "Ref": "PublicSubnet"
911 | }
912 | },
913 | "Type": "AWS::EC2::Instance"
914 | },
915 | "NATIPAddress": {
916 | "DependsOn": "GatewayToInternet",
917 | "Properties": {
918 | "Domain": "vpc",
919 | "InstanceId": {
920 | "Ref": "NATDevice"
921 | }
922 | },
923 | "Type": "AWS::EC2::EIP"
924 | },
925 | "NATSecurityGroup": {
926 | "Properties": {
927 | "GroupDescription": "Enable internal access to the NAT device",
928 | "SecurityGroupEgress": [
929 | {
930 | "CidrIp": "0.0.0.0/0",
931 | "FromPort": "80",
932 | "IpProtocol": "tcp",
933 | "ToPort": "80"
934 | },
935 | {
936 | "CidrIp": "0.0.0.0/0",
937 | "FromPort": "443",
938 | "IpProtocol": "tcp",
939 | "ToPort": "443"
940 | }
941 | ],
942 | "SecurityGroupIngress": [
943 | {
944 | "CidrIp": "0.0.0.0/0",
945 | "FromPort": "80",
946 | "IpProtocol": "tcp",
947 | "ToPort": "80"
948 | },
949 | {
950 | "CidrIp": "0.0.0.0/0",
951 | "FromPort": "443",
952 | "IpProtocol": "tcp",
953 | "ToPort": "443"
954 | },
955 | {
956 | "CidrIp": {
957 | "Ref": "SSHFrom"
958 | },
959 | "FromPort": "22",
960 | "IpProtocol": "tcp",
961 | "ToPort": "22"
962 | }
963 | ],
964 | "VpcId": {
965 | "Ref": "VPC"
966 | }
967 | },
968 | "Type": "AWS::EC2::SecurityGroup"
969 | },
970 | "OutBoundPrivateNetworkAclEntry": {
971 | "Properties": {
972 | "CidrBlock": "0.0.0.0/0",
973 | "Egress": "true",
974 | "NetworkAclId": {
975 | "Ref": "PrivateNetworkAcl"
976 | },
977 | "PortRange": {
978 | "From": "0",
979 | "To": "65535"
980 | },
981 | "Protocol": "6",
982 | "RuleAction": "allow",
983 | "RuleNumber": "100"
984 | },
985 | "Type": "AWS::EC2::NetworkAclEntry"
986 | },
987 | "OutboundPublicNetworkAclEntry": {
988 | "Properties": {
989 | "CidrBlock": "0.0.0.0/0",
990 | "Egress": "true",
991 | "NetworkAclId": {
992 | "Ref": "PublicNetworkAcl"
993 | },
994 | "PortRange": {
995 | "From": "0",
996 | "To": "65535"
997 | },
998 | "Protocol": "6",
999 | "RuleAction": "allow",
1000 | "RuleNumber": "100"
1001 | },
1002 | "Type": "AWS::EC2::NetworkAclEntry"
1003 | },
1004 | "PrivateElasticLoadBalancer": {
1005 | "Properties": {
1006 | "HealthCheck": {
1007 | "HealthyThreshold": "3",
1008 | "Interval": "90",
1009 | "Target": "HTTP:8098/",
1010 | "Timeout": "60",
1011 | "UnhealthyThreshold": "5"
1012 | },
1013 | "Listeners": [
1014 | {
1015 | "InstancePort": "8098",
1016 | "LoadBalancerPort": "8098",
1017 | "Protocol": "HTTP"
1018 | },
1019 | {
1020 | "InstancePort": "8087",
1021 | "LoadBalancerPort": "8087",
1022 | "Protocol": "TCP"
1023 | }
1024 | ],
1025 | "Scheme": "internal",
1026 | "SecurityGroups": [
1027 | {
1028 | "Ref": "PrivateLoadBalancerSecurityGroup"
1029 | }
1030 | ],
1031 | "Subnets": [
1032 | {
1033 | "Ref": "PrivateSubnet"
1034 | }
1035 | ]
1036 | },
1037 | "Type": "AWS::ElasticLoadBalancing::LoadBalancer"
1038 | },
1039 | "PrivateLoadBalancerSecurityGroup": {
1040 | "Properties": {
1041 | "GroupDescription": "Private ELB Security Group",
1042 | "SecurityGroupIngress": [
1043 | {
1044 | "FromPort": "8087",
1045 | "IpProtocol": "tcp",
1046 | "SourceSecurityGroupId": {
1047 | "Ref": "BastionSecurityGroup"
1048 | },
1049 | "ToPort": "8087"
1050 | },
1051 | {
1052 | "FromPort": "8098",
1053 | "IpProtocol": "tcp",
1054 | "SourceSecurityGroupId": {
1055 | "Ref": "BastionSecurityGroup"
1056 | },
1057 | "ToPort": "8098"
1058 | }
1059 | ],
1060 | "VpcId": {
1061 | "Ref": "VPC"
1062 | }
1063 | },
1064 | "Type": "AWS::EC2::SecurityGroup"
1065 | },
1066 | "PrivateNetworkAcl": {
1067 | "Properties": {
1068 | "Tags": [
1069 | {
1070 | "Key": "Application",
1071 | "Value": {
1072 | "Ref": "AWS::StackName"
1073 | }
1074 | },
1075 | {
1076 | "Key": "Network",
1077 | "Value": "Private"
1078 | }
1079 | ],
1080 | "VpcId": {
1081 | "Ref": "VPC"
1082 | }
1083 | },
1084 | "Type": "AWS::EC2::NetworkAcl"
1085 | },
1086 | "PrivateRoute": {
1087 | "Properties": {
1088 | "DestinationCidrBlock": "0.0.0.0/0",
1089 | "InstanceId": {
1090 | "Ref": "NATDevice"
1091 | },
1092 | "RouteTableId": {
1093 | "Ref": "PrivateRouteTable"
1094 | }
1095 | },
1096 | "Type": "AWS::EC2::Route"
1097 | },
1098 | "PrivateRouteTable": {
1099 | "Properties": {
1100 | "Tags": [
1101 | {
1102 | "Key": "Application",
1103 | "Value": {
1104 | "Ref": "AWS::StackName"
1105 | }
1106 | },
1107 | {
1108 | "Key": "Network",
1109 | "Value": "Private"
1110 | }
1111 | ],
1112 | "VpcId": {
1113 | "Ref": "VPC"
1114 | }
1115 | },
1116 | "Type": "AWS::EC2::RouteTable"
1117 | },
1118 | "PrivateSubnet": {
1119 | "Properties": {
1120 | "CidrBlock": {
1121 | "Fn::FindInMap": [
1122 | "SubnetConfig",
1123 | "Private",
1124 | "CIDR"
1125 | ]
1126 | },
1127 | "Tags": [
1128 | {
1129 | "Key": "Application",
1130 | "Value": {
1131 | "Ref": "AWS::StackName"
1132 | }
1133 | },
1134 | {
1135 | "Key": "Network",
1136 | "Value": "Private"
1137 | }
1138 | ],
1139 | "VpcId": {
1140 | "Ref": "VPC"
1141 | }
1142 | },
1143 | "Type": "AWS::EC2::Subnet"
1144 | },
1145 | "PrivateSubnetNetworkAclAssociation": {
1146 | "Properties": {
1147 | "NetworkAclId": {
1148 | "Ref": "PrivateNetworkAcl"
1149 | },
1150 | "SubnetId": {
1151 | "Ref": "PrivateSubnet"
1152 | }
1153 | },
1154 | "Type": "AWS::EC2::SubnetNetworkAclAssociation"
1155 | },
1156 | "PrivateSubnetRouteTableAssociation": {
1157 | "Properties": {
1158 | "RouteTableId": {
1159 | "Ref": "PrivateRouteTable"
1160 | },
1161 | "SubnetId": {
1162 | "Ref": "PrivateSubnet"
1163 | }
1164 | },
1165 | "Type": "AWS::EC2::SubnetRouteTableAssociation"
1166 | },
1167 | "PublicNetworkAcl": {
1168 | "Properties": {
1169 | "Tags": [
1170 | {
1171 | "Key": "Application",
1172 | "Value": {
1173 | "Ref": "AWS::StackName"
1174 | }
1175 | },
1176 | {
1177 | "Key": "Network",
1178 | "Value": "Public"
1179 | }
1180 | ],
1181 | "VpcId": {
1182 | "Ref": "VPC"
1183 | }
1184 | },
1185 | "Type": "AWS::EC2::NetworkAcl"
1186 | },
1187 | "PublicRoute": {
1188 | "Properties": {
1189 | "DestinationCidrBlock": "0.0.0.0/0",
1190 | "GatewayId": {
1191 | "Ref": "InternetGateway"
1192 | },
1193 | "RouteTableId": {
1194 | "Ref": "PublicRouteTable"
1195 | }
1196 | },
1197 | "Type": "AWS::EC2::Route"
1198 | },
1199 | "PublicRouteTable": {
1200 | "Properties": {
1201 | "Tags": [
1202 | {
1203 | "Key": "Application",
1204 | "Value": {
1205 | "Ref": "AWS::StackName"
1206 | }
1207 | },
1208 | {
1209 | "Key": "Network",
1210 | "Value": "Public"
1211 | }
1212 | ],
1213 | "VpcId": {
1214 | "Ref": "VPC"
1215 | }
1216 | },
1217 | "Type": "AWS::EC2::RouteTable"
1218 | },
1219 | "PublicSubnet": {
1220 | "Properties": {
1221 | "CidrBlock": {
1222 | "Fn::FindInMap": [
1223 | "SubnetConfig",
1224 | "Public",
1225 | "CIDR"
1226 | ]
1227 | },
1228 | "Tags": [
1229 | {
1230 | "Key": "Application",
1231 | "Value": {
1232 | "Ref": "AWS::StackName"
1233 | }
1234 | },
1235 | {
1236 | "Key": "Network",
1237 | "Value": "Public"
1238 | }
1239 | ],
1240 | "VpcId": {
1241 | "Ref": "VPC"
1242 | }
1243 | },
1244 | "Type": "AWS::EC2::Subnet"
1245 | },
1246 | "PublicSubnetNetworkAclAssociation": {
1247 | "Properties": {
1248 | "NetworkAclId": {
1249 | "Ref": "PublicNetworkAcl"
1250 | },
1251 | "SubnetId": {
1252 | "Ref": "PublicSubnet"
1253 | }
1254 | },
1255 | "Type": "AWS::EC2::SubnetNetworkAclAssociation"
1256 | },
1257 | "PublicSubnetRouteTableAssociation": {
1258 | "Properties": {
1259 | "RouteTableId": {
1260 | "Ref": "PublicRouteTable"
1261 | },
1262 | "SubnetId": {
1263 | "Ref": "PublicSubnet"
1264 | }
1265 | },
1266 | "Type": "AWS::EC2::SubnetRouteTableAssociation"
1267 | },
1268 | "RolePolicies": {
1269 | "Properties": {
1270 | "PolicyDocument": {
1271 | "Statement": [
1272 | {
1273 | "Action": [
1274 | "ec2:DescribeAvailabilityZones",
1275 | "ec2:DescribeInstanceStatus",
1276 | "ec2:DescribeInstances",
1277 | "ec2:DescribeRegions",
1278 | "ec2:DescribeTags"
1279 | ],
1280 | "Effect": "Allow",
1281 | "Resource": "*"
1282 | }
1283 | ]
1284 | },
1285 | "PolicyName": "root",
1286 | "Roles": [
1287 | {
1288 | "Ref": "RootRole"
1289 | }
1290 | ]
1291 | },
1292 | "Type": "AWS::IAM::Policy"
1293 | },
1294 | "RootInstanceProfile": {
1295 | "Properties": {
1296 | "Path": "/",
1297 | "Roles": [
1298 | {
1299 | "Ref": "RootRole"
1300 | }
1301 | ]
1302 | },
1303 | "Type": "AWS::IAM::InstanceProfile"
1304 | },
1305 | "RootRole": {
1306 | "Properties": {
1307 | "AssumeRolePolicyDocument": {
1308 | "Statement": [
1309 | {
1310 | "Action": [
1311 | "sts:AssumeRole"
1312 | ],
1313 | "Effect": "Allow",
1314 | "Principal": {
1315 | "Service": [
1316 | "ec2.amazonaws.com"
1317 | ]
1318 | }
1319 | }
1320 | ]
1321 | },
1322 | "Path": "/"
1323 | },
1324 | "Type": "AWS::IAM::Role"
1325 | },
1326 | "VPC": {
1327 | "Properties": {
1328 | "CidrBlock": {
1329 | "Fn::FindInMap": [
1330 | "SubnetConfig",
1331 | "VPC",
1332 | "CIDR"
1333 | ]
1334 | },
1335 | "Tags": [
1336 | {
1337 | "Key": "Application",
1338 | "Value": {
1339 | "Ref": "AWS::StackName"
1340 | }
1341 | },
1342 | {
1343 | "Key": "Network",
1344 | "Value": "Public"
1345 | }
1346 | ]
1347 | },
1348 | "Type": "AWS::EC2::VPC"
1349 | }
1350 | }
1351 | }
--------------------------------------------------------------------------------
/tools/join_riak_cluster.py:
--------------------------------------------------------------------------------
1 | #! /usr/bin/env python
2 |
3 | import sys
4 | from boto import ec2
5 | import boto.utils
6 | from subprocess import Popen, PIPE, STDOUT
7 | import time
8 |
9 | def runcmd(cmd):
10 | print "Running: %s" % cmd
11 | p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
12 | output = p.stdout.read()
13 | output.strip()
14 | return output
15 |
16 | def get_group_info(ec2conn, instance_id) :
17 | reservations = ec2conn.get_all_instances(instance_id)
18 | instance = [i for r in reservations for i in r.instances][0]
19 | if instance.tags.has_key('aws:autoscaling:groupName'):
20 | private_ip, first_node, nodes_total, stack_id, node_number, instance = get_autoscale_info(ec2conn, instance)
21 | elif instance.tags.has_key('stackId'):
22 | private_ip, first_node, nodes_total, stack_id, node_number, instance = get_reservation_info(ec2conn, instance)
23 | else:
24 | return None
25 | return private_ip, first_node, nodes_total, stack_id, node_number, instance
26 |
27 | def get_autoscale_info(ec2conn, instance) :
28 | autoscale_group = instance.tags['aws:autoscaling:groupName']
29 | private_ip = instance.private_ip_address
30 | filters = {'tag:aws:autoscaling:groupName': '%s*' % autoscale_group, 'instance-state-name': 'running'}
31 | reservations = ec2conn.get_all_instances(filters=filters)
32 | instances = [i for r in reservations for i in r.instances]
33 | sorted_instances = sorted(instances, key=lambda i: (i.launch_time, i.id))
34 | first_node = sorted_instances[0]
35 | node_number = [si.id for si in sorted_instances].index(instance.id) + 1
36 | return private_ip, first_node, len(sorted_instances), autoscale_group, node_number, instance
37 |
38 |
39 | def get_reservation_info(ec2conn, instance) :
40 | found = False
41 | while found != True:
42 | try:
43 | stack_id = instance.tags['stackId']
44 | nodes_total = int(instance.tags['nodesTotal'])
45 | node_number = int(instance.tags['nodeNumber'])
46 | break
47 | except:
48 | time.sleep(10)
49 | print'Waiting for tags'
50 | private_ip = instance.private_ip_address
51 |
52 | filters = {'tag:stackId': stack_id, 'tag:nodeNumber': '1' }
53 | reservations = ec2conn.get_all_instances(filters=filters)
54 | instances = [i for r in reservations for i in r.instances]
55 | first_node = instances[0]
56 | return private_ip, first_node, int(nodes_total), stack_id, node_number, instance
57 |
58 | def plan_commit(total_nodes):
59 | ready = False
60 | committed = False
61 | planned = False
62 | #wait until the ring is in a ready phase
63 | while ready == False:
64 | output=runcmd('riak-admin ring-status|grep "Ring Ready"')
65 | if output.find('Ring Ready: true') != -1:
66 | ready = True
67 | break
68 | time.sleep(5)
69 | #keep planning until total nodes are equal to staged nodes
70 | while planned == False:
71 | output=runcmd("riak-admin cluster plan")
72 | print output
73 | staged_nodes = output.count("valid")
74 | print "Total Nodes: %s Staged Nodes %s" % (str(total_nodes), str(staged_nodes))
75 | if staged_nodes == total_nodes:
76 | break
77 | time.sleep(5)
78 | #commit changes to the cluster
79 | while committed == False:
80 | output=runcmd("riak-admin cluster commit")
81 | print output
82 | if output.find('Cluster changes committed') != -1:
83 | committed = True
84 | return
85 | time.sleep(5)
86 |
87 | instance_data = boto.utils.get_instance_metadata()
88 | instance_id = instance_data["instance-id"]
89 |
90 | # connect to region of the current instance rather than default of us-east-1
91 | zone = instance_data['placement']['availability-zone']
92 | region_name = zone[:-1]
93 |
94 | print "Connecting to region %s" % region_name
95 | ec2conn = ec2.connect_to_region(region_name)
96 | private_ip, first_node, nodes_total, stack_id, node_number, instance = get_group_info(ec2conn, instance_id)
97 |
98 | print "Instance belongs to %s. Finding first node" % stack_id
99 | print "Node number for this machine is %s." % node_number
100 | print "Node count for this cluster is %s." % nodes_total
101 | print "First node in the cluster is %s" % first_node.private_ip_address
102 | print "Private IP for this machine is %s." % private_ip
103 |
104 |
105 | output = runcmd ('riak-admin member-status 2> /dev/null | grep ^valid|wc -l')
106 | node_count = int(output)
107 | print "Current node count is %s" % str(node_count)
108 | joined = False
109 |
110 | if node_count > 1 :
111 | print "Node already in cluster."
112 | joined = True
113 | sys.exit(0)
114 |
115 | if private_ip == first_node.private_ip_address:
116 | print 'This is the first node in the cluster. Nodes will join it.'
117 | sys.exit(0)
118 |
119 | #looping through the join until I get a successful request
120 | while joined == False:
121 | print "Joining node to node: %s." % first_node.private_ip_address
122 | cmd = ["riak-admin", "cluster", "join", "riak@%s" % first_node.private_ip_address]
123 | output = runcmd(" ".join(cmd))
124 | print output
125 | if output.find('Success: staged join request') != -1:
126 | joined = True
127 | break
128 | time.sleep(3)
129 |
130 | if node_number == nodes_total:
131 | print 'This is the last node to join the cluster. Stage and plan.'
132 | plan_commit(nodes_total)
133 |
--------------------------------------------------------------------------------