├── .gitignore ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── TODO ├── ami ├── bencher │ ├── packer.json │ ├── resources │ │ └── ndbench │ │ │ ├── Log4jInit.java │ │ │ ├── log4j.properties │ │ │ └── web.xml │ └── scripts │ │ └── bencher-provision.sh ├── cassandra3 │ ├── packer.json │ ├── resources │ │ ├── etc │ │ │ └── cassandra │ │ │ │ └── default.conf │ │ │ │ ├── cassandra.yaml │ │ │ │ ├── jvm.options │ │ │ │ └── logback.xml │ │ ├── initd-disable-transparent-hugepages │ │ ├── limits.conf │ │ └── sysctl.conf │ └── scripts │ │ └── cassandra-provision.sh └── rocksandra │ ├── packer.json │ ├── resources │ ├── etc │ │ └── cassandra │ │ │ └── default.conf │ │ │ ├── cassandra.yaml │ │ │ ├── jvm.options │ │ │ └── logback.xml │ ├── initd-disable-transparent-hugepages │ ├── limits.conf │ └── sysctl.conf │ └── scripts │ └── cassandra-provision.sh ├── bin ├── ssh └── ssh_all └── cloudformation.yaml /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | /ami/cassandra3/resources/cassandra.rpm 3 | /ami/rocksandra/resources/rocksandra.rpm 4 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Code of Conduct 2 | 3 | Facebook has adopted a Code of Conduct that we expect project participants to adhere to. Please [read the full text](https://code.facebook.com/codeofconduct) so that you can understand what actions will and will not be tolerated. 4 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to Cassandra AWS Benchmark 2 | 3 | ## Code of Conduct 4 | The code of conduct is described in [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md) 5 | 6 | ## Contributor License Agreement ("CLA") 7 | 8 | In order to accept your pull request, we need you to submit a CLA. You 9 | only need to do this once, so if you've done this for another Facebook 10 | open source project, you're good to go. If you are submitting a pull 11 | request for the first time, just let us know that you have completed 12 | the CLA and we can cross-check with your GitHub username. 13 | 14 | Complete your CLA here: 15 | 16 | If you prefer to sign a paper copy, we can send you a PDF. Send us an 17 | e-mail or create a new github issue to request the CLA in PDF format. 18 | 19 | ## License 20 | By contributing to Cassandra AWS Benchmark, you agree that your contributions will be licensed under the LICENSE file in the root directory of this source tree. -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # cassandra-aws-benchmark 2 | Scripts and templates for cassandra benchmark environment provision 3 | 4 | ## Requirements 5 | * aws cli: `pip install awscli --upgrade --user` and `aws configure` 6 | 7 | ## Create benchmark environment 8 | 9 | create cloudformation stack: 10 | 11 | ```shell 12 | aws cloudformation create-stack --stack-name MY_STACK_NAME \ 13 | --disable-rollback \ 14 | --template-body file://cloudformation.yaml \ 15 | --parameters \ 16 | ParameterKey=CassandraAMI,ParameterValue=MY_CASSANDRA_AMI_ID \ 17 | ParameterKey=KeyName,ParameterValue=MY_SSH_KEY_NAME \ 18 | ParameterKey=VPCSubnetId,ParameterValue=MY_VPC_SUBNET_ID \ 19 | ParameterKey=InstanceProfile,ParameterValue=NAME_OF_INSTANCE_PROFILE_WITH_EC2_AND_AUOSCALEGROUP_READONLY_ACCESS \ 20 | --capabilities CAPABILITY_IAM 21 | ``` 22 | 23 | update cloudformation stack: 24 | 25 | ```shell 26 | aws cloudformation udpate-stack --stack-name MY_STACK_NAME \ 27 | --template-body file://cloudformation.yaml \ 28 | --parameters \ 29 | ParameterKey=CassandraAMI,ParameterValue=MY_CASSANDRA_AMI_ID \ 30 | ParameterKey=KeyName,ParameterValue=MY_SSH_KEY_NAME \ 31 | ParameterKey=VPCSubnetId,ParameterValue=MY_VPC_SUBNET_ID \ 32 | ParameterKey=InstanceProfile,ParameterValue=NAME_OF_INSTANCE_PROFILE_WITH_EC2_AND_AUOSCALEGROUP_READONLY_ACCESS \ 33 | --capabilities CAPABILITY_IAM 34 | ``` 35 | 36 | ## Lastest Prebuilt Cassandra AMI 37 | * cassandra 3.0.15: ami-09cc7371 (us-west-2) 38 | * rocksandra: ami-b770cdcf (us-west-2) 39 | 40 | 41 | ## Build Cassandra AMI Myself 42 | prerequists: packer: `brew install packer` 43 | 44 | cassandra3x 45 | ``` 46 | $> cd ami/cassandra3x 47 | $> wget -O resources/cassandra.rpm https://www.apache.org/dist/cassandra/redhat/30x/cassandra-3.0.15-1.noarch.rpm 48 | $> packer build -var "image_version=$(date +%s)" packer.json 49 | ``` 50 | 51 | ndbench 52 | ``` 53 | $> cd ami/bencher 54 | $> packer build -var "image_version=$(date +%s)" packer.json 55 | ``` 56 | 57 | ## License 58 | Cassandra AWS Benchmark is Apache 2.0 licensed, as found in the LICENSE file. 59 | 60 | -------------------------------------------------------------------------------- /TODO: -------------------------------------------------------------------------------- 1 | v build rocksandra ami 2 | v multi node ndbench cluster 3 | v ebs volumn support 4 | v better default ndbnech config 5 | v benchmark with cassandra stress-tool 6 | v switch to xfs 7 | v remove or sample backfill log otherwise it fills up disk quickly 8 | o use /etc/cassandra/conf/default.conf/jvm.option replace /etc/default/cassandra 9 | * lockdown instanceprofile permissions 10 | * setup security group 11 | * friendly cli (maybe rake) 12 | * export cassandra metrics to cloudwatch 13 | * install jmxterm or jmxsh 14 | 15 | -------------------------------------------------------------------------------- /ami/bencher/packer.json: -------------------------------------------------------------------------------- 1 | { 2 | "variables": { 3 | "aws_access_key": "", 4 | "aws_secret_key": "", 5 | "aws_region": "us-west-2", 6 | "aws_ami_image": "ami-a142e9d9", 7 | "aws_instance_type": "m4.large", 8 | "image_version" : "0.0.0" 9 | }, 10 | "builders": [ 11 | { 12 | "type": "amazon-ebs", 13 | "access_key": "{{user `aws_access_key`}}", 14 | "secret_key": "{{user `aws_secret_key`}}", 15 | "region": "{{user `aws_region`}}", 16 | "source_ami": "{{user `aws_ami_image`}}", 17 | "instance_type": "{{user `aws_instance_type`}}", 18 | "ssh_username": "ec2-user", 19 | "ami_name": "bencher-{{user `image_version`}}", 20 | "ami_groups": "all", 21 | "tags": { 22 | "Name": "bencher-{{user `image_version`}}", 23 | "Description": "benchmark tools" 24 | } 25 | } 26 | ], 27 | "provisioners": [ 28 | { 29 | "type": "file", 30 | "source": "resources", 31 | "destination": "/home/ec2-user/" 32 | }, 33 | { 34 | "type": "shell", 35 | "scripts": [ 36 | "scripts/bencher-provision.sh" 37 | ] 38 | } 39 | ] 40 | } 41 | -------------------------------------------------------------------------------- /ami/bencher/resources/ndbench/Log4jInit.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2018-present, Facebook, Inc. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"); 5 | * you may not use this file except in compliance with the License. 6 | * You may obtain a copy of the License at 7 | * 8 | * http://www.apache.org/licenses/LICENSE-2.0 9 | * 10 | * Unless required by applicable law or agreed to in writing, software 11 | * distributed under the License is distributed on an "AS IS" BASIS, 12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | * See the License for the specific language governing permissions and 14 | * limitations under the License. 15 | */ 16 | package com.netflix.ndbench.defaultimpl; 17 | 18 | import org.apache.log4j.PropertyConfigurator; 19 | import javax.servlet.http.HttpServlet; 20 | import javax.servlet.http.HttpServletRequest; 21 | import javax.servlet.http.HttpServletResponse; 22 | import java.io.PrintWriter; 23 | import java.io.IOException; 24 | 25 | public class Log4jInit extends HttpServlet { 26 | 27 | public 28 | void init() { 29 | String prefix = getServletContext().getRealPath("/"); 30 | String file = getInitParameter("log4j-init-file"); 31 | // if the log4j-init-file is not set, then no point in trying 32 | if(file != null) { 33 | PropertyConfigurator.configure(prefix+file); 34 | } 35 | } 36 | 37 | public 38 | void doGet(HttpServletRequest req, HttpServletResponse res) { 39 | } 40 | } 41 | -------------------------------------------------------------------------------- /ami/bencher/resources/ndbench/log4j.properties: -------------------------------------------------------------------------------- 1 | log4j.debug=TRUE 2 | log4j.rootLogger=INFO, R 3 | 4 | log4j.appender.R=org.apache.log4j.RollingFileAppender 5 | log4j.appender.R.File=/var/log/ndbench/ndbench.log 6 | log4j.appender.R.MaxFileSize=512MB 7 | log4j.appender.R.MaxBackupIndex=2 8 | log4j.appender.R.layout=org.apache.log4j.PatternLayout 9 | log4j.appender.R.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss.SSSS} %p %t %c %m%n -------------------------------------------------------------------------------- /ami/bencher/resources/ndbench/web.xml: -------------------------------------------------------------------------------- 1 | 16 | 17 | 20 | 21 | com.netflix.ndbench.defaultimpl.InjectedWebListener 22 | 23 | 24 | 25 | guiceFilter 26 | com.google.inject.servlet.GuiceFilter 27 | 28 | 29 | 30 | guiceFilter 31 | /REST/* 32 | 33 | 34 | 35 | 36 | log4j-init 37 | com.netflix.ndbench.defaultimpl.Log4jInit 38 | 39 | 40 | log4j-init-file 41 | WEB-INF/classes/log4j.properties 42 | 43 | 44 | 1 45 | 46 | 47 | -------------------------------------------------------------------------------- /ami/bencher/scripts/bencher-provision.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # Copyright 2018-present, Facebook, Inc. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | # 17 | set -e -x 18 | 19 | sudo yum install -y epel-release 20 | sudo yum update -y 21 | 22 | sudo yum remove -y java-1.7.0-openjdk 23 | sudo yum install -y java-1.8.0-openjdk-devel 24 | sudo yum install -y tomcat8 25 | sudo yum install -y ntp 26 | sudo yum install -y git-core 27 | 28 | 29 | # build and install ndbench 30 | 31 | ## setup log dir 32 | sudo mkdir /var/log/ndbench 33 | sudo chown tomcat:tomcat /var/log/ndbench 34 | 35 | 36 | ## checkout code 37 | cd ~ 38 | git clone https://github.com/wpc/ndbench.git 39 | cd ndbench 40 | 41 | ## config logging with fix size rotations otherwise ndbench log will fill up disks fairly quick 42 | cp ~/resources/ndbench/log4j.properties ndbench-web/src/main/resources/ 43 | cp ~/resources/ndbench/Log4jInit.java ndbench-web/src/main/java/com/netflix/ndbench/defaultimpl/Log4jInit.java 44 | cp ~/resources/ndbench/web.xml ndbench-web/src/main/webapp/WEB-INF/web.xml 45 | 46 | 47 | ## build and deploy to tomcat8 48 | ./gradlew clean build 49 | sudo cp ./ndbench-web/build/libs/ndbench-web-0.4.0-SNAPSHOT.war /var/lib/tomcat8/webapps/ROOT.war 50 | 51 | -------------------------------------------------------------------------------- /ami/cassandra3/packer.json: -------------------------------------------------------------------------------- 1 | { 2 | "variables": { 3 | "aws_access_key": "", 4 | "aws_secret_key": "", 5 | "aws_region": "us-west-2", 6 | "aws_ami_image": "ami-a142e9d9", 7 | "aws_instance_type": "i3.large", 8 | "image_version" : "0.0.0" 9 | }, 10 | "builders": [ 11 | { 12 | "type": "amazon-ebs", 13 | "access_key": "{{user `aws_access_key`}}", 14 | "secret_key": "{{user `aws_secret_key`}}", 15 | "region": "{{user `aws_region`}}", 16 | "source_ami": "{{user `aws_ami_image`}}", 17 | "instance_type": "{{user `aws_instance_type`}}", 18 | "ssh_username": "ec2-user", 19 | "ami_name": "cassandra3-benchmark-{{user `image_version`}}", 20 | "ami_groups": "all", 21 | "tags": { 22 | "Name": "cassandra3-benchmark-{{user `image_version`}}", 23 | "Description": "Cassandra 3 image for benchmark purpose" 24 | } 25 | } 26 | ], 27 | "provisioners": [ 28 | { 29 | "type": "file", 30 | "source": "resources", 31 | "destination": "/home/ec2-user/" 32 | }, 33 | 34 | { 35 | "type": "shell", 36 | "scripts": [ 37 | "scripts/cassandra-provision.sh" 38 | ] 39 | } 40 | ] 41 | } 42 | -------------------------------------------------------------------------------- /ami/cassandra3/resources/etc/cassandra/default.conf/cassandra.yaml: -------------------------------------------------------------------------------- 1 | # 2 | # Copyright 2018-present, Facebook, Inc. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | # 16 | auto_bootstrap: true 17 | authenticator: AllowAllAuthenticator 18 | authorizer: AllowAllAuthorizer 19 | auto_snapshot: true 20 | client_encryption_options: 21 | enabled: false 22 | keystore: conf/.keystore 23 | keystore_password: cassandra 24 | cluster_name: benchmark 25 | commitlog_segment_size_in_mb: 32 26 | commitlog_sync: periodic 27 | commitlog_sync_period_in_ms: 10000 28 | commitlog_total_space_in_mb: 4096 29 | compaction_throughput_mb_per_sec: 150 30 | concurrent_reads: 32 31 | concurrent_writes: 32 32 | concurrent_compactors: 10 33 | cross_node_timeout: true 34 | disk_failure_policy: stop 35 | dynamic_snitch_badness_threshold: 0.25 36 | dynamic_snitch_reset_interval_in_ms: 600000 37 | dynamic_snitch_update_interval_in_ms: 100 38 | endpoint_snitch: GossipingPropertyFileSnitch 39 | hinted_handoff_enabled: true 40 | max_hints_delivery_threads: 4 41 | max_hint_window_in_ms: 14400000 42 | hinted_handoff_throttle_in_kb: 5120 43 | incremental_backups: false 44 | inter_dc_tcp_nodelay: true 45 | internode_compression: dc 46 | key_cache_keys_to_save: 0 47 | key_cache_save_period: 14400 48 | key_cache_size_in_mb: 512 49 | listen_address: "__REPLACE_WITH_LOCAL_IP__" 50 | disk_access_mode: mmap_index_only 51 | native_transport_port: 9042 52 | num_tokens: __REPLACE_WITH_NUM_TOKENS__ 53 | partitioner: org.apache.cassandra.dht.Murmur3Partitioner 54 | phi_convict_threshold: 8 55 | range_request_timeout_in_ms: 10000 56 | read_request_timeout_in_ms: 4000 57 | request_scheduler: org.apache.cassandra.scheduler.NoScheduler 58 | request_timeout_in_ms: 10000 59 | row_cache_save_period: 0 60 | row_cache_size_in_mb: 0 61 | rpc_address: '0.0.0.0' 62 | rpc_keepalive: true 63 | rpc_max_threads: 8 64 | rpc_min_threads: 8 65 | rpc_port: 9160 66 | rpc_server_type: hsha 67 | seed_provider: 68 | - class_name: org.apache.cassandra.locator.SimpleSeedProvider 69 | parameters: 70 | - seeds: "__REPLACE_WITH_SEED_IPS__" 71 | server_encryption_options: 72 | internode_encryption: none 73 | keystore: conf/.keystore 74 | keystore_password: cassandra 75 | truststore: conf/.truststore 76 | truststore_password: cassandra 77 | snapshot_before_compaction: false 78 | ssl_storage_port: 7001 79 | start_native_transport: true 80 | start_rpc: true 81 | storage_port: 7000 82 | thrift_framed_transport_size_in_mb: 512 83 | trickle_fsync_interval_in_kb: 10240 84 | trickle_fsync: true 85 | truncate_request_timeout_in_ms: 60000 86 | write_request_timeout_in_ms: 4000 87 | thrift_max_message_length_in_mb: 512 88 | memtable_allocation_type: heap_buffers 89 | broadcast_rpc_address: "__REPLACE_WITH_LOCAL_IP__" 90 | batchlog_replay_throttle_in_kb: 1024 91 | streaming_socket_timeout_in_ms: 86400000 92 | tombstone_failure_threshold: 100000 93 | tombstone_warn_threshold: 10000 94 | sstable_preemptive_open_interval_in_mb: 50 95 | batch_size_warn_threshold_in_kb: 300 96 | batch_size_fail_threshold_in_kb: 500 97 | commit_failure_policy: stop 98 | concurrent_counter_writes: 128 99 | counter_cache_keys_to_save: 100 100 | counter_cache_save_period: 7200 101 | counter_cache_size_in_mb: 0 102 | counter_write_request_timeout_in_ms: 5000 103 | memtable_cleanup_threshold: 0.66 104 | data_file_directories: 105 | - "/var/lib/cassandra/data" 106 | commitlog_directory: "/var/lib/cassandra/commitlog" 107 | saved_caches_directory: "/var/lib/cassandra/saved_caches" 108 | hints_directory: "/var/lib/cassandra/hints" 109 | -------------------------------------------------------------------------------- /ami/cassandra3/resources/etc/cassandra/default.conf/jvm.options: -------------------------------------------------------------------------------- 1 | ########################################################################### 2 | # jvm.options # 3 | # # 4 | # - all flags defined here will be used by cassandra to startup the JVM # 5 | # - one flag should be specified per line # 6 | # - lines that do not start with '-' will be ignored # 7 | # - only static flags are accepted (no variables or parameters) # 8 | # - dynamic flags will be appended to these on cassandra-env # 9 | ########################################################################### 10 | 11 | ################# 12 | # HEAP SETTINGS # 13 | ################# 14 | 15 | # Heap size is automatically calculated by cassandra-env based on this 16 | # formula: max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB)) 17 | # That is: 18 | # - calculate 1/2 ram and cap to 1024MB 19 | # - calculate 1/4 ram and cap to 8192MB 20 | # - pick the max 21 | # 22 | # For production use you may wish to adjust this for your environment. 23 | # If that's the case, uncomment the -Xmx and Xms options below to override the 24 | # automatic calculation of JVM heap memory. 25 | # 26 | # It is recommended to set min (-Xms) and max (-Xmx) heap sizes to 27 | # the same value to avoid stop-the-world GC pauses during resize, and 28 | # so that we can lock the heap in memory on startup to prevent any 29 | # of it from being swapped out. 30 | #-Xms4G 31 | #-Xmx4G 32 | 33 | # Young generation size is automatically calculated by cassandra-env 34 | # based on this formula: min(100 * num_cores, 1/4 * heap size) 35 | # 36 | # The main trade-off for the young generation is that the larger it 37 | # is, the longer GC pause times will be. The shorter it is, the more 38 | # expensive GC will be (usually). 39 | # 40 | # It is not recommended to set the young generation size if using the 41 | # G1 GC, since that will override the target pause-time goal. 42 | # More info: http://www.oracle.com/technetwork/articles/java/g1gc-1984535.html 43 | # 44 | # The example below assumes a modern 8-core+ machine for decent 45 | # times. If in doubt, and if you do not particularly want to tweak, go 46 | # 100 MB per physical CPU core. 47 | #-Xmn800M 48 | 49 | ################# 50 | # GC SETTINGS # 51 | ################# 52 | 53 | ### CMS Settings 54 | 55 | #-XX:+UseParNewGC 56 | #-XX:+UseConcMarkSweepGC 57 | #-XX:+CMSParallelRemarkEnabled 58 | #-XX:SurvivorRatio=8 59 | #-XX:MaxTenuringThreshold=1 60 | #-XX:CMSInitiatingOccupancyFraction=75 61 | #-XX:+UseCMSInitiatingOccupancyOnly 62 | #-XX:CMSWaitDuration=10000 63 | #-XX:+CMSParallelInitialMarkEnabled 64 | #-XX:+CMSEdenChunksRecordAlways 65 | # some JVMs will fill up their heap when accessed via JMX, see CASSANDRA-6541 66 | #-XX:+CMSClassUnloadingEnabled 67 | 68 | ### G1 Settings (experimental, comment previous section and uncomment section below to enable) 69 | 70 | ## Use the Hotspot garbage-first collector. 71 | -XX:+UseG1GC 72 | # 73 | ## Have the JVM do less remembered set work during STW, instead 74 | ## preferring concurrent GC. Reduces p99.9 latency. 75 | #-XX:G1RSetUpdatingPauseTimePercent=5 76 | # 77 | ## Main G1GC tunable: lowering the pause target will lower throughput and vise versa. 78 | ## 200ms is the JVM default and lowest viable setting 79 | ## 1000ms increases throughput. Keep it smaller than the timeouts in cassandra.yaml. 80 | -XX:MaxGCPauseMillis=300 81 | -XX:MaxTenuringThreshold=1 82 | 83 | ## Optional G1 Settings 84 | 85 | # Save CPU time on large (>= 16GB) heaps by delaying region scanning 86 | # until the heap is 70% full. The default in Hotspot 8u40 is 40%. 87 | #-XX:InitiatingHeapOccupancyPercent=70 88 | 89 | # For systems with > 8 cores, the default ParallelGCThreads is 5/8 the number of logical cores. 90 | # Otherwise equal to the number of cores when 8 or less. 91 | # Machines with > 10 cores should try setting these to <= full cores. 92 | #-XX:ParallelGCThreads=16 93 | # By default, ConcGCThreads is 1/4 of ParallelGCThreads. 94 | # Setting both to the same value can reduce STW durations. 95 | #-XX:ConcGCThreads=16 96 | 97 | ### GC logging options -- uncomment to enable 98 | 99 | -XX:+PrintGCDetails 100 | -XX:+PrintGCDateStamps 101 | -XX:+PrintHeapAtGC 102 | -XX:+PrintTenuringDistribution 103 | -XX:+PrintGCApplicationStoppedTime 104 | -XX:+PrintPromotionFailure 105 | #-XX:PrintFLSStatistics=1 106 | -Xloggc:/var/log/cassandra/gc.log 107 | -XX:+UseGCLogFileRotation 108 | -XX:NumberOfGCLogFiles=10 109 | -XX:GCLogFileSize=10M 110 | 111 | ################### 112 | # EXTRA SETTINGS # 113 | ################### 114 | 115 | # disable consistent rangemovement to allow us bootstrap all none seed nodes 116 | # at same time. 117 | -Dcassandra.consistent.rangemovement=false -------------------------------------------------------------------------------- /ami/cassandra3/resources/etc/cassandra/default.conf/logback.xml: -------------------------------------------------------------------------------- 1 | 16 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | INFO 31 | 32 | ${cassandra.logdir}/system.log 33 | 34 | ${cassandra.logdir}/system.log.%i.zip 35 | 1 36 | 20 37 | 38 | 39 | 20MB 40 | 41 | 42 | %-5level [%thread] %date{ISO8601} %F:%L - %msg%n 43 | 44 | 45 | 46 | 47 | 48 | 49 | ${cassandra.logdir}/debug.log 50 | 51 | ${cassandra.logdir}/debug.log.%i.zip 52 | 1 53 | 20 54 | 55 | 56 | 20MB 57 | 58 | 59 | %-5level [%thread] %date{ISO8601} %F:%L - %msg%n 60 | 61 | 62 | 63 | 64 | 65 | 66 | 1024 67 | 0 68 | true 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | INFO 77 | 78 | 79 | %-5level %date{HH:mm:ss,SSS} %msg%n 80 | 81 | 82 | 83 | 86 | 87 | 88 | 89 | 90 | 91 | 94 | 95 | 96 | 97 | 98 | 99 | -------------------------------------------------------------------------------- /ami/cassandra3/resources/initd-disable-transparent-hugepages: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | ### BEGIN INIT INFO 3 | # Provides: disable-transparent-hugepages 4 | # Required-Start: $local_fs 5 | # Required-Stop: 6 | # X-Start-Before: mongod mongodb-mms-automation-agent 7 | # Default-Start: 2 3 4 5 8 | # Default-Stop: 0 1 6 9 | # Short-Description: Disable Linux transparent huge pages 10 | # Description: Disable Linux transparent huge pages, to improve 11 | # database performance. 12 | ### END INIT INFO 13 | 14 | case $1 in 15 | start) 16 | if [ -d /sys/kernel/mm/transparent_hugepage ]; then 17 | thp_path=/sys/kernel/mm/transparent_hugepage 18 | elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then 19 | thp_path=/sys/kernel/mm/redhat_transparent_hugepage 20 | else 21 | return 0 22 | fi 23 | 24 | echo 'never' > ${thp_path}/enabled 25 | echo 'never' > ${thp_path}/defrag 26 | 27 | re='^[0-1]+$' 28 | if [[ $(cat ${thp_path}/khugepaged/defrag) =~ $re ]] 29 | then 30 | # RHEL 7 31 | echo 0 > ${thp_path}/khugepaged/defrag 32 | else 33 | # RHEL 6 34 | echo 'no' > ${thp_path}/khugepaged/defrag 35 | fi 36 | 37 | unset re 38 | unset thp_path 39 | ;; 40 | esac 41 | -------------------------------------------------------------------------------- /ami/cassandra3/resources/limits.conf: -------------------------------------------------------------------------------- 1 | # /etc/security/limits.conf 2 | # 3 | #This file sets the resource limits for the users logged in via PAM. 4 | #It does not affect resource limits of the system services. 5 | # 6 | #Also note that configuration files in /etc/security/limits.d directory, 7 | #which are read in alphabetical order, override the settings in this 8 | #file in case the domain is the same or more specific. 9 | #That means for example that setting a limit for wildcard domain here 10 | #can be overriden with a wildcard setting in a config file in the 11 | #subdirectory, but a user specific setting here can be overriden only 12 | #with a user specific setting in the subdirectory. 13 | # 14 | #Each line describes a limit for a user in the form: 15 | # 16 | # 17 | # 18 | #Where: 19 | # can be: 20 | # - a user name 21 | # - a group name, with @group syntax 22 | # - the wildcard *, for default entry 23 | # - the wildcard %, can be also used with %group syntax, 24 | # for maxlogin limit 25 | # 26 | # can have the two values: 27 | # - "soft" for enforcing the soft limits 28 | # - "hard" for enforcing hard limits 29 | # 30 | # can be one of the following: 31 | # - core - limits the core file size (KB) 32 | # - data - max data size (KB) 33 | # - fsize - maximum filesize (KB) 34 | # - memlock - max locked-in-memory address space (KB) 35 | # - nofile - max number of open file descriptors 36 | # - rss - max resident set size (KB) 37 | # - stack - max stack size (KB) 38 | # - cpu - max CPU time (MIN) 39 | # - nproc - max number of processes 40 | # - as - address space limit (KB) 41 | # - maxlogins - max number of logins for this user 42 | # - maxsyslogins - max number of logins on the system 43 | # - priority - the priority to run user process with 44 | # - locks - max number of file locks the user can hold 45 | # - sigpending - max number of pending signals 46 | # - msgqueue - max memory used by POSIX message queues (bytes) 47 | # - nice - max nice priority allowed to raise to values: [-20, 19] 48 | # - rtprio - max realtime priority 49 | # 50 | # 51 | root soft nofile 32768 52 | root hard nofile 32768 53 | root soft memlock unlimited 54 | root hard memlock unlimited 55 | root soft as unlimited 56 | root hard as unlimited 57 | root hard nproc 32768 58 | root soft nproc 32768 59 | * soft nofile 32768 60 | * hard nofile 32768 61 | * soft memlock unlimited 62 | * hard memlock unlimited 63 | * soft as unlimited 64 | * hard as unlimited 65 | * soft nproc 32768 66 | * hard nproc 32768 67 | 68 | # End of file 69 | -------------------------------------------------------------------------------- /ami/cassandra3/resources/sysctl.conf: -------------------------------------------------------------------------------- 1 | net.core.rmem_max = 16777216 2 | net.core.wmem_max = 16777216 3 | net.core.rmem_default = 16777216 4 | net.core.wmem_default = 16777216 5 | net.core.optmem_max = 40960 6 | net.ipv4.tcp_rmem = 4096 87380 16777216 7 | net.ipv4.tcp_wmem = 4096 65536 16777216 8 | vm.max_map_count = 1048575 9 | xen.independent_wallclock = 1 10 | -------------------------------------------------------------------------------- /ami/cassandra3/scripts/cassandra-provision.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # Copyright 2018-present, Facebook, Inc. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | # 17 | set -e -x 18 | 19 | # Install packages 20 | sudo cp -r /home/ec2-user/resources/ /root/ 21 | 22 | sudo yum install -y epel-release 23 | sudo yum update -y 24 | sudo yum install -y wget 25 | sudo yum remove -y java-1.7.0-openjdk 26 | sudo yum install -y java-1.8.0-openjdk-devel 27 | sudo yum install -y ntp 28 | sudo yum install -y jna 29 | sudo yum install -y jemalloc 30 | sudo yum install -y sysstat 31 | sudo yum install -y dstat 32 | sudo yum install -y htop 33 | sudo yum install -y xfsprogs 34 | sudo yum install -y xfsdump 35 | 36 | # Tune OS 37 | ## Remove constraint on processes 38 | sudo rm -f /etc/security/limits.d/*-nproc.conf 39 | ## Copy our limits.conf 40 | sudo cp ~/resources/limits.conf /etc/security/limits.conf 41 | ## Copy our sysctl.conf 42 | sudo cp ~/resources/sysctl.conf /etc/sysctl.conf 43 | ## Disable huge pages which can cause a CPU spike. 44 | sudo cp ~/resources/initd-disable-transparent-hugepages /etc/init.d/disable-transparent-hugepages 45 | sudo chmod 755 /etc/init.d/disable-transparent-hugepages 46 | 47 | # Install Cassandra 48 | sudo rpm -ivh ~/resources/cassandra.rpm 49 | 50 | # Cassandra configs 51 | sudo cp ~/resources/etc/cassandra/default.conf/cassandra.yaml /etc/cassandra/default.conf/cassandra.yaml 52 | sudo chmod 644 /etc/cassandra/default.conf/cassandra.yaml 53 | 54 | sudo cp ~/resources/etc/cassandra/default.conf/jvm.options /etc/cassandra/default.conf/jvm.options 55 | sudo chmod 644 /etc/cassandra/default.conf/jvm.options 56 | 57 | sudo cp ~/resources/etc/cassandra/default.conf/logback.xml /etc/cassandra/default.conf/logback.xml 58 | sudo chmod 644 /etc/cassandra/default.conf/logback.xml 59 | 60 | -------------------------------------------------------------------------------- /ami/rocksandra/packer.json: -------------------------------------------------------------------------------- 1 | { 2 | "variables": { 3 | "aws_access_key": "", 4 | "aws_secret_key": "", 5 | "aws_region": "us-west-2", 6 | "aws_ami_image": "ami-a142e9d9", 7 | "aws_instance_type": "i3.large", 8 | "image_version" : "0.0.0" 9 | }, 10 | "builders": [ 11 | { 12 | "type": "amazon-ebs", 13 | "access_key": "{{user `aws_access_key`}}", 14 | "secret_key": "{{user `aws_secret_key`}}", 15 | "region": "{{user `aws_region`}}", 16 | "source_ami": "{{user `aws_ami_image`}}", 17 | "instance_type": "{{user `aws_instance_type`}}", 18 | "ssh_username": "ec2-user", 19 | "ami_name": "rocksandra-benchmark-{{user `image_version`}}", 20 | "ami_groups": "all", 21 | "tags": { 22 | "Name": "rocksandra-benchmark-{{user `image_version`}}", 23 | "Description": "Rocksandra image for benchmark purpose" 24 | } 25 | } 26 | ], 27 | "provisioners": [ 28 | { 29 | "type": "file", 30 | "source": "resources", 31 | "destination": "/home/ec2-user/" 32 | }, 33 | 34 | { 35 | "type": "shell", 36 | "scripts": [ 37 | "scripts/cassandra-provision.sh" 38 | ] 39 | } 40 | ] 41 | } 42 | -------------------------------------------------------------------------------- /ami/rocksandra/resources/etc/cassandra/default.conf/cassandra.yaml: -------------------------------------------------------------------------------- 1 | # 2 | # Copyright 2018-present, Facebook, Inc. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | # 16 | auto_bootstrap: true 17 | authenticator: AllowAllAuthenticator 18 | authorizer: AllowAllAuthorizer 19 | auto_snapshot: true 20 | client_encryption_options: 21 | enabled: false 22 | keystore: conf/.keystore 23 | keystore_password: cassandra 24 | cluster_name: benchmark 25 | commitlog_segment_size_in_mb: 32 26 | commitlog_sync: periodic 27 | commitlog_sync_period_in_ms: 10000 28 | commitlog_total_space_in_mb: 4096 29 | compaction_throughput_mb_per_sec: 150 30 | concurrent_reads: 32 31 | concurrent_writes: 32 32 | concurrent_compactors: 4 33 | cross_node_timeout: true 34 | disk_failure_policy: stop 35 | dynamic_snitch_badness_threshold: 0.25 36 | dynamic_snitch_reset_interval_in_ms: 600000 37 | dynamic_snitch_update_interval_in_ms: 100 38 | endpoint_snitch: GossipingPropertyFileSnitch 39 | hinted_handoff_enabled: true 40 | max_hints_delivery_threads: 4 41 | max_hint_window_in_ms: 14400000 42 | hinted_handoff_throttle_in_kb: 5120 43 | incremental_backups: false 44 | inter_dc_tcp_nodelay: true 45 | internode_compression: dc 46 | key_cache_keys_to_save: 0 47 | key_cache_save_period: 14400 48 | key_cache_size_in_mb: 512 49 | listen_address: "__REPLACE_WITH_LOCAL_IP__" 50 | disk_access_mode: mmap_index_only 51 | native_transport_port: 9042 52 | num_tokens: __REPLACE_WITH_NUM_TOKENS__ 53 | partitioner: org.apache.cassandra.dht.Murmur3Partitioner 54 | phi_convict_threshold: 8 55 | range_request_timeout_in_ms: 10000 56 | read_request_timeout_in_ms: 4000 57 | request_scheduler: org.apache.cassandra.scheduler.NoScheduler 58 | request_timeout_in_ms: 10000 59 | row_cache_save_period: 0 60 | row_cache_size_in_mb: 0 61 | rpc_address: '0.0.0.0' 62 | rpc_keepalive: true 63 | rpc_max_threads: 8 64 | rpc_min_threads: 8 65 | rpc_port: 9160 66 | rpc_server_type: hsha 67 | seed_provider: 68 | - class_name: org.apache.cassandra.locator.SimpleSeedProvider 69 | parameters: 70 | - seeds: "__REPLACE_WITH_SEED_IPS__" 71 | server_encryption_options: 72 | internode_encryption: none 73 | keystore: conf/.keystore 74 | keystore_password: cassandra 75 | truststore: conf/.truststore 76 | truststore_password: cassandra 77 | snapshot_before_compaction: false 78 | ssl_storage_port: 7001 79 | start_native_transport: true 80 | start_rpc: true 81 | storage_port: 7000 82 | thrift_framed_transport_size_in_mb: 512 83 | trickle_fsync_interval_in_kb: 10240 84 | trickle_fsync: true 85 | truncate_request_timeout_in_ms: 60000 86 | write_request_timeout_in_ms: 4000 87 | thrift_max_message_length_in_mb: 512 88 | memtable_allocation_type: heap_buffers 89 | broadcast_rpc_address: "__REPLACE_WITH_LOCAL_IP__" 90 | batchlog_replay_throttle_in_kb: 1024 91 | streaming_socket_timeout_in_ms: 86400000 92 | tombstone_failure_threshold: 100000 93 | tombstone_warn_threshold: 10000 94 | sstable_preemptive_open_interval_in_mb: 50 95 | batch_size_warn_threshold_in_kb: 300 96 | batch_size_fail_threshold_in_kb: 500 97 | commit_failure_policy: stop 98 | concurrent_counter_writes: 128 99 | counter_cache_keys_to_save: 100 100 | counter_cache_save_period: 7200 101 | counter_cache_size_in_mb: 0 102 | counter_write_request_timeout_in_ms: 5000 103 | memtable_cleanup_threshold: 0.66 104 | data_file_directories: 105 | - "/var/lib/cassandra/data" 106 | commitlog_directory: "/var/lib/cassandra/commitlog" 107 | saved_caches_directory: "/var/lib/cassandra/saved_caches" 108 | hints_directory: "/var/lib/cassandra/hints" 109 | -------------------------------------------------------------------------------- /ami/rocksandra/resources/etc/cassandra/default.conf/jvm.options: -------------------------------------------------------------------------------- 1 | ########################################################################### 2 | # jvm.options # 3 | # # 4 | # - all flags defined here will be used by cassandra to startup the JVM # 5 | # - one flag should be specified per line # 6 | # - lines that do not start with '-' will be ignored # 7 | # - only static flags are accepted (no variables or parameters) # 8 | # - dynamic flags will be appended to these on cassandra-env # 9 | ########################################################################### 10 | 11 | ################# 12 | # HEAP SETTINGS # 13 | ################# 14 | 15 | # Heap size is automatically calculated by cassandra-env based on this 16 | # formula: max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB)) 17 | # That is: 18 | # - calculate 1/2 ram and cap to 1024MB 19 | # - calculate 1/4 ram and cap to 8192MB 20 | # - pick the max 21 | # 22 | # For production use you may wish to adjust this for your environment. 23 | # If that's the case, uncomment the -Xmx and Xms options below to override the 24 | # automatic calculation of JVM heap memory. 25 | # 26 | # It is recommended to set min (-Xms) and max (-Xmx) heap sizes to 27 | # the same value to avoid stop-the-world GC pauses during resize, and 28 | # so that we can lock the heap in memory on startup to prevent any 29 | # of it from being swapped out. 30 | #-Xms4G 31 | #-Xmx4G 32 | 33 | # Young generation size is automatically calculated by cassandra-env 34 | # based on this formula: min(100 * num_cores, 1/4 * heap size) 35 | # 36 | # The main trade-off for the young generation is that the larger it 37 | # is, the longer GC pause times will be. The shorter it is, the more 38 | # expensive GC will be (usually). 39 | # 40 | # It is not recommended to set the young generation size if using the 41 | # G1 GC, since that will override the target pause-time goal. 42 | # More info: http://www.oracle.com/technetwork/articles/java/g1gc-1984535.html 43 | # 44 | # The example below assumes a modern 8-core+ machine for decent 45 | # times. If in doubt, and if you do not particularly want to tweak, go 46 | # 100 MB per physical CPU core. 47 | #-Xmn800M 48 | 49 | ################# 50 | # GC SETTINGS # 51 | ################# 52 | 53 | ### CMS Settings 54 | 55 | #-XX:+UseParNewGC 56 | #-XX:+UseConcMarkSweepGC 57 | #-XX:+CMSParallelRemarkEnabled 58 | #-XX:SurvivorRatio=8 59 | #-XX:MaxTenuringThreshold=1 60 | #-XX:CMSInitiatingOccupancyFraction=75 61 | #-XX:+UseCMSInitiatingOccupancyOnly 62 | #-XX:CMSWaitDuration=10000 63 | #-XX:+CMSParallelInitialMarkEnabled 64 | #-XX:+CMSEdenChunksRecordAlways 65 | # some JVMs will fill up their heap when accessed via JMX, see CASSANDRA-6541 66 | #-XX:+CMSClassUnloadingEnabled 67 | 68 | ### G1 Settings (experimental, comment previous section and uncomment section below to enable) 69 | 70 | ## Use the Hotspot garbage-first collector. 71 | -XX:+UseG1GC 72 | # 73 | ## Have the JVM do less remembered set work during STW, instead 74 | ## preferring concurrent GC. Reduces p99.9 latency. 75 | #-XX:G1RSetUpdatingPauseTimePercent=5 76 | # 77 | ## Main G1GC tunable: lowering the pause target will lower throughput and vise versa. 78 | ## 200ms is the JVM default and lowest viable setting 79 | ## 1000ms increases throughput. Keep it smaller than the timeouts in cassandra.yaml. 80 | -XX:MaxGCPauseMillis=300 81 | -XX:MaxTenuringThreshold=1 82 | 83 | ## Optional G1 Settings 84 | 85 | # Save CPU time on large (>= 16GB) heaps by delaying region scanning 86 | # until the heap is 70% full. The default in Hotspot 8u40 is 40%. 87 | #-XX:InitiatingHeapOccupancyPercent=70 88 | 89 | # For systems with > 8 cores, the default ParallelGCThreads is 5/8 the number of logical cores. 90 | # Otherwise equal to the number of cores when 8 or less. 91 | # Machines with > 10 cores should try setting these to <= full cores. 92 | #-XX:ParallelGCThreads=16 93 | # By default, ConcGCThreads is 1/4 of ParallelGCThreads. 94 | # Setting both to the same value can reduce STW durations. 95 | #-XX:ConcGCThreads=16 96 | 97 | ### GC logging options -- uncomment to enable 98 | 99 | -XX:+PrintGCDetails 100 | -XX:+PrintGCDateStamps 101 | -XX:+PrintHeapAtGC 102 | -XX:+PrintTenuringDistribution 103 | -XX:+PrintGCApplicationStoppedTime 104 | -XX:+PrintPromotionFailure 105 | #-XX:PrintFLSStatistics=1 106 | -Xloggc:/var/log/cassandra/gc.log 107 | -XX:+UseGCLogFileRotation 108 | -XX:NumberOfGCLogFiles=10 109 | -XX:GCLogFileSize=10M 110 | 111 | ################### 112 | # Rocks SETTINGS # 113 | ################### 114 | 115 | -Dcassandra.consistent.rangemovement=false 116 | -Dcassandra.rocksdb.keyspace=dev1 117 | -Dcassandra.rocksdb.dir=/var/lib/cassandra/data/rocksdbsharded 118 | -Dcassandra.rocksdb.stream.dir=/var/lib/data/cassandra/rocksdbstream/ 119 | -Dcassandra.rocksdb.stream.sst_size=67108864 120 | -Dcassandra.rocksdb.thrift=false 121 | -Dcassandra.rocksdb.thrift.port=28000 122 | -Dcassandra.rocksdb.double_write=false 123 | -Dcassandra.rocksdb.max_levels=4 124 | -Dcassandra.rocksdb.rate_mbytes_per_second=150 125 | -Dcassandra.rocksdb.block_cache_size_mbytes=128 126 | -Dcassandra.rocksdb.write_buffer_size_mbytes=512 127 | -Dcassandra.rocksdb.max_mbytes_for_level_base=1024 128 | -Dcassandra.rocksdb.disable_write_to_commitlog=false 129 | -Dcassandra.rocksdb.merge_operands_limit=0 130 | -Dcassandra.rocksdb.num_shard=10 131 | -Dcassandra.rocksdb.cache_index_and_filter_blocks=true 132 | -Dcassandra.rocksdb.pin_l0_filter_and_index_blocks_in_cache=true 133 | -------------------------------------------------------------------------------- /ami/rocksandra/resources/etc/cassandra/default.conf/logback.xml: -------------------------------------------------------------------------------- 1 | 16 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | INFO 31 | 32 | ${cassandra.logdir}/system.log 33 | 34 | ${cassandra.logdir}/system.log.%i.zip 35 | 1 36 | 20 37 | 38 | 39 | 20MB 40 | 41 | 42 | %-5level [%thread] %date{ISO8601} %F:%L - %msg%n 43 | 44 | 45 | 46 | 47 | 48 | 49 | ${cassandra.logdir}/debug.log 50 | 51 | ${cassandra.logdir}/debug.log.%i.zip 52 | 1 53 | 20 54 | 55 | 56 | 20MB 57 | 58 | 59 | %-5level [%thread] %date{ISO8601} %F:%L - %msg%n 60 | 61 | 62 | 63 | 64 | 65 | 66 | 1024 67 | 0 68 | true 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | INFO 77 | 78 | 79 | %-5level %date{HH:mm:ss,SSS} %msg%n 80 | 81 | 82 | 83 | 86 | 87 | 88 | 89 | 90 | 91 | 94 | 95 | 96 | 97 | 98 | 99 | -------------------------------------------------------------------------------- /ami/rocksandra/resources/initd-disable-transparent-hugepages: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | ### BEGIN INIT INFO 3 | # Provides: disable-transparent-hugepages 4 | # Required-Start: $local_fs 5 | # Required-Stop: 6 | # X-Start-Before: mongod mongodb-mms-automation-agent 7 | # Default-Start: 2 3 4 5 8 | # Default-Stop: 0 1 6 9 | # Short-Description: Disable Linux transparent huge pages 10 | # Description: Disable Linux transparent huge pages, to improve 11 | # database performance. 12 | ### END INIT INFO 13 | 14 | case $1 in 15 | start) 16 | if [ -d /sys/kernel/mm/transparent_hugepage ]; then 17 | thp_path=/sys/kernel/mm/transparent_hugepage 18 | elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then 19 | thp_path=/sys/kernel/mm/redhat_transparent_hugepage 20 | else 21 | return 0 22 | fi 23 | 24 | echo 'never' > ${thp_path}/enabled 25 | echo 'never' > ${thp_path}/defrag 26 | 27 | re='^[0-1]+$' 28 | if [[ $(cat ${thp_path}/khugepaged/defrag) =~ $re ]] 29 | then 30 | # RHEL 7 31 | echo 0 > ${thp_path}/khugepaged/defrag 32 | else 33 | # RHEL 6 34 | echo 'no' > ${thp_path}/khugepaged/defrag 35 | fi 36 | 37 | unset re 38 | unset thp_path 39 | ;; 40 | esac 41 | -------------------------------------------------------------------------------- /ami/rocksandra/resources/limits.conf: -------------------------------------------------------------------------------- 1 | # /etc/security/limits.conf 2 | # 3 | #This file sets the resource limits for the users logged in via PAM. 4 | #It does not affect resource limits of the system services. 5 | # 6 | #Also note that configuration files in /etc/security/limits.d directory, 7 | #which are read in alphabetical order, override the settings in this 8 | #file in case the domain is the same or more specific. 9 | #That means for example that setting a limit for wildcard domain here 10 | #can be overriden with a wildcard setting in a config file in the 11 | #subdirectory, but a user specific setting here can be overriden only 12 | #with a user specific setting in the subdirectory. 13 | # 14 | #Each line describes a limit for a user in the form: 15 | # 16 | # 17 | # 18 | #Where: 19 | # can be: 20 | # - a user name 21 | # - a group name, with @group syntax 22 | # - the wildcard *, for default entry 23 | # - the wildcard %, can be also used with %group syntax, 24 | # for maxlogin limit 25 | # 26 | # can have the two values: 27 | # - "soft" for enforcing the soft limits 28 | # - "hard" for enforcing hard limits 29 | # 30 | # can be one of the following: 31 | # - core - limits the core file size (KB) 32 | # - data - max data size (KB) 33 | # - fsize - maximum filesize (KB) 34 | # - memlock - max locked-in-memory address space (KB) 35 | # - nofile - max number of open file descriptors 36 | # - rss - max resident set size (KB) 37 | # - stack - max stack size (KB) 38 | # - cpu - max CPU time (MIN) 39 | # - nproc - max number of processes 40 | # - as - address space limit (KB) 41 | # - maxlogins - max number of logins for this user 42 | # - maxsyslogins - max number of logins on the system 43 | # - priority - the priority to run user process with 44 | # - locks - max number of file locks the user can hold 45 | # - sigpending - max number of pending signals 46 | # - msgqueue - max memory used by POSIX message queues (bytes) 47 | # - nice - max nice priority allowed to raise to values: [-20, 19] 48 | # - rtprio - max realtime priority 49 | # 50 | # 51 | root soft nofile 32768 52 | root hard nofile 32768 53 | root soft memlock unlimited 54 | root hard memlock unlimited 55 | root soft as unlimited 56 | root hard as unlimited 57 | root hard nproc 32768 58 | root soft nproc 32768 59 | * soft nofile 32768 60 | * hard nofile 32768 61 | * soft memlock unlimited 62 | * hard memlock unlimited 63 | * soft as unlimited 64 | * hard as unlimited 65 | * soft nproc 32768 66 | * hard nproc 32768 67 | 68 | # End of file 69 | -------------------------------------------------------------------------------- /ami/rocksandra/resources/sysctl.conf: -------------------------------------------------------------------------------- 1 | net.core.rmem_max = 16777216 2 | net.core.wmem_max = 16777216 3 | net.core.rmem_default = 16777216 4 | net.core.wmem_default = 16777216 5 | net.core.optmem_max = 40960 6 | net.ipv4.tcp_rmem = 4096 87380 16777216 7 | net.ipv4.tcp_wmem = 4096 65536 16777216 8 | vm.max_map_count = 1048575 9 | xen.independent_wallclock = 1 10 | -------------------------------------------------------------------------------- /ami/rocksandra/scripts/cassandra-provision.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # Copyright 2018-present, Facebook, Inc. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | # 17 | set -e -x 18 | 19 | # Install packages 20 | sudo cp -r /home/ec2-user/resources/ /root/ 21 | 22 | sudo yum install -y epel-release 23 | sudo yum update -y 24 | sudo yum install -y wget 25 | sudo yum remove -y java-1.7.0-openjdk 26 | sudo yum install -y java-1.8.0-openjdk-devel 27 | sudo yum install -y ntp 28 | sudo yum install -y jna 29 | sudo yum install -y jemalloc 30 | sudo yum install -y sysstat 31 | sudo yum install -y dstat 32 | sudo yum install -y htop 33 | sudo yum install -y xfsprogs 34 | sudo yum install -y xfsdump 35 | 36 | 37 | # Tune OS 38 | ## Remove constraint on processes 39 | sudo rm -f /etc/security/limits.d/*-nproc.conf 40 | ## Copy our limits.conf 41 | sudo cp ~/resources/limits.conf /etc/security/limits.conf 42 | ## Copy our sysctl.conf 43 | sudo cp ~/resources/sysctl.conf /etc/sysctl.conf 44 | ## Disable huge pages which can cause a CPU spike. 45 | sudo cp ~/resources/initd-disable-transparent-hugepages /etc/init.d/disable-transparent-hugepages 46 | sudo chmod 755 /etc/init.d/disable-transparent-hugepages 47 | 48 | # Install Cassandra 49 | sudo rpm -ivh ~/resources/rocksandra.rpm 50 | 51 | # Cassandra configs 52 | sudo cp ~/resources/etc/cassandra/default.conf/cassandra.yaml /etc/cassandra/default.conf/cassandra.yaml 53 | sudo chmod 644 /etc/cassandra/default.conf/cassandra.yaml 54 | 55 | sudo cp ~/resources/etc/cassandra/default.conf/jvm.options /etc/cassandra/default.conf/jvm.options 56 | sudo chmod 644 /etc/cassandra/default.conf/jvm.options 57 | 58 | sudo cp ~/resources/etc/cassandra/default.conf/logback.xml /etc/cassandra/default.conf/logback.xml 59 | sudo chmod 644 /etc/cassandra/default.conf/logback.xml 60 | 61 | -------------------------------------------------------------------------------- /bin/ssh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Connect or execute ssh command in single hosts with specified named tag and index 4 | # usage: 5 | # ssh into first cassandra node 6 | # ./bin/ssh ~/.ssh/sshkey mystackname/cassandra 1 7 | # execute `nodetool status` on second cassandra node 8 | # ./bin/ssh ~/.ssh/sshkey mystackname/cassandra 2 "nodetool status" 9 | 10 | host=$(aws ec2 describe-instances \ 11 | --filters Name=tag:Name,Values=$2 Name=instance-state-name,Values=running\ 12 | | jq -r ".Reservations[] | .Instances[] | .PublicDnsName" \ 13 | | head -n$3 \ 14 | | tail -n1) 15 | 16 | ssh \ 17 | -o StrictHostKeyChecking=no \ 18 | -o UserKnownHostsFile=/dev/null \ 19 | -i $1 ec2-user@$host \ 20 | "$4" 21 | 22 | 23 | -------------------------------------------------------------------------------- /bin/ssh_all: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Execute command via ssh in all cassandra or bencher instances 4 | # usage: 5 | # ./bin/ssh_all ~/.ssh/sshkey mystackname/cassandra "nodetool info" 6 | # ./bin/ssh_all ~/.ssh/sshkey mystackname/bencher "tail -n 5 /var/log/ndbench/ndbench.log" 7 | 8 | aws ec2 describe-instances \ 9 | --filters Name=tag:Name,Values=$2 Name=instance-state-name,Values=running \ 10 | | jq -r ".Reservations[] | .Instances[] | .PublicDnsName" \ 11 | | xargs -I{} ssh \ 12 | -o StrictHostKeyChecking=no \ 13 | -o UserKnownHostsFile=/dev/null \ 14 | -i $1 ec2-user@{} \ 15 | "$3" 16 | -------------------------------------------------------------------------------- /cloudformation.yaml: -------------------------------------------------------------------------------- 1 | # 2 | # Copyright 2018-present, Facebook, Inc. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | # 16 | AWSTemplateFormatVersion: '2010-09-09' 17 | Description: CF template for provision single DC cassandra cluster for benchmark purpose 18 | Parameters: 19 | KeyName: 20 | Description: Name of an existing EC2 KeyPair to enable SSH access to the instance 21 | Type: AWS::EC2::KeyPair::KeyName 22 | ConstraintDescription: must be the name of an existing EC2 KeyPair. 23 | Default: sshkey 24 | NoneSeedFleetSize: 25 | Type: Number 26 | Description: Number of nodes that is not seed. ClusterSize=NoneSeedFleetSize+1 27 | Default: '2' 28 | InstanceType: 29 | Description: Node EC2 instance type 30 | Type: String 31 | Default: i3.xlarge 32 | AllowedValues: 33 | - i3.xlarge 34 | - i3.2xlarge 35 | - i3.4xlarge 36 | - i3.8xlarge 37 | - i3.16xlarge 38 | ConstraintDescription: must be a valid EC2 instance type. 39 | AWSRegion: 40 | Type: String 41 | Description: Which aws region we should use 42 | Default: us-west-2 43 | CassandraAMI: 44 | Type: String 45 | Description: Cassandra AMI ID. Make sure you use the one corresponding to the 46 | region setting 47 | Default: ami-9fd06fe7 48 | BencherAMI: 49 | Type: String 50 | Description: NDBnech AMI ID. Make sure you use the one corresponding to the region 51 | setting 52 | Default: ami-4aab1432 53 | BencherInstanceType: 54 | Description: EC2 instance type for Bencher node 55 | Type: String 56 | Default: c5.2xlarge 57 | BencherFleetSize: 58 | Type: Number 59 | Description: Number of bencher nodes for generating load 60 | Default: '4' 61 | VPCSubnetId: 62 | Type: String 63 | Description: VPC subnet id you want to all machines located to 64 | ResourceGroup: 65 | Type: String 66 | Description: Add group tag for your resources for easier to manage 67 | Default: ig_benchmark 68 | InstanceProfile: 69 | Type: String 70 | Description: Name of a precreated instance profile that has permission to ec2::Describ* and autoscalegroup::Describe* 71 | 72 | Mappings: 73 | Scripts: 74 | Cassandra: 75 | DiskMount: | 76 | #!/bin/bash 77 | set -x -e 78 | mount_options=rw,noatime,attr2,inode64,allocsize=64k,noquota 79 | disk_count=$(find /dev/ -name "nvme*n1" -type b | wc -l) 80 | disk_list=$(find /dev/ -name "nvme*n1" -type b | xargs) 81 | if [ $disk_count == "1" ]; 82 | then 83 | # i3.xlarge, i3.2xlarge only have one disk, so just format and mount it 84 | mkfs.xfs -s size=4096 $disk_list 85 | mount -o $mount_options $disk_list /var/lib/cassandra 86 | else 87 | # on i3.4xlarge and above setup raid0 to max flash throughput 88 | mdadm --create --verbose --level=0 /dev/md0 --name=DATA --raid-devices=$disk_count $disk_list 89 | set +e 90 | mdadm --wait /dev/md0 91 | set -e 92 | mkfs.xfs -s size=4096 /dev/md0 93 | mount -o $mount_options /dev/md0 /var/lib/cassandra 94 | fi 95 | mkdir /var/lib/cassandra/{data,commitlog,hints,saved_caches} 96 | chown -R cassandra:cassandra /var/lib/cassandra 97 | 98 | SetupSchema: | 99 | #!/bin/bash 100 | set -x -e 101 | 102 | /usr/bin/cqlsh -e \ 103 | "CREATE KEYSPACE IF NOT EXISTS dev1 \ 104 | WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': '1'} \ 105 | AND durable_writes = true;" 106 | 107 | # skip C* WAL for rocksandra since rocks db already have one. 108 | if grep -q cassandra.rocksdb /etc/cassandra/default.conf/jvm.options 109 | then 110 | /usr/bin/cqlsh -e "ALTER KEYSPACE dev1 with durable_writes = false;" 111 | fi 112 | 113 | /usr/bin/cqlsh -e \ 114 | "CREATE TABLE IF NOT EXISTS dev1.emp ( \ 115 | emp_uname text PRIMARY KEY, \ 116 | emp_dept text, \ 117 | emp_first text, \ 118 | emp_last text \ 119 | );" 120 | 121 | ConfigLocalIp: >- 122 | sed -i "s|__REPLACE_WITH_LOCAL_IP__|`curl -s http://169.254.169.254/latest/meta-data/local-ipv4`|g" /etc/cassandra/default.conf/cassandra.yaml 123 | 124 | ConfigSeedIpsOnSeedItself: >- 125 | sed -i "s|__REPLACE_WITH_SEED_IPS__|`curl -s http://169.254.169.254/latest/meta-data/local-ipv4`|g" /etc/cassandra/default.conf/cassandra.yaml 126 | 127 | ConfigNumTokens: >- 128 | sed -i "s|__REPLACE_WITH_NUM_TOKENS__|256|g" /etc/cassandra/default.conf/cassandra.yaml 129 | 130 | ConfigHeapSize: | 131 | #!/bin/bash 132 | set -x -e 133 | 134 | # heap_cap is max memory we can use. for rocksandar we set it to 12g since 135 | # memtables are hold in rocksdb 136 | if grep -q cassandra.rocksdb /etc/cassandra/default.conf/jvm.options 137 | then 138 | heap_cap=12288 139 | else 140 | heap_cap=65532 141 | fi 142 | # set heap size to 1/4 of system memory, cap to $heap_cap 143 | system_memory_in_mb=`free -m | awk '/:/ {print $2;exit}'` 144 | heap_size_in_mb=`expr $system_memory_in_mb / 4` 145 | if [ "$heap_size_in_mb" -gt "$heap_cap" ] 146 | then 147 | heap_size_in_mb="$heap_cap" 148 | fi 149 | # write new setting to config 150 | echo "\n" >> /etc/cassandra/default.conf/jvm.options 151 | echo "-Xms${heap_size_in_mb}M" >> /etc/cassandra/default.conf/jvm.options 152 | echo "-Xmx${heap_size_in_mb}M" >> /etc/cassandra/default.conf/jvm.options 153 | 154 | 155 | Outputs: 156 | CassandraSeedNode: 157 | Description: Public IP of cassandra seed node 158 | Value: 159 | Fn::GetAtt: 160 | - SeedNodeInstance 161 | - PublicIp 162 | 163 | Resources: 164 | # setup single ec2 instance as seed node 165 | SeedNodeInstance: 166 | Type: AWS::EC2::Instance 167 | Properties: 168 | InstanceType: !Ref InstanceType 169 | EbsOptimized: 'true' 170 | KeyName: !Ref KeyName 171 | ImageId: !Ref CassandraAMI 172 | SubnetId: !Ref VPCSubnetId 173 | Tags: 174 | - Key: Name 175 | Value: 176 | Fn::Sub: >- 177 | ${AWS::StackName}/cassandra 178 | - Key: group 179 | Value: !Ref ResourceGroup 180 | UserData: 181 | Fn::Base64: 182 | Fn::Sub: | 183 | #! /bin/bash -v 184 | # Helper function 185 | function error_exit 186 | { 187 | /opt/aws/bin/cfn-signal -e 1 -r "$1" '${CassandraSeedWaitHandle}' 188 | exit 1 189 | } 190 | # Install packages 191 | /opt/aws/bin/cfn-init -s ${AWS::StackId} -r SeedNodeInstance --region ${AWS::Region} 192 | # All is well so signal success 193 | /opt/aws/bin/cfn-signal -e $? -r "SeedNodeInstance setup complete" '${CassandraSeedWaitHandle}' 194 | Metadata: 195 | AWS::CloudFormation::Init: 196 | configSets: 197 | default: 198 | - configure_hosts 199 | - configure_cassandra 200 | - start_service 201 | - setup_schema 202 | configure_hosts: 203 | commands: 204 | 01-create_hosts_entry-on_boot: 205 | command: echo "`curl -s http://169.254.169.254/latest/meta-data/local-ipv4` `hostname`" >>/etc/hosts 206 | test: test ! -f .create_hosts_entry-semaphore 207 | 02-signal_startup_complete: 208 | command: touch .create_hosts_entry-semaphore 209 | files: 210 | /etc/cfn/cfn-hup.conf: 211 | content: 212 | Fn::Sub: | 213 | [main] 214 | stack=${AWS::StackId} 215 | region=${AWS::Region} 216 | interval=1 217 | mode: '000400' 218 | owner: root 219 | group: root 220 | /etc/cfn/hooks.d/cfn-auto-reloader.conf: 221 | content: 222 | Fn::Sub: | 223 | [cfn-auto-reloader-hook] 224 | triggers=post.update 225 | path=Resources.SeedNodeInstance.Metadata.AWS::CloudFormation::Init 226 | action=/opt/aws/bin/cfn-init -v -s ${AWS::StackId} --resource SeedNodeInstance --configsets default --region ${AWS::Region} 227 | runas=root 228 | mode: '000400' 229 | owner: root 230 | group: root 231 | configure_cassandra: 232 | commands: 233 | 01-config_local_ip: 234 | command: !FindInMap [ Scripts, Cassandra, ConfigLocalIp ] 235 | 02-config_seed_ips: 236 | command: !FindInMap [ Scripts, Cassandra, ConfigSeedIpsOnSeedItself ] 237 | 03-config_tokens: 238 | command: !FindInMap [ Scripts, Cassandra, ConfigNumTokens ] 239 | 04_config_heap: 240 | command: !FindInMap [ Scripts, Cassandra, ConfigHeapSize ] 241 | 05_mount_data_disk: 242 | command: !FindInMap [ Scripts, Cassandra, DiskMount ] 243 | start_service: 244 | services: 245 | sysvinit: 246 | cfn-hup: 247 | enabled: 'true' 248 | ensureRunning: 'true' 249 | files: 250 | - /etc/cfn/cfn-hup.conf 251 | - /etc/cfn/hooks.d/cfn-auto-reloader.conf 252 | commands: 253 | 01-restart-cassandra: 254 | command: /sbin/service cassandra restart 255 | setup_schema: 256 | commands: 257 | 01-wait-for-serverup: 258 | command: /bin/sleep 120 259 | 02-schema: 260 | command: !FindInMap [ Scripts, Cassandra, SetupSchema ] 261 | 262 | CassandraSeedWaitHandle: 263 | Type: AWS::CloudFormation::WaitConditionHandle 264 | CassandraSeedWaitCondition: 265 | Type: AWS::CloudFormation::WaitCondition 266 | DependsOn: SeedNodeInstance 267 | Properties: 268 | Handle: 269 | Ref: CassandraSeedWaitHandle 270 | Timeout: '3600' 271 | 272 | # setup auto scaling group for none seed nodes 273 | CassandraNonSeedFleet: 274 | Type: AWS::AutoScaling::AutoScalingGroup 275 | DependsOn: CassandraSeedWaitCondition 276 | UpdatePolicy: 277 | AutoScalingRollingUpdate: 278 | MaxBatchSize: '1' 279 | MinInstancesInService: '0' 280 | PauseTime: PT0M30S 281 | Properties: 282 | LaunchConfigurationName: !Ref CassandraNonSeedLaunchConfig 283 | MinSize: !Ref NoneSeedFleetSize 284 | MaxSize: !Ref NoneSeedFleetSize 285 | DesiredCapacity: !Ref NoneSeedFleetSize 286 | VPCZoneIdentifier: [ !Ref VPCSubnetId ] 287 | Tags: 288 | - Key: Name 289 | Value: 290 | Fn::Sub: >- 291 | ${AWS::StackName}/cassandra 292 | PropagateAtLaunch: 'true' 293 | - Key: group 294 | Value: !Ref ResourceGroup 295 | PropagateAtLaunch: 'true' 296 | 297 | 298 | CassandraNonSeedLaunchConfig: 299 | Type: AWS::AutoScaling::LaunchConfiguration 300 | Properties: 301 | InstanceType: !Ref InstanceType 302 | EbsOptimized: 'true' 303 | KeyName: !Ref KeyName 304 | ImageId: !Ref CassandraAMI 305 | UserData: 306 | Fn::Base64: 307 | Fn::Sub: | 308 | #! /bin/bash -v 309 | yum update -y 310 | # Helper function 311 | function error_exit 312 | { 313 | /opt/aws/bin/cfn-signal -e 1 -r "$1" '${CassandraNonSeedWaitHandle}' 314 | exit 1 315 | } 316 | # Install packages 317 | /opt/aws/bin/cfn-init -s ${AWS::StackId} -r CassandraNonSeedLaunchConfig --region ${AWS::Region} 318 | # All is well so signal success 319 | /opt/aws/bin/cfn-signal -e $? -r "Cassandra instance setup complete" '${CassandraNonSeedWaitHandle}' 320 | Metadata: 321 | AWS::CloudFormation::Init: 322 | configSets: 323 | default: 324 | - configure_hosts 325 | - configure_cassandra 326 | - start_service 327 | configure_hosts: 328 | commands: 329 | 01-create_hosts_entry-on_boot: 330 | command: echo "`curl -s http://169.254.169.254/latest/meta-data/local-ipv4` `hostname`" >>/etc/hosts 331 | test: test ! -f .create_hosts_entry-semaphore 332 | 02-signal_startup_complete: 333 | command: touch .create_hosts_entry-semaphore 334 | files: 335 | /etc/cfn/cfn-hup.conf: 336 | content: 337 | Fn::Sub: | 338 | [main] 339 | stack=${AWS::StackId} 340 | region=${AWS::Region} 341 | interval=1 342 | mode: '000400' 343 | owner: root 344 | group: root 345 | /etc/cfn/hooks.d/cfn-auto-reloader.conf: 346 | content: 347 | Fn::Sub: | 348 | [cfn-auto-reloader-hook] 349 | triggers=post.update 350 | path=Resources.CassandraNonSeedLaunchConfig.Metadata.AWS::CloudFormation::Init 351 | action=/opt/aws/bin/cfn-init -v -s ${AWS::StackId} --resource CassandraNonSeedLaunchConfig --configsets default --region ${AWS::Region} 352 | runas=root 353 | mode: '000400' 354 | owner: root 355 | group: root 356 | configure_cassandra: 357 | commands: 358 | 01-config_local_ip: 359 | command: !FindInMap [ Scripts, Cassandra, ConfigLocalIp ] 360 | 02-config_seed_ips: 361 | command: 362 | Fn::Sub: >- 363 | sed -i "s|__REPLACE_WITH_SEED_IPS__|${SeedNodeInstance.PrivateIp}|g" /etc/cassandra/default.conf/cassandra.yaml 364 | 03-config_tokens: 365 | command: !FindInMap [ Scripts, Cassandra, ConfigNumTokens ] 366 | 04_config_heap: 367 | command: !FindInMap [ Scripts, Cassandra, ConfigHeapSize ] 368 | 05_mount_data_disk: 369 | command: !FindInMap [ Scripts, Cassandra, DiskMount ] 370 | start_service: 371 | services: 372 | sysvinit: 373 | cfn-hup: 374 | enabled: 'true' 375 | ensureRunning: 'true' 376 | files: 377 | - /etc/cfn/cfn-hup.conf 378 | - /etc/cfn/hooks.d/cfn-auto-reloader.conf 379 | commands: 380 | 01-restart-cassandra: 381 | command: /sbin/service cassandra restart 382 | CassandraNonSeedWaitHandle: 383 | Type: AWS::CloudFormation::WaitConditionHandle 384 | CassandraNonSeedWaitCondition: 385 | Type: AWS::CloudFormation::WaitCondition 386 | DependsOn: CassandraNonSeedFleet 387 | Properties: 388 | Handle: 389 | Ref: CassandraNonSeedWaitHandle 390 | Timeout: '3600' 391 | Count: '1' 392 | 393 | # setup auto scaling group for multiple bencher nodes with ndbench 394 | BencherFleet: 395 | Type: AWS::AutoScaling::AutoScalingGroup 396 | DependsOn: CassandraNonSeedWaitCondition 397 | UpdatePolicy: 398 | AutoScalingRollingUpdate: 399 | MaxBatchSize: "2" 400 | MinInstancesInService: '0' 401 | PauseTime: PT5S 402 | Properties: 403 | VPCZoneIdentifier: [ !Ref VPCSubnetId ] 404 | LaunchConfigurationName: !Ref BencherLaunchConfig 405 | MinSize: !Ref BencherFleetSize 406 | MaxSize: !Ref BencherFleetSize 407 | DesiredCapacity: !Ref BencherFleetSize 408 | Tags: 409 | - Key: Name 410 | Value: 411 | Fn::Sub: >- 412 | ${AWS::StackName}/bencher 413 | PropagateAtLaunch: 'true' 414 | - Key: group 415 | Value: !Ref ResourceGroup 416 | PropagateAtLaunch: 'true' 417 | 418 | 419 | BencherLaunchConfig: 420 | Type: AWS::AutoScaling::LaunchConfiguration 421 | Properties: 422 | InstanceType: !Ref BencherInstanceType 423 | IamInstanceProfile: !Ref InstanceProfile 424 | KeyName: !Ref KeyName 425 | ImageId: !Ref BencherAMI 426 | UserData: 427 | Fn::Base64: 428 | Fn::Sub: | 429 | #! /bin/bash -v 430 | # Helper function 431 | function error_exit 432 | { 433 | /opt/aws/bin/cfn-signal -e 1 -r "$1" '${BencherWaitHandle}' 434 | exit 1 435 | } 436 | # Install packages 437 | /opt/aws/bin/cfn-init -s ${AWS::StackId} -r BencherLaunchConfig --region ${AWS::Region} 438 | # All is well so signal success 439 | /opt/aws/bin/cfn-signal -e $? -r "bencher setup complete. Last error: $(grep ERROR /var/log/cfn-init.log | tail -n 1)" '${BencherWaitHandle}' 440 | Metadata: 441 | AWS::CloudFormation::Init: 442 | configSets: 443 | default: 444 | - configure_hosts 445 | - configure_ndbench 446 | - start_service 447 | configure_hosts: 448 | files: 449 | /etc/cfn/cfn-hup.conf: 450 | content: 451 | Fn::Sub: |- 452 | [main] 453 | stack=${AWS::StackId} 454 | region=${AWS::Region} 455 | interval=1 456 | mode: '000400' 457 | owner: root 458 | group: root 459 | /etc/cfn/hooks.d/cfn-auto-reloader.conf: 460 | content: 461 | Fn::Sub: | 462 | [cfn-auto-reloader-hook] 463 | triggers=post.update 464 | path=Resources.BencherLaunchConfig.Metadata.AWS::CloudFormation::Init 465 | action=/opt/aws/bin/cfn-init -v -s ${AWS::StackId} --resource BencherLaunchConfig --configsets default --region ${AWS::Region} 466 | runas=root 467 | mode: '000400' 468 | owner: root 469 | group: root 470 | configure_ndbench: 471 | files: 472 | /etc/sysconfig/tomcat8: 473 | content: 474 | Fn::Sub: | 475 | export DISCOVERY_ENV=AWS_ASG 476 | export JAVA_OPTS="$JAVA_OPTS -Dndbench.config.dataSize=2000" 477 | export JAVA_OPTS="$JAVA_OPTS -Dndbench.config.numBackfill=64" 478 | export JAVA_OPTS="$JAVA_OPTS -Dndbench.config.numKeys=250000000" 479 | export JAVA_OPTS="$JAVA_OPTS -Dndbench.config.numReaders=64" 480 | export JAVA_OPTS="$JAVA_OPTS -Dndbench.config.numValues=100" 481 | export JAVA_OPTS="$JAVA_OPTS -Dndbench.config.numWriters=64" 482 | export JAVA_OPTS="$JAVA_OPTS -Dndbench.config.readRateLimit=10000" 483 | export JAVA_OPTS="$JAVA_OPTS -Dndbench.config.writeRateLimit=10000" 484 | export JAVA_OPTS="$JAVA_OPTS -Dndbench.config.cass.cluster=benchmark" 485 | export JAVA_OPTS="$JAVA_OPTS -Dndbench.config.cass.host=${SeedNodeInstance.PrivateIp}" 486 | mode: '000644' 487 | owner: root 488 | group: root 489 | start_service: 490 | services: 491 | sysvinit: 492 | cfn-hup: 493 | enabled: 'true' 494 | ensureRunning: 'true' 495 | files: 496 | - /etc/cfn/cfn-hup.conf 497 | - /etc/cfn/hooks.d/cfn-auto-reloader.conf 498 | commands: 499 | 01-restart-tomcat: 500 | command: /sbin/service tomcat8 restart 501 | 502 | BencherWaitHandle: 503 | Type: AWS::CloudFormation::WaitConditionHandle 504 | 505 | BencherWaitCondition: 506 | Type: AWS::CloudFormation::WaitCondition 507 | DependsOn: BencherFleet 508 | Properties: 509 | Handle: 510 | Ref: BencherWaitHandle 511 | Timeout: '3600' 512 | Count: '1' 513 | --------------------------------------------------------------------------------