├── .github └── PULL_REQUEST_TEMPLATE.md ├── Berksfile ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── layer-custom ├── recipes │ ├── allocation-awareness.rb │ ├── cloudwatch-custom.rb │ ├── esmonit.rb │ └── esplugins.rb └── templates │ └── default │ ├── elasticsearch.cloudwatch-custom.sh.erb │ └── elasticsearch.monitrc.conf.erb ├── opsworks-elasticsearch-11-10.template └── opsworks-elasticsearch.template /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | *Issue #, if available:* 2 | 3 | *Description of changes:* 4 | 5 | 6 | By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. 7 | -------------------------------------------------------------------------------- /Berksfile: -------------------------------------------------------------------------------- 1 | source 'https://api.berkshelf.com' 2 | 3 | cookbook 'apt', '>= 2.3.8', git: 'git://github.com/opscode-cookbooks/apt.git', tag: 'v2.3.8' 4 | cookbook 'ark', git: 'git://github.com/opscode-cookbooks/ark.git', tag: 'v0.4.2' 5 | cookbook 'bluepill', '>= 2.3.1', git: 'git://github.com/opscode-cookbooks/bluepill.git', tag: 'v2.3.1' 6 | cookbook 'build-essential', '>= 1.4.2', git: 'git://github.com/opscode-cookbooks/build-essential.git', tag: 'v1.4.2' 7 | cookbook 'java', '>= 1.19.2', git: 'git://github.com/opscode-cookbooks/java.git', tag: 'v1.21.2' 8 | cookbook 'monit', '>= 0.6', git: 'git://github.com/apsoto/monit.git' 9 | cookbook 'ohai', '>= 1.1.12', git: 'git://github.com/opscode-cookbooks/ohai.git', tag: 'v1.1.12' 10 | cookbook 'python', '>= 1.4.6', git: 'git://github.com/poise/python.git', tag: 'v1.4.6' 11 | cookbook 'rsyslog', '>= 1.11.0', git: 'git://github.com/opscode-cookbooks/rsyslog.git', tag: 'v1.11.0' 12 | cookbook 'windows', '>= 1.30.0', git: 'git://github.com/opscode-cookbooks/windows.git', tag: 'v1.30.2' 13 | cookbook 'yum', git: 'git://github.com/opscode-cookbooks/yum.git', tag: 'v3.1.0' 14 | cookbook 'yum-epel', '>= 0.3.4', git: 'git://github.com/opscode-cookbooks/yum-epel.git', tag: 'v0.3.4' 15 | cookbook 'elasticsearch', git: 'git://github.com/elasticsearch/cookbook-elasticsearch.git', tag: '0.3.7' 16 | cookbook 'mongodb', git: 'git://github.com/edelight/chef-mongodb' 17 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check [existing open](https://github.com/${GITHUB_ORG}/${GITHUB_REPO}/issues), or [recently closed](https://github.com/${GITHUB_ORG}/${GITHUB_REPO}/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels ((enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/${GITHUB_ORG}/${GITHUB_REPO}/labels/help%20wanted) issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](https://github.com/${GITHUB_ORG}/${GITHUB_REPO}/blob/master/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "{}" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright {yyyy} {name of copyright owner} 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | opsworks-elasticsearch-cookbook 2 | =============================== 3 | 4 | AWS OpsWorks Elasticsearch sample cookbook referenced in [this blog] (http://blogs.aws.amazon.com/application-management/post/Tx3MEVKS0A4G7R5/Deploying-Elasticsearch-with-OpsWorks) 5 | -------------------------------------------------------------------------------- /layer-custom/recipes/allocation-awareness.rb: -------------------------------------------------------------------------------- 1 | node.default[:elasticsearch][:custom_config] = {'node.rack_id' => "#{node[:opsworks][:instance][:availability_zone]}"} 2 | 3 | -------------------------------------------------------------------------------- /layer-custom/recipes/cloudwatch-custom.rb: -------------------------------------------------------------------------------- 1 | # Create script that pushes custom metrics into Cloudwatch 2 | # 3 | template "/usr/local/bin/cloudwatch-custom.sh" do 4 | source "elasticsearch.cloudwatch-custom.sh.erb" 5 | mode "0550" 6 | owner "root" 7 | group "root" 8 | end 9 | 10 | # Cron it so that it runs every minute 11 | # 12 | cron "cloudwatch-custom" do 13 | hour "*" 14 | minute "*" 15 | weekday "*" 16 | command "/usr/local/bin/cloudwatch-custom.sh" 17 | end 18 | -------------------------------------------------------------------------------- /layer-custom/recipes/esmonit.rb: -------------------------------------------------------------------------------- 1 | template "/etc/monit.d/elasticsearch-monit.conf" do 2 | source "elasticsearch.monitrc.conf.erb" 3 | mode 0440 4 | owner "root" 5 | group "root" 6 | end -------------------------------------------------------------------------------- /layer-custom/recipes/esplugins.rb: -------------------------------------------------------------------------------- 1 | script "install_plugin_es_head" do 2 | interpreter "bash" 3 | user "root" 4 | cwd "#{node.elasticsearch[:dir]}/elasticsearch-#{node.elasticsearch[:version]}/bin/" 5 | code <<-EOH 6 | plugin -install mobz/elasticsearch-head 7 | EOH 8 | not_if { File.exist?("#{node.elasticsearch[:dir]}/elasticsearch-#{node.elasticsearch[:version]}/plugins/head") } 9 | end 10 | 11 | #notifies :restart, 'service[elasticsearch]' unless node.elasticsearch[:skip_restart] 12 | -------------------------------------------------------------------------------- /layer-custom/templates/default/elasticsearch.cloudwatch-custom.sh.erb: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | aws cloudwatch put-metric-data --region <%= node[:opsworks][:instance][:region] %> --namespace "ElasticSearch-<%= node.elasticsearch[:cluster][:name] %>" --metric-name "number_of_data_nodes" --unit "Count" --value "`curl -s 'http://localhost:9200/_cluster/health' | python -c 'import json,sys;obj=json.load(sys.stdin);print obj["number_of_data_nodes"]'`" 3 | -------------------------------------------------------------------------------- /layer-custom/templates/default/elasticsearch.monitrc.conf.erb: -------------------------------------------------------------------------------- 1 | # ------------------------------------------ 2 | # Monit configuration file for ElasticSearch 3 | # ------------------------------------------ 4 | 5 | check process elasticsearch with pidfile <%= node.elasticsearch[:pid_file] %> 6 | start program = "/etc/init.d/elasticsearch restart" with timeout 60 seconds 7 | stop program = "/etc/init.d/elasticsearch stop" 8 | if cpu > 90% for 15 cycles then alert 9 | if totalmem > 90% for 15 cycles then alert 10 | if loadavg(15min) greater than 10 for 50 cycles then alert 11 | group elasticsearch 12 | 13 | <% if node.monit[:notify_email] %> 14 | check host elasticsearch_connection with address 0.0.0.0 15 | if failed url http://0.0.0.0:<%= node.elasticsearch[:http][:port] %>/ with timeout 15 seconds then alert 16 | group elasticsearch 17 | 18 | check host elasticsearch_cluster_health with address 0.0.0.0 19 | if failed url http://0.0.0.0:<%= node.elasticsearch[:http][:port] %>/_cluster/health 20 | and content == 'green' 21 | with timeout 60 seconds 22 | then alert 23 | alert <%= node.monit[:notify_email] %> with mail-format { 24 | subject: [monit] elasticsearch: CLUSTER HEALTH PROBLEM at <%= node.hostname %> 25 | message: [<%= node.hostname %>] $SERVICE $ACTION 26 | <% if node.monit[:http_auth] && node.cloud %> 27 | -- 28 | http://<%= node.monit[:http_auth]['username'] %>:<%= node.monit[:http_auth]['password_encoded'] %>@<%= node.cloud.public_hostname %>:2812/elasticsearch_cluster_health 29 | <% end %> 30 | } 31 | group elasticsearch 32 | <% end %> 33 | -------------------------------------------------------------------------------- /opsworks-elasticsearch-11-10.template: -------------------------------------------------------------------------------- 1 | { 2 | "AWSTemplateFormatVersion": "2010-09-09", 3 | "Parameters": { 4 | "DefaultOWRoles": { 5 | "Description": "Yes = have OpsWorks use existing IAM roles from another OpsWorks stack, or No=create roles in the CloudFormation stack.", 6 | "Type": "String", 7 | "Default": "Yes", 8 | "AllowedValues": [ 9 | "Yes", 10 | "No" 11 | ] 12 | }, 13 | "WebUsername": { 14 | "Description": "WebUsername", 15 | "Type": "String", 16 | "Default": "username" 17 | }, 18 | "WebPassword": { 19 | "Description": "WebPassword", 20 | "Type": "String", 21 | "Default": "password" 22 | }, 23 | "CookbookRepo": { 24 | "Description": "GitURL", 25 | "Type": "String", 26 | "Default": "https://github.com/amazonwebservices/opsworks-elasticsearch-cookbook" 27 | } 28 | }, 29 | "Conditions": { 30 | "DefaultOWRolesCondition": { 31 | "Fn::Equals": [ 32 | { 33 | "Ref": "DefaultOWRoles" 34 | }, 35 | "Yes" 36 | ] 37 | }, 38 | "NotDefaultOWRolesCondition": { 39 | "Fn::Equals": [ 40 | { 41 | "Ref": "DefaultOWRoles" 42 | }, 43 | "No" 44 | ] 45 | } 46 | }, 47 | "Resources": { 48 | "ServiceRole": { 49 | "Condition": "NotDefaultOWRolesCondition", 50 | "Type": "AWS::IAM::Role", 51 | "Properties": { 52 | "AssumeRolePolicyDocument": { 53 | "Statement": [ 54 | { 55 | "Effect": "Allow", 56 | "Principal": { 57 | "Service": [ 58 | "opsworks.amazonaws.com" 59 | ] 60 | }, 61 | "Action": [ 62 | "sts:AssumeRole" 63 | ] 64 | } 65 | ] 66 | }, 67 | "Path": "/", 68 | "Policies": [ 69 | { 70 | "PolicyName": "opsworks-service", 71 | "PolicyDocument": { 72 | "Statement": [ 73 | { 74 | "Effect": "Allow", 75 | "Action": [ 76 | "ec2:*", 77 | "iam:PassRole", 78 | "cloudwatch:GetMetricStatistics", 79 | "elasticloadbalancing:*" 80 | ], 81 | "Resource": "*" 82 | } 83 | ] 84 | } 85 | } 86 | ] 87 | } 88 | }, 89 | "OpsWorksEC2Role": { 90 | "Condition": "NotDefaultOWRolesCondition", 91 | "Type": "AWS::IAM::Role", 92 | "Properties": { 93 | "AssumeRolePolicyDocument": { 94 | "Statement": [ 95 | { 96 | "Effect": "Allow", 97 | "Principal": { 98 | "Service": [ 99 | "ec2.amazonaws.com" 100 | ] 101 | }, 102 | "Action": [ 103 | "sts:AssumeRole" 104 | ] 105 | } 106 | ] 107 | }, 108 | "Path": "/", 109 | "Policies": [ 110 | { 111 | "PolicyName": "opsworks-ec2-role", 112 | "PolicyDocument": { 113 | "Statement": [ 114 | { 115 | "Effect": "Allow", 116 | "Action": [ 117 | "ec2:DescribeInstances", 118 | "ec2:DescribeRegions", 119 | "ec2:DescribeSecurityGroups", 120 | "ec2:DescribeTags", 121 | "cloudwatch:PutMetricData" 122 | ], 123 | "Resource": "*" 124 | } 125 | ] 126 | } 127 | } 128 | ] 129 | } 130 | }, 131 | "InstanceRole": { 132 | "Condition": "NotDefaultOWRolesCondition", 133 | "Type": "AWS::IAM::InstanceProfile", 134 | "Properties": { 135 | "Path": "/", 136 | "Roles": [ 137 | { 138 | "Ref": "OpsWorksEC2Role" 139 | } 140 | ] 141 | } 142 | }, 143 | "myStack": { 144 | "Type": "AWS::OpsWorks::Stack", 145 | "Properties": { 146 | "Name": { 147 | "Ref": "AWS::StackName" 148 | }, 149 | "CustomJson": { 150 | "java": { 151 | "jdk_version": "7", 152 | "oracle": { 153 | "accept_oracle_download_terms": "true" 154 | }, 155 | "accept_license_agreement": "true", 156 | "install_flavor": "oracle" 157 | }, 158 | "elasticsearch": { 159 | "nginx": { 160 | "users": [ 161 | { 162 | "username": { 163 | "Ref": "WebUsername" 164 | }, 165 | "password": { 166 | "Ref": "WebPassword" 167 | } 168 | } 169 | ], 170 | "allow_cluster_api": "true", 171 | "port": 80 172 | }, 173 | "cluster": { 174 | "name": "opsworks-elasticsearch" 175 | }, 176 | "gateway": { 177 | "expected_nodes": 3 178 | }, 179 | "discovery": { 180 | "type": "ec2", 181 | "zen": { 182 | "minimum_master_nodes": 2, 183 | "ping": { 184 | "multicast": { 185 | "enabled": false 186 | } 187 | } 188 | }, 189 | "ec2": { 190 | "tag": { 191 | "opsworks:stack": { 192 | "Ref": "AWS::StackName" 193 | } 194 | } 195 | } 196 | }, 197 | "path": { 198 | "data": "/mnt/elasticsearch-data" 199 | }, 200 | "cloud": { 201 | "aws": { 202 | "region": "us-east-1" 203 | } 204 | }, 205 | "custom_config": { 206 | "cluster.routing.allocation.awareness.attributes": "rack_id" 207 | } 208 | } 209 | }, 210 | "ServiceRoleArn": { 211 | "Fn::If": [ 212 | "DefaultOWRolesCondition", 213 | { 214 | "Fn::Join": [ 215 | "", 216 | [ 217 | "arn:aws:iam::", 218 | { 219 | "Ref": "AWS::AccountId" 220 | }, 221 | ":role/aws-opsworks-service-role" 222 | ] 223 | ] 224 | }, 225 | { 226 | "Fn::GetAtt": [ 227 | "ServiceRole", 228 | "Arn" 229 | ] 230 | } 231 | ] 232 | }, 233 | "DefaultInstanceProfileArn": { 234 | "Fn::If": [ 235 | "DefaultOWRolesCondition", 236 | { 237 | "Fn::Join": [ 238 | "", 239 | [ 240 | "arn:aws:iam::", 241 | { 242 | "Ref": "AWS::AccountId" 243 | }, 244 | ":instance-profile/aws-opsworks-ec2-role" 245 | ] 246 | ] 247 | }, 248 | { 249 | "Fn::GetAtt": [ 250 | "InstanceRole", 251 | "Arn" 252 | ] 253 | } 254 | ] 255 | }, 256 | "ConfigurationManager": { 257 | "Name" : "Chef", 258 | "Version" : "11.10" 259 | }, 260 | "UseCustomCookbooks": "true", 261 | "CustomCookbooksSource": { 262 | "Type": "git", 263 | "Url": { 264 | "Ref": "CookbookRepo" 265 | } 266 | } 267 | } 268 | }, 269 | "elasticsearchtest": { 270 | "Type": "AWS::EC2::SecurityGroup", 271 | "Properties": { 272 | "GroupDescription": "so that ES cluster can find other nodes" 273 | } 274 | }, 275 | "searchLayer": { 276 | "Type": "AWS::OpsWorks::Layer", 277 | "Properties": { 278 | "StackId": { 279 | "Ref": "myStack" 280 | }, 281 | "Name": "Search", 282 | "Type": "custom", 283 | "Shortname": "search", 284 | "CustomRecipes": { 285 | "Setup": [ 286 | "apt", 287 | "ark", 288 | "elasticsearch", 289 | "elasticsearch::aws", 290 | "elasticsearch::proxy", 291 | "java", 292 | "layer-custom::esplugins", 293 | "layer-custom::allocation-awareness", 294 | "layer-custom::esmonit", 295 | "layer-custom::cloudwatch-custom" 296 | ] 297 | }, 298 | "EnableAutoHealing": "true", 299 | "AutoAssignElasticIps": "false", 300 | "AutoAssignPublicIps": "true", 301 | "VolumeConfigurations": [ 302 | { 303 | "MountPoint": "/mnt/elasticsearch-data", 304 | "NumberOfDisks": 1, 305 | "Size": 100 306 | } 307 | ], 308 | "CustomSecurityGroupIds": [ 309 | { 310 | "Fn::GetAtt": [ 311 | "elasticsearchtest", 312 | "GroupId" 313 | ] 314 | } 315 | ] 316 | } 317 | } 318 | } 319 | } 320 | -------------------------------------------------------------------------------- /opsworks-elasticsearch.template: -------------------------------------------------------------------------------- 1 | { 2 | "AWSTemplateFormatVersion": "2010-09-09", 3 | "Parameters": { 4 | "DefaultOWRoles": { 5 | "Description": "Yes = have OpsWorks use existing IAM roles, or No=create roles in the CloudFormation stack.", 6 | "Type": "String", 7 | "Default": "No", 8 | "AllowedValues": [ 9 | "Yes", 10 | "No" 11 | ] 12 | }, 13 | "WebUsername": { 14 | "Description": "WebUsername", 15 | "Type": "String", 16 | "Default": "username" 17 | }, 18 | "WebPassword": { 19 | "Description": "WebPassword", 20 | "Type": "String", 21 | "Default": "password" 22 | }, 23 | "S3Archive": { 24 | "Description": "S3ArchiveURL", 25 | "Type": "String", 26 | "Default": "ENTER YOUR S3 ARCHIVE URL" 27 | } 28 | }, 29 | "Conditions": { 30 | "DefaultOWRolesCondition": { 31 | "Fn::Equals": [ 32 | { 33 | "Ref": "DefaultOWRoles" 34 | }, 35 | "Yes" 36 | ] 37 | }, 38 | "NotDefaultOWRolesCondition": { 39 | "Fn::Equals": [ 40 | { 41 | "Ref": "DefaultOWRoles" 42 | }, 43 | "No" 44 | ] 45 | } 46 | }, 47 | "Resources": { 48 | "ServiceRole": { 49 | "Condition": "NotDefaultOWRolesCondition", 50 | "Type": "AWS::IAM::Role", 51 | "Properties": { 52 | "AssumeRolePolicyDocument": { 53 | "Statement": [ 54 | { 55 | "Effect": "Allow", 56 | "Principal": { 57 | "Service": [ 58 | "opsworks.amazonaws.com" 59 | ] 60 | }, 61 | "Action": [ 62 | "sts:AssumeRole" 63 | ] 64 | } 65 | ] 66 | }, 67 | "Path": "/", 68 | "Policies": [ 69 | { 70 | "PolicyName": "opsworks-service", 71 | "PolicyDocument": { 72 | "Statement": [ 73 | { 74 | "Effect": "Allow", 75 | "Action": [ 76 | "ec2:*", 77 | "iam:PassRole", 78 | "cloudwatch:GetMetricStatistics", 79 | "elasticloadbalancing:*" 80 | ], 81 | "Resource": "*" 82 | } 83 | ] 84 | } 85 | } 86 | ] 87 | } 88 | }, 89 | "OpsWorksEC2Role": { 90 | "Condition": "NotDefaultOWRolesCondition", 91 | "Type": "AWS::IAM::Role", 92 | "Properties": { 93 | "AssumeRolePolicyDocument": { 94 | "Statement": [ 95 | { 96 | "Effect": "Allow", 97 | "Principal": { 98 | "Service": [ 99 | "ec2.amazonaws.com" 100 | ] 101 | }, 102 | "Action": [ 103 | "sts:AssumeRole" 104 | ] 105 | } 106 | ] 107 | }, 108 | "Path": "/", 109 | "Policies": [ 110 | { 111 | "PolicyName": "opsworks-ec2-role", 112 | "PolicyDocument": { 113 | "Statement": [ 114 | { 115 | "Effect": "Allow", 116 | "Action": [ 117 | "ec2:DescribeInstances", 118 | "ec2:DescribeRegions", 119 | "ec2:DescribeSecurityGroups", 120 | "ec2:DescribeTags", 121 | "cloudwatch:PutMetricData" 122 | ], 123 | "Resource": "*" 124 | } 125 | ] 126 | } 127 | } 128 | ] 129 | } 130 | }, 131 | "InstanceRole": { 132 | "Condition": "NotDefaultOWRolesCondition", 133 | "Type": "AWS::IAM::InstanceProfile", 134 | "Properties": { 135 | "Path": "/", 136 | "Roles": [ 137 | { 138 | "Ref": "OpsWorksEC2Role" 139 | } 140 | ] 141 | } 142 | }, 143 | "myStack": { 144 | "Type": "AWS::OpsWorks::Stack", 145 | "Properties": { 146 | "Name": { 147 | "Ref": "AWS::StackName" 148 | }, 149 | "CustomJson": { 150 | "java": { 151 | "jdk_version": "7", 152 | "oracle": { 153 | "accept_oracle_download_terms": "true" 154 | }, 155 | "accept_license_agreement": "true", 156 | "install_flavor": "oracle" 157 | }, 158 | "elasticsearch": { 159 | "nginx": { 160 | "users": [ 161 | { 162 | "username": { 163 | "Ref": "WebUsername" 164 | }, 165 | "password": { 166 | "Ref": "WebPassword" 167 | } 168 | } 169 | ], 170 | "allow_cluster_api": "true", 171 | "port": 80 172 | }, 173 | "cluster": { 174 | "name": "opsworks-elasticsearch" 175 | }, 176 | "gateway": { 177 | "expected_nodes": 3 178 | }, 179 | "discovery": { 180 | "type": "ec2", 181 | "zen": { 182 | "minimum_master_nodes": 2, 183 | "ping": { 184 | "multicast": { 185 | "enabled": false 186 | } 187 | } 188 | }, 189 | "ec2": { 190 | "tag": { 191 | "opsworks:stack": { 192 | "Ref": "AWS::StackName" 193 | } 194 | } 195 | } 196 | }, 197 | "path": { 198 | "data": "/mnt/elasticsearch-data" 199 | }, 200 | "cloud": { 201 | "aws": { 202 | "region": "us-east-1" 203 | } 204 | }, 205 | "custom_config": { 206 | "cluster.routing.allocation.awareness.attributes": "rack_id" 207 | } 208 | } 209 | }, 210 | "ServiceRoleArn": { 211 | "Fn::If": [ 212 | "DefaultOWRolesCondition", 213 | { 214 | "Fn::Join": [ 215 | "", 216 | [ 217 | "arn:aws:iam::", 218 | { 219 | "Ref": "AWS::AccountId" 220 | }, 221 | ":role/aws-opsworks-service-role" 222 | ] 223 | ] 224 | }, 225 | { 226 | "Fn::GetAtt": [ 227 | "ServiceRole", 228 | "Arn" 229 | ] 230 | } 231 | ] 232 | }, 233 | "DefaultInstanceProfileArn": { 234 | "Fn::If": [ 235 | "DefaultOWRolesCondition", 236 | { 237 | "Fn::Join": [ 238 | "", 239 | [ 240 | "arn:aws:iam::", 241 | { 242 | "Ref": "AWS::AccountId" 243 | }, 244 | ":instance-profile/aws-opsworks-ec2-role" 245 | ] 246 | ] 247 | }, 248 | { 249 | "Fn::GetAtt": [ 250 | "InstanceRole", 251 | "Arn" 252 | ] 253 | } 254 | ] 255 | }, 256 | "UseCustomCookbooks": "true", 257 | "CustomCookbooksSource": { 258 | "Type": "archive", 259 | "Url": { 260 | "Ref": "S3Archive" 261 | } 262 | } 263 | } 264 | }, 265 | "elasticsearchtest": { 266 | "Type": "AWS::EC2::SecurityGroup", 267 | "Properties": { 268 | "GroupDescription": "so that ES cluster can find other nodes" 269 | } 270 | }, 271 | "searchLayer": { 272 | "Type": "AWS::OpsWorks::Layer", 273 | "Properties": { 274 | "StackId": { 275 | "Ref": "myStack" 276 | }, 277 | "Name": "Search", 278 | "Type": "custom", 279 | "Shortname": "search", 280 | "CustomRecipes": { 281 | "Setup": [ 282 | "apt", 283 | "ark", 284 | "elasticsearch", 285 | "elasticsearch::aws", 286 | "elasticsearch::proxy", 287 | "java", 288 | "layer-custom::esplugins", 289 | "layer-custom::allocation-awareness", 290 | "layer-custom::esmonit", 291 | "layer-custom::cloudwatch-custom" 292 | ] 293 | }, 294 | "EnableAutoHealing": "true", 295 | "AutoAssignElasticIps": "false", 296 | "AutoAssignPublicIps": "true", 297 | "VolumeConfigurations": [ 298 | { 299 | "MountPoint": "/mnt/elasticsearch-data", 300 | "NumberOfDisks": 1, 301 | "Size": 100 302 | } 303 | ], 304 | "CustomSecurityGroupIds": [ 305 | { 306 | "Fn::GetAtt": [ 307 | "elasticsearchtest", 308 | "GroupId" 309 | ] 310 | } 311 | ] 312 | } 313 | }, 314 | "myInstance1": { 315 | "Type": "AWS::OpsWorks::Instance", 316 | "Properties": { 317 | "StackId": { 318 | "Ref": "myStack" 319 | }, 320 | "LayerIds": [ 321 | { 322 | "Ref": "searchLayer" 323 | } 324 | ], 325 | "InstanceType": "m3.large" 326 | } 327 | }, 328 | "myInstance2": { 329 | "Type": "AWS::OpsWorks::Instance", 330 | "Properties": { 331 | "StackId": { 332 | "Ref": "myStack" 333 | }, 334 | "LayerIds": [ 335 | { 336 | "Ref": "searchLayer" 337 | } 338 | ], 339 | "InstanceType": "m3.large" 340 | } 341 | }, 342 | "myInstance3": { 343 | "Type": "AWS::OpsWorks::Instance", 344 | "Properties": { 345 | "StackId": { 346 | "Ref": "myStack" 347 | }, 348 | "LayerIds": [ 349 | { 350 | "Ref": "searchLayer" 351 | } 352 | ], 353 | "InstanceType": "m3.large" 354 | } 355 | } 356 | } 357 | } 358 | --------------------------------------------------------------------------------