├── .gitignore ├── LICENSE-2.0.txt ├── NOTICE.txt ├── README.md ├── fabfile.py ├── fabric-install.md ├── scripts └── oracle-java6.sh ├── solr-fabric-guide.md └── templates ├── solr-upstart.conf ├── zoo.cfg └── zookeeper-upstart.conf /.gitignore: -------------------------------------------------------------------------------- 1 | *.py[cod] 2 | 3 | # C extensions 4 | *.so 5 | 6 | # Packages 7 | *.egg 8 | *.egg-info 9 | dist 10 | build 11 | eggs 12 | parts 13 | bin 14 | var 15 | sdist 16 | develop-eggs 17 | .installed.cfg 18 | lib 19 | lib64 20 | 21 | # Installer logs 22 | pip-log.txt 23 | 24 | # Unit test / coverage reports 25 | .coverage 26 | .tox 27 | nosetests.xml 28 | 29 | # Translations 30 | *.mo 31 | 32 | # Mr Developer 33 | .mr.developer.cfg 34 | .project 35 | .pydevproject 36 | -------------------------------------------------------------------------------- /LICENSE-2.0.txt: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | -------------------------------------------------------------------------------- /NOTICE.txt: -------------------------------------------------------------------------------- 1 | Solr-Fabric 2 | Copyright 2013 LucidWorks, Inc. 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | solr-fabric 2 | =========== 3 | 4 | [Fabric](http://www.fabfile.org/) scripts for installing SolrCloud 5 | 6 | See [Solr Fabric Guide](./solr-fabric-guide.md) 7 | -------------------------------------------------------------------------------- /fabfile.py: -------------------------------------------------------------------------------- 1 | # 2 | # fabfile.py -- install SolrCloud with fabric 3 | # 4 | 5 | # See the NOTICE file distributed with this work for additional 6 | # information regarding copyright ownership. 7 | # 8 | # LucidWorks, Inc. licenses this file to you under the Apache License, 9 | # Version 2.0 (the "License"); you may not use this file except in 10 | # compliance with the License. You may obtain a copy of the License at 11 | # 12 | # http://www.apache.org/licenses/LICENSE-2.0 13 | # 14 | # Unless required by applicable law or agreed to in writing, software 15 | # distributed under the License is distributed on an "AS IS" BASIS, 16 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 17 | # implied. See the License for the specific language governing 18 | # permissions and limitations under the License. 19 | 20 | from fabric.api import * 21 | from fabric.contrib.files import * 22 | from fabric.contrib.project import * 23 | from sets import Set 24 | from fabric.task_utils import merge 25 | import time, os 26 | 27 | # define roles for our cluster: 3 zookeeper nodes and 4 solr nodes 28 | env.roledefs.update({ 29 | 'zookeeper': [ 'vm110', 'vm111', 'vm112' ], 30 | 'solr': [ 'vm110', 'vm111', 'vm112', 'vm113' ], 31 | }) 32 | 33 | # If you have a different username on the nodes, set it here: 34 | #env.user = "ubuntu" 35 | 36 | # URLs to download ZooKeeper and Solr from 37 | env.zookeeper_url = 'http://www.mirrorservice.org/sites/ftp.apache.org/zookeeper/zookeeper-3.4.5/zookeeper-3.4.5.tar.gz' 38 | env.solr_url = 'http://archive.apache.org/dist/lucene/solr/4.4.0/solr-4.4.0.tgz' 39 | 40 | ### You don't need to change anything below here 41 | 42 | env.always_use_pty = False 43 | env.forward_agent = True 44 | env.use_ssh_config = True 45 | 46 | # Name of the directory on the nodes where we install our components, 47 | # relative to the remote user directory 48 | MY_DIR_NAME = 'solr-fabric' 49 | 50 | env.num_shards = 2 51 | 52 | # prepare local filenames for distribution 53 | env.zookeeper_tgz = os.path.basename(env.zookeeper_url) 54 | env.solr_tgz = os.path.basename(env.solr_url) 55 | 56 | # prepare remote directory names for extracted distributions 57 | env.my_dir_path = os.path.expanduser("/home/{0}/{1}".format(env.user, MY_DIR_NAME)) 58 | env.zookeeper_dir = os.path.join(env.my_dir_path, re.sub(r'\.tar\.gz$', '', env.zookeeper_tgz)) 59 | env.solr_dir = os.path.join(env.my_dir_path, re.sub(r'\.tgz$', '', env.solr_tgz)) 60 | 61 | # names for upstart services. Use a prefix to prevent accidentally overwriting a system package 62 | env.zookeeper_service = 'my_zookeeper' 63 | env.solr_service = 'my_solr' 64 | env.solr_home = 'solr' #If you plan to use multicore. Run: fab stop_solr, fab bootstrap_multicore_solrcloud and fab start_solr 65 | env.solr_port = 8983 #Note if you modfiy this, please update example/etc/jetty.xml 66 | 67 | # create an 'all' role containing all hosts 68 | env.roledefs.update({ 'all': merge([], ['zookeeper', 'solr'], [], env.roledefs) }) 69 | 70 | env.first_solr = env.roledefs['solr'][0] 71 | env.first_zookeeper = env.roledefs['zookeeper'][0] 72 | 73 | # local jinja2 templates directory 74 | TEMPLATES = "./templates" 75 | 76 | @roles('all') 77 | def test_ssh(): 78 | """ Run 'hostname' on all hosts, to test ssh. """ 79 | run("hostname") 80 | 81 | @roles('all') 82 | def test_ping(): 83 | """ Run 'ping' on all hosts, to test hostname resolution. """ 84 | for host in env.roledefs['all']: 85 | run("ping -c 1 {0}".format(host)) 86 | 87 | @roles('all') 88 | def copy_ssh_key(ssh_pub_key = "~/.ssh/id_dsa.pub"): 89 | """ Append a public key to .ssh/authorized_keys on the nodes, for password-less ssh. """ 90 | ssh_pub_key_path = os.path.expanduser(ssh_pub_key) 91 | remote = "solr-fabric-key.pem" 92 | put(ssh_pub_key_path, remote) 93 | run("mkdir -p ~/.ssh") 94 | run("cat {0} >> ~/.ssh/authorized_keys".format(remote)) 95 | run("rm {0}".format(remote)) 96 | 97 | @roles('all') 98 | def setup_sudoers(): 99 | """ Add the user to sudoers, allowing password-less sudo. """ 100 | append("/etc/sudoers", "{0} ALL=(ALL) NOPASSWD:ALL".format(env.user), use_sudo=True) 101 | 102 | @roles('all') 103 | def create_my_dir(): 104 | """ Create a directory on the nodes to hold our ZooKeeper and Solr installs. """ 105 | run("mkdir -p {0}".format(env.my_dir_path)) 106 | 107 | @roles('all') 108 | def install_oracle_java(): 109 | """ Install Oracle java. """ 110 | # This runs on all the individual nodes. That is a little slow and 111 | # wasteful, as it does repeated download from Oracle. We could try 112 | # to download once from Oracle to the laptop, but that adds more 113 | # complexity and depends on what OS you run on. We could speed 114 | # things up with a fabric @parallel decorator, but then the output 115 | # becomes more confusing, which is not ideal for a tutorial. 116 | # Really, you want to install Java in your VM base image so you 117 | # can skip all this. 118 | execute("create_my_dir") 119 | script = os.path.join(env.my_dir_path, "oracle-java6.sh") 120 | put("scripts/oracle-java6.sh", script) 121 | sudo("bash -x {0}".format(script)) 122 | 123 | @roles('all') 124 | def java_version(): 125 | """ Print the Java version. """ 126 | run("java -version") 127 | 128 | def download_zookeeper(): 129 | """ Download ZooKeeper to the local directory. """ 130 | if os.path.exists(env.zookeeper_tgz): 131 | puts("{0} already exists".format(env.zookeeper_tgz)) 132 | return 133 | local("wget {0}".format(env.zookeeper_url)) 134 | 135 | def download_solr(): 136 | """ Download Solr to the local directory. """ 137 | if os.path.exists(env.solr_tgz): 138 | puts("{0} already exists".format(env.solr_tgz)) 139 | return 140 | local("wget {0}".format(env.solr_url)) 141 | 142 | @roles('zookeeper') 143 | def copy_zookeeper(): 144 | """ Upload ZooKeeper to the nodes. """ 145 | put(env.zookeeper_tgz, os.path.join(env.my_dir_path, env.zookeeper_tgz)) 146 | 147 | @roles('solr') 148 | def copy_solr(): 149 | """ Upload Solr to the nodes. """ 150 | put(env.solr_tgz, os.path.join(env.my_dir_path, env.solr_tgz)) 151 | 152 | @roles('zookeeper') 153 | def extract_zookeeper(): 154 | """ Extract ZooKeeper """ 155 | with cd(env.my_dir_path): 156 | run("tar xf {0}".format(env.zookeeper_tgz)) 157 | 158 | def configure_zookeeper_id(zk_id): 159 | """ Write the zookeeper node id to the my_id file. """ 160 | with cd(env.zookeeper_dir): 161 | run("mkdir -p data") 162 | run("echo {0} > data/myid".format(zk_id)) 163 | context = { "hosts": env.roledefs['zookeeper'], "path": env.zookeeper_dir } 164 | upload_template(filename='zoo.cfg', destination='conf/', template_dir=TEMPLATES, context=context, use_jinja=True) 165 | 166 | @roles('zookeeper') 167 | def upstart_zookeeper(): 168 | """ Write an Upstart script for ZooKeeper. """ 169 | context = { "user": env.user, "group": env.user, "path": env.zookeeper_dir } 170 | upload_template(filename='zookeeper-upstart.conf', destination='/etc/init/{0}.conf'.format(env.zookeeper_service), 171 | template_dir=TEMPLATES, context=context, use_sudo=True, use_jinja=True) 172 | 173 | @roles('zookeeper') 174 | def zookeeper_upstart_log(): 175 | """ Tail the Upstart log for ZooKeeper. """ 176 | sudo("tail /var/log/upstart/{0}.log".format(env.zookeeper_service)) 177 | 178 | @roles('zookeeper') 179 | def check_zookeeper(): 180 | """ Ask ZooKeeper if it is ok, and report its leader/follower mode. """ 181 | out = run("echo ruok | nc localhost 2181") 182 | if not "imok" in out: 183 | abort("zookeeper not ok") 184 | run("echo stat | nc localhost 2181 | grep Mode:") 185 | 186 | @hosts(env.first_zookeeper) 187 | def show_zookeeper(): 188 | """ Show zookeeper content. """ 189 | with cd(env.zookeeper_dir): 190 | run("echo ls / | ./bin/zkCli.sh") 191 | run("echo ls /live_nodes | ./bin/zkCli.sh") 192 | run("echo get /overseer | ./bin/zkCli.sh") 193 | 194 | def wait_for_zookeeper(): 195 | """ Wait for ZooKeeper to come up and elect a leader. """ 196 | sleep_seconds = 3 197 | while True: 198 | complete = True 199 | for host in env.roledefs['zookeeper']: 200 | with settings(host_string=host): 201 | mode = run("echo stat | nc localhost 2181 | grep Mode:", warn_only=True) 202 | if not ("Mode: follower" in mode or "Mode: leader" in mode): 203 | complete = False 204 | break 205 | if complete: 206 | puts("got a leader, and all nodes are up") 207 | return 208 | else: 209 | puts("zookeeper cluster not complete yet; sleeping {0} seconds".format(sleep_seconds)) 210 | time.sleep(sleep_seconds) 211 | 212 | def configure_zookeeper(): 213 | """ Configure ZooKeeper. """ 214 | zk_id = 0 215 | for host in env.roledefs['zookeeper']: 216 | zk_id = zk_id + 1 217 | execute(configure_zookeeper_id, zk_id, hosts=[host]) 218 | 219 | @roles('solr') 220 | def extract_solr(): 221 | """ Extract Solr. """ 222 | with cd(env.my_dir_path): 223 | run("tar xf {0}".format(env.solr_tgz)) 224 | 225 | def zookeeper_hostports(): 226 | """ Return a comma-separated list of ZooKeeper hostname:port pairs. """ 227 | return ",".join([ "{0}:2181".format(host) for host in env.roledefs['zookeeper'] ]) 228 | 229 | @hosts(env.first_solr) 230 | def bootstrap_solrcloud(): 231 | """ Bootstrap SolrCloud. """ 232 | # See http://docs.lucidworks.com/display/solr/Command+Line+Utilities 233 | zkhost = "{0}:2181".format(env.first_zookeeper) 234 | collection = "collection1" 235 | conf_set = "configuration1" 236 | with cd(os.path.join(env.solr_dir, "example")): 237 | # jetty has not run yet, so the webapp has not been extracted; do it here ourselves 238 | run("mkdir solr-webapp-tmp; (cd solr-webapp-tmp; jar xvf ../webapps/solr.war)") 239 | zk_cli = "java -classpath solr-webapp-tmp/WEB-INF/lib/*:./lib/ext/* org.apache.solr.cloud.ZkCLI" 240 | run("{0} -cmd upconfig -zkhost {1} -d solr/{2}/conf/ -n {3}".format(zk_cli, zkhost, collection, conf_set)) 241 | run("{0} -cmd linkconfig -zkhost {1} -collection {2} -confname {3} -solrhome {4}".format(zk_cli, zkhost, collection, conf_set, env.solr_home)) 242 | run("{0} -cmd bootstrap -zkhost {1} -solrhome {2}".format(zk_cli, zkhost, env.solr_home)) 243 | run("{0} -cmd upconfig -zkhost {1} -d solr/{2}/conf/ -n {3}".format(zk_cli, zkhost, collection, conf_set)) 244 | run("rm -fr solr-webapp-tmp") 245 | 246 | @hosts(env.first_solr) 247 | def bootstrap_multicore_solrcloud(): 248 | """ Bootstrap multicore SolrCloud. """ 249 | # See http://docs.lucidworks.com/display/solr/Command+Line+Utilities 250 | zkhost = "{0}:2181".format(env.first_zookeeper) 251 | with cd(os.path.join(env.solr_dir, "example")): 252 | # jetty has not run yet, so the webapp has not been extracted; do it here ourselves 253 | run("mkdir solr-webapp-tmp; (cd solr-webapp-tmp; jar xvf ../webapps/solr.war)") 254 | zk_cli = "java -classpath solr-webapp-tmp/WEB-INF/lib/*:./lib/ext/* org.apache.solr.cloud.ZkCLI" 255 | run("{0} -cmd bootstrap -zkhost {1} -solrhome {2}".format(zk_cli, zkhost, env.solr_home)) 256 | run("rm -fr solr-webapp-tmp") 257 | 258 | def solrcloud_url(): 259 | """ Print a URL for the Solr Admin interface. """ 260 | puts("http://{0}:{1}/solr/#/~cloud".format(env.first_solr, env.solr_port)) 261 | 262 | def wait_for_port(port, max_wait=60, interval=5): 263 | """ Wait for a TCP port to be listened to. """ 264 | while True: 265 | started = time.time() 266 | status = run("netstat --listen --numeric | grep ':{0} ' || echo no".format(port)) 267 | if status != "no": 268 | return 269 | delta = time.time() - started 270 | if delta > max_wait: 271 | raise Exception("port {0} still not listening after {1} seconds".format(port, delta)) 272 | time.sleep(interval) 273 | 274 | def wait_for_solr(): 275 | """ Wait for Solr nodes to come up. """ 276 | execute('wait_for_solr_ports') 277 | execute('wait_for_solr_in_zookeeper') 278 | execute('solr_clusterstate') 279 | 280 | @roles('solr') 281 | def wait_for_solr_ports(): 282 | """ Wait for Solr ports to be listened on. """ 283 | status = sudo("service {0} status".format(env.solr_service)) 284 | if "running" not in status: 285 | abort("solr not running") 286 | execute('wait_for_port', env.solr_port) 287 | 288 | def wait_for_solr_in_zookeeper(): 289 | """ Wait for Solr data to appear in ZooKeeper. """ 290 | sleep_seconds = 3 291 | # look in zookeeper for the nodes 292 | while True: 293 | complete = True 294 | with(settings(host_string=env.first_zookeeper)): 295 | with cd(env.zookeeper_dir): 296 | result = run("echo get /live_nodes | ./bin/zkCli.sh", warn_only=True) 297 | if not "numChildren = {0}".format(len(env.roledefs['solr'])) in result: 298 | complete = False 299 | if complete: 300 | puts("got all nodes in zookeeper") 301 | break 302 | else: 303 | puts("not complete yet; sleeping {0} seconds".format(sleep_seconds)) 304 | time.sleep(sleep_seconds) 305 | 306 | @hosts(env.first_zookeeper) 307 | def solr_clusterstate(): 308 | """ Display cluster state. """ 309 | with cd(env.zookeeper_dir): 310 | run("echo get /clusterstate.json | ./bin/zkCli.sh", warn_only=True) 311 | 312 | @roles('solr') 313 | def upstart_solr(): 314 | """ Write an Upstart script for Solr. """ 315 | context = { "host": env.host, "user": env.user, "group": env.user, "path": os.path.join(env.solr_dir, "example"), 316 | "num_shards": env.num_shards, "zookeeper_hostports": zookeeper_hostports(), 'solr_home': env.solr_home} 317 | upload_template(filename='solr-upstart.conf', destination='/etc/init/{0}.conf'.format(env.solr_service), 318 | template_dir=TEMPLATES, context=context, use_sudo=True, use_jinja=True) 319 | 320 | @roles('zookeeper') 321 | def start_zookeeper(): 322 | """ Start ZooKeeper. """ 323 | sudo("service {0} start".format(env.zookeeper_service)) 324 | 325 | @roles('zookeeper') 326 | def stop_zookeeper(): 327 | """ Stop ZooKeeper. """ 328 | sudo("service {0} stop".format(env.zookeeper_service)) 329 | 330 | @roles('solr') 331 | def start_solr(): 332 | """ Start Solr. """ 333 | if not "running" in sudo("service {0} status".format(env.solr_service)): 334 | sudo("service {0} start".format(env.solr_service)) 335 | 336 | @roles('solr') 337 | def stop_solr(): 338 | """ Stop Solr. """ 339 | sudo("service {0} stop".format(env.solr_service)) 340 | 341 | @roles('solr') 342 | def solr_upstart_log(): 343 | """ Tail the Upstart log for Solr. """ 344 | sudo("tail /var/log/upstart/{0}.log".format(env.solr_service)) 345 | 346 | @roles('solr') 347 | def solr_status(): 348 | """ Report the service status for Solr, and print the Solr cores status. """ 349 | sudo("service {0} status".format(env.solr_service)) 350 | run("wget -O - http://localhost:{0}/solr/admin/cores?action=STATUS".format(env.solr_port)) 351 | 352 | 353 | ### Below here are top-level tasks 354 | 355 | def download(): 356 | """ Download ZooKeeper and Solr. """ 357 | execute('download_zookeeper') 358 | execute('download_solr') 359 | 360 | def install_zookeeper(): 361 | """ Install the ZooKeeper nodes. """ 362 | execute('download_zookeeper') 363 | execute('copy_zookeeper') 364 | execute('extract_zookeeper') 365 | execute('configure_zookeeper') 366 | execute('upstart_zookeeper') 367 | execute('start_zookeeper') 368 | 369 | def install_solr(): 370 | """ Install the Solr nodes. """ 371 | execute('download_solr') 372 | execute('copy_solr') 373 | execute('extract_solr') 374 | execute('bootstrap_solrcloud') 375 | execute('upstart_solr') 376 | execute('start_solr') 377 | 378 | def everything(): 379 | """ Do everything. """ 380 | execute('test_ssh') 381 | execute('copy_ssh_key') 382 | execute('setup_sudoers') 383 | execute('create_my_dir') 384 | execute('install_oracle_java') 385 | execute('java_version') 386 | execute('install_solr_and_zookeeper') 387 | execute('sample_data') 388 | execute('sample_query') 389 | execute('display_status') 390 | 391 | def install_solr_and_zookeeper(): 392 | """ Install ZooKeeper and Solr. """ 393 | execute('create_my_dir') 394 | execute('download') 395 | execute('install_zookeeper') 396 | execute('wait_for_zookeeper') 397 | execute('install_solr') 398 | execute('wait_for_solr') 399 | 400 | @hosts(env.first_solr) 401 | def sample_data(): 402 | """ Load the "books" sample data. """ 403 | with cd(os.path.join(env.solr_dir, "example/exampledocs")): 404 | run("curl -sS 'http://{0}:{1}/solr/update/json?commit=true' --data-binary @books.json -H 'Content-type:application/json'".format(env.first_solr, env.solr_port)) 405 | 406 | @hosts(env.first_solr) 407 | def sample_query(): 408 | """ Do a query. """ 409 | local("curl -sS 'http://{0}:{1}/solr/select?q=name:monsters&wt=json&indent=true'".format(env.first_solr, env.solr_port)) 410 | 411 | @hosts(env.first_solr) 412 | def sample_query_all(): 413 | """ Do a query for all documents. """ 414 | local("curl -sS 'http://{0}:{1}/solr/select?q=*:*&wt=json&indent=true'".format(env.first_solr, env.solr_port)) 415 | 416 | @roles('solr') 417 | def sample_query_all_distrib_false(): 418 | """ Do a query for all documents, against each node, with distrib=false. """ 419 | run("curl -sS 'http://localhost:{0}/solr/select?q=*:*&wt=json&indent=true&distrib=false'".format(env.solr_port)) 420 | 421 | @roles('solr') 422 | def display_status(): 423 | """ Show Solr core status. """ 424 | run("""curl -sS "http://localhost:{0}/solr/admin/cores?action=STATUS&indent=true&wt=json" """.format(env.solr_port)) 425 | 426 | @roles('zookeeper') 427 | def uninstall_zookeeper_upstart(): 428 | """ Uninstall ZooKeeper Upstart script. """ 429 | sudo("rm -f /etc/init/{0}.conf".format(env.zookeeper_service)) 430 | 431 | @roles('solr') 432 | def uninstall_solr_upstart(): 433 | """ Uninstall Solr Upstart script. """ 434 | sudo("rm -f /etc/init/{0}.conf".format(env.solr_service)) 435 | 436 | @roles('all') 437 | def uninstall_mydir(): 438 | """ Remove our directory on the nodes. """ 439 | run("rm -fr {0}".format(env.my_dir_path)) 440 | 441 | @roles('all') 442 | def uninstall(): 443 | """ Uninstall ZooKeeper and Solr. """ 444 | execute('uninstall_solr_upstart') 445 | execute('uninstall_zookeeper_upstart') 446 | execute('uninstall_mydir') 447 | -------------------------------------------------------------------------------- /fabric-install.md: -------------------------------------------------------------------------------- 1 | Installing Fabric and its dependencies 2 | ====================================== 3 | 4 | [Fabric](http://www.fabfile.org/) itself takes a little effort to setup, but it's worth it. 5 | 6 | The Fabric community is quite active, and we will use the latest version. 7 | 8 | We will run Fabric from Python's 9 | [virtualenv](http://www.virtualenv.org/en/latest/), which isolates the installation in a 10 | virtual environment that is easy to remove and re-install. 11 | 12 | If you do not already have virtualenv installed, do that first: 13 | 14 | #### On Ubuntu 15 | 16 | sudo apt-get install python-virtualenv 17 | 18 | and use the (outdated) Ubuntu fabric package to install Fabric's dependencies (such as compiler and python header files required for the crypto library): 19 | 20 | apt-cache depends fabric | grep Depends: | sed 's/^ *Depends: //' | xargs sudo apt-get ---yes install 21 | sudo apt-get --yes install python-dev 22 | 23 | 24 | ### On OSX 25 | 26 | You need a system compiler. If you have a developer.apple.com account you can download and 27 | install the command-line tools separately; if not, download Xcode from the App Store, and install the 28 | "Command Line Tools" component in the Downloads pane in Preferences. 29 | 30 | You need a recent Python, pip, and virtual_env. 31 | I strongly recommend [homebrew](http://brew.sh), 32 | so you can install a fresh python and virtualenv with: 33 | 34 | brew install python 35 | sudo pip install virtualenv 36 | 37 | ### Installing Fabric 38 | 39 | Now we can create a virtualenv to install fabric into: 40 | 41 | VIRTUAL_ENV_DIR=$HOME/fabric 42 | virtualenv "$VIRTUAL_ENV_DIR" 43 | source "$VIRTUAL_ENV_DIR/bin/activate" 44 | 45 | This will change your bash prompt to include a "(fabric)" prefix. 46 | Whenever you want to run fabric, just re-activate your virtualenv with that source command. 47 | 48 | And, finally, we're ready to actually install Fabric: 49 | 50 | pip install fabric jinja2 51 | 52 | To verify you can run it, invoke it to print out its versions: 53 | 54 | (fabric)mak@crab$ fab -V 55 | Fabric 1.5.1 56 | Paramiko 1.9.0 57 | 58 | Yay! 59 | -------------------------------------------------------------------------------- /scripts/oracle-java6.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # Install Oracle Java6 on Ubuntu 4 | # Based on http://www.webupd8.org/2012/01/install-oracle-java-jdk-7-in-ubuntu-via.html 5 | 6 | set -e 7 | 8 | function install_oracle_java() { 9 | export DEBIAN_FRONTEND=noninteractive 10 | 11 | apt-get --yes purge openjdk* 12 | 13 | which add-apt-repository || apt-get --yes install python-software-properties software-properties-common 14 | 15 | # to skip the license screen: 16 | /usr/bin/debconf-set-selections <> /etc/environment" 34 | fi 35 | 36 | if ! grep -q 'env_keep+=JAVA_HOME' /etc/sudoers; then 37 | ed /etc/sudoers <