├── LICENSE ├── README.md ├── base └── Dockerfile ├── client.sh ├── docker-compose.yml ├── rebuild.sh ├── refresh.sh ├── storm-nimbus ├── Dockerfile └── start-supervisor.sh ├── storm-supervisor └── Dockerfile ├── storm-ui └── Dockerfile ├── storm ├── Dockerfile ├── cluster.xml ├── config-supervisord.sh ├── start-supervisor.sh └── storm.yaml └── zookeeper ├── Dockerfile └── start-zk.sh /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, and 10 | distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by the copyright 13 | owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all other entities 16 | that control, are controlled by, or are under common control with that entity. 17 | For the purposes of this definition, "control" means (i) the power, direct or 18 | indirect, to cause the direction or management of such entity, whether by 19 | contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the 20 | outstanding shares, or (iii) beneficial ownership of such entity. 21 | 22 | "You" (or "Your") shall mean an individual or Legal Entity exercising 23 | permissions granted by this License. 24 | 25 | "Source" form shall mean the preferred form for making modifications, including 26 | but not limited to software source code, documentation source, and configuration 27 | files. 28 | 29 | "Object" form shall mean any form resulting from mechanical transformation or 30 | translation of a Source form, including but not limited to compiled object code, 31 | generated documentation, and conversions to other media types. 32 | 33 | "Work" shall mean the work of authorship, whether in Source or Object form, made 34 | available under the License, as indicated by a copyright notice that is included 35 | in or attached to the work (an example is provided in the Appendix below). 36 | 37 | "Derivative Works" shall mean any work, whether in Source or Object form, that 38 | is based on (or derived from) the Work and for which the editorial revisions, 39 | annotations, elaborations, or other modifications represent, as a whole, an 40 | original work of authorship. For the purposes of this License, Derivative Works 41 | shall not include works that remain separable from, or merely link (or bind by 42 | name) to the interfaces of, the Work and Derivative Works thereof. 43 | 44 | "Contribution" shall mean any work of authorship, including the original version 45 | of the Work and any modifications or additions to that Work or Derivative Works 46 | thereof, that is intentionally submitted to Licensor for inclusion in the Work 47 | by the copyright owner or by an individual or Legal Entity authorized to submit 48 | on behalf of the copyright owner. For the purposes of this definition, 49 | "submitted" means any form of electronic, verbal, or written communication sent 50 | to the Licensor or its representatives, including but not limited to 51 | communication on electronic mailing lists, source code control systems, and 52 | issue tracking systems that are managed by, or on behalf of, the Licensor for 53 | the purpose of discussing and improving the Work, but excluding communication 54 | that is conspicuously marked or otherwise designated in writing by the copyright 55 | owner as "Not a Contribution." 56 | 57 | "Contributor" shall mean Licensor and any individual or Legal Entity on behalf 58 | of whom a Contribution has been received by Licensor and subsequently 59 | incorporated within the Work. 60 | 61 | 2. Grant of Copyright License. 62 | 63 | Subject to the terms and conditions of this License, each Contributor hereby 64 | grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, 65 | irrevocable copyright license to reproduce, prepare Derivative Works of, 66 | publicly display, publicly perform, sublicense, and distribute the Work and such 67 | Derivative Works in Source or Object form. 68 | 69 | 3. Grant of Patent License. 70 | 71 | Subject to the terms and conditions of this License, each Contributor hereby 72 | grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, 73 | irrevocable (except as stated in this section) patent license to make, have 74 | made, use, offer to sell, sell, import, and otherwise transfer the Work, where 75 | such license applies only to those patent claims licensable by such Contributor 76 | that are necessarily infringed by their Contribution(s) alone or by combination 77 | of their Contribution(s) with the Work to which such Contribution(s) was 78 | submitted. If You institute patent litigation against any entity (including a 79 | cross-claim or counterclaim in a lawsuit) alleging that the Work or a 80 | Contribution incorporated within the Work constitutes direct or contributory 81 | patent infringement, then any patent licenses granted to You under this License 82 | for that Work shall terminate as of the date such litigation is filed. 83 | 84 | 4. Redistribution. 85 | 86 | You may reproduce and distribute copies of the Work or Derivative Works thereof 87 | in any medium, with or without modifications, and in Source or Object form, 88 | provided that You meet the following conditions: 89 | 90 | You must give any other recipients of the Work or Derivative Works a copy of 91 | this License; and 92 | You must cause any modified files to carry prominent notices stating that You 93 | changed the files; and 94 | You must retain, in the Source form of any Derivative Works that You distribute, 95 | all copyright, patent, trademark, and attribution notices from the Source form 96 | of the Work, excluding those notices that do not pertain to any part of the 97 | Derivative Works; and 98 | If the Work includes a "NOTICE" text file as part of its distribution, then any 99 | Derivative Works that You distribute must include a readable copy of the 100 | attribution notices contained within such NOTICE file, excluding those notices 101 | that do not pertain to any part of the Derivative Works, in at least one of the 102 | following places: within a NOTICE text file distributed as part of the 103 | Derivative Works; within the Source form or documentation, if provided along 104 | with the Derivative Works; or, within a display generated by the Derivative 105 | Works, if and wherever such third-party notices normally appear. The contents of 106 | the NOTICE file are for informational purposes only and do not modify the 107 | License. You may add Your own attribution notices within Derivative Works that 108 | You distribute, alongside or as an addendum to the NOTICE text from the Work, 109 | provided that such additional attribution notices cannot be construed as 110 | modifying the License. 111 | You may add Your own copyright statement to Your modifications and may provide 112 | additional or different license terms and conditions for use, reproduction, or 113 | distribution of Your modifications, or for any such Derivative Works as a whole, 114 | provided Your use, reproduction, and distribution of the Work otherwise complies 115 | with the conditions stated in this License. 116 | 117 | 5. Submission of Contributions. 118 | 119 | Unless You explicitly state otherwise, any Contribution intentionally submitted 120 | for inclusion in the Work by You to the Licensor shall be under the terms and 121 | conditions of this License, without any additional terms or conditions. 122 | Notwithstanding the above, nothing herein shall supersede or modify the terms of 123 | any separate license agreement you may have executed with Licensor regarding 124 | such Contributions. 125 | 126 | 6. Trademarks. 127 | 128 | This License does not grant permission to use the trade names, trademarks, 129 | service marks, or product names of the Licensor, except as required for 130 | reasonable and customary use in describing the origin of the Work and 131 | reproducing the content of the NOTICE file. 132 | 133 | 7. Disclaimer of Warranty. 134 | 135 | Unless required by applicable law or agreed to in writing, Licensor provides the 136 | Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, 137 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, 138 | including, without limitation, any warranties or conditions of TITLE, 139 | NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are 140 | solely responsible for determining the appropriateness of using or 141 | redistributing the Work and assume any risks associated with Your exercise of 142 | permissions under this License. 143 | 144 | 8. Limitation of Liability. 145 | 146 | In no event and under no legal theory, whether in tort (including negligence), 147 | contract, or otherwise, unless required by applicable law (such as deliberate 148 | and grossly negligent acts) or agreed to in writing, shall any Contributor be 149 | liable to You for damages, including any direct, indirect, special, incidental, 150 | or consequential damages of any character arising as a result of this License or 151 | out of the use or inability to use the Work (including but not limited to 152 | damages for loss of goodwill, work stoppage, computer failure or malfunction, or 153 | any and all other commercial damages or losses), even if such Contributor has 154 | been advised of the possibility of such damages. 155 | 156 | 9. Accepting Warranty or Additional Liability. 157 | 158 | While redistributing the Work or Derivative Works thereof, You may choose to 159 | offer, and charge a fee for, acceptance of support, warranty, indemnity, or 160 | other liability obligations and/or rights consistent with this License. However, 161 | in accepting such obligations, You may act only on Your own behalf and on Your 162 | sole responsibility, not on behalf of any other Contributor, and only if You 163 | agree to indemnify, defend, and hold each Contributor harmless for any liability 164 | incurred by, or claims asserted against, such Contributor by reason of your 165 | accepting any such warranty or additional liability. 166 | 167 | END OF TERMS AND CONDITIONS 168 | 169 | Copyright 2016 Carey Tzou 170 | 171 | Licensed under the Apache License, Version 2.0 (the "License"); 172 | you may not use this file except in compliance with the License. 173 | You may obtain a copy of the License at 174 | 175 | http://www.apache.org/licenses/LICENSE-2.0 176 | 177 | Unless required by applicable law or agreed to in writing, software 178 | distributed under the License is distributed on an "AS IS" BASIS, 179 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 180 | See the License for the specific language governing permissions and 181 | limitations under the License. 182 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | storm-mono-docker 2 | ============ 3 | 4 | Dockerfiles for building a storm cluster. Inspired by [https://github.com/ptgoetz/storm-vagrant](https://github.com/ptgoetz/storm-vagrant) 5 | 6 | The images are available directly from [https://index.docker.io](https://index.docker.io) 7 | 8 | ## Pre-Requisites 9 | 10 | - install docker-compose [http://docs.docker.com/compose/install/](http://docs.docker.com/compose/install/) 11 | 12 | ## Usage 13 | 14 | Start a cluster: 15 | 16 | - ```docker-compose up -d``` 17 | 18 | - The ```docker-compose up``` command aggregates the output of each container. When the command exits, all containers are stopped. Running ```docker-compose up -d``` starts the containers in the background and leaves them running. 19 | 20 | Destroy a cluster: 21 | 22 | - ```docker-compose stop``` 23 | 24 | Add more supervisors: 25 | 26 | - ```docker-compose scale supervisor=4``` 27 | 28 | Kill and rm all docker cache: 29 | 30 | - ```docker kill $(docker ps -a -f "name=stormmonodocker" -q) ; docker rm $(docker ps -a -f "name=stormmonodocker" -q)``` 31 | 32 | Remove all docker image: 33 | 34 | - ```docker rmi $(docker images | grep "^ziyunhx" | awk "{print $3}")``` 35 | 36 | ## Building 37 | 38 | - ```rebuild.sh``` 39 | 40 | ## FAQ 41 | ### How can I access Storm UI from my host? 42 | Take a look at docker-compose.yml: 43 | 44 | ui: 45 | image: ziyunhx/storm-ui 46 | ports: 47 | - "49080:8080" 48 | 49 | This tells Docker to expose the Docker UI container's port 8080 as port 49080 on the host
50 | 51 | If you are running docker natively you can use localhost. If you're using boot2docker, then do: 52 | 53 | $ boot2docker ip 54 | The VM's Host only interface IP address is: 192.168.59.103 55 | 56 | Which returns your docker VM's IP.
57 | So, to open storm UI, type the following in your browser: 58 | 59 | localhost:49080 60 | 61 | or 62 | 63 | 192.168.59.103:49080 64 | 65 | in my case. 66 | 67 | ### How can I deploy a topology? 68 | Since the nimbus host and port are not default, you need to specify where the nimbus host is, and what is the nimbus port number.
69 | Following the example above, after discovering the nimbus host IP (could be localhost, could be our docker VM ip as in the case of boot2docker), run the following command: 70 | 71 | storm jar target/your-topology-fat-jar.jar com.your.package.AndTopology topology-name -c nimbus.host=192.168.59.103 -c nimbus.thrift.port=49627 72 | 73 | If the above method is not successful, please change the storm.yaml file like this: 74 | 75 | nimbus.seeds: ["192.168.59.103"] 76 | nimbus.thrift.port: 49627 77 | 78 | And run the following command: 79 | 80 | storm jar target/your-topology-fat-jar.jar com.your.package.AndTopology topology-name 81 | 82 | ### How can I connect to one of the containers? 83 | Find the forwarded ssh port for the container you wish to connect to (use `docker-compose ps`) 84 | 85 | $ ssh root@`boot2docker ip` -p $CONTAINER_PORT 86 | 87 | The password is 'ziyunhxpass' (from: https://registry.hub.docker.com/u/ziyunhx/base/dockerfile/). 88 | -------------------------------------------------------------------------------- /base/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM ubuntu:trusty 2 | 3 | RUN apt-get update; apt-get install -y unzip openjdk-7-jre-headless wget supervisor docker.io openssh-server 4 | ENV JAVA_HOME /usr/lib/jvm/java-7-openjdk-amd64/ 5 | RUN echo 'root:ziyunhxpass' | chpasswd 6 | RUN mkdir /var/run/sshd 7 | RUN sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config 8 | RUN cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime 9 | 10 | EXPOSE 22 11 | -------------------------------------------------------------------------------- /client.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | docker run -i -t --link stormdocker_nimbus_1:nimbus -p 22 ziyunhx/storm /bin/bash -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | zookeeper: 2 | image: ziyunhx/zookeeper 3 | ports: 4 | - "49181:2181" 5 | - "22" 6 | nimbus: 7 | image: ziyunhx/storm-nimbus:1.2.2 8 | ports: 9 | - "49773:3773" 10 | - "49772:3772" 11 | - "49627:6627" 12 | - "22" 13 | links: 14 | - zookeeper:zk 15 | supervisor: 16 | image: ziyunhx/storm-supervisor:1.2.2 17 | ports: 18 | - "8000" 19 | - "22" 20 | links: 21 | - nimbus:nimbus 22 | - zookeeper:zk 23 | ui: 24 | image: ziyunhx/storm-ui:1.2.2 25 | ports: 26 | - "49080:8080" 27 | - "22" 28 | links: 29 | - nimbus:nimbus 30 | - zookeeper:zk 31 | -------------------------------------------------------------------------------- /rebuild.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | docker build -t="ziyunhx/zookeeper" zookeeper 4 | docker build -t="ziyunhx/storm-nimbus" storm-nimbus 5 | docker build -t="ziyunhx/storm-supervisor" storm-supervisor 6 | docker build -t="ziyunhx/storm-ui" storm-ui -------------------------------------------------------------------------------- /refresh.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | docker pull ziyunhx/zookeeper 4 | docker pull ziyunhx/storm-nimbus 5 | docker pull ziyunhx/storm-supervisor 6 | docker pull ziyunhx/storm-ui -------------------------------------------------------------------------------- /storm-nimbus/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM ziyunhx/storm:1.2.2 2 | 3 | RUN /usr/bin/config-supervisord.sh nimbus 4 | RUN /usr/bin/config-supervisord.sh drpc 5 | 6 | EXPOSE 6627 7 | EXPOSE 3772 8 | EXPOSE 3773 9 | ADD start-supervisor.sh /usr/bin/start-supervisor.sh 10 | CMD /usr/bin/start-supervisor.sh 11 | -------------------------------------------------------------------------------- /storm-nimbus/start-supervisor.sh: -------------------------------------------------------------------------------- 1 | export IP=`hostname -i` 2 | sed -i -e "s/%zookeeper%/$ZK_PORT_2181_TCP_ADDR/g" $STORM_HOME/conf/storm.yaml 3 | sed -i -e "s/%nimbus%/$IP/g" $STORM_HOME/conf/storm.yaml 4 | 5 | echo "storm.local.hostname: `hostname -i`" >> $STORM_HOME/conf/storm.yaml 6 | 7 | /usr/sbin/sshd && supervisord 8 | -------------------------------------------------------------------------------- /storm-supervisor/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM ziyunhx/storm:1.2.2 2 | 3 | RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF 4 | RUN apt-get update 5 | RUN apt-get install -y apt-transport-https 6 | RUN echo "deb https://download.mono-project.com/repo/ubuntu stable-trusty main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list 7 | RUN apt-get update 8 | RUN apt-get install -y mono-complete 9 | 10 | RUN wget -q https://packages.microsoft.com/config/ubuntu/14.04/packages-microsoft-prod.deb 11 | RUN dpkg -i packages-microsoft-prod.deb 12 | RUN apt-get update 13 | RUN apt-get install -y dotnet-sdk-2.1 14 | 15 | EXPOSE 6700 16 | EXPOSE 6701 17 | EXPOSE 6702 18 | EXPOSE 6703 19 | EXPOSE 8000 20 | 21 | RUN /usr/bin/config-supervisord.sh supervisor 22 | RUN /usr/bin/config-supervisord.sh logviewer 23 | CMD /usr/bin/start-supervisor.sh 24 | -------------------------------------------------------------------------------- /storm-ui/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM ziyunhx/storm:1.2.2 2 | 3 | RUN /usr/bin/config-supervisord.sh ui 4 | 5 | EXPOSE 8080 6 | CMD /usr/bin/start-supervisor.sh 7 | -------------------------------------------------------------------------------- /storm/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM ziyunhx/base 2 | 3 | RUN wget -q -O - http://mirrors.sonic.net/apache/storm/apache-storm-1.2.2/apache-storm-1.2.2.tar.gz | tar -xzf - -C /opt 4 | 5 | ENV STORM_HOME /opt/apache-storm-1.2.2 6 | RUN groupadd storm; useradd --gid storm --home-dir /home/storm --create-home --shell /bin/bash storm; chown -R storm:storm $STORM_HOME; mkdir /var/log/storm ; chown -R storm:storm /var/log/storm 7 | 8 | RUN ln -s $STORM_HOME/bin/storm /usr/bin/storm 9 | 10 | ADD storm.yaml $STORM_HOME/conf/storm.yaml 11 | ADD cluster.xml $STORM_HOME/logback/cluster.xml 12 | ADD config-supervisord.sh /usr/bin/config-supervisord.sh 13 | ADD start-supervisor.sh /usr/bin/start-supervisor.sh 14 | 15 | RUN echo [supervisord] | tee -a /etc/supervisor/supervisord.conf ; echo nodaemon=true | tee -a /etc/supervisor/supervisord.conf 16 | -------------------------------------------------------------------------------- /storm/cluster.xml: -------------------------------------------------------------------------------- 1 | 2 | 16 | 17 | 18 | 19 | %d{yyyy-MM-dd HH:mm:ss.SSS} %c{1.} [%p] %msg%n 20 | 21 | 22 | 25 | 26 | ${pattern} 27 | 28 | 29 | 30 | 31 | 32 | 33 | 36 | 37 | ${pattern} 38 | 39 | 40 | 41 | 42 | 43 | 44 | 47 | 48 | ${pattern} 49 | 50 | 51 | 52 | 53 | 54 | 55 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | -------------------------------------------------------------------------------- /storm/config-supervisord.sh: -------------------------------------------------------------------------------- 1 | echo [program:storm-$1] | tee -a /etc/supervisor/conf.d/storm-$1.conf 2 | echo command=storm $1 | tee -a /etc/supervisor/conf.d/storm-$1.conf 3 | echo directory=/home/storm | tee -a /etc/supervisor/conf.d/storm-$1.conf 4 | echo autorestart=true | tee -a /etc/supervisor/conf.d/storm-$1.conf 5 | echo user=root | tee -a /etc/supervisor/conf.d/storm-$1.conf 6 | -------------------------------------------------------------------------------- /storm/start-supervisor.sh: -------------------------------------------------------------------------------- 1 | sed -i -e "s/%zookeeper%/$ZK_PORT_2181_TCP_ADDR/g" $STORM_HOME/conf/storm.yaml 2 | sed -i -e "s/%nimbus%/$NIMBUS_PORT_6627_TCP_ADDR/g" $STORM_HOME/conf/storm.yaml 3 | 4 | echo "storm.local.hostname: `hostname -i`" >> $STORM_HOME/conf/storm.yaml 5 | 6 | /usr/sbin/sshd && supervisord 7 | -------------------------------------------------------------------------------- /storm/storm.yaml: -------------------------------------------------------------------------------- 1 | storm.zookeeper.servers: 2 | - "%zookeeper%" 3 | nimbus.seeds: ["%nimbus%"] 4 | drpc.servers: 5 | - "%nimbus%" 6 | drpc.port: 3772 7 | drpc.invocations.port: 3773 8 | -------------------------------------------------------------------------------- /zookeeper/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM ziyunhx/base 2 | 3 | RUN wget -q -O - http://mirror.vorboss.net/apache/zookeeper/zookeeper-3.4.9/zookeeper-3.4.9.tar.gz | tar -xzf - -C /opt 4 | RUN mv /opt/zookeeper-3.4.9/conf/zoo_sample.cfg /opt/zookeeper-3.4.9/conf/zoo.cfg 5 | 6 | ENV ZK_HOME /opt/zookeeper-3.4.9 7 | RUN sed -i "s|/tmp/zookeeper|$ZK_HOME/data|g" $ZK_HOME/conf/zoo.cfg; mkdir $ZK_HOME/data 8 | 9 | ADD start-zk.sh /usr/bin/start-zk.sh 10 | 11 | EXPOSE 2181 2888 3888 12 | 13 | WORKDIR /opt/zookeeper-3.4.9 14 | VOLUME ["/opt/zookeeper-3.4.9/conf", "/opt/zookeeper-3.4.9/data"] 15 | 16 | CMD /usr/sbin/sshd && start-zk.sh -------------------------------------------------------------------------------- /zookeeper/start-zk.sh: -------------------------------------------------------------------------------- 1 | sed -i -r 's|#(log4j.appender.ROLLINGFILE.MaxBackupIndex.*)|\1|g' $ZK_HOME/conf/log4j.properties 2 | sed -i -r 's|#autopurge|autopurge|g' $ZK_HOME/conf/zoo.cfg 3 | 4 | /opt/zookeeper-3.4.9/bin/zkServer.sh start-foreground --------------------------------------------------------------------------------