├── .gitignore ├── LICENSE ├── README.md ├── kafka.0.10.1.0 ├── docker-compose.yaml ├── kafka-manager-1.3.2.1.zip ├── kafka-manager.Dockerfile ├── kafka.0.10.1.0.Dockerfile ├── start_kafka.sh └── zookeeper.Dockerfile ├── kafka.0.8.2.2 ├── docker-compose.yml ├── kafka.Dockerfile └── zookeeper.Dockerfile ├── kafka.0.9.0.1 ├── docker-compose.yaml ├── kafka.0.9.0.1.Dockerfile └── zookeeper.Dockerfile └── kafka.1.1.0 ├── docker-compose.yaml ├── kafka-manager.Dockerfile ├── kafka.1.1.0.Dockerfile ├── start_kafka.sh └── zookeeper.Dockerfile /.gitignore: -------------------------------------------------------------------------------- 1 | tools 2 | ca 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "{}" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright {yyyy} {name of copyright owner} 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Build Kafka cluster with docker-compose 2 | 3 | ------ 4 | 5 | # 集群管理方法 6 | 7 | ## 构建image 8 | ```bash 9 | cd demokafka.0.10.1.0 10 | docker-compose build 11 | ``` 12 | 13 | ## 启动集群 14 | ```bash 15 | cd demokafka.0.10.1.0 16 | docker-compose up -d 17 | ``` 18 | 19 | ## 停止集群 20 | ```bash 21 | cd demokafka.0.10.1.0 22 | docker-compose stop 23 | ``` 24 | 25 | ## 删除集群 26 | ```bash 27 | cd demokafka.0.10.1.0 28 | docker-compose rm -f 29 | ``` 30 | 31 | # 集群配置说明 32 | 33 | ## 主要配置 34 | - ***ZOOKEEPER_CONNECT:*** zookeeper0:12181,zookeeper1:12182,zookeeper2:12183/kafka 35 | - ***BROKER_ID:*** 0 36 | - ***LISTENERS:*** PLAINTEXT://kafka0:19092,SSL://kafka0:29092 37 | - ***ZOOKEEPER_SESSION_TIMEOUT:*** 3600000 38 | - ***CONNECT_REST_PORT:*** 18083 39 | 40 | ## 其它配置 41 | 对于其它配置，配置名应以`KAFKA_PROPERTY_`开头，并且配置名须将点号换成下划线，如`auto.create.topics.enable`应以以下方式配置 42 | `KAFKA_PROPERTY_AUTO_CREATE_TOPICS_ENABLE: "false"` 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | -------------------------------------------------------------------------------- /kafka.0.10.1.0/docker-compose.yaml: -------------------------------------------------------------------------------- 1 | version: '2.0' 2 | services: 3 | zookeeper0: 4 | build: 5 | context: . 6 | dockerfile: zookeeper.Dockerfile 7 | image: jason/zookeeper:3.4.8 8 | container_name: zookeeper0 9 | hostname: zookeeper0 10 | ports: 11 | - "12181:12181" 12 | - "12888:12888" 13 | - "13888:13888" 14 | expose: 15 | - 12181 16 | - 12888 17 | - 13888 18 | environment: 19 | ZOOKEEPER_PORT: 12181 20 | ZOOKEEPER_ID: 0 21 | ZOOKEEPER_SERVERS: server.0=zookeeper0:12888:13888 server.1=zookeeper1:12889:13889 server.2=zookeeper2:12890:13890 22 | zookeeper1: 23 | build: 24 | context: . 25 | dockerfile: zookeeper.Dockerfile 26 | image: jason/zookeeper:3.4.8 27 | container_name: zookeeper1 28 | hostname: zookeeper1 29 | ports: 30 | - "12182:12182" 31 | - "12889:12889" 32 | - "13889:13889" 33 | expose: 34 | - 12182 35 | - 12889 36 | - 13889 37 | environment: 38 | ZOOKEEPER_PORT: 12182 39 | ZOOKEEPER_ID: 1 40 | ZOOKEEPER_SERVERS: server.0=zookeeper0:12888:13888 server.1=zookeeper1:12889:13889 server.2=zookeeper2:12890:13890 41 | # depends_on: 42 | # - zookeeper0 43 | zookeeper2: 44 | build: 45 | context: . 46 | dockerfile: zookeeper.Dockerfile 47 | image: jason/zookeeper:3.4.8 48 | container_name: zookeeper2 49 | hostname: zookeeper2 50 | ports: 51 | - "12183:12183" 52 | - "12890:12890" 53 | - "13890:13890" 54 | expose: 55 | - 12183 56 | - 12890 57 | - 13890 58 | environment: 59 | ZOOKEEPER_PORT: 12183 60 | ZOOKEEPER_ID: 2 61 | ZOOKEEPER_SERVERS: server.0=zookeeper0:12888:13888 server.1=zookeeper1:12889:13889 server.2=zookeeper2:12890:13890 62 | # depends_on: 63 | # - zookeeper1 64 | kafka0: 65 | build: 66 | context: . 67 | dockerfile: kafka.0.10.1.0.Dockerfile 68 | image: jason/kafka:0.10.1.0 69 | container_name: kafka0 70 | hostname: kafka0 71 | ports: 72 | - "19092:19092" 73 | - "29092:29092" 74 | - "18083:18083" 75 | - "12345:12345" 76 | environment: 77 | ZOOKEEPER_CONNECT: zookeeper0:12181,zookeeper1:12182,zookeeper2:12183/kafka 78 | BROKER_ID: 0 79 | LISTENERS: PLAINTEXT://kafka0:19092,SSL://kafka0:29092 80 | ZOOKEEPER_SESSION_TIMEOUT: 3600000 81 | CONNECT_REST_PORT: 18083 82 | KAFKA_PROPERTY_AUTO_CREATE_TOPICS_ENABLE: "false" 83 | KAFKA_PROPERTY_SSL_CLIENT_AUTH: required 84 | KAFKA_PROPERTY_DELETE_TOPIC_ENABLE: "true" 85 | PORT_JMX: 12345 86 | KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" 87 | volumes: 88 | - /var/run/docker.sock:/var/run/docker.sock 89 | - ./ca:/opt/kafka/ca 90 | depends_on: 91 | - zookeeper0 92 | - zookeeper1 93 | - zookeeper2 94 | expose: 95 | - 9092 96 | # links: 97 | # - zookeeper 98 | kafka1: 99 | build: 100 | context: . 101 | dockerfile: kafka.0.10.1.0.Dockerfile 102 | image: jason/kafka:0.10.1.0 103 | container_name: kafka1 104 | hostname: kafka1 105 | ports: 106 | - "19093:19093" 107 | - "29093:29093" 108 | - "18084:18084" 109 | - "12346:12346" 110 | environment: 111 | ZOOKEEPER_CONNECT: zookeeper0:12181,zookeeper1:12182,zookeeper2:12183/kafka 112 | BROKER_ID: 1 113 | LISTENERS: PLAINTEXT://kafka1:19093,SSL://kafka1:29093 114 | ZOOKEEPER_SESSION_TIMEOUT: 3600000 115 | CONNECT_REST_PORT: 18084 116 | KAFKA_PROPERTY_AUTO_CREATE_TOPICS_ENABLE: "false" 117 | KAFKA_PROPERTY_SSL_CLIENT_AUTH: required 118 | KAFKA_PROPERTY_DELETE_TOPIC_ENABLE: "true" 119 | PORT_JMX: 12346 120 | KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=kafka1" 121 | volumes: 122 | - /var/run/docker.sock:/var/run/docker.sock 123 | - ./ca:/opt/kafka/ca 124 | depends_on: 125 | - zookeeper0 126 | - zookeeper1 127 | - zookeeper2 128 | expose: 129 | - 9093 130 | # links: 131 | # - zookeeper 132 | kafka2: 133 | build: . 134 | build: 135 | context: . 136 | dockerfile: kafka.0.10.1.0.Dockerfile 137 | image: jason/kafka:0.10.1.0 138 | container_name: kafka2 139 | hostname: kafka2 140 | ports: 141 | - "19094:19094" 142 | - "29094:29094" 143 | - "18085:18085" 144 | - "12347:12347" 145 | environment: 146 | ZOOKEEPER_CONNECT: zookeeper0:12181,zookeeper1:12182,zookeeper2:12183/kafka 147 | BROKER_ID: 2 148 | LISTENERS: PLAINTEXT://kafka2:19094,SSL://kafka2:29094 149 | ZOOKEEPER_SESSION_TIMEOUT: 3600000 150 | CONNECT_REST_PORT: 18085 151 | KAFKA_PROPERTY_AUTO_CREATE_TOPICS_ENABLE: "false" 152 | KAFKA_PROPERTY_SSL_CLIENT_AUTH: required 153 | KAFKA_PROPERTY_DELETE_TOPIC_ENABLE: "true" 154 | PORT_JMX: 12347 155 | KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=kafka2" 156 | volumes: 157 | - /var/run/docker.sock:/var/run/docker.sock 158 | - ./ca:/opt/kafka/ca 159 | depends_on: 160 | - zookeeper0 161 | - zookeeper1 162 | - zookeeper2 163 | expose: 164 | - 9094 165 | kafka3: 166 | build: 167 | context: . 168 | dockerfile: kafka.0.10.1.0.Dockerfile 169 | image: jason/kafka:0.10.1.0 170 | container_name: kafka3 171 | hostname: kafka3 172 | ports: 173 | - "19095:19095" 174 | - "29095:29095" 175 | - "18086:18086" 176 | - "12348:12348" 177 | environment: 178 | ZOOKEEPER_CONNECT: zookeeper0:12181,zookeeper1:12182,zookeeper2:12183/kafka 179 | BROKER_ID: 3 180 | LISTENERS: PLAINTEXT://kafka3:19095,SSL://kafka3:29095 181 | ZOOKEEPER_SESSION_TIMEOUT: 3600000 182 | CONNECT_REST_PORT: 18086 183 | KAFKA_PROPERTY_AUTO_CREATE_TOPICS_ENABLE: "false" 184 | KAFKA_PROPERTY_SSL_CLIENT_AUTH: required 185 | KAFKA_PROPERTY_DELETE_TOPIC_ENABLE: "true" 186 | PORT_JMX: 12348 187 | KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" 188 | volumes: 189 | - /var/run/docker.sock:/var/run/docker.sock 190 | - ./ca:/opt/kafka/ca 191 | depends_on: 192 | - zookeeper0 193 | - zookeeper1 194 | - zookeeper2 195 | expose: 196 | - 19095 197 | - 29095 198 | # links: 199 | # - zookeeper 200 | kafka4: 201 | build: 202 | context: . 203 | dockerfile: kafka.0.10.1.0.Dockerfile 204 | image: jason/kafka:0.10.1.0 205 | container_name: kafka4 206 | hostname: kafka4 207 | ports: 208 | - "19096:19096" 209 | - "29096:29096" 210 | - "18087:18087" 211 | - "12349:12349" 212 | environment: 213 | ZOOKEEPER_CONNECT: zookeeper0:12181,zookeeper1:12182,zookeeper2:12183/kafka 214 | BROKER_ID: 4 215 | LISTENERS: PLAINTEXT://kafka4:19096,SSL://kafka4:29096 216 | ZOOKEEPER_SESSION_TIMEOUT: 3600000 217 | CONNECT_REST_PORT: 18087 218 | KAFKA_PROPERTY_AUTO_CREATE_TOPICS_ENABLE: "false" 219 | KAFKA_PROPERTY_SSL_CLIENT_AUTH: required 220 | KAFKA_PROPERTY_DELETE_TOPIC_ENABLE: "true" 221 | PORT_JMX: 12349 222 | KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=kafka1" 223 | volumes: 224 | - /var/run/docker.sock:/var/run/docker.sock 225 | - ./ca:/opt/kafka/ca 226 | depends_on: 227 | - zookeeper0 228 | - zookeeper1 229 | - zookeeper2 230 | expose: 231 | - 29096 232 | - 19096 233 | # links: 234 | # - zookeeper 235 | kafka5: 236 | build: . 237 | build: 238 | context: . 239 | dockerfile: kafka.0.10.1.0.Dockerfile 240 | image: jason/kafka:0.10.1.0 241 | container_name: kafka5 242 | hostname: kafka5 243 | ports: 244 | - "19097:19097" 245 | - "29097:29097" 246 | - "18088:18088" 247 | - "12350:12350" 248 | environment: 249 | ZOOKEEPER_CONNECT: zookeeper0:12181,zookeeper1:12182,zookeeper2:12183/kafka 250 | BROKER_ID: 5 251 | LISTENERS: PLAINTEXT://kafka5:19097,SSL://kafka5:29097 252 | ZOOKEEPER_SESSION_TIMEOUT: 3600000 253 | CONNECT_REST_PORT: 18088 254 | KAFKA_PROPERTY_AUTO_CREATE_TOPICS_ENABLE: "false" 255 | KAFKA_PROPERTY_SSL_CLIENT_AUTH: required 256 | KAFKA_PROPERTY_DELETE_TOPIC_ENABLE: "true" 257 | PORT_JMX: 12350 258 | KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=kafka2" 259 | volumes: 260 | - /var/run/docker.sock:/var/run/docker.sock 261 | - ./ca:/opt/kafka/ca 262 | depends_on: 263 | - zookeeper0 264 | - zookeeper1 265 | - zookeeper2 266 | expose: 267 | - 19097 268 | - 29097 269 | kafka-manager: 270 | build: . 271 | build: 272 | context: . 273 | dockerfile: kafka-manager.Dockerfile 274 | image: jason/kafka-manager:1.0 275 | container_name: kafka-manager 276 | hostname: kafka-manager 277 | ports: 278 | - "38080:38080" 279 | environment: 280 | ZK_HOSTS: zookeeper0:12181 281 | PORT: 38080 282 | expose: 283 | - 38080 284 | -------------------------------------------------------------------------------- /kafka.0.10.1.0/kafka-manager-1.3.2.1.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/habren/kafka-docker/fc7e4a56eed66231eb8d7131b3a26b0014aefa47/kafka.0.10.1.0/kafka-manager-1.3.2.1.zip -------------------------------------------------------------------------------- /kafka.0.10.1.0/kafka-manager.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM centos:6.6 2 | 3 | RUN mkdir /etc/yum.repos.d/backup &&\ 4 | mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/backup/ &&\ 5 | curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo 6 | 7 | RUN yum -y install nc vim lsof wget tar bzip2 unzip vim-enhanced passwd sudo yum-utils hostname net-tools rsync man git make automake cmake patch logrotate python-devel libpng-devel libjpeg-devel pwgen python-pip 8 | 9 | RUN mkdir /opt/java &&\ 10 | wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u171-b11/512cd62ec5174c3487ac17c61aaa89e8/jdk-8u171-linux-x64.tar.gz -P /opt/java 11 | 12 | ENV JAVA_HOME "/opt/java/jdk1.8.0_171" 13 | 14 | RUN tar zxvf /opt/java/jdk-8u171-linux-x64.tar.gz -C /opt/java &&\ 15 | sed -i "/^PATH/i export JAVA_HOME="$JAVA_HOME /root/.bash_profile &&\ 16 | sed -i "s%^PATH.*$%&:"$JAVA_HOME"/bin%g" /root/.bash_profile &&\ 17 | source /root/.bash_profile 18 | 19 | COPY kafka-manager-1.3.2.1.zip /opt 20 | 21 | RUN unzip /opt/kafka-manager-1.3.2.1.zip -d /opt &&\ 22 | mv /opt/kafka-manager-1.3.2.1 /opt/kafka-manager 23 | 24 | WORKDIR /opt/kafka-manager 25 | 26 | ENTRYPOINT ["bin/kafka-manager", "-Dhttp.port=38080"] 27 | 28 | -------------------------------------------------------------------------------- /kafka.0.10.1.0/kafka.0.10.1.0.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM centos:6.6 2 | 3 | RUN mkdir /etc/yum.repos.d/backup &&\ 4 | mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/backup/ &&\ 5 | curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo 6 | 7 | RUN yum -y install nc vim lsof wget tar bzip2 unzip vim-enhanced passwd sudo yum-utils hostname net-tools rsync man git make automake cmake patch logrotate python-devel libpng-devel libjpeg-devel pwgen python-pip 8 | 9 | RUN mkdir /opt/java &&\ 10 | wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u171-b11/512cd62ec5174c3487ac17c61aaa89e8/jdk-8u171-linux-x64.tar.gz -P /opt/java 11 | 12 | ENV KAFKA_VERSION "0.10.1.0" 13 | 14 | RUN mkdir /opt/kafka &&\ 15 | wget https://archive.apache.org/dist/kafka/$KAFKA_VERSION/kafka_2.11-$KAFKA_VERSION.tgz -P /opt/kafka 16 | 17 | ENV JAVA_HOME "/opt/java/jdk1.8.0_171" 18 | 19 | RUN tar zxvf /opt/java/jdk-8u171-linux-x64.tar.gz -C /opt/java &&\ 20 | sed -i "/^PATH/i export JAVA_HOME="$JAVA_HOME /root/.bash_profile &&\ 21 | sed -i "s%^PATH.*$%&:"$JAVA_HOME"/bin%g" /root/.bash_profile &&\ 22 | source /root/.bash_profile 23 | 24 | RUN tar zxvf /opt/kafka/kafka*.tgz -C /opt/kafka &&\ 25 | sed -i 's/num.partitions.*$/num.partitions=3/g' /opt/kafka/kafka_2.11-$KAFKA_VERSION/config/server.properties 26 | 27 | RUN sed -i '0,/^if/s%^if%export JAVA_HOME='$JAVA_HOME'\nexport PATH=$PATH:$JAVA_HOME/bin\nif%' /opt/kafka/kafka_2.11-0.10.1.0/bin/kafka-run-class.sh 28 | 29 | COPY start_kafka.sh /opt/kafka 30 | 31 | #RUN yum install -y tcpdump dstat sysstat 32 | 33 | EXPOSE 9092 34 | 35 | WORKDIR /opt/kafka/kafka_2.11-$KAFKA_VERSION 36 | 37 | ENTRYPOINT ["sh", "/opt/kafka/start_kafka.sh"] 38 | -------------------------------------------------------------------------------- /kafka.0.10.1.0/start_kafka.sh: -------------------------------------------------------------------------------- 1 | source /root/.bash_profile 2 | 3 | KAFKA_HOME=/opt/kafka/kafka_2.11-$KAFKA_VERSION 4 | 5 | pushd $KAFKA_HOME 6 | [ ! -z $ZOOKEEPER_CONNECT ] && sed -i 's%.*zookeeper.connect=.*$%zookeeper.connect='$ZOOKEEPER_CONNECT'%g' config/server.properties 7 | [ ! -z $BROKER_ID ] && sed -i 's%broker.id=.*$%broker.id='$BROKER_ID'%g' config/server.properties 8 | sed -i 's%#listeners=.*$%listeners=PLAINTEXT://'$(hostname -i)':9092%g' config/server.properties 9 | [ ! -z $LISTENERS ] && sed -i 's%listeners=.*$%listeners='$LISTENERS'%g' config/server.properties 10 | [ ! -z $ZOOKEEPER_SESSION_TIMEOUT ] && sed -i 's%zookeeper.connection.timeout.ms.*$%zookeeper.connection.timeout.ms='$ZOOKEEPER_SESSION_TIMEOUT'%g' config/server.properties 11 | [ ! -z $CONNECT_REST_PORT ] && echo rest.port=$CONNECT_REST_PORT >> /opt/kafka/kafka_2.11-0.10.1.0/config/connect-standalone.properties && echo -e '\nrest.port='$CONNECT_REST_PORT >> config/connect-distributed.properties 12 | [ ! -z $PORT_JMX ] && export JMX_PORT=$PORT_JMX 13 | popd 14 | 15 | 16 | if [[ $(echo $LISTENERS | tr '[:upper:]' '[:lower:]') =~ ssl ]] 17 | then 18 | pushd /opt/kafka 19 | mkdir /opt/kafka/ssl 20 | country=US ; [ ! -z $SSL_C ] && country=$SSL_C 21 | organization=UNKNOWN ; [ ! -z $SSL_O ] && organization=$SSL_O 22 | organizationalunit=UNKNOWN ; [ ! -z $SSL_OU ] && organizationalunit=$SSL_OU 23 | province=UNKNOWN ; [ ! -z $SSL_ST ] && province=$SSL_ST 24 | commonname=US ; [ ! -z $SSL_CN ] && commonname=$SSL_CN 25 | locality=UNKNOWN ; [ ! -z $SSL_L ] && locality=$SSL_L 26 | ssl_password=jasonguo ; [ ! -z $SSL_PASSWORD ] && ssl_password=$SSL_PASSWORD 27 | 28 | if [ -f ca/ca.crt ] || [ -f ca/ca.key ] 29 | then 30 | sleep 5s 31 | else 32 | openssl req -new -x509 -keyout ca/ca.key -out ca/ca.crt -days 365 -passout pass:$ssl_password -subj "/C="$country"/ST="$province"/L="$locality"/O="$organization"/OU="$organizationalunit"/CN="$commonname 33 | fi 34 | 35 | keytool -keystore ssl/$HOSTNAME.keystore.jks -alias $HOSTNAME -validity 365 -storepass $ssl_password -keypass $ssl_password -genkey -dname "CN="$commonname", OU="$organizationalunit", O="$organization", L="$locality", ST="$province", C="$country 36 | 37 | keytool -v -keystore ssl/$HOSTNAME.truststore.jks -alias CARoot -import -file ca/ca.crt -storepass $ssl_password <> config/server.properties 83 | fi 84 | fi 85 | done 86 | popd 87 | 88 | cd $KAFKA_HOME 89 | bin/kafka-server-start.sh config/server.properties 90 | -------------------------------------------------------------------------------- /kafka.0.10.1.0/zookeeper.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM centos:6.6 2 | 3 | RUN mkdir /etc/yum.repos.d/backup &&\ 4 | mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/backup/ &&\ 5 | curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo 6 | 7 | RUN yum -y install nc vim lsof wget tar bzip2 unzip vim-enhanced passwd sudo yum-utils hostname net-tools rsync man git make automake cmake patch logrotate python-devel libpng-devel libjpeg-devel pwgen python-pip 8 | 9 | RUN mkdir /opt/java &&\ 10 | wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u171-b11/512cd62ec5174c3487ac17c61aaa89e8/jdk-8u171-linux-x64.tar.gz -P /opt/java 11 | 12 | RUN tar zxvf /opt/java/jdk-8u171-linux-x64.tar.gz -C /opt/java &&\ 13 | JAVA_HOME=/opt/java/jdk1.8.0_171 &&\ 14 | sed -i "/^PATH/i export JAVA_HOME=$JAVA_HOME" /root/.bash_profile &&\ 15 | sed -i "s%^PATH.*$%&:$JAVA_HOME/bin%g" /root/.bash_profile &&\ 16 | source /root/.bash_profile 17 | 18 | ENV ZOOKEEPER_VERSION "3.4.8" 19 | 20 | RUN mkdir /opt/zookeeper &&\ 21 | wget http://archive.apache.org/dist/zookeeper/zookeeper-$ZOOKEEPER_VERSION/zookeeper-$ZOOKEEPER_VERSION.tar.gz -P /opt/zookeeper 22 | 23 | RUN tar zxvf /opt/zookeeper/zookeeper*.tar.gz -C /opt/zookeeper 24 | 25 | RUN echo "source /root/.bash_profile" > /opt/zookeeper/start.sh &&\ 26 | echo "cp /opt/zookeeper/zookeeper-"$ZOOKEEPER_VERSION"/conf/zoo_sample.cfg /opt/zookeeper/zookeeper-"$ZOOKEEPER_VERSION"/conf/zoo.cfg" >> /opt/zookeeper/start.sh &&\ 27 | echo "[ ! -z $""ZOOKEEPER_PORT"" ] && sed -i 's%.*clientPort=.*$%clientPort='$""ZOOKEEPER_PORT'""%g' /opt/zookeeper/zookeeper-"$ZOOKEEPER_VERSION"/conf/zoo.cfg" >> /opt/zookeeper/start.sh &&\ 28 | echo "[ ! -z $""ZOOKEEPER_ID"" ] && mkdir -p /tmp/zookeeper && echo $""ZOOKEEPER_ID > /tmp/zookeeper/myid" >> /opt/zookeeper/start.sh &&\ 29 | echo "[[ ! -z $""ZOOKEEPER_SERVERS"" ]] && for server in $""ZOOKEEPER_SERVERS""; do echo $""server"" >> /opt/zookeeper/zookeeper-"$ZOOKEEPER_VERSION"/conf/zoo.cfg; done" >> /opt/zookeeper/start.sh &&\ 30 | echo "/opt/zookeeper/zookeeper-$"ZOOKEEPER_VERSION"/bin/zkServer.sh start-foreground" >> /opt/zookeeper/start.sh 31 | 32 | EXPOSE 2181 33 | 34 | WORKDIR /opt/zookeeper/zookeeper-$ZOOKEEPER_VERSION 35 | 36 | ENTRYPOINT ["sh", "/opt/zookeeper/start.sh"] 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | -------------------------------------------------------------------------------- /kafka.0.8.2.2/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '2.0' 2 | services: 3 | zookeeper0: 4 | build: 5 | context: . 6 | dockerfile: zookeeper.Dockerfile 7 | image: jason/zookeeper:3.4.6 8 | container_name: zookeeper0 9 | hostname: zookeeper0 10 | ports: 11 | - "2181:2181" 12 | - "2888:2888" 13 | - "3888:3888" 14 | expose: 15 | - 2181 16 | - 2888 17 | - 3888 18 | environment: 19 | ZOOKEEPER_PORT: 2181 20 | ZOOKEEPER_ID: 0 21 | ZOOKEEPER_SERVERS: server.0=zookeeper0:2888:3888 server.1=zookeeper1:28881:38881 server.2=zookeeper2:28882:38882 22 | zookeeper1: 23 | build: 24 | context: . 25 | dockerfile: zookeeper.Dockerfile 26 | image: jason/zookeeper:3.4.6 27 | container_name: zookeeper1 28 | hostname: zookeeper1 29 | ports: 30 | - "2182:2182" 31 | - "28881:28881" 32 | - "38881:38881" 33 | expose: 34 | - 2182 35 | - 2888 36 | - 3888 37 | environment: 38 | ZOOKEEPER_PORT: 2182 39 | ZOOKEEPER_ID: 1 40 | ZOOKEEPER_SERVERS: server.0=zookeeper0:2888:3888 server.1=zookeeper1:28881:38881 server.2=zookeeper2:28882:38882 41 | # depends_on: 42 | # - zookeeper0 43 | zookeeper2: 44 | build: 45 | context: . 46 | dockerfile: zookeeper.Dockerfile 47 | image: jason/zookeeper:3.4.6 48 | container_name: zookeeper2 49 | hostname: zookeeper2 50 | ports: 51 | - "2183:2183" 52 | - "28882:28882" 53 | - "38882:38882" 54 | expose: 55 | - 2183 56 | - 2888 57 | - 3888 58 | environment: 59 | ZOOKEEPER_PORT: 2183 60 | ZOOKEEPER_ID: 2 61 | ZOOKEEPER_SERVERS: server.0=zookeeper0:2888:3888 server.1=zookeeper1:28881:38881 server.2=zookeeper2:28882:38882 62 | # depends_on: 63 | # - zookeeper1 64 | kafka0: 65 | build: 66 | context: . 67 | dockerfile: kafka.Dockerfile 68 | image: jason/kafka:0.8.2.2 69 | container_name: kafka0 70 | hostname: kafka0 71 | ports: 72 | - "9092:9092" 73 | environment: 74 | ZOOKEEPER_CONNECT: zookeeper0:2181,zookeeper1:2182,zookeeper2:2183/kafka 75 | BROKER_ID: 0 76 | BROKER_PORT: 9092 77 | ADVERTISED_HOST_NAME: kafka0 78 | HOST_NAME: kafka0 79 | ZOOKEEPER_SESSION_TIMEOUT: 3600000 80 | volumes: 81 | - /var/run/docker.sock:/var/run/docker.sock 82 | depends_on: 83 | - zookeeper0 84 | - zookeeper1 85 | - zookeeper2 86 | expose: 87 | - 9092 88 | # links: 89 | # - zookeeper 90 | kafka1: 91 | build: 92 | context: . 93 | dockerfile: kafka.Dockerfile 94 | image: jason/kafka:0.8.2.2 95 | container_name: kafka1 96 | hostname: kafka1 97 | ports: 98 | - "9093:9093" 99 | environment: 100 | ZOOKEEPER_CONNECT: zookeeper0:2181,zookeeper1:2182,zookeeper2:2183/kafka 101 | BROKER_ID: 1 102 | BROKER_PORT: 9093 103 | ADVERTISED_HOST_NAME: kafka1 104 | HOST_NAME: kafka1 105 | ZOOKEEPER_SESSION_TIMEOUT: 3600000 106 | volumes: 107 | - /var/run/docker.sock:/var/run/docker.sock 108 | depends_on: 109 | - zookeeper0 110 | - zookeeper1 111 | - zookeeper2 112 | expose: 113 | - 9093 114 | # links: 115 | # - zookeeper 116 | kafka2: 117 | build: . 118 | build: 119 | context: . 120 | dockerfile: kafka.Dockerfile 121 | image: jason/kafka:0.8.2.2 122 | container_name: kafka2 123 | hostname: kafka2 124 | ports: 125 | - "9094:9094" 126 | environment: 127 | ZOOKEEPER_CONNECT: zookeeper0:2181,zookeeper1:2182,zookeeper2:2183/kafka 128 | BROKER_ID: 2 129 | BROKER_PORT: 9094 130 | ADVERTISED_HOST_NAME: kafka2 131 | HOST_NAME: kafka2 132 | ZOOKEEPER_SESSION_TIMEOUT: 3600000 133 | volumes: 134 | - /var/run/docker.sock:/var/run/docker.sock 135 | depends_on: 136 | - zookeeper0 137 | - zookeeper1 138 | - zookeeper2 139 | expose: 140 | - 9094 141 | # links: 142 | # - zookeeper -------------------------------------------------------------------------------- /kafka.0.8.2.2/kafka.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM centos:6.6 2 | 3 | RUN mkdir /etc/yum.repos.d/backup &&\ 4 | mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/backup/ &&\ 5 | curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo 6 | 7 | RUN yum -y install tcpdump dtat sysstat vim lsof wget tar bzip2 unzip vim-enhanced passwd sudo yum-utils hostname net-tools rsync man git make automake cmake patch logrotate python-devel libpng-devel libjpeg-devel pwgen python-pip 8 | 9 | RUN mkdir /opt/java &&\ 10 | wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u102-b14/jdk-8u102-linux-x64.tar.gz -P /opt/java 11 | 12 | ENV KAFKA_VERSION "0.8.2.2" 13 | 14 | RUN mkdir /opt/kafka &&\ 15 | wget http://apache.fayea.com/kafka/$KAFKA_VERSION/kafka_2.11-$KAFKA_VERSION.tgz -P /opt/kafka 16 | 17 | RUN tar zxvf /opt/java/jdk-8u102-linux-x64.tar.gz -C /opt/java &&\ 18 | JAVA_HOME=/opt/java/jdk1.8.0_102 &&\ 19 | sed -i "/^PATH/i export JAVA_HOME=$JAVA_HOME" /root/.bash_profile &&\ 20 | sed -i "s%^PATH.*$%&:$JAVA_HOME/bin%g" /root/.bash_profile &&\ 21 | source /root/.bash_profile 22 | 23 | RUN tar zxvf /opt/kafka/kafka*.tgz -C /opt/kafka &&\ 24 | sed -i 's/num.partitions.*$/num.partitions=3/g' /opt/kafka/kafka_2.11-$KAFKA_VERSION/config/server.properties 25 | 26 | RUN echo "source /root/.bash_profile" > /opt/kafka/start.sh &&\ 27 | echo "cd /opt/kafka/kafka_2.11-"$KAFKA_VERSION >> /opt/kafka/start.sh &&\ 28 | #echo "sed -i 's%zookeeper.connect=.*$%zookeeper.connect=zookeeper:2181%g' /opt/kafka/kafka_2.11-"$KAFKA_VERSION"/config/server.properties" >> /opt/kafka/start.sh &&\ 29 | echo "[ ! -z $""ZOOKEEPER_CONNECT"" ] && sed -i 's%.*zookeeper.connect=.*$%zookeeper.connect='$""ZOOKEEPER_CONNECT'""%g' /opt/kafka/kafka_2.11-"$KAFKA_VERSION"/config/server.properties" >> /opt/kafka/start.sh &&\ 30 | echo "[ ! -z $""BROKER_ID"" ] && sed -i 's%broker.id=.*$%broker.id='$""BROKER_ID'""%g' /opt/kafka/kafka_2.11-"$KAFKA_VERSION"/config/server.properties" >> /opt/kafka/start.sh &&\ 31 | echo "[ ! -z $""BROKER_PORT"" ] && sed -i 's%port=.*$%port='$""BROKER_PORT'""%g' /opt/kafka/kafka_2.11-"$KAFKA_VERSION"/config/server.properties" >> /opt/kafka/start.sh &&\ 32 | echo "sed -i 's%#advertised.host.name=.*$%advertised.host.name='$""(hostname -i)'""%g' /opt/kafka/kafka_2.11-"$KAFKA_VERSION"/config/server.properties" >> /opt/kafka/start.sh &&\ 33 | echo "[ ! -z $""ADVERTISED_HOST_NAME"" ] && sed -i 's%advertised.host.name=.*$%advertised.host.name='$""ADVERTISED_HOST_NAME'""%g' /opt/kafka/kafka_2.11-"$KAFKA_VERSION"/config/server.properties" >> /opt/kafka/start.sh &&\ 34 | echo "sed -i 's%#host.name=.*$%host.name='$""(hostname -i)'""%g' /opt/kafka/kafka_2.11-"$KAFKA_VERSION"/config/server.properties" >> /opt/kafka/start.sh &&\ 35 | echo "[ ! -z $""HOST_NAME"" ] && sed -i 's%host.name=.*$%host.name='$""HOST_NAME'""%g' /opt/kafka/kafka_2.11-"$KAFKA_VERSION"/config/server.properties" >> /opt/kafka/start.sh &&\ 36 | echo "[ ! -z $""ZOOKEEPER_SESSION_TIMEOUT"" ] && sed -i 's%zookeeper.connection.timeout.ms.*$%zookeeper.connection.timeout.ms='$""ZOOKEEPER_SESSION_TIMEOUT'""%g' /opt/kafka/kafka_2.11-"$KAFKA_VERSION"/config/server.properties" >> /opt/kafka/start.sh &&\ 37 | echo "delete.topic.enable=true" >> /opt/kafka/kafka_2.11-$KAFKA_VERSION/config/server.properties &&\ 38 | echo "bin/kafka-server-start.sh config/server.properties" >> /opt/kafka/start.sh &&\ 39 | chmod a+x /opt/kafka/start.sh 40 | 41 | RUN yum install -y nc 42 | 43 | EXPOSE 9092 44 | 45 | WORKDIR /opt/kafka/kafka_2.11-$KAFKA_VERSION 46 | 47 | ENTRYPOINT ["sh", "/opt/kafka/start.sh"] 48 | -------------------------------------------------------------------------------- /kafka.0.8.2.2/zookeeper.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM centos:6.6 2 | 3 | RUN mkdir /etc/yum.repos.d/backup &&\ 4 | mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/backup/ &&\ 5 | curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo 6 | 7 | RUN yum -y install tcpdump dtat sysstat vim lsof wget tar bzip2 unzip vim-enhanced passwd sudo yum-utils hostname net-tools rsync man git make automake cmake patch logrotate python-devel libpng-devel libjpeg-devel pwgen python-pip 8 | 9 | RUN mkdir /opt/java &&\ 10 | wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u102-b14/jdk-8u102-linux-x64.tar.gz -P /opt/java 11 | 12 | RUN tar zxvf /opt/java/jdk-8u102-linux-x64.tar.gz -C /opt/java &&\ 13 | JAVA_HOME=/opt/java/jdk1.8.0_102 &&\ 14 | sed -i "/^PATH/i export JAVA_HOME=$JAVA_HOME" /root/.bash_profile &&\ 15 | sed -i "s%^PATH.*$%&:$JAVA_HOME/bin%g" /root/.bash_profile &&\ 16 | source /root/.bash_profile 17 | 18 | ENV ZOOKEEPER_VERSION "3.4.6" 19 | 20 | RUN mkdir /opt/zookeeper &&\ 21 | wget http://mirror.olnevhost.net/pub/apache/zookeeper/zookeeper-$ZOOKEEPER_VERSION/zookeeper-$ZOOKEEPER_VERSION.tar.gz -P /opt/zookeeper 22 | 23 | RUN tar zxvf /opt/zookeeper/zookeeper*.tar.gz -C /opt/zookeeper 24 | 25 | RUN echo "source /root/.bash_profile" > /opt/zookeeper/start.sh &&\ 26 | echo "cp /opt/zookeeper/zookeeper-"$ZOOKEEPER_VERSION"/conf/zoo_sample.cfg /opt/zookeeper/zookeeper-"$ZOOKEEPER_VERSION"/conf/zoo.cfg" >> /opt/zookeeper/start.sh &&\ 27 | echo "[ ! -z $""ZOOKEEPER_PORT"" ] && sed -i 's%.*clientPort=.*$%clientPort='$""ZOOKEEPER_PORT'""%g' /opt/zookeeper/zookeeper-"$ZOOKEEPER_VERSION"/conf/zoo.cfg" >> /opt/zookeeper/start.sh &&\ 28 | echo "[ ! -z $""ZOOKEEPER_ID"" ] && mkdir -p /tmp/zookeeper && echo $""ZOOKEEPER_ID > /tmp/zookeeper/myid" >> /opt/zookeeper/start.sh &&\ 29 | echo "[[ ! -z $""ZOOKEEPER_SERVERS"" ]] && for server in $""ZOOKEEPER_SERVERS""; do echo $""server"" >> /opt/zookeeper/zookeeper-"$ZOOKEEPER_VERSION"/conf/zoo.cfg; done" >> /opt/zookeeper/start.sh &&\ 30 | echo "/opt/zookeeper/zookeeper-$"ZOOKEEPER_VERSION"/bin/zkServer.sh start-foreground" >> /opt/zookeeper/start.sh 31 | 32 | RUN yum install -y nc 33 | 34 | EXPOSE 2181 35 | 36 | WORKDIR /opt/zookeeper/zookeeper-$ZOOKEEPER_VERSION 37 | 38 | ENTRYPOINT ["sh", "/opt/zookeeper/start.sh"] 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | -------------------------------------------------------------------------------- /kafka.0.9.0.1/docker-compose.yaml: -------------------------------------------------------------------------------- 1 | version: '2.0' 2 | services: 3 | zookeeper0: 4 | build: 5 | context: . 6 | dockerfile: zookeeper.Dockerfile 7 | image: jason/zookeeper:3.4.6 8 | container_name: zookeeper0 9 | hostname: zookeeper0 10 | ports: 11 | - "2181:2181" 12 | - "2888:2888" 13 | - "3888:3888" 14 | expose: 15 | - 2181 16 | - 2888 17 | - 3888 18 | environment: 19 | ZOOKEEPER_PORT: 2181 20 | ZOOKEEPER_ID: 0 21 | ZOOKEEPER_SERVERS: server.0=zookeeper0:2888:3888 server.1=zookeeper1:28881:38881 server.2=zookeeper2:28882:38882 22 | zookeeper1: 23 | build: 24 | context: . 25 | dockerfile: zookeeper.Dockerfile 26 | image: jason/zookeeper:3.4.6 27 | container_name: zookeeper1 28 | hostname: zookeeper1 29 | ports: 30 | - "2182:2182" 31 | - "28881:28881" 32 | - "38881:38881" 33 | expose: 34 | - 2182 35 | - 2888 36 | - 3888 37 | environment: 38 | ZOOKEEPER_PORT: 2182 39 | ZOOKEEPER_ID: 1 40 | ZOOKEEPER_SERVERS: server.0=zookeeper0:2888:3888 server.1=zookeeper1:28881:38881 server.2=zookeeper2:28882:38882 41 | # depends_on: 42 | # - zookeeper0 43 | zookeeper2: 44 | build: 45 | context: . 46 | dockerfile: zookeeper.Dockerfile 47 | image: jason/zookeeper:3.4.6 48 | container_name: zookeeper2 49 | hostname: zookeeper2 50 | ports: 51 | - "2183:2183" 52 | - "28882:28882" 53 | - "38882:38882" 54 | expose: 55 | - 2183 56 | - 2888 57 | - 3888 58 | environment: 59 | ZOOKEEPER_PORT: 2183 60 | ZOOKEEPER_ID: 2 61 | ZOOKEEPER_SERVERS: server.0=zookeeper0:2888:3888 server.1=zookeeper1:28881:38881 server.2=zookeeper2:28882:38882 62 | # depends_on: 63 | # - zookeeper1 64 | kafka0: 65 | build: 66 | context: . 67 | dockerfile: kafka.0.9.0.1.Dockerfile 68 | image: jason/kafka:0.9.0.1 69 | container_name: kafka0 70 | hostname: kafka0 71 | ports: 72 | - "9092:9092" 73 | environment: 74 | ZOOKEEPER_CONNECT: zookeeper0:2181,zookeeper1:2182,zookeeper2:2183/kafka 75 | BROKER_ID: 0 76 | BROKER_PORT: 9092 77 | ADVERTISED_HOST_NAME: kafka0 78 | HOST_NAME: kafka0 79 | ZOOKEEPER_SESSION_TIMEOUT: 3600000 80 | volumes: 81 | - /var/run/docker.sock:/var/run/docker.sock 82 | depends_on: 83 | - zookeeper0 84 | - zookeeper1 85 | - zookeeper2 86 | expose: 87 | - 9092 88 | # links: 89 | # - zookeeper 90 | kafka1: 91 | build: 92 | context: . 93 | dockerfile: kafka.0.9.0.1.Dockerfile 94 | image: jason/kafka:0.9.0.1 95 | container_name: kafka1 96 | hostname: kafka1 97 | ports: 98 | - "9093:9093" 99 | environment: 100 | ZOOKEEPER_CONNECT: zookeeper0:2181,zookeeper1:2182,zookeeper2:2183/kafka 101 | BROKER_ID: 1 102 | BROKER_PORT: 9093 103 | ADVERTISED_HOST_NAME: kafka1 104 | HOST_NAME: kafka1 105 | ZOOKEEPER_SESSION_TIMEOUT: 3600000 106 | volumes: 107 | - /var/run/docker.sock:/var/run/docker.sock 108 | depends_on: 109 | - zookeeper0 110 | - zookeeper1 111 | - zookeeper2 112 | expose: 113 | - 9093 114 | # links: 115 | # - zookeeper 116 | kafka2: 117 | build: . 118 | build: 119 | context: . 120 | dockerfile: kafka.0.9.0.1.Dockerfile 121 | image: jason/kafka:0.9.0.1 122 | container_name: kafka2 123 | hostname: kafka2 124 | ports: 125 | - "9094:9094" 126 | environment: 127 | ZOOKEEPER_CONNECT: zookeeper0:2181,zookeeper1:2182,zookeeper2:2183/kafka 128 | BROKER_ID: 2 129 | BROKER_PORT: 9094 130 | ADVERTISED_HOST_NAME: kafka2 131 | HOST_NAME: kafka2 132 | ZOOKEEPER_SESSION_TIMEOUT: 3600000 133 | volumes: 134 | - /var/run/docker.sock:/var/run/docker.sock 135 | depends_on: 136 | - zookeeper0 137 | - zookeeper1 138 | - zookeeper2 139 | expose: 140 | - 9094 141 | # links: 142 | # - zookeeper -------------------------------------------------------------------------------- /kafka.0.9.0.1/kafka.0.9.0.1.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM centos:6.6 2 | 3 | RUN mkdir /etc/yum.repos.d/backup &&\ 4 | mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/backup/ &&\ 5 | curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo 6 | 7 | RUN yum -y install vim lsof wget tar bzip2 unzip vim-enhanced passwd sudo yum-utils hostname net-tools rsync man git make automake cmake patch logrotate python-devel libpng-devel libjpeg-devel pwgen python-pip 8 | 9 | RUN mkdir /opt/java &&\ 10 | wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u102-b14/jdk-8u102-linux-x64.tar.gz -P /opt/java 11 | 12 | ENV KAFKA_VERSION "0.9.0.1" 13 | 14 | RUN mkdir /opt/kafka &&\ 15 | wget http://apache.fayea.com/kafka/$KAFKA_VERSION/kafka_2.11-$KAFKA_VERSION.tgz -P /opt/kafka 16 | 17 | RUN tar zxvf /opt/java/jdk-8u102-linux-x64.tar.gz -C /opt/java &&\ 18 | JAVA_HOME=/opt/java/jdk1.8.0_102 &&\ 19 | sed -i "/^PATH/i export JAVA_HOME=$JAVA_HOME" /root/.bash_profile &&\ 20 | sed -i "s%^PATH.*$%&:$JAVA_HOME/bin%g" /root/.bash_profile &&\ 21 | source /root/.bash_profile 22 | 23 | RUN tar zxvf /opt/kafka/kafka*.tgz -C /opt/kafka &&\ 24 | sed -i 's/num.partitions.*$/num.partitions=3/g' /opt/kafka/kafka_2.11-$KAFKA_VERSION/config/server.properties 25 | 26 | RUN echo "source /root/.bash_profile" > /opt/kafka/start.sh &&\ 27 | echo "cd /opt/kafka/kafka_2.11-"$KAFKA_VERSION >> /opt/kafka/start.sh &&\ 28 | #echo "sed -i 's%zookeeper.connect=.*$%zookeeper.connect=zookeeper:2181%g' /opt/kafka/kafka_2.11-"$KAFKA_VERSION"/config/server.properties" >> /opt/kafka/start.sh &&\ 29 | echo "[ ! -z $""ZOOKEEPER_CONNECT"" ] && sed -i 's%.*zookeeper.connect=.*$%zookeeper.connect='$""ZOOKEEPER_CONNECT'""%g' /opt/kafka/kafka_2.11-"$KAFKA_VERSION"/config/server.properties" >> /opt/kafka/start.sh &&\ 30 | echo "[ ! -z $""BROKER_ID"" ] && sed -i 's%broker.id=.*$%broker.id='$""BROKER_ID'""%g' /opt/kafka/kafka_2.11-"$KAFKA_VERSION"/config/server.properties" >> /opt/kafka/start.sh &&\ 31 | echo "[ ! -z $""BROKER_PORT"" ] && sed -i 's%port=.*$%port='$""BROKER_PORT'""%g' /opt/kafka/kafka_2.11-"$KAFKA_VERSION"/config/server.properties" >> /opt/kafka/start.sh &&\ 32 | echo "sed -i 's%#advertised.host.name=.*$%advertised.host.name='$""(hostname -i)'""%g' /opt/kafka/kafka_2.11-"$KAFKA_VERSION"/config/server.properties" >> /opt/kafka/start.sh &&\ 33 | echo "[ ! -z $""ADVERTISED_HOST_NAME"" ] && sed -i 's%advertised.host.name=.*$%advertised.host.name='$""ADVERTISED_HOST_NAME'""%g' /opt/kafka/kafka_2.11-"$KAFKA_VERSION"/config/server.properties" >> /opt/kafka/start.sh &&\ 34 | echo "sed -i 's%#host.name=.*$%host.name='$""(hostname -i)'""%g' /opt/kafka/kafka_2.11-"$KAFKA_VERSION"/config/server.properties" >> /opt/kafka/start.sh &&\ 35 | echo "[ ! -z $""HOST_NAME"" ] && sed -i 's%host.name=.*$%host.name='$""HOST_NAME'""%g' /opt/kafka/kafka_2.11-"$KAFKA_VERSION"/config/server.properties" >> /opt/kafka/start.sh &&\ 36 | echo "[ ! -z $""ZOOKEEPER_SESSION_TIMEOUT"" ] && sed -i 's%zookeeper.connection.timeout.ms.*$%zookeeper.connection.timeout.ms='$""ZOOKEEPER_SESSION_TIMEOUT'""%g' /opt/kafka/kafka_2.11-"$KAFKA_VERSION"/config/server.properties" >> /opt/kafka/start.sh &&\ 37 | echo "delete.topic.enable=true" >> /opt/kafka/kafka_2.11-$KAFKA_VERSION/config/server.properties &&\ 38 | echo "bin/kafka-server-start.sh config/server.properties" >> /opt/kafka/start.sh &&\ 39 | chmod a+x /opt/kafka/start.sh 40 | 41 | RUN yum install -y nc 42 | 43 | EXPOSE 9092 44 | 45 | WORKDIR /opt/kafka/kafka_2.11-$KAFKA_VERSION 46 | 47 | ENTRYPOINT ["sh", "/opt/kafka/start.sh"] 48 | -------------------------------------------------------------------------------- /kafka.0.9.0.1/zookeeper.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM centos:6.6 2 | 3 | RUN mkdir /etc/yum.repos.d/backup &&\ 4 | mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/backup/ &&\ 5 | curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo 6 | 7 | RUN yum -y install vim lsof wget tar bzip2 unzip vim-enhanced passwd sudo yum-utils hostname net-tools rsync man git make automake cmake patch logrotate python-devel libpng-devel libjpeg-devel pwgen python-pip 8 | 9 | RUN mkdir /opt/java &&\ 10 | wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u102-b14/jdk-8u102-linux-x64.tar.gz -P /opt/java 11 | 12 | RUN tar zxvf /opt/java/jdk-8u102-linux-x64.tar.gz -C /opt/java &&\ 13 | JAVA_HOME=/opt/java/jdk1.8.0_102 &&\ 14 | sed -i "/^PATH/i export JAVA_HOME=$JAVA_HOME" /root/.bash_profile &&\ 15 | sed -i "s%^PATH.*$%&:$JAVA_HOME/bin%g" /root/.bash_profile &&\ 16 | source /root/.bash_profile 17 | 18 | ENV ZOOKEEPER_VERSION "3.4.6" 19 | 20 | RUN mkdir /opt/zookeeper &&\ 21 | wget http://mirror.olnevhost.net/pub/apache/zookeeper/zookeeper-$ZOOKEEPER_VERSION/zookeeper-$ZOOKEEPER_VERSION.tar.gz -P /opt/zookeeper 22 | 23 | RUN tar zxvf /opt/zookeeper/zookeeper*.tar.gz -C /opt/zookeeper 24 | 25 | RUN echo "source /root/.bash_profile" > /opt/zookeeper/start.sh &&\ 26 | echo "cp /opt/zookeeper/zookeeper-"$ZOOKEEPER_VERSION"/conf/zoo_sample.cfg /opt/zookeeper/zookeeper-"$ZOOKEEPER_VERSION"/conf/zoo.cfg" >> /opt/zookeeper/start.sh &&\ 27 | echo "[ ! -z $""ZOOKEEPER_PORT"" ] && sed -i 's%.*clientPort=.*$%clientPort='$""ZOOKEEPER_PORT'""%g' /opt/zookeeper/zookeeper-"$ZOOKEEPER_VERSION"/conf/zoo.cfg" >> /opt/zookeeper/start.sh &&\ 28 | echo "[ ! -z $""ZOOKEEPER_ID"" ] && mkdir -p /tmp/zookeeper && echo $""ZOOKEEPER_ID > /tmp/zookeeper/myid" >> /opt/zookeeper/start.sh &&\ 29 | echo "[[ ! -z $""ZOOKEEPER_SERVERS"" ]] && for server in $""ZOOKEEPER_SERVERS""; do echo $""server"" >> /opt/zookeeper/zookeeper-"$ZOOKEEPER_VERSION"/conf/zoo.cfg; done" >> /opt/zookeeper/start.sh &&\ 30 | echo "/opt/zookeeper/zookeeper-$"ZOOKEEPER_VERSION"/bin/zkServer.sh start-foreground" >> /opt/zookeeper/start.sh 31 | 32 | RUN yum install -y nc 33 | 34 | EXPOSE 2181 35 | 36 | WORKDIR /opt/zookeeper/zookeeper-$ZOOKEEPER_VERSION 37 | 38 | ENTRYPOINT ["sh", "/opt/zookeeper/start.sh"] 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | -------------------------------------------------------------------------------- /kafka.1.1.0/docker-compose.yaml: -------------------------------------------------------------------------------- 1 | version: '2.0' 2 | services: 3 | zookeeper0: 4 | build: 5 | context: . 6 | dockerfile: zookeeper.Dockerfile 7 | image: jason/zookeeper:3.4.12 8 | container_name: zookeeper0 9 | hostname: zookeeper0 10 | ports: 11 | - "12181:12181" 12 | - "12888:12888" 13 | - "13888:13888" 14 | expose: 15 | - 12181 16 | - 12888 17 | - 13888 18 | environment: 19 | ZOOKEEPER_PORT: 12181 20 | ZOOKEEPER_ID: 0 21 | ZOOKEEPER_SERVERS: server.0=zookeeper0:12888:13888 server.1=zookeeper1:12889:13889 server.2=zookeeper2:12890:13890 22 | zookeeper1: 23 | build: 24 | context: . 25 | dockerfile: zookeeper.Dockerfile 26 | image: jason/zookeeper:3.4.12 27 | container_name: zookeeper1 28 | hostname: zookeeper1 29 | ports: 30 | - "12182:12182" 31 | - "12889:12889" 32 | - "13889:13889" 33 | expose: 34 | - 12182 35 | - 12889 36 | - 13889 37 | environment: 38 | ZOOKEEPER_PORT: 12182 39 | ZOOKEEPER_ID: 1 40 | ZOOKEEPER_SERVERS: server.0=zookeeper0:12888:13888 server.1=zookeeper1:12889:13889 server.2=zookeeper2:12890:13890 41 | # depends_on: 42 | # - zookeeper0 43 | zookeeper2: 44 | build: 45 | context: . 46 | dockerfile: zookeeper.Dockerfile 47 | image: jason/zookeeper:3.4.12 48 | container_name: zookeeper2 49 | hostname: zookeeper2 50 | ports: 51 | - "12183:12183" 52 | - "12890:12890" 53 | - "13890:13890" 54 | expose: 55 | - 12183 56 | - 12890 57 | - 13890 58 | environment: 59 | ZOOKEEPER_PORT: 12183 60 | ZOOKEEPER_ID: 2 61 | ZOOKEEPER_SERVERS: server.0=zookeeper0:12888:13888 server.1=zookeeper1:12889:13889 server.2=zookeeper2:12890:13890 62 | # depends_on: 63 | # - zookeeper1 64 | kafka0: 65 | build: 66 | context: . 67 | dockerfile: kafka.1.1.0.Dockerfile 68 | image: jason/kafka:1.1.0 69 | container_name: kafka0 70 | hostname: kafka0 71 | ports: 72 | - "19092:19092" 73 | - "29092:29092" 74 | - "18083:18083" 75 | - "12345:12345" 76 | environment: 77 | ZOOKEEPER_CONNECT: zookeeper0:12181,zookeeper1:12182,zookeeper2:12183/kafka 78 | BROKER_ID: 0 79 | LISTENERS: PLAINTEXT://kafka0:19092,SSL://kafka0:29092 80 | ZOOKEEPER_SESSION_TIMEOUT: 3600000 81 | CONNECT_REST_PORT: 18083 82 | KAFKA_PROPERTY_AUTO_CREATE_TOPICS_ENABLE: "false" 83 | KAFKA_PROPERTY_SSL_CLIENT_AUTH: required 84 | KAFKA_PROPERTY_DELETE_TOPIC_ENABLE: "true" 85 | PORT_JMX: 12345 86 | KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" 87 | volumes: 88 | - /var/run/docker.sock:/var/run/docker.sock 89 | - ./ca:/opt/kafka/ca 90 | depends_on: 91 | - zookeeper0 92 | - zookeeper1 93 | - zookeeper2 94 | expose: 95 | - 9092 96 | # links: 97 | # - zookeeper 98 | kafka1: 99 | build: 100 | context: . 101 | dockerfile: kafka.1.1.0.Dockerfile 102 | image: jason/kafka:1.1.0 103 | container_name: kafka1 104 | hostname: kafka1 105 | ports: 106 | - "19093:19093" 107 | - "29093:29093" 108 | - "18084:18084" 109 | - "12346:12346" 110 | environment: 111 | ZOOKEEPER_CONNECT: zookeeper0:12181,zookeeper1:12182,zookeeper2:12183/kafka 112 | BROKER_ID: 1 113 | LISTENERS: PLAINTEXT://kafka1:19093,SSL://kafka1:29093 114 | ZOOKEEPER_SESSION_TIMEOUT: 3600000 115 | CONNECT_REST_PORT: 18084 116 | KAFKA_PROPERTY_AUTO_CREATE_TOPICS_ENABLE: "false" 117 | KAFKA_PROPERTY_SSL_CLIENT_AUTH: required 118 | KAFKA_PROPERTY_DELETE_TOPIC_ENABLE: "true" 119 | PORT_JMX: 12346 120 | KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=kafka1" 121 | volumes: 122 | - /var/run/docker.sock:/var/run/docker.sock 123 | - ./ca:/opt/kafka/ca 124 | depends_on: 125 | - zookeeper0 126 | - zookeeper1 127 | - zookeeper2 128 | expose: 129 | - 9093 130 | # links: 131 | # - zookeeper 132 | kafka2: 133 | build: . 134 | build: 135 | context: . 136 | dockerfile: kafka.1.1.0.Dockerfile 137 | image: jason/kafka:1.1.0 138 | container_name: kafka2 139 | hostname: kafka2 140 | ports: 141 | - "19094:19094" 142 | - "29094:29094" 143 | - "18085:18085" 144 | - "12347:12347" 145 | environment: 146 | ZOOKEEPER_CONNECT: zookeeper0:12181,zookeeper1:12182,zookeeper2:12183/kafka 147 | BROKER_ID: 2 148 | LISTENERS: PLAINTEXT://kafka2:19094,SSL://kafka2:29094 149 | ZOOKEEPER_SESSION_TIMEOUT: 3600000 150 | CONNECT_REST_PORT: 18085 151 | KAFKA_PROPERTY_AUTO_CREATE_TOPICS_ENABLE: "false" 152 | KAFKA_PROPERTY_SSL_CLIENT_AUTH: required 153 | KAFKA_PROPERTY_DELETE_TOPIC_ENABLE: "true" 154 | PORT_JMX: 12347 155 | KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=kafka2" 156 | volumes: 157 | - /var/run/docker.sock:/var/run/docker.sock 158 | - ./ca:/opt/kafka/ca 159 | depends_on: 160 | - zookeeper0 161 | - zookeeper1 162 | - zookeeper2 163 | expose: 164 | - 9094 165 | kafka3: 166 | build: 167 | context: . 168 | dockerfile: kafka.1.1.0.Dockerfile 169 | image: jason/kafka:1.1.0 170 | container_name: kafka3 171 | hostname: kafka3 172 | ports: 173 | - "19095:19095" 174 | - "29095:29095" 175 | - "18086:18086" 176 | - "12348:12348" 177 | environment: 178 | ZOOKEEPER_CONNECT: zookeeper0:12181,zookeeper1:12182,zookeeper2:12183/kafka 179 | BROKER_ID: 3 180 | LISTENERS: PLAINTEXT://kafka3:19095,SSL://kafka3:29095 181 | ZOOKEEPER_SESSION_TIMEOUT: 3600000 182 | CONNECT_REST_PORT: 18086 183 | KAFKA_PROPERTY_AUTO_CREATE_TOPICS_ENABLE: "false" 184 | KAFKA_PROPERTY_SSL_CLIENT_AUTH: required 185 | KAFKA_PROPERTY_DELETE_TOPIC_ENABLE: "true" 186 | PORT_JMX: 12348 187 | KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" 188 | volumes: 189 | - /var/run/docker.sock:/var/run/docker.sock 190 | - ./ca:/opt/kafka/ca 191 | depends_on: 192 | - zookeeper0 193 | - zookeeper1 194 | - zookeeper2 195 | expose: 196 | - 19095 197 | - 29095 198 | # links: 199 | # - zookeeper 200 | kafka4: 201 | build: 202 | context: . 203 | dockerfile: kafka.1.1.0.Dockerfile 204 | image: jason/kafka:1.1.0 205 | container_name: kafka4 206 | hostname: kafka4 207 | ports: 208 | - "19096:19096" 209 | - "29096:29096" 210 | - "18087:18087" 211 | - "12349:12349" 212 | environment: 213 | ZOOKEEPER_CONNECT: zookeeper0:12181,zookeeper1:12182,zookeeper2:12183/kafka 214 | BROKER_ID: 4 215 | LISTENERS: PLAINTEXT://kafka4:19096,SSL://kafka4:29096 216 | ZOOKEEPER_SESSION_TIMEOUT: 3600000 217 | CONNECT_REST_PORT: 18087 218 | KAFKA_PROPERTY_AUTO_CREATE_TOPICS_ENABLE: "false" 219 | KAFKA_PROPERTY_SSL_CLIENT_AUTH: required 220 | KAFKA_PROPERTY_DELETE_TOPIC_ENABLE: "true" 221 | PORT_JMX: 12349 222 | KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=kafka1" 223 | volumes: 224 | - /var/run/docker.sock:/var/run/docker.sock 225 | - ./ca:/opt/kafka/ca 226 | depends_on: 227 | - zookeeper0 228 | - zookeeper1 229 | - zookeeper2 230 | expose: 231 | - 29096 232 | - 19096 233 | # links: 234 | # - zookeeper 235 | kafka5: 236 | build: . 237 | build: 238 | context: . 239 | dockerfile: kafka.1.1.0.Dockerfile 240 | image: jason/kafka:1.1.0 241 | container_name: kafka5 242 | hostname: kafka5 243 | ports: 244 | - "19097:19097" 245 | - "29097:29097" 246 | - "18088:18088" 247 | - "12350:12350" 248 | environment: 249 | ZOOKEEPER_CONNECT: zookeeper0:12181,zookeeper1:12182,zookeeper2:12183/kafka 250 | BROKER_ID: 5 251 | LISTENERS: PLAINTEXT://kafka5:19097,SSL://kafka5:29097 252 | ZOOKEEPER_SESSION_TIMEOUT: 3600000 253 | CONNECT_REST_PORT: 18088 254 | KAFKA_PROPERTY_AUTO_CREATE_TOPICS_ENABLE: "false" 255 | KAFKA_PROPERTY_SSL_CLIENT_AUTH: required 256 | KAFKA_PROPERTY_DELETE_TOPIC_ENABLE: "true" 257 | PORT_JMX: 12350 258 | KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=kafka2" 259 | volumes: 260 | - /var/run/docker.sock:/var/run/docker.sock 261 | - ./ca:/opt/kafka/ca 262 | depends_on: 263 | - zookeeper0 264 | - zookeeper1 265 | - zookeeper2 266 | expose: 267 | - 19097 268 | - 29097 269 | kafka-manager: 270 | build: . 271 | build: 272 | context: . 273 | dockerfile: kafka-manager.Dockerfile 274 | image: jason/kafka-manager:1.3.3 275 | container_name: kafka-manager 276 | hostname: kafka-manager 277 | ports: 278 | - "38080:38080" 279 | environment: 280 | ZK_HOSTS: zookeeper0:12181,zookeeper1:12182,zookeeper2:12183 281 | PORT: 38080 282 | expose: 283 | - 38080 284 | -------------------------------------------------------------------------------- /kafka.1.1.0/kafka-manager.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM centos:6.6 2 | 3 | RUN mkdir /etc/yum.repos.d/backup &&\ 4 | mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/backup/ &&\ 5 | curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo 6 | 7 | RUN yum -y install nc vim lsof wget tar bzip2 unzip vim-enhanced passwd sudo yum-utils hostname net-tools rsync man git make automake cmake patch logrotate python-devel libpng-devel libjpeg-devel pwgen python-pip 8 | 9 | RUN mkdir /opt/java &&\ 10 | wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" https://edelivery.oracle.com/otn-pub/java/jdk/8u172-b11/a58eab1ec242421181065cdc37240b08/jdk-8u172-linux-x64.tar.gz -P /opt/java 11 | 12 | ENV JAVA_HOME "/opt/java/jdk1.8.0_172" 13 | 14 | RUN tar zxvf /opt/java/jdk-8u172-linux-x64.tar.gz -C /opt/java &&\ 15 | sed -i "/^PATH/i export JAVA_HOME="$JAVA_HOME /root/.bash_profile &&\ 16 | sed -i "s%^PATH.*$%&:"$JAVA_HOME"/bin%g" /root/.bash_profile &&\ 17 | source /root/.bash_profile 18 | 19 | ENV KAFKA_MANAGER_VERSION "1.3.3.17" 20 | 21 | RUN wget https://github.com/yahoo/kafka-manager/archive/$KAFKA_MANAGER_VERSION.zip -P /opt/ 22 | 23 | RUN unzip /opt/$KAFKA_MANAGER_VERSION.zip -d /opt/ &&\ 24 | mv /opt/kafka-manager-$KAFKA_MANAGER_VERSION /opt/kafka-manager 25 | 26 | RUN cd /opt/kafka-manager &&\ 27 | PATH=$JAVA_HOME/bin:$PATH &&\ 28 | ./sbt clean dist 29 | 30 | ENV PATH $JAVA_HOME/bin:$PATH 31 | 32 | RUN cp /opt/kafka-manager/target/universal/kafka-manager-$KAFKA_MANAGER_VERSION.zip /tmp/ &&\ 33 | rm -rf /opt/kafka-manager/ &&\ 34 | unzip /tmp/kafka-manager-$KAFKA_MANAGER_VERSION.zip -d /opt/ &&\ 35 | mv /opt/kafka-manager-$KAFKA_MANAGER_VERSION /opt/kafka-manager &&\ 36 | rm -rf /tmp/* /root/.sbt /root/.ivy2 37 | 38 | WORKDIR /opt/kafka-manager/ 39 | 40 | ENTRYPOINT ["bin/kafka-manager", "-Dhttp.port=38080"] 41 | 42 | -------------------------------------------------------------------------------- /kafka.1.1.0/kafka.1.1.0.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM centos:6.6 2 | 3 | RUN mkdir /etc/yum.repos.d/backup &&\ 4 | mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/backup/ &&\ 5 | curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo 6 | 7 | RUN yum -y install nc vim lsof wget tar bzip2 unzip vim-enhanced passwd sudo yum-utils hostname net-tools rsync man git make automake cmake patch logrotate python-devel libpng-devel libjpeg-devel pwgen python-pip tcpdump dstat sysstat 8 | 9 | RUN mkdir /opt/java &&\ 10 | wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" https://edelivery.oracle.com/otn-pub/java/jdk/8u172-b11/a58eab1ec242421181065cdc37240b08/jdk-8u172-linux-x64.tar.gz -P /opt/java 11 | 12 | ENV KAFKA_VERSION "1.1.0" 13 | 14 | RUN mkdir /opt/kafka &&\ 15 | wget http://www-eu.apache.org/dist/kafka/$KAFKA_VERSION/kafka_2.11-$KAFKA_VERSION.tgz -P /opt/kafka 16 | 17 | ENV JAVA_HOME "/opt/java/jdk1.8.0_172" 18 | 19 | RUN tar zxvf /opt/java/jdk-8u172-linux-x64.tar.gz -C /opt/java &&\ 20 | sed -i "/^PATH/i export JAVA_HOME="$JAVA_HOME /root/.bash_profile &&\ 21 | sed -i "s%^PATH.*$%&:"$JAVA_HOME"/bin%g" /root/.bash_profile &&\ 22 | source /root/.bash_profile 23 | 24 | RUN tar zxvf /opt/kafka/kafka*.tgz -C /opt/kafka &&\ 25 | sed -i 's/num.partitions.*$/num.partitions=3/g' /opt/kafka/kafka_2.11-$KAFKA_VERSION/config/server.properties 26 | 27 | RUN sed -i '0,/^if/s%^if%export JAVA_HOME='$JAVA_HOME'\nexport PATH=$PATH:$JAVA_HOME/bin\nif%' /opt/kafka/kafka_2.11-$KAFKA_VERSION/bin/kafka-run-class.sh 28 | 29 | COPY start_kafka.sh /opt/kafka 30 | 31 | 32 | EXPOSE 9092 33 | 34 | WORKDIR /opt/kafka/kafka_2.11-$KAFKA_VERSION 35 | 36 | ENTRYPOINT ["sh", "/opt/kafka/start_kafka.sh"] 37 | -------------------------------------------------------------------------------- /kafka.1.1.0/start_kafka.sh: -------------------------------------------------------------------------------- 1 | source /root/.bash_profile 2 | 3 | KAFKA_HOME=/opt/kafka/kafka_2.11-$KAFKA_VERSION 4 | 5 | pushd $KAFKA_HOME 6 | [ ! -z $ZOOKEEPER_CONNECT ] && sed -i 's%.*zookeeper.connect=.*$%zookeeper.connect='$ZOOKEEPER_CONNECT'%g' config/server.properties 7 | [ ! -z $BROKER_ID ] && sed -i 's%broker.id=.*$%broker.id='$BROKER_ID'%g' config/server.properties 8 | sed -i 's%#listeners=.*$%listeners=PLAINTEXT://'$(hostname -i)':9092%g' config/server.properties 9 | [ ! -z $LISTENERS ] && sed -i 's%listeners=.*$%listeners='$LISTENERS'%g' config/server.properties 10 | [ ! -z $ZOOKEEPER_SESSION_TIMEOUT ] && sed -i 's%zookeeper.connection.timeout.ms.*$%zookeeper.connection.timeout.ms='$ZOOKEEPER_SESSION_TIMEOUT'%g' config/server.properties 11 | [ ! -z $CONNECT_REST_PORT ] && echo rest.port=$CONNECT_REST_PORT >> /opt/kafka/kafka_2.11-$KAFKA_VERSION/config/connect-standalone.properties && echo -e '\nrest.port='$CONNECT_REST_PORT >> config/connect-distributed.properties 12 | [ ! -z $PORT_JMX ] && export JMX_PORT=$PORT_JMX 13 | popd 14 | 15 | 16 | if [[ $(echo $LISTENERS | tr '[:upper:]' '[:lower:]') =~ ssl ]] 17 | then 18 | pushd /opt/kafka 19 | mkdir /opt/kafka/ssl 20 | country=US ; [ ! -z $SSL_C ] && country=$SSL_C 21 | organization=UNKNOWN ; [ ! -z $SSL_O ] && organization=$SSL_O 22 | organizationalunit=UNKNOWN ; [ ! -z $SSL_OU ] && organizationalunit=$SSL_OU 23 | province=UNKNOWN ; [ ! -z $SSL_ST ] && province=$SSL_ST 24 | commonname=US ; [ ! -z $SSL_CN ] && commonname=$SSL_CN 25 | locality=UNKNOWN ; [ ! -z $SSL_L ] && locality=$SSL_L 26 | ssl_password=jasonguo ; [ ! -z $SSL_PASSWORD ] && ssl_password=$SSL_PASSWORD 27 | 28 | if [ -f ca/ca.crt ] || [ -f ca/ca.key ] 29 | then 30 | sleep 5s 31 | else 32 | openssl req -new -x509 -keyout ca/ca.key -out ca/ca.crt -days 365 -passout pass:$ssl_password -subj "/C="$country"/ST="$province"/L="$locality"/O="$organization"/OU="$organizationalunit"/CN="$commonname 33 | fi 34 | 35 | keytool -keystore ssl/$HOSTNAME.keystore.jks -alias $HOSTNAME -validity 365 -storepass $ssl_password -keypass $ssl_password -genkey -dname "CN="$commonname", OU="$organizationalunit", O="$organization", L="$locality", ST="$province", C="$country 36 | 37 | keytool -v -keystore ssl/$HOSTNAME.truststore.jks -alias CARoot -import -file ca/ca.crt -storepass $ssl_password <> config/server.properties 84 | fi 85 | fi 86 | done 87 | popd 88 | 89 | cd $KAFKA_HOME 90 | bin/kafka-server-start.sh config/server.properties 91 | -------------------------------------------------------------------------------- /kafka.1.1.0/zookeeper.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM centos:6.6 2 | 3 | RUN mkdir /etc/yum.repos.d/backup &&\ 4 | mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/backup/ &&\ 5 | curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo 6 | 7 | RUN yum -y install nc vim lsof wget tar bzip2 unzip vim-enhanced passwd sudo yum-utils hostname net-tools rsync man git make automake cmake patch logrotate python-devel libpng-devel libjpeg-devel pwgen python-pip 8 | 9 | RUN mkdir /opt/java &&\ 10 | wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" https://edelivery.oracle.com/otn-pub/java/jdk/8u172-b11/a58eab1ec242421181065cdc37240b08/jdk-8u172-linux-x64.tar.gz -P /opt/java 11 | 12 | ENV JAVA_HOME "/opt/java/jdk1.8.0_172" 13 | 14 | RUN tar zxvf /opt/java/jdk-8u172-linux-x64.tar.gz -C /opt/java &&\ 15 | sed -i "/^PATH/i export JAVA_HOME="$JAVA_HOME /root/.bash_profile &&\ 16 | sed -i "s%^PATH.*$%&:"$JAVA_HOME"/bin%g" /root/.bash_profile &&\ 17 | source /root/.bash_profile 18 | 19 | ENV ZOOKEEPER_VERSION "3.4.12" 20 | 21 | RUN mkdir /opt/zookeeper &&\ 22 | wget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-$ZOOKEEPER_VERSION/zookeeper-$ZOOKEEPER_VERSION.tar.gz -P /opt/zookeeper 23 | 24 | RUN tar zxvf /opt/zookeeper/zookeeper*.tar.gz -C /opt/zookeeper 25 | 26 | RUN echo "source /root/.bash_profile" > /opt/zookeeper/start.sh &&\ 27 | echo "cp /opt/zookeeper/zookeeper-"$ZOOKEEPER_VERSION"/conf/zoo_sample.cfg /opt/zookeeper/zookeeper-"$ZOOKEEPER_VERSION"/conf/zoo.cfg" >> /opt/zookeeper/start.sh &&\ 28 | echo "[ ! -z $""ZOOKEEPER_PORT"" ] && sed -i 's%.*clientPort=.*$%clientPort='$""ZOOKEEPER_PORT'""%g' /opt/zookeeper/zookeeper-"$ZOOKEEPER_VERSION"/conf/zoo.cfg" >> /opt/zookeeper/start.sh &&\ 29 | echo "[ ! -z $""ZOOKEEPER_ID"" ] && mkdir -p /tmp/zookeeper && echo $""ZOOKEEPER_ID > /tmp/zookeeper/myid" >> /opt/zookeeper/start.sh &&\ 30 | echo "[[ ! -z $""ZOOKEEPER_SERVERS"" ]] && for server in $""ZOOKEEPER_SERVERS""; do echo $""server"" >> /opt/zookeeper/zookeeper-"$ZOOKEEPER_VERSION"/conf/zoo.cfg; done" >> /opt/zookeeper/start.sh &&\ 31 | echo "/opt/zookeeper/zookeeper-$"ZOOKEEPER_VERSION"/bin/zkServer.sh start-foreground" >> /opt/zookeeper/start.sh 32 | 33 | EXPOSE 2181 34 | 35 | WORKDIR /opt/zookeeper/zookeeper-$ZOOKEEPER_VERSION 36 | 37 | ENTRYPOINT ["sh", "/opt/zookeeper/start.sh"] 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | --------------------------------------------------------------------------------