├── .gitignore ├── LICENSE ├── README.md ├── Vagrantfile ├── roles ├── java │ ├── defaults │ │ └── main.yml │ ├── tasks │ │ └── main.yml │ └── templates │ │ └── profile.java8.sh.j2 ├── kafka │ ├── defaults │ │ └── main.yml │ ├── handlers │ │ └── main.yml │ ├── tasks │ │ └── main.yml │ └── templates │ │ ├── kafka-properties.j2 │ │ └── kafka-systemd.j2 └── zookeeper │ ├── defaults │ └── main.yml │ ├── handlers │ └── main.yml │ ├── tasks │ └── main.yml │ └── templates │ ├── myid │ ├── zoo.cfg │ └── zookeeper-systemd.j2 ├── site.retry └── site.yml /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | 5 | # C extensions 6 | *.so 7 | 8 | # Distribution / packaging 9 | .Python 10 | env/ 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | lib/ 17 | lib64/ 18 | parts/ 19 | sdist/ 20 | var/ 21 | *.egg-info/ 22 | .installed.cfg 23 | *.egg 24 | 25 | # PyInstaller 26 | # Usually these files are written by a python script from a template 27 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 28 | *.manifest 29 | *.spec 30 | 31 | # Installer logs 32 | pip-log.txt 33 | pip-delete-this-directory.txt 34 | 35 | # Unit test / coverage reports 36 | htmlcov/ 37 | .tox/ 38 | .coverage 39 | .cache 40 | nosetests.xml 41 | coverage.xml 42 | 43 | # Translations 44 | *.mo 45 | *.pot 46 | 47 | # Django stuff: 48 | *.log 49 | 50 | # Sphinx documentation 51 | docs/_build/ 52 | 53 | # PyBuilder 54 | target/ 55 | .vagrant 56 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | License 2 | 3 | This software is licensed under the Apache 2 license, quoted below. 4 | 5 | Copyright (C) 2009-2016 Lloyd Chan 6 | 7 | Licensed under the Apache License, Version 2.0 (the "License"); you may not use this project except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0. 8 | 9 | Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. 10 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Kafka CentOS virtual cluster 2 | 3 | Easily create a local Kafka cluster w/ Zookeeper quorum via Vagrant + Ansible. Or just use the Ansible playbooks. 4 | 5 | The main differences between this and [Wirbelsturm](https://github.com/miguno/wirbelsturm) are: 6 | 7 | - Focus on Ansible and playbooks that can be used to provision Zk + Kafka w/o Vagrant 8 | - No Storm provisioning as of writing 9 | 10 | ## Usage 11 | 12 | Depending on your hardware and network connection, the script execution might take between 10-20 minutes. 13 | 14 | ### 1. Install Vagrant, VirtualBox and Ansible on your machine 15 | 16 | For Mac, this can be done with Homebrew: 17 | ``` 18 | brew install caskroom/cask/brew-cask 19 | brew cask install virtualbox 20 | brew install vagrant 21 | brew install ansible 22 | ``` 23 | 24 | Make sure you are running Ansible v1.7.2 or higher with `ansible --version`. 25 | 26 | For other systems, checkout the installation pages of [Vagrant](https://docs.vagrantup.com/v2/installation/), [VirtualBox](https://www.virtualbox.org/wiki/Downloads) and [Ansible](http://docs.ansible.com/intro_installation.html). 27 | 28 | ### 2. Clone this repo 29 | 30 | ``` 31 | git clone git@github.com:lloydmeta/ansible-kafka-cluster.git 32 | cd ansible-kafka-cluster 33 | ``` 34 | 35 | 36 | ### 3. Start the cluster with Vagrant 37 | 38 | ``` 39 | vagrant up 40 | ``` 41 | 42 | ### 4. SSH into the first and second Kafka nodes 43 | 44 | In separate terminals, run: 45 | 46 | ``` 47 | vagrant ssh kafka-node-1 48 | ``` 49 | 50 | ``` 51 | vagrant ssh kafka-node-2 52 | ``` 53 | 54 | ### 5. Create a replicated topic in kafka-node-1 and play with it 55 | 56 | ``` 57 | export LOG_DIR=/tmp # otherwise we get a warning 58 | cd /etc/kafka_2.11-0.8.2.1 59 | 60 | bin/kafka-topics.sh --create --zookeeper zk-node-1:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic 61 | 62 | # Look at the topic description to make sure it is indeed replicated 63 | bin/kafka-topics.sh --describe --zookeeper zk-node-1:2181 --topic my-replicated-topic 64 | 65 | # Send a few messages 66 | bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic 67 | ... 68 | my test message 1 69 | my test message 2 70 | ^C 71 | ``` 72 | 73 | ### 6. In kafka-node-2, consume a few messages 74 | ``` 75 | export LOG_DIR=/tmp # otherwise we get a warning 76 | cd /etc/kafka_2.11-0.8.2.1 77 | 78 | bin/kafka-console-consumer.sh --zookeeper zk-node-1:2181 --from-beginning --topic my-replicated-topic 79 | ... 80 | my test message 1 81 | my test message 2 82 | ^C 83 | ``` 84 | 85 | ### 7. Suspend a Kafka node 86 | 87 | From your host, take down a Kafka node 88 | 89 | ``` 90 | vagrant suspend kafka-node-3 91 | ``` 92 | 93 | Wait 6 seconds; this is the default heartbeat frequency for Kafka that tells Zookeeper that a node has gone down. 94 | 95 | Then verify on any other Kafka node that your replicated topic is now missing one member under isr (in-sync replica), but otherwise still works. If this takes some time, 96 | 97 | ``` 98 | bin/kafka-topics.sh --describe --zookeeper zk-node-1:2181 --topic my-replicated-topic 99 | ``` 100 | 101 | Feel free to send and consume messages while the node is down. Note though, that you may have to restart some of the shell-based consumer/producers that are already running because they don't work well in disaster situations.. 102 | 103 | ### 8. Bring the Kafka node back up, put it to sleep, etc. 104 | 105 | -------------------------------------------------------------------------------- /Vagrantfile: -------------------------------------------------------------------------------- 1 | # -*- mode: ruby -*- 2 | # vi: set ft=ruby : 3 | 4 | require 'ipaddr' 5 | 6 | unless Vagrant.has_plugin?("vagrant-hostmanager") 7 | raise 'vagrant-hostmanager is not installed! run "vagrant plugin install vagrant-hostmanager" to fix' 8 | end 9 | 10 | VAGRANTFILE_API_VERSION = "2" 11 | 12 | class IpAssigner 13 | 14 | def self.next_ip(current_ip) 15 | IPAddr.new(current_ip).succ.to_s 16 | end 17 | 18 | def self.generate(start_ip, number_of_addresses) 19 | number_of_addresses.times.inject([start_ip]) { |acc, i| 20 | nekst = next_ip(acc.last) 21 | acc + [nekst] 22 | } 23 | end 24 | 25 | end 26 | 27 | Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| 28 | 29 | config.vm.box = 'hfm4/centos7' 30 | config.ssh.insert_key = false 31 | config.hostmanager.enabled = true 32 | config.hostmanager.manage_host = true 33 | config.hostmanager.ignore_private_ip = false 34 | config.hostmanager.include_offline = true 35 | 36 | accept_oracle_licence = true # set to false if you don't agree (will not install Java8 for you) 37 | 38 | private_network_begin = "192.168.5.100" # private ip will start incrementing from this 39 | zk_vm_memory_mb = 256 40 | zk_port = 2181 41 | kafka_vm_memory_mb = 512 42 | kafka_port = 9092 43 | 44 | # < ------- These need to be set in group vars if using Ansible w/o Vagrant ------- 45 | 46 | # The follwing variables will need to be passed manually if you want to use the Ansible 47 | # playbooks w/o Vagrant. They are set in this Vagrantfile in this manner because it allows us to easily 48 | # increase or decrease the cluster sizes. 49 | 50 | # Note that zk_id must be unique for each host in the cluster. It should ideally not change 51 | # throughout the lifetime of the Zookeeper installation on a given machine. 52 | zk_cluster_info = { 53 | 'zk-node-1' => { :zk_id => 1 }, 54 | 'zk-node-2' => { :zk_id => 2 }, 55 | 'zk-node-3' => { :zk_id => 3 } 56 | } 57 | 58 | # Note that broker_id must be unique for each host in the cluster. It should ideally not change 59 | # throughout the lifetime of the Kafka installation on a given machine. 60 | kafka_cluster_info = { 61 | 'kafka-node-1' => { :broker_id => 1 }, 62 | 'kafka-node-2' => { :broker_id => 2 }, 63 | 'kafka-node-3' => { :broker_id => 3 } 64 | } 65 | 66 | ## ------- These need to be set in group vars if using Ansible w/o Vagrant ------- > 67 | 68 | # Helper to make new ips 69 | zk_ips = IpAssigner.generate( 70 | private_network_begin, 71 | zk_cluster_info.size) 72 | 73 | kafka_ips = IpAssigner.generate( 74 | IpAssigner.next_ip(zk_ips.last || private_network_begin), 75 | kafka_cluster_info.size) 76 | 77 | zk_cluster = Hash[zk_cluster_info.map.with_index { |(k, v), idx| 78 | [k, v.merge( 79 | :ip => zk_ips[idx], 80 | :memory => zk_vm_memory_mb, 81 | :client_port => zk_port, 82 | :client_forward_to => zk_port + idx )] 83 | }] 84 | 85 | kafka_cluster = Hash[kafka_cluster_info.map.with_index { |(k, v), idx| 86 | [k, v.merge( 87 | :ip => kafka_ips[idx], 88 | :memory => kafka_vm_memory_mb, 89 | :client_port => kafka_port, 90 | :client_forward_to => kafka_port + idx )] 91 | }] 92 | 93 | total_cluster = zk_cluster.merge(kafka_cluster) 94 | 95 | total_cluster.each_with_index do |(short_name, info), idx| 96 | 97 | config.vm.define short_name do |host| 98 | host.vm.network :forwarded_port, guest: info[:client_port], host: info[:client_forward_to] 99 | host.vm.network :private_network, ip: info[:ip] 100 | host.vm.hostname = short_name 101 | host.vm.provider :virtualbox do |vb| 102 | vb.customize ["modifyvm", :id, "--memory", info[:memory]] 103 | end 104 | 105 | # This allows us to provision everything in one go, in parallel. 106 | if idx == (total_cluster.size - 1) 107 | host.vm.provision :ansible do |ansible| 108 | ansible.playbook = "site.yml" 109 | ansible.groups = { 110 | "zk" => zk_cluster.keys, 111 | "kafka" => kafka_cluster.keys 112 | } 113 | ansible.verbose = 'vv' 114 | ansible.sudo = true 115 | ansible.limit = 'all' # otherwise, Ansible only runs on the current host... 116 | ansible.extra_vars = { 117 | accept_oracle_licence: accept_oracle_licence, 118 | vagrant_zk_client_port: zk_port, 119 | vagrant_zk_cluster_info: zk_cluster_info, 120 | vagrant_kafka_cluster_info: kafka_cluster_info 121 | } 122 | end 123 | end 124 | 125 | end 126 | 127 | end 128 | 129 | end 130 | -------------------------------------------------------------------------------- /roles/java/defaults/main.yml: -------------------------------------------------------------------------------- 1 | jdk8_version: '1.8.0_40' 2 | jdk8_pkg_version: '8' 3 | symlink_to: 'java8' 4 | src_base_dir: "/tmp" 5 | accept_oracle_licence: false 6 | -------------------------------------------------------------------------------- /roles/java/tasks/main.yml: -------------------------------------------------------------------------------- 1 | - name: install wget 2 | yum: name=wget state=present 3 | tags: java8 4 | 5 | - name: create /usr/java directory 6 | file: path=/usr/java state=directory owner=root group=root mode=0755 7 | tags: java8 8 | 9 | - name: check jdk8 package downloaded 10 | command: test -f {{ src_base_dir }}/jdk-{{ jdk8_pkg_version }}-linux-x64.tar.gz 11 | register: jdk8_downloaded 12 | failed_when: jdk8_downloaded.rc not in [0, 1] 13 | changed_when: False 14 | tags: java8 15 | 16 | - name: download jdk8 package 17 | command: > 18 | wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u40-b26/jdk-8u40-linux-x64.tar.gz" -O {{ src_base_dir }}/jdk-{{ jdk8_pkg_version }}-linux-x64.tar.gz 19 | tags: java8 20 | when: jdk8_downloaded.rc == 1 21 | 22 | - name: extract jdk8 23 | command: > 24 | tar -xf {{ src_base_dir }}/jdk-{{ jdk8_pkg_version }}-linux-x64.tar.gz 25 | chdir=/usr/java 26 | creates=/usr/java/jdk{{ jdk8_version }} 27 | tags: java8 28 | 29 | - name: symlink to /usr/local 30 | file: > 31 | state=link 32 | src=/usr/java/jdk{{ jdk8_version }} 33 | dest=/usr/local/{{ symlink_to }} 34 | owner=root group=root 35 | tags: java8 36 | 37 | - name: /etc/profile.d/java8.sh 38 | template: > 39 | src=profile.java8.sh.j2 40 | dest=/etc/profile.d/{{ symlink_to }}.sh 41 | owner=root group=root mode=0644 42 | tags: java8 43 | -------------------------------------------------------------------------------- /roles/java/templates/profile.java8.sh.j2: -------------------------------------------------------------------------------- 1 | JAVA_HOME=/usr/local/{{ symlink_to }} 2 | export JAVA_HOME 3 | PATH=$JAVA_HOME/bin:$PATH 4 | -------------------------------------------------------------------------------- /roles/kafka/defaults/main.yml: -------------------------------------------------------------------------------- 1 | kafka: 2 | version: 0.8.2.1 3 | scala_version: 2.11 4 | mirror: http://ftp.meisei-u.ac.jp/mirror/apache/dist 5 | data_dir: /var/lib/kafka 6 | install_dir: "/etc" 7 | # This does not have to be every Zookeeper host, but the more the better 8 | # by default, we assume this is run at the same time as Zookeeper provisioning 9 | zk_hosts: "{{ groups['zk'] }}" # This does not have to be every Zookeeper host 10 | zk_client_port: "{{ vagrant_zk_client_port }}"# Should be set properly if not using Vagrant 11 | kafka_cluster_info: "{{ vagrant_kafka_cluster_info }}" 12 | # If not using Vagrant, 13 | # Should be a list of node hostnames to node info 14 | # Example: 15 | # kafka_cluster_info: 16 | # kafka-node-1: 17 | # broker_id: 1 18 | # kafka-node-2: 19 | # broker_id: 2 20 | # kafka-node-3: 21 | # broker_id: 3 22 | -------------------------------------------------------------------------------- /roles/kafka/handlers/main.yml: -------------------------------------------------------------------------------- 1 | - name: start kafka 2 | service: name=kafka state=started enabled=yes 3 | 4 | - name: restart kafka 5 | service: name=kafka state=restarted enabled=yes 6 | -------------------------------------------------------------------------------- /roles/kafka/tasks/main.yml: -------------------------------------------------------------------------------- 1 | - name: disable firewalld 2 | service: name=firewalld state=stopped enabled=no 3 | ignore_errors: True #when firewalld is stopped 4 | tags: kafka 5 | 6 | - name: install tools 7 | yum: name="@Development tools" state=present 8 | tags: kafka 9 | 10 | - name: create group 11 | group: 12 | name: kafka 13 | state: present 14 | tags: kafka 15 | 16 | - name: create user 17 | user: 18 | name: kafka 19 | group: kafka 20 | tags: kafka 21 | 22 | - name: Setting internal variable 23 | set_fact: 24 | kafka_name: kafka_{{ kafka.scala_version }}-{{ kafka.version }} 25 | tags: kafka 26 | 27 | - name: Setting internal variable 28 | set_fact: 29 | kafka_dir: "{{ kafka.install_dir }}/{{ kafka_name }}" 30 | tags: kafka 31 | 32 | - name: check if tar has been downloaded 33 | command: test -f /tmp/{{ kafka_name }}.tgz 34 | register: kafka_tar_downloaded 35 | failed_when: kafka_tar_downloaded.rc not in [0, 1] 36 | changed_when: False 37 | tags: kafka 38 | 39 | - name: Ensure Kafka tar is downloaded 40 | get_url: 41 | url: "{{ kafka.mirror }}/kafka/{{ kafka.version }}/{{ kafka_name }}.tgz" 42 | dest: /tmp 43 | tags: kafka 44 | when: kafka_tar_downloaded.rc == 1 45 | 46 | - name: Ensure tar is extracted 47 | command: tar xzf /tmp/{{ kafka_name }}.tgz chdir="{{ kafka.install_dir }}" 48 | tags: kafka 49 | 50 | # Config and start Kafka 51 | - name: Ensures data dir {{ kafka.data_dir }} exists 52 | file: 53 | path: "{{ kafka.data_dir }}" 54 | state: directory 55 | owner: kafka 56 | group: kafka 57 | tags: kafka 58 | 59 | - name: Copy real config 60 | template: 61 | src: kafka-properties.j2 62 | dest: "{{ kafka_dir }}/config/real-server.properties" 63 | notify: restart kafka 64 | tags: kafka 65 | 66 | - name: systemd start script 67 | template: 68 | src: kafka-systemd.j2 69 | dest: /etc/systemd/system/kafka.service 70 | owner: root 71 | group: root 72 | mode: 644 73 | notify: start kafka 74 | tags: kafka 75 | -------------------------------------------------------------------------------- /roles/kafka/templates/kafka-properties.j2: -------------------------------------------------------------------------------- 1 | # Licensed to the Apache Software Foundation (ASF) under one or more 2 | # contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright ownership. 4 | # The ASF licenses this file to You under the Apache License, Version 2.0 5 | # (the "License"); you may not use this file except in compliance with 6 | # the License. You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | # see kafka.server.KafkaConfig for additional details and defaults 16 | 17 | ############################# Server Basics ############################# 18 | 19 | # The id of the broker. This must be set to a unique integer for each broker. 20 | broker.id={{ kafka_cluster_info[ansible_hostname]["broker_id"] }} 21 | 22 | ############################# Socket Server Settings ############################# 23 | 24 | # The port the socket server listens on 25 | port=9092 26 | 27 | # Hostname the broker will bind to. If not set, the server will bind to all interfaces 28 | #host.name=localhost 29 | 30 | # Hostname the broker will advertise to producers and consumers. If not set, it uses the 31 | # value for "host.name" if configured. Otherwise, it will use the value returned from 32 | # java.net.InetAddress.getCanonicalHostName(). 33 | #advertised.host.name= 34 | 35 | # The port to publish to ZooKeeper for clients to use. If this is not set, 36 | # it will publish the same port that the broker binds to. 37 | #advertised.port= 38 | 39 | # The number of threads handling network requests 40 | num.network.threads=3 41 | 42 | # The number of threads doing disk I/O 43 | num.io.threads=8 44 | 45 | # The send buffer (SO_SNDBUF) used by the socket server 46 | socket.send.buffer.bytes=102400 47 | 48 | # The receive buffer (SO_RCVBUF) used by the socket server 49 | socket.receive.buffer.bytes=102400 50 | 51 | # The maximum size of a request that the socket server will accept (protection against OOM) 52 | socket.request.max.bytes=104857600 53 | 54 | 55 | ############################# Log Basics ############################# 56 | 57 | # A comma seperated list of directories under which to store log files 58 | log.dirs={{ kafka.data_dir }} 59 | 60 | # The default number of log partitions per topic. More partitions allow greater 61 | # parallelism for consumption, but this will also result in more files across 62 | # the brokers. 63 | num.partitions=1 64 | 65 | # The number of threads per data directory to be used for log recovery at startup and flushing at shutdown. 66 | # This value is recommended to be increased for installations with data dirs located in RAID array. 67 | num.recovery.threads.per.data.dir=1 68 | 69 | ############################# Log Flush Policy ############################# 70 | 71 | # Messages are immediately written to the filesystem but by default we only fsync() to sync 72 | # the OS cache lazily. The following configurations control the flush of data to disk. 73 | # There are a few important trade-offs here: 74 | # 1. Durability: Unflushed data may be lost if you are not using replication. 75 | # 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush. 76 | # 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to exceessive seeks. 77 | # The settings below allow one to configure the flush policy to flush data after a period of time or 78 | # every N messages (or both). This can be done globally and overridden on a per-topic basis. 79 | 80 | # The number of messages to accept before forcing a flush of data to disk 81 | #log.flush.interval.messages=10000 82 | 83 | # The maximum amount of time a message can sit in a log before we force a flush 84 | #log.flush.interval.ms=1000 85 | 86 | ############################# Log Retention Policy ############################# 87 | 88 | # The following configurations control the disposal of log segments. The policy can 89 | # be set to delete segments after a period of time, or after a given size has accumulated. 90 | # A segment will be deleted whenever *either* of these criteria are met. Deletion always happens 91 | # from the end of the log. 92 | 93 | # The minimum age of a log file to be eligible for deletion 94 | log.retention.hours=168 95 | 96 | # A size-based retention policy for logs. Segments are pruned from the log as long as the remaining 97 | # segments don't drop below log.retention.bytes. 98 | #log.retention.bytes=1073741824 99 | 100 | # The maximum size of a log segment file. When this size is reached a new log segment will be created. 101 | log.segment.bytes=1073741824 102 | 103 | # The interval at which log segments are checked to see if they can be deleted according 104 | # to the retention policies 105 | log.retention.check.interval.ms=300000 106 | 107 | # By default the log cleaner is disabled and the log retention policy will default to just delete segments after their retention expires. 108 | # If log.cleaner.enable=true is set the cleaner will be enabled and individual logs can then be marked for log compaction. 109 | log.cleaner.enable=false 110 | 111 | ############################# Zookeeper ############################# 112 | 113 | # Zookeeper connection string (see zookeeper docs for details). 114 | # This is a comma separated host:port pairs, each corresponding to a zk 115 | # server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002". 116 | # You can also append an optional chroot string to the urls to specify the 117 | # root directory for all kafka znodes. 118 | zookeeper.connect={{ zk_hosts | join(":{0},".format(zk_client_port)) }}:{{ zk_client_port }} 119 | 120 | # Timeout in ms for connecting to zookeeper 121 | zookeeper.connection.timeout.ms=6000 122 | -------------------------------------------------------------------------------- /roles/kafka/templates/kafka-systemd.j2: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=Kafka 3 | Before= 4 | After=network.target 5 | 6 | [Service] 7 | User=kafka 8 | Environment=JAVA_HOME=/usr/local/java8 9 | CHDIR= {{ kafka.data_dir }} 10 | ExecStart={{ kafka_dir }}/bin/kafka-server-start.sh {{ kafka_dir }}/config/real-server.properties 11 | Restart=on-abort 12 | 13 | [Install] 14 | WantedBy=multi-user.target 15 | -------------------------------------------------------------------------------- /roles/zookeeper/defaults/main.yml: -------------------------------------------------------------------------------- 1 | zk: 2 | version: 3.4.6 3 | mirror: http://ftp.meisei-u.ac.jp/mirror/apache/dist 4 | data_dir: /var/lib/zookeeper 5 | install_dir: "/etc" 6 | tick_time: 2000 7 | zk_cluster_info: "{{ vagrant_zk_cluster_info }}" 8 | # If not using Vagrant, 9 | # Should be a list of node hostnames to node info 10 | # Example: 11 | # zk_cluster_info: 12 | # zk-node-1: 13 | # zk_id: 1 14 | # zk-node-2: 15 | # zk_id: 2 16 | # zk-node-3: 17 | # zk_id: 3 18 | -------------------------------------------------------------------------------- /roles/zookeeper/handlers/main.yml: -------------------------------------------------------------------------------- 1 | - name: start zookeeper 2 | service: name=zookeeper state=started enabled=yes 3 | 4 | - name: restart zookeeper 5 | service: name=zookeeper state=restarted enabled=yes 6 | -------------------------------------------------------------------------------- /roles/zookeeper/tasks/main.yml: -------------------------------------------------------------------------------- 1 | - name: disable firewalld 2 | service: name=firewalld state=stopped enabled=no 3 | ignore_errors: True #when firewalld is stopped 4 | tags: zookeeper 5 | 6 | # Otherwise, Zookeeper ends up not working when using ailases 7 | - name: remove own host from 127.0.0.1 aliases 8 | lineinfile: > 9 | dest=/etc/hosts 10 | state=present 11 | regexp='^(127\.0\.0\.1.*){{ inventory_hostname }}(.*)$' 12 | line='\1\2' 13 | backup=yes 14 | backrefs=yes 15 | tags: zookeeper 16 | notify: restart zookeeper 17 | 18 | - name: create group 19 | group: 20 | name: zookeeper 21 | state: present 22 | tags: zookeeper 23 | 24 | - name: create user 25 | user: 26 | name: zookeeper 27 | group: zookeeper 28 | tags: zookeeper 29 | 30 | - name: Setting internal variable 31 | set_fact: 32 | zk_name: zookeeper-{{ zk.version }} 33 | tags: zookeeper 34 | 35 | - name: Zookeeper | Setting internal variable 36 | set_fact: 37 | zk_dir: "{{ zk.install_dir }}/{{ zk_name }}" 38 | tags: zookeeper 39 | 40 | - name: check if tar has been downloaded 41 | command: test -f /tmp/{{ zk_name }}.tar.gz 42 | register: zookeeper_tar_downloaded 43 | failed_when: zookeeper_tar_downloaded.rc not in [0, 1] 44 | changed_when: False 45 | tags: zookeeper 46 | 47 | - name: Ensure Zookeeper tar is downloaded 48 | get_url: 49 | url: "{{ zk.mirror }}/zookeeper/{{ zk_name }}/{{ zk_name }}.tar.gz" 50 | dest: /tmp 51 | tags: zookeeper 52 | when: zookeeper_tar_downloaded.rc == 1 53 | 54 | - name: Ensure tar is extracted 55 | command: tar xzf /tmp/{{ zk_name }}.tar.gz chdir="{{ zk.install_dir }}" 56 | tags: zookeeper 57 | 58 | # Config and start Zookeeper 59 | - name: Ensure zoo.conf is present 60 | template: 61 | src: zoo.cfg 62 | dest: "{{ zk_dir }}/conf/zoo.cfg" 63 | notify: restart zookeeper 64 | tags: zookeeper 65 | 66 | - name: Ensures data dir {{ zk.data_dir }} exists 67 | file: 68 | path: "{{ zk.data_dir }}" 69 | state: directory 70 | owner: zookeeper 71 | group: zookeeper 72 | tags: zookeeper 73 | 74 | - name: Ensure myid is set in {{ zk.data_dir }}/myid 75 | template: 76 | src: myid 77 | dest: "{{ zk.data_dir }}/myid" 78 | notify: restart zookeeper 79 | tags: zookeeper 80 | 81 | - name: systemd start script 82 | template: 83 | src: zookeeper-systemd.j2 84 | dest: /etc/systemd/system/zookeeper.service 85 | owner: root 86 | group: root 87 | mode: 644 88 | notify: start zookeeper 89 | tags: zookeeper 90 | -------------------------------------------------------------------------------- /roles/zookeeper/templates/myid: -------------------------------------------------------------------------------- 1 | {{ zk_cluster_info[inventory_hostname]["zk_id"] }} 2 | -------------------------------------------------------------------------------- /roles/zookeeper/templates/zoo.cfg: -------------------------------------------------------------------------------- 1 | tickTime={{ zk.tick_time }} 2 | dataDir={{ zk.data_dir }} 3 | clientPort=2181 4 | initLimit=5 5 | syncLimit=2 6 | {% for name in groups["zk"] %} 7 | server.{{ loop.index }}={{ name }}:2888:3888 8 | {% endfor %} 9 | -------------------------------------------------------------------------------- /roles/zookeeper/templates/zookeeper-systemd.j2: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=Zookeeper 3 | Before= 4 | After=network.target 5 | 6 | [Service] 7 | Type=forking 8 | User=zookeeper 9 | Environment=ZOO_LOG_DIR={{ zk.data_dir }}/log JAVA_HOME=/usr/local/java8 10 | CHDIR= {{ zk.data_dir }} 11 | ExecStart={{ zk_dir }}/bin/zkServer.sh start 12 | Restart=on-abort 13 | 14 | [Install] 15 | WantedBy=multi-user.target 16 | -------------------------------------------------------------------------------- /site.retry: -------------------------------------------------------------------------------- 1 | zk-node-1 2 | zk-node-2 3 | zk-node-3 4 | -------------------------------------------------------------------------------- /site.yml: -------------------------------------------------------------------------------- 1 | - hosts: all 2 | roles: 3 | - { role: java, when: accept_oracle_licence } 4 | 5 | - hosts: zk 6 | roles: 7 | - zookeeper 8 | 9 | - hosts: kafka 10 | roles: 11 | - kafka 12 | --------------------------------------------------------------------------------