├── README.md ├── kafka-broker-SERVER_NUMBER.jass.conf ├── server-sasl-brokers-zookeeper.properties ├── zookeeper ├── kafka ├── README-Kerberos.md ├── README-Kafka-brokers-SASL-only.md └── README-Zookeeper.md /README.md: -------------------------------------------------------------------------------- 1 | # Instructions for setting up Kafka, ZooKeeper and Kerberos with SASL and SSL 2 | 3 | See the [README-Kerberos.md](README-Kerberos.md) file for instructions for setting up Kerberos. 4 | 5 | See the [README-Zookeeper.md](README-Zookeeper.md) file for instructions for setting up ZooKeeper with SASL. 6 | 7 | See the [README-Kafka.md](README-Kafka-brokers-SASL-only.md) file for instructions for setting up Kafka brokers with SASL. 8 | 9 | (TODO) See the [README-Kafka.md](README-Kafka-brokers-add-SSSL.md) file for instructions for adding SSL to your Kafka brokers. 10 | -------------------------------------------------------------------------------- /kafka-broker-SERVER_NUMBER.jass.conf: -------------------------------------------------------------------------------- 1 | // This is for the broker acting as a server to Kafka clients 2 | KafkaServer { 3 | com.sun.security.auth.module.Krb5LoginModule required 4 | useKeyTab=true 5 | keyTab="/home/kafka/kafka_2.11-1.0.0/config/kafka.server-SERVER_NUMBER.yourdomain.com.keytab" 6 | storeKey=true 7 | useTicketCache=false 8 | serviceName="kafka" 9 | principal="kafka/server-SERVER_NUMBER.yourdomain.com@YOURDOMAIN.COM"; 10 | }; 11 | 12 | // This is for the broker acting as a client to ZooKeeper 13 | Client { 14 | com.sun.security.auth.module.Krb5LoginModule required 15 | useKeyTab=true 16 | keyTab="/home/kafka/kafka_2.11-1.0.0/config/kafka.server-SERVER_NUMBER.yourdomain.com.keytab" 17 | storeKey=true 18 | useTicketCache=false 19 | serviceName="zookeeper" 20 | principal="kafka/server-SERVER_NUMBER.yourdomain.com@YOURDOMAIN.COM"; 21 | }; 22 | -------------------------------------------------------------------------------- /server-sasl-brokers-zookeeper.properties: -------------------------------------------------------------------------------- 1 | broker.id=1 2 | listeners=SASL_PLAINTEXT://0.0.0.0:9092 3 | advertised.listeners=SASL_PLAINTEXT://server-01.yourdomain.com:9092 4 | security.inter.broker.protocol=SASL_PLAINTEXT 5 | super.users=kafka 6 | delete.topic.enable=true 7 | auto.create.topics.enable=false 8 | num.network.threads=3 9 | num.io.threads=8 10 | socket.send.buffer.bytes=102400 11 | socket.receive.buffer.bytes=102400 12 | socket.request.max.bytes=104857600 13 | log.dirs=/home/kafka/log 14 | num.partitions=1 15 | num.recovery.threads.per.data.dir=1 16 | offsets.topic.replication.factor=1 17 | transaction.state.log.replication.factor=1 18 | transaction.state.log.min.isr=1 19 | log.retention.hours=168 20 | log.segment.bytes=1073741824 21 | log.retention.check.interval.ms=300000 22 | zookeeper.connect=zookeeper-server-01.yourdomain.com:2181,zookeeper-server-02.yourdomain.com:2181,zookeeper-server-03.yourdomain.com:2181/apps/kafka-cluster-demo 23 | zookeeper.connection.timeout.ms=6000 24 | group.initial.rebalance.delay.ms=0 25 | -------------------------------------------------------------------------------- /zookeeper: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # 3 | # This script was adapted from https://gist.github.com/bejean/b9ff72c6d2143e16e35d 4 | # 5 | # Purpose: This script starts and stops the Zookeeper daemon 6 | # 7 | # chkconfig: - 90 10 8 | # description: ZooKeeper daemon 9 | 10 | ### BEGIN INIT INFO 11 | # Provides: zookeeper 12 | # Required-Start: $network $local_fs $remote_fs 13 | # Required-Stop: $network $local_fs $remote_fs 14 | # Should-Start: 15 | # Should-Stop: 16 | # Default-Start: 2 3 4 5 17 | # Default-Stop: 0 1 6 18 | # Short-Description: Controls Zookeeper as a Service 19 | ### END INIT INFO 20 | 21 | ZK_INSTALL_DIR="/opt/zookeeper" 22 | 23 | if [ ! -d "$ZK_INSTALL_DIR" ]; then 24 | echo "$ZK_INSTALL_DIR not found! Please check the ZK_INSTALL_DIR setting in your $0 script." 25 | exit 1 26 | fi 27 | 28 | # Specify the user to run Zookeeper as; if not set, then ZooKeeper will not run. 29 | RUNAS="zookeeper" 30 | 31 | # verify the specified run as user exists 32 | runas_uid="`id -u "$RUNAS"`" 33 | if [ $? -ne 0 ]; then 34 | echo "User $RUNAS not found! Please create the $RUNAS user before running this script." 35 | exit 1 36 | fi 37 | 38 | case "$1" in 39 | start|stop|restart|status) 40 | ZK_CMD="$1" 41 | ;; 42 | *) 43 | echo "Usage: $0 {start|stop|restart|status}" 44 | exit 45 | esac 46 | 47 | if [ -n "$RUNAS" ]; then 48 | su -c "\"$ZK_INSTALL_DIR/bin/zkServer.sh\" $ZK_CMD" - "$RUNAS" 49 | fi 50 | 51 | exit 0 52 | -------------------------------------------------------------------------------- /kafka: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # 3 | # This script was adapted from https://gist.github.com/bejean/b9ff72c6d2143e16e35d and 4 | # https://gist.github.com/superscott/a1c67871cdd54b0c8693 5 | # 6 | # Other references: 7 | # https://stackoverflow.com/questions/41014742/where-will-be-nohup-file-created-stored 8 | # 9 | # Purpose: This script starts and stops the Kafka daemon 10 | # 11 | # chkconfig: - 90 10 12 | # description: Kafka daemon 13 | 14 | ### BEGIN INIT INFO 15 | # Provides: kafka 16 | # Required-Start: $network $local_fs $remote_fs 17 | # Required-Stop: $network $local_fs $remote_fs 18 | # Should-Start: 19 | # Should-Stop: 20 | # Default-Start: 2 3 4 5 21 | # Default-Stop: 0 1 6 22 | # Short-Description: Controls Kafka as a Service 23 | ### END INIT INFO 24 | 25 | INSTALL_DIR="/opt/kafka" 26 | 27 | if [ ! -d "$INSTALL_DIR" ]; then 28 | echo "$INSTALL_DIR not found! Please check the INSTALL_DIR setting in your $0 script." 29 | exit 1 30 | fi 31 | 32 | # Specify the user to run Kafka as; if not set, then Kafka will not run. 33 | RUNAS="kafka" 34 | 35 | # verify the specified run as user exists 36 | runas_uid="`id -u "$RUNAS"`" 37 | if [ $? -ne 0 ]; then 38 | echo "User $RUNAS not found! Please create the $RUNAS user before running this script." 39 | exit 1 40 | fi 41 | 42 | DAEMON_NAME=kafka 43 | MEMORY_LIMITS="-Xmx256M -Xms128M" 44 | KAFKA_HEAP_OPTS="-Djava.security.auth.login.config=$INSTALL_DIR/config/jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf $MEMORY_LIMITS" 45 | 46 | get_pid () { 47 | echo `ps ax | grep -i 'kafka.Kafka' | grep -v grep | grep -iv consumer | grep -iv producer | grep -i server-sasl-brokers-zookeeper.properties | awk '{print $1}'` 48 | } 49 | 50 | case "$1" in 51 | start) 52 | # Start daemon. 53 | pid=$( get_pid ) 54 | if [ -n "$pid" ] 55 | then 56 | echo "Kafka was already running (pid: $pid)" 57 | else 58 | echo "Starting $DAEMON_NAME"; 59 | if [ -n "$RUNAS" ]; then 60 | sudo su -c "export KAFKA_HEAP_OPTS=\"$KAFKA_HEAP_OPTS\" && nohup $INSTALL_DIR/bin/kafka-server-start.sh -daemon $INSTALL_DIR/config/server-sasl-brokers-zookeeper.properties > /tmp/nohup.out" - "$RUNAS" 61 | else 62 | echo "RUNAS user not specified" 63 | fi 64 | fi 65 | ;; 66 | 67 | stop) 68 | # Stop daemons. 69 | echo "Shutting down $DAEMON_NAME"; 70 | pid=$( get_pid ) 71 | if [ -n "$pid" ] 72 | then 73 | kill -9 $pid 74 | else 75 | echo "Kafka was not Running" 76 | fi 77 | ;; 78 | 79 | restart) 80 | $0 stop 81 | sleep 2 82 | $0 start 83 | ;; 84 | 85 | status) 86 | pid=$( get_pid ) 87 | if [ -n "$pid" ] 88 | then 89 | echo "Kafka is Running as PID: $pid" 90 | else 91 | echo "Kafka is not Running" 92 | fi 93 | ;; 94 | 95 | *) 96 | echo "Usage: $0 {start|stop|restart|status}" 97 | exit 98 | esac 99 | 100 | exit 0 101 | -------------------------------------------------------------------------------- /README-Kerberos.md: -------------------------------------------------------------------------------- 1 | # Installing and Setting up Kerberos on Debian Stretch (Debian 9.2) 2 | 3 | ## Introduction 4 | 5 | Instructions for installing and configuring Kerberos v5 on an AWS EC2 Debian Stretch instance follow. 6 | 7 | Commands that are indented are to be run not at the `bash` prompt but instead at the `kadmin` prompt that should be present at that point in the instructions. 8 | 9 | ## Steps 10 | 11 | ### Primary and alternative KDCs 12 | 13 | Follow the steps below for primary, secondary, etc, KDCs. 14 | 15 | 1. Get a static IP address. If using AWS, this would be an Elastic IP. If using Google Cloud, you would reserve a static IP address and then link it to your VM instance. 16 | 17 | 2. Decide on a subdomain, e.g. `foo-01.mydomain.com`, and create a DNS entry to direct `foo-01.mydomain.com` to the above static IP address. 18 | 19 | 3. Create an EC2 instance, Google Cloud VM instance, or get a VPS. If using EC2, you might consider one of the Debian Stretch community AMIs (I chose `debian-stretch-hvm-x86_64-gp2-2017-12-17-65723`). I selected a T2 Micro with 1 GB RAM and 8 GB Magnetic Storage. For the Security Group, allow traffic to ports 22 (TCP, for SSH) and 88 (UDP, for Kerberos). Assign your static IP address to this instance. 20 | 21 | If using Google Compute Cloud, currently Debian 9 Stretch is the default VM image. 22 | 23 | 4. This step is relevant only for ZooKeeper servers, but might be useful for Kerberos and Kafka servers as well: make sure that a reverse DNS look up on your static IP address resolves to `foo-01.mydomain.com`. If you're using AWS, see [here](https://aws.amazon.com/blogs/aws/reverse-dns-for-ec2s-elastic-ip-addresses/) for how to request AWS to set up the mapping for a reverse DNS lookup. If a reverse DNS lookup does not resolve to `foo-01.mydomain.com`, then connecting to your ZooKeeper Server from the ZooKeeper client will not work. 24 | 25 | If using Google Cloud, the process is easier - you can [create a PTR record](https://cloud.google.com/compute/docs/instances/create-ptr-record) and then verify it by creating a TXT record as per the instructions you will get at the page for creating a PTR record. 26 | 27 | 5. Make sure that port 22 is open. SSH into your instance, and run the `update` and `upgrade` commands, and install `java`: 28 | 29 | ``` 30 | sudo apt-get update && sudo apt-get -y upgrade 31 | sudo apt-get install -y default-jre 32 | ``` 33 | 34 | 6. Change the hostname on the EC2 instance/VPS to be the FQDN `foo-01.mydomain.com`. Instructions are [available at this aws link for the Amazon Linux AMI](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-hostname.html). For Debian Stretch, you would replace the contents of `/etc/hostname` with the following 35 | 36 | `foo-01.mydomain.com` 37 | 38 | and then reboot (`sudo shutdown -r now`) 39 | 40 | On Google Cloud, this step is not necessary. 41 | 42 | 7. (a) For Kerberos servers (this is not necessary for ZooKeeper servers and Kafka servers, though you can do this for those too), add the static IP address and subdomain to your hosts file (`/etc/hosts`): 43 | 44 | `w.x.y.z foo-01` 45 | 46 | This will allow kerberos to resolve the subdomain to the static IP address. 47 | 48 | (b) For Kafka nodes, and optionally for all other nodes, also add a line with the loop-back address and the fully qualified domain name: 49 | 50 | `127.0.0.1 foo-01.mydomain.com` 51 | 52 | 53 | 8. To install and configure Kerberos on the EC2 instance, execute the following commands. If this is fresh install, you can omit the first lines that delete and remove: 54 | 55 | ``` 56 | sudo rm -rf /etc/krb5conf 57 | sudo rm -rf /var/lib/krb5kdc 58 | sudo rm -rf /etc/krb5.conf 59 | sudo apt purge -y krb5-kdc krb5-admin-server krb5-config krb5-locales krb5-user krb5.conf 60 | sudo apt-get install -y krb5-{kdc,admin-server} 61 | ``` 62 | 63 | This last command will prompt you for information. Accept the REALM it chooses (MYDOMAIN.COM). Enter the hostnames of all of the servers: 64 | 65 | `foo-01.mydomain.com foo-02.mydomain.com foo-03.mydomain.com` 66 | 67 | If you mess things up, re-execute the above commands. Likely, you will get an error after executing the above along the lines of `Cannot open DB2 database '/var/li…`. That's OK - continue with the steps below. 68 | 69 | 9. Set up a new realm (you will have to create a master password): 70 | 71 | `sudo kdb5_util create -s -r MYDOMAIN.COM` 72 | 73 | 74 | 10. Edit `/etc/krb5.conf` as follows: 75 | 76 | Under `[libdefaults]` add: `rdns = false` 77 | 78 | Tack the following on to the end of the file (starting under [domain_realm] 79 | 80 | ``` 81 | .mydomain.com = MYDOMAIN.COM 82 | mydomain.com = MYDOMAIN.COM 83 | 84 | [logging] 85 | kdc = FILE:/var/log/kerberos/krb5kdc.log 86 | admin_server = FILE:/var/log/kerberos/kadmin.log 87 | default = FILE:/var/log/kerberos/krb5lib.log 88 | ``` 89 | 90 | 11. Create the log directory and file: 91 | 92 | ``` 93 | sudo mkdir /var/log/kerberos 94 | sudo touch /var/log/kerberos/kadmin.log 95 | sudo chmod a+rw /var/log/kerberos/kadmin.log 96 | ``` 97 | 98 | 12. Edit the `/etc/krb5kdc/kadm5.acl` access control list file, and uncomment or add this line: 99 | 100 | `*/admin *` 101 | 102 | This will give admin privileges to any principal you add with the form `x/admin@y` 103 | 104 | 13. Restart the server and the kdc: 105 | 106 | `sudo invoke-rc.d krb5-admin-server restart && sudo invoke-rc.d krb5-kdc restart` 107 | 108 | 109 | 14. Test that there are no tickets already: 110 | 111 | `klist # should output: klist: No credentials cache found (filename: /tmp/krb5cc_1000)` 112 | 113 | ### Primary KDC 114 | 115 | If this will be the Primary KDC, then create an admin principal (it will prompt you for a user password): 116 | 117 | ``` 118 | sudo kadmin.local 119 | addprinc yourname/admin 120 | quit 121 | ``` 122 | 123 | Create a ticket: 124 | 125 | ``` 126 | KRB5_TRACE=/dev/stdout kinit yourname/admin 127 | ``` 128 | 129 | Test that the ticket was created: 130 | 131 | ``` 132 | klist # should output some issued date, expire date, etc. 133 | ``` 134 | 135 | While we are here, lets create principals for our ZooKeeper ensemble and our Kafka cluster. Let us assume that your zookeeper server will be `zookeeper-server-0X.yourdomain.com`. Execute the following (you will have to choose a password for each): 136 | 137 | ``` 138 | sudo kadmin.local 139 | addprinc zookeeperclient/whatever 140 | addprinc zookeeper/zookeeper-server-01.yourdomain.com 141 | addprinc zookeeper/zookeeper-server-02.yourdomain.com 142 | addprinc zookeeper/zookeeper-server-03.yourdomain.com 143 | addprinc kafka/server-01.yourdomain.com 144 | addprinc kafka/server-02.yourdomain.com 145 | addprinc consumer1/whatever 146 | addprinc producer1/whatever 147 | ``` 148 | 149 | Export the keytabs - Zookeeper servers and clients will use these (you can name the `keytab` files whatever you want, you don't have to follow the convention I have used below): 150 | 151 | ``` 152 | ktadd -k /etc/security/zookeeperclient.whatever.keytab zookeeperclient/whatever 153 | ktadd -k /etc/security/zookeeper.zookeeper-server-01.yourdomain.com.keytab zookeeper/zookeeper-server-01.yourdomain.com 154 | ktadd -k /etc/security/zookeeper.zookeeper-server-02.yourdomain.com.keytab zookeeper/zookeeper-server-02.yourdomain.com 155 | ktadd -k /etc/security/zookeeper.zookeeper-server-03.yourdomain.com.keytab zookeeper/zookeeper-server-03.yourdomain.com 156 | ktadd -k /etc/security/kafka.server-01.yourdomain.com.keytab kafka/server-01.yourdomain.com 157 | ktadd -k /etc/security/kafka.server-02.yourdomain.com.keytab kafka/server-02.yourdomain.com 158 | ktadd -k /etc/security/consumer1.whatever.keytab consumer1/whatever 159 | ktadd -k /etc/security/producer1.whatever.keytab producer1/whatever 160 | quit 161 | ``` 162 | 163 | The next step is to [set up and test connection a to a Zookeeper ensemble](README-Zookeeper.md). 164 | 165 | ## References 166 | 167 | https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-hostname.html 168 | 169 | https://debian-administration.org/article/570/MIT_Kerberos_installation_on_Debian 170 | 171 | https://help.ubuntu.com/lts/serverguide/kerberos.html 172 | 173 | https://serverfault.com/questions/592893/completely-uninstall-kerberos-on-ubuntu-server 174 | 175 | https://help.ubuntu.com/community/Kerberos 176 | 177 | https://aws.amazon.com/blogs/aws/reverse-dns-for-ec2s-elastic-ip-addresses 178 | -------------------------------------------------------------------------------- /README-Kafka-brokers-SASL-only.md: -------------------------------------------------------------------------------- 1 | # Installing and setting up a Kafka (broker) cluster with SASL 2 | 3 | ## Introduction 4 | 5 | Instructions for setting up a Kafka cluster and enabling SASL authentication between the brokers and ZooKeeper follow. 6 | 7 | My setup will be as follows: 8 | 9 | - 2 nodes, each with just one broker. I was going to have a node with two brokers, but then I figured... why? There does not seem to be any advantage in such an arrangement. 10 | - The `/chroot/path` will be `/apps/kafka-cluster-demo`. This will be a znode in the ZooKeeper cluster that I will create manually. I will give `cdrw` permissions to the brokers for this znode. This is like giving a namespace for my Kafka cluster, so that multiple applications can use the same ZooKeeper ensemble without interfering with each other. 11 | 12 | ## Steps 13 | 14 | ### Setting up the first Kafka node. 15 | 16 | Instructions for setting up the server first Kafka node follow. After you have set up the server for the first Kafka node, you can follow these same steps for setting up the servers for the other nodes in the cluster, with the exception that you would use a different static IP address for each and change the subdomain from `kerberos-server-01` to `server-k`, where `k` is `02`, `03`, etc. (Here, we will assume that the Kafka nodes are on servers that could be running other applications, so we will call them `server-k` instead of `kafka-server-k`). 17 | 18 | Steps 1 through 7, are the same as [those for setting up Kerberos](README-Kerberos.md), except that for this demonstration the subdomain will be `server-01` (so you will replace the contents of `/etc/hostname` with `server-01.mydomain.com` and add `YOUR.STATIC.IP.ADDRESS server-01` and `YOUR.STATIC.IP.ADDRESS server-01.yourdomain.com` to `/etc/hosts`). Also, make sure that the following ports are open: 9092, 9093. If you will be running multiple brokers on a server, then open 2 additional ports for each additional broker. 19 | 20 | At this point you should have your Kafka servers set up. What makes a Kafka server a Kafka node is running at least one Kafka broker on the server. Below are instructions for setting up broker-10. 21 | 22 | 8. In your ZooKeeper ensemble, create a znode to be used as the `/chroot/path` for this demo. I will use `/apps/kafka-cluster-demo` as my `/chroot/path` You can connect using the `zktestclient` from the [ZooKeeper setup instructions](README-ZooKeeper.md). Also set the permissions so that all brokers 23 | 24 | ``` 25 | create /apps 26 | setAcl /apps sasl:zktestclient/whatever@YOURDOMAIN.COM:crwd 27 | create /apps/kafka-cluster-demo "" 28 | setAcl /apps/kafka-cluster-demo sasl:zktestclient/whatever@YOURDOMAIN.COM:crwda,sasl:kafka/server-01.yourdomain.com@YOURDOMAIN.COM:crw,sasl:kafka/server-02.yourdomain.com@YOURDOMAIN.COM:crw 29 | ``` 30 | 31 | 9. Create a `kafka` user: `sudo adduser kafka`, and choose a password. 32 | 33 | 10. `su` to this user, and [download](https://kafka.apache.org/downloads) Kafka 1.0 (don't automatically use the mirror below - choose the mirror that the above link suggests for you): 34 | 35 | ``` 36 | sudo su - kafka 37 | mkdir ~/log 38 | wget -c http://apache.mirror.globo.tech/kafka/1.0.0/kafka_2.11-1.0.0.tgz 39 | tar -zxvf kafka_2.11-1.0.0.tgz 40 | exit 41 | sudo ln -s /home/kafka/kafka_2.11-1.0.0 /opt/kafka 42 | sudo su - kafka 43 | cd kafka_2.11-1.0.0 44 | ``` 45 | 46 | 11. Create a copy of the `server.properties` file (in the `config/` directory) for this exercise: 47 | 48 | ``` 49 | cp config/server.properties config/server-sasl-brokers-zookeeper.properties 50 | ``` 51 | 52 | Edit the new file so that it contains the following (add or edit existing lines): 53 | 54 | ``` 55 | broker.id=1 # for other brokers, replace the 1 with SERVER_NUMBER (which is the same as NODE_NUMBER, since we will have only one node per server). 56 | listeners=SASL_PLAINTEXT://0.0.0.0:9092 57 | advertised.listeners=SASL_PLAINTEXT://server-01.yourdomain.com:9092 # change server-01 as appropriate 58 | security.inter.broker.protocol=SASL_PLAINTEXT 59 | super.users=kafka 60 | delete.topic.enable=true 61 | auto.create.topics.enable 62 | log.dirs=/home/kafka/log 63 | zookeeper.connect=zookeeper-server-01.yourdomain.com:2181,zookeeper-server-02.yourdomain.com:2181,zookeeper-server-03.yourdomain.com:2181/apps/kafka-cluster-demo # Include all of your ZooKeeper servers - I use 3. 64 | ``` 65 | 66 | A sample server config file is available [here](server-sasl-brokers-zookeeper.properties) 67 | 68 | 12. Create a `jaas.conf` file in the `config/` directory: 69 | 70 | ``` 71 | KafkaServer { 72 | com.sun.security.auth.module.Krb5LoginModule required 73 | useKeyTab=true 74 | keyTab="/home/kafka/kafka_2.11-1.0.0/config/kafka.server-01.yourdomain.com.keytab" 75 | storeKey=true 76 | useTicketCache=false 77 | serviceName="kafka" 78 | principal="kafka/server-01.yourdomain.com@YOURDOMAIN.COM"; 79 | }; 80 | 81 | // This is for the broker acting as a client to ZooKeeper 82 | Client { 83 | com.sun.security.auth.module.Krb5LoginModule required 84 | useKeyTab=true 85 | keyTab="/home/kafka/kafka_2.11-1.0.0/config/kafka.server-01.yourdomain.com.keytab" 86 | storeKey=true 87 | useTicketCache=false 88 | serviceName="zookeeper" 89 | principal="kafka/server-01.yourdomain.com@YOURDOMAIN.COM"; 90 | }; 91 | ``` 92 | 93 | 13. Place the keytab file cited in the above `jaas.conf` file (that were created in [the Kerberos setup instructions](README-Kerberos.md) - `kafka.server-01.yourdomain.com.keytab`) in the `config/` directory. 94 | 95 | 14. Test that you can start the broker without any errors: 96 | 97 | ``` 98 | KAFKA_HEAP_OPTS="-Djava.security.auth.login.config=/home/kafka/kafka_2.11-1.0.0/config/jaas.conf -Dsun.security.krb5.debug=true -Djava.security.krb5.conf=/etc/krb5.conf -Xmx256M -Xms128M" \ 99 | bin/kafka-server-start.sh config/server-sasl-brokers-zookeeper.properties 100 | ``` 101 | 102 | Note that I have added the JVMFLAGS `-Xmx256M` and `-Xms128M` to limit the amount of memory that Kafka will use, because I am using an AWS t2.miro instance, which has only 1 GB of RAM. If I don't add these flags, I will get an out of memory error. 103 | 104 | 15. If the previous step succeeds, then test that you can write to a Kafka topic on this broker with a producer, and read from it using a consumer. Make sure that the broker you started from the step above is still running. 105 | 106 | a. Open a separate terminal window, log in as Kafka, and switch to the `/home/kafka/kafka_2.11-1.0.0/` directory. Create a test topic to publish to: 107 | 108 | KAFKA_OPTS="-Djava.security.auth.login.config=/home/kafka/kafka_2.11-1.0.0/config/jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf" \ 109 | bin/kafka-topics.sh --create \ 110 | --zookeeper zookeeper-server-01.yourdomain.com:2181,zookeeper-server-02.yourdomain.com:2181,zookeeper-server-03.yourdomain.com:2181/apps/kafka-cluster-demo \ 111 | --replication-factor 1 \ 112 | --partitions 1 \ 113 | --topic test-topic 114 | 115 | You should see `Created topic "test-topic".` as the output of this command. 116 | 117 | b. Now list all topics, just to confirm (not really necessary) that the test topic `test-topic` was created: 118 | 119 | KAFKA_OPTS="-Djava.security.auth.login.config=/home/kafka/kafka_2.11-1.0.0/config/jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf" \ 120 | bin/kafka-topics.sh --list \ 121 | --zookeeper zookeeper-server-01.yourdomain.com:2181,zookeeper-server-02.yourdomain.com:2181,zookeeper-server-03.yourdomain.com:2181/apps/kafka-cluster-demo 122 | 123 | The output should be `test-topic`. You can get detail on the topic by executing the above command using `--describe` instead of `--list`. 124 | 125 | c. At this point, attempts to write to or read from this topic will fail, because no producers and no consumers have authorizations to do so. Lets us grant these authorizations for the producer: 126 | 127 | KAFKA_HEAP_OPTS="-Djava.security.auth.login.config=/home/kafka/kafka_2.11-1.0.0/config/jaas.conf -Dsun.security.krb5.debug=true -Djava.security.krb5.conf=/etc/krb5.conf -Xmx256M -Xms128M" \ 128 | bin/kafka-acls.sh --authorizer-properties \ 129 | zookeeper.connect=zookeeper-server-01.yourdomain.com:2181,zookeeper-server-02.yourdomain.com:2181,zookeeper-server-03.yourdomain.com:2181/apps/kafka-cluster-demo \ 130 | --add --allow-principal User:producer1/whatever@EIGENROUTE.COM --producer --topic test-topic 131 | 132 | and for the consumer: 133 | 134 | KAFKA_HEAP_OPTS="-Djava.security.auth.login.config=/home/kafka/kafka_2.11-1.0.0/config/jaas.conf -Dsun.security.krb5.debug=true -Djava.security.krb5.conf=/etc/krb5.conf -Xmx256M -Xms128M" \ 135 | bin/kafka-acls.sh --authorizer-properties \ 136 | zookeeper.connect=zookeeper-server-01.yourdomain.com:2181,zookeeper-server-02.yourdomain.com:2181,zookeeper-server-03.yourdomain.com:2181/apps/kafka-cluster-demo \ 137 | --add --allow-principal User:consumer1/whatever@EIGENROUTE.COM --consumer --topic test-topic --group Group-1 138 | 139 | ### Testing that you can write to and read from a topic 140 | 141 | 16. Now we will configure and start producers and consumers on a separate machine. You can use your local machine. Navigate to or create a directory for testing the consumer and producer. 142 | 143 | ``` 144 | cd /path/to/your-directory/ 145 | wget -c http://apache.mirror.globo.tech/kafka/1.0.0/kafka_2.11-1.0.0.tgz 146 | tar -zxvf kafka_2.11-1.0.0.tgz 147 | rm kafka_2.11-1.0.0.tgz 148 | cd kafka_2.11-1.0.0 149 | ``` 150 | 151 | Place the `producer1.whatever.keytab` and `consumer.whatever.keytab` files you created [earlier](README-kerberos.md) in the `config/` directory. In this directory, create a file `sasl-producer.properties` with the following contents: 152 | 153 | ``` 154 | bootstrap.servers=server-01.yourdomain.com:9092 155 | security.protocol=SASL_PLAINTEXT 156 | sasl.mechanism=GSSAPI 157 | sasl.kerberos.service.name=kafka 158 | sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \ 159 | useKeyTab=true \ 160 | storeKey=true \ 161 | keyTab="/path/to/producer1.whatever.keytab" \ 162 | principal="producer1/whatever@YOURDOMAIN.COM"; 163 | ``` 164 | 165 | And create a file `sasl-consumer.properties` with the following contents: 166 | 167 | ``` 168 | bootstrap.servers=server-01.yourdomain.com:9092 169 | security.protocol=SASL_PLAINTEXT 170 | sasl.mechanism=GSSAPI 171 | sasl.kerberos.service.name=kafka 172 | sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \ 173 | useKeyTab=true \ 174 | storeKey=true \ 175 | keyTab="/path/to/consumer1.whatever.keytab" \ 176 | principal="consumer1/whatever@YOURDOMAIN.COM"; 177 | ``` 178 | 179 | 17. In this same terminal window, start the producer: 180 | 181 | ``` 182 | KAFA_HEAP_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Dsun.security.krb5.debug=true" \ 183 | bin/kafka-console-producer.sh --broker-list server-01.yourdomain.com:9092 \ 184 | --topic test-topic --producer.config config/sasl-producer.properties 185 | ``` 186 | 187 | In a different terminal, navigate to the same directory that you were in above, and start the consumer: 188 | 189 | ``` 190 | KAFA_HEAP_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Dsun.security.krb5.debug=true" \ 191 | bin/kafka-console-consumer.sh --bootstrap-server server-01.yourdomain.com:9092 \ 192 | --topic test-topic --consumer.config config/sasl-consumer.properties --from-beginning 193 | ``` 194 | 195 | Start entering text in the producer window, hit enter, and check that this same text shows up in the consumer window. If so, your single node cluster works, and you can now stop the producer and the consumer. If not, open an issue in this repo, so that I can fix these instructions. 196 | 197 | 18. Delete this test topic: 198 | 199 | ``` 200 | sudo su - kafka 201 | cd /home/kafka/kafka_2.11-1/ 202 | KAFKA_HEAP_OPTS="-Djava.security.auth.login.config=/home/kafka/kafka_2.11-1.0.0/config/jaas.conf -Dsun.security.krb5.debug=true -Djava.security.krb5.conf=/etc/krb5.conf -Xmx256M -Xms128M" \ 203 | bin/kafka-run-class.sh kafka.admin.TopicCommand \ 204 | --zookeeper zookeeper-server-01.yourdomain.com:2181,zookeeper-server-02.yourdomain.com:2181,zookeeper-server-03.yourdomain.com:2181/apps/kafka-cluster-demo \ 205 | --delete --topic test-topic 206 | ``` 207 | 208 | Note: deleting topics is flaky. You might have to connect to Zookeeper as `super` and delete the topic manually by executing the following: 209 | 210 | ``` 211 | rmr /apps/kafka-cluster-demo/brokers/topics/test-topic 212 | ``` 213 | 214 | (the reference for this is [here](https://stackoverflow.com/a/33538299/2251463)) 215 | 216 | ### Make the Kafka broker a service 217 | 218 | 19. If the previous step succeeds, then make this Kafka broker a service using the `init.d` script [here](kafka). Go back to the windown where your Kafka broker is running, and stop the broker. Place this script in the `/etc/init.d/` directory, make it world executable, make it a service: 219 | 220 | ``` 221 | exit # to get you back to the admin user, which can sudo 222 | cd /etc/init.d 223 | sudo chmod a+x kafka 224 | sudo update-rc.d kakfa defaults 225 | sudo service kafka start 226 | ``` 227 | 228 | ### Setting up the second Kafka node/broker 229 | 230 | 20. Set up a second Kafka server, just as you set up the first Kafka server - repeat steps 1 though 7, 9 through 14, and step 19 above, using a different broker ID and subdomain. I will use a broker ID of `2` and the subdomain `server-02`. 231 | 232 | 21. Now, log into the server of the first broker (server-01) as kafka, navigate to the `kafka_2.11-1.0.0/` directory, and create a topic. Basically, execute step 15 above again, except name the new test topic `test-topic2`. `describe` the new topic, and grant the appropriate permissions to the producer and consumer for the new topic. 233 | 234 | ``` 235 | KAFKA_OPTS="-Djava.security.auth.login.config=/home/kafka/kafka_2.11-1.0.0/config/jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf" \ 236 | bin/kafka-topics.sh --create \ 237 | --zookeeper zookeeper-server-01.yourdomain.com:2181,zookeeper-server-02.yourdomain.com:2181,zookeeper-server-03.yourdomain.com:2181/apps/kafka-cluster-demo \ 238 | --replication-factor 2 \ 239 | --partitions 9 \ 240 | --topic test-topic2 241 | 242 | KAFKA_OPTS="-Djava.security.auth.login.config=/home/kafka/kafka_2.11-1.0.0/config/jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf" \ 243 | bin/kafka-topics.sh --describe \ 244 | --zookeeper zookeeper-server-01.yourdomain.com:2181,zookeeper-server-02.yourdomain.com:2181,zookeeper-server-03.yourdomain.com:2181/apps/kafka-cluster-demo 245 | ``` 246 | 247 | The output should show that `test-topic2` has 9 partitions, that the leader of each partition is either 1 or 2, and that the Isrs are either 1,2 or 2,1. If instead you get output showing that, for all partitions, the leader is "none" and the Isrs are blank, then resart both brokers. I found this to be necessary if I create the topic on Broker 2 instead of Broker 1. See [here](https://stackoverflow.com/questions/48143483/kafka-producer-in-a-multi-broker-multi-server-cluster-cannot-write-to-newly-cre) for more info. Weird. 248 | 249 | Now grant the necessary permissions for the producer and the consumer: 250 | 251 | ``` 252 | KAFKA_HEAP_OPTS="-Djava.security.auth.login.config=/home/kafka/kafka_2.11-1.0.0/config/jaas.conf -Dsun.security.krb5.debug=true -Djava.security.krb5.conf=/etc/krb5.conf -Xmx256M -Xms128M" \ 253 | bin/kafka-acls.sh --authorizer-properties \ 254 | zookeeper.connect=zookeeper-server-01.yourdomain.com:2181,zookeeper-server-02.yourdomain.com:2181,zookeeper-server-03.yourdomain.com:2181/apps/kafka-cluster-demo \ 255 | --add --allow-principal User:producer1/whatever@EIGENROUTE.COM --producer --topic test-topic2 256 | 257 | KAFKA_HEAP_OPTS="-Djava.security.auth.login.config=/home/kafka/kafka_2.11-1.0.0/config/jaas.conf -Dsun.security.krb5.debug=true -Djava.security.krb5.conf=/etc/krb5.conf -Xmx256M -Xms128M" \ 258 | bin/kafka-acls.sh --authorizer-properties \ 259 | zookeeper.connect=zookeeper-server-01.yourdomain.com:2181,zookeeper-server-02.yourdomain.com:2181,zookeeper-server-03.yourdomain.com:2181/apps/kafka-cluster-demo \ 260 | --add --allow-principal User:consumer1/whatever@EIGENROUTE.COM --consumer --topic test-topic2 --group Group-1 261 | ``` 262 | 263 | This also might not work on the broker server on which you are executing the command, as you might have to execute the command on the other broker. Yes, more Kafka weirdness. For me, the producer command failed on the Broker 1 server with the following error: 264 | 265 | ``` 266 | Error while executing ACL command: org.apache.zookeeper.KeeperException$NoAuthException: 267 | KeeperErrorCode = NoAuth for /kafka-acl-changes/acl_changes_0000000000 268 | org.I0Itec.zkclient.exception.ZkException: org.apache.zookeeper.KeeperException$NoAuthException: 269 | KeeperErrorCode = NoAuth for /kafka-acl-changes/acl_changes_0000000000 270 | 271 | ``` 272 | 273 | When I checked the permissions on ZooKeeper of the `/apps/kafka-cluster-demo/kafka-acl-changes/acl_changes_0000000000` node, I saw that only Broker 2 has write permission on this node: 274 | 275 | ``` 276 | [zk: zookeeper-server-03.eigenroute.com:2181(CONNECTED) 0] getAcl /apps/kafka-cluster-demo/kafka-acl-changes/acl_changes_0000000000 277 | 'world,'anyone 278 | : r 279 | 'sasl,'kafka/server-02.eigenroute.com@EIGENROUTE.COM 280 | : cdrwa 281 | [zk: zookeeper-server-03.eigenroute.com:2181(CONNECTED) 1] 282 | ``` 283 | 284 | So I had to set the permissions using the Broker 2 server. This worked - the producer was able to write to and the consumer was able to read from the topic. 285 | 286 | 22. On your local machine, as before, start a producer and consumer in separate terminal tabs. Use the same producer and consumers and before, except set the topic to `test-topic2` for both. Enter text in the producer tab and you should see it output in the consumer tab. 287 | 288 | 289 | Well this was a pain. I found two instances of Kafka weirdness. It shouldn't matter on which Broker server you create topics or grant permissions, but apparently it does. 290 | 291 | Next up: SSL between Kafka clients and Brokers. 292 | 293 | 294 | ## References 295 | 296 | Kumar, Manish; Chanchal Singh. Building Data Streaming Applications with Apache Kafka. Packt Publishing. August 2017. Publication reference 1170817. 297 | 298 | Estrada, Raul. Apache Kafka 1.0 Cookbook. Packt Publishing. December 2017. Publication reference 1211217. 299 | 300 | https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/ 301 | 302 | https://www.gitbook.com/book/jaceklaskowski/apache-kafka/details 303 | 304 | https://kafka.apache.org/ 305 | 306 | https://stackoverflow.com/a/33538299/2251463 307 | 308 | https://stackoverflow.com/questions/48143483/kafka-producer-in-a-multi-broker-multi-server-cluster-cannot-write-to-newly-cre 309 | -------------------------------------------------------------------------------- /README-Zookeeper.md: -------------------------------------------------------------------------------- 1 | # Installing and Setting up a Zookeeper ensemble with SASL, using Debian Stretch (Debian 9.2) 2 | 3 | ## Introduction 4 | 5 | Instructions for setting up a ZooKeeper ensemble and enabling SASL follow. 6 | 7 | ## Steps 8 | 9 | ### Setting up the first Zookeeper server. 10 | 11 | Instructions for setting up the first Zookeeper server follow. After you have set up the first ZooKeeper server, you can follow these same steps for setting up the other servers in the ensemble, with the exception that you would use a different static IP address and change the subdomain from `zookeeper-server-01` to `zookeeper-server-k`, where `k` is `02`, `03`, etc. 12 | 13 | Steps 1 through 7, are the same as [those for setting up Kerberos](README-Kerberos.md), except that for this demonstration the subdomain will be `zookeeper-server-01` (so you will replace the contents of `/etc/hostname` with `zookeeper-server-01.mydomain.com` and add `YOUR.STATIC.IP.ADDRESS zookeeper-server-01` to `/etc/hosts`). Also, make sure that the following ports are open: 2181, 2888, 3888. 14 | 15 | 7. Add a user zookeeper: 16 | 17 | `sudo adduser zookeeper` 18 | 19 | 8. Switch to this user, and download and unpack the latest stable version of zookeeper (don't automatically use the link below - use a download link available from [here](https://www.apache.org/dyn/closer.cgi/zookeeper/)). I'm using 3.4.11, which seems to be the latest stable version as of the time of this post: 20 | 21 | ``` 22 | sudo su - zookeeper 23 | mkdir zk 24 | cd zk 25 | wget -c http://apache.mirror.globo.tech/zookeeper/zookeeper-3.4.11/zookeeper-3.4.11.tar.gz 26 | tar -zxvf zookeeper-3.4.11.tar.gz 27 | rm zookeeper-3.4.11.tar.gz 28 | cd zookeeper-3.4.11 29 | ``` 30 | 31 | 9. In the `conf/` directory, create a file `zoo.cfg-ensemble` with the following contents: 32 | 33 | ``` 34 | tickTime=2000 35 | initLimit=10 36 | syncLimit=5 37 | dataDir=/home/zookeeper/dataDir 38 | dataLogDir=/home/zookeeper/dataLogDir 39 | clientPort=2181 40 | # server.1=0.0.0.0:2888:3888 41 | # server.2=zookeeper-server-02.yourdomain.com:2888:3888 42 | # server.3=zookeeper-server-03.yourdomain.com:2888:3888 43 | authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider 44 | requireClientAuthScheme=sasl 45 | jaasLoginRenew=3600000 46 | ``` 47 | 48 | Note: replace `zookeeper-server-0X` with `0.0.0.0` when X is the id of the Zookeeper instance you are setting up, and all other `server.X` lines should have `zookeeper-server-0X`. See [this StackOverflow post](https://stackoverflow.com/questions/30940981/zookeeper-error-cannot-open-channel-to-x-at-election-address) for more information on why this must be the case when using EC2. 49 | 50 | For now, the `server.X` lines are commented out, in order to test the ZooKeeper server in StandAlone mode. Later we will uncomment these lines so that we can use ZooKeeper as an ensemble of servers. 51 | 52 | 10. Create the `dataDir` and `dataLogDir` folders, and a `jaas folder`, and create a symlink from `zoo.cfg` to `zoo.cfg-ensemble`: 53 | 54 | ``` 55 | mkdir /home/zookeeper/dataDir 56 | mkdir /home/zookeeper/dataLogDir 57 | mkdir /home/zookeeper/jaas 58 | ln -s /home/zookeeper/zk/zookeeper-3.4.11/conf/zoo.cfg-ensemble /home/zookeeper/zk/zookeeper-3.4.11/conf/zoo.cfg 59 | exit 60 | sudo ln -s /home/zookeeper/zk/zookeeper-3.4.11 /opt/zookeeper 61 | sudo su - zookeeper 62 | cd zk/zookeeper-3.4.11 63 | ``` 64 | 65 | 11. In the `/home/zookeeper/zk/zookeeper-3.4.11/conf/log4j.properties` file, decide where you will place log files. Junqueira & Reed (see References below) recommend writing log files to a separate drive. If you want to keep them on the same drive for now, then replace the lines: 66 | 67 | ``` 68 | zookeeper.log.dir=. 69 | ... 70 | zookeeper.tracelog.dir=. 71 | ``` 72 | 73 | with 74 | 75 | ``` 76 | zookeeper.log.dir=/home/zookeeper/log 77 | ... 78 | zookeeper.tracelog.dir=/home/zookeeper/log 79 | ``` 80 | 81 | and create the log directory: 82 | 83 | ``` 84 | mkdir /home/zookeeper/log 85 | ``` 86 | 87 | You will have to also figure out how to get `zkServer.sh` to stop overriding values there. [See this (seemingly dead) issue](https://issues.apache.org/jira/browse/ZOOKEEPER-2170) for more details. It seems that the `log4j.properties` file is ignored in favor of the environment variables `ZOO_LOG_DIR` and `ZOO_LOG4J_PROP`. 88 | 89 | If you decide to log, remember to periodically clear out the log files so that you do not run out of file storage space. 90 | 91 | 12. In the `/home/zookeeper/jaas/` folder, place the `zookeeper-server-01.keytab` file that you created when setting up Kerberos. Make the `zookeeper` user the owner of this file, and make sure this user has read and write permissions on the file. Also in this folder, create a `jaas.conf` file with the following contents: 92 | 93 | ``` 94 | Server { 95 | com.sun.security.auth.module.Krb5LoginModule required 96 | useTicketCache=false 97 | useKeyTab=true 98 | keyTab="/home/zookeeper/jaas/zookeeper-server-01.keytab" 99 | storeKey=true 100 | principal="zookeeper/zookeeper-server-01@YOURDOMAIN.COM"; 101 | }; 102 | ``` 103 | 104 | Note that you would use `02` or `03` instead of `01` in the above file according to the ensemble number (id) that you are setting up. 105 | 106 | 13. In `/home/zookeeper/dataDir/`, create a file `myid` and place a solitary `1` in this file: 107 | 108 | ``` 109 | echo "1" > /home/zookeeper/dataDir/myid 110 | ``` 111 | 112 | `1` is used here because this is `zookeeper-server-01`. For `zookeeper-server-02`, one would use a `2` instead, etc. 113 | 114 | 115 | 14. In the `/etc/` folder, place the `/etc/krb5.conf` file from [the Kerberos server that you set up earlier](README-Kerberos.md), and make sure it is world readable. 116 | 117 | 15. Repeat the above steps for the other ZooKeeper servers in the ensemble (`zookeeper-server-02` and `zookeeper-server-03`) 118 | 119 | 16. To test that the server starts without errors, execute the following command from `/home/zookeeper/zk/zookeeper-3.4.11/`: 120 | 121 | ``` 122 | JVMFLAGS="-Djava.security.auth.login.config=/home/zookeeper/jaas/jaas.conf -Dsun.security.krb5.debug=true" ZOO_LOG_DIR="/home/zookeeper/log" ZOO_LOG4J_PROP=TRACE,CONSOLE,ROLLINGFILE bin/zkServer.sh start-foreground 123 | ``` 124 | 125 | The output should be similar to the following: 126 | 127 | ``` 128 | ... 129 | >>> KrbAsRep cons in KrbAsReq.getReply zookeeper/zookeeper-server-01.yourdomain.com 130 | 2017-12-26 20:46:20,231 [myid:] - INFO [main:Login@297] - Server successfully logged in. 131 | 2017-12-26 20:46:20,234 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181 132 | 2017-12-26 20:46:20,235 [myid:] - INFO [Thread-1:Login$1@130] - TGT refresh thread started. 133 | 2017-12-26 20:46:20,235 [myid:] - INFO [Thread-1:Login@305] - TGT valid starting at: Tue Dec 26 20:46:20 UTC 2017 134 | 2017-12-26 20:46:20,236 [myid:] - INFO [Thread-1:Login@306] - TGT expires: Wed Dec 27 06:46:20 UTC 2017 135 | 2017-12-26 20:46:20,236 [myid:] - INFO [Thread-1:Login$1@185] - TGT refresh sleeping until: Wed Dec 27 04:58:21 UTC 2017 136 | ``` 137 | 138 | ### Testing the ZooKeeper server in StandAlone mode. 139 | 140 | 17. On another machine (EC2 instance, personal, whatever - as long as it is not the same machine as the server, though it can be), download ZooKeeper as before to a directory of your choosing - I'll assume you have downloaded it to your home directory: 141 | 142 | ``` 143 | cd ~ 144 | wget -c http://apache.mirror.globo.tech/zookeeper/zookeeper-3.4.11/zookeeper-3.4.11.tar.gz 145 | tar -zxvf zookeeper-3.4.11.tar.gz 146 | rm zookeeper-3.4.11.tar.gz 147 | cd zookeeper-3.4.11 148 | ``` 149 | 150 | 18. Place the `/etc/security/zookeeperclient.whatever.keytab` that you created in the [Kerberos](README-Kerberos.md) section in some directory, say `~/jaas/` (create this directory). Make sure you have read permission on it. 151 | 152 | 19. Also in the `~/jaas/` section, create a file called `jaas.conf` and add the following content to it: 153 | 154 | ``` 155 | Client { 156 | com.sun.security.auth.module.Krb5LoginModule required 157 | useTicketCache=false 158 | useKeyTab=true 159 | keyTab="~/jaas/zookeeperclient.whatever.keytab" 160 | storeKey=true 161 | principal="zookeeperclient/whatever@YOURDOMAIN.COM"; 162 | }; 163 | ``` 164 | 20. Place the `/etc/krb5.conf` file from the Kerberos server that you set up in the `~/jaas/` directory of the client machine, and give yourself read permissions on it. 165 | 166 | 21. From the `~/zk/zookeeper-3.4.11/` directory, run the following command: 167 | 168 | ``` 169 | JVMFLAGS="-Djava.security.auth.login.config=/full/path/to/jaas/jaas.conf -Dsun.security.krb5.debug=true -Djava.security.krb5.conf=/full/path/to/jass/krb5.conf" bin/zkCli.sh -server zookeeper-server-01.yourdomain.com:2181 170 | ``` 171 | 172 | If successful, the terminal output should be similar to the following: 173 | 174 | ``` 175 | ... 176 | 0290: 78 C5 E1 84 61 9E 17 78 CC 01 82 23 AB B2 5D EF x...a..x...#..]. 177 | 02A0: 32 50 7D ED 19 EC 8E 5B C3 5B DD 3B 2P.....[.[.; 178 | 179 | Krb5Context.unwrap: token=[05 04 01 ff 00 0c 00 00 00 00 00 00 2a d9 04 e7 01 01 00 00 5b 29 03 0d 76 14 1f 76 0d 1d 3e 9a ] 180 | Krb5Context.unwrap: data=[01 01 00 00 ] 181 | Krb5Context.wrap: data=[01 01 00 00 7a 6b 74 65 73 74 63 6c 69 65 6e 74 2f 6b 65 72 62 65 72 6f 73 2d 73 65 72 76 65 72 2d 30 32 2e 65 69 67 65 6e 72 6f 75 74 65 2e 63 6f 6d 40 45 49 47 45 4e 52 4f 55 54 45 2e 43 4f 4d ] 182 | Krb5Context.wrap: token=[05 04 00 ff 00 0c 00 00 00 00 00 00 2a d9 04 e7 01 01 00 00 7a 6b 74 65 73 74 63 6c 69 65 6e 74 2f 6b 65 72 62 65 72 6f 73 2d 73 65 72 76 65 72 2d 30 32 2e 65 69 67 65 6e 72 6f 75 74 65 2e 63 6f 6d 40 45 49 47 45 4e 52 4f 55 54 45 2e 43 4f 4d 6d 17 f8 84 22 96 99 2f 09 28 f2 5c ] 183 | 184 | WATCHER:: 185 | 186 | WatchedEvent state:SaslAuthenticated type:None path:null 187 | ``` 188 | 189 | If you get a message like `WatchedEvent state:AuthFailed type:None path:null` somewhere in the output (not necessarily at the end of the output), please open an issue in this repository and provide details on the error you are getting, and provide the configuration files and paths. 190 | 191 | 22. If you were able to SASL authenticate to the ZooKeeper server in StandAlone mode, try executing some ZooKeeper commands: 192 | 193 | ``` 194 | [zk: zookeeper-server-01.yourdomain.com:2181(CONNECTED) 0] ls / 195 | [zookeeper] 196 | [zk: zookeeper-server-01.yourdomain.com:2181(CONNECTED) 1] create /mynode "some data" 197 | Created /mynode 198 | [zk: zookeeper-server-01.yourdomain.com:2181(CONNECTED) 2] ls / 199 | [mynode, zookeeper] 200 | [zk: zookeeper-server-01.yourdomain.com:2181(CONNECTED) 3] get /mynode 201 | some data 202 | cZxid = 0xb4 203 | ctime = Wed Dec 27 01:08:00 UTC 2017 204 | mZxid = 0xb4 205 | mtime = Wed Dec 27 01:08:00 UTC 2017 206 | pZxid = 0xb4 207 | cversion = 0 208 | dataVersion = 0 209 | aclVersion = 0 210 | ephemeralOwner = 0x0 211 | dataLength = 9 212 | numChildren = 0 213 | [zk: zookeeper-server-01.yourdomain.com:2181(CONNECTED) 4] delete /mynode 214 | [zk: zookeeper-server-01.yourdomain.com:2181(CONNECTED) 5] ls / 215 | [zookeeper] 216 | [zk: zookeeper-server-01.yourdomain.com:2181(CONNECTED) 6] 217 | ``` 218 | 219 | If you were able to achieve something similar to the above, then continue. 220 | 221 | 23. Repeat the above for additional ZooKeeper servers to test that you can get them each working in StandAlone mode. 222 | 223 | ### Running a ZooKeeper ensemble. 224 | 225 | 24. Stop all of the ZooKeeper servers that you created and started. On all of the servers, uncomment the `# server.X` lines. Start all of the servers. 226 | 227 | In the output of each of the servers, you should see one with `LEADING` and others with `FOLLOWING - LEADER ELECTION TOOK ...`. If this is the case, then you have the ensemble working correctly. 228 | 229 | ### Restricting access using ACLs. 230 | 231 | 25. At this point, anyone can connect to your ZooKeeper ensemble, and use it and abuse it. We cannot (as over version 3.4) prevent people from connecting to it, but we can prevent people from creating, reading, writing, or deleting znodes, or from modifying ACLs. 232 | 233 | Return to you ZooKeeper client machine, and connect without SASL: 234 | 235 | ``` 236 | bin/zkCli.sh -server zookeeper-server-01.yourdomain.com:2181,zookeeper-server-02.yourdomain.com:2181,zookeeper-server-03.yourdomain.com:2181 237 | ``` 238 | 239 | and play around, by creating a node, reading it, deleting it, getting the ACL of the root znode, etc: 240 | 241 | ``` 242 | [...] ls / 243 | [zookeeper] 244 | [...] create /badnode "vulnerable" 245 | Created /badnode 246 | [...] ls / 247 | [badnode, zookeeper] 248 | [...] getAcl /badnode 249 | 'world,'anyone 250 | : cdrwa 251 | [...] delete /badnode 252 | [...] ls / 253 | [zookeeper] 254 | [...] getAcl / 255 | 'world,'anyone 256 | : cdrwa 257 | ``` 258 | 259 | Let us create a new node and set permissions on it so that only our `zktestclient/whatever` user may read, write, or delete it, or create child znodes under it. 260 | 261 | ``` 262 | [...] create /newnode "for zk sasl client" 263 | Created /newnode 264 | [...] ls / 265 | [zookeeper, newnode] 266 | [...] get /newnode 267 | for zk sasl client 268 | cZxid = 0x1e0000001a 269 | ctime = Wed Dec 27 04:48:37 UTC 2017 270 | mZxid = 0x1e0000001a 271 | mtime = Wed Dec 27 04:48:37 UTC 2017 272 | pZxid = 0x1e0000001a 273 | cversion = 0 274 | dataVersion = 0 275 | aclVersion = 0 276 | ephemeralOwner = 0x0 277 | dataLength = 18 278 | numChildren = 0 279 | [...] setAcl /newnode sasl:zktestclient/whatever@YOURDOMAIN.COM:crwd 280 | cZxid = 0x1e0000001a 281 | ctime = Wed Dec 27 04:48:37 UTC 2017 282 | mZxid = 0x1e0000001a 283 | mtime = Wed Dec 27 04:48:37 UTC 2017 284 | pZxid = 0x1e0000001a 285 | cversion = 0 286 | dataVersion = 0 287 | aclVersion = 1 288 | ephemeralOwner = 0x0 289 | dataLength = 18 290 | numChildren = 0 291 | [...] getAcl /newnode 292 | 'sasl,'sasl:zktestclient/whatever@YOURDOMAIN.COM:crwd 293 | : cdrw 294 | [...] get /newnode 295 | Authentication is not valid : /newnode 296 | [...] create /newnode/childofnewnode "child of new node" 297 | Authentication is not valid : /newnode/childofnewnode 298 | [...] ls / 299 | [zookeeper, newnode] 300 | ``` 301 | 302 | So we created a new znode `/newnode`, but set the ACL for it such that we cannot create child nodes under it nor can we read it - only `zktestclient/whatever` can. 303 | 304 | 26. Let us now SASL authenticate as `zktestclient/whatever`. On your client machine, execute the following command: 305 | 306 | ``` 307 | JVMFLAGS="-Djava.security.auth.login.config=/full/path/to/jaas/jaas.conf -Dsun.security.krb5.debug=true -Djava.security.krb5.conf=/full/path/to/jass/krb5.conf" bin/zkCli.sh -server zookeeper-server-01.yourdomain.com:2181,zookeeper-server-02.yourdomain.com:2181,zookeeper-server-03.yourdomain.com:2181 308 | ``` 309 | 310 | and reproduce the following: 311 | 312 | ``` 313 | [...] ls / 314 | [zookeeper, newnode] 315 | [...] get /newnode 316 | for zk sasl client 317 | cZxid = 0x1e0000001a 318 | ctime = Wed Dec 27 04:48:37 UTC 2017 319 | mZxid = 0x1e0000001a 320 | mtime = Wed Dec 27 04:48:37 UTC 2017 321 | pZxid = 0x1e0000001a 322 | cversion = 0 323 | dataVersion = 0 324 | aclVersion = 1 325 | ephemeralOwner = 0x0 326 | dataLength = 18 327 | numChildren = 0 328 | [...] create /newnode/childofnewnode "child of new node" 329 | Created /newnode/childofnewnode 330 | [...] ls / 331 | [zookeeper, newnode] 332 | [...] get /newnode/childofnewnode 333 | child of new node 334 | cZxid = 0x1e0000001f 335 | ctime = Wed Dec 27 05:05:16 UTC 2017 336 | mZxid = 0x1e0000001f 337 | mtime = Wed Dec 27 05:05:16 UTC 2017 338 | pZxid = 0x1e0000001f 339 | cversion = 0 340 | dataVersion = 0 341 | aclVersion = 0 342 | ephemeralOwner = 0x0 343 | dataLength = 17 344 | numChildren = 0 345 | setAcl /newnode/childofnewnode sasl:zktestclient/whatever@YOURDOMAIN.COM:crwd 346 | ``` 347 | 348 | We see that, now that we are authenticated as the authorized user, we can create a child znode under our znode, and read that znode and our original znode (its parent). Let us restrict access to the parent znode (`/`). 349 | 350 | 27. Set the ACL on the root znode, using any connected user. Because the the root znode currently has the ACL `world:anyone:cdrwa`, anyone can set its ACL. 351 | 352 | ``` 353 | setAcl /newnode/childofnewnode sasl:zktestclient/whatever@YOURDOMAIN.COM:crwd 354 | ``` 355 | 356 | The root znode is now restricted to only `zktestclient/whatever` and the super user. Remember to set the ACLs as above for all nodes that you create. 357 | 358 | 28. Try connecting to the other ZooKeeper servers to see whether they hold the same state: 359 | 360 | ``` 361 | JVMFLAGS="-Djava.security.auth.login.config=/full/path/to/jaas/jaas.conf -Dsun.security.krb5.debug=true -Djava.security.krb5.conf=/full/path/to/jass/krb5.conf" bin/zkCli.sh -server zookeeper-server-03.yourdomain.com:2181 362 | ``` 363 | 364 | Get the ACL for `/newnode/childofnewnode`, etc. Do this for three nodes. The results should be the same. 365 | 366 | 29. For all of your ZooKeeper servers, make ZooKeeper a service. There is a nice `init.d` script [here](https://gist.github.com/bejean/b9ff72c6d2143e16e35d) that you can adapt. More info from [debian-administration.org is here](https://debian-administration.org/article/28/Making_scripts_run_at_boot_time_with_Debian). I have copied the aforementioned script [here - zookeeper](zookeeper), and adapted it slightly. Execute the following to set up ZooKeeper as a service: 367 | 368 | In /home/zookeeper/.profile, add the following: 369 | 370 | ``` 371 | export ZOO_LOG_DIR="/home/zookeeper/log" 372 | export ZOO_JAAS_CONF="/home/zookeeper/jaas/jaas.conf" 373 | export JVMFLAGS="-Djava.security.auth.login.config=$ZOO_JAAS_CONF" 374 | export ZOO_LOG4J_PROP=TRACE,ROLLINGFILE 375 | ``` 376 | 377 | Create an init.d script for ZooKeeper: 378 | 379 | ``` 380 | sudo vi /etc/init.d/zookeeper 381 | ``` 382 | 383 | Paste the [zookeeper](zookeeper) script in the editor and save it. 384 | 385 | ``` 386 | sudo chown root. /etc/init.d/zookeeper 387 | sudo chmod 755 /etc/init.d/zookeeper 388 | sudo update-rc.d zookeeper defaults 389 | sudo service zookeeper start 390 | ``` 391 | 392 | Now ZooKeeper should start automatically after rebooting. 393 | 394 | ### ZooKeeper super user (optional) 395 | 396 | 30. ACLs do not apply to super users. Instructions for activating and authenticating as a super user follow. On one of the ZooKeeper servers we will activate the super user login and set the super user password. Choose a ZooKeeper server machine, and execute the following from the `~/zk/zookeeper-3.4.11/` directory in a different bash (terminal) session than the one that is running the server: 397 | 398 | ``` 399 | export ZK_CLASSPATH=/home/zookeeper/zk/zookeeper-3.4.11/bin/../build/classes:/home/zookeeper/zk/zookeeper-3.4.11/bin/../build/lib/*.jar:/home/zookeeper/zk/zookeeper-3.4.11/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/zookeeper/zk/zookeeper-3.4.11/bin/../lib/slf4j-api-1.6.1.jar:/home/zookeeper/zk/zookeeper-3.4.11/bin/../lib/netty-3.10.5.Final.jar:/home/zookeeper/zk/zookeeper-3.4.11/bin/../lib/log4j-1.2.16.jar:/home/zookeeper/zk/zookeeper-3.4.11/bin/../lib/jline-0.9.94.jar:/home/zookeeper/zk/zookeeper-3.4.11/bin/../lib/audience-annotations-0.5.0.jar:/home/zookeeper/zk/zookeeper-3.4.11/bin/../zookeeper-3.4.11.jar:/home/zookeeper/zk/zookeeper-3.4.11/bin/../src/java/lib/*.jar:/home/zookeeper/zk/zookeeper-3.4.11/bin/../conf: 400 | java -cp $ZK_CLASSPATH org.apache.zookeeper.server.auth.DigestAuthenticationProvider super:some-secret-password 401 | ``` 402 | 403 | (You can get the class path by running `zkEnv.sh` after uncommenting the last line in this file, which echo's the class path to the terminal). 404 | 405 | The output should be similar to: 406 | 407 | ``` 408 | super:some-secret-password->super:Bl5S86TbxiWTRBCdXR1pfGuau48= 409 | ``` 410 | 411 | Stop the ZooKeeper server on this machine, and start it again using the following command: 412 | 413 | ``` 414 | JVMFLAGS="-Djava.security.auth.login.config=/home/zookeeper/jaas/jaas.conf -Dsun.security.krb5.debug=true -Dzookeeper.DigestAuthenticationProvider.superDigest=super:Bl5S86TbxiWTRBCdXR1pfGuau48=" ZOO_LOG_DIR="/home/zookeeper/log" ZOO_LOG4J_PROP=TRACE,ROLLINGFILE,CONSOLE bin/zkServer.sh start-foreground 415 | ``` 416 | 417 | 31. In a separate bash session on the same machine that you are running this ZooKeeper server instance with super user enabled (you can use the same session that you used above to generate the super user password), connect to this ZooKeeper server using a non-SASL authenticated client: 418 | 419 | ``` 420 | cd ~/zk/zookeeper-3.4.11 421 | bin/zkCli.sh -server localhost:2181 422 | ``` 423 | 424 | Note that this has to be done on the same machine as the ZooKeeper server, since traffic between ZooKeeper servers and clients is unencrypted (as of ZooKeeper 3.4), so you want to avoid sending the super user password over an unsecured network. 425 | 426 | 32. Authenticate as the super user: 427 | 428 | ``` 429 | addauth digest super:some-secret-password 430 | ``` 431 | 432 | You can now set ACLs, obtain information on any znode by running `get` on that znode, etc. 433 | 434 | ## References 435 | 436 | Junqueira, Flavio; Reed, Benjamin. ZooKeeper: Distributed Process Coordination (Kindle Location 743). O'Reilly Media. Kindle Edition. 437 | 438 | https://stackoverflow.com/questions/30940981/zookeeper-error-cannot-open-channel-to-x-at-election-address 439 | 440 | https://gist.github.com/bejean/b9ff72c6d2143e16e35d 441 | 442 | https://debian-administration.org/article/28/Making_scripts_run_at_boot_time_with_Debian 443 | 444 | https://issues.apache.org/jira/browse/ZOOKEEPER-2170 445 | --------------------------------------------------------------------------------