└── README.md /README.md: -------------------------------------------------------------------------------- 1 | # kafka-cheatsheet 2 | 3 | ## Install Kafka with Zookeeper on Windows 4 | 1. Install WSL2 using `wsl --install` 5 | 6 | 2. Install Java JDK version 11 7 | 8 | 3. Download Apache Kafka from https://kafka.apache.org/downloads under Binary Downloads section or use wget command 9 | 10 | ```bash 11 | wget https://archive.apache.org/dist/kafka/3.8.0/kafka_2.13-3.8.0.tgz 12 | ``` 13 | Add below system run command in `.bashrc` 14 | 15 | ```bash 16 | PATH="$PATH:~/kafka_2.13-3.8.0/bin" 17 | ``` 18 | 19 | 4. Extract the contents on WSL2 and setup the $PATH environment variables to easy access the Kafka binaries 20 | ```bash 21 | tar xzf kafka_2.13-3.0.0.tgz 22 | mv kafka_2.13-3.0.0 ~ 23 | ``` 24 | 25 | 5. Start Zookeeper using the binaries in WSL2 26 | ```bash 27 | zookeeper-server-start.sh ~/kafka_2.13-3.8.0/config/zookeeper.properties 28 | ``` 29 | 30 | 6. Start Kafka using the binaries in an another process in WSL2 31 | 32 | ```bash 33 | kafka-server-start.sh ~/kafka_2.13-3.8.0/config/server.properties 34 | ``` 35 | 36 | ## Kafka Topics commands 37 | 38 | 1. Create: 39 | 40 | ```bash 41 | kafka-topics.sh --bootstrap-server localhost:9092 --topic first_topic --create 42 | 43 | kafka-topics.sh --bootstrap-server localhost:9092 --topic second_topic --create --partitions 3 44 | 45 | kafka-topics.sh --bootstrap-server localhost:9092 --topic third_topic --create --partitions 3 --replication-factor 1 46 | ``` 47 | 48 | 2. List: 49 | 50 | ```bash 51 | kafka-topics.sh --bootstrap-server localhost:9092 --list 52 | ``` 53 | 3. Describe: 54 | 55 | ```bash 56 | kafka-topics.sh --bootstrap-server localhost:9092 --topic first_topic --describe 57 | ``` 58 | 4. Delete: 59 | 60 | ```bash 61 | kafka-topics.sh --bootstrap-server localhost:9092 --topic first_topic --delete // works only when delete.topic.enable=true 62 | ``` 63 | 64 | ## Kafka Console Producer commands 65 | 66 | 1. Basic producing: 67 | 68 | ```bash 69 | kafka-console-producer.sh --bootstrap-server localhost:9092 --topic first_topic 70 | >Hello 71 | >May name is Sudheer 72 | >This is a kafka demo 73 | ``` 74 | 75 | 2. Producer with properties: 76 | 77 | ```bash 78 | kafka-console-producer.sh --bootstrap-server localhost:9092 --topic first_topic --producer-property acks=all 79 | >The producer supports properties through arguments 80 | >Let's broker acknowledge the message 81 | ``` 82 | 83 | 3. Producting to non-existing topic: 84 | 85 | ```bash 86 | kafka-console-producer.sh --bootstrap-server localhost:9092 --topic new_topic 87 | > hello new topic! 88 | ``` 89 | 4. Producting with keys: 90 | 91 | ```bash 92 | kafka-console-producer.sh --bootstrap-server localhost:9092 --topic first_topic --property parse.key=true --property key.separator=* 93 | >veggies: potato 94 | >fruits: banana 95 | ``` 96 | 97 | ## Kafka Console Consumer commands 98 | 99 | 1. Consuming messages: 100 | 101 | ```bash 102 | kafka-topics.sh --bootstrap-server localhost:9092 --topic fourth_topic --create --partitions 3 103 | kafka-console-producer.sh --bootstrap-server localhost:9092 --producer-property partitioner.class=org.apache.kafka.clients.producer.RoundRobinPartitioner --topic fourth_topic 104 | >one 105 | >two 106 | >three 107 | 108 | kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic fourth_topic 109 | one 110 | two 111 | three 112 | ``` 113 | 114 | 2. Consuming from beginning: 115 | 116 | ```bash 117 | kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic fourth_topic --from-beginning 118 | one 119 | three 120 | two 121 | ``` 122 | 123 | 3. Message formatting with key, value, partitioning and timestamp info: 124 | 125 | ```bash 126 | kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic fourth_topic --formatter kafka.tools.DefaultMessageFormatter --property print.timestamp=true --property print.key=true --property print.value=true --property print.partition=true --from-beginning 127 | 128 | CreateTime:1724466964735 Partition:1 null one 129 | CreateTime:1724466968330 Partition:0 null three 130 | CreateTime:1724466966554 Partition:2 null two 131 | ``` 132 | 133 | ## Kafka Console Consumer group commands 134 | 135 | **Prerequisites:** Create a new topic with 3 partitions 136 | 137 | ```bash 138 | kafka-topics.sh --bootstrap-server localhost:9092 --topic fourth_topic --create --partitions 3 139 | ``` 140 | 141 | 1. Create a first consumer in consumer group: 142 | ```bash 143 | root@Sonu:~# kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic fourth_topic --group my-first-group 144 | ``` 145 | 146 | 2. Create a producer to produce messages: 147 | ```bash 148 | kafka-console-producer.sh --bootstrap-server localhost:9092 --producer-property partitioner.class=org.apache.kafka.clients.producer.RoundRobinPartitioner --topic fourth_topic 149 | >one 150 | >two 151 | >three 152 | >four 153 | ``` 154 | 155 | 3. Create more consumers in a group and message spread: 156 | ```bash 157 | kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic fourth_topic --group my-first-group 158 | one 159 | four 160 | kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic fourth_topic --group my-first-group 161 | three 162 | kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic fourth_topic --group my-first-group 163 | two 164 | ``` 165 | 166 | 4. Create a new group with from beginning option: 167 | ```bash 168 | kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic fourth_topic --group my-fourth-group --from-beginning 169 | two 170 | one 171 | four 172 | three 173 | ``` 174 | 175 | ## Kafka Consumer Group commands 176 | 1. List consumer groups 177 | 178 | ```bash 179 | kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list 180 | my-fourth-group 181 | my-first-group 182 | my-third-group 183 | my-second-group 184 | ``` 185 | 186 | 2. Describe a consumer group 187 | 188 | ```bash 189 | kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-first-group 190 | 191 | GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID 192 | my-first-group fourth_topic 0 3 3 0 console-consumer-e6eb8120-3613-4b28-b722-6917474e8d46 /127.0.0.1 console-consumer 193 | my-first-group fourth_topic 1 3 3 0 console-consumer-e6eb8120-3613-4b28-b722-6917474e8d46 /127.0.0.1 console-consumer 194 | my-first-group fourth_topic 2 2 2 0 console-consumer-e91f28d3-403b-45b4-89a7-1fb60566fe95 /127.0.0.1 console-consumer 195 | ``` 196 | 197 | 3. Reset offsets 198 | 199 | ```bash 200 | kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-first-group --reset-offsets --to-earliest --topic fourth_topic --execute 201 | 202 | WARN [AdminClient clientId=adminclient-1] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics. (org.apache.kafka.clients.admin.KafkaAdminClient) 203 | 204 | GROUP TOPIC PARTITION NEW-OFFSET 205 | my-first-group fourth_topic 0 0 206 | my-first-group fourth_topic 1 0 207 | my-first-group fourth_topic 2 0 208 | ``` 209 | 210 | --------------------------------------------------------------------------------