├── .github ├── FUNDING.yml └── workflows │ └── awesome-bot.yml └── README.md /.github/FUNDING.yml: -------------------------------------------------------------------------------- 1 | github: [dharmeshkakadia] 2 | -------------------------------------------------------------------------------- /.github/workflows/awesome-bot.yml: -------------------------------------------------------------------------------- 1 | name: awesome-bot 2 | 3 | on: [pull_request] 4 | 5 | jobs: 6 | awesome-bot-link-check: 7 | runs-on: ubuntu-latest 8 | steps: 9 | - uses: actions/checkout@v1 10 | - uses: docker://dkhamsing/awesome_bot:latest 11 | with: 12 | args: --allow-redirect /github/workspace/README.md --request-delay 1 13 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # awesome-kafka 2 | Everything about Apache Kafka 3 | 4 | Share your Kafka ❤️ through pull requests :) 5 | 6 | What can you expect to see here? 7 | 8 | * [Development](#development) 9 | * [Client libraries](#client-libraries) 10 | * [Kafka Streams libraries](#kafka-streams-libraries) 11 | * [KSQL](#ksql) 12 | * [Connectors](#connectors) 13 | * [Producers](#producers) 14 | * [Consumers](#consumers) 15 | * [Transformations](#transformations) 16 | * [Testing](#testing) 17 | * [Operations](#operations) 18 | * [CI/CD](#cicd) 19 | * [Operational Utilities](#operational-utilities) 20 | * [Monitoring](#monitoring) 21 | * [Performance tools](#performance-tools) 22 | * [Security](#security) 23 | * [Audit](#audit) 24 | * [Mirroring](#mirroring) 25 | * [Backup](#backup) 26 | * [Tools](#tools) 27 | * [Metadata Management](#metadata-management) 28 | * [Schema Management](#schema-management) 29 | * [UI](#ui) 30 | * [Deployment](#deployment) 31 | * [Kubernetes](#kubernetes) 32 | * [Docker](#docker) 33 | * [Integrations](#integrations) 34 | * [Alternative Implementations](#alternative-implementations) 35 | * [Project built on top of Kafka](#projects-built-on-top-of-Kafka) 36 | * [Unsorted](#unsorted) 37 | 38 | 39 | ## Development 40 | 41 | ### Client libraries 42 | 43 | * [Go](https://github.com/Shopify/sarama) 44 | * [Go](https://github.com/jdamick/kafka) 45 | * [Go](https://github.com/optiopay/kafka) 46 | * [Go Kafka streaming](https://github.com/lovoo/goka) 47 | * [Node](https://kafka.js.org) 48 | * [Node](https://github.com/oleksiyk/kafka) 49 | * [Node](https://github.com/nodefluent/node-sinek) 50 | * [Node Kafka streams](https://github.com/nodefluent/kafka-streams) 51 | * [Node librdkafka](https://github.com/nodefluent/node-rdkafka) 52 | * [Php](https://github.com/weiboad/kafka-php) 53 | * [Php](https://github.com/arnaud-lb/php-rdkafka) 54 | * [Perl](https://github.com/TrackingSoft/Kafka) 55 | * [Python (asyncio)](https://github.com/aio-libs/aiokafka) 56 | * [Python (kafka-python)](https://github.com/dpkp/kafka-python) 57 | * [Python (pykafka)](https://github.com/Parsely/pykafka) 58 | * [Ruby](https://github.com/zendesk/ruby-kafka) 59 | * [Phobos - Ruby App framework](https://github.com/klarna/phobos) 60 | * [Karafka - Ruby and Rails framework](https://github.com/karafka/karafka) 61 | * [Dart](https://github.com/dart-kafka/kafka) 62 | * [JRuby](https://github.com/joekiller/jruby-kafka) 63 | * [Elixir](https://github.com/kafkaex/kafka_ex) 64 | * [Erlang](https://github.com/klarna/brod) 65 | * [Rust](https://github.com/spicavigo/kafka-rust) 66 | * [LinkedIn wrapper library](https://github.com/linkedin/li-apache-kafka-clients) 67 | * [kasper - a lightweight library for processing Kafka topics](https://github.com/movio/kasper) 68 | * and [others](https://cwiki.apache.org/confluence/display/KAFKA/Clients) 69 | 70 | ### Kafka Streams Libraries 71 | * [Scala](https://github.com/lightbend/kafka-streams-scala) 72 | * [faust - Python Stream Processing](https://github.com/robinhood/faust) 73 | * [Southpaw - Streaming left joins for data change capture](https://github.com/jwplayer/southpaw) 74 | 75 | ### KSQL 76 | * [KSQL JDBC Driver](https://github.com/mmolimar/ksql-jdbc-driver) 77 | * [KSQL Addon - User Defined Function (UDF) for Machine Learning](https://github.com/kaiwaehner/ksql-machine-learning-udf) 78 | 79 | ### Connectors 80 | 81 | * [JDBC](https://github.com/confluentinc/kafka-connect-jdbc) 82 | * [Elasticsearch](https://github.com/confluentinc/kafka-connect-elasticsearch) 83 | * [HDFS](https://github.com/confluentinc/kafka-connect-hdfs) 84 | * [Beast - Kafka to BigQuery Sink](https://github.com/gojek/beast) 85 | * [BigQuery](https://github.com/nodefluent/bigquery-kafka-connect) 86 | * [SQL](https://github.com/nodefluent/sequelize-kafka-connect) 87 | * [GCloud pubsub](https://github.com/nodefluent/gcloud-pubsub-kafka-connect) 88 | * [BigTable](https://github.com/nodefluent/bigtable-kafka-connect) 89 | * [Google Cloud Storage (GCS)](https://github.com/aiven/aiven-kafka-connect-gcs) 90 | * [HBase](https://github.com/mravi/kafka-connect-hbase) 91 | * [FileSystem](https://github.com/mmolimar/kafka-connect-fs) 92 | * [Streaming files from a local filesystem](https://github.com/streamthoughts/kafka-connect-file-pulse) 93 | * [kafka-connect-protobuf-converter](https://github.com/blueapron/kafka-connect-protobuf-converter) 94 | * [IBM MQ Source](https://github.com/ibm-messaging/kafka-connect-mq-source) 95 | * [IBM MQ Sink](https://github.com/ibm-messaging/kafka-connect-mq-sink) 96 | * [Azure IoT Hub](https://github.com/Azure/toketi-kafka-connect-iothub) 97 | * [Pulsar](https://github.com/streamnative/pulsar-io-kafka) 98 | * [HTTP](https://github.com/thomaskwscott/kafka-connect-http) 99 | 100 | ### Producers 101 | * [Maxwell's daemon - a mysql-to-json kafka producer](https://github.com/zendesk/maxwell) 102 | 103 | ### Consumers 104 | * [Spark](https://github.com/dibbhatt/kafka-spark-consumer) 105 | * [Storm](http://storm.apache.org/) 106 | * [Mapreduce](https://github.com/Conductor/kangaroo) 107 | * [Divolte](https://github.com/divolte/divolte-kafka-consumer) 108 | * [Azure Data Explorer/Kusto](https://github.com/Azure/kafka-sink-azure-kusto) 109 | * [Firehose](https://github.com/odpf/firehose) - Firehose is an extensible, no-code, and cloud-native service to load real-time streaming data from Kafka to data stores, data lakes, and analytical storage systems. 110 | * [Raccoon](https://github.com/odpf/raccoon) - Raccoon is a high-throughput, low-latency service to collect events in real-time from your web, mobile apps, and services using multiple network protocols and publish to Kafka. 111 | 112 | ### Transformations 113 | * [Dagger](https://github.com/odpf/dagger) - Dagger is an easy-to-use, SQL framework to process built on top of Apache Flink for stateful processing of Kafka data. 114 | * [Single Message Transformations (SMT)](https://github.com/aiven/aiven-kafka-connect-transforms) 115 | 116 | ### Testing 117 | * [Mocked Streams - Topology testing library for Kafka Streams (no broker required)](https://github.com/jpzk/mockedstreams) 118 | * [Unit-testing for embedded kafka services](https://github.com/Landoop/kafka-testing) 119 | * [Kafka JUnit](https://github.com/salesforce/kafka-junit) 120 | * [Kafka Unit](https://github.com/chbatey/kafka-unit) 121 | * [Kafka for JUnit](https://github.com/mguenther/kafka-junit) 122 | * [Karate and Kafka](https://github.com/karatelabs/karate-examples/blob/main/kafka/README.md) 123 | * [embedded-kafka - in-memory Kafka instance to run your tests against](https://github.com/embeddedkafka/embedded-kafka) 124 | * [Kafka Containers](https://www.testcontainers.org/modules/kafka/) 125 | * [Mockintosh - through asynchronous Kafka actors](https://mockintosh.io/) 126 | 127 | 128 | ## Operations 129 | 130 | ### CI/CD 131 | * [kafka-gitops](https://github.com/devshawn/kafka-gitops) 132 | 133 | ### Operational Utilities 134 | * [Cruise control](https://github.com/linkedin/cruise-control) 135 | * [DoctorKafka](https://github.com/pinterest/doctorkafka/) and its successor [Orion](https://github.com/pinterest/orion) 136 | * [Brucke - Inter-cluster bridge of kafka topics](https://github.com/klarna/brucke) 137 | * [Kafka-Kit](https://github.com/DataDog/kafka-kit) 138 | * [Secor - service for persisting Kafka logs](https://github.com/pinterest/secor) 139 | * [kafkabalancer](https://github.com/CAFxX/kafkabalancer) 140 | * [Yelp Kafka-Utils](https://github.com/Yelp/kafka-utils) 141 | * [Cerner Common Kafka utilities](https://github.com/cerner/common-kafka) 142 | * [connectctl - manage kafka connect connectors easily](https://github.com/90poe/connectctl) 143 | 144 | ### Monitoring 145 | * [Kafka Offset Exporter](https://github.com/echojc/kafka-offset-exporter) 146 | * [Fluent plugin](https://github.com/fluent/fluent-plugin-kafka) 147 | * [Burrow - Kafka Consumer Lag Checking](https://github.com/linkedin/Burrow) 148 | * [Remora](https://github.com/zalando-incubator/remora) 149 | * [Kafka Offset Monitoring](https://github.com/Morningstar/kafka-offset-monitor) 150 | * [kafka-eagle](https://github.com/smartloli/kafka-eagle) 151 | * [Kafka-prometheus](https://github.com/streamthoughts/kafka-monitoring-suite-demo-prometheus) 152 | * [Consumer lag monitoring](https://github.com/omarsmak/kafka-consumer-lag-monitoring) 153 | 154 | ### Performance tools 155 | * [sangrenel](https://github.com/jamiealquiza/sangrenel) 156 | * [KLoadGen - load generator plugin for jmeter](https://github.com/corunet/kloadgen) 157 | 158 | ### Security 159 | * [Kafka Security Manager](https://github.com/simplesteph/kafka-security-manager) 160 | * [Kafka security playbook](https://github.com/Dabz/kafka-security-playbook) 161 | 162 | ### Audit 163 | * [chaperone - A Kafka audit system](https://github.com/uber/chaperone) 164 | 165 | ### Mirroring 166 | * [MirrorMaker](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330) 167 | * [Message Handler For MirrorMaker to change topic names](https://github.com/opencore/mirrormaker_topic_rename) 168 | * [uReplicator](https://github.com/uber/uReplicator) 169 | * [MirrorTool for Kafka Connect](https://github.com/Comcast/MirrorTool-for-Kafka-Connect) 170 | * [Mirus](https://github.com/salesforce/mirus) 171 | 172 | ### Backup 173 | * [Backup and Restore topics & offsets](https://github.com/itadventurer/kafka-backup) 174 | 175 | ### Tools 176 | 177 | * [Karapace - Your Kafka essentials in one tool](https://github.com/aiven/karapace) 178 | * [REST-proxy](https://github.com/confluentinc/kafka-rest) 179 | * [Kafka-Pixy - gRPC/REST Proxy](https://github.com/mailgun/kafka-pixy) 180 | * [Kafka Proxy providing SASL/PLAIN authentication and SSL](https://github.com/grepplabs/kafka-proxy) 181 | * [Reactive/Akka API](https://github.com/akka/reactive-kafka) 182 | * [kafkat](https://github.com/airbnb/kafkat) 183 | * [kt](https://github.com/fgeller/kt) 184 | * [kafka-connect-tools](https://github.com/datamountaineer/kafka-connect-tools) 185 | * [Node Kafka connect](https://github.com/nodefluent/kafka-connect) 186 | * [Kafka-monitor](https://github.com/linkedin/kafka-monitor) 187 | * [Kafka Auto completion](https://github.com/Landoop/kafka-autocomplete) 188 | * [Kafka Cheet Sheet](https://github.com/Landoop/kafka-cheat-sheet) 189 | * [Kafka Write Protocol encoder/decoder](https://github.com/klarna/kafka_protocol) 190 | * [Winton Kafka Streams](https://github.com/wintoncode/winton-kafka-streams) 191 | * [Spring-kafka](https://github.com/spring-projects/spring-kafka) 192 | * [Maven quick start for Kafka Connect connectors](https://github.com/jcustenborder/kafka-connect-archtype) 193 | * [Trifecta](https://github.com/ldaniels528/trifecta) 194 | * [Kafka Specs](https://github.com/streamthoughts/kafka-specs) 195 | * [Strimzi Kafka CLI](https://github.com/systemcraftsman/strimzi-kafka-cli) 196 | 197 | ### Metadata Management 198 | * [Go library for Kafka metadata in Zookeeper](https://github.com/wvanbergen/kazoo-go) 199 | * [Meteor](https://github.com/odpf/meteor) - Meteor is a scalable, easy-to-use, extensible metadata collection framework from the different cloud providers and on-prem sources and publish to Kafka. 200 | 201 | ### Schema Management 202 | * [Stencil](https://github.com/odpf/stencil) - Stencil is a Protobuf schema registry for Kafka that enables you to create, maintain and consume protobuf messages and APIs dynamically, efficiently, and reliably. 203 | * [Schema registry](https://github.com/confluentinc/schema-registry) 204 | * [Node Schema registry](https://github.com/nodefluent/schema-registry) 205 | * [Schema Registry Transfer SMT](https://github.com/cricket007/schema-registry-transfer-smt) 206 | * [Expedia Stream Registry](https://github.com/ExpediaGroup/stream-registry) 207 | 208 | ### UI 209 | * [kafka-manager](https://github.com/yahoo/kafka-manager) 210 | * [Kafka HQ](https://github.com/tchiotludo/kafkahq) 211 | * [Kafdrop 3.x](https://github.com/obsidiandynamics/kafdrop) – web UI for Kafka. For versions older than 0.10.0 use [Kafdrop 2.x](https://github.com/HomeAdvisor/Kafdrop). 212 | * [Kafka Webview](https://github.com/SourceLabOrg/kafka-webview) 213 | * [Kafka Topics UI](https://github.com/Landoop/kafka-topics-ui) 214 | * [Kafka Connect UI](https://github.com/Landoop/kafka-connect-ui) 215 | * [Kafka Schema UI](https://github.com/Landoop/schema-registry-ui) 216 | * [Kafkawize - self service topic management portal](https://github.com/muralibasani/kafkawize) 217 | * [Kadmin - Kafka Producer/Consumer UI](https://github.com/BetterCloud/kadmin) 218 | * [Kowl](https://github.com/cloudhut/kowl) 219 | * [Kafka Streams Topology Sketch Diagram Visualization](https://github.com/zz85/kafka-streams-viz) 220 | * [Web based GUI](https://github.com/provectus/kafka-ui) 221 | 222 | ## Deployment 223 | 224 | ### Kubernetes 225 | * [Kafka k8s operator](https://github.com/krallistic/kafka-operator) 226 | * [Strimzi](https://github.com/strimzi/strimzi-kafka-operator) Operator for deploying and running Apache Kafka on Kubernetes 227 | * [konsumerator - Kafka Consumer Operator](https://github.com/lwolf/konsumerator) 228 | 229 | ### Docker 230 | * [fast-data-dev](https://github.com/Landoop/fast-data-dev) 231 | 232 | ### Integrations 233 | 234 | * [Ecosystem](https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem) 235 | * [hermes](https://github.com/allegro/hermes) 236 | * [IBMStreams](https://github.com/IBMStreams/streamsx.kafka) 237 | * [Brooklin](https://github.com/linkedin/brooklin) 238 | 239 | ## Alternative Implementations 240 | 241 | * [Go](https://github.com/travisjeffery/jocko) 242 | * [C/C++](https://github.com/edenhill/librdkafka) 243 | 244 | ## Projects built on top of Kafka 245 | * [In-Memory Cache Backed by Kafka](https://github.com/rayokota/kcache) 246 | * [ransactional Metadata Store Backed by Kafka](https://github.com/rayokota/keta) 247 | 248 | ## Unsorted 249 | 250 | * https://github.com/Landoop/kafka-connect-kcql-smt 251 | * https://github.com/nodefluent/purpur 252 | * [Azkarra Streams - lightweight java framework which makes easy to create production-ready Kafka Streams applications](https://github.com/streamthoughts/azkarra-streams) 253 | --------------------------------------------------------------------------------