├── .gitignore ├── README.md ├── clients.md ├── connectors.md ├── integrations.md ├── learning.md ├── libraries.md ├── social.md ├── testing.md └── tools.md /.gitignore: -------------------------------------------------------------------------------- 1 | # Created by .ignore support plugin (hsz.mobi) 2 | ### JetBrains template 3 | # Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio and WebStorm 4 | # Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839 5 | 6 | # User-specific stuff 7 | .idea 8 | 9 | awesome-kafka.iml 10 | 11 | # Sensitive or high-churn files 12 | .idea/**/dataSources/ 13 | .idea/**/dataSources.ids 14 | .idea/**/dataSources.local.xml 15 | .idea/**/sqlDataSources.xml 16 | .idea/**/dynamic.xml 17 | .idea/**/uiDesigner.xml 18 | 19 | # Gradle 20 | .idea/**/gradle.xml 21 | .idea/**/libraries 22 | 23 | # CMake 24 | cmake-build-debug/ 25 | cmake-build-release/ 26 | 27 | # Mongo Explorer plugin 28 | .idea/**/mongoSettings.xml 29 | 30 | # File-based project format 31 | *.iws 32 | 33 | # IntelliJ 34 | out/ 35 | 36 | # mpeltonen/sbt-idea plugin 37 | .idea_modules/ 38 | 39 | # JIRA plugin 40 | atlassian-ide-plugin.xml 41 | 42 | # Cursive Clojure plugin 43 | .idea/replstate.xml 44 | 45 | # Crashlytics plugin (for Android Studio and IntelliJ) 46 | com_crashlytics_export_strings.xml 47 | crashlytics.properties 48 | crashlytics-build.properties 49 | fabric.properties 50 | 51 | # Editor-based Rest Client 52 | .idea/httpRequests 53 | ### Windows template 54 | # Windows thumbnail cache files 55 | Thumbs.db 56 | ehthumbs.db 57 | ehthumbs_vista.db 58 | 59 | # Dump file 60 | *.stackdump 61 | 62 | # Folder config file 63 | [Dd]esktop.ini 64 | 65 | # Recycle Bin used on file shares 66 | $RECYCLE.BIN/ 67 | 68 | # Windows Installer files 69 | *.cab 70 | *.msi 71 | *.msix 72 | *.msm 73 | *.msp 74 | 75 | # Windows shortcuts 76 | *.lnk 77 | ### Linux template 78 | *~ 79 | 80 | # temporary files which can be created if a process still has a handle open of a deleted file 81 | .fuse_hidden* 82 | 83 | # KDE directory preferences 84 | .directory 85 | 86 | # Linux trash folder which might appear on any partition or disk 87 | .Trash-* 88 | 89 | # .nfs files are created when an open file is removed but is still being accessed 90 | .nfs* 91 | ### macOS template 92 | # General 93 | .DS_Store 94 | .AppleDouble 95 | .LSOverride 96 | 97 | # Icon must end with two \r 98 | Icon 99 | 100 | # Thumbnails 101 | ._* 102 | 103 | # Files that might appear in the root of a volume 104 | .DocumentRevisions-V100 105 | .fseventsd 106 | .Spotlight-V100 107 | .TemporaryItems 108 | .Trashes 109 | .VolumeIcon.icns 110 | .com.apple.timemachine.donotpresent 111 | 112 | # Directories potentially created on remote AFP share 113 | .AppleDB 114 | .AppleDesktop 115 | Network Trash Folder 116 | Temporary Items 117 | .apdisk 118 | 119 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # awesome-kafka 2 | 3 | This list is for anyone wishing to learn about [Apache Kafka](http://kafka.apache.org/), but do not have a starting point. 4 | 5 | #### How to contribute 6 | 7 | Fork the repository, create a contribution, and create a pull request against monksy/awesome-kafka:master. 8 | 9 | 10 | Table of Contents 11 | ================= 12 | 13 | * [Learning/Resources](learning.md) 14 | * [Tools/Utilities/Monitoring](tools.md) 15 | * [Client Libraries](clients.md) 16 | * [Libraries with Kafka Support](libraries.md) 17 | * [Kafka Connectors](connectors.md) 18 | * [Testing](testing.md) 19 | * [Projects with Kafka Integrations](integrations.md) 20 | * [Social Resources](social.md) 21 | 22 | ## Requests for Help/TODO 23 | 24 | - Learning: Organize articles into tutorials, case studies, product related, reference, etc 25 | - Create a section for resources that will help to quickstart learning for kafka and kafka streams 26 | - Add about sections per each page. 27 | - Merge in repo: https://github.com/semantalytics/awesome-kafka 28 | - Find Example projects with kafka 29 | - Organize learning.md by the types/subsystem that they are using 30 | - Add section headers 31 | - Find blogs that are dedicated to Kafka 32 | - Create sections for all the sub-sections 33 | - Kafka Streams 34 | - KSQL 35 | - Kafka Connect 36 | - etc 37 | - Solicit for help from the community 38 | - Get involved with Kafka community (gitter/slack/irc/mailing list) 39 | - Fix links in learning 40 | - Verify links work 41 | 42 | ## Where did this information come from? 43 | 44 | - Originally from: https://github.com/infoslack/awesome-kafka 45 | - Client Libraries, Tools, etc from https://github.com/dharmeshkakadia/awesome-kafka 46 | - Awesome-data-engineering list for kafka: https://github.com/monksy/awesome-data-engineering -------------------------------------------------------------------------------- /clients.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | ### Go 4 | 5 | * [Shopify's Sarama](https://github.com/Shopify/sarama) 6 | * [JDamick's Kafka](https://github.com/jdamick/kafka) 7 | * [Optionpay's Kafka](https://github.com/optiopay/kafka) 8 | * [Go Kafka streaming](https://github.com/lovoo/goka) 9 | 10 | ### Javascript/Node 11 | 12 | * [Oleksiyk's Kafka](https://github.com/oleksiyk/kafka) 13 | * [Node Sinek](https://github.com/nodefluent/node-sinek) 14 | * [Node Kafka streams](https://github.com/nodefluent/kafka-streams) 15 | * [Node librdkafka](https://github.com/nodefluent/node-rdkafka) 16 | * [kafka-node](https://github.com/SOHU-Co/kafka-node) Node.js client for Apache Kafka 0.9+ 17 | * [Kafka-logger](https://github.com/uber/kafka-logger) Kafka-winston logger for nodejs from uber 18 | 19 | ### PHP 20 | * [Kafka-php](https://github.com/weiboad/kafka-php) 21 | * [php-rdkafka](https://github.com/arnaud-lb/php-rdkafka) 22 | 23 | ### Ruby 24 | 25 | * [Ruby](https://github.com/zendesk/ruby-kafka) 26 | * [Phobos - Ruby App framework](https://github.com/klarna/phobos) 27 | * [Karafka - Ruby and Rails framework](https://github.com/karafka/karafka) 28 | * [JRuby](https://github.com/joekiller/jruby-kafka) 29 | 30 | ### Rust 31 | * [kafka](https://github.com/spicavigo/kafka-rust) pure rust, has major issues and at the time without maintainer 32 | * [rdkafka](https://github.com/fede1024/rust-rdkafka) wrapping librdkafka, full async support 33 | * [schema_registry_converter](https://github.com/gklijs/schema_registry_converter) for using Confluent schema registry 34 | 35 | ### Other 36 | 37 | * [Python](https://github.com/Parsely/pykafka) 38 | * [Perl](https://github.com/TrackingSoft/Kafka) 39 | * [Dart](https://github.com/dart-kafka/kafka) 40 | * [Elixir](https://github.com/kafkaex/kafka_ex) 41 | * [Erlang](https://github.com/klarna/brod) 42 | * [LinkedIn wrapper library](https://github.com/linkedin/li-apache-kafka-clients) 43 | * [kasper - a lightweight library for processing Kafka topics](https://github.com/movio/kasper) 44 | * [librdkafka](https://github.com/edenhill/librdkafka) The Apache Kafka C/C++ library 45 | 46 | 47 | ### Kafka Streams Libraries 48 | * [Scala](https://github.com/lightbend/kafka-streams-scala) *Merged into Kafka 2.0* - [KAFKA-6670](https://issues.apache.org/jira/browse/KAFKA-6670) 49 | * [faust - Python Stream Processing](https://github.com/robinhood/faust) 50 | * [Python](https://github.com/wintoncode/winton-kafka-streams) 51 | 52 | ### KSQl 53 | * [KSQL JDBC Driver](https://github.com/mmolimar/ksql-jdbc-driver) 54 | -------------------------------------------------------------------------------- /connectors.md: -------------------------------------------------------------------------------- 1 | * [Apache Camel Kafka Connectors](https://camel.apache.org/camel-kafka-connector/latest/) - 340+ Apache Camel components as Kafka Connect connectors like Slack, Google Sheets, AWS S3 among others 2 | * [JDBC](https://github.com/confluentinc/kafka-connect-jdbc) 3 | * [Elasticsearch](https://github.com/confluentinc/kafka-connect-elasticsearch) 4 | * [Elasticsearch 5 with XPack support](https://github.com/Landoop/stream-reactor/tree/master/kafka-connect-elastic5) 5 | * [Elasticsearch 6 with XPack support](https://github.com/Landoop/stream-reactor/tree/master/kafka-connect-elastic6) 6 | * [HDFS](https://github.com/confluentinc/kafka-connect-hdfs) 7 | * [BigQuery](https://github.com/wepay/kafka-connect-bigquery) 8 | * [SQL](https://github.com/nodefluent/sequelize-kafka-connect) 9 | * [GCloud pubsub](https://github.com/nodefluent/gcloud-pubsub-kafka-connect) 10 | * [BigTable](https://github.com/nodefluent/bigtable-kafka-connect) 11 | * [HBase](https://github.com/mravi/kafka-connect-hbase) 12 | * [FileSystem](https://github.com/mmolimar/kafka-connect-fs) 13 | * [kafka-connect-protobuf-converter](https://github.com/blueapron/kafka-connect-protobuf-converter) 14 | * [IBM MQ](https://github.com/ibm-messaging/kafka-connect-mq-source) 15 | * [Azure IoT Hub](https://github.com/Azure/toketi-kafka-connect-iothub) 16 | * [FTP](https://github.com/Landoop/stream-reactor/tree/master/kafka-connect-ftp) 17 | * [Cassandra](https://github.com/Landoop/stream-reactor/tree/master/kafka-connect-cassandra) 18 | * [Redis](https://github.com/Landoop/stream-reactor/tree/master/kafka-connect-redis) 19 | * [MongoDB](https://github.com/Landoop/stream-reactor/tree/master/kafka-connect-mongodb) 20 | * [MQTT](https://github.com/Landoop/stream-reactor/tree/master/kafka-connect-mqtt) 21 | * [Hazelcast](https://github.com/Landoop/stream-reactor/tree/master/kafka-connect-hazelcast) 22 | * [kafka-connect-rest](https://github.com/llofberg/kafka-connect-rest) 23 | * [kafka-connect-aws-lambda](https://github.com/llofberg/kafka-connect-aws-lambda) 24 | * [kafka-connect-s3-parquet](https://github.com/llofberg/kafka-connect-s3-parquet) 25 | -------------------------------------------------------------------------------- /integrations.md: -------------------------------------------------------------------------------- 1 | * [Ecosystem](https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem) 2 | * [asyncio](https://github.com/aio-libs/aiokafka) 3 | * [hermes](https://github.com/allegro/hermes) 4 | * [pg-kafka](https://github.com/xstevens/pg_kafka) A PostgreSQL extension to produce messages to Apache Kafka 5 | * [Secor](https://github.com/pinterest/secor) Pinterest's Kafka to S3 distributed consumer 6 | * [Camus](https://github.com/linkedin/camus) LinkedIn's Kafka to HDFS pipeline. 7 | * [BottledWater](https://github.com/confluentinc/bottledwater-pg) Change data capture from PostgreSQL into Kafka 8 | -------------------------------------------------------------------------------- /learning.md: -------------------------------------------------------------------------------- 1 | 2 | Table of Contents 3 | ================= 4 | * [Articles](#articles) 5 | * [Talks / Presentations](#talks--presentations) 6 | * [Books](#books) 7 | * [Courses](#courses) 8 | * [Blogs](#blogs) 9 | * [Papers](#papers) 10 | 11 | ## Articles 12 | * [Kafka Cheet Sheet](https://github.com/Landoop/kafka-cheat-sheet) 13 | * [Apache Kafka for Beginners](http://blog.cloudera.com/blog/2014/09/apache-kafka-for-beginners/) 14 | * [Introduction to Kafka](http://sysadvent.blogspot.com.br/2014/12/day-4-introduction-to-kafka.html) 15 | * [Understanding Kafka with Factorio](https://hackernoon.com/understanding-kafka-with-factorio-74e8fc9bf181) 16 | * [Introducing Kafka Streams: Stream Processing Made Simple](http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple) 17 | * [Apache Kafka, Samza, and the Unix Philosophy of Distributed Data](http://www.confluent.io/blog/apache-kafka-samza-and-the-unix-philosophy-of-distributed-data) 18 | * [Using logs to build a solid data infrastructure (or: why dual writes are a bad idea)](http://www.confluent.io/blog/using-logs-to-build-a-solid-data-infrastructure-or-why-dual-writes-are-a-bad-idea/) 19 | * [Bottled Water: Real-time integration of PostgreSQL and Kafka](http://www.confluent.io/blog/bottled-water-real-time-integration-of-postgresql-and-kafka/) 20 | * [Deploying Apache Kafka on AWS Elastic Block Store (EBS)](http://www.confluent.io/blog/deploying-apache-kafka-on-aws-elastic-block-store-ebs) 21 | * [How We Monitor and Run Kafka At Scale](http://www.confluent.io/blog/how-we-monitor-and-run-kafka-at-scale-signalfx) 22 | * [Monitoring Kafka performance metrics](https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics/) 23 | * [Kafka for uSwitch's Event Pipeline](http://oobaloo.co.uk/kafka-for-uswitchs-event-pipeline) 24 | * [Next Gen Real-time Streaming with Storm-Kafka Integration](http://blog.infochimps.com/2012/10/30/next-gen-real-time-streaming-storm-kafka-integration/) 25 | * [all netflix tech blog kafka tagged articles](http://techblog.netflix.com/search/label/kafka) 26 | * [all linked in tech blog kafka tagged articles](https://engineering.linkedin.com/blog/topic/kafka) 27 | * [How to use Apache Kafka to transform a batch pipeline into a real-time one](https://medium.com/@stephane.maarek/how-to-use-apache-kafka-to-transform-a-batch-pipeline-into-a-real-time-one-831b48a6ad85) 28 | * [Event Sourcing: What it is and why it's awesome](https://dev.to/barryosull/event-sourcing-what-it-is-and-why-its-awesome) 29 | * [Maven quick start for Kafka Connect connectors](https://github.com/jcustenborder/kafka-connect-archtype) 30 | * [Introduction to Stream Processing using Kafka Streams](https://www.loginradius.com/engineering/blog/stream-processing-using-kafka/) 31 | 32 | ## Talks / Presentations 33 | 34 | * [Introduction to Apache Kafka by Joe Stein](https://www.youtube.com/watch?v=qc33qMUvR7c) 35 | * [Apache Kafka and the Next 700 Stream Processing Systems by Jay Kreps](https://www.youtube.com/watch?v=9RMOc0SwRro) 36 | * [Using Logs To Build a Solid Data Infrastructure - Martin Kleppmann (LinkedIn) ](http://www.ustream.tv/recorded/61479591) 37 | * [I ♥ Logs: Apache Kafka and Real-Time Data Integration](https://www.youtube.com/watch?v=aJuo_bLSW6s) 38 | * [Apache Kafka: Real-time Streaming and Data Pipelines with Apache Kafka by Joe Stein](https://www.youtube.com/watch?v=InAKDEk7H0M) 39 | * [Chris Curtin - AJUG - Apache Kafka](https://vimeo.com/63040812) 40 | * [Getting started guide on Apache Kafka by Learning Journal](https://www.youtube.com/playlist?list=PLkz1SCf5iB4enAR00Z46JwY9GGkaS2NON) 41 | * [Short demo of how to do joins in Kafka Streams](https://youtu.be/U_FTlAx9rGA) 42 | * [Developing with the Go client for Apache Kafka](http://www.slideshare.net/charmalloc/developing-with-the-go-client-for-apache-kafka) 43 | * [Verisign - Apache Kafka 0.8 basic training](http://www.slideshare.net/miguno/apache-kafka-08-basic-training-verisign) 44 | * [Developing Real-Time Data Pipelines with Apache Kafka](http://www.slideshare.net/charmalloc/developingwithapachekafka-29910685) 45 | * [Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303) | AWS re:Invent 2013](http://www.slideshare.net/AmazonWebServices/infrastructure-at-scale-apache-kafka-twitter-storm-elastic-search-arc303-aws-reinvent-2013) 46 | * [Real-time streaming and data pipelines with Apache Kafka](http://www.slideshare.net/charmalloc/real-timestreamingdata-pipelinesapachekafka) 47 | * [Building a Real-time Data Pipeline: Apache Kafka at LinkedIn](http://www.slideshare.net/Hadoop_Summit/building-a-realtime-data-pipeline-apache-kafka-at-linkedin) 48 | * [Kafka replication apachecon_2013](http://www.slideshare.net/junrao/kafka-replication-apachecon2013) 49 | * [Introduction and Overview of Apache Kafka](http://www.slideshare.net/mumrah/kafka-talk-tri-hug) 50 | 51 | ## Books 52 | 53 | * [Making sense of stream processing](http://www.confluent.io/making-sense-of-stream-processing-ebook) 54 | * [Designing Data-Intensive Applications](http://shop.oreilly.com/product/0636920032175.do) 55 | * [Effective Kafka - a hands-on guide to building robust and scalable event-driven applications](https://apachekafkabook.com/) 56 | * [Kafka: The Definitive Guide](https://www.confluent.io/wp-content/uploads/confluent-kafka-definitive-guide-complete.pdf) 57 | * [A fast-paced introduction to every aspect of working with Kafka you need to really reap its benefits.](https://www.manning.com/books/kafka-in-action) 58 | * [By the end of the book, you'll be ready to use Kafka Streams in your projects to reap the benefits of the insight your data holds quickly and easily.](https://www.manning.com/books/kafka-streams-in-action) 59 | * [Kafka Streams in Action, Second Edition](https://www.manning.com/books/kafka-streams-in-action-second-edition) 60 | * [Streaming Data Pipelines with Kafka](https://www.manning.com/books/streaming-data-pipelines-with-kafka) 61 | 62 | ## Courses 63 | 64 | * [Udemy Apache Kafka Series - Learning Apache Kafka for Beginners](https://goo.gl/GaCKQN) - All Level 65 | * [Udemy Apache Kafka Series - Kafka Connect Hands-on Learning](https://goo.gl/wLLLY9) - Intermediate 66 | * [Udemy Apache Kafka Series - Kafka Streams for Data Processing](https://goo.gl/bro314) - Intermediate 67 | * [Udemy Apache Kafka Series - Kafka Cluster Setup & Administration](https://goo.gl/1uYAuU) - Expert 68 | * [Udemy Apache Kafka Series - Confluent Schema Registry & REST Proxy](https://goo.gl/XgWcVz) - Intermediate 69 | 70 | ## Blogs 71 | 72 | ## Papers 73 | 74 | * [Building a Replicated Logging System with Apache Kafka](http://www.vldb.org/pvldb/vol8/p1654-wang.pdf) 75 | * [Kafka: a Distributed Messaging System for Log Processing](http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf) 76 | * [Building LinkedIn’s Real-time Activity Data Pipeline](http://sites.computer.org/debull/A12june/pipeline.pdf) 77 | * [The Log: What every software engineer should know about real-time data's unifying abstraction](https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying) 78 | 79 | 80 | -------------------------------------------------------------------------------- /libraries.md: -------------------------------------------------------------------------------- 1 | 2 | ## Kafka 3 | 4 | * [Reactive/Akka API](https://github.com/akka/reactive-kafka) 5 | * [Exlir Kafka Write Protocol encoder/decoder](https://github.com/klarna/kafka_protocol) 6 | * [Spring-kafka](https://github.com/spring-projects/spring-kafka) 7 | * [Jackdaw — Simple configuration and mocking of Kafka clients in Java](https://github.com/obsidiandynamics/jackdaw) 8 | 9 | ## KSQL 10 | 11 | * [KSQL Addon - User Defined Function (UDF) for Machine Learning](https://github.com/kaiwaehner/ksql-machine-learning-udf) 12 | 13 | ## Kafka Connect 14 | 15 | * [Node Kafka connect](https://github.com/nodefluent/kafka-connect) 16 | -------------------------------------------------------------------------------- /social.md: -------------------------------------------------------------------------------- 1 | 2 | ## Forums 3 | 4 | * [r/ApacheKafka](https://reddit.com/r/ApacheKafka) 5 | * [StackOverflow](http://stackoverflow.com/questions/tagged/apache-kafka-connect+or+apache-kafka+or+confluent-kafka+or+KSQL) 6 | 7 | ## Chats 8 | 9 | * IRC: chat.freenode.net in #apache-kafka 10 | * [Confluent Community Slack](https://launchpass.com/confluentcommunity) 11 | 12 | ## Mailing Lists 13 | 14 | * User's list: users-subscribe@kafka.apache.org 15 | * Developer's list: dev-subscribe@kafka.apache.org 16 | * [Confluent Platform Mailing list](https://groups.google.com/forum/#!forum/confluent-platform) 17 | -------------------------------------------------------------------------------- /testing.md: -------------------------------------------------------------------------------- 1 | * [Mocked Streams - Topology testing library for Kafka Streams (no broker required)](https://github.com/jpzk/mockedstreams) 2 | * [Unit-testing for embedded kafka services](https://github.com/Landoop/kafka-testing) 3 | * [Synthetic avro message generator for Kafka](https://github.com/Landoop/landoop-avro-generator) 4 | * [Kafka JUnit](https://github.com/salesforce/kafka-junit) 5 | * [Kafka Unit](https://github.com/chbatey/kafka-unit) 6 | * [Kafka for JUnit](https://github.com/mguenther/kafka-junit) 7 | * [embedded-kafka - in-memory Kafka instance to run your tests against](https://github.com/embeddedkafka/embedded-kafka) 8 | -------------------------------------------------------------------------------- /tools.md: -------------------------------------------------------------------------------- 1 | Table of Contents 2 | ================= 3 | 4 | * [Monitoring](#monitoring) 5 | * [Utility](#utility) 6 | * [Deployment Resources](#deployment-resources) 7 | * [Dashboards](#dashboards) 8 | * [Mirroring](#mirroring) 9 | * [Schema](#schema) 10 | * [Management](#management) 11 | * [Security](#security) 12 | * [Audit](#audit) 13 | * [Archival](#archival) 14 | * [Integrations](#integrations) 15 | * [Administration](#administration) 16 | * [Load Testing/Performance](#load-testingperformance) 17 | 18 | ## Monitoring 19 | * [Availablity Monitor](https://github.com/Microsoft/Availability-Monitor-for-Kafka) 20 | * [Burrow](https://github.com/linkedin/Burrow) A utility used to monitor consumer groups for lag and report when it exceeds a criteria. 21 | * [Kafka Offset Exporter](https://github.com/echojc/kafka-offset-exporter) 22 | * [Fluent plugin](https://github.com/fluent/fluent-plugin-kafka) 23 | * [Remora](https://github.com/zalando-incubator/remora) 24 | * [Kafka Offset Monitoring](https://github.com/Morningstar/kafka-offset-monitor) 25 | * [kafka-eagle](https://github.com/smartloli/kafka-eagle) 26 | * [Kafka-monitor](https://github.com/linkedin/kafka-monitor) 27 | 28 | ## Utility 29 | * [Burrower](https://github.com/splee/burrower) A utiltiy to convert the output from Burrow and report it to InfluxDB 30 | * [Brucke - Inter-cluster bridge of kafka topics](https://github.com/klarna/brucke) 31 | * [Kafka-Kit](https://github.com/DataDog/kafka-kit) 32 | * [Yelp Kafka-Utils](https://github.com/Yelp/kafka-utils) 33 | * [kt](https://github.com/fgeller/kt) 34 | * [kafka-connect-tools](https://github.com/datamountaineer/kafka-connect-tools) 35 | * [Kafka Auto completion](https://github.com/Landoop/kafka-autocomplete) 36 | * [Go library for Kafka metadata in Zookeeper](https://github.com/wvanbergen/kazoo-go) 37 | * [kafkacat](https://github.com/edenhill/kafkacat) Generic command line non-JVM Apache Kafka producer and consumer 38 | * [Strimzi Kafka CLI](https://github.com/systemcraftsman/strimzi-kafka-cli) A CLI for Strimzi Kafka Operator 39 | 40 | 41 | ## Deployment Resources 42 | * [Strimzi](https://github.com/strimzi/strimzi-kafka-operator) Operator for deploying and running Apache Kafka on Kubernetes and OpenShift 43 | * [Kafka Helm Charts](https://github.com/helm/charts/tree/master/incubator/kafka) A helm chart to deploy Apache Kafka on Kubernetes 44 | * [Non-Helm Kafka Deployment for Kubernetes](https://github.com/Yolean/kubernetes-kafka) 45 | * [Docker: fast-data-dev](https://github.com/Landoop/fast-data-dev) 46 | * [kafka-docker](https://github.com/wurstmeister/kafka-docker) Kafka in Docker 47 | 48 | ## Dashboards 49 | * [Yahoo Kafka Manager](https://github.com/yahoo/kafka-manager) 50 | * [Kafka Webview](https://github.com/SourceLabOrg/kafka-webview) 51 | * [Kafka HQ](https://github.com/tchiotludo/kafkahq) 52 | * [Kafka Tool](http://www.kafkatool.com/)- Commercial Tool for managing Kafka 0.8 53 | * [Kafdrop](https://github.com/HomeAdvisor/Kafdrop) 54 | * [Kafdrop 3](https://github.com/obsidiandynamics/kafdrop) — Reboot of Kafdrop, adding support for Kafka 2.x and Kubernetes 55 | * [Kafka Topics UI](https://github.com/Landoop/kafka-topics-ui) 56 | * [Kafka Connect UI](https://github.com/Landoop/kafka-connect-ui) 57 | * [Kafka Schema UI](https://github.com/Landoop/schema-registry-ui) 58 | * [UI for Apache Kafka](https://github.com/kafbat/kafka-ui) 59 | 60 | 61 | ## Mirroring 62 | * [MirrorMaker](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330) 63 | * [uReplicator](https://github.com/uber/uReplicator) 64 | 65 | ## Schema 66 | * [apicurio-registry](https://github.com/Apicurio/apicurio-registry) - Apicurio API/schema registry (includes UI) 67 | * [Schema registry](https://github.com/confluentinc/schema-registry) 68 | * [Node Schema registry](https://github.com/nodefluent/schema-registry) 69 | * [Stencil](https://github.com/odpf/stencil) - Stencil is a Protobuf schema registry for Kafka that enables you to create, maintain and consume protobuf messages and APIs dynamically, efficiently, and reliably. 70 | 71 | ## Management 72 | * [Cruise control](https://github.com/linkedin/cruise-control) Manager for large Kafka Clusters 73 | * [DoctorKafka](https://github.com/pinterest/doctorkafka/) Monitors for broker failure and rebalences the cluster 74 | * [kafkabalancer](https://github.com/CAFxX/kafkabalancer) Rebalences data accross Kafka Brokers 75 | 76 | ## Security 77 | * [Kafka Security Manager](https://github.com/simplesteph/kafka-security-manager) 78 | 79 | ### Audit 80 | * [chaperone - A Kafka audit system](https://github.com/uber/chaperone) 81 | 82 | ## Archival 83 | * [Secor - service for persisting Kafka logs](https://github.com/pinterest/secor) 84 | 85 | ## Integrations 86 | * [REST-proxy](https://github.com/confluentinc/kafka-rest) 87 | * [strimzi-http-bridge](https://github.com/strimzi/strimzi-kafka-bridge) - Strimzi Kafka Bridge (AMQP & HTTP) 88 | * [Kafka-Pixy - gRPC/REST Proxy](https://github.com/mailgun/kafka-pixy) 89 | * [Kafka Proxy providing SASL/PLAIN authentication and SSL](https://github.com/grepplabs/kafka-proxy) 90 | * [Firehose](https://github.com/odpf/firehose) - Firehose is an extensible, no-code, and cloud-native service to load real-time streaming data from Kafka to data stores, data lakes, and analytical storage systems. 91 | * [Dagger](https://github.com/odpf/dagger) - Dagger is an easy-to-use, SQL framework to process built on top of Apache Flink for stateful processing of Kafka data. 92 | * [Raccoon](https://github.com/odpf/raccoon) - Raccoon is a high-throughput, low-latency service to collect events in real-time from your web, mobile apps, and services using multiple network protocols and publish to Kafka. 93 | * [Meteor](https://github.com/odpf/meteor) - Meteor is a scalable, easy-to-use, extensible metadata collection framework from the different cloud providers and on-prem sources and publish to Kafka. 94 | 95 | ## Administration 96 | * [kafkat](https://github.com/airbnb/kafkat) Simplified command-line administration for Kafka brokers 97 | 98 | ## Load Testing/Performance 99 | 100 | * [sangrenel](https://github.com/jamiealquiza/sangrenel) - Load Generation and monitoring 101 | 102 | ## Encryption 103 | * [kafkacrypto](https://github.com/tmcqueen-materials/kafkacrypto) - Message layer encryption for Kafka 104 | 105 | 106 | 107 | --------------------------------------------------------------------------------