├── README.adoc ├── quarkus-kafka-producer ├── .dockerignore ├── .gitignore ├── .mvn │ └── wrapper │ │ ├── MavenWrapperDownloader.java │ │ ├── maven-wrapper.jar │ │ └── maven-wrapper.properties ├── README.md ├── mvnw ├── mvnw.cmd ├── pom.xml └── src │ └── main │ ├── docker │ ├── Dockerfile.jvm │ └── Dockerfile.native │ ├── java │ └── org │ │ └── acme │ │ ├── ATDData.java │ │ ├── DataProducer.java │ │ ├── RNData.java │ │ └── dbs-kafka.code-workspace │ └── resources │ ├── META-INF │ └── resources │ │ └── index.html │ └── application.properties └── quarkus-kafka-streaming ├── .dockerignore ├── .gitignore ├── .mvn └── wrapper │ ├── MavenWrapperDownloader.java │ ├── maven-wrapper.jar │ └── maven-wrapper.properties ├── README.md ├── mvnw ├── mvnw.cmd ├── pom.xml └── src └── main ├── docker ├── Dockerfile.jvm └── Dockerfile.native ├── java └── org │ └── acme │ ├── DataProcessor.java │ ├── InteractiveQueries.java │ └── KafkaStreaming.java └── resources ├── META-INF └── resources │ └── index.html └── application.properties /README.adoc: -------------------------------------------------------------------------------- 1 | = Data Streaming with Kafka and Quarkus 2 | 3 | ==== Problem Statement 4 | 5 | The problem statement is described here 6 | https://developers.redhat.com/blog/2020/09/28/build-a-data-streaming-pipeline-using-kafka-streams-and-quarkus/ 7 | 8 | ==== Pre-requisite 9 | 10 | 1. Make sure you have docker and kakfa cli available in your laptop/env 11 | 12 | ==== Step 1: Run Kafka Cluster 13 | 14 | Make sure you have docker installed on your laptop. Run following command to start a kafka cluster 15 | ---- 16 | $ git clone https://github.com/wurstmeister/kafka-docker.git 17 | $ cd kafka-docker 18 | ---- 19 | 20 | open docker-compose-single-broker.yml and change the following line 21 | KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100 22 | 23 | to 24 | 25 | KAFKA_ADVERTISED_HOST_NAME: localhost 26 | 27 | ---- 28 | $ docker-compose -f docker-compose-single-broker.yml up -d 29 | ---- 30 | 31 | ==== Step 2: Create Kafka Topics 32 | 33 | We will now create left-stream-topic, right-stream-topic, stream-stream-outerjoin, processed-topic topics using kafka cli 34 | 35 | ---- 36 | $ cd 37 | $ ./kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic left-stream-topic 38 | $ ./kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic right-stream-topic 39 | $ ./kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic stream-stream-outerjoin 40 | $ ./kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic processed-topic 41 | 42 | ---- 43 | 44 | ==== Step 3: Run the streaming application 45 | 46 | ---- 47 | $ git clone https://github.com/kgshukla/data-streaming-kafka-quarkus.git 48 | $ cd data-streaming-kafka-quarkus/quarkus-kafka-streaming 49 | 50 | # start the application in dev mode. You could also package the application as a jar file and start using 'java -jar' command 51 | $ ./mvnw compile quarkus:dev 52 | 53 | ---- 54 | open a new terminal and initialize the application. This step wasn't needed if the application was defined with ApplicationScoped notation 55 | 56 | ---- 57 | $ curl localhost:8080/startstream 58 | ---- 59 | 60 | ==== Step 4: Run the producer application 61 | 62 | Open a new terminal and run the following commands 63 | ---- 64 | $ cd data-streaming-kafka-quarkus/quarkus-kafka-producer 65 | 66 | $ ./mvnw compile quarkus:dev 67 | 68 | ---- 69 | 70 | ==== Step 5: Watch the processed-topic topic 71 | 72 | ---- 73 | $ cd 74 | $ ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic processed-topic --property print.key=true --property print.timestamp=true 75 | ---- 76 | 77 | ==== Step 6: Simulate various use cases 78 | 79 | Use case 1 - Send few records to left-stream-topic, right-stream-topic topics 80 | 81 | Couple of records will be missing in left-stream-topic. You will notice that punctuator will process those records which have been put into state store. 82 | 83 | ---- 84 | $ curl localhost:8082/sendfewrecords 85 | ---- 86 | 87 | Use case 2 - Send one record to left-stream-topic only 88 | 89 | You will notice that punctuator will process those records which have been put into state store. 90 | 91 | ---- 92 | $ curl localhost:8082/sendoneleftrecord 93 | 94 | # to see records that are present in the state store and not yet processed 95 | $ curl localhost:8080/storedata 96 | ---- 97 | 98 | Use case 3 - send 100000 records to both left-stream-topic, right-stream-topic topics 99 | 100 | ---- 101 | $ curl localhost:8082/sendmanyrecords 102 | ---- 103 | -------------------------------------------------------------------------------- /quarkus-kafka-producer/.dockerignore: -------------------------------------------------------------------------------- 1 | * 2 | !target/*-runner 3 | !target/*-runner.jar 4 | !target/lib/* -------------------------------------------------------------------------------- /quarkus-kafka-producer/.gitignore: -------------------------------------------------------------------------------- 1 | # Eclipse 2 | .project 3 | .classpath 4 | .settings/ 5 | bin/ 6 | 7 | # IntelliJ 8 | .idea 9 | *.ipr 10 | *.iml 11 | *.iws 12 | 13 | # NetBeans 14 | nb-configuration.xml 15 | 16 | # Visual Studio Code 17 | .vscode 18 | 19 | # OSX 20 | .DS_Store 21 | 22 | # Vim 23 | *.swp 24 | *.swo 25 | 26 | # patch 27 | *.orig 28 | *.rej 29 | 30 | # Maven 31 | target/ 32 | pom.xml.tag 33 | pom.xml.releaseBackup 34 | pom.xml.versionsBackup 35 | release.properties -------------------------------------------------------------------------------- /quarkus-kafka-producer/.mvn/wrapper/MavenWrapperDownloader.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2007-present the original author or authors. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"); 5 | * you may not use this file except in compliance with the License. 6 | * You may obtain a copy of the License at 7 | * 8 | * http://www.apache.org/licenses/LICENSE-2.0 9 | * 10 | * Unless required by applicable law or agreed to in writing, software 11 | * distributed under the License is distributed on an "AS IS" BASIS, 12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | * See the License for the specific language governing permissions and 14 | * limitations under the License. 15 | */ 16 | import java.net.*; 17 | import java.io.*; 18 | import java.nio.channels.*; 19 | import java.util.Properties; 20 | 21 | public class MavenWrapperDownloader { 22 | 23 | private static final String WRAPPER_VERSION = "0.5.6"; 24 | /** 25 | * Default URL to download the maven-wrapper.jar from, if no 'downloadUrl' is provided. 26 | */ 27 | private static final String DEFAULT_DOWNLOAD_URL = "https://repo.maven.apache.org/maven2/io/takari/maven-wrapper/" 28 | + WRAPPER_VERSION + "/maven-wrapper-" + WRAPPER_VERSION + ".jar"; 29 | 30 | /** 31 | * Path to the maven-wrapper.properties file, which might contain a downloadUrl property to 32 | * use instead of the default one. 33 | */ 34 | private static final String MAVEN_WRAPPER_PROPERTIES_PATH = 35 | ".mvn/wrapper/maven-wrapper.properties"; 36 | 37 | /** 38 | * Path where the maven-wrapper.jar will be saved to. 39 | */ 40 | private static final String MAVEN_WRAPPER_JAR_PATH = 41 | ".mvn/wrapper/maven-wrapper.jar"; 42 | 43 | /** 44 | * Name of the property which should be used to override the default download url for the wrapper. 45 | */ 46 | private static final String PROPERTY_NAME_WRAPPER_URL = "wrapperUrl"; 47 | 48 | public static void main(String args[]) { 49 | System.out.println("- Downloader started"); 50 | File baseDirectory = new File(args[0]); 51 | System.out.println("- Using base directory: " + baseDirectory.getAbsolutePath()); 52 | 53 | // If the maven-wrapper.properties exists, read it and check if it contains a custom 54 | // wrapperUrl parameter. 55 | File mavenWrapperPropertyFile = new File(baseDirectory, MAVEN_WRAPPER_PROPERTIES_PATH); 56 | String url = DEFAULT_DOWNLOAD_URL; 57 | if(mavenWrapperPropertyFile.exists()) { 58 | FileInputStream mavenWrapperPropertyFileInputStream = null; 59 | try { 60 | mavenWrapperPropertyFileInputStream = new FileInputStream(mavenWrapperPropertyFile); 61 | Properties mavenWrapperProperties = new Properties(); 62 | mavenWrapperProperties.load(mavenWrapperPropertyFileInputStream); 63 | url = mavenWrapperProperties.getProperty(PROPERTY_NAME_WRAPPER_URL, url); 64 | } catch (IOException e) { 65 | System.out.println("- ERROR loading '" + MAVEN_WRAPPER_PROPERTIES_PATH + "'"); 66 | } finally { 67 | try { 68 | if(mavenWrapperPropertyFileInputStream != null) { 69 | mavenWrapperPropertyFileInputStream.close(); 70 | } 71 | } catch (IOException e) { 72 | // Ignore ... 73 | } 74 | } 75 | } 76 | System.out.println("- Downloading from: " + url); 77 | 78 | File outputFile = new File(baseDirectory.getAbsolutePath(), MAVEN_WRAPPER_JAR_PATH); 79 | if(!outputFile.getParentFile().exists()) { 80 | if(!outputFile.getParentFile().mkdirs()) { 81 | System.out.println( 82 | "- ERROR creating output directory '" + outputFile.getParentFile().getAbsolutePath() + "'"); 83 | } 84 | } 85 | System.out.println("- Downloading to: " + outputFile.getAbsolutePath()); 86 | try { 87 | downloadFileFromURL(url, outputFile); 88 | System.out.println("Done"); 89 | System.exit(0); 90 | } catch (Throwable e) { 91 | System.out.println("- Error downloading"); 92 | e.printStackTrace(); 93 | System.exit(1); 94 | } 95 | } 96 | 97 | private static void downloadFileFromURL(String urlString, File destination) throws Exception { 98 | if (System.getenv("MVNW_USERNAME") != null && System.getenv("MVNW_PASSWORD") != null) { 99 | String username = System.getenv("MVNW_USERNAME"); 100 | char[] password = System.getenv("MVNW_PASSWORD").toCharArray(); 101 | Authenticator.setDefault(new Authenticator() { 102 | @Override 103 | protected PasswordAuthentication getPasswordAuthentication() { 104 | return new PasswordAuthentication(username, password); 105 | } 106 | }); 107 | } 108 | URL website = new URL(urlString); 109 | ReadableByteChannel rbc; 110 | rbc = Channels.newChannel(website.openStream()); 111 | FileOutputStream fos = new FileOutputStream(destination); 112 | fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE); 113 | fos.close(); 114 | rbc.close(); 115 | } 116 | 117 | } 118 | -------------------------------------------------------------------------------- /quarkus-kafka-producer/.mvn/wrapper/maven-wrapper.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kgshukla/data-streaming-kafka-quarkus/65612de9813264d203ab9dcdf68bd461871e01db/quarkus-kafka-producer/.mvn/wrapper/maven-wrapper.jar -------------------------------------------------------------------------------- /quarkus-kafka-producer/.mvn/wrapper/maven-wrapper.properties: -------------------------------------------------------------------------------- 1 | distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.6.3/apache-maven-3.6.3-bin.zip 2 | wrapperUrl=https://repo.maven.apache.org/maven2/io/takari/maven-wrapper/0.5.6/maven-wrapper-0.5.6.jar 3 | -------------------------------------------------------------------------------- /quarkus-kafka-producer/README.md: -------------------------------------------------------------------------------- 1 | # quarkus-kafka-producer project 2 | 3 | This project uses Quarkus, the Supersonic Subatomic Java Framework. 4 | 5 | If you want to learn more about Quarkus, please visit its website: https://quarkus.io/ . 6 | 7 | ## Running the application in dev mode 8 | 9 | You can run your application in dev mode that enables live coding using: 10 | ``` 11 | ./mvnw quarkus:dev 12 | ``` 13 | 14 | ## Packaging and running the application 15 | 16 | The application can be packaged using `./mvnw package`. 17 | It produces the `quarkus-kafka-producer-1.0.0-SNAPSHOT-runner.jar` file in the `/target` directory. 18 | Be aware that it’s not an _über-jar_ as the dependencies are copied into the `target/lib` directory. 19 | 20 | The application is now runnable using `java -jar target/quarkus-kafka-producer-1.0.0-SNAPSHOT-runner.jar`. 21 | 22 | ## Creating a native executable 23 | 24 | You can create a native executable using: `./mvnw package -Pnative`. 25 | 26 | Or, if you don't have GraalVM installed, you can run the native executable build in a container using: `./mvnw package -Pnative -Dquarkus.native.container-build=true`. 27 | 28 | You can then execute your native executable with: `./target/quarkus-kafka-producer-1.0.0-SNAPSHOT-runner` 29 | 30 | If you want to learn more about building native executables, please consult https://quarkus.io/guides/building-native-image. -------------------------------------------------------------------------------- /quarkus-kafka-producer/mvnw: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # ---------------------------------------------------------------------------- 3 | # Licensed to the Apache Software Foundation (ASF) under one 4 | # or more contributor license agreements. See the NOTICE file 5 | # distributed with this work for additional information 6 | # regarding copyright ownership. The ASF licenses this file 7 | # to you under the Apache License, Version 2.0 (the 8 | # "License"); you may not use this file except in compliance 9 | # with the License. You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, 14 | # software distributed under the License is distributed on an 15 | # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 16 | # KIND, either express or implied. See the License for the 17 | # specific language governing permissions and limitations 18 | # under the License. 19 | # ---------------------------------------------------------------------------- 20 | 21 | # ---------------------------------------------------------------------------- 22 | # Maven Start Up Batch script 23 | # 24 | # Required ENV vars: 25 | # ------------------ 26 | # JAVA_HOME - location of a JDK home dir 27 | # 28 | # Optional ENV vars 29 | # ----------------- 30 | # M2_HOME - location of maven2's installed home dir 31 | # MAVEN_OPTS - parameters passed to the Java VM when running Maven 32 | # e.g. to debug Maven itself, use 33 | # set MAVEN_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 34 | # MAVEN_SKIP_RC - flag to disable loading of mavenrc files 35 | # ---------------------------------------------------------------------------- 36 | 37 | if [ -z "$MAVEN_SKIP_RC" ] ; then 38 | 39 | if [ -f /etc/mavenrc ] ; then 40 | . /etc/mavenrc 41 | fi 42 | 43 | if [ -f "$HOME/.mavenrc" ] ; then 44 | . "$HOME/.mavenrc" 45 | fi 46 | 47 | fi 48 | 49 | # OS specific support. $var _must_ be set to either true or false. 50 | cygwin=false; 51 | darwin=false; 52 | mingw=false 53 | case "`uname`" in 54 | CYGWIN*) cygwin=true ;; 55 | MINGW*) mingw=true;; 56 | Darwin*) darwin=true 57 | # Use /usr/libexec/java_home if available, otherwise fall back to /Library/Java/Home 58 | # See https://developer.apple.com/library/mac/qa/qa1170/_index.html 59 | if [ -z "$JAVA_HOME" ]; then 60 | if [ -x "/usr/libexec/java_home" ]; then 61 | export JAVA_HOME="`/usr/libexec/java_home`" 62 | else 63 | export JAVA_HOME="/Library/Java/Home" 64 | fi 65 | fi 66 | ;; 67 | esac 68 | 69 | if [ -z "$JAVA_HOME" ] ; then 70 | if [ -r /etc/gentoo-release ] ; then 71 | JAVA_HOME=`java-config --jre-home` 72 | fi 73 | fi 74 | 75 | if [ -z "$M2_HOME" ] ; then 76 | ## resolve links - $0 may be a link to maven's home 77 | PRG="$0" 78 | 79 | # need this for relative symlinks 80 | while [ -h "$PRG" ] ; do 81 | ls=`ls -ld "$PRG"` 82 | link=`expr "$ls" : '.*-> \(.*\)$'` 83 | if expr "$link" : '/.*' > /dev/null; then 84 | PRG="$link" 85 | else 86 | PRG="`dirname "$PRG"`/$link" 87 | fi 88 | done 89 | 90 | saveddir=`pwd` 91 | 92 | M2_HOME=`dirname "$PRG"`/.. 93 | 94 | # make it fully qualified 95 | M2_HOME=`cd "$M2_HOME" && pwd` 96 | 97 | cd "$saveddir" 98 | # echo Using m2 at $M2_HOME 99 | fi 100 | 101 | # For Cygwin, ensure paths are in UNIX format before anything is touched 102 | if $cygwin ; then 103 | [ -n "$M2_HOME" ] && 104 | M2_HOME=`cygpath --unix "$M2_HOME"` 105 | [ -n "$JAVA_HOME" ] && 106 | JAVA_HOME=`cygpath --unix "$JAVA_HOME"` 107 | [ -n "$CLASSPATH" ] && 108 | CLASSPATH=`cygpath --path --unix "$CLASSPATH"` 109 | fi 110 | 111 | # For Mingw, ensure paths are in UNIX format before anything is touched 112 | if $mingw ; then 113 | [ -n "$M2_HOME" ] && 114 | M2_HOME="`(cd "$M2_HOME"; pwd)`" 115 | [ -n "$JAVA_HOME" ] && 116 | JAVA_HOME="`(cd "$JAVA_HOME"; pwd)`" 117 | fi 118 | 119 | if [ -z "$JAVA_HOME" ]; then 120 | javaExecutable="`which javac`" 121 | if [ -n "$javaExecutable" ] && ! [ "`expr \"$javaExecutable\" : '\([^ ]*\)'`" = "no" ]; then 122 | # readlink(1) is not available as standard on Solaris 10. 123 | readLink=`which readlink` 124 | if [ ! `expr "$readLink" : '\([^ ]*\)'` = "no" ]; then 125 | if $darwin ; then 126 | javaHome="`dirname \"$javaExecutable\"`" 127 | javaExecutable="`cd \"$javaHome\" && pwd -P`/javac" 128 | else 129 | javaExecutable="`readlink -f \"$javaExecutable\"`" 130 | fi 131 | javaHome="`dirname \"$javaExecutable\"`" 132 | javaHome=`expr "$javaHome" : '\(.*\)/bin'` 133 | JAVA_HOME="$javaHome" 134 | export JAVA_HOME 135 | fi 136 | fi 137 | fi 138 | 139 | if [ -z "$JAVACMD" ] ; then 140 | if [ -n "$JAVA_HOME" ] ; then 141 | if [ -x "$JAVA_HOME/jre/sh/java" ] ; then 142 | # IBM's JDK on AIX uses strange locations for the executables 143 | JAVACMD="$JAVA_HOME/jre/sh/java" 144 | else 145 | JAVACMD="$JAVA_HOME/bin/java" 146 | fi 147 | else 148 | JAVACMD="`which java`" 149 | fi 150 | fi 151 | 152 | if [ ! -x "$JAVACMD" ] ; then 153 | echo "Error: JAVA_HOME is not defined correctly." >&2 154 | echo " We cannot execute $JAVACMD" >&2 155 | exit 1 156 | fi 157 | 158 | if [ -z "$JAVA_HOME" ] ; then 159 | echo "Warning: JAVA_HOME environment variable is not set." 160 | fi 161 | 162 | CLASSWORLDS_LAUNCHER=org.codehaus.plexus.classworlds.launcher.Launcher 163 | 164 | # traverses directory structure from process work directory to filesystem root 165 | # first directory with .mvn subdirectory is considered project base directory 166 | find_maven_basedir() { 167 | 168 | if [ -z "$1" ] 169 | then 170 | echo "Path not specified to find_maven_basedir" 171 | return 1 172 | fi 173 | 174 | basedir="$1" 175 | wdir="$1" 176 | while [ "$wdir" != '/' ] ; do 177 | if [ -d "$wdir"/.mvn ] ; then 178 | basedir=$wdir 179 | break 180 | fi 181 | # workaround for JBEAP-8937 (on Solaris 10/Sparc) 182 | if [ -d "${wdir}" ]; then 183 | wdir=`cd "$wdir/.."; pwd` 184 | fi 185 | # end of workaround 186 | done 187 | echo "${basedir}" 188 | } 189 | 190 | # concatenates all lines of a file 191 | concat_lines() { 192 | if [ -f "$1" ]; then 193 | echo "$(tr -s '\n' ' ' < "$1")" 194 | fi 195 | } 196 | 197 | BASE_DIR=`find_maven_basedir "$(pwd)"` 198 | if [ -z "$BASE_DIR" ]; then 199 | exit 1; 200 | fi 201 | 202 | ########################################################################################## 203 | # Extension to allow automatically downloading the maven-wrapper.jar from Maven-central 204 | # This allows using the maven wrapper in projects that prohibit checking in binary data. 205 | ########################################################################################## 206 | if [ -r "$BASE_DIR/.mvn/wrapper/maven-wrapper.jar" ]; then 207 | if [ "$MVNW_VERBOSE" = true ]; then 208 | echo "Found .mvn/wrapper/maven-wrapper.jar" 209 | fi 210 | else 211 | if [ "$MVNW_VERBOSE" = true ]; then 212 | echo "Couldn't find .mvn/wrapper/maven-wrapper.jar, downloading it ..." 213 | fi 214 | if [ -n "$MVNW_REPOURL" ]; then 215 | jarUrl="$MVNW_REPOURL/io/takari/maven-wrapper/0.5.6/maven-wrapper-0.5.6.jar" 216 | else 217 | jarUrl="https://repo.maven.apache.org/maven2/io/takari/maven-wrapper/0.5.6/maven-wrapper-0.5.6.jar" 218 | fi 219 | while IFS="=" read key value; do 220 | case "$key" in (wrapperUrl) jarUrl="$value"; break ;; 221 | esac 222 | done < "$BASE_DIR/.mvn/wrapper/maven-wrapper.properties" 223 | if [ "$MVNW_VERBOSE" = true ]; then 224 | echo "Downloading from: $jarUrl" 225 | fi 226 | wrapperJarPath="$BASE_DIR/.mvn/wrapper/maven-wrapper.jar" 227 | if $cygwin; then 228 | wrapperJarPath=`cygpath --path --windows "$wrapperJarPath"` 229 | fi 230 | 231 | if command -v wget > /dev/null; then 232 | if [ "$MVNW_VERBOSE" = true ]; then 233 | echo "Found wget ... using wget" 234 | fi 235 | if [ -z "$MVNW_USERNAME" ] || [ -z "$MVNW_PASSWORD" ]; then 236 | wget "$jarUrl" -O "$wrapperJarPath" 237 | else 238 | wget --http-user=$MVNW_USERNAME --http-password=$MVNW_PASSWORD "$jarUrl" -O "$wrapperJarPath" 239 | fi 240 | elif command -v curl > /dev/null; then 241 | if [ "$MVNW_VERBOSE" = true ]; then 242 | echo "Found curl ... using curl" 243 | fi 244 | if [ -z "$MVNW_USERNAME" ] || [ -z "$MVNW_PASSWORD" ]; then 245 | curl -o "$wrapperJarPath" "$jarUrl" -f 246 | else 247 | curl --user $MVNW_USERNAME:$MVNW_PASSWORD -o "$wrapperJarPath" "$jarUrl" -f 248 | fi 249 | 250 | else 251 | if [ "$MVNW_VERBOSE" = true ]; then 252 | echo "Falling back to using Java to download" 253 | fi 254 | javaClass="$BASE_DIR/.mvn/wrapper/MavenWrapperDownloader.java" 255 | # For Cygwin, switch paths to Windows format before running javac 256 | if $cygwin; then 257 | javaClass=`cygpath --path --windows "$javaClass"` 258 | fi 259 | if [ -e "$javaClass" ]; then 260 | if [ ! -e "$BASE_DIR/.mvn/wrapper/MavenWrapperDownloader.class" ]; then 261 | if [ "$MVNW_VERBOSE" = true ]; then 262 | echo " - Compiling MavenWrapperDownloader.java ..." 263 | fi 264 | # Compiling the Java class 265 | ("$JAVA_HOME/bin/javac" "$javaClass") 266 | fi 267 | if [ -e "$BASE_DIR/.mvn/wrapper/MavenWrapperDownloader.class" ]; then 268 | # Running the downloader 269 | if [ "$MVNW_VERBOSE" = true ]; then 270 | echo " - Running MavenWrapperDownloader.java ..." 271 | fi 272 | ("$JAVA_HOME/bin/java" -cp .mvn/wrapper MavenWrapperDownloader "$MAVEN_PROJECTBASEDIR") 273 | fi 274 | fi 275 | fi 276 | fi 277 | ########################################################################################## 278 | # End of extension 279 | ########################################################################################## 280 | 281 | export MAVEN_PROJECTBASEDIR=${MAVEN_BASEDIR:-"$BASE_DIR"} 282 | if [ "$MVNW_VERBOSE" = true ]; then 283 | echo $MAVEN_PROJECTBASEDIR 284 | fi 285 | MAVEN_OPTS="$(concat_lines "$MAVEN_PROJECTBASEDIR/.mvn/jvm.config") $MAVEN_OPTS" 286 | 287 | # For Cygwin, switch paths to Windows format before running java 288 | if $cygwin; then 289 | [ -n "$M2_HOME" ] && 290 | M2_HOME=`cygpath --path --windows "$M2_HOME"` 291 | [ -n "$JAVA_HOME" ] && 292 | JAVA_HOME=`cygpath --path --windows "$JAVA_HOME"` 293 | [ -n "$CLASSPATH" ] && 294 | CLASSPATH=`cygpath --path --windows "$CLASSPATH"` 295 | [ -n "$MAVEN_PROJECTBASEDIR" ] && 296 | MAVEN_PROJECTBASEDIR=`cygpath --path --windows "$MAVEN_PROJECTBASEDIR"` 297 | fi 298 | 299 | # Provide a "standardized" way to retrieve the CLI args that will 300 | # work with both Windows and non-Windows executions. 301 | MAVEN_CMD_LINE_ARGS="$MAVEN_CONFIG $@" 302 | export MAVEN_CMD_LINE_ARGS 303 | 304 | WRAPPER_LAUNCHER=org.apache.maven.wrapper.MavenWrapperMain 305 | 306 | exec "$JAVACMD" \ 307 | $MAVEN_OPTS \ 308 | -classpath "$MAVEN_PROJECTBASEDIR/.mvn/wrapper/maven-wrapper.jar" \ 309 | "-Dmaven.home=${M2_HOME}" "-Dmaven.multiModuleProjectDirectory=${MAVEN_PROJECTBASEDIR}" \ 310 | ${WRAPPER_LAUNCHER} $MAVEN_CONFIG "$@" 311 | -------------------------------------------------------------------------------- /quarkus-kafka-producer/mvnw.cmd: -------------------------------------------------------------------------------- 1 | @REM ---------------------------------------------------------------------------- 2 | @REM Licensed to the Apache Software Foundation (ASF) under one 3 | @REM or more contributor license agreements. See the NOTICE file 4 | @REM distributed with this work for additional information 5 | @REM regarding copyright ownership. The ASF licenses this file 6 | @REM to you under the Apache License, Version 2.0 (the 7 | @REM "License"); you may not use this file except in compliance 8 | @REM with the License. You may obtain a copy of the License at 9 | @REM 10 | @REM http://www.apache.org/licenses/LICENSE-2.0 11 | @REM 12 | @REM Unless required by applicable law or agreed to in writing, 13 | @REM software distributed under the License is distributed on an 14 | @REM "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 15 | @REM KIND, either express or implied. See the License for the 16 | @REM specific language governing permissions and limitations 17 | @REM under the License. 18 | @REM ---------------------------------------------------------------------------- 19 | 20 | @REM ---------------------------------------------------------------------------- 21 | @REM Maven Start Up Batch script 22 | @REM 23 | @REM Required ENV vars: 24 | @REM JAVA_HOME - location of a JDK home dir 25 | @REM 26 | @REM Optional ENV vars 27 | @REM M2_HOME - location of maven2's installed home dir 28 | @REM MAVEN_BATCH_ECHO - set to 'on' to enable the echoing of the batch commands 29 | @REM MAVEN_BATCH_PAUSE - set to 'on' to wait for a keystroke before ending 30 | @REM MAVEN_OPTS - parameters passed to the Java VM when running Maven 31 | @REM e.g. to debug Maven itself, use 32 | @REM set MAVEN_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 33 | @REM MAVEN_SKIP_RC - flag to disable loading of mavenrc files 34 | @REM ---------------------------------------------------------------------------- 35 | 36 | @REM Begin all REM lines with '@' in case MAVEN_BATCH_ECHO is 'on' 37 | @echo off 38 | @REM set title of command window 39 | title %0 40 | @REM enable echoing by setting MAVEN_BATCH_ECHO to 'on' 41 | @if "%MAVEN_BATCH_ECHO%" == "on" echo %MAVEN_BATCH_ECHO% 42 | 43 | @REM set %HOME% to equivalent of $HOME 44 | if "%HOME%" == "" (set "HOME=%HOMEDRIVE%%HOMEPATH%") 45 | 46 | @REM Execute a user defined script before this one 47 | if not "%MAVEN_SKIP_RC%" == "" goto skipRcPre 48 | @REM check for pre script, once with legacy .bat ending and once with .cmd ending 49 | if exist "%HOME%\mavenrc_pre.bat" call "%HOME%\mavenrc_pre.bat" 50 | if exist "%HOME%\mavenrc_pre.cmd" call "%HOME%\mavenrc_pre.cmd" 51 | :skipRcPre 52 | 53 | @setlocal 54 | 55 | set ERROR_CODE=0 56 | 57 | @REM To isolate internal variables from possible post scripts, we use another setlocal 58 | @setlocal 59 | 60 | @REM ==== START VALIDATION ==== 61 | if not "%JAVA_HOME%" == "" goto OkJHome 62 | 63 | echo. 64 | echo Error: JAVA_HOME not found in your environment. >&2 65 | echo Please set the JAVA_HOME variable in your environment to match the >&2 66 | echo location of your Java installation. >&2 67 | echo. 68 | goto error 69 | 70 | :OkJHome 71 | if exist "%JAVA_HOME%\bin\java.exe" goto init 72 | 73 | echo. 74 | echo Error: JAVA_HOME is set to an invalid directory. >&2 75 | echo JAVA_HOME = "%JAVA_HOME%" >&2 76 | echo Please set the JAVA_HOME variable in your environment to match the >&2 77 | echo location of your Java installation. >&2 78 | echo. 79 | goto error 80 | 81 | @REM ==== END VALIDATION ==== 82 | 83 | :init 84 | 85 | @REM Find the project base dir, i.e. the directory that contains the folder ".mvn". 86 | @REM Fallback to current working directory if not found. 87 | 88 | set MAVEN_PROJECTBASEDIR=%MAVEN_BASEDIR% 89 | IF NOT "%MAVEN_PROJECTBASEDIR%"=="" goto endDetectBaseDir 90 | 91 | set EXEC_DIR=%CD% 92 | set WDIR=%EXEC_DIR% 93 | :findBaseDir 94 | IF EXIST "%WDIR%"\.mvn goto baseDirFound 95 | cd .. 96 | IF "%WDIR%"=="%CD%" goto baseDirNotFound 97 | set WDIR=%CD% 98 | goto findBaseDir 99 | 100 | :baseDirFound 101 | set MAVEN_PROJECTBASEDIR=%WDIR% 102 | cd "%EXEC_DIR%" 103 | goto endDetectBaseDir 104 | 105 | :baseDirNotFound 106 | set MAVEN_PROJECTBASEDIR=%EXEC_DIR% 107 | cd "%EXEC_DIR%" 108 | 109 | :endDetectBaseDir 110 | 111 | IF NOT EXIST "%MAVEN_PROJECTBASEDIR%\.mvn\jvm.config" goto endReadAdditionalConfig 112 | 113 | @setlocal EnableExtensions EnableDelayedExpansion 114 | for /F "usebackq delims=" %%a in ("%MAVEN_PROJECTBASEDIR%\.mvn\jvm.config") do set JVM_CONFIG_MAVEN_PROPS=!JVM_CONFIG_MAVEN_PROPS! %%a 115 | @endlocal & set JVM_CONFIG_MAVEN_PROPS=%JVM_CONFIG_MAVEN_PROPS% 116 | 117 | :endReadAdditionalConfig 118 | 119 | SET MAVEN_JAVA_EXE="%JAVA_HOME%\bin\java.exe" 120 | set WRAPPER_JAR="%MAVEN_PROJECTBASEDIR%\.mvn\wrapper\maven-wrapper.jar" 121 | set WRAPPER_LAUNCHER=org.apache.maven.wrapper.MavenWrapperMain 122 | 123 | set DOWNLOAD_URL="https://repo.maven.apache.org/maven2/io/takari/maven-wrapper/0.5.6/maven-wrapper-0.5.6.jar" 124 | 125 | FOR /F "tokens=1,2 delims==" %%A IN ("%MAVEN_PROJECTBASEDIR%\.mvn\wrapper\maven-wrapper.properties") DO ( 126 | IF "%%A"=="wrapperUrl" SET DOWNLOAD_URL=%%B 127 | ) 128 | 129 | @REM Extension to allow automatically downloading the maven-wrapper.jar from Maven-central 130 | @REM This allows using the maven wrapper in projects that prohibit checking in binary data. 131 | if exist %WRAPPER_JAR% ( 132 | if "%MVNW_VERBOSE%" == "true" ( 133 | echo Found %WRAPPER_JAR% 134 | ) 135 | ) else ( 136 | if not "%MVNW_REPOURL%" == "" ( 137 | SET DOWNLOAD_URL="%MVNW_REPOURL%/io/takari/maven-wrapper/0.5.6/maven-wrapper-0.5.6.jar" 138 | ) 139 | if "%MVNW_VERBOSE%" == "true" ( 140 | echo Couldn't find %WRAPPER_JAR%, downloading it ... 141 | echo Downloading from: %DOWNLOAD_URL% 142 | ) 143 | 144 | powershell -Command "&{"^ 145 | "$webclient = new-object System.Net.WebClient;"^ 146 | "if (-not ([string]::IsNullOrEmpty('%MVNW_USERNAME%') -and [string]::IsNullOrEmpty('%MVNW_PASSWORD%'))) {"^ 147 | "$webclient.Credentials = new-object System.Net.NetworkCredential('%MVNW_USERNAME%', '%MVNW_PASSWORD%');"^ 148 | "}"^ 149 | "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; $webclient.DownloadFile('%DOWNLOAD_URL%', '%WRAPPER_JAR%')"^ 150 | "}" 151 | if "%MVNW_VERBOSE%" == "true" ( 152 | echo Finished downloading %WRAPPER_JAR% 153 | ) 154 | ) 155 | @REM End of extension 156 | 157 | @REM Provide a "standardized" way to retrieve the CLI args that will 158 | @REM work with both Windows and non-Windows executions. 159 | set MAVEN_CMD_LINE_ARGS=%* 160 | 161 | %MAVEN_JAVA_EXE% %JVM_CONFIG_MAVEN_PROPS% %MAVEN_OPTS% %MAVEN_DEBUG_OPTS% -classpath %WRAPPER_JAR% "-Dmaven.multiModuleProjectDirectory=%MAVEN_PROJECTBASEDIR%" %WRAPPER_LAUNCHER% %MAVEN_CONFIG% %* 162 | if ERRORLEVEL 1 goto error 163 | goto end 164 | 165 | :error 166 | set ERROR_CODE=1 167 | 168 | :end 169 | @endlocal & set ERROR_CODE=%ERROR_CODE% 170 | 171 | if not "%MAVEN_SKIP_RC%" == "" goto skipRcPost 172 | @REM check for post script, once with legacy .bat ending and once with .cmd ending 173 | if exist "%HOME%\mavenrc_post.bat" call "%HOME%\mavenrc_post.bat" 174 | if exist "%HOME%\mavenrc_post.cmd" call "%HOME%\mavenrc_post.cmd" 175 | :skipRcPost 176 | 177 | @REM pause the script if MAVEN_BATCH_PAUSE is set to 'on' 178 | if "%MAVEN_BATCH_PAUSE%" == "on" pause 179 | 180 | if "%MAVEN_TERMINATE_CMD%" == "on" exit %ERROR_CODE% 181 | 182 | exit /B %ERROR_CODE% 183 | -------------------------------------------------------------------------------- /quarkus-kafka-producer/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 4 | 4.0.0 5 | org.acme 6 | quarkus-kafka-producer 7 | 1.0.0-SNAPSHOT 8 | 9 | 3.8.1 10 | true 11 | 1.8 12 | 1.8 13 | UTF-8 14 | UTF-8 15 | 1.5.2.Final 16 | quarkus-bom 17 | io.quarkus 18 | 1.5.2.Final 19 | 2.22.1 20 | 21 | 22 | 23 | 24 | ${quarkus.platform.group-id} 25 | ${quarkus.platform.artifact-id} 26 | ${quarkus.platform.version} 27 | pom 28 | import 29 | 30 | 31 | 32 | 33 | 34 | io.quarkus 35 | quarkus-resteasy 36 | 37 | 38 | io.quarkus 39 | quarkus-junit5 40 | test 41 | 42 | 43 | io.rest-assured 44 | rest-assured 45 | test 46 | 47 | 48 | io.quarkus 49 | quarkus-kafka-client 50 | 51 | 52 | io.quarkus 53 | quarkus-spring-web 54 | 55 | 56 | 57 | 58 | 59 | io.quarkus 60 | quarkus-maven-plugin 61 | ${quarkus-plugin.version} 62 | 63 | 64 | 65 | build 66 | 67 | 68 | 69 | 70 | 71 | maven-compiler-plugin 72 | ${compiler-plugin.version} 73 | 74 | 75 | maven-surefire-plugin 76 | ${surefire-plugin.version} 77 | 78 | 79 | org.jboss.logmanager.LogManager 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | native 88 | 89 | 90 | native 91 | 92 | 93 | 94 | 95 | 96 | maven-failsafe-plugin 97 | ${surefire-plugin.version} 98 | 99 | 100 | 101 | integration-test 102 | verify 103 | 104 | 105 | 106 | ${project.build.directory}/${project.build.finalName}-runner 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | native 116 | 117 | 118 | 119 | 120 | -------------------------------------------------------------------------------- /quarkus-kafka-producer/src/main/docker/Dockerfile.jvm: -------------------------------------------------------------------------------- 1 | #### 2 | # This Dockerfile is used in order to build a container that runs the Quarkus application in JVM mode 3 | # 4 | # Before building the docker image run: 5 | # 6 | # mvn package 7 | # 8 | # Then, build the image with: 9 | # 10 | # docker build -f src/main/docker/Dockerfile.jvm -t quarkus/quarkus-kafka-producer-jvm . 11 | # 12 | # Then run the container using: 13 | # 14 | # docker run -i --rm -p 8080:8080 quarkus/quarkus-kafka-producer-jvm 15 | # 16 | ### 17 | FROM registry.access.redhat.com/ubi8/ubi-minimal:8.1 18 | 19 | ARG JAVA_PACKAGE=java-11-openjdk-headless 20 | ARG RUN_JAVA_VERSION=1.3.5 21 | 22 | ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en' 23 | 24 | # Install java and the run-java script 25 | # Also set up permissions for user `1001` 26 | RUN microdnf install curl ca-certificates ${JAVA_PACKAGE} \ 27 | && microdnf update \ 28 | && microdnf clean all \ 29 | && mkdir /deployments \ 30 | && chown 1001 /deployments \ 31 | && chmod "g+rwX" /deployments \ 32 | && chown 1001:root /deployments \ 33 | && curl https://repo1.maven.org/maven2/io/fabric8/run-java-sh/${RUN_JAVA_VERSION}/run-java-sh-${RUN_JAVA_VERSION}-sh.sh -o /deployments/run-java.sh \ 34 | && chown 1001 /deployments/run-java.sh \ 35 | && chmod 540 /deployments/run-java.sh \ 36 | && echo "securerandom.source=file:/dev/urandom" >> /etc/alternatives/jre/lib/security/java.security 37 | 38 | # Configure the JAVA_OPTIONS, you can add -XshowSettings:vm to also display the heap size. 39 | ENV JAVA_OPTIONS="-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager" 40 | 41 | COPY target/lib/* /deployments/lib/ 42 | COPY target/*-runner.jar /deployments/app.jar 43 | 44 | EXPOSE 8080 45 | USER 1001 46 | 47 | ENTRYPOINT [ "/deployments/run-java.sh" ] -------------------------------------------------------------------------------- /quarkus-kafka-producer/src/main/docker/Dockerfile.native: -------------------------------------------------------------------------------- 1 | #### 2 | # This Dockerfile is used in order to build a container that runs the Quarkus application in native (no JVM) mode 3 | # 4 | # Before building the docker image run: 5 | # 6 | # mvn package -Pnative -Dquarkus.native.container-build=true 7 | # 8 | # Then, build the image with: 9 | # 10 | # docker build -f src/main/docker/Dockerfile.native -t quarkus/quarkus-kafka-producer . 11 | # 12 | # Then run the container using: 13 | # 14 | # docker run -i --rm -p 8080:8080 quarkus/quarkus-kafka-producer 15 | # 16 | ### 17 | FROM registry.access.redhat.com/ubi8/ubi-minimal:8.1 18 | WORKDIR /work/ 19 | COPY target/*-runner /work/application 20 | 21 | # set up permissions for user `1001` 22 | RUN chmod 775 /work /work/application \ 23 | && chown -R 1001 /work \ 24 | && chmod -R "g+rwX" /work \ 25 | && chown -R 1001:root /work 26 | 27 | EXPOSE 8080 28 | USER 1001 29 | 30 | CMD ["./application", "-Dquarkus.http.host=0.0.0.0"] -------------------------------------------------------------------------------- /quarkus-kafka-producer/src/main/java/org/acme/ATDData.java: -------------------------------------------------------------------------------- 1 | package org.acme; 2 | 3 | public class ATDData { 4 | String key; 5 | String value; 6 | 7 | public ATDData(String key, String value) { 8 | this.key = key; 9 | this.value = value; 10 | } 11 | 12 | String getKey() { 13 | return key; 14 | } 15 | 16 | String getValue() { 17 | return value; 18 | } 19 | } -------------------------------------------------------------------------------- /quarkus-kafka-producer/src/main/java/org/acme/DataProducer.java: -------------------------------------------------------------------------------- 1 | package org.acme; 2 | 3 | import java.util.HashMap; 4 | import java.util.Map; 5 | import java.util.Properties; 6 | 7 | import org.springframework.web.bind.annotation.RestController; 8 | import org.springframework.web.bind.annotation.RequestMapping; 9 | import org.apache.kafka.clients.producer.KafkaProducer; 10 | import org.apache.kafka.clients.producer.Producer; 11 | import org.apache.kafka.clients.producer.ProducerRecord; 12 | 13 | @RestController 14 | public class DataProducer { 15 | 16 | private final String LEFT_STREAM_TOPIC = "left-stream-topic"; 17 | private final String RIGHT_STREAM_TOPIC = "right-stream-topic"; 18 | 19 | private static final Map ATD; 20 | static { 21 | ATD = new HashMap<>(); 22 | ATD.put(1, new ATDData("A", "ATD Txn - A")); 23 | ATD.put(2, new ATDData("B", "ATD Txn - B")); 24 | ATD.put(3, new ATDData("C", "ATD Txn - C")); 25 | ATD.put(4, new ATDData("D", "ATD Txn - D")); 26 | ATD.put(5, new ATDData("E", "ATD Txn - E")); 27 | ATD.put(6, new ATDData("F", "ATD Txn - F")); 28 | ATD.put(7, new ATDData("G", "ATD Txn - G")); 29 | ATD.put(8, new ATDData("H", "ATD Txn - H")); 30 | } 31 | 32 | 33 | private static final Map RN; 34 | static { 35 | RN = new HashMap(); 36 | RN.put(1, new RNData("A", "RN Txn - A")); 37 | RN.put(2, new RNData("B", "RN Txn - B")); 38 | RN.put(3, new RNData("C", "RN Txn - C")); 39 | RN.put(4, new RNData("D", "RN Txn - D")); 40 | RN.put(5, new RNData("E", "RN Txn - E")); 41 | RN.put(6, new RNData("F", "RN Txn - F")); 42 | RN.put(7, new RNData("G", "RN Txn - G")); 43 | RN.put(8, new RNData("H", "RN Txn - H")); 44 | RN.put(9, new RNData("I", "RN Txn - I")); 45 | RN.put(10, new RNData("J", "RN Txn - J")); 46 | } 47 | 48 | private Properties props; 49 | Producer producer; 50 | 51 | private void setup() { 52 | props = new Properties(); 53 | props.put("bootstrap.servers", "localhost:9092"); 54 | props.put("acks", "all"); 55 | props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); 56 | props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); 57 | 58 | producer = new KafkaProducer<>(props); 59 | } 60 | 61 | @RequestMapping("/sendmanyrecords") 62 | public void sendManyRecords() { 63 | setup(); 64 | try { 65 | for (int i = 0; i < 10000; i++) { 66 | producer.send(new ProducerRecord(LEFT_STREAM_TOPIC, 67 | "A".concat(String.valueOf(i)), 68 | "ATD - txn ".concat(String.valueOf(i)) )); 69 | 70 | try { 71 | Thread.sleep(3); 72 | } catch (InterruptedException e) {} 73 | 74 | producer.send(new ProducerRecord(RIGHT_STREAM_TOPIC, 75 | "A".concat(String.valueOf(i)), 76 | "RN - txn ".concat(String.valueOf(i)))); 77 | } 78 | } finally { 79 | producer.close(); 80 | } 81 | } 82 | 83 | @RequestMapping("/sendfewrecords") 84 | public void sendFewRecords() { 85 | setup(); 86 | int max = 3; 87 | int min = 1; 88 | 89 | try { 90 | for (int i = 0; i < 10; i++) { 91 | if (ATD.containsKey(i + 1)) { 92 | ATDData data = ATD.get(i+1); 93 | producer.send(new ProducerRecord(LEFT_STREAM_TOPIC, 94 | data.getKey(), 95 | data.getValue())); 96 | } 97 | if (RN.containsKey(i + 1)) { 98 | int sleep_time = (int)(Math.random()*((max-min)+1))+min; 99 | try { 100 | Thread.sleep(sleep_time*1000); 101 | } catch (InterruptedException e) {} 102 | 103 | RNData rnData = RN.get(i+1); 104 | producer.send(new ProducerRecord(RIGHT_STREAM_TOPIC, 105 | rnData.getKey(), 106 | rnData.getValue())); 107 | } 108 | } 109 | } finally { 110 | producer.close(); 111 | } 112 | } 113 | 114 | @RequestMapping("/sendoneleftrecord") 115 | public void sendOneLeftRecord() { 116 | setup(); 117 | try { 118 | for (int i = 0; i < 1; i++) { 119 | if (ATD.containsKey(i + 1)) { 120 | ATDData data = ATD.get(i+1); 121 | producer.send(new ProducerRecord(LEFT_STREAM_TOPIC, 122 | data.getKey(), 123 | data.getValue())); 124 | } 125 | } 126 | } finally { 127 | producer.close(); 128 | } 129 | 130 | } 131 | } -------------------------------------------------------------------------------- /quarkus-kafka-producer/src/main/java/org/acme/RNData.java: -------------------------------------------------------------------------------- 1 | package org.acme; 2 | 3 | public class RNData { 4 | String key; 5 | String value; 6 | 7 | public RNData(String key, String value) { 8 | this.key = key; 9 | this.value = value; 10 | } 11 | 12 | String getKey() { 13 | return key; 14 | } 15 | 16 | String getValue() { 17 | return value; 18 | } 19 | } -------------------------------------------------------------------------------- /quarkus-kafka-producer/src/main/java/org/acme/dbs-kafka.code-workspace: -------------------------------------------------------------------------------- 1 | { 2 | "folders": [ 3 | { 4 | "path": "../../../../.." 5 | }, 6 | { 7 | "path": "../../../../../../quarkus-kafka-streaming" 8 | } 9 | ], 10 | "settings": { 11 | "java.configuration.updateBuildConfiguration": "automatic" 12 | } 13 | } -------------------------------------------------------------------------------- /quarkus-kafka-producer/src/main/resources/META-INF/resources/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | quarkus-kafka-producer - 1.0.0-SNAPSHOT 6 | 99 | 100 | 101 | 102 | 105 | 106 |
107 |
108 |

Congratulations, you have created a new Quarkus application.

109 | 110 |

Why do you see this?

111 | 112 |

This page is served by Quarkus. The source is in 113 | src/main/resources/META-INF/resources/index.html.

114 | 115 |

What can I do from here?

116 | 117 |

If not already done, run the application in dev mode using: mvn compile quarkus:dev. 118 |

119 |
    120 |
  • Add REST resources, Servlets, functions and other services in src/main/java.
  • 121 |
  • Your static assets are located in src/main/resources/META-INF/resources.
  • 122 |
  • Configure your application in src/main/resources/application.properties. 123 |
  • 124 |
125 | 126 |

How do I get rid of this page?

127 |

Just delete the src/main/resources/META-INF/resources/index.html file.

128 |
129 |
130 |
131 |

Application

132 |
    133 |
  • GroupId: org.acme
  • 134 |
  • ArtifactId: quarkus-kafka-producer
  • 135 |
  • Version: 1.0.0-SNAPSHOT
  • 136 |
  • Quarkus Version: 1.3.2.Final-redhat-00001
  • 137 |
138 |
139 |
140 |

Next steps

141 | 146 |
147 |
148 |
149 | 150 | 151 | 152 | -------------------------------------------------------------------------------- /quarkus-kafka-producer/src/main/resources/application.properties: -------------------------------------------------------------------------------- 1 | # Configuration file 2 | # key = value 3 | quarkus.http.port=8082 -------------------------------------------------------------------------------- /quarkus-kafka-streaming/.dockerignore: -------------------------------------------------------------------------------- 1 | * 2 | !target/*-runner 3 | !target/*-runner.jar 4 | !target/lib/* -------------------------------------------------------------------------------- /quarkus-kafka-streaming/.gitignore: -------------------------------------------------------------------------------- 1 | # Eclipse 2 | .project 3 | .classpath 4 | .settings/ 5 | bin/ 6 | 7 | # IntelliJ 8 | .idea 9 | *.ipr 10 | *.iml 11 | *.iws 12 | 13 | # NetBeans 14 | nb-configuration.xml 15 | 16 | # Visual Studio Code 17 | .vscode 18 | 19 | # OSX 20 | .DS_Store 21 | 22 | # Vim 23 | *.swp 24 | *.swo 25 | 26 | # patch 27 | *.orig 28 | *.rej 29 | 30 | # Maven 31 | target/ 32 | pom.xml.tag 33 | pom.xml.releaseBackup 34 | pom.xml.versionsBackup 35 | release.properties -------------------------------------------------------------------------------- /quarkus-kafka-streaming/.mvn/wrapper/MavenWrapperDownloader.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2007-present the original author or authors. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"); 5 | * you may not use this file except in compliance with the License. 6 | * You may obtain a copy of the License at 7 | * 8 | * http://www.apache.org/licenses/LICENSE-2.0 9 | * 10 | * Unless required by applicable law or agreed to in writing, software 11 | * distributed under the License is distributed on an "AS IS" BASIS, 12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | * See the License for the specific language governing permissions and 14 | * limitations under the License. 15 | */ 16 | import java.net.*; 17 | import java.io.*; 18 | import java.nio.channels.*; 19 | import java.util.Properties; 20 | 21 | public class MavenWrapperDownloader { 22 | 23 | private static final String WRAPPER_VERSION = "0.5.6"; 24 | /** 25 | * Default URL to download the maven-wrapper.jar from, if no 'downloadUrl' is provided. 26 | */ 27 | private static final String DEFAULT_DOWNLOAD_URL = "https://repo.maven.apache.org/maven2/io/takari/maven-wrapper/" 28 | + WRAPPER_VERSION + "/maven-wrapper-" + WRAPPER_VERSION + ".jar"; 29 | 30 | /** 31 | * Path to the maven-wrapper.properties file, which might contain a downloadUrl property to 32 | * use instead of the default one. 33 | */ 34 | private static final String MAVEN_WRAPPER_PROPERTIES_PATH = 35 | ".mvn/wrapper/maven-wrapper.properties"; 36 | 37 | /** 38 | * Path where the maven-wrapper.jar will be saved to. 39 | */ 40 | private static final String MAVEN_WRAPPER_JAR_PATH = 41 | ".mvn/wrapper/maven-wrapper.jar"; 42 | 43 | /** 44 | * Name of the property which should be used to override the default download url for the wrapper. 45 | */ 46 | private static final String PROPERTY_NAME_WRAPPER_URL = "wrapperUrl"; 47 | 48 | public static void main(String args[]) { 49 | System.out.println("- Downloader started"); 50 | File baseDirectory = new File(args[0]); 51 | System.out.println("- Using base directory: " + baseDirectory.getAbsolutePath()); 52 | 53 | // If the maven-wrapper.properties exists, read it and check if it contains a custom 54 | // wrapperUrl parameter. 55 | File mavenWrapperPropertyFile = new File(baseDirectory, MAVEN_WRAPPER_PROPERTIES_PATH); 56 | String url = DEFAULT_DOWNLOAD_URL; 57 | if(mavenWrapperPropertyFile.exists()) { 58 | FileInputStream mavenWrapperPropertyFileInputStream = null; 59 | try { 60 | mavenWrapperPropertyFileInputStream = new FileInputStream(mavenWrapperPropertyFile); 61 | Properties mavenWrapperProperties = new Properties(); 62 | mavenWrapperProperties.load(mavenWrapperPropertyFileInputStream); 63 | url = mavenWrapperProperties.getProperty(PROPERTY_NAME_WRAPPER_URL, url); 64 | } catch (IOException e) { 65 | System.out.println("- ERROR loading '" + MAVEN_WRAPPER_PROPERTIES_PATH + "'"); 66 | } finally { 67 | try { 68 | if(mavenWrapperPropertyFileInputStream != null) { 69 | mavenWrapperPropertyFileInputStream.close(); 70 | } 71 | } catch (IOException e) { 72 | // Ignore ... 73 | } 74 | } 75 | } 76 | System.out.println("- Downloading from: " + url); 77 | 78 | File outputFile = new File(baseDirectory.getAbsolutePath(), MAVEN_WRAPPER_JAR_PATH); 79 | if(!outputFile.getParentFile().exists()) { 80 | if(!outputFile.getParentFile().mkdirs()) { 81 | System.out.println( 82 | "- ERROR creating output directory '" + outputFile.getParentFile().getAbsolutePath() + "'"); 83 | } 84 | } 85 | System.out.println("- Downloading to: " + outputFile.getAbsolutePath()); 86 | try { 87 | downloadFileFromURL(url, outputFile); 88 | System.out.println("Done"); 89 | System.exit(0); 90 | } catch (Throwable e) { 91 | System.out.println("- Error downloading"); 92 | e.printStackTrace(); 93 | System.exit(1); 94 | } 95 | } 96 | 97 | private static void downloadFileFromURL(String urlString, File destination) throws Exception { 98 | if (System.getenv("MVNW_USERNAME") != null && System.getenv("MVNW_PASSWORD") != null) { 99 | String username = System.getenv("MVNW_USERNAME"); 100 | char[] password = System.getenv("MVNW_PASSWORD").toCharArray(); 101 | Authenticator.setDefault(new Authenticator() { 102 | @Override 103 | protected PasswordAuthentication getPasswordAuthentication() { 104 | return new PasswordAuthentication(username, password); 105 | } 106 | }); 107 | } 108 | URL website = new URL(urlString); 109 | ReadableByteChannel rbc; 110 | rbc = Channels.newChannel(website.openStream()); 111 | FileOutputStream fos = new FileOutputStream(destination); 112 | fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE); 113 | fos.close(); 114 | rbc.close(); 115 | } 116 | 117 | } 118 | -------------------------------------------------------------------------------- /quarkus-kafka-streaming/.mvn/wrapper/maven-wrapper.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kgshukla/data-streaming-kafka-quarkus/65612de9813264d203ab9dcdf68bd461871e01db/quarkus-kafka-streaming/.mvn/wrapper/maven-wrapper.jar -------------------------------------------------------------------------------- /quarkus-kafka-streaming/.mvn/wrapper/maven-wrapper.properties: -------------------------------------------------------------------------------- 1 | distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.6.3/apache-maven-3.6.3-bin.zip 2 | wrapperUrl=https://repo.maven.apache.org/maven2/io/takari/maven-wrapper/0.5.6/maven-wrapper-0.5.6.jar 3 | -------------------------------------------------------------------------------- /quarkus-kafka-streaming/README.md: -------------------------------------------------------------------------------- 1 | # quarkus-kafka-streaming project 2 | 3 | This project uses Quarkus, the Supersonic Subatomic Java Framework. 4 | 5 | If you want to learn more about Quarkus, please visit its website: https://quarkus.io/ . 6 | 7 | ## Running the application in dev mode 8 | 9 | You can run your application in dev mode that enables live coding using: 10 | ``` 11 | ./mvnw quarkus:dev 12 | ``` 13 | 14 | ## Packaging and running the application 15 | 16 | The application can be packaged using `./mvnw package`. 17 | It produces the `quarkus-kafka-streaming-1.0.0-SNAPSHOT-runner.jar` file in the `/target` directory. 18 | Be aware that it’s not an _über-jar_ as the dependencies are copied into the `target/lib` directory. 19 | 20 | The application is now runnable using `java -jar target/quarkus-kafka-streaming-1.0.0-SNAPSHOT-runner.jar`. 21 | 22 | ## Creating a native executable 23 | 24 | You can create a native executable using: `./mvnw package -Pnative`. 25 | 26 | Or, if you don't have GraalVM installed, you can run the native executable build in a container using: `./mvnw package -Pnative -Dquarkus.native.container-build=true`. 27 | 28 | You can then execute your native executable with: `./target/quarkus-kafka-streaming-1.0.0-SNAPSHOT-runner` 29 | 30 | If you want to learn more about building native executables, please consult https://quarkus.io/guides/building-native-image. -------------------------------------------------------------------------------- /quarkus-kafka-streaming/mvnw: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # ---------------------------------------------------------------------------- 3 | # Licensed to the Apache Software Foundation (ASF) under one 4 | # or more contributor license agreements. See the NOTICE file 5 | # distributed with this work for additional information 6 | # regarding copyright ownership. The ASF licenses this file 7 | # to you under the Apache License, Version 2.0 (the 8 | # "License"); you may not use this file except in compliance 9 | # with the License. You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, 14 | # software distributed under the License is distributed on an 15 | # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 16 | # KIND, either express or implied. See the License for the 17 | # specific language governing permissions and limitations 18 | # under the License. 19 | # ---------------------------------------------------------------------------- 20 | 21 | # ---------------------------------------------------------------------------- 22 | # Maven Start Up Batch script 23 | # 24 | # Required ENV vars: 25 | # ------------------ 26 | # JAVA_HOME - location of a JDK home dir 27 | # 28 | # Optional ENV vars 29 | # ----------------- 30 | # M2_HOME - location of maven2's installed home dir 31 | # MAVEN_OPTS - parameters passed to the Java VM when running Maven 32 | # e.g. to debug Maven itself, use 33 | # set MAVEN_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 34 | # MAVEN_SKIP_RC - flag to disable loading of mavenrc files 35 | # ---------------------------------------------------------------------------- 36 | 37 | if [ -z "$MAVEN_SKIP_RC" ] ; then 38 | 39 | if [ -f /etc/mavenrc ] ; then 40 | . /etc/mavenrc 41 | fi 42 | 43 | if [ -f "$HOME/.mavenrc" ] ; then 44 | . "$HOME/.mavenrc" 45 | fi 46 | 47 | fi 48 | 49 | # OS specific support. $var _must_ be set to either true or false. 50 | cygwin=false; 51 | darwin=false; 52 | mingw=false 53 | case "`uname`" in 54 | CYGWIN*) cygwin=true ;; 55 | MINGW*) mingw=true;; 56 | Darwin*) darwin=true 57 | # Use /usr/libexec/java_home if available, otherwise fall back to /Library/Java/Home 58 | # See https://developer.apple.com/library/mac/qa/qa1170/_index.html 59 | if [ -z "$JAVA_HOME" ]; then 60 | if [ -x "/usr/libexec/java_home" ]; then 61 | export JAVA_HOME="`/usr/libexec/java_home`" 62 | else 63 | export JAVA_HOME="/Library/Java/Home" 64 | fi 65 | fi 66 | ;; 67 | esac 68 | 69 | if [ -z "$JAVA_HOME" ] ; then 70 | if [ -r /etc/gentoo-release ] ; then 71 | JAVA_HOME=`java-config --jre-home` 72 | fi 73 | fi 74 | 75 | if [ -z "$M2_HOME" ] ; then 76 | ## resolve links - $0 may be a link to maven's home 77 | PRG="$0" 78 | 79 | # need this for relative symlinks 80 | while [ -h "$PRG" ] ; do 81 | ls=`ls -ld "$PRG"` 82 | link=`expr "$ls" : '.*-> \(.*\)$'` 83 | if expr "$link" : '/.*' > /dev/null; then 84 | PRG="$link" 85 | else 86 | PRG="`dirname "$PRG"`/$link" 87 | fi 88 | done 89 | 90 | saveddir=`pwd` 91 | 92 | M2_HOME=`dirname "$PRG"`/.. 93 | 94 | # make it fully qualified 95 | M2_HOME=`cd "$M2_HOME" && pwd` 96 | 97 | cd "$saveddir" 98 | # echo Using m2 at $M2_HOME 99 | fi 100 | 101 | # For Cygwin, ensure paths are in UNIX format before anything is touched 102 | if $cygwin ; then 103 | [ -n "$M2_HOME" ] && 104 | M2_HOME=`cygpath --unix "$M2_HOME"` 105 | [ -n "$JAVA_HOME" ] && 106 | JAVA_HOME=`cygpath --unix "$JAVA_HOME"` 107 | [ -n "$CLASSPATH" ] && 108 | CLASSPATH=`cygpath --path --unix "$CLASSPATH"` 109 | fi 110 | 111 | # For Mingw, ensure paths are in UNIX format before anything is touched 112 | if $mingw ; then 113 | [ -n "$M2_HOME" ] && 114 | M2_HOME="`(cd "$M2_HOME"; pwd)`" 115 | [ -n "$JAVA_HOME" ] && 116 | JAVA_HOME="`(cd "$JAVA_HOME"; pwd)`" 117 | fi 118 | 119 | if [ -z "$JAVA_HOME" ]; then 120 | javaExecutable="`which javac`" 121 | if [ -n "$javaExecutable" ] && ! [ "`expr \"$javaExecutable\" : '\([^ ]*\)'`" = "no" ]; then 122 | # readlink(1) is not available as standard on Solaris 10. 123 | readLink=`which readlink` 124 | if [ ! `expr "$readLink" : '\([^ ]*\)'` = "no" ]; then 125 | if $darwin ; then 126 | javaHome="`dirname \"$javaExecutable\"`" 127 | javaExecutable="`cd \"$javaHome\" && pwd -P`/javac" 128 | else 129 | javaExecutable="`readlink -f \"$javaExecutable\"`" 130 | fi 131 | javaHome="`dirname \"$javaExecutable\"`" 132 | javaHome=`expr "$javaHome" : '\(.*\)/bin'` 133 | JAVA_HOME="$javaHome" 134 | export JAVA_HOME 135 | fi 136 | fi 137 | fi 138 | 139 | if [ -z "$JAVACMD" ] ; then 140 | if [ -n "$JAVA_HOME" ] ; then 141 | if [ -x "$JAVA_HOME/jre/sh/java" ] ; then 142 | # IBM's JDK on AIX uses strange locations for the executables 143 | JAVACMD="$JAVA_HOME/jre/sh/java" 144 | else 145 | JAVACMD="$JAVA_HOME/bin/java" 146 | fi 147 | else 148 | JAVACMD="`which java`" 149 | fi 150 | fi 151 | 152 | if [ ! -x "$JAVACMD" ] ; then 153 | echo "Error: JAVA_HOME is not defined correctly." >&2 154 | echo " We cannot execute $JAVACMD" >&2 155 | exit 1 156 | fi 157 | 158 | if [ -z "$JAVA_HOME" ] ; then 159 | echo "Warning: JAVA_HOME environment variable is not set." 160 | fi 161 | 162 | CLASSWORLDS_LAUNCHER=org.codehaus.plexus.classworlds.launcher.Launcher 163 | 164 | # traverses directory structure from process work directory to filesystem root 165 | # first directory with .mvn subdirectory is considered project base directory 166 | find_maven_basedir() { 167 | 168 | if [ -z "$1" ] 169 | then 170 | echo "Path not specified to find_maven_basedir" 171 | return 1 172 | fi 173 | 174 | basedir="$1" 175 | wdir="$1" 176 | while [ "$wdir" != '/' ] ; do 177 | if [ -d "$wdir"/.mvn ] ; then 178 | basedir=$wdir 179 | break 180 | fi 181 | # workaround for JBEAP-8937 (on Solaris 10/Sparc) 182 | if [ -d "${wdir}" ]; then 183 | wdir=`cd "$wdir/.."; pwd` 184 | fi 185 | # end of workaround 186 | done 187 | echo "${basedir}" 188 | } 189 | 190 | # concatenates all lines of a file 191 | concat_lines() { 192 | if [ -f "$1" ]; then 193 | echo "$(tr -s '\n' ' ' < "$1")" 194 | fi 195 | } 196 | 197 | BASE_DIR=`find_maven_basedir "$(pwd)"` 198 | if [ -z "$BASE_DIR" ]; then 199 | exit 1; 200 | fi 201 | 202 | ########################################################################################## 203 | # Extension to allow automatically downloading the maven-wrapper.jar from Maven-central 204 | # This allows using the maven wrapper in projects that prohibit checking in binary data. 205 | ########################################################################################## 206 | if [ -r "$BASE_DIR/.mvn/wrapper/maven-wrapper.jar" ]; then 207 | if [ "$MVNW_VERBOSE" = true ]; then 208 | echo "Found .mvn/wrapper/maven-wrapper.jar" 209 | fi 210 | else 211 | if [ "$MVNW_VERBOSE" = true ]; then 212 | echo "Couldn't find .mvn/wrapper/maven-wrapper.jar, downloading it ..." 213 | fi 214 | if [ -n "$MVNW_REPOURL" ]; then 215 | jarUrl="$MVNW_REPOURL/io/takari/maven-wrapper/0.5.6/maven-wrapper-0.5.6.jar" 216 | else 217 | jarUrl="https://repo.maven.apache.org/maven2/io/takari/maven-wrapper/0.5.6/maven-wrapper-0.5.6.jar" 218 | fi 219 | while IFS="=" read key value; do 220 | case "$key" in (wrapperUrl) jarUrl="$value"; break ;; 221 | esac 222 | done < "$BASE_DIR/.mvn/wrapper/maven-wrapper.properties" 223 | if [ "$MVNW_VERBOSE" = true ]; then 224 | echo "Downloading from: $jarUrl" 225 | fi 226 | wrapperJarPath="$BASE_DIR/.mvn/wrapper/maven-wrapper.jar" 227 | if $cygwin; then 228 | wrapperJarPath=`cygpath --path --windows "$wrapperJarPath"` 229 | fi 230 | 231 | if command -v wget > /dev/null; then 232 | if [ "$MVNW_VERBOSE" = true ]; then 233 | echo "Found wget ... using wget" 234 | fi 235 | if [ -z "$MVNW_USERNAME" ] || [ -z "$MVNW_PASSWORD" ]; then 236 | wget "$jarUrl" -O "$wrapperJarPath" 237 | else 238 | wget --http-user=$MVNW_USERNAME --http-password=$MVNW_PASSWORD "$jarUrl" -O "$wrapperJarPath" 239 | fi 240 | elif command -v curl > /dev/null; then 241 | if [ "$MVNW_VERBOSE" = true ]; then 242 | echo "Found curl ... using curl" 243 | fi 244 | if [ -z "$MVNW_USERNAME" ] || [ -z "$MVNW_PASSWORD" ]; then 245 | curl -o "$wrapperJarPath" "$jarUrl" -f 246 | else 247 | curl --user $MVNW_USERNAME:$MVNW_PASSWORD -o "$wrapperJarPath" "$jarUrl" -f 248 | fi 249 | 250 | else 251 | if [ "$MVNW_VERBOSE" = true ]; then 252 | echo "Falling back to using Java to download" 253 | fi 254 | javaClass="$BASE_DIR/.mvn/wrapper/MavenWrapperDownloader.java" 255 | # For Cygwin, switch paths to Windows format before running javac 256 | if $cygwin; then 257 | javaClass=`cygpath --path --windows "$javaClass"` 258 | fi 259 | if [ -e "$javaClass" ]; then 260 | if [ ! -e "$BASE_DIR/.mvn/wrapper/MavenWrapperDownloader.class" ]; then 261 | if [ "$MVNW_VERBOSE" = true ]; then 262 | echo " - Compiling MavenWrapperDownloader.java ..." 263 | fi 264 | # Compiling the Java class 265 | ("$JAVA_HOME/bin/javac" "$javaClass") 266 | fi 267 | if [ -e "$BASE_DIR/.mvn/wrapper/MavenWrapperDownloader.class" ]; then 268 | # Running the downloader 269 | if [ "$MVNW_VERBOSE" = true ]; then 270 | echo " - Running MavenWrapperDownloader.java ..." 271 | fi 272 | ("$JAVA_HOME/bin/java" -cp .mvn/wrapper MavenWrapperDownloader "$MAVEN_PROJECTBASEDIR") 273 | fi 274 | fi 275 | fi 276 | fi 277 | ########################################################################################## 278 | # End of extension 279 | ########################################################################################## 280 | 281 | export MAVEN_PROJECTBASEDIR=${MAVEN_BASEDIR:-"$BASE_DIR"} 282 | if [ "$MVNW_VERBOSE" = true ]; then 283 | echo $MAVEN_PROJECTBASEDIR 284 | fi 285 | MAVEN_OPTS="$(concat_lines "$MAVEN_PROJECTBASEDIR/.mvn/jvm.config") $MAVEN_OPTS" 286 | 287 | # For Cygwin, switch paths to Windows format before running java 288 | if $cygwin; then 289 | [ -n "$M2_HOME" ] && 290 | M2_HOME=`cygpath --path --windows "$M2_HOME"` 291 | [ -n "$JAVA_HOME" ] && 292 | JAVA_HOME=`cygpath --path --windows "$JAVA_HOME"` 293 | [ -n "$CLASSPATH" ] && 294 | CLASSPATH=`cygpath --path --windows "$CLASSPATH"` 295 | [ -n "$MAVEN_PROJECTBASEDIR" ] && 296 | MAVEN_PROJECTBASEDIR=`cygpath --path --windows "$MAVEN_PROJECTBASEDIR"` 297 | fi 298 | 299 | # Provide a "standardized" way to retrieve the CLI args that will 300 | # work with both Windows and non-Windows executions. 301 | MAVEN_CMD_LINE_ARGS="$MAVEN_CONFIG $@" 302 | export MAVEN_CMD_LINE_ARGS 303 | 304 | WRAPPER_LAUNCHER=org.apache.maven.wrapper.MavenWrapperMain 305 | 306 | exec "$JAVACMD" \ 307 | $MAVEN_OPTS \ 308 | -classpath "$MAVEN_PROJECTBASEDIR/.mvn/wrapper/maven-wrapper.jar" \ 309 | "-Dmaven.home=${M2_HOME}" "-Dmaven.multiModuleProjectDirectory=${MAVEN_PROJECTBASEDIR}" \ 310 | ${WRAPPER_LAUNCHER} $MAVEN_CONFIG "$@" 311 | -------------------------------------------------------------------------------- /quarkus-kafka-streaming/mvnw.cmd: -------------------------------------------------------------------------------- 1 | @REM ---------------------------------------------------------------------------- 2 | @REM Licensed to the Apache Software Foundation (ASF) under one 3 | @REM or more contributor license agreements. See the NOTICE file 4 | @REM distributed with this work for additional information 5 | @REM regarding copyright ownership. The ASF licenses this file 6 | @REM to you under the Apache License, Version 2.0 (the 7 | @REM "License"); you may not use this file except in compliance 8 | @REM with the License. You may obtain a copy of the License at 9 | @REM 10 | @REM http://www.apache.org/licenses/LICENSE-2.0 11 | @REM 12 | @REM Unless required by applicable law or agreed to in writing, 13 | @REM software distributed under the License is distributed on an 14 | @REM "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 15 | @REM KIND, either express or implied. See the License for the 16 | @REM specific language governing permissions and limitations 17 | @REM under the License. 18 | @REM ---------------------------------------------------------------------------- 19 | 20 | @REM ---------------------------------------------------------------------------- 21 | @REM Maven Start Up Batch script 22 | @REM 23 | @REM Required ENV vars: 24 | @REM JAVA_HOME - location of a JDK home dir 25 | @REM 26 | @REM Optional ENV vars 27 | @REM M2_HOME - location of maven2's installed home dir 28 | @REM MAVEN_BATCH_ECHO - set to 'on' to enable the echoing of the batch commands 29 | @REM MAVEN_BATCH_PAUSE - set to 'on' to wait for a keystroke before ending 30 | @REM MAVEN_OPTS - parameters passed to the Java VM when running Maven 31 | @REM e.g. to debug Maven itself, use 32 | @REM set MAVEN_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 33 | @REM MAVEN_SKIP_RC - flag to disable loading of mavenrc files 34 | @REM ---------------------------------------------------------------------------- 35 | 36 | @REM Begin all REM lines with '@' in case MAVEN_BATCH_ECHO is 'on' 37 | @echo off 38 | @REM set title of command window 39 | title %0 40 | @REM enable echoing by setting MAVEN_BATCH_ECHO to 'on' 41 | @if "%MAVEN_BATCH_ECHO%" == "on" echo %MAVEN_BATCH_ECHO% 42 | 43 | @REM set %HOME% to equivalent of $HOME 44 | if "%HOME%" == "" (set "HOME=%HOMEDRIVE%%HOMEPATH%") 45 | 46 | @REM Execute a user defined script before this one 47 | if not "%MAVEN_SKIP_RC%" == "" goto skipRcPre 48 | @REM check for pre script, once with legacy .bat ending and once with .cmd ending 49 | if exist "%HOME%\mavenrc_pre.bat" call "%HOME%\mavenrc_pre.bat" 50 | if exist "%HOME%\mavenrc_pre.cmd" call "%HOME%\mavenrc_pre.cmd" 51 | :skipRcPre 52 | 53 | @setlocal 54 | 55 | set ERROR_CODE=0 56 | 57 | @REM To isolate internal variables from possible post scripts, we use another setlocal 58 | @setlocal 59 | 60 | @REM ==== START VALIDATION ==== 61 | if not "%JAVA_HOME%" == "" goto OkJHome 62 | 63 | echo. 64 | echo Error: JAVA_HOME not found in your environment. >&2 65 | echo Please set the JAVA_HOME variable in your environment to match the >&2 66 | echo location of your Java installation. >&2 67 | echo. 68 | goto error 69 | 70 | :OkJHome 71 | if exist "%JAVA_HOME%\bin\java.exe" goto init 72 | 73 | echo. 74 | echo Error: JAVA_HOME is set to an invalid directory. >&2 75 | echo JAVA_HOME = "%JAVA_HOME%" >&2 76 | echo Please set the JAVA_HOME variable in your environment to match the >&2 77 | echo location of your Java installation. >&2 78 | echo. 79 | goto error 80 | 81 | @REM ==== END VALIDATION ==== 82 | 83 | :init 84 | 85 | @REM Find the project base dir, i.e. the directory that contains the folder ".mvn". 86 | @REM Fallback to current working directory if not found. 87 | 88 | set MAVEN_PROJECTBASEDIR=%MAVEN_BASEDIR% 89 | IF NOT "%MAVEN_PROJECTBASEDIR%"=="" goto endDetectBaseDir 90 | 91 | set EXEC_DIR=%CD% 92 | set WDIR=%EXEC_DIR% 93 | :findBaseDir 94 | IF EXIST "%WDIR%"\.mvn goto baseDirFound 95 | cd .. 96 | IF "%WDIR%"=="%CD%" goto baseDirNotFound 97 | set WDIR=%CD% 98 | goto findBaseDir 99 | 100 | :baseDirFound 101 | set MAVEN_PROJECTBASEDIR=%WDIR% 102 | cd "%EXEC_DIR%" 103 | goto endDetectBaseDir 104 | 105 | :baseDirNotFound 106 | set MAVEN_PROJECTBASEDIR=%EXEC_DIR% 107 | cd "%EXEC_DIR%" 108 | 109 | :endDetectBaseDir 110 | 111 | IF NOT EXIST "%MAVEN_PROJECTBASEDIR%\.mvn\jvm.config" goto endReadAdditionalConfig 112 | 113 | @setlocal EnableExtensions EnableDelayedExpansion 114 | for /F "usebackq delims=" %%a in ("%MAVEN_PROJECTBASEDIR%\.mvn\jvm.config") do set JVM_CONFIG_MAVEN_PROPS=!JVM_CONFIG_MAVEN_PROPS! %%a 115 | @endlocal & set JVM_CONFIG_MAVEN_PROPS=%JVM_CONFIG_MAVEN_PROPS% 116 | 117 | :endReadAdditionalConfig 118 | 119 | SET MAVEN_JAVA_EXE="%JAVA_HOME%\bin\java.exe" 120 | set WRAPPER_JAR="%MAVEN_PROJECTBASEDIR%\.mvn\wrapper\maven-wrapper.jar" 121 | set WRAPPER_LAUNCHER=org.apache.maven.wrapper.MavenWrapperMain 122 | 123 | set DOWNLOAD_URL="https://repo.maven.apache.org/maven2/io/takari/maven-wrapper/0.5.6/maven-wrapper-0.5.6.jar" 124 | 125 | FOR /F "tokens=1,2 delims==" %%A IN ("%MAVEN_PROJECTBASEDIR%\.mvn\wrapper\maven-wrapper.properties") DO ( 126 | IF "%%A"=="wrapperUrl" SET DOWNLOAD_URL=%%B 127 | ) 128 | 129 | @REM Extension to allow automatically downloading the maven-wrapper.jar from Maven-central 130 | @REM This allows using the maven wrapper in projects that prohibit checking in binary data. 131 | if exist %WRAPPER_JAR% ( 132 | if "%MVNW_VERBOSE%" == "true" ( 133 | echo Found %WRAPPER_JAR% 134 | ) 135 | ) else ( 136 | if not "%MVNW_REPOURL%" == "" ( 137 | SET DOWNLOAD_URL="%MVNW_REPOURL%/io/takari/maven-wrapper/0.5.6/maven-wrapper-0.5.6.jar" 138 | ) 139 | if "%MVNW_VERBOSE%" == "true" ( 140 | echo Couldn't find %WRAPPER_JAR%, downloading it ... 141 | echo Downloading from: %DOWNLOAD_URL% 142 | ) 143 | 144 | powershell -Command "&{"^ 145 | "$webclient = new-object System.Net.WebClient;"^ 146 | "if (-not ([string]::IsNullOrEmpty('%MVNW_USERNAME%') -and [string]::IsNullOrEmpty('%MVNW_PASSWORD%'))) {"^ 147 | "$webclient.Credentials = new-object System.Net.NetworkCredential('%MVNW_USERNAME%', '%MVNW_PASSWORD%');"^ 148 | "}"^ 149 | "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; $webclient.DownloadFile('%DOWNLOAD_URL%', '%WRAPPER_JAR%')"^ 150 | "}" 151 | if "%MVNW_VERBOSE%" == "true" ( 152 | echo Finished downloading %WRAPPER_JAR% 153 | ) 154 | ) 155 | @REM End of extension 156 | 157 | @REM Provide a "standardized" way to retrieve the CLI args that will 158 | @REM work with both Windows and non-Windows executions. 159 | set MAVEN_CMD_LINE_ARGS=%* 160 | 161 | %MAVEN_JAVA_EXE% %JVM_CONFIG_MAVEN_PROPS% %MAVEN_OPTS% %MAVEN_DEBUG_OPTS% -classpath %WRAPPER_JAR% "-Dmaven.multiModuleProjectDirectory=%MAVEN_PROJECTBASEDIR%" %WRAPPER_LAUNCHER% %MAVEN_CONFIG% %* 162 | if ERRORLEVEL 1 goto error 163 | goto end 164 | 165 | :error 166 | set ERROR_CODE=1 167 | 168 | :end 169 | @endlocal & set ERROR_CODE=%ERROR_CODE% 170 | 171 | if not "%MAVEN_SKIP_RC%" == "" goto skipRcPost 172 | @REM check for post script, once with legacy .bat ending and once with .cmd ending 173 | if exist "%HOME%\mavenrc_post.bat" call "%HOME%\mavenrc_post.bat" 174 | if exist "%HOME%\mavenrc_post.cmd" call "%HOME%\mavenrc_post.cmd" 175 | :skipRcPost 176 | 177 | @REM pause the script if MAVEN_BATCH_PAUSE is set to 'on' 178 | if "%MAVEN_BATCH_PAUSE%" == "on" pause 179 | 180 | if "%MAVEN_TERMINATE_CMD%" == "on" exit %ERROR_CODE% 181 | 182 | exit /B %ERROR_CODE% 183 | -------------------------------------------------------------------------------- /quarkus-kafka-streaming/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 4 | 4.0.0 5 | org.acme 6 | quarkus-kafka-streaming 7 | 1.0.0-SNAPSHOT 8 | 9 | 3.8.1 10 | true 11 | 1.8 12 | 1.8 13 | UTF-8 14 | UTF-8 15 | 1.5.2.Final 16 | quarkus-bom 17 | io.quarkus 18 | 1.5.2.Final 19 | 2.22.1 20 | 21 | 22 | 23 | 24 | ${quarkus.platform.group-id} 25 | ${quarkus.platform.artifact-id} 26 | ${quarkus.platform.version} 27 | pom 28 | import 29 | 30 | 31 | 32 | 33 | 34 | io.quarkus 35 | quarkus-resteasy 36 | 37 | 38 | io.quarkus 39 | quarkus-junit5 40 | test 41 | 42 | 43 | io.rest-assured 44 | rest-assured 45 | test 46 | 47 | 48 | io.quarkus 49 | quarkus-kafka-client 50 | 51 | 52 | io.quarkus 53 | quarkus-spring-web 54 | 55 | 56 | io.quarkus 57 | quarkus-kafka-streams 58 | 59 | 60 | 61 | 62 | 63 | io.quarkus 64 | quarkus-maven-plugin 65 | ${quarkus-plugin.version} 66 | 67 | 68 | 69 | build 70 | 71 | 72 | 73 | 74 | 75 | maven-compiler-plugin 76 | ${compiler-plugin.version} 77 | 78 | 79 | maven-surefire-plugin 80 | ${surefire-plugin.version} 81 | 82 | 83 | org.jboss.logmanager.LogManager 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | native 92 | 93 | 94 | native 95 | 96 | 97 | 98 | 99 | 100 | maven-failsafe-plugin 101 | ${surefire-plugin.version} 102 | 103 | 104 | 105 | integration-test 106 | verify 107 | 108 | 109 | 110 | ${project.build.directory}/${project.build.finalName}-runner 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | native 120 | 121 | 122 | 123 | 124 | -------------------------------------------------------------------------------- /quarkus-kafka-streaming/src/main/docker/Dockerfile.jvm: -------------------------------------------------------------------------------- 1 | #### 2 | # This Dockerfile is used in order to build a container that runs the Quarkus application in JVM mode 3 | # 4 | # Before building the docker image run: 5 | # 6 | # mvn package 7 | # 8 | # Then, build the image with: 9 | # 10 | # docker build -f src/main/docker/Dockerfile.jvm -t quarkus/quarkus-kafka-streaming-jvm . 11 | # 12 | # Then run the container using: 13 | # 14 | # docker run -i --rm -p 8080:8080 quarkus/quarkus-kafka-streaming-jvm 15 | # 16 | ### 17 | FROM registry.access.redhat.com/ubi8/ubi-minimal:8.1 18 | 19 | ARG JAVA_PACKAGE=java-11-openjdk-headless 20 | ARG RUN_JAVA_VERSION=1.3.5 21 | 22 | ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en' 23 | 24 | # Install java and the run-java script 25 | # Also set up permissions for user `1001` 26 | RUN microdnf install curl ca-certificates ${JAVA_PACKAGE} \ 27 | && microdnf update \ 28 | && microdnf clean all \ 29 | && mkdir /deployments \ 30 | && chown 1001 /deployments \ 31 | && chmod "g+rwX" /deployments \ 32 | && chown 1001:root /deployments \ 33 | && curl https://repo1.maven.org/maven2/io/fabric8/run-java-sh/${RUN_JAVA_VERSION}/run-java-sh-${RUN_JAVA_VERSION}-sh.sh -o /deployments/run-java.sh \ 34 | && chown 1001 /deployments/run-java.sh \ 35 | && chmod 540 /deployments/run-java.sh \ 36 | && echo "securerandom.source=file:/dev/urandom" >> /etc/alternatives/jre/lib/security/java.security 37 | 38 | # Configure the JAVA_OPTIONS, you can add -XshowSettings:vm to also display the heap size. 39 | ENV JAVA_OPTIONS="-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager" 40 | 41 | COPY target/lib/* /deployments/lib/ 42 | COPY target/*-runner.jar /deployments/app.jar 43 | 44 | EXPOSE 8080 45 | USER 1001 46 | 47 | ENTRYPOINT [ "/deployments/run-java.sh" ] -------------------------------------------------------------------------------- /quarkus-kafka-streaming/src/main/docker/Dockerfile.native: -------------------------------------------------------------------------------- 1 | #### 2 | # This Dockerfile is used in order to build a container that runs the Quarkus application in native (no JVM) mode 3 | # 4 | # Before building the docker image run: 5 | # 6 | # mvn package -Pnative -Dquarkus.native.container-build=true 7 | # 8 | # Then, build the image with: 9 | # 10 | # docker build -f src/main/docker/Dockerfile.native -t quarkus/quarkus-kafka-streaming . 11 | # 12 | # Then run the container using: 13 | # 14 | # docker run -i --rm -p 8080:8080 quarkus/quarkus-kafka-streaming 15 | # 16 | ### 17 | FROM registry.access.redhat.com/ubi8/ubi-minimal:8.1 18 | WORKDIR /work/ 19 | COPY target/*-runner /work/application 20 | 21 | # set up permissions for user `1001` 22 | RUN chmod 775 /work /work/application \ 23 | && chown -R 1001 /work \ 24 | && chmod -R "g+rwX" /work \ 25 | && chown -R 1001:root /work 26 | 27 | EXPOSE 8080 28 | USER 1001 29 | 30 | CMD ["./application", "-Dquarkus.http.host=0.0.0.0"] -------------------------------------------------------------------------------- /quarkus-kafka-streaming/src/main/java/org/acme/DataProcessor.java: -------------------------------------------------------------------------------- 1 | package org.acme; 2 | 3 | import java.time.Duration; 4 | 5 | import org.apache.kafka.streams.KeyValue; 6 | import org.apache.kafka.streams.processor.Processor; 7 | import org.apache.kafka.streams.processor.ProcessorContext; 8 | import org.apache.kafka.streams.processor.PunctuationType; 9 | import org.apache.kafka.streams.processor.Punctuator; 10 | import org.apache.kafka.streams.state.KeyValueIterator; 11 | import org.apache.kafka.streams.state.KeyValueStore; 12 | 13 | 14 | public class DataProcessor implements Processor{ 15 | 16 | private ProcessorContext context; 17 | private KeyValueStore kvStore; 18 | private final String STORE_NAME = "missing_data_store"; 19 | 20 | @Override 21 | public void init(ProcessorContext context) { 22 | this.context = context; 23 | kvStore = (KeyValueStore) context.getStateStore(STORE_NAME); 24 | 25 | // schedule a punctuate() method every 10 seconds based on stream-time 26 | this.context.schedule(Duration.ofSeconds(60), PunctuationType.WALL_CLOCK_TIME, 27 | new Punctuator(){ 28 | 29 | @Override 30 | public void punctuate(long timestamp) { 31 | System.out.println("Scheduled punctuator called at "+timestamp); 32 | KeyValueIterator iter = kvStore.all(); 33 | while (iter.hasNext()) { 34 | KeyValue entry = iter.next(); 35 | System.out.println(" Processed key: "+entry.key+" and value: "+entry.value+" and sending to processed-topic topic"); 36 | context.forward(entry.key, entry.value.toString()); 37 | kvStore.put(entry.key, null); 38 | } 39 | iter.close(); 40 | 41 | // commit the current processing progress 42 | context.commit(); 43 | } 44 | } 45 | ); 46 | 47 | } 48 | 49 | @Override 50 | public void process(String key, String value) { 51 | if(value.contains("null")) { 52 | if (kvStore.get(key) != null) { 53 | // this means that the other value arrived first 54 | // you have both the values now and can process the record 55 | String newvalue = value.concat(" ").concat(kvStore.get(key)); 56 | process(key, newvalue); 57 | 58 | // remove the entry from the statestore (if any left or right record came first as an event) 59 | kvStore.delete(key); 60 | 61 | context.forward(key, newvalue); 62 | } else { 63 | // add to state store as either left or right data is missing 64 | System.out.println("Incomplete value: "+value+" detected. Putting into statestore for later processing"); 65 | kvStore.put(key, value); 66 | } 67 | } else { 68 | processRecord(key, value); 69 | 70 | // remove the entry from the statestore (if any left or right record came first as an event) 71 | kvStore.delete(key); 72 | 73 | //forward the processed data to processed-topic topic 74 | context.forward(key, value); 75 | } 76 | context.commit(); 77 | } 78 | 79 | @Override 80 | public void close() { 81 | // TODO Auto-generated method stub 82 | 83 | } 84 | 85 | private void processRecord (String key, String value) { 86 | System.out.println("==== Record Processed ==== key: "+key+" and value: "+value); 87 | } 88 | 89 | } -------------------------------------------------------------------------------- /quarkus-kafka-streaming/src/main/java/org/acme/InteractiveQueries.java: -------------------------------------------------------------------------------- 1 | package org.acme; 2 | 3 | import javax.enterprise.context.ApplicationScoped; 4 | import org.apache.kafka.streams.KafkaStreams; 5 | import org.apache.kafka.streams.KeyValue; 6 | import org.apache.kafka.streams.state.ReadOnlyKeyValueStore; 7 | import org.apache.kafka.streams.state.KeyValueIterator; 8 | import org.apache.kafka.streams.state.QueryableStoreTypes; 9 | import org.apache.kafka.streams.errors.InvalidStateStoreException; 10 | 11 | @ApplicationScoped 12 | public class InteractiveQueries { 13 | 14 | KafkaStreams streams; 15 | 16 | private final String STORE_NAME = "missing_data_store"; 17 | 18 | public void setStream(KafkaStreams ks) { 19 | this.streams = ks; 20 | } 21 | 22 | public String getRecords() { 23 | ReadOnlyKeyValueStore store = getStore(); 24 | String mesg = ""; 25 | for (KeyValueIterator it = store.all(); it.hasNext();) { 26 | KeyValue kv = it.next(); 27 | mesg = mesg.concat("\n").concat("Not Processed Records -- key: "+kv.key +" and value: "+kv.value); 28 | } 29 | return mesg.concat("\n"); 30 | } 31 | 32 | private ReadOnlyKeyValueStore getStore() { 33 | while (true) { 34 | try { 35 | return streams.store(STORE_NAME, QueryableStoreTypes.keyValueStore()); 36 | } catch (InvalidStateStoreException e) { 37 | // ignore, store not ready yet 38 | } 39 | } 40 | } 41 | } -------------------------------------------------------------------------------- /quarkus-kafka-streaming/src/main/java/org/acme/KafkaStreaming.java: -------------------------------------------------------------------------------- 1 | package org.acme; 2 | 3 | import org.springframework.web.bind.annotation.RestController; 4 | import org.springframework.web.bind.annotation.RequestMapping; 5 | 6 | import java.util.Properties; 7 | import java.util.Map; 8 | import java.util.HashMap; 9 | 10 | import javax.inject.Inject; 11 | 12 | import java.time.Duration; 13 | 14 | import org.apache.kafka.common.serialization.Serdes; 15 | import org.apache.kafka.streams.KafkaStreams; 16 | import org.apache.kafka.streams.StreamsBuilder; 17 | import org.apache.kafka.streams.StreamsConfig; 18 | import org.apache.kafka.streams.Topology; 19 | import org.apache.kafka.streams.kstream.*; 20 | import org.apache.kafka.streams.processor.ProcessorSupplier; 21 | import org.apache.kafka.streams.processor.Processor; 22 | import org.apache.kafka.streams.state.StoreBuilder; 23 | import org.apache.kafka.streams.state.Stores; 24 | import org.apache.kafka.streams.state.KeyValueStore; 25 | 26 | @RestController 27 | public class KafkaStreaming { 28 | 29 | private KafkaStreams streamsOuterJoin; 30 | private final String LEFT_STREAM_TOPIC = "left-stream-topic"; 31 | private final String RIGHT_STREAM_TOPIC = "right-stream-topic"; 32 | private final String OUTER_JOIN_STREAM_OUT_TOPIC = "stream-stream-outerjoin"; 33 | private final String PROCESSED_STREAM_OUT_TOPIC = "processed-topic"; 34 | private final String STORE_NAME = "missing_data_store"; 35 | 36 | // private final String INNER_JOIN_STREAM_OUT_TOPIC = "stream-stream-innerjoin"; 37 | // private KafkaStreams streamsInnerJoin; 38 | 39 | private final String KAFKA_APP_ID = "outerjoin"; 40 | private final String KAFKA_SERVER_NAME = "localhost:9092"; 41 | 42 | @Inject 43 | InteractiveQueries interactiveQueries; 44 | 45 | @RequestMapping("/startstream/") 46 | public void startStreamStreamOuterJoin() { 47 | 48 | Properties props = new Properties(); 49 | props.put(StreamsConfig.APPLICATION_ID_CONFIG, KAFKA_APP_ID); 50 | props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, KAFKA_SERVER_NAME); 51 | props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass()); 52 | props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass()); 53 | 54 | final StreamsBuilder builder = new StreamsBuilder(); 55 | 56 | KStream leftSource = builder.stream(LEFT_STREAM_TOPIC); 57 | KStream rightSource = builder.stream(RIGHT_STREAM_TOPIC); 58 | 59 | 60 | 61 | // build the state store that will eventually store all unprocessed items 62 | Map changelogConfig = new HashMap<>(); 63 | // override min.insync.replicas 64 | //changelogConfig.put("min.insyc.replicas", "1"); 65 | 66 | StoreBuilder> stateStore = Stores.keyValueStoreBuilder( 67 | Stores.persistentKeyValueStore(STORE_NAME), 68 | Serdes.String(), 69 | Serdes.String()) 70 | .withLoggingEnabled(changelogConfig); 71 | //.withLoggingDisabled(); 72 | 73 | // do the outer join 74 | // change the value to be a mix of both streams value 75 | // have a moving window of 5 seconds 76 | // output the last value received for a specific key during the window 77 | // push the data to OUTER_JOIN_STREAM_OUT_TOPIC topic 78 | leftSource.outerJoin(rightSource, 79 | (leftValue, rightValue) -> "left=" + leftValue + ", right=" + rightValue, 80 | JoinWindows.of(Duration.ofSeconds(5))) 81 | .groupByKey() 82 | .reduce(((key, lastValue) -> lastValue)) 83 | .toStream() 84 | .to(OUTER_JOIN_STREAM_OUT_TOPIC); 85 | 86 | 87 | // build the streams topology 88 | final Topology topology = builder.build(); 89 | 90 | // add another stream that reads data from OUTER_JOIN_STREAM_OUT_TOPIC topic 91 | topology.addSource("Source", OUTER_JOIN_STREAM_OUT_TOPIC); 92 | 93 | // add a processor to the stream so that each record is processed 94 | topology.addProcessor("StateProcessor", 95 | new ProcessorSupplier() 96 | { public Processor get() { 97 | return new DataProcessor(); 98 | }}, 99 | "Source"); 100 | 101 | topology.addSink("Sink", PROCESSED_STREAM_OUT_TOPIC, "StateProcessor"); 102 | 103 | // add a state store so that the processor can push the records which it didnt process 104 | topology.addStateStore(stateStore, "StateProcessor"); 105 | 106 | streamsOuterJoin = new KafkaStreams(topology, props); 107 | interactiveQueries.setStream(streamsOuterJoin); 108 | streamsOuterJoin.start(); 109 | } 110 | 111 | @RequestMapping("/storedata") 112 | public String getStoreData() { 113 | return interactiveQueries.getRecords(); 114 | } 115 | } -------------------------------------------------------------------------------- /quarkus-kafka-streaming/src/main/resources/META-INF/resources/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | quarkus-kafka-streaming - 1.0.0-SNAPSHOT 6 | 99 | 100 | 101 | 102 | 105 | 106 |
107 |
108 |

Congratulations, you have created a new Quarkus application.

109 | 110 |

Why do you see this?

111 | 112 |

This page is served by Quarkus. The source is in 113 | src/main/resources/META-INF/resources/index.html.

114 | 115 |

What can I do from here?

116 | 117 |

If not already done, run the application in dev mode using: mvn compile quarkus:dev. 118 |

119 |
    120 |
  • Add REST resources, Servlets, functions and other services in src/main/java.
  • 121 |
  • Your static assets are located in src/main/resources/META-INF/resources.
  • 122 |
  • Configure your application in src/main/resources/application.properties. 123 |
  • 124 |
125 | 126 |

How do I get rid of this page?

127 |

Just delete the src/main/resources/META-INF/resources/index.html file.

128 |
129 |
130 |
131 |

Application

132 |
    133 |
  • GroupId: org.acme
  • 134 |
  • ArtifactId: quarkus-kafka-streaming
  • 135 |
  • Version: 1.0.0-SNAPSHOT
  • 136 |
  • Quarkus Version: 1.3.2.Final-redhat-00001
  • 137 |
138 |
139 |
140 |

Next steps

141 | 146 |
147 |
148 |
149 | 150 | 151 | 152 | -------------------------------------------------------------------------------- /quarkus-kafka-streaming/src/main/resources/application.properties: -------------------------------------------------------------------------------- 1 | # Configuration file 2 | # key = value 3 | # quarkus.http.port=8081 4 | # streams options 5 | kafka-streams.cache.max.bytes.buffering=10240 6 | kafka-streams.commit.interval.ms=1000 7 | kafka-streams.metadata.max.age.ms=500 8 | kafka-streams.auto.offset.reset=earliest 9 | kafka-streams.consumer.session.timeout.ms=250 10 | kafka-streams.consumer.heartbeat.interval.ms=200 11 | 12 | 13 | quarkus.kafka-streams.bootstrap-servers=localhost:9092 14 | quarkus.kafka-streams.application-id=outerjoin 15 | quarkus.kafka-streams.application-server=localhost:9090 16 | quarkus.kafka-streams.topics=left-stream-topic,right-stream-topic,stream-stream-outerjoin,processed-topic --------------------------------------------------------------------------------