├── BankAppArch.png ├── CQREventSourcingAnalytics.png ├── CQRSEventSourcingStep1.png ├── CQRSState.png ├── ClassicalBI.png ├── CommandToEvent.png ├── Cqrs.png ├── CqrsIndexes.png ├── Dockerfile ├── EventSourcing.png ├── EventSourcingSnapshotting.png ├── README.md ├── RetervingState.png ├── bank-app-mini-statement-supervisor.json ├── debezium-jdbc └── Dockerfile ├── docker-compose.yaml ├── jdbc-sink.json ├── mvnw ├── mvnw.cmd ├── mysqlsource.json ├── pom.xml └── src ├── main ├── java │ └── com │ │ └── gonnect │ │ └── debezium │ │ └── kafka │ │ └── bank │ │ └── account │ │ ├── CdcApplication.java │ │ ├── api │ │ ├── MoneyWithdrawalApis.java │ │ └── MoneyWithdrawalCommand.java │ │ ├── cqrssink │ │ ├── BankOperation.java │ │ ├── CqrdReadModelUpdater.java │ │ ├── CqrsReadModel.java │ │ ├── DebitCardCdc.java │ │ ├── DebitCardCdcMessage.java │ │ └── MiniStatement.java │ │ ├── model │ │ ├── DebitCard.java │ │ ├── DebitCardNotFoundException.java │ │ ├── InSufficientMoneyException.java │ │ └── MoneyWithdrawal.java │ │ ├── repository │ │ ├── DebitCardRepository.java │ │ └── MoneyWithdrawalRepository.java │ │ └── service │ │ └── MoneyWithdrawalService.java └── resources │ ├── application.properties │ ├── data-mysql.sql │ └── schema.sql └── test └── java └── com └── gonnect └── debezium └── kafka └── bank └── account └── CdcApplicationTests.java /BankAppArch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mgorav/CqrsWithCDC/a119b76f5a0f63e0c3b14452f0c02fd0d2e77d11/BankAppArch.png -------------------------------------------------------------------------------- /CQREventSourcingAnalytics.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mgorav/CqrsWithCDC/a119b76f5a0f63e0c3b14452f0c02fd0d2e77d11/CQREventSourcingAnalytics.png -------------------------------------------------------------------------------- /CQRSEventSourcingStep1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mgorav/CqrsWithCDC/a119b76f5a0f63e0c3b14452f0c02fd0d2e77d11/CQRSEventSourcingStep1.png -------------------------------------------------------------------------------- /CQRSState.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mgorav/CqrsWithCDC/a119b76f5a0f63e0c3b14452f0c02fd0d2e77d11/CQRSState.png -------------------------------------------------------------------------------- /ClassicalBI.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mgorav/CqrsWithCDC/a119b76f5a0f63e0c3b14452f0c02fd0d2e77d11/ClassicalBI.png -------------------------------------------------------------------------------- /CommandToEvent.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mgorav/CqrsWithCDC/a119b76f5a0f63e0c3b14452f0c02fd0d2e77d11/CommandToEvent.png -------------------------------------------------------------------------------- /Cqrs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mgorav/CqrsWithCDC/a119b76f5a0f63e0c3b14452f0c02fd0d2e77d11/Cqrs.png -------------------------------------------------------------------------------- /CqrsIndexes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mgorav/CqrsWithCDC/a119b76f5a0f63e0c3b14452f0c02fd0d2e77d11/CqrsIndexes.png -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | # Start with a base image containing Java runtime 2 | FROM openjdk:8-jdk-alpine 3 | 4 | # Add a volume pointing to /tmp 5 | VOLUME /tmp 6 | 7 | # Make port 8888 available to the world outside this container 8 | EXPOSE 8080 9 | 10 | # The application's jar file 11 | ARG JAR_FILE=target/SpringCloudStreamDebeziumKafka-0.0.1.jar 12 | 13 | # Add the application's jar to the container 14 | ADD ${JAR_FILE} SpringCloudStreamDebeziumKafka-0.0.1.jar 15 | 16 | # Run the jar file 17 | ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/SpringCloudStreamDebeziumKafka-0.0.1.jar"] -------------------------------------------------------------------------------- /EventSourcing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mgorav/CqrsWithCDC/a119b76f5a0f63e0c3b14452f0c02fd0d2e77d11/EventSourcing.png -------------------------------------------------------------------------------- /EventSourcingSnapshotting.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mgorav/CqrsWithCDC/a119b76f5a0f63e0c3b14452f0c02fd0d2e77d11/EventSourcingSnapshotting.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # CQRS + Event Sourcing Using CDC + OLAP Using Druid 2 | 3 | CQRS & Event Sourcing are "THE BUZZ" words these days and how it fits in real time analytics. Microservices architecture causes segregation of applications (also called mushrooming of application landscape). This posses challenge for the Data Engineering teams. The proposed architecture (in the GitHub blog) demystifies CQRS + Event Sourcing in Real time analytics and hence bridges the gap between Full Stack Developers & Data Engineer. 4 | 5 | ![alt text](./Cqrs.png) 6 | 7 | **NOTE: CQRS + Event Sourcing = Elegant DDD** 8 | 9 | Event Souring - “All changes to an application state are stored as a sequence of events.” Martin Fowler 10 | - Change made to state are tracked as events 11 | - Event are stored in event store (any database) 12 | - Use stored events and summation of all these events (always arrive in current state) 13 | 14 | **NOTE**  Event Sourcing is not part of CQRS 15 | 16 | ![alt text](./EventSourcing.png) 17 | 18 | 19 | The Change Data Capture (CDC) provides an easy mechanism to implement these. Further CDC, in the world of micro-services, data services & real time analytics is a central part of a modern architecture these days.Check out my GitHub project which demonstrates: 20 | 1. CQRS 21 | 2. Event Sourcing 22 | 3. CDC using Debezium (good bye to expensive CDC products & complicated integration) 23 | 4. Kafka Connect 24 | 5. Kafka 25 | 6. Real time streaming 26 | 7. Spring Cloud Stream 27 | 28 | 29 | ## CQRS Bank Application - Using CQRS + Event Sourcing with Events relay using CDC 30 | A bank application which demonstrates CQRS design pattern. This application performs following operations: 31 | 32 | 1. _Money withdrawal using debit card_ 33 | 34 | 2. _List all the money withdrawal (mini bank statement)_ 35 | 36 | This application uses following two tables for above operations: 37 | 38 | 1. _**debit_card**_ 39 | 40 | 2. _**money_withdrawal**_ 41 | 42 | A debit card withdrawl operation is stored in debit_card table. Once the transaciton is successfully committed in the debit_card table, using Debezium's & Kafka connect the CDC is moved to Kafka. Once the message arrive in Kafka topic, using Spring Cloud Stream Stream Listener, an entry in made to money_withdrawl table. This table is used to create mini statement (query) 43 | 44 | **_NOTE_** For sake of simplicity same DB is used but as can be seen - "a command to perform debit operation" is separated from mini statement. 45 | 46 | 47 | Following picture shows architecture of this application: 48 | 49 | ![alt text](./BankAppArch.png) 50 | 51 | 52 | To demonstrate OLAP capabilities, this application write mini statement to Kafka topic - "ministatement". From this topic, Druid picks it up and provide fast querying abilities 53 | 54 | 55 | ## Pre-requisite 56 | 57 | 1. _MySQL_ 58 | 59 | 2. _Apache Kafka_ 60 | 61 | 3. _Kafka Connect_ 62 | 63 | 4. _Debezium_ 64 | 65 | 5. _Spring Cloud Stream_ 66 | 67 | 6. _Zookeeper_ 68 | 69 | 7. Druid 70 | 71 | 8. _Docker_ 72 | 73 | **_NOTE_**: This application is completely dockerized. 74 | 75 | ## Run Application 76 | 77 | **The complete reference architecture implementation is dockerized. Hence it takes just few minutes to run this application locally or any cloud provider of your choice.** 78 | 79 | Execute following steps to run the application: 80 | 81 | - _Build bank app_ 82 | 83 | ```bash 84 | mvn clean install -DskipTests 85 | ``` 86 | 87 | - _Run bank application complete infrastructure:_ 88 | 89 | ```bash 90 | docker-compose up 91 | ``` 92 | 93 | - _Instruct Kafka Connect to tail transaction log of MySQL DB and start sending messages as CDC to Kafka:_ 94 | 95 | ```bash 96 | curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" http://localhost:8083/connectors/ -d @mysqlsource.json --verbose 97 | ``` 98 | 99 | - _Instruct Druid connect with Kafka_ 100 | 101 | ```bash 102 | curl -XPOST -H'Content-Type: application/json' -d @bank-app-mini-statement-supervisor.json http://localhost:8081/druid/indexer/v1/supervisor 103 | ``` 104 | 105 | - _Money withdrawal operation:_ 106 | 107 | ```bash 108 | curl http://localhost:8080/moneywithdrawals -X POST --header 'Content-Type: application/json' -d '{"debitCard":"123456789", "amount": 10.00}' --verbose 109 | ``` 110 | 111 | - _Mini statement fetching operation (query/read model)_ 112 | 113 | ```bash 114 | curl http://localhost:8080/moneywithdrawals?debitCardId=123456789 --verbose 115 | ``` 116 | 117 | # CQRS,Event Sourcing & Its Usage In Analytics Demystified 118 | _In section some of key concepts of CQRS & Event Sourcing are explained along with it's usage in analytics_ 119 | 120 | **Command Vs Event** 121 | 122 | _Command – “submit order”_ 123 | 124 | - A request (imperative statement) 125 | - May fail 126 | - May affect multiple aggregates 127 | 128 | **NOTE**: _~~Rebuild Aggregate State from Commands~~_ 129 | 130 | _Event – “order submitted”_ 131 | 132 | - Statement of fact (past tense) 133 | - Never fails 134 | - May affect a single aggregate 135 | 136 | **NOTE:** _Rebuild Aggregate State from Events_ 137 | 138 | 139 | **Command to Event** 140 | 141 | Following digram explain creation of _**Event** from a **Command**_ 142 | 143 | 144 | ![alt text](./CommandToEvent.png) 145 | 146 | 147 | Once a **Command** is converted to an **Event**, _**the state**_ can be reterived as shown below: 148 | 149 | ![alt text](./RetervingState.png) 150 | 151 | 152 | **Classical BI** 153 | 154 | A classical BI is shown below: **_NOT A NEW IDEA_** 155 | 156 | ![alt text](./ClassicalBI.png) 157 | 158 | 159 | **Tying knots together - CQRS & Event Sourcing** 160 | 161 | Below shows **_CQRS_** & **_Event Sourcing_**: 162 | 163 | ![alt text](./CQRSEventSourcingStep1.png) 164 | 165 | But "?" is still missing block. This missing "?" is called **_"State" or "Materialized View"_** 166 | 167 | ![alt text](./CQRSState.png) 168 | 169 | - Query a RDMS?  Old style 170 | - RDMS is option (will become bottleneck as volume of data increases) 171 | - Views are optimized for specific query use cases 172 | - multiple view of same events 173 | - Update is asynchronously  delayed  eventual consistency  build to scale 174 | - Easy to evolve or fix 175 | - Change or fix logic; rebuild view from events  event log is the source of truth not the view 176 | - Views can be rebuilt from the events 177 | 178 | Indexing is key concerns when "Command" is separated from "Query". Following diagram shows architecture to achieve this: 179 | 180 | ![alt text](./CqrsIndexes.png) 181 | 182 | **Snapshotting** 183 | Snapshotting is one of the key concepts in "Event Sourcing". Following diagram shows this: 184 | 185 | ![alt text](./EventSourcingSnapshotting.png) 186 | 187 | **Event sourcing/cqrs drawback** 188 | 189 | - No “One-Size-Fits-All” 190 | - Multiple “topic’ implementation 191 | - Delayed reads 192 | - No ACID transactions 193 | - Additional complexity 194 | 195 | **Event sourcing/cqrs benefits** 196 | 197 | - No “One-Size-Fits-All” 198 | - “topic’ are optimized for usecases 199 | - Eventual consistency 200 | - History, temporal queries 201 | - Robust for data corruption 202 | 203 | 204 | **The Complete Picture** 205 | 206 | The below picture shows the end to end architecture using CQRS & Event Sourcing 207 | 208 | ![alt text](./CQREventSourcingAnalytics.png) 209 | -------------------------------------------------------------------------------- /RetervingState.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mgorav/CqrsWithCDC/a119b76f5a0f63e0c3b14452f0c02fd0d2e77d11/RetervingState.png -------------------------------------------------------------------------------- /bank-app-mini-statement-supervisor.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "kafka", 3 | "dataSchema": { 4 | "dataSource": "bankapp", 5 | "parser": { 6 | "type": "string", 7 | "parseSpec": { 8 | "format": "json", 9 | "timestampSpec": { 10 | "column": "time", 11 | "format": "auto" 12 | }, 13 | "dimensionsSpec": { 14 | "dimensions": [ 15 | "id", 16 | "amount", 17 | "debitCardId", 18 | "transactionDate" 19 | ] 20 | } 21 | } 22 | }, 23 | "metricsSpec" : [], 24 | "granularitySpec": { 25 | "type": "uniform", 26 | "segmentGranularity": "DAY", 27 | "queryGranularity": "NONE", 28 | "rollup": false 29 | } 30 | }, 31 | "tuningConfig": { 32 | "type": "kafka", 33 | "reportParseExceptions": false 34 | }, 35 | "ioConfig": { 36 | "topic": "ministatement", 37 | "replicas": 1, 38 | "taskDuration": "PT10M", 39 | "completionTimeout": "PT20M", 40 | "consumerProperties": { 41 | "bootstrap.servers": "localhost:9092" 42 | } 43 | } 44 | } 45 | -------------------------------------------------------------------------------- /debezium-jdbc/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM debezium/connect:0.7 2 | ENV KAFKA_CONNECT_JDBC_DIR=$KAFKA_CONNECT_PLUGINS_DIR/kafka-connect-jdbc 3 | 4 | # Deploy Kafka Connect JDBC 5 | RUN mkdir $KAFKA_CONNECT_JDBC_DIR && cd $KAFKA_CONNECT_JDBC_DIR &&\ 6 | curl -sO http://packages.confluent.io/maven/io/confluent/kafka-connect-jdbc/3.3.0/kafka-connect-jdbc-3.3.0.jar 7 | 8 | 9 | -------------------------------------------------------------------------------- /docker-compose.yaml: -------------------------------------------------------------------------------- 1 | version: '2' 2 | services: 3 | bank-app: 4 | build: . 5 | ports: 6 | - 8080:8080 7 | links: 8 | - mysql 9 | - kafka 10 | zookeeper: 11 | image: debezium/zookeeper:0.8 12 | ports: 13 | - 2181:2181 14 | - 2888:2888 15 | - 3888:3888 16 | kafka: 17 | image: debezium/kafka:0.8 18 | ports: 19 | - 9092:9092 20 | links: 21 | - zookeeper 22 | environment: 23 | - ZOOKEEPER_CONNECT=zookeeper:2181 24 | - ADVERTISED_LISTENERS=PLAINTEXT://192.168.1.17:9092 25 | mysql: 26 | image: debezium/example-mysql:0.8 27 | ports: 28 | - 3306:3306 29 | environment: 30 | - MYSQL_ROOT_PASSWORD=debezium 31 | - MYSQL_USER=mysqluser 32 | - MYSQL_PASSWORD=mysqlpw 33 | connect: 34 | image: debezium/connect-jdbc-es:0.8 35 | build: 36 | context: debezium-jdbc 37 | ports: 38 | - 8083:8083 39 | - 5005:5005 40 | links: 41 | - kafka 42 | - mysql 43 | environment: 44 | - BOOTSTRAP_SERVERS=kafka:9092 45 | - GROUP_ID=1 46 | - CONFIG_STORAGE_TOPIC=my_connect_configs 47 | - OFFSET_STORAGE_TOPIC=my_connect_offsets 48 | druid: 49 | image: fokkodriesprong/docker-druid 50 | ports: 51 | - 8081:8081 52 | - 8082:8082 53 | depends_on: 54 | - kafka 55 | links: 56 | - kafka 57 | superset: 58 | image: fokkodriesprong/docker-superset 59 | ports: 60 | - 8088:8088 61 | links: 62 | - druid 63 | -------------------------------------------------------------------------------- /jdbc-sink.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "jdbc-sink", 3 | "config": { 4 | "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector", 5 | "tasks.max": "1", 6 | "topics": "customers", 7 | "connection.url": "jdbc:postgresql://postgres:5432/inventory?user=postgresuser&password=postgrespw", 8 | "transforms": "unwrap", 9 | "transforms.unwrap.type": "io.debezium.transforms.UnwrapFromEnvelope", 10 | "auto.create": "true", 11 | "insert.mode": "upsert", 12 | "pk.fields": "id", 13 | "pk.mode": "record_value" 14 | } 15 | } 16 | -------------------------------------------------------------------------------- /mvnw: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # ---------------------------------------------------------------------------- 3 | # Licensed to the Apache Software Foundation (ASF) under one 4 | # or more contributor license agreements. See the NOTICE file 5 | # distributed with this work for additional information 6 | # regarding copyright ownership. The ASF licenses this file 7 | # to you under the Apache License, Version 2.0 (the 8 | # "License"); you may not use this file except in compliance 9 | # with the License. You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, 14 | # software distributed under the License is distributed on an 15 | # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 16 | # KIND, either express or implied. See the License for the 17 | # specific language governing permissions and limitations 18 | # under the License. 19 | # ---------------------------------------------------------------------------- 20 | 21 | # ---------------------------------------------------------------------------- 22 | # Maven2 Start Up Batch script 23 | # 24 | # Required ENV vars: 25 | # ------------------ 26 | # JAVA_HOME - location of a JDK home dir 27 | # 28 | # Optional ENV vars 29 | # ----------------- 30 | # M2_HOME - location of maven2's installed home dir 31 | # MAVEN_OPTS - parameters passed to the Java VM when running Maven 32 | # e.g. to debug Maven itself, use 33 | # set MAVEN_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 34 | # MAVEN_SKIP_RC - flag to disable loading of mavenrc files 35 | # ---------------------------------------------------------------------------- 36 | 37 | if [ -z "$MAVEN_SKIP_RC" ] ; then 38 | 39 | if [ -f /etc/mavenrc ] ; then 40 | . /etc/mavenrc 41 | fi 42 | 43 | if [ -f "$HOME/.mavenrc" ] ; then 44 | . "$HOME/.mavenrc" 45 | fi 46 | 47 | fi 48 | 49 | # OS specific support. $var _must_ be set to either true or false. 50 | cygwin=false; 51 | darwin=false; 52 | mingw=false 53 | case "`uname`" in 54 | CYGWIN*) cygwin=true ;; 55 | MINGW*) mingw=true;; 56 | Darwin*) darwin=true 57 | # Use /usr/libexec/java_home if available, otherwise fall back to /Library/Java/Home 58 | # See https://developer.apple.com/library/mac/qa/qa1170/_index.html 59 | if [ -z "$JAVA_HOME" ]; then 60 | if [ -x "/usr/libexec/java_home" ]; then 61 | export JAVA_HOME="`/usr/libexec/java_home`" 62 | else 63 | export JAVA_HOME="/Library/Java/Home" 64 | fi 65 | fi 66 | ;; 67 | esac 68 | 69 | if [ -z "$JAVA_HOME" ] ; then 70 | if [ -r /etc/gentoo-release ] ; then 71 | JAVA_HOME=`java-config --jre-home` 72 | fi 73 | fi 74 | 75 | if [ -z "$M2_HOME" ] ; then 76 | ## resolve links - $0 may be a link to maven's home 77 | PRG="$0" 78 | 79 | # need this for relative symlinks 80 | while [ -h "$PRG" ] ; do 81 | ls=`ls -ld "$PRG"` 82 | link=`expr "$ls" : '.*-> \(.*\)$'` 83 | if expr "$link" : '/.*' > /dev/null; then 84 | PRG="$link" 85 | else 86 | PRG="`dirname "$PRG"`/$link" 87 | fi 88 | done 89 | 90 | saveddir=`pwd` 91 | 92 | M2_HOME=`dirname "$PRG"`/.. 93 | 94 | # make it fully qualified 95 | M2_HOME=`cd "$M2_HOME" && pwd` 96 | 97 | cd "$saveddir" 98 | # echo Using m2 at $M2_HOME 99 | fi 100 | 101 | # For Cygwin, ensure paths are in UNIX format before anything is touched 102 | if $cygwin ; then 103 | [ -n "$M2_HOME" ] && 104 | M2_HOME=`cygpath --unix "$M2_HOME"` 105 | [ -n "$JAVA_HOME" ] && 106 | JAVA_HOME=`cygpath --unix "$JAVA_HOME"` 107 | [ -n "$CLASSPATH" ] && 108 | CLASSPATH=`cygpath --path --unix "$CLASSPATH"` 109 | fi 110 | 111 | # For Mingw, ensure paths are in UNIX format before anything is touched 112 | if $mingw ; then 113 | [ -n "$M2_HOME" ] && 114 | M2_HOME="`(cd "$M2_HOME"; pwd)`" 115 | [ -n "$JAVA_HOME" ] && 116 | JAVA_HOME="`(cd "$JAVA_HOME"; pwd)`" 117 | # TODO classpath? 118 | fi 119 | 120 | if [ -z "$JAVA_HOME" ]; then 121 | javaExecutable="`which javac`" 122 | if [ -n "$javaExecutable" ] && ! [ "`expr \"$javaExecutable\" : '\([^ ]*\)'`" = "no" ]; then 123 | # readlink(1) is not available as standard on Solaris 10. 124 | readLink=`which readlink` 125 | if [ ! `expr "$readLink" : '\([^ ]*\)'` = "no" ]; then 126 | if $darwin ; then 127 | javaHome="`dirname \"$javaExecutable\"`" 128 | javaExecutable="`cd \"$javaHome\" && pwd -P`/javac" 129 | else 130 | javaExecutable="`readlink -f \"$javaExecutable\"`" 131 | fi 132 | javaHome="`dirname \"$javaExecutable\"`" 133 | javaHome=`expr "$javaHome" : '\(.*\)/bin'` 134 | JAVA_HOME="$javaHome" 135 | export JAVA_HOME 136 | fi 137 | fi 138 | fi 139 | 140 | if [ -z "$JAVACMD" ] ; then 141 | if [ -n "$JAVA_HOME" ] ; then 142 | if [ -x "$JAVA_HOME/jre/sh/java" ] ; then 143 | # IBM's JDK on AIX uses strange locations for the executables 144 | JAVACMD="$JAVA_HOME/jre/sh/java" 145 | else 146 | JAVACMD="$JAVA_HOME/bin/java" 147 | fi 148 | else 149 | JAVACMD="`which java`" 150 | fi 151 | fi 152 | 153 | if [ ! -x "$JAVACMD" ] ; then 154 | echo "Error: JAVA_HOME is not defined correctly." >&2 155 | echo " We cannot execute $JAVACMD" >&2 156 | exit 1 157 | fi 158 | 159 | if [ -z "$JAVA_HOME" ] ; then 160 | echo "Warning: JAVA_HOME environment variable is not set." 161 | fi 162 | 163 | CLASSWORLDS_LAUNCHER=org.codehaus.plexus.classworlds.launcher.Launcher 164 | 165 | # traverses directory structure from process work directory to filesystem root 166 | # first directory with .mvn subdirectory is considered project base directory 167 | find_maven_basedir() { 168 | 169 | if [ -z "$1" ] 170 | then 171 | echo "Path not specified to find_maven_basedir" 172 | return 1 173 | fi 174 | 175 | basedir="$1" 176 | wdir="$1" 177 | while [ "$wdir" != '/' ] ; do 178 | if [ -d "$wdir"/.mvn ] ; then 179 | basedir=$wdir 180 | break 181 | fi 182 | # workaround for JBEAP-8937 (on Solaris 10/Sparc) 183 | if [ -d "${wdir}" ]; then 184 | wdir=`cd "$wdir/.."; pwd` 185 | fi 186 | # end of workaround 187 | done 188 | echo "${basedir}" 189 | } 190 | 191 | # concatenates all lines of a file 192 | concat_lines() { 193 | if [ -f "$1" ]; then 194 | echo "$(tr -s '\n' ' ' < "$1")" 195 | fi 196 | } 197 | 198 | BASE_DIR=`find_maven_basedir "$(pwd)"` 199 | if [ -z "$BASE_DIR" ]; then 200 | exit 1; 201 | fi 202 | 203 | ########################################################################################## 204 | # Extension to allow automatically downloading the maven-wrapper.jar from Maven-central 205 | # This allows using the maven wrapper in projects that prohibit checking in binary data. 206 | ########################################################################################## 207 | if [ -r "$BASE_DIR/.mvn/wrapper/maven-wrapper.jar" ]; then 208 | if [ "$MVNW_VERBOSE" = true ]; then 209 | echo "Found .mvn/wrapper/maven-wrapper.jar" 210 | fi 211 | else 212 | if [ "$MVNW_VERBOSE" = true ]; then 213 | echo "Couldn't find .mvn/wrapper/maven-wrapper.jar, downloading it ..." 214 | fi 215 | jarUrl="https://repo.maven.apache.org/maven2/io/takari/maven-wrapper/0.4.2/maven-wrapper-0.4.2.jar" 216 | while IFS="=" read key value; do 217 | case "$key" in (wrapperUrl) jarUrl="$value"; break ;; 218 | esac 219 | done < "$BASE_DIR/.mvn/wrapper/maven-wrapper.properties" 220 | if [ "$MVNW_VERBOSE" = true ]; then 221 | echo "Downloading from: $jarUrl" 222 | fi 223 | wrapperJarPath="$BASE_DIR/.mvn/wrapper/maven-wrapper.jar" 224 | 225 | if command -v wget > /dev/null; then 226 | if [ "$MVNW_VERBOSE" = true ]; then 227 | echo "Found wget ... using wget" 228 | fi 229 | wget "$jarUrl" -O "$wrapperJarPath" 230 | elif command -v curl > /dev/null; then 231 | if [ "$MVNW_VERBOSE" = true ]; then 232 | echo "Found curl ... using curl" 233 | fi 234 | curl -o "$wrapperJarPath" "$jarUrl" 235 | else 236 | if [ "$MVNW_VERBOSE" = true ]; then 237 | echo "Falling back to using Java to download" 238 | fi 239 | javaClass="$BASE_DIR/.mvn/wrapper/MavenWrapperDownloader.java" 240 | if [ -e "$javaClass" ]; then 241 | if [ ! -e "$BASE_DIR/.mvn/wrapper/MavenWrapperDownloader.class" ]; then 242 | if [ "$MVNW_VERBOSE" = true ]; then 243 | echo " - Compiling MavenWrapperDownloader.java ..." 244 | fi 245 | # Compiling the Java class 246 | ("$JAVA_HOME/bin/javac" "$javaClass") 247 | fi 248 | if [ -e "$BASE_DIR/.mvn/wrapper/MavenWrapperDownloader.class" ]; then 249 | # Running the downloader 250 | if [ "$MVNW_VERBOSE" = true ]; then 251 | echo " - Running MavenWrapperDownloader.java ..." 252 | fi 253 | ("$JAVA_HOME/bin/java" -cp .mvn/wrapper MavenWrapperDownloader "$MAVEN_PROJECTBASEDIR") 254 | fi 255 | fi 256 | fi 257 | fi 258 | ########################################################################################## 259 | # End of extension 260 | ########################################################################################## 261 | 262 | export MAVEN_PROJECTBASEDIR=${MAVEN_BASEDIR:-"$BASE_DIR"} 263 | if [ "$MVNW_VERBOSE" = true ]; then 264 | echo $MAVEN_PROJECTBASEDIR 265 | fi 266 | MAVEN_OPTS="$(concat_lines "$MAVEN_PROJECTBASEDIR/.mvn/jvm.config") $MAVEN_OPTS" 267 | 268 | # For Cygwin, switch paths to Windows format before running java 269 | if $cygwin; then 270 | [ -n "$M2_HOME" ] && 271 | M2_HOME=`cygpath --path --windows "$M2_HOME"` 272 | [ -n "$JAVA_HOME" ] && 273 | JAVA_HOME=`cygpath --path --windows "$JAVA_HOME"` 274 | [ -n "$CLASSPATH" ] && 275 | CLASSPATH=`cygpath --path --windows "$CLASSPATH"` 276 | [ -n "$MAVEN_PROJECTBASEDIR" ] && 277 | MAVEN_PROJECTBASEDIR=`cygpath --path --windows "$MAVEN_PROJECTBASEDIR"` 278 | fi 279 | 280 | WRAPPER_LAUNCHER=org.apache.maven.wrapper.MavenWrapperMain 281 | 282 | exec "$JAVACMD" \ 283 | $MAVEN_OPTS \ 284 | -classpath "$MAVEN_PROJECTBASEDIR/.mvn/wrapper/maven-wrapper.jar" \ 285 | "-Dmaven.home=${M2_HOME}" "-Dmaven.multiModuleProjectDirectory=${MAVEN_PROJECTBASEDIR}" \ 286 | ${WRAPPER_LAUNCHER} $MAVEN_CONFIG "$@" 287 | -------------------------------------------------------------------------------- /mvnw.cmd: -------------------------------------------------------------------------------- 1 | @REM ---------------------------------------------------------------------------- 2 | @REM Licensed to the Apache Software Foundation (ASF) under one 3 | @REM or more contributor license agreements. See the NOTICE file 4 | @REM distributed with this work for additional information 5 | @REM regarding copyright ownership. The ASF licenses this file 6 | @REM to you under the Apache License, Version 2.0 (the 7 | @REM "License"); you may not use this file except in compliance 8 | @REM with the License. You may obtain a copy of the License at 9 | @REM 10 | @REM http://www.apache.org/licenses/LICENSE-2.0 11 | @REM 12 | @REM Unless required by applicable law or agreed to in writing, 13 | @REM software distributed under the License is distributed on an 14 | @REM "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 15 | @REM KIND, either express or implied. See the License for the 16 | @REM specific language governing permissions and limitations 17 | @REM under the License. 18 | @REM ---------------------------------------------------------------------------- 19 | 20 | @REM ---------------------------------------------------------------------------- 21 | @REM Maven2 Start Up Batch script 22 | @REM 23 | @REM Required ENV vars: 24 | @REM JAVA_HOME - location of a JDK home dir 25 | @REM 26 | @REM Optional ENV vars 27 | @REM M2_HOME - location of maven2's installed home dir 28 | @REM MAVEN_BATCH_ECHO - set to 'on' to enable the echoing of the batch commands 29 | @REM MAVEN_BATCH_PAUSE - set to 'on' to wait for a key stroke before ending 30 | @REM MAVEN_OPTS - parameters passed to the Java VM when running Maven 31 | @REM e.g. to debug Maven itself, use 32 | @REM set MAVEN_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 33 | @REM MAVEN_SKIP_RC - flag to disable loading of mavenrc files 34 | @REM ---------------------------------------------------------------------------- 35 | 36 | @REM Begin all REM lines with '@' in case MAVEN_BATCH_ECHO is 'on' 37 | @echo off 38 | @REM set title of command window 39 | title %0 40 | @REM enable echoing my setting MAVEN_BATCH_ECHO to 'on' 41 | @if "%MAVEN_BATCH_ECHO%" == "on" echo %MAVEN_BATCH_ECHO% 42 | 43 | @REM set %HOME% to equivalent of $HOME 44 | if "%HOME%" == "" (set "HOME=%HOMEDRIVE%%HOMEPATH%") 45 | 46 | @REM Execute a user defined script before this one 47 | if not "%MAVEN_SKIP_RC%" == "" goto skipRcPre 48 | @REM check for pre script, once with legacy .bat ending and once with .cmd ending 49 | if exist "%HOME%\mavenrc_pre.bat" call "%HOME%\mavenrc_pre.bat" 50 | if exist "%HOME%\mavenrc_pre.cmd" call "%HOME%\mavenrc_pre.cmd" 51 | :skipRcPre 52 | 53 | @setlocal 54 | 55 | set ERROR_CODE=0 56 | 57 | @REM To isolate internal variables from possible post scripts, we use another setlocal 58 | @setlocal 59 | 60 | @REM ==== START VALIDATION ==== 61 | if not "%JAVA_HOME%" == "" goto OkJHome 62 | 63 | echo. 64 | echo Error: JAVA_HOME not found in your environment. >&2 65 | echo Please set the JAVA_HOME variable in your environment to match the >&2 66 | echo location of your Java installation. >&2 67 | echo. 68 | goto error 69 | 70 | :OkJHome 71 | if exist "%JAVA_HOME%\bin\java.exe" goto init 72 | 73 | echo. 74 | echo Error: JAVA_HOME is set to an invalid directory. >&2 75 | echo JAVA_HOME = "%JAVA_HOME%" >&2 76 | echo Please set the JAVA_HOME variable in your environment to match the >&2 77 | echo location of your Java installation. >&2 78 | echo. 79 | goto error 80 | 81 | @REM ==== END VALIDATION ==== 82 | 83 | :init 84 | 85 | @REM Find the project base dir, i.e. the directory that contains the folder ".mvn". 86 | @REM Fallback to current working directory if not found. 87 | 88 | set MAVEN_PROJECTBASEDIR=%MAVEN_BASEDIR% 89 | IF NOT "%MAVEN_PROJECTBASEDIR%"=="" goto endDetectBaseDir 90 | 91 | set EXEC_DIR=%CD% 92 | set WDIR=%EXEC_DIR% 93 | :findBaseDir 94 | IF EXIST "%WDIR%"\.mvn goto baseDirFound 95 | cd .. 96 | IF "%WDIR%"=="%CD%" goto baseDirNotFound 97 | set WDIR=%CD% 98 | goto findBaseDir 99 | 100 | :baseDirFound 101 | set MAVEN_PROJECTBASEDIR=%WDIR% 102 | cd "%EXEC_DIR%" 103 | goto endDetectBaseDir 104 | 105 | :baseDirNotFound 106 | set MAVEN_PROJECTBASEDIR=%EXEC_DIR% 107 | cd "%EXEC_DIR%" 108 | 109 | :endDetectBaseDir 110 | 111 | IF NOT EXIST "%MAVEN_PROJECTBASEDIR%\.mvn\jvm.config" goto endReadAdditionalConfig 112 | 113 | @setlocal EnableExtensions EnableDelayedExpansion 114 | for /F "usebackq delims=" %%a in ("%MAVEN_PROJECTBASEDIR%\.mvn\jvm.config") do set JVM_CONFIG_MAVEN_PROPS=!JVM_CONFIG_MAVEN_PROPS! %%a 115 | @endlocal & set JVM_CONFIG_MAVEN_PROPS=%JVM_CONFIG_MAVEN_PROPS% 116 | 117 | :endReadAdditionalConfig 118 | 119 | SET MAVEN_JAVA_EXE="%JAVA_HOME%\bin\java.exe" 120 | set WRAPPER_JAR="%MAVEN_PROJECTBASEDIR%\.mvn\wrapper\maven-wrapper.jar" 121 | set WRAPPER_LAUNCHER=org.apache.maven.wrapper.MavenWrapperMain 122 | 123 | set DOWNLOAD_URL="https://repo.maven.apache.org/maven2/io/takari/maven-wrapper/0.4.2/maven-wrapper-0.4.2.jar" 124 | FOR /F "tokens=1,2 delims==" %%A IN (%MAVEN_PROJECTBASEDIR%\.mvn\wrapper\maven-wrapper.properties) DO ( 125 | IF "%%A"=="wrapperUrl" SET DOWNLOAD_URL=%%B 126 | ) 127 | 128 | @REM Extension to allow automatically downloading the maven-wrapper.jar from Maven-central 129 | @REM This allows using the maven wrapper in projects that prohibit checking in binary data. 130 | if exist %WRAPPER_JAR% ( 131 | echo Found %WRAPPER_JAR% 132 | ) else ( 133 | echo Couldn't find %WRAPPER_JAR%, downloading it ... 134 | echo Downloading from: %DOWNLOAD_URL% 135 | powershell -Command "(New-Object Net.WebClient).DownloadFile('%DOWNLOAD_URL%', '%WRAPPER_JAR%')" 136 | echo Finished downloading %WRAPPER_JAR% 137 | ) 138 | @REM End of extension 139 | 140 | %MAVEN_JAVA_EXE% %JVM_CONFIG_MAVEN_PROPS% %MAVEN_OPTS% %MAVEN_DEBUG_OPTS% -classpath %WRAPPER_JAR% "-Dmaven.multiModuleProjectDirectory=%MAVEN_PROJECTBASEDIR%" %WRAPPER_LAUNCHER% %MAVEN_CONFIG% %* 141 | if ERRORLEVEL 1 goto error 142 | goto end 143 | 144 | :error 145 | set ERROR_CODE=1 146 | 147 | :end 148 | @endlocal & set ERROR_CODE=%ERROR_CODE% 149 | 150 | if not "%MAVEN_SKIP_RC%" == "" goto skipRcPost 151 | @REM check for post script, once with legacy .bat ending and once with .cmd ending 152 | if exist "%HOME%\mavenrc_post.bat" call "%HOME%\mavenrc_post.bat" 153 | if exist "%HOME%\mavenrc_post.cmd" call "%HOME%\mavenrc_post.cmd" 154 | :skipRcPost 155 | 156 | @REM pause the script if MAVEN_BATCH_PAUSE is set to 'on' 157 | if "%MAVEN_BATCH_PAUSE%" == "on" pause 158 | 159 | if "%MAVEN_TERMINATE_CMD%" == "on" exit %ERROR_CODE% 160 | 161 | exit /B %ERROR_CODE% 162 | -------------------------------------------------------------------------------- /mysqlsource.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "inventory-connector", 3 | "config": { 4 | "connector.class": "io.debezium.connector.mysql.MySqlConnector", 5 | "tasks.max": "1", 6 | "database.hostname": "mysql", 7 | "database.port": "3306", 8 | "database.user": "debezium", 9 | "database.password": "dbz", 10 | "database.server.id": "184054", 11 | "database.server.name": "dbserver1", 12 | "database.whitelist": "inventory", 13 | "database.history.kafka.bootstrap.servers": "kafka:9092", 14 | "database.history.kafka.topic": "schema-changes.inventory", 15 | "transforms": "route", 16 | "transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter", 17 | "transforms.route.regex": "([^.]+)\\.([^.]+)\\.([^.]+)", 18 | "transforms.route.replacement": "$3" 19 | } 20 | } 21 | -------------------------------------------------------------------------------- /pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 4 | 4.0.0 5 | 6 | org.springframework.boot 7 | spring-boot-starter-parent 8 | 2.0.4.RELEASE 9 | 10 | 11 | com.gonnect.debezium.kafka 12 | SpringCloudStreamDebeziumKafka 13 | 0.0.1 14 | cdc 15 | CQRS Using CDC Capture 16 | 17 | 18 | UTF-8 19 | UTF-8 20 | 1.8 21 | Finchley.SR1 22 | 23 | 24 | 25 | 26 | 27 | org.springframework.cloud 28 | spring-cloud-dependencies 29 | ${spring-cloud.version} 30 | pom 31 | import 32 | 33 | 34 | 35 | 36 | 37 | 38 | org.springframework.boot 39 | spring-boot-starter-data-jpa 40 | 41 | 42 | org.springframework.boot 43 | spring-boot-starter-web 44 | 45 | 46 | org.springframework.boot 47 | spring-boot-starter-actuator 48 | 49 | 50 | 51 | org.springframework.cloud 52 | spring-cloud-stream-binder-kafka 53 | 54 | 55 | org.springframework.cloud 56 | spring-cloud-stream 57 | 58 | 59 | 60 | mysql 61 | mysql-connector-java 62 | 63 | 64 | 65 | org.projectlombok 66 | lombok 67 | 1.18.0 68 | provided 69 | 70 | 71 | 72 | org.springframework.boot 73 | spring-boot-starter-test 74 | test 75 | 76 | 77 | 78 | com.h2database 79 | h2 80 | test 81 | 82 | 83 | 84 | org.springframework.cloud 85 | spring-cloud-stream-test-support 86 | test 87 | 88 | 89 | 90 | 91 | 92 | 93 | org.springframework.boot 94 | spring-boot-maven-plugin 95 | 96 | 97 | 98 | 99 | 100 | 101 | spring-snapshots 102 | Spring Snapshots 103 | https://repo.spring.io/libs-snapshot-local 104 | 105 | true 106 | 107 | 108 | false 109 | 110 | 111 | 112 | spring-milestones 113 | Spring Milestones 114 | https://repo.spring.io/libs-milestone-local 115 | 116 | false 117 | 118 | 119 | 120 | spring-releases 121 | Spring Releases 122 | https://repo.spring.io/release 123 | 124 | false 125 | 126 | 127 | 128 | 129 | 130 | spring-snapshots 131 | Spring Snapshots 132 | https://repo.spring.io/libs-snapshot-local 133 | 134 | true 135 | 136 | 137 | false 138 | 139 | 140 | 141 | spring-milestones 142 | Spring Milestones 143 | https://repo.spring.io/libs-milestone-local 144 | 145 | false 146 | 147 | 148 | 149 | spring-releases 150 | Spring Releases 151 | https://repo.spring.io/libs-release-local 152 | 153 | false 154 | 155 | 156 | 157 | 158 | 159 | -------------------------------------------------------------------------------- /src/main/java/com/gonnect/debezium/kafka/bank/account/CdcApplication.java: -------------------------------------------------------------------------------- 1 | package com.gonnect.debezium.kafka.bank.account; 2 | 3 | import org.springframework.boot.SpringApplication; 4 | import org.springframework.boot.autoconfigure.SpringBootApplication; 5 | 6 | @SpringBootApplication 7 | public class CdcApplication { 8 | 9 | public static void main(String[] args) { 10 | SpringApplication.run(CdcApplication.class, args); 11 | } 12 | 13 | } 14 | 15 | -------------------------------------------------------------------------------- /src/main/java/com/gonnect/debezium/kafka/bank/account/api/MoneyWithdrawalApis.java: -------------------------------------------------------------------------------- 1 | package com.gonnect.debezium.kafka.bank.account.api; 2 | 3 | import com.gonnect.debezium.kafka.bank.account.model.MoneyWithdrawal; 4 | import com.gonnect.debezium.kafka.bank.account.repository.MoneyWithdrawalRepository; 5 | import com.gonnect.debezium.kafka.bank.account.service.MoneyWithdrawalService; 6 | import org.springframework.http.ResponseEntity; 7 | import org.springframework.web.bind.annotation.GetMapping; 8 | import org.springframework.web.bind.annotation.PostMapping; 9 | import org.springframework.web.bind.annotation.RequestBody; 10 | import org.springframework.web.bind.annotation.RestController; 11 | 12 | import javax.websocket.server.PathParam; 13 | import java.util.List; 14 | 15 | @RestController("/moneywithdrawals") 16 | public class MoneyWithdrawalApis { 17 | 18 | private final MoneyWithdrawalRepository moneyWithdrawalRepository; 19 | private final MoneyWithdrawalService moneyWithdrawalService; 20 | 21 | MoneyWithdrawalApis(MoneyWithdrawalRepository moneyWithdrawalRepository, MoneyWithdrawalService moneyWithdrawalService) { 22 | this.moneyWithdrawalRepository = moneyWithdrawalRepository; 23 | this.moneyWithdrawalService = moneyWithdrawalService; 24 | } 25 | 26 | @PostMapping 27 | ResponseEntity withdraw(@RequestBody MoneyWithdrawalCommand moneyWithdrawalCommand) { 28 | moneyWithdrawalService.withdraw(moneyWithdrawalCommand.getDebitCard(), moneyWithdrawalCommand.getAmount()); 29 | return ResponseEntity.ok().build(); 30 | } 31 | 32 | @GetMapping 33 | ResponseEntity> withdrawals(@PathParam("debitCardId") String debitCardId) { 34 | return ResponseEntity.ok().body(moneyWithdrawalRepository.findByDebitCardId(debitCardId)); 35 | } 36 | } 37 | -------------------------------------------------------------------------------- /src/main/java/com/gonnect/debezium/kafka/bank/account/api/MoneyWithdrawalCommand.java: -------------------------------------------------------------------------------- 1 | package com.gonnect.debezium.kafka.bank.account.api; 2 | 3 | import lombok.AllArgsConstructor; 4 | import lombok.Data; 5 | import lombok.NoArgsConstructor; 6 | 7 | import java.math.BigDecimal; 8 | 9 | @Data 10 | @NoArgsConstructor 11 | @AllArgsConstructor 12 | public class MoneyWithdrawalCommand { 13 | 14 | private String debitCard; 15 | private BigDecimal amount; 16 | } 17 | -------------------------------------------------------------------------------- /src/main/java/com/gonnect/debezium/kafka/bank/account/cqrssink/BankOperation.java: -------------------------------------------------------------------------------- 1 | package com.gonnect.debezium.kafka.bank.account.cqrssink; 2 | 3 | public class BankOperation { 4 | 5 | private String op; 6 | private String ts_ms; 7 | private DebitCardCdc before; 8 | private DebitCardCdc after; 9 | 10 | String getOp() { 11 | return op; 12 | } 13 | 14 | void setOp(String op) { 15 | this.op = op; 16 | } 17 | 18 | DebitCardCdc getBefore() { 19 | return before; 20 | } 21 | 22 | void setBefore(DebitCardCdc before) { 23 | this.before = before; 24 | } 25 | 26 | String getTs_ms() { 27 | return ts_ms; 28 | } 29 | 30 | void setTs_ms(String ts_ms) { 31 | this.ts_ms = ts_ms; 32 | } 33 | 34 | DebitCardCdc getAfter() { 35 | return after; 36 | } 37 | 38 | void setAfter(DebitCardCdc after) { 39 | this.after = after; 40 | } 41 | } 42 | -------------------------------------------------------------------------------- /src/main/java/com/gonnect/debezium/kafka/bank/account/cqrssink/CqrdReadModelUpdater.java: -------------------------------------------------------------------------------- 1 | package com.gonnect.debezium.kafka.bank.account.cqrssink; 2 | 3 | import com.gonnect.debezium.kafka.bank.account.model.MoneyWithdrawal; 4 | import com.gonnect.debezium.kafka.bank.account.repository.MoneyWithdrawalRepository; 5 | import lombok.extern.slf4j.Slf4j; 6 | import org.springframework.cloud.stream.annotation.StreamListener; 7 | import org.springframework.cloud.stream.messaging.Processor; 8 | import org.springframework.cloud.stream.messaging.Sink; 9 | import org.springframework.messaging.handler.annotation.SendTo; 10 | import org.springframework.stereotype.Service; 11 | 12 | import java.math.BigDecimal; 13 | import java.util.Date; 14 | 15 | import static org.springframework.cloud.stream.messaging.Sink.INPUT; 16 | import static org.springframework.cloud.stream.messaging.Source.OUTPUT; 17 | 18 | @Service 19 | @Slf4j 20 | public class CqrdReadModelUpdater { 21 | 22 | private final MoneyWithdrawalRepository moneyWithdrawalRepository; 23 | 24 | CqrdReadModelUpdater(MoneyWithdrawalRepository moneyWithdrawalRepository) { 25 | this.moneyWithdrawalRepository = moneyWithdrawalRepository; 26 | } 27 | 28 | @StreamListener(INPUT) 29 | @SendTo(OUTPUT) 30 | public MiniStatement process(DebitCardCdcMessage message) { 31 | MiniStatement miniStatement = new MiniStatement(); 32 | if(message.isUpdate()) { 33 | MoneyWithdrawal moneyWithdrawal = saveWithdrawalFrom(message); 34 | miniStatement.setId(moneyWithdrawal.getId()); 35 | miniStatement.setAmount(moneyWithdrawal.getAmount()); 36 | miniStatement.setDebitCardId(moneyWithdrawal.getDebitCardId()); 37 | miniStatement.setTransactionDate(new Date(System.currentTimeMillis())); 38 | 39 | } 40 | 41 | log.info("Producing mini statement: " + miniStatement.toString()); 42 | 43 | return miniStatement; 44 | } 45 | 46 | private MoneyWithdrawal saveWithdrawalFrom(DebitCardCdcMessage message) { 47 | String debitCardId = message.getPayload().getBefore().getId(); 48 | BigDecimal withdrawalAmount 49 | = balanceAfter(message).subtract(balanceBefore(message)); 50 | return moneyWithdrawalRepository.save(new MoneyWithdrawal(withdrawalAmount, debitCardId)); 51 | } 52 | 53 | private BigDecimal balanceAfter(DebitCardCdcMessage message) { 54 | return message.getPayload().getAfter().getUsed_limit(); 55 | } 56 | 57 | private BigDecimal balanceBefore(DebitCardCdcMessage message) { 58 | return message.getPayload().getBefore().getUsed_limit(); 59 | } 60 | } 61 | -------------------------------------------------------------------------------- /src/main/java/com/gonnect/debezium/kafka/bank/account/cqrssink/CqrsReadModel.java: -------------------------------------------------------------------------------- 1 | package com.gonnect.debezium.kafka.bank.account.cqrssink; 2 | 3 | import org.springframework.cloud.stream.annotation.EnableBinding; 4 | import org.springframework.cloud.stream.messaging.Sink; 5 | import org.springframework.context.annotation.Configuration; 6 | 7 | @Configuration 8 | @EnableBinding(Sink.class) 9 | public class CqrsReadModel { 10 | } 11 | -------------------------------------------------------------------------------- /src/main/java/com/gonnect/debezium/kafka/bank/account/cqrssink/DebitCardCdc.java: -------------------------------------------------------------------------------- 1 | package com.gonnect.debezium.kafka.bank.account.cqrssink; 2 | 3 | import java.math.BigDecimal; 4 | import java.math.BigInteger; 5 | import java.util.Base64; 6 | 7 | public class DebitCardCdc { 8 | private String id; 9 | private String used_limit; 10 | 11 | BigDecimal getUsed_limit() { 12 | return new BigDecimal(new BigInteger(Base64.getDecoder().decode(used_limit)), 2); 13 | } 14 | 15 | void setUsed_limit(String used_limit) { 16 | this.used_limit = used_limit; 17 | } 18 | 19 | public String getId() { 20 | return id; 21 | } 22 | 23 | public void setId(String id) { 24 | this.id = id; 25 | } 26 | } 27 | -------------------------------------------------------------------------------- /src/main/java/com/gonnect/debezium/kafka/bank/account/cqrssink/DebitCardCdcMessage.java: -------------------------------------------------------------------------------- 1 | package com.gonnect.debezium.kafka.bank.account.cqrssink; 2 | 3 | public class DebitCardCdcMessage { 4 | 5 | private BankOperation payload; 6 | 7 | public DebitCardCdcMessage() { 8 | } 9 | 10 | public DebitCardCdcMessage(BankOperation payload) { 11 | this.payload = payload; 12 | } 13 | 14 | BankOperation getPayload() { 15 | return payload; 16 | } 17 | 18 | void setPayload(BankOperation payload) { 19 | this.payload = payload; 20 | } 21 | 22 | boolean isUpdate() { 23 | return payload.getOp().equals("u"); 24 | } 25 | } 26 | -------------------------------------------------------------------------------- /src/main/java/com/gonnect/debezium/kafka/bank/account/cqrssink/MiniStatement.java: -------------------------------------------------------------------------------- 1 | package com.gonnect.debezium.kafka.bank.account.cqrssink; 2 | 3 | import lombok.Getter; 4 | import lombok.NoArgsConstructor; 5 | import lombok.Setter; 6 | 7 | import java.math.BigDecimal; 8 | import java.util.Date; 9 | 10 | @NoArgsConstructor 11 | @Getter 12 | @Setter 13 | public class MiniStatement { 14 | 15 | private String id; 16 | private BigDecimal amount; 17 | private String debitCardId; 18 | private Date transactionDate; 19 | } 20 | -------------------------------------------------------------------------------- /src/main/java/com/gonnect/debezium/kafka/bank/account/model/DebitCard.java: -------------------------------------------------------------------------------- 1 | package com.gonnect.debezium.kafka.bank.account.model; 2 | 3 | import lombok.Getter; 4 | import lombok.NoArgsConstructor; 5 | 6 | import javax.persistence.Entity; 7 | import javax.persistence.Id; 8 | import java.math.BigDecimal; 9 | import java.util.UUID; 10 | 11 | @Entity 12 | @NoArgsConstructor 13 | public class DebitCard { 14 | 15 | @Id 16 | @Getter private String id; 17 | private BigDecimal initialLimit; 18 | private BigDecimal usedLimit = BigDecimal.ZERO; 19 | 20 | public DebitCard(BigDecimal limit) { 21 | this.initialLimit = limit; 22 | this.id = UUID.randomUUID().toString(); 23 | } 24 | 25 | public void withdraw(BigDecimal amount) { 26 | if (thereIsMoneyToWithdraw(amount)) { 27 | usedLimit = usedLimit.add(amount); 28 | } else { 29 | throw new InSufficientMoneyException(id, amount, availableBalance()); 30 | } 31 | } 32 | 33 | public BigDecimal availableBalance() { 34 | return initialLimit.subtract(usedLimit); 35 | } 36 | 37 | private boolean thereIsMoneyToWithdraw(BigDecimal amount) { 38 | return availableBalance().compareTo(amount) >= 0; 39 | } 40 | 41 | } 42 | -------------------------------------------------------------------------------- /src/main/java/com/gonnect/debezium/kafka/bank/account/model/DebitCardNotFoundException.java: -------------------------------------------------------------------------------- 1 | package com.gonnect.debezium.kafka.bank.account.model; 2 | 3 | import java.math.BigDecimal; 4 | 5 | public class DebitCardNotFoundException extends RuntimeException { 6 | 7 | public DebitCardNotFoundException(String message) { 8 | super(message); 9 | } 10 | 11 | 12 | } 13 | -------------------------------------------------------------------------------- /src/main/java/com/gonnect/debezium/kafka/bank/account/model/InSufficientMoneyException.java: -------------------------------------------------------------------------------- 1 | package com.gonnect.debezium.kafka.bank.account.model; 2 | 3 | import java.math.BigDecimal; 4 | 5 | class InSufficientMoneyException extends RuntimeException { 6 | 7 | InSufficientMoneyException(String cardNo, BigDecimal wanted, BigDecimal availableBalance) { 8 | super(String.format("Card %s not able to withdraw %s. Balance is %s", cardNo, wanted, availableBalance)); 9 | } 10 | 11 | 12 | } 13 | -------------------------------------------------------------------------------- /src/main/java/com/gonnect/debezium/kafka/bank/account/model/MoneyWithdrawal.java: -------------------------------------------------------------------------------- 1 | package com.gonnect.debezium.kafka.bank.account.model; 2 | 3 | import lombok.Getter; 4 | import lombok.NoArgsConstructor; 5 | import lombok.Setter; 6 | import lombok.ToString; 7 | 8 | import javax.persistence.Entity; 9 | import javax.persistence.Id; 10 | import java.math.BigDecimal; 11 | import java.util.UUID; 12 | 13 | @Entity 14 | @NoArgsConstructor 15 | @Getter 16 | @Setter 17 | @ToString 18 | public class MoneyWithdrawal { 19 | 20 | @Id 21 | private String id; 22 | private @Getter BigDecimal amount; 23 | private String debitCardId; 24 | 25 | public MoneyWithdrawal(BigDecimal amount, String debitCardId) { 26 | this.id = UUID.randomUUID().toString(); 27 | this.amount = amount; 28 | this.debitCardId = debitCardId; 29 | } 30 | } 31 | -------------------------------------------------------------------------------- /src/main/java/com/gonnect/debezium/kafka/bank/account/repository/DebitCardRepository.java: -------------------------------------------------------------------------------- 1 | package com.gonnect.debezium.kafka.bank.account.repository; 2 | 3 | import com.gonnect.debezium.kafka.bank.account.model.DebitCard; 4 | import org.springframework.data.jpa.repository.JpaRepository; 5 | 6 | public interface DebitCardRepository extends JpaRepository { 7 | } 8 | -------------------------------------------------------------------------------- /src/main/java/com/gonnect/debezium/kafka/bank/account/repository/MoneyWithdrawalRepository.java: -------------------------------------------------------------------------------- 1 | package com.gonnect.debezium.kafka.bank.account.repository; 2 | 3 | import com.gonnect.debezium.kafka.bank.account.model.MoneyWithdrawal; 4 | import org.springframework.data.jpa.repository.JpaRepository; 5 | 6 | import java.util.List; 7 | 8 | public interface MoneyWithdrawalRepository extends JpaRepository { 9 | 10 | 11 | List findByDebitCardId(String debitCardId); 12 | } 13 | -------------------------------------------------------------------------------- /src/main/java/com/gonnect/debezium/kafka/bank/account/service/MoneyWithdrawalService.java: -------------------------------------------------------------------------------- 1 | package com.gonnect.debezium.kafka.bank.account.service; 2 | 3 | import com.gonnect.debezium.kafka.bank.account.model.DebitCard; 4 | import com.gonnect.debezium.kafka.bank.account.model.DebitCardNotFoundException; 5 | import com.gonnect.debezium.kafka.bank.account.repository.DebitCardRepository; 6 | import com.gonnect.debezium.kafka.bank.account.repository.MoneyWithdrawalRepository; 7 | import org.springframework.stereotype.Service; 8 | 9 | import javax.transaction.Transactional; 10 | import java.math.BigDecimal; 11 | 12 | @Service 13 | public class MoneyWithdrawalService { 14 | 15 | private final DebitCardRepository debitCardRepository; 16 | private final MoneyWithdrawalRepository moneyWithdrawalRepository; 17 | 18 | MoneyWithdrawalService(DebitCardRepository debitCardRepository, MoneyWithdrawalRepository moneyWithdrawalRepository) { 19 | this.debitCardRepository = debitCardRepository; 20 | this.moneyWithdrawalRepository = moneyWithdrawalRepository; 21 | } 22 | 23 | 24 | @Transactional 25 | public void withdraw(String debitCardId, BigDecimal amount) { 26 | 27 | DebitCard debitCard = debitCardRepository.findById(debitCardId) 28 | .orElseThrow(() -> new DebitCardNotFoundException("Cannot find debit card with id " + debitCardId)); 29 | debitCard.withdraw(amount); 30 | } 31 | } 32 | -------------------------------------------------------------------------------- /src/main/resources/application.properties: -------------------------------------------------------------------------------- 1 | spring.datasource.url=jdbc:mysql://mysql:3306/inventory?autoReconnect=true&useSSL=false 2 | spring.datasource.username=mysqluser 3 | spring.datasource.password=mysqlpw 4 | spring.jpa.hibernate.ddl-auto=none 5 | spring.datasource.initialization-mode=always 6 | spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQL5Dialect 7 | 8 | spring.cloud.stream.kafka.binder.brokers=kafka:9092 9 | 10 | spring.cloud.stream.bindings.input.destination=debit_card 11 | spring.cloud.stream.bindings.input.contentType=application/json 12 | 13 | 14 | spring.cloud.stream.bindings.output.destination=ministatement 15 | spring.cloud.stream.bindings.output.contentType=application/json 16 | 17 | spring.cloud.stream.kafka.binder.transaction.producer.configuration.retries=1 18 | spring.cloud.stream.kafka.binder.transaction.producer.configuration.acks=all 19 | 20 | 21 | 22 | spring.datasource.platform=mysql 23 | -------------------------------------------------------------------------------- /src/main/resources/data-mysql.sql: -------------------------------------------------------------------------------- 1 | INSERT IGNORE INTO debit_card (ID, INITIAL_LIMIT, USED_LIMIT) VALUES 2 | ('123456789', 10000, 0); 3 | 4 | COMMIT; -------------------------------------------------------------------------------- /src/main/resources/schema.sql: -------------------------------------------------------------------------------- 1 | CREATE TABLE IF NOT EXISTS debit_card ( 2 | id CHAR(36), 3 | initial_limit DECIMAL(18,2) NOT NULL, 4 | used_limit DECIMAL(18,2) NOT NULL, 5 | PRIMARY KEY (ID) 6 | ); 7 | 8 | CREATE TABLE IF NOT EXISTS money_withdrawal ( 9 | id CHAR(36) PRIMARY KEY, 10 | debit_card_id CHAR(36) NOT NULL, 11 | amount DECIMAL(18,2) NOT NULL 12 | ); 13 | 14 | -------------------------------------------------------------------------------- /src/test/java/com/gonnect/debezium/kafka/bank/account/CdcApplicationTests.java: -------------------------------------------------------------------------------- 1 | package com.gonnect.debezium.kafka.bank.account; 2 | 3 | import org.junit.Test; 4 | import org.junit.runner.RunWith; 5 | import org.springframework.boot.test.context.SpringBootTest; 6 | import org.springframework.test.context.junit4.SpringRunner; 7 | 8 | @RunWith(SpringRunner.class) 9 | @SpringBootTest 10 | public class CdcApplicationTests { 11 | 12 | @Test 13 | public void contextLoads() { 14 | } 15 | 16 | } 17 | 18 | --------------------------------------------------------------------------------