├── assets ├── gh-secrets.png ├── gatling-report.png ├── gh-actions-run.png ├── k8s-active-active-DR-T0.png ├── k8s-active-active-DR-T1.png ├── k8s-active-active-DR-T2.png └── active-active pattern.svg ├── .gitignore ├── perftest ├── src │ └── test │ │ ├── resources │ │ └── application.conf │ │ └── java │ │ └── org │ │ └── github │ │ └── ogomezso │ │ └── perftest │ │ ├── ChuckSaysNoProxySimulation.java │ │ └── ChuckSaysProxySimulation.java └── pom.xml ├── ccloud-resources └── cluster-linking │ ├── create-mirror-topic-west-north.sh │ ├── create-mirror-noproxy-topic-north-west.sh │ ├── create-mirror-noproxy-topic-west-north.sh │ ├── create-mirror-topic-north-west.sh │ ├── cl-west-north.properties │ ├── cl-north-west.properties │ ├── cluster-linking-north-west.sh │ └── cluster-linking-west-north.sh ├── java-consumer ├── src │ └── main │ │ └── java │ │ └── org │ │ └── github │ │ └── ogomezso │ │ └── javaconsumer │ │ ├── infrastructure │ │ └── kafka │ │ │ ├── ConsumerAdapter.java │ │ │ ├── ChuckService.java │ │ │ ├── KafkaConfig.java │ │ │ └── ChuckConsumer.java │ │ └── App.java ├── config │ └── config.properties ├── Dockerfile └── pom.xml ├── java-producer ├── src │ └── main │ │ ├── config │ │ └── config.properties │ │ └── java │ │ └── org │ │ └── github │ │ └── ogomezso │ │ └── javaproducer │ │ ├── domain │ │ ├── ChuckFactPort.java │ │ ├── model │ │ │ └── ChuckFact.java │ │ └── ChuckFactService.java │ │ ├── infrastructure │ │ ├── rest │ │ │ ├── model │ │ │ │ └── ChuckFactResponse.java │ │ │ ├── FactResponseMapper.java │ │ │ └── ChuckController.java │ │ └── kafka │ │ │ ├── ChuckAdapter.java │ │ │ ├── KafkaConfig.java │ │ │ ├── ChuckService.java │ │ │ └── ChuckProducer.java │ │ └── App.java ├── config │ └── config.properties ├── Dockerfile └── pom.xml ├── k8s-resources ├── proxy │ ├── kafka-proxy-configmap.yaml │ ├── java-cloud-producer-configmap.yaml │ ├── java-cloud-consumer-configmap.yaml │ ├── java-cloud-consumer.yaml │ └── java-cloud-producer.yaml └── no-proxy │ ├── java-cloud-producer-noproxy-configmap.yaml │ ├── java-cloud-consumer-noproxy-configmap.yaml │ ├── java-cloud-consumer-noproxy.yaml │ └── java-cloud-producer-noproxy.yaml ├── .github └── workflows │ ├── set-proxy-to-west.yaml │ ├── set-proxy-to-north.yaml │ ├── set-producer-to-west.yaml │ ├── set-consumer-to-west.yaml │ ├── set-consumer-to-north.yaml │ └── set-producer-to-north.yaml ├── local-environment └── kafka-proxy │ └── docker-compose.yaml └── README.md /assets/gh-secrets.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ogomezso/disaster-recovery-playground/HEAD/assets/gh-secrets.png -------------------------------------------------------------------------------- /assets/gatling-report.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ogomezso/disaster-recovery-playground/HEAD/assets/gatling-report.png -------------------------------------------------------------------------------- /assets/gh-actions-run.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ogomezso/disaster-recovery-playground/HEAD/assets/gh-actions-run.png -------------------------------------------------------------------------------- /assets/k8s-active-active-DR-T0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ogomezso/disaster-recovery-playground/HEAD/assets/k8s-active-active-DR-T0.png -------------------------------------------------------------------------------- /assets/k8s-active-active-DR-T1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ogomezso/disaster-recovery-playground/HEAD/assets/k8s-active-active-DR-T1.png -------------------------------------------------------------------------------- /assets/k8s-active-active-DR-T2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ogomezso/disaster-recovery-playground/HEAD/assets/k8s-active-active-DR-T2.png -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .vscode/** 2 | azure-credentials/** 3 | **/target/** 4 | .idea/** 5 | .env 6 | **/west-north.properties 7 | **/north-west.properties 8 | -------------------------------------------------------------------------------- /perftest/src/test/resources/application.conf: -------------------------------------------------------------------------------- 1 | no-proxy { 2 | base-url="http://localhost:8080" #setup your svc url here 3 | } 4 | 5 | proxy { 6 | base-url="http://localhost:8081" 7 | } -------------------------------------------------------------------------------- /ccloud-resources/cluster-linking/create-mirror-topic-west-north.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source ../.env 3 | 4 | confluent kafka mirror create west-$TOPIC --link west-north-clink --environment $ENV --source-topic $TOPIC --cluster $CCLOUD_NORTH_CLUSTERID 5 | -------------------------------------------------------------------------------- /java-consumer/src/main/java/org/github/ogomezso/javaconsumer/infrastructure/kafka/ConsumerAdapter.java: -------------------------------------------------------------------------------- 1 | package org.github.ogomezso.javaconsumer.infrastructure.kafka; 2 | 3 | public interface ConsumerAdapter { 4 | 5 | void pollMessages(); 6 | } 7 | -------------------------------------------------------------------------------- /ccloud-resources/cluster-linking/create-mirror-noproxy-topic-north-west.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source ../.env 3 | 4 | confluent kafka mirror create north-$NP_TOPIC --link north-west-clink --environment $ENV --source-topic $NP_TOPIC --cluster $CCLOUD_WEST_CLUSTERID 5 | -------------------------------------------------------------------------------- /ccloud-resources/cluster-linking/create-mirror-noproxy-topic-west-north.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source ../.env 3 | 4 | confluent kafka mirror create west-$NP_TOPIC --link west-north-clink --environment $ENV --source-topic $NP_TOPIC --cluster $CCLOUD_NORTH_CLUSTERID 5 | -------------------------------------------------------------------------------- /ccloud-resources/cluster-linking/create-mirror-topic-north-west.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source ../.env 3 | 4 | confluent kafka mirror create north-chuck-java-topic --link north-west-clink --environment $ENV --source-topic chuck-java-topic --cluster $CCLOUD_WEST_CLUSTERID 5 | -------------------------------------------------------------------------------- /java-producer/src/main/config/config.properties: -------------------------------------------------------------------------------- 1 | bootstrap.servers=localhost:30001,localhost:30002,localhost:30003 2 | client.id=chuck-java-producer 3 | key.serializer=org.apache.kafka.common.serialization.StringSerializer 4 | value.serializer=org.apache.kafka.common.serialization.StringSerializer -------------------------------------------------------------------------------- /java-producer/src/main/java/org/github/ogomezso/javaproducer/domain/ChuckFactPort.java: -------------------------------------------------------------------------------- 1 | package org.github.ogomezso.javaproducer.domain; 2 | 3 | import org.github.ogomezso.javaproducer.domain.model.ChuckFact; 4 | 5 | public interface ChuckFactPort { 6 | 7 | ChuckFact buildFact(); 8 | } 9 | -------------------------------------------------------------------------------- /java-producer/src/main/java/org/github/ogomezso/javaproducer/domain/model/ChuckFact.java: -------------------------------------------------------------------------------- 1 | package org.github.ogomezso.javaproducer.domain.model; 2 | 3 | import lombok.Builder; 4 | import lombok.Value; 5 | 6 | @Builder 7 | @Value 8 | public class ChuckFact { 9 | String id; 10 | Long timestamp; 11 | String fact; 12 | } 13 | -------------------------------------------------------------------------------- /java-producer/src/main/java/org/github/ogomezso/javaproducer/infrastructure/rest/model/ChuckFactResponse.java: -------------------------------------------------------------------------------- 1 | package org.github.ogomezso.javaproducer.infrastructure.rest.model; 2 | 3 | import lombok.Builder; 4 | import lombok.Data; 5 | 6 | @Builder 7 | @Data 8 | public class ChuckFactResponse { 9 | 10 | private Long timestamp; 11 | private String fact; 12 | } 13 | -------------------------------------------------------------------------------- /ccloud-resources/cluster-linking/cl-west-north.properties: -------------------------------------------------------------------------------- 1 | cluster.link.prefix=west_ 2 | 3 | bootstrap.servers=$CCLOUD_WEST_URL 4 | security.protocol=SASL_SSL 5 | sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='$CCLOUD_WEST_APIKEY' password='$CCLOUD_WEST_PASSWD'; 6 | sasl.mechanism=PLAIN 7 | client.dns.lookup=use_all_dns_ips 8 | session.timeout.ms=45000 9 | 10 | 11 | -------------------------------------------------------------------------------- /ccloud-resources/cluster-linking/cl-north-west.properties: -------------------------------------------------------------------------------- 1 | cluster.link.prefix=north- 2 | 3 | bootstrap.servers="$CCLOUD_NORTH_URL" 4 | security.protocol=SASL_SSL 5 | sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='$CCLOUD_NORTH_APIKEY' password='$CCLOUD_NORTH_PASSWD'; 6 | sasl.mechanism=PLAIN 7 | client.dns.lookup=use_all_dns_ips 8 | session.timeout.ms=45000 9 | 10 | 11 | -------------------------------------------------------------------------------- /java-producer/src/main/java/org/github/ogomezso/javaproducer/infrastructure/kafka/ChuckAdapter.java: -------------------------------------------------------------------------------- 1 | package org.github.ogomezso.javaproducer.infrastructure.kafka; 2 | 3 | import org.github.ogomezso.javaproducer.domain.model.ChuckFact; 4 | 5 | import com.fasterxml.jackson.core.JsonProcessingException; 6 | 7 | public interface ChuckAdapter { 8 | 9 | ChuckFact sendFact(String topic) throws JsonProcessingException; 10 | } 11 | -------------------------------------------------------------------------------- /k8s-resources/proxy/kafka-proxy-configmap.yaml: -------------------------------------------------------------------------------- 1 | kind: ConfigMap 2 | apiVersion: v1 3 | metadata: 4 | namespace: clients 5 | name: kafka-proxy-config 6 | annotations: 7 | reloader.stakater.com/match: "true" 8 | data: 9 | jaas.config: |- 10 | KafkaServer { 11 | org.apache.kafka.common.security.plain.PlainLoginModule required 12 | username="${CCLOUD_APIKEY}" 13 | password="${CCLOUD_PASSWD}" 14 | }; 15 | bootstrap_servers: ${CCLOUD_URL} -------------------------------------------------------------------------------- /java-producer/config/config.properties: -------------------------------------------------------------------------------- 1 | bootstrap.servers= 2 | ssl.endpoint.identification.algorithm=https 3 | security.protocol=SASL_SSL 4 | sasl.mechanism=PLAIN 5 | sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='' password=''; 6 | client.id=chuck-java-producer 7 | topic=chuck-java-topic 8 | key.serializer=org.apache.kafka.common.serialization.StringSerializer 9 | value.serializer=org.apache.kafka.common.serialization.StringSerializer -------------------------------------------------------------------------------- /java-consumer/src/main/java/org/github/ogomezso/javaconsumer/infrastructure/kafka/ChuckService.java: -------------------------------------------------------------------------------- 1 | package org.github.ogomezso.javaconsumer.infrastructure.kafka; 2 | 3 | import java.util.Properties; 4 | 5 | public class ChuckService implements ConsumerAdapter { 6 | 7 | private final ChuckConsumer consumer; 8 | 9 | public ChuckService(Properties config) { 10 | this.consumer = new ChuckConsumer(config); 11 | } 12 | 13 | @Override 14 | public void pollMessages() { 15 | consumer.pollMessage(); 16 | } 17 | 18 | } 19 | -------------------------------------------------------------------------------- /java-producer/src/main/java/org/github/ogomezso/javaproducer/infrastructure/kafka/KafkaConfig.java: -------------------------------------------------------------------------------- 1 | package org.github.ogomezso.javaproducer.infrastructure.kafka; 2 | 3 | import java.util.Properties; 4 | 5 | import org.apache.kafka.clients.producer.KafkaProducer; 6 | 7 | import lombok.RequiredArgsConstructor; 8 | 9 | @RequiredArgsConstructor 10 | public class KafkaConfig { 11 | 12 | static KafkaProducer createKafkaProducer(Properties appConfig) { 13 | return new KafkaProducer<>(appConfig); 14 | } 15 | 16 | } 17 | -------------------------------------------------------------------------------- /k8s-resources/proxy/java-cloud-producer-configmap.yaml: -------------------------------------------------------------------------------- 1 | kind: ConfigMap 2 | apiVersion: v1 3 | metadata: 4 | namespace: clients 5 | name: java-cloud-producer-config 6 | annotations: 7 | reloader.stakater.com/match: "true" 8 | data: 9 | config.properties: |- 10 | bootstrap.servers=localhost:32400 11 | key.serializer=org.apache.kafka.common.serialization.StringSerializer 12 | value.serializer=org.apache.kafka.common.serialization.StringSerializer 13 | client.id=chuck-java-producer 14 | topic=chuck-java-topic 15 | -------------------------------------------------------------------------------- /java-consumer/config/config.properties: -------------------------------------------------------------------------------- 1 | bootstrap.servers= 2 | ssl.endpoint.identification.algorithm=https 3 | security.protocol=SASL_SSL 4 | sasl.mechanism=PLAIN 5 | sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='' password=''; 6 | key.deserializer=org.apache.kafka.common.serialization.StringDeserializer 7 | value.deserializer=org.apache.kafka.common.serialization.StringDeserializer 8 | client.id=chuck-java-consumer 9 | group.id=chuck-java-cg 10 | topic=.*chuck-java-topic -------------------------------------------------------------------------------- /java-producer/src/main/java/org/github/ogomezso/javaproducer/infrastructure/rest/FactResponseMapper.java: -------------------------------------------------------------------------------- 1 | package org.github.ogomezso.javaproducer.infrastructure.rest; 2 | 3 | import org.github.ogomezso.javaproducer.domain.model.ChuckFact; 4 | import org.github.ogomezso.javaproducer.infrastructure.rest.model.ChuckFactResponse; 5 | 6 | class FactResponseMapper { 7 | 8 | ChuckFactResponse toResponse(ChuckFact fact) { 9 | return ChuckFactResponse.builder() 10 | .fact(fact.getFact()) 11 | .timestamp(fact.getTimestamp()) 12 | .build(); 13 | } 14 | } 15 | -------------------------------------------------------------------------------- /k8s-resources/proxy/java-cloud-consumer-configmap.yaml: -------------------------------------------------------------------------------- 1 | kind: ConfigMap 2 | apiVersion: v1 3 | metadata: 4 | namespace: clients 5 | name: java-cloud-consumer-config 6 | annotations: 7 | reloader.stakater.com/match: "true" 8 | data: 9 | config.properties: |- 10 | bootstrap.servers=localhost:32400 11 | key.deserializer=org.apache.kafka.common.serialization.StringDeserializer 12 | value.deserializer=org.apache.kafka.common.serialization.StringDeserializer 13 | client.id=chuck-java-consumer 14 | group.id=chuck-java-cg 15 | topic=.*chuck-java-topic 16 | -------------------------------------------------------------------------------- /ccloud-resources/cluster-linking/cluster-linking-north-west.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source ../.env 3 | 4 | cat <> north-west.properties 5 | cluster.link.prefix=north- 6 | bootstrap.servers=$CCLOUD_NORTH_URL 7 | security.protocol=SASL_SSL 8 | sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='$CCLOUD_NORTH_APIKEY' password='$CCLOUD_NORTH_PASSWD'; 9 | sasl.mechanism=PLAIN 10 | client.dns.lookup=use_all_dns_ips 11 | session.timeout.ms=45000 12 | EOF 13 | 14 | confluent kafka link create north-west-clink \ 15 | --config-file north-west.properties \ 16 | --environment "$ENV" \ 17 | --cluster "$CCLOUD_WEST_CLUSTERID" -------------------------------------------------------------------------------- /ccloud-resources/cluster-linking/cluster-linking-west-north.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source ../.env 3 | 4 | cat <> west-north.properties 5 | cluster.link.prefix=west- 6 | bootstrap.servers=$CCLOUD_WEST_URL 7 | security.protocol=SASL_SSL 8 | sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='$CCLOUD_WEST_APIKEY' password='$CCLOUD_WEST_PASSWD'; 9 | sasl.mechanism=PLAIN 10 | client.dns.lookup=use_all_dns_ips 11 | session.timeout.ms=45000 12 | EOF 13 | 14 | confluent kafka link create west-north-clink \ 15 | --config-file west-north.properties \ 16 | --environment "$ENV" \ 17 | --cluster "$CCLOUD_NORTH_CLUSTERID" 18 | -------------------------------------------------------------------------------- /java-producer/Dockerfile: -------------------------------------------------------------------------------- 1 | ARG BUILD_HOME=/app 2 | 3 | # Builder ############################################ 4 | FROM maven:3.8.5-openjdk-17-slim AS builder 5 | 6 | RUN apt-get update \ 7 | && apt-get install -y wget \ 8 | && rm -rf /var/lib/apt/lists/* 9 | 10 | ENV APP_HOME=$BUILD_HOME 11 | WORKDIR $APP_HOME 12 | 13 | COPY src $APP_HOME/src 14 | COPY pom.xml $APP_HOME 15 | RUN mvn -f $APP_HOME/pom.xml clean package 16 | 17 | # App ############################################ 18 | FROM openjdk:17-slim 19 | 20 | COPY --from=builder $APP_HOME/target/java-producer-jar-with-dependencies.jar app.jar 21 | 22 | EXPOSE 38080 23 | 24 | ENV APP_NAME="java-producer" 25 | 26 | CMD ["java", "-jar", "app.jar","/config/config.properties"] 27 | -------------------------------------------------------------------------------- /java-consumer/Dockerfile: -------------------------------------------------------------------------------- 1 | ARG BUILD_HOME=/app 2 | 3 | # Builder ############################################ 4 | FROM maven:3.8.5-openjdk-17-slim AS builder 5 | 6 | RUN apt-get update \ 7 | && apt-get install -y wget \ 8 | && rm -rf /var/lib/apt/lists/* 9 | 10 | ENV APP_HOME=$BUILD_HOME 11 | WORKDIR $APP_HOME 12 | 13 | COPY src $APP_HOME/src 14 | COPY config $APP_HOME/config 15 | COPY pom.xml $APP_HOME 16 | RUN mvn -f $APP_HOME/pom.xml clean package 17 | 18 | # App ############################################ 19 | FROM openjdk:17-slim 20 | 21 | COPY --from=builder $APP_HOME/target/java-consumer-jar-with-dependencies.jar app.jar 22 | 23 | EXPOSE 38080 24 | 25 | ARG APP_NAME="java-consumer" 26 | 27 | CMD ["java", "-jar", "app.jar","/config/config.properties"] 28 | -------------------------------------------------------------------------------- /java-consumer/src/main/java/org/github/ogomezso/javaconsumer/infrastructure/kafka/KafkaConfig.java: -------------------------------------------------------------------------------- 1 | package org.github.ogomezso.javaconsumer.infrastructure.kafka; 2 | 3 | import java.util.Properties; 4 | import java.util.regex.Pattern; 5 | 6 | import org.apache.kafka.clients.consumer.KafkaConsumer; 7 | 8 | import lombok.RequiredArgsConstructor; 9 | 10 | @RequiredArgsConstructor 11 | public class KafkaConfig { 12 | 13 | static KafkaConsumer createKafkaConsumer(Properties config) { 14 | 15 | final KafkaConsumer consumer = new KafkaConsumer<>(config); 16 | 17 | Pattern topicPattern = Pattern.compile(config.getProperty("topic", "*chuck-java-topic")); 18 | 19 | consumer.subscribe(topicPattern); 20 | 21 | return consumer; 22 | } 23 | } 24 | -------------------------------------------------------------------------------- /java-producer/src/main/java/org/github/ogomezso/javaproducer/domain/ChuckFactService.java: -------------------------------------------------------------------------------- 1 | package org.github.ogomezso.javaproducer.domain; 2 | 3 | import java.sql.Timestamp; 4 | import java.time.Instant; 5 | import java.util.UUID; 6 | 7 | import org.github.ogomezso.javaproducer.domain.model.ChuckFact; 8 | 9 | import com.github.javafaker.Faker; 10 | 11 | import lombok.RequiredArgsConstructor; 12 | 13 | @RequiredArgsConstructor 14 | public class ChuckFactService implements ChuckFactPort { 15 | 16 | private final Faker faker = new Faker(); 17 | 18 | @Override 19 | public ChuckFact buildFact() { 20 | return ChuckFact.builder() 21 | .id(UUID.randomUUID().toString()) 22 | .timestamp(Timestamp.from(Instant.now()).getTime()) 23 | .fact(faker.chuckNorris().fact()) 24 | .build(); 25 | } 26 | } 27 | -------------------------------------------------------------------------------- /k8s-resources/no-proxy/java-cloud-producer-noproxy-configmap.yaml: -------------------------------------------------------------------------------- 1 | kind: ConfigMap 2 | apiVersion: v1 3 | metadata: 4 | namespace: noproxy-clients 5 | name: java-cloud-producer-noproxy-config 6 | annotations: 7 | reloader.stakater.com/match: "true" 8 | data: 9 | config.properties: |- 10 | bootstrap.servers=${CCLOUD_URL} 11 | ssl.endpoint.identification.algorithm=https 12 | security.protocol=SASL_SSL 13 | sasl.mechanism=PLAIN 14 | sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='${CCLOUD_APIKEY}' password='${CCLOUD_PASSWD}'; 15 | client.id=chuck-java-noproxy-producer 16 | key.serializer=org.apache.kafka.common.serialization.StringSerializer 17 | value.serializer=org.apache.kafka.common.serialization.StringSerializer 18 | topic=chuck-java-np-topic 19 | -------------------------------------------------------------------------------- /.github/workflows/set-proxy-to-west.yaml: -------------------------------------------------------------------------------- 1 | name: Point kafka proxy to West Cluster 2 | on: 3 | workflow_dispatch: 4 | 5 | jobs: 6 | build: 7 | runs-on: ubuntu-latest 8 | steps: 9 | - uses: actions/checkout@master 10 | 11 | # Set the target AKS cluster. 12 | - uses: Azure/aks-set-context@v1 13 | with: 14 | creds: '${{ secrets.AZURE_CREDENTIALS }}' 15 | cluster-name: ogomezso-aks 16 | resource-group: ogomezso 17 | - uses: swdotcom/update-and-apply-kubernetes-configs@v1 18 | with: 19 | k8-config-file-paths: k8s-resources/proxy/kafka-proxy-configmap.yaml 20 | replacement-method: defined 21 | env: 22 | CCLOUD_URL: ${{ secrets.CCLOUD_WEST_URL }} 23 | CCLOUD_APIKEY: ${{ secrets.CCLOUD_WEST_APIKEY }} 24 | CCLOUD_PASSWD: ${{ secrets.CCLOUD_WEST_PASSWD }} -------------------------------------------------------------------------------- /.github/workflows/set-proxy-to-north.yaml: -------------------------------------------------------------------------------- 1 | name: Point kafka proxy to North Cluster 2 | on: 3 | workflow_dispatch: 4 | 5 | jobs: 6 | build: 7 | runs-on: ubuntu-latest 8 | steps: 9 | - uses: actions/checkout@master 10 | 11 | # Set the target AKS cluster. 12 | - uses: Azure/aks-set-context@v1 13 | with: 14 | creds: '${{ secrets.AZURE_CREDENTIALS }}' 15 | cluster-name: ogomezso-aks 16 | resource-group: ogomezso 17 | - uses: swdotcom/update-and-apply-kubernetes-configs@v1 18 | with: 19 | k8-config-file-paths: k8s-resources/proxy/kafka-proxy-configmap.yaml 20 | replacement-method: defined 21 | env: 22 | CCLOUD_URL: ${{ secrets.CCLOUD_NORTH_URL }} 23 | CCLOUD_APIKEY: ${{ secrets.CCLOUD_NORTH_APIKEY }} 24 | CCLOUD_PASSWD: ${{ secrets.CCLOUD_NORTH_PASSWD }} 25 | -------------------------------------------------------------------------------- /.github/workflows/set-producer-to-west.yaml: -------------------------------------------------------------------------------- 1 | name: Point producer to West Cluster 2 | on: 3 | workflow_dispatch: 4 | 5 | jobs: 6 | build: 7 | runs-on: ubuntu-latest 8 | steps: 9 | - uses: actions/checkout@master 10 | 11 | # Set the target AKS cluster. 12 | - uses: Azure/aks-set-context@v1 13 | with: 14 | creds: '${{ secrets.AZURE_CREDENTIALS }}' 15 | cluster-name: ogomezso-aks 16 | resource-group: ogomezso 17 | - uses: swdotcom/update-and-apply-kubernetes-configs@v1 18 | with: 19 | k8-config-file-paths: k8s-resources/no-proxy/java-cloud-producer-noproxy-configmap.yaml 20 | replacement-method: defined 21 | env: 22 | CCLOUD_URL: ${{ secrets.CCLOUD_WEST_URL }} 23 | CCLOUD_APIKEY: ${{ secrets.CCLOUD_WEST_APIKEY }} 24 | CCLOUD_PASSWD: ${{ secrets.CCLOUD_WEST_PASSWD }} -------------------------------------------------------------------------------- /k8s-resources/no-proxy/java-cloud-consumer-noproxy-configmap.yaml: -------------------------------------------------------------------------------- 1 | kind: ConfigMap 2 | apiVersion: v1 3 | metadata: 4 | namespace: noproxy-clients 5 | name: java-cloud-consumer-noproxy-config 6 | annotations: 7 | reloader.stakater.com/match: "true" 8 | data: 9 | config.properties: |- 10 | bootstrap.servers=${CCLOUD_URL} 11 | ssl.endpoint.identification.algorithm=https 12 | security.protocol=SASL_SSL 13 | sasl.mechanism=PLAIN 14 | sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='${CCLOUD_APIKEY}' password='${CCLOUD_PASSWD}'; 15 | key.deserializer=org.apache.kafka.common.serialization.StringDeserializer 16 | value.deserializer=org.apache.kafka.common.serialization.StringDeserializer 17 | client.id=chuck-java-noproxy-consumer 18 | group.id=chuck-java-noproxy-cg 19 | topic=.*chuck-java-np-topic 20 | -------------------------------------------------------------------------------- /.github/workflows/set-consumer-to-west.yaml: -------------------------------------------------------------------------------- 1 | name: Point consumer to West Cluster 2 | on: 3 | workflow_dispatch: 4 | 5 | jobs: 6 | build: 7 | runs-on: ubuntu-latest 8 | steps: 9 | - uses: actions/checkout@master 10 | 11 | # Set the target AKS cluster. 12 | - uses: Azure/aks-set-context@v1 13 | with: 14 | creds: '${{ secrets.AZURE_CREDENTIALS }}' 15 | cluster-name: ogomezso-aks 16 | resource-group: ogomezso 17 | - uses: swdotcom/update-and-apply-kubernetes-configs@v1 18 | with: 19 | k8-config-file-paths: k8s-resources/no-proxy/java-cloud-consumer-noproxy-configmap.yaml 20 | replacement-method: defined 21 | env: 22 | CCLOUD_URL: ${{ secrets.CCLOUD_WEST_URL }} 23 | CCLOUD_APIKEY: ${{ secrets.CCLOUD_WEST_APIKEY }} 24 | CCLOUD_PASSWD: ${{ secrets.CCLOUD_WEST_PASSWD }} 25 | -------------------------------------------------------------------------------- /.github/workflows/set-consumer-to-north.yaml: -------------------------------------------------------------------------------- 1 | name: Point consumer to North Cluster 2 | on: 3 | workflow_dispatch: 4 | 5 | jobs: 6 | build: 7 | runs-on: ubuntu-latest 8 | steps: 9 | - uses: actions/checkout@master 10 | 11 | # Set the target AKS cluster. 12 | - uses: Azure/aks-set-context@v1 13 | with: 14 | creds: '${{ secrets.AZURE_CREDENTIALS }}' 15 | cluster-name: ogomezso-aks 16 | resource-group: ogomezso 17 | - uses: swdotcom/update-and-apply-kubernetes-configs@v1 18 | with: 19 | k8-config-file-paths: k8s-resources/no-proxy/java-cloud-consumer-noproxy-configmap.yaml 20 | replacement-method: defined 21 | env: 22 | CCLOUD_URL: ${{ secrets.CCLOUD_NORTH_URL }} 23 | CCLOUD_APIKEY: ${{ secrets.CCLOUD_NORTH_APIKEY }} 24 | CCLOUD_PASSWD: ${{ secrets.CCLOUD_NORTH_PASSWD }} 25 | -------------------------------------------------------------------------------- /.github/workflows/set-producer-to-north.yaml: -------------------------------------------------------------------------------- 1 | name: Point producer to North Cluster 2 | on: 3 | workflow_dispatch: 4 | 5 | jobs: 6 | build: 7 | runs-on: ubuntu-latest 8 | steps: 9 | - uses: actions/checkout@master 10 | 11 | # Set the target AKS cluster. 12 | - uses: Azure/aks-set-context@v1 13 | with: 14 | creds: '${{ secrets.AZURE_CREDENTIALS }}' 15 | cluster-name: ogomezso-aks 16 | resource-group: ogomezso 17 | - uses: swdotcom/update-and-apply-kubernetes-configs@v1 18 | with: 19 | k8-config-file-paths: k8s-resources/no-proxy/java-cloud-producer-noproxy-configmap.yaml 20 | replacement-method: defined 21 | env: 22 | CCLOUD_URL: ${{ secrets.CCLOUD_NORTH_URL }} 23 | CCLOUD_APIKEY: ${{ secrets.CCLOUD_NORTH_APIKEY }} 24 | CCLOUD_PASSWD: ${{ secrets.CCLOUD_NORTH_PASSWD }} 25 | -------------------------------------------------------------------------------- /java-consumer/src/main/java/org/github/ogomezso/javaconsumer/App.java: -------------------------------------------------------------------------------- 1 | package org.github.ogomezso.javaconsumer; 2 | 3 | import java.io.FileInputStream; 4 | import java.io.FileNotFoundException; 5 | import java.io.IOException; 6 | import java.io.InputStream; 7 | import java.net.URISyntaxException; 8 | import java.util.Properties; 9 | 10 | import org.github.ogomezso.javaconsumer.infrastructure.kafka.ChuckService; 11 | import org.github.ogomezso.javaconsumer.infrastructure.kafka.ConsumerAdapter; 12 | 13 | public class App { 14 | 15 | public static void main(String[] args) { 16 | 17 | try (InputStream input = new FileInputStream(args[0])) { 18 | Properties config = new Properties(); 19 | config.load(input); 20 | ConsumerAdapter chuckAdapter = new ChuckService(config); 21 | chuckAdapter.pollMessages(); 22 | } catch (IOException ex) { 23 | ex.printStackTrace(); 24 | } 25 | } 26 | } 27 | -------------------------------------------------------------------------------- /java-consumer/src/main/java/org/github/ogomezso/javaconsumer/infrastructure/kafka/ChuckConsumer.java: -------------------------------------------------------------------------------- 1 | package org.github.ogomezso.javaconsumer.infrastructure.kafka; 2 | 3 | import java.time.Duration; 4 | import java.util.Properties; 5 | 6 | import org.apache.kafka.clients.consumer.ConsumerRecords; 7 | import org.apache.kafka.clients.consumer.KafkaConsumer; 8 | 9 | public class ChuckConsumer { 10 | 11 | private final KafkaConsumer plainConsumer; 12 | 13 | public ChuckConsumer(Properties config) { 14 | this.plainConsumer = KafkaConfig.createKafkaConsumer(config); 15 | } 16 | 17 | public void pollMessage() { 18 | while (true) { 19 | final ConsumerRecords consumerRecords = plainConsumer.poll(Duration.ofMillis(500)); 20 | 21 | consumerRecords.forEach(record -> { 22 | System.out.printf("Consumer Record:(%s, %s, %d, %d), from Topic: %s\n", 23 | record.key(), record.value(), 24 | record.partition(), record.offset(), record.topic()); 25 | }); 26 | } 27 | } 28 | 29 | } 30 | -------------------------------------------------------------------------------- /java-producer/src/main/java/org/github/ogomezso/javaproducer/infrastructure/rest/ChuckController.java: -------------------------------------------------------------------------------- 1 | package org.github.ogomezso.javaproducer.infrastructure.rest; 2 | 3 | import java.util.Properties; 4 | 5 | import org.github.ogomezso.javaproducer.infrastructure.kafka.ChuckAdapter; 6 | import org.github.ogomezso.javaproducer.infrastructure.kafka.ChuckService; 7 | 8 | import com.fasterxml.jackson.core.JsonProcessingException; 9 | import com.fasterxml.jackson.databind.ObjectMapper; 10 | 11 | public class ChuckController { 12 | 13 | private final FactResponseMapper mapper = new FactResponseMapper(); 14 | private final ChuckAdapter adapter; 15 | 16 | private final String topic; 17 | 18 | private final ObjectMapper objectMapper = new ObjectMapper(); 19 | 20 | public ChuckController(Properties appConfig) { 21 | this.adapter = new ChuckService(appConfig); 22 | this.topic = appConfig.getProperty("topic", "chuck-java-topic"); 23 | } 24 | 25 | public String sendFact() throws JsonProcessingException { 26 | return objectMapper.writeValueAsString(mapper.toResponse(adapter.sendFact(topic))); 27 | } 28 | } -------------------------------------------------------------------------------- /java-producer/src/main/java/org/github/ogomezso/javaproducer/App.java: -------------------------------------------------------------------------------- 1 | package org.github.ogomezso.javaproducer; 2 | 3 | import static spark.Spark.port; 4 | import static spark.Spark.post; 5 | 6 | import java.io.FileInputStream; 7 | import java.io.FileNotFoundException; 8 | import java.io.IOException; 9 | import java.io.InputStream; 10 | import java.net.URISyntaxException; 11 | import java.util.Properties; 12 | 13 | import org.github.ogomezso.javaproducer.infrastructure.rest.ChuckController; 14 | 15 | import lombok.extern.slf4j.Slf4j; 16 | 17 | @Slf4j 18 | public class App { 19 | 20 | public static void main(String[] args) { 21 | 22 | try (InputStream input = new FileInputStream(args[0])) { 23 | Properties config = new Properties(); 24 | config.load(input); 25 | ChuckController controller = new ChuckController(config); 26 | port(8080); 27 | post("/chuck-says", (req, res) -> { 28 | log.info("Plain Json request received"); 29 | res.header("Content-Type", "application/json"); 30 | return controller.sendFact(); 31 | }); 32 | } catch (IOException ex) { 33 | ex.printStackTrace(); 34 | } 35 | 36 | } 37 | } 38 | 39 | -------------------------------------------------------------------------------- /java-producer/src/main/java/org/github/ogomezso/javaproducer/infrastructure/kafka/ChuckService.java: -------------------------------------------------------------------------------- 1 | package org.github.ogomezso.javaproducer.infrastructure.kafka; 2 | 3 | import java.util.Properties; 4 | 5 | import org.github.ogomezso.javaproducer.domain.ChuckFactPort; 6 | import org.github.ogomezso.javaproducer.domain.ChuckFactService; 7 | import org.github.ogomezso.javaproducer.domain.model.ChuckFact; 8 | 9 | import com.fasterxml.jackson.core.JsonProcessingException; 10 | import com.fasterxml.jackson.databind.ObjectMapper; 11 | 12 | public class ChuckService implements ChuckAdapter { 13 | 14 | private final ChuckFactPort chuckFactPort = new ChuckFactService(); 15 | private final ChuckProducer producer; 16 | private final ObjectMapper objectMapper = new ObjectMapper(); 17 | 18 | public ChuckService(Properties appConfig) { 19 | this.producer = new ChuckProducer(appConfig); 20 | } 21 | 22 | @Override 23 | public ChuckFact sendFact(String topic) throws JsonProcessingException { 24 | ChuckFact fact = chuckFactPort.buildFact(); 25 | String message = objectMapper.writeValueAsString(fact); 26 | producer.produceJsonMessage(topic,message); 27 | return fact; 28 | } 29 | } 30 | -------------------------------------------------------------------------------- /java-producer/src/main/java/org/github/ogomezso/javaproducer/infrastructure/kafka/ChuckProducer.java: -------------------------------------------------------------------------------- 1 | package org.github.ogomezso.javaproducer.infrastructure.kafka; 2 | 3 | import java.util.Properties; 4 | 5 | import org.apache.kafka.clients.producer.KafkaProducer; 6 | import org.apache.kafka.clients.producer.ProducerRecord; 7 | 8 | 9 | import lombok.extern.slf4j.Slf4j; 10 | 11 | @Slf4j 12 | public class ChuckProducer { 13 | 14 | private final KafkaProducer plainProducer; 15 | 16 | public ChuckProducer(Properties appConfig) { 17 | this.plainProducer = KafkaConfig.createKafkaProducer(appConfig); 18 | } 19 | 20 | public void produceJsonMessage(String topic, String msg) { 21 | 22 | ProducerRecord record = new ProducerRecord<>(topic, msg); 23 | 24 | plainProducer.send(record, (recordMetadata, exception) -> { 25 | if (exception == null) { 26 | log.info("Record written to offset " + 27 | recordMetadata.offset() + " timestamp " + 28 | recordMetadata.timestamp()); 29 | } else { 30 | log.error("An error occurred"); 31 | exception.printStackTrace(System.err); 32 | } 33 | }); 34 | } 35 | } 36 | -------------------------------------------------------------------------------- /perftest/src/test/java/org/github/ogomezso/perftest/ChuckSaysNoProxySimulation.java: -------------------------------------------------------------------------------- 1 | package org.github.ogomezso.perftest; 2 | 3 | 4 | import static io.gatling.javaapi.core.CoreDsl.exec; 5 | import static io.gatling.javaapi.core.CoreDsl.scenario; 6 | import static io.gatling.javaapi.core.OpenInjectionStep.atOnceUsers; 7 | import static io.gatling.javaapi.http.HttpDsl.http; 8 | 9 | import com.typesafe.config.Config; 10 | import com.typesafe.config.ConfigFactory; 11 | 12 | import io.gatling.javaapi.core.ScenarioBuilder; 13 | import io.gatling.javaapi.core.Simulation; 14 | import io.gatling.javaapi.http.HttpProtocolBuilder; 15 | 16 | public class ChuckSaysNoProxySimulation extends Simulation { 17 | 18 | Config config = ConfigFactory.load(); 19 | private final HttpProtocolBuilder httpProtocol = http 20 | .baseUrl(config.getString("no-proxy.base-url")) 21 | .inferHtmlResources() 22 | .acceptHeader("*/*"); 23 | 24 | private final ScenarioBuilder scn = scenario("ChuckSaysNoProxyProducer").repeat(1000).on( 25 | exec( 26 | http("Chuck Says with proxy") 27 | .post("/chuck-says") 28 | ) 29 | ); 30 | 31 | { 32 | setUp(scn.injectOpen(atOnceUsers(10))).protocols(httpProtocol); 33 | } 34 | } 35 | -------------------------------------------------------------------------------- /perftest/src/test/java/org/github/ogomezso/perftest/ChuckSaysProxySimulation.java: -------------------------------------------------------------------------------- 1 | package org.github.ogomezso.perftest; 2 | 3 | 4 | import static io.gatling.javaapi.core.CoreDsl.exec; 5 | import static io.gatling.javaapi.core.CoreDsl.scenario; 6 | import static io.gatling.javaapi.core.OpenInjectionStep.atOnceUsers; 7 | import static io.gatling.javaapi.http.HttpDsl.http; 8 | 9 | import com.typesafe.config.Config; 10 | import com.typesafe.config.ConfigFactory; 11 | 12 | import io.gatling.javaapi.core.ScenarioBuilder; 13 | import io.gatling.javaapi.core.Simulation; 14 | import io.gatling.javaapi.http.HttpProtocolBuilder; 15 | 16 | public class ChuckSaysProxySimulation extends Simulation { 17 | 18 | Config config = ConfigFactory.load(); 19 | private final HttpProtocolBuilder httpProxyProtocol = http 20 | .baseUrl(config.getString("proxy.base-url")) 21 | .inferHtmlResources() 22 | .acceptHeader("*/*"); 23 | 24 | private final ScenarioBuilder scn = scenario("ChuckSaysProxyProducer").repeat(1000).on( 25 | exec( 26 | http("Chuck Says with proxy") 27 | .post("/chuck-says") 28 | ) 29 | ); 30 | 31 | { 32 | setUp(scn.injectOpen(atOnceUsers(10))).protocols(httpProxyProtocol); 33 | } 34 | } 35 | -------------------------------------------------------------------------------- /perftest/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 5 | 4.0.0 6 | 7 | org.github.ogomezso.perftest 8 | perftest 9 | 1.0-SNAPSHOT 10 | 11 | 12 | 11 13 | 11 14 | UTF-8 15 | 16 | 17 | 18 | io.gatling.highcharts 19 | gatling-charts-highcharts 20 | 3.8.4 21 | 22 | 23 | 24 | 25 | 26 | 27 | io.gatling 28 | gatling-maven-plugin 29 | 4.2.7 30 | 31 | true 32 | 33 | 34 | 35 | 36 | -------------------------------------------------------------------------------- /k8s-resources/no-proxy/java-cloud-consumer-noproxy.yaml: -------------------------------------------------------------------------------- 1 | kind: Role 2 | apiVersion: rbac.authorization.k8s.io/v1 3 | metadata: 4 | namespace: noproxy-clients 5 | name: namespace-reader 6 | rules: 7 | - apiGroups: ["", "extensions", "apps"] 8 | resources: ["configmaps", "pods", "services", "endpoints", "secrets"] 9 | verbs: ["get", "list", "watch"] 10 | 11 | --- 12 | kind: RoleBinding 13 | apiVersion: rbac.authorization.k8s.io/v1 14 | metadata: 15 | name: namespace-reader-binding 16 | namespace: noproxy-clients 17 | subjects: 18 | - kind: ServiceAccount 19 | name: default 20 | apiGroup: "" 21 | roleRef: 22 | kind: Role 23 | name: namespace-reader 24 | apiGroup: "" 25 | --- 26 | apiVersion: apps/v1 27 | kind: Deployment 28 | metadata: 29 | creationTimestamp: null 30 | labels: 31 | app: java-cloud-consumer-noproxy 32 | name: java-cloud-consumer-noproxy 33 | namespace: noproxy-clients 34 | annotations: 35 | configmap.reloader.stakater.com/reload: "java-cloud-consumer-noproxy-config" 36 | spec: 37 | replicas: 1 38 | selector: 39 | matchLabels: 40 | app: java-cloud-consumer-noproxy 41 | strategy: {} 42 | template: 43 | metadata: 44 | creationTimestamp: null 45 | labels: 46 | app: java-cloud-consumer-noproxy 47 | spec: 48 | containers: 49 | - name: java-cloud-consumer-noproxy 50 | image: ogomezso/java-cloud-consumer:0.0.1 51 | resources: {} 52 | volumeMounts: 53 | - name: app-config-volume 54 | mountPath: /config 55 | volumes: 56 | - name: app-config-volume 57 | configMap: 58 | name: java-cloud-consumer-noproxy-config 59 | 60 | -------------------------------------------------------------------------------- /k8s-resources/no-proxy/java-cloud-producer-noproxy.yaml: -------------------------------------------------------------------------------- 1 | kind: Role 2 | apiVersion: rbac.authorization.k8s.io/v1 3 | metadata: 4 | namespace: noproxy-clients 5 | name: namespace-reader 6 | rules: 7 | - apiGroups: ["", "extensions", "apps"] 8 | resources: ["configmaps", "pods", "services", "endpoints", "secrets"] 9 | verbs: ["get", "list", "watch"] 10 | 11 | --- 12 | kind: RoleBinding 13 | apiVersion: rbac.authorization.k8s.io/v1 14 | metadata: 15 | name: namespace-reader-binding 16 | namespace: noproxy-clients 17 | subjects: 18 | - kind: ServiceAccount 19 | name: default 20 | apiGroup: "" 21 | roleRef: 22 | kind: Role 23 | name: namespace-reader 24 | apiGroup: "" 25 | --- 26 | apiVersion: apps/v1 27 | kind: Deployment 28 | metadata: 29 | creationTimestamp: null 30 | labels: 31 | app: java-cloud-producer-noproxy 32 | name: java-cloud-producer-noproxy 33 | namespace: noproxy-clients 34 | annotations: 35 | configmap.reloader.stakater.com/reload: "java-cloud-producer-noproxy-config" 36 | spec: 37 | replicas: 1 38 | selector: 39 | matchLabels: 40 | app: java-cloud-producer-noproxy 41 | strategy: {} 42 | template: 43 | metadata: 44 | creationTimestamp: null 45 | labels: 46 | app: java-cloud-producer-noproxy 47 | spec: 48 | containers: 49 | - name: java-cloud-producer-noproxy 50 | image: ogomezso/java-cloud-producer:0.0.2 51 | resources: {} 52 | volumeMounts: 53 | - name: app-config-volume 54 | mountPath: /config 55 | volumes: 56 | - name: app-config-volume 57 | configMap: 58 | name: java-cloud-producer-noproxy-config 59 | --- 60 | apiVersion: v1 61 | kind: Service 62 | metadata: 63 | creationTimestamp: null 64 | labels: 65 | app: java-cloud-producer-noproxy 66 | name: java-cloud-producer-noproxy 67 | namespace: noproxy-clients 68 | spec: 69 | type: LoadBalancer 70 | ports: 71 | - port: 8080 72 | protocol: TCP 73 | targetPort: 8080 74 | selector: 75 | app: java-cloud-producer-noproxy 76 | status: 77 | loadBalancer: {} 78 | -------------------------------------------------------------------------------- /k8s-resources/proxy/java-cloud-consumer.yaml: -------------------------------------------------------------------------------- 1 | kind: Role 2 | apiVersion: rbac.authorization.k8s.io/v1 3 | metadata: 4 | namespace: clients 5 | name: namespace-reader 6 | rules: 7 | - apiGroups: ["", "extensions", "apps"] 8 | resources: ["configmaps", "pods", "services", "endpoints", "secrets"] 9 | verbs: ["get", "list", "watch"] 10 | 11 | --- 12 | kind: RoleBinding 13 | apiVersion: rbac.authorization.k8s.io/v1 14 | metadata: 15 | name: namespace-reader-binding 16 | namespace: clients 17 | subjects: 18 | - kind: ServiceAccount 19 | name: default 20 | apiGroup: "" 21 | roleRef: 22 | kind: Role 23 | name: namespace-reader 24 | apiGroup: "" 25 | --- 26 | apiVersion: apps/v1 27 | kind: Deployment 28 | metadata: 29 | creationTimestamp: null 30 | annotations: 31 | configmap.reloader.stakater.com/reload: "java-cloud-consumer-config,kafka-proxy-config" 32 | labels: 33 | app: java-cloud-consumer 34 | name: java-cloud-consumer 35 | namespace: clients 36 | spec: 37 | replicas: 1 38 | selector: 39 | matchLabels: 40 | app: java-cloud-consumer 41 | strategy: {} 42 | template: 43 | metadata: 44 | creationTimestamp: null 45 | labels: 46 | app: java-cloud-consumer 47 | spec: 48 | containers: 49 | - name: java-cloud-consumer 50 | image: ogomezso/java-cloud-consumer:0.0.1 51 | resources: {} 52 | volumeMounts: 53 | - name: app-config-volume 54 | mountPath: /config 55 | - name: kafka-proxy 56 | image: grepplabs/kafka-proxy:latest 57 | args: 58 | - server 59 | - --log-format=json 60 | - --bootstrap-server-mapping=$(BOOTSTRAP_SERVERS),127.0.0.1:32400 61 | - '--tls-enable' 62 | - '--tls-insecure-skip-verify' 63 | - '--sasl-enable' 64 | - '--sasl-jaas-config-file=/config/jaas.config' 65 | env: 66 | - name: BOOTSTRAP_SERVERS 67 | valueFrom: 68 | configMapKeyRef: 69 | name: kafka-proxy-config 70 | key: bootstrap_servers 71 | volumeMounts: 72 | - name: kafka-proxy-config-volume 73 | mountPath: "/config" 74 | ports: 75 | - name: metrics 76 | containerPort: 9080 77 | livenessProbe: 78 | httpGet: 79 | path: /health 80 | port: 9080 81 | initialDelaySeconds: 5 82 | periodSeconds: 3 83 | readinessProbe: 84 | httpGet: 85 | path: /health 86 | port: 9080 87 | initialDelaySeconds: 5 88 | periodSeconds: 10 89 | timeoutSeconds: 5 90 | successThreshold: 2 91 | failureThreshold: 5 92 | volumes: 93 | - name: app-config-volume 94 | configMap: 95 | name: java-cloud-consumer-config 96 | - name: kafka-proxy-config-volume 97 | configMap: 98 | name: kafka-proxy-config 99 | -------------------------------------------------------------------------------- /k8s-resources/proxy/java-cloud-producer.yaml: -------------------------------------------------------------------------------- 1 | kind: Role 2 | apiVersion: rbac.authorization.k8s.io/v1 3 | metadata: 4 | namespace: clients 5 | name: namespace-reader 6 | rules: 7 | - apiGroups: ["", "extensions", "apps"] 8 | resources: ["configmaps", "pods", "services", "endpoints", "secrets"] 9 | verbs: ["get", "list", "watch"] 10 | 11 | --- 12 | kind: RoleBinding 13 | apiVersion: rbac.authorization.k8s.io/v1 14 | metadata: 15 | name: namespace-reader-binding 16 | namespace: clients 17 | subjects: 18 | - kind: ServiceAccount 19 | name: default 20 | apiGroup: "" 21 | roleRef: 22 | kind: Role 23 | name: namespace-reader 24 | apiGroup: "" 25 | --- 26 | apiVersion: apps/v1 27 | kind: Deployment 28 | metadata: 29 | creationTimestamp: null 30 | annotations: 31 | configmap.reloader.stakater.com/reload: "java-cloud-producer-config,kafka-proxy-config" 32 | labels: 33 | app: java-cloud-producer 34 | name: java-cloud-producer 35 | namespace: clients 36 | spec: 37 | replicas: 1 38 | selector: 39 | matchLabels: 40 | app: java-cloud-producer 41 | strategy: {} 42 | template: 43 | metadata: 44 | creationTimestamp: null 45 | labels: 46 | app: java-cloud-producer 47 | spec: 48 | containers: 49 | - name: java-cloud-producer 50 | image: ogomezso/java-cloud-producer:0.0.4 51 | resources: {} 52 | volumeMounts: 53 | - name: app-config-volume 54 | mountPath: /config 55 | - name: kafka-proxy 56 | image: grepplabs/kafka-proxy:latest 57 | args: 58 | - server 59 | - --log-format=json 60 | - --bootstrap-server-mapping=$(BOOTSTRAP_SERVERS),127.0.0.1:32400 61 | - '--tls-enable' 62 | - '--tls-insecure-skip-verify' 63 | - '--sasl-enable' 64 | - '--sasl-jaas-config-file=/config/jaas.config' 65 | env: 66 | - name: BOOTSTRAP_SERVERS 67 | valueFrom: 68 | configMapKeyRef: 69 | name: kafka-proxy-config 70 | key: bootstrap_servers 71 | volumeMounts: 72 | - name: kafka-proxy-config-volume 73 | mountPath: "/config" 74 | ports: 75 | - name: metrics 76 | containerPort: 9080 77 | livenessProbe: 78 | httpGet: 79 | path: /health 80 | port: 9080 81 | initialDelaySeconds: 5 82 | periodSeconds: 3 83 | readinessProbe: 84 | httpGet: 85 | path: /health 86 | port: 9080 87 | initialDelaySeconds: 5 88 | periodSeconds: 10 89 | timeoutSeconds: 5 90 | successThreshold: 2 91 | failureThreshold: 5 92 | volumes: 93 | - name: app-config-volume 94 | configMap: 95 | name: java-cloud-producer-config 96 | - name: kafka-proxy-config-volume 97 | configMap: 98 | name: kafka-proxy-config 99 | --- 100 | apiVersion: v1 101 | kind: Service 102 | metadata: 103 | creationTimestamp: null 104 | labels: 105 | app: java-cloud-producer 106 | name: java-cloud-producer 107 | namespace: clients 108 | spec: 109 | type: LoadBalancer 110 | ports: 111 | - port: 8080 112 | protocol: TCP 113 | targetPort: 8080 114 | selector: 115 | app: java-cloud-producer 116 | status: 117 | loadBalancer: {} 118 | -------------------------------------------------------------------------------- /java-consumer/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4.0.0 4 | 5 | otlp.example 6 | java-consumer 7 | 1.0-SNAPSHOT 8 | 9 | 10 | confluent 11 | https://packages.confluent.io/maven/ 12 | 13 | 14 | 15 | 17 16 | 17 17 | 3.3.1 18 | java-consumer 19 | 20 | 21 | 22 | org.projectlombok 23 | lombok 24 | 1.18.24 25 | true 26 | 27 | 28 | org.slf4j 29 | slf4j-simple 30 | 2.0.4 31 | 32 | 33 | com.github.javafaker 34 | javafaker 35 | 1.0.2 36 | 37 | 38 | com.fasterxml.jackson.core 39 | jackson-databind 40 | 2.14.0 41 | 42 | 43 | kafka-clients 44 | org.apache.kafka 45 | ${kafka.version} 46 | 47 | 48 | 49 | 50 | 51 | org.apache.maven.plugins 52 | maven-jar-plugin 53 | 2.4 54 | 55 | ${finalName} 56 | 57 | 58 | true 59 | org.github.ogomezso.javaconsumer.App 60 | dependency-jars/ 61 | 62 | 63 | 64 | 65 | 66 | org.apache.maven.plugins 67 | maven-compiler-plugin 68 | 3.8.1 69 | 70 | 17 71 | 72 | 73 | 74 | org.apache.maven.plugins 75 | maven-assembly-plugin 76 | 77 | 78 | 79 | attached 80 | 81 | package 82 | 83 | ${finalName} 84 | 85 | jar-with-dependencies 86 | 87 | 88 | 89 | org.github.ogomezso.javaconsumer.App 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | -------------------------------------------------------------------------------- /java-producer/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4.0.0 4 | 5 | org.example 6 | java-producer 7 | 1.0-SNAPSHOT 8 | 9 | 10 | confluent 11 | https://packages.confluent.io/maven/ 12 | 13 | 14 | 15 | 17 16 | 17 17 | 3.3.1 18 | java-producer 19 | 20 | 21 | 22 | com.sparkjava 23 | spark-core 24 | 2.9.4 25 | 26 | 27 | org.projectlombok 28 | lombok 29 | 1.18.24 30 | true 31 | 32 | 33 | org.slf4j 34 | slf4j-simple 35 | 2.0.4 36 | 37 | 38 | com.github.javafaker 39 | javafaker 40 | 1.0.2 41 | 42 | 43 | com.fasterxml.jackson.core 44 | jackson-databind 45 | 2.14.0 46 | 47 | 48 | kafka-clients 49 | org.apache.kafka 50 | ${kafka.version} 51 | 52 | 53 | org.junit.jupiter 54 | junit-jupiter-api 55 | 5.9.0 56 | test 57 | 58 | 59 | 60 | 61 | 62 | org.slf4j 63 | slf4j-api 64 | 2.0.4 65 | 66 | 67 | 68 | 69 | 70 | 71 | org.apache.maven.plugins 72 | maven-jar-plugin 73 | 2.4 74 | 75 | ${finalName} 76 | 77 | 78 | true 79 | org.github.ogomezso.javaproducer.App 80 | dependency-jars/ 81 | 82 | 83 | 84 | 85 | 86 | org.apache.maven.plugins 87 | maven-compiler-plugin 88 | 3.8.1 89 | 90 | 17 91 | 92 | 93 | 94 | org.apache.maven.plugins 95 | maven-assembly-plugin 96 | 97 | 98 | 99 | attached 100 | 101 | package 102 | 103 | ${finalName} 104 | 105 | jar-with-dependencies 106 | 107 | 108 | 109 | org.github.ogomezso.javaproducer.App 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | -------------------------------------------------------------------------------- /local-environment/kafka-proxy/docker-compose.yaml: -------------------------------------------------------------------------------- 1 | version: '3.7' 2 | 3 | services: 4 | 5 | kafka-proxy: 6 | image: grepplabs/kafka-proxy:v0.3.1-all 7 | ports: 8 | - "30001-30003:30001-30003" 9 | command: server --bootstrap-server-mapping=dc1_broker1:19092,0.0.0.0:30001,localhost:30001 --bootstrap-server-mapping=dc1_broker2:19092,0.0.0.0:30002,localhost:30002 --bootstrap-server-mapping=dc1_broker3:19092,0.0.0.0:30003,localhost:30003 10 | depends_on: 11 | - dc1_broker 12 | 13 | dc1_zookeeper: 14 | image: confluentinc/cp-zookeeper:7.2.0 15 | ports: 16 | - "12181:2181" 17 | - "1999:1999" 18 | environment: 19 | ZOOKEEPER_CLIENT_PORT: 12181 20 | ZOOKEEPER_TICK_TIME: 2000 21 | KAFKA_JMX_PORT: 1999 22 | KAFKA_JMX_HOSTNAME: dc1_zookeeper 23 | 24 | dc1_broker1: 25 | image: confluentinc/cp-server:7.2.0 26 | depends_on: 27 | - dc1_zookeeper 28 | ports: 29 | - "19101:9101" 30 | environment: 31 | KAFKA_BROKER_ID: 1 32 | KAFKA_ZOOKEEPER_CONNECT: dc1_zookeeper:12181 33 | KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT 34 | KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://dc1_broker1:19092 35 | KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter 36 | KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3 37 | KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 38 | KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 3 39 | KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 3 40 | KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 41 | KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3 42 | KAFKA_JMX_PORT: 9101 43 | KAFKA_JMX_HOSTNAME: dc1_broker1 44 | KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://dc1_schema-registry:8081 45 | CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: dc1_broker1:19092 46 | CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 3 47 | CONFLUENT_METRICS_ENABLE: 'true' 48 | CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous' 49 | 50 | dc1_broker2: 51 | image: confluentinc/cp-server:7.2.0 52 | depends_on: 53 | - dc1_zookeeper 54 | ports: 55 | - "19102:9101" 56 | environment: 57 | KAFKA_BROKER_ID: 2 58 | KAFKA_ZOOKEEPER_CONNECT: dc1_zookeeper:12181 59 | KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT 60 | KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://dc1_broker2:19092 61 | KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter 62 | KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3 63 | KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 64 | KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 3 65 | KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 3 66 | KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 67 | KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3 68 | KAFKA_JMX_PORT: 9101 69 | KAFKA_JMX_HOSTNAME: dc1_broker2 70 | KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://dc1_schema-registry:8081 71 | CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: dc1_broker2:19092 72 | CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 3 73 | CONFLUENT_METRICS_ENABLE: 'true' 74 | CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous' 75 | 76 | dc1_broker3: 77 | image: confluentinc/cp-server:7.2.0 78 | depends_on: 79 | - dc1_zookeeper 80 | ports: 81 | - "19103:9101" 82 | environment: 83 | KAFKA_BROKER_ID: 3 84 | KAFKA_ZOOKEEPER_CONNECT: dc1_zookeeper:12181 85 | KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT 86 | KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://dc1_broker3:19092 87 | KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter 88 | KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3 89 | KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 90 | KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 3 91 | KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 3 92 | KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 93 | KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3 94 | KAFKA_JMX_PORT: 9101 95 | KAFKA_JMX_HOSTNAME: dc1_broker3 96 | KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://dc1_schema-registry:8081 97 | CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: dc1_broker3:19092 98 | CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 3 99 | CONFLUENT_METRICS_ENABLE: 'true' 100 | CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous' 101 | 102 | dc1_schema-registry: 103 | image: confluentinc/cp-schema-registry:7.2.0 104 | depends_on: 105 | - dc1_broker1 106 | - dc1_broker2 107 | - dc1_broker3 108 | ports: 109 | - "8081:8081" 110 | environment: 111 | SCHEMA_REGISTRY_HOST_NAME: dc1_schema-registry 112 | SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: dc1_broker1:19092,dc1_broker2:19092,dc1_broker3:19092 113 | SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081 114 | 115 | dc1_control-center: 116 | image: confluentinc/cp-enterprise-control-center:7.2.0 117 | depends_on: 118 | - dc1_broker1 119 | - dc1_schema-registry 120 | ports: 121 | - "9021:9021" 122 | environment: 123 | CONTROL_CENTER_BOOTSTRAP_SERVERS: dc1_broker1:19092 124 | CONTROL_CENTER_SCHEMA_REGISTRY_URL: http://dc1_schema-registry:8081 125 | CONTROL_CENTER_REPLICATION_FACTOR: 3 126 | CONTROL_CENTER_INTERNAL_TOPICS_PARTITIONS: 3 127 | CONTROL_CENTER_MONITORING_INTERCEPTOR_TOPIC_PARTITIONS: 3 128 | CONFLUENT_METRICS_TOPIC_REPLICATION: 3 129 | PORT: 9021 130 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Disaster Recovery Playground 2 | 3 | ## Project Goal 4 | 5 | Reproduce different architectural patterns for Confluent Cluster Disaster Recovery and their failover and failback procedures. 6 | 7 | ## Active-Active Platform over Confluent Cloud with Clients hosted on K8s Cluster with automatic restart on config changes 8 | 9 | ## Motivation 10 | 11 | The main pain on all DR failback/failover strategy is the actual need to reconnection on all the clients to be able to reload metadata from the new cluster, how we can know which and where are this clients?, on distributed architecture patterns this is not always possible. 12 | 13 | On this scenario we tried to solve this problem taking advantage of kubernetes features regarging centraized configuration. 14 | ### Solution Design 15 | 16 | **Note**: On this example the producer and consumer are connected to one DC at a time but be aware that this scenario supports clients producing and consuming simultaneously on both DCs 17 | 18 | On T0 we good our "origin cluster" up and running 19 | 20 | ![T0](./assets/k8s-active-active-DR-T0.png) 21 | 22 | On T1 there is a lost of service on CCloud DC1 23 | 24 | ![T1](./assets/k8s-active-active-DR-T1.png) 25 | 26 | On T2 we change the ConfigMap containing the connection info and Realoader trigger a rolling update on all the clients that observes that CM 27 | ![T2](./assets/k8s-active-active-DR-T2.png) 28 | 29 | 30 | ### Scenario Topology 31 | 32 | ![Active-Active Patter](./assets/active-active%20pattern.svg) 33 | 34 | 1. Two Confluent Cloud Clusters following the active-active DR pattern: 35 | Topic Topology Considerations: 36 | a. Topics are created of both DC 37 | b. Topics are replicated to the opposite DC, prefixed with their DC id 38 | Client Considerations: 39 | a. On this pattern clients can produce and consume from both DCS at the same time. 40 | b. Producers only produce on the non-prefixed topics 41 | c. Consumers will consume from both prefixed and non-prefixed topics allowing you to consume data produced on both DC 42 | 43 | 2. Kubernetes Cluster deployed (connectivity to CCloud cluster required): 44 | This K8s cluster will be used **just to deploy clients inside** 45 | 46 | 3. [Reloader](https://github.com/stakater/Reloader) pod deployed on your preferred namespace (default namespace can be a good option so any other one will have access to it). 47 | 48 | Reloader is an open source tool that taking advantage of POD standard annotation give them the possibility to observe changes on one or many given config maps and perform a rolling update in case of it. 49 | For standard deployment you can execute: 50 | 51 | ```bash 52 | kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml 53 | ``` 54 | 55 | 4. Create topics you want to use on both DC if auto.create.topics.enable is false 56 | 57 | 5. **Cluster Linking**: To setup the `active-active DR pattern` you will need: 58 | 59 | 1. Create a `.env` file under `ccloud-resources` folder it will contain: 60 | 61 | ```properties 62 | export ENV= # CCLOUD Environment you're working on # 63 | export TOPIC= # Default topic where clients will produce/consume # 64 | export NP_TOPIC= # Topic where no proxy clients will produce/consume # 65 | export CCLOUD_NORTH_CLUSTERID= # DC-1 Cluster Id # 66 | export CCLOUD_NORTH_URL= # DC-1 Bootstrap server # 67 | export CCLOUD_NORTH_APIKEY= # DC-1 App key # 68 | export CCLOUD_NORTH_PASSWD= # DC-1 password # 69 | export CCLOUD_WEST_CLUSTERID= # DC-2 Cluster Id # 70 | export CCLOUD_WEST_URL= # DC-2 Bootstrap url # 71 | export CCLOUD_WEST_APIKEY= # DC-2 Api Key # 72 | export CCLOUD_WEST_PASSWD= # DC-2 Password # 73 | ``` 74 | * The following scripts will use `.env` file to configure the resource creation 75 | * All mirror topics will be prefixed with `-` 76 | 2. Run `ccloud-resources/cluster-linking/cluster-linking-north-west.sh` to create DC-1 to DC-2 `cluster link` 77 | 3. Run `ccloud-resources/cluster-linking/create-mirror-topic-north-west.sh` to create the mirror topic on DC2 cluster for proxy based clients 78 | 4. Run `ccloud-resources/cluster-linking/create-mirror-noproxy-topic-north-west.sh` to create the mirror topic for on DC-2 cluster for non proxy based clients 79 | 5. Run `ccloud-resources/cluster-linking/cluster-linking-west-north.sh` to create DC-2 to DC-1 `cluster link` 80 | 6. Run `ccloud-resources/cluster-linking/create-mirror-topic-west-north.sh` to create mirror topic on DC1 for proxy based clients 81 | 7. Run `ccloud-resources/cluster-linking/create-mirror-noproxy-topic-west-north.sh` to create mirror topic on DC1 for non proxy based clients 82 | 83 | 6. **Java Clients (producer/consumer)** images on a available registry (you can use the resource definition provided under k8's resources folder) 84 | 85 | * ConfigMaps/Secrets need to be annotated to make `reloader` aware of the config changes: 86 | 87 | ```yaml 88 | annotations: 89 | reloader.stakater.com/match: "true" 90 | ``` 91 | 92 | * Client application `deployment` will be annotated with config maps that we want to actively listen changes: 93 | 94 | ```yaml 95 | annotations: 96 | configmap.reloader.stakater.com/reload: "java-cloud-producer-config,kafka-proxy-config" 97 | ``` 98 | 99 | On this example we will be using two kind of clients: 100 | - **Sidecar Proxy Based Clients**: Clients will be deployed with a [grepplabs/kafka-proxy](https://github.com/grepplabs/kafka-proxy) container as sidecar, that will act as a layer 4/7 kafka protocol aware proxy. The kafka proxy sidecar will manage all cluster connections so client configuration just need to setup the boker list as localhost on the port that proxy is opened. This way all the cluster related configuration (broker list, secrets, etc) is outside the client and managed by the proxy itself. It gives you the possibility to rely on platform teams to set up all the cluster information preventing the developer teams to have to take care of it. 101 | - **Non-Proxy Based Clients**: Plain K8s Based Kafka Client, on this case the configuration management is all inside the client config maps. 102 | 103 | 1. **Java Client with Proxy** 104 | * Resources are intended to be created on a `clients` namespace. 105 | 106 | 107 | 1. Create 2 configmap using the file `k8s-resources/proxy/kafka-proxy-configmap.yaml` as a template. For each configmap, add the details of DC Cluster and API Key/Secret. This will be the config map the produce refers when it start: 108 | 109 | ```bash 110 | kubectl apply -f k8s-resources/proxy/kafka-proxy-configmap.yaml 111 | ``` 112 | 113 | 2. Producer using a Proxy 114 | 1. Configure how the producer will call the proxy (as a sidecar): 115 | 116 | ```bash 117 | kubectl apply -f k8s-resources/proxy/java-cloud-producer-configmap.yaml 118 | ``` 119 | 120 | 2. Run the producer: 121 | 122 | ```bash 123 | kubectl apply -f k8s-resources/proxy/java-cloud-producer.yaml 124 | ``` 125 | 126 | 3. Send Messages using the producer: 127 | 128 | Get the EXTERNAL-IP where the producer expose its API: 129 | 130 | ```bash 131 | kubectl get svc -n clients 132 | ``` 133 | 134 | and send the message using this command: 135 | 136 | ```bash 137 | curl -X POST http://:8080/chuck-says 138 | ``` 139 | 3. Consumer using Proxy 140 | 1. Apply consumer config: 141 | 142 | ````bash 143 | kubectl apply -f k8s-resources/proxy/java-cloud-consumer-configmap.yaml 144 | ```` 145 | 2. Apply consumer resources: 146 | 147 | ```bash 148 | kubectl apply -f k8s-resources/proxy/java-cloud-consumer.yaml 149 | ``` 150 | 151 | * The consumer will consume from a topic group based on pattern that covers the topology described on `active-active pattern` 152 | 153 | 3. When disaster happens we will change the `k8s-resources/proxy/kafka-proxy-configmap.yaml` with the values of DC2 Cluster 154 | 155 | 4. `Reloader` will trigger a `rolling update` on any deployment annotated to listen changes of this CM. 156 | 157 | 5. As failback procedure we just need to restore the connection data on `k8s-resources/proxy/kafka-proxy-configmap.yaml` to the DC 1 values and after a `rolling update` clients will be again connecting to the original cluster. 158 | 159 | 2. **Java Client without Proxy** 160 | * Resources are intended to be created on a `nproxy-clients` namespace. 161 | 162 | 1. Producer without proxy 163 | 1. Create a configmap using the file `k8s-resources/no-proxy/java-cloud-producer-noproxy-configmap.yaml` as a template. For each configmap, add the details of DC Cluster and API Key/Secret. This will be the config map the produce refers when it start: 164 | 165 | ```bash 166 | kubectl apply -f k8s-resources/no-proxy/java-cloud-producer-noproxy-configmap.yaml 167 | ``` 168 | 169 | 2. Configure how the producer that will call CCloud Cluster directly: 170 | 171 | ```bash 172 | kubectl apply -f k8s-resources/no-proxy/java-cloud-producer-noproxy.yaml 173 | ``` 174 | 175 | 3. Send Messages using the producer: 176 | 177 | Get the EXTERNAL-IP where the producer expese its API: 178 | 179 | ```bash 180 | kubectl get svc -n noproxy-clients 181 | ``` 182 | 183 | and send the message using this command: 184 | 185 | ```bash 186 | curl -X POST http://:8080/chuck-says 187 | ``` 188 | 189 | 4. When disaster happens we will change the `k8s-resources/no-proxy/java-cloud-producer-noproxy-configmap.yaml` with the values of DC2 Cluster 190 | 191 | 5. `Reloader` will trigger a `rolling update` on any deployment annotated to listen changes of this CM. 192 | 193 | 6. As failback procedure we just need to restore the connection data on `k8s-resources/no-proxy/java-cloud-producer-noproxy-configmap.yaml` to the DC 1 values and after a `rolling update` clients will be again connecting to the original cluster. 194 | 195 | 2. Consumer without proxy 196 | 1. Create a config map using `k8s-resources/no-proxy/java-cloud-consumer-noproxy-configmap.yaml` as template as consumer it will contain the cluster information 197 | ```bash 198 | kubectl apply -f k8s-resources/no-proxy/java-cloud-consumer-noproxy.yaml 199 | ``` 200 | 2. Apply the consumer resources: 201 | ```bash 202 | kubectl apply -f k8s-resources/no-proxy/java-cloud-consumer.yaml 203 | ``` 204 | 3. `Consumer Client` will be created configure to consume from a pattern of topics that matches with the `active-active pattern` naming defined. 205 | 206 | ## Failover/failback automation: 207 | 208 | The change of the proper configuration (kafka-proxy or client config) on the subsequent config map is fully automated based `git hub actions` and `Azure AKS` you can find the actions definition under `.github/workflows` folder. 209 | 210 | ### Setting up your Repository to run the action: 211 | 212 | First thing you will need is get yout `AZURE_CREDENTIALS` by running: 213 | 214 | ```bash 215 | az ad sp create-for-rbac --name "chuck" --role contributor --scopes /subscriptions//resourceGroups/ --sdk-auth 216 | ``` 217 | 218 | and set the json response as repository secret on github. 219 | 220 | You also will need to setup secrets for the CCLOUD resources config. 221 | 222 | Find the secrets needed on the image bellow. 223 | 224 | ![gh-secrets.png](assets/gh-secrets.png) 225 | 226 | ### Running the action: 227 | 228 | Just go to the `Actions` tab on your repo choose the action and branch from you want to perform. 229 | 230 | ![gh-actions-run.png](assets/gh-actions-run.png) 231 | 232 | For `proxy clients` a change on `proxy configuration` will be enough to change configuration on both `consumer` and `producer` and perform a rolling update. 233 | 234 | In case of `no proxy clients` the configuration is change directly on the client so you need to apply on both of them separately. 235 | 236 | ## Performance Test 237 | 238 | A basic performance test suite is implemented using [Gatling](https://gatling.io). 239 | 240 | The test will run 1000 request for 10 concurrent users for proxy and no proxy producers 241 | 242 | To run it set up `perftest/src/test/resources/application.conf` file with the public endpoints for your proxy and no-proxy clients go to perftest folder and run: 243 | 244 | ```bash 245 | mvn gatling:test 246 | ``` 247 | 248 | after that you can find the report folders with a `index.html` file under `perftest/target/gatling` 249 | 250 | ![gatling-report.png](assets/gatling-report.png) 251 | -------------------------------------------------------------------------------- /assets/active-active pattern.svg: -------------------------------------------------------------------------------- 1 | --------------------------------------------------------------------------------