├── README.md ├── cluster-link-update-offset-sync.config ├── consumer-offset-filter.sh ├── consumer-offset-validation-old.sh ├── consumer-offset-validation-src-noauth-old.sh ├── consumer-offset-validation-src-noauth.sh ├── consumer-offset-validation.sh ├── example-consumer-filter.txt ├── example-promote.txt ├── failover-mirror-topic.sh ├── mirroring-validation.sh ├── promote-mirror-topic.sh └── topic-delete.sh /README.md: -------------------------------------------------------------------------------- 1 | # ccloud-migration-scripts 2 | 3 | Scripts to help automate the migration of applications to a new Confluent Cloud cluster using Cluster Linking 4 | 5 | ## Prerequisites 6 | 7 | 1. Working cluster link between the source and destination cluster 8 | 2. API keys created for the destination cluster (and the source cluster if migrating between Confluent Cloud clusters) 9 | 3. Required ACLs (if applicable) 10 | https://docs.confluent.io/cloud/current/multi-cloud/cluster-linking/security-cloud.html 11 | 4. Mirror topics created (either automatically with the cluster link configuration or manually) 12 | 5. Kafka and Confluent CLI must be installed and referenced in your $PATH 13 | 14 | ## Description 15 | 16 | Typically, the scripts will be run in the following order as part of the migration: 17 | 18 | 1. mirroring-validation.sh 19 | 2. consumer-offset-validation.sh 20 | 3. consumer-offset-filter.sh 21 | 4. promote-mirror-topic.sh 22 | 23 | ### mirroring-validation.sh 24 | 25 | This script validates that all source partition data is being appropriately mirrored to the destination. During migrations, issues have been observed where a few partitions are not being mirrored correctly between the source and destination. This is typically caught when the mirror lag and last source fetch offset are equal and non-zero 26 | 27 | Two log files are output: one that does not filter partitions with no data on the source and one that does 28 | 29 | #### Example execution: 30 | 31 | ./mirroring-validation.sh \ 32 | --input-file { input file } \ 33 | --link-id { link name } \ 34 | --cluster { destination cluster ID } \ 35 | --environment { environment ID } \ 36 | --all-active-mirror-topics 37 | 38 | #### Example properties files: 39 | 40 | bootstrap.servers={ bootstrap URL } 41 | ssl.endpoint.identification.algorithm=https 42 | security.protocol=SASL_SSL 43 | sasl.mechanism=PLAIN 44 | sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="{ username }" password="{ password }"; 45 | 46 | #### Example output: 47 | 48 | *File 1* 49 | 50 | TOPIC PARTITION MIRROR_LAG STATUS LAST_FETCH_OFFSET 51 | example.topic1 1 56347 ACTIVE 56347 52 | example.topic1 5 57283 ACTIVE 57283 53 | example.topic1 4 0 ACTIVE 0 54 | example.topic1 7 0 ACTIVE 0 55 | 56 | *File 2* 57 | 58 | TOPIC PARTITION MIRROR_LAG STATUS LAST_FETCH_OFFSET 59 | example.topic1 1 56347 ACTIVE 56347 60 | example.topic1 5 57283 ACTIVE 57283 61 | 62 | ### consumer-offset-validation.sh 63 | 64 | This script validates that consumer offsets are successfully synced over the cluster link from source to destination. If no input file is provided, ALL consumer groups are compared 65 | 66 | A legend is printed with each consumer group to display which side of the diff is the source and which is the destination 67 | 68 | #### Example execution: 69 | 70 | ./consumer-offset-validation.sh \ 71 | --dest-bootstrap-server { DEST bootstrap URL } \ 72 | --src-bootstrap-server { SRC bootstrap URL } \ 73 | --command-config-dest { DEST properties } \ 74 | --command-config-src { SRC properties } \ 75 | --remove-unused-consumers \ 76 | --input-file { list of target consumers } 77 | 78 | #### Example properties files: 79 | 80 | bootstrap.servers={ bootstrap URL } 81 | ssl.endpoint.identification.algorithm=https 82 | security.protocol=SASL_SSL 83 | sasl.mechanism=PLAIN 84 | sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="{ username }" password="{ password }"; 85 | 86 | #### Example diff file output **without removing** unused consumers: 87 | 88 | =========================== Group Name: example-v1 ============================ 89 | 1c1 90 | < =================== SRC =================== 91 | --- 92 | > =================== DEST =================== 93 | 94 | =========================== Group Name: example-v2 ============================ 95 | 1c1 96 | < =================== SRC =================== 97 | --- 98 | > =================== DEST =================== 99 | 9,10d8 100 | < example-v2 example-topic2 14 - - 101 | < example-v2 example-topic2 15 - - 102 | 103 | =========================== Group Name: example-v3 ============================ 104 | 1c1 105 | < =================== OSK =================== 106 | --- 107 | > =================== CCLOUD =================== 108 | 3,6c3,6 109 | < example-v3 example-topic3 0 77198 0 110 | < example-v3 example-topic3 1 37189 0 111 | < example-v3 example-topic3 2 83083 1 112 | < example-v3 example-topic3 3 107846 0 113 | --- 114 | > example-v3 example-topic3 0 77197 1 115 | > example-v3 example-topic3 1 37188 2 116 | > example-v3 example-topic3 2 83082 2 117 | > example-v3 example-topic3 3 107846 1 118 | 8,9c8,9 119 | < example-v3 example-topic3 5 70157 0 120 | < example-v3 example-topic3 6 44621 0 121 | --- 122 | > example-v3 example-topic3 5 70156 1 123 | > example-v3 example-topic3 6 44620 1 124 | 125 | #### Example diff file output **removing** unused consumers (removing null offset data): 126 | 127 | =========================== Group Name: example-v1 ============================ 128 | 1c1 129 | < =================== SRC =================== 130 | --- 131 | > =================== DEST =================== 132 | 133 | =========================== Group Name: example-v2 ============================ 134 | 1c1 135 | < =================== SRC =================== 136 | --- 137 | > =================== DEST =================== 138 | 139 | =========================== Group Name: example-v3 ============================ 140 | 1c1 141 | < =================== OSK =================== 142 | --- 143 | > =================== CCLOUD =================== 144 | 3,6c3,6 145 | < example-v3 example-topic3 0 77198 0 146 | < example-v3 example-topic3 1 37189 0 147 | < example-v3 example-topic3 2 83083 1 148 | < example-v3 example-topic3 3 107846 0 149 | --- 150 | > example-v3 example-topic3 0 77197 1 151 | > example-v3 example-topic3 1 37188 2 152 | > example-v3 example-topic3 2 83082 2 153 | > example-v3 example-topic3 3 107846 1 154 | 8,9c8,9 155 | < example-v3 example-topic3 5 70157 0 156 | < example-v3 example-topic3 6 44621 0 157 | --- 158 | > example-v3 example-topic3 5 70156 1 159 | > example-v3 example-topic3 6 44620 1 160 | 161 | ### consumer-offset-filter.sh 162 | 163 | This script accepts a list of consumer groups in an input file to exclude from the offset sync functionality of the cluster link 164 | 165 | A config file is output that is used to update the **consumer.offset.group.filters** and exclude target consumer groups 166 | 167 | #### Example execution: 168 | 169 | ./consumer-offset-filter.sh \ 170 | --input-file { input file } \ 171 | --link-id { link name } \ 172 | --cluster { destination cluster ID } \ 173 | --environment { environment ID } 174 | 175 | #### Example output: 176 | 177 | ============================================================================ 178 | =============== Create New Configuration File for Link Update ============== 179 | 180 | consumer.offset.group.filters={"groupFilters":[ {"name": "*", "patternType": "LITERAL", "filterType": "INCLUDE"},{"name": "cg-test-app1", "patternType": "LITERAL", "filterType": "EXCLUDE"},{"name": "cg-test2-app1", "patternType": "LITERAL", "filterType": "EXCLUDE"},{"name": "cg-test2-app2", "patternType": "LITERAL", "filterType": "EXCLUDE"}]} 181 | 182 | ============================================================================ 183 | ========== Update the Cluster Link Consumer Group Offset Syncing =========== 184 | 185 | Updated cluster link "test-link". 186 | 187 | ============================================================================ 188 | ================= Validate the Updated Link Configuration ================== 189 | 190 | Config Name | Config Value | Read Only | Sensitive | Source | Synonyms 191 | -----------------------------------------+----------------------------------------------+-----------+-----------+-----------------------------+----------- 192 | ... 193 | consumer.offset.group.filters | {"groupFilters":[ {"name": | false | false | DYNAMIC_CLUSTER_LINK_CONFIG | [] 194 | | "*", "patternType": | | | | 195 | | "LITERAL", "filterType": | | | | 196 | | "INCLUDE"},{"name": | | | | 197 | | "cg-test-app1", "patternType": | | | | 198 | | "LITERAL", "filterType": | | | | 199 | | "EXCLUDE"},{"name": | | | | 200 | | "cg-test2-app1", | | | | 201 | | "patternType": | | | | 202 | | "LITERAL", "filterType": | | | | 203 | | "EXCLUDE"},{"name": | | | | 204 | | "cg-test2-app2", | | | | 205 | | "patternType": "LITERAL", | | | | 206 | | "filterType": "EXCLUDE"}]} | | | | 207 | consumer.offset.sync.enable | true | false | false | DYNAMIC_CLUSTER_LINK_CONFIG | [] 208 | 209 | ### promote-mirror-topic.sh 210 | 211 | This script accepts a list of mirror topics to promote and validates successful promotion 212 | 213 | #### Example execution: 214 | 215 | ./promote-mirror-topic.sh \ 216 | --input-file { input file } \ 217 | --link-id { link name } \ 218 | --cluster { destination cluster ID } \ 219 | --environment { environment ID } 220 | 221 | #### Example output: 222 | 223 | ============================================================================ 224 | =========== Displaying mirror topic current status pre-promotion =========== 225 | 226 | Mirror Topic: test 227 | Command: confluent kafka mirror describe test --link test-link --cluster lkc-12rq15 --environment env-z8xj3 228 | 229 | Link Name | Mirror Topic Name | Partition | Partition Mirror Lag | Source Topic Name | Mirror Status | Status Time Ms | Last Source Fetch Offset 230 | ------------+-------------------+-----------+----------------------+-------------------+---------------+----------------+--------------------------- 231 | test-link | test | 1 | 0 | test | ACTIVE | 1646336492686 | 7 232 | test-link | test | 0 | 0 | test | ACTIVE | 1646336492686 | 3 233 | test-link | test | 3 | 0 | test | ACTIVE | 1646336492686 | 1 234 | test-link | test | 2 | 0 | test | ACTIVE | 1646336492686 | 3 235 | test-link | test | 5 | 0 | test | ACTIVE | 1646336492686 | 1 236 | test-link | test | 4 | 0 | test | ACTIVE | 1646336492686 | 6 237 | 238 | Mirror Topic: test2 239 | Command: confluent kafka mirror describe test2 --link test-link --cluster lkc-12rq15 --environment env-z8xj3 240 | 241 | Link Name | Mirror Topic Name | Partition | Partition Mirror Lag | Source Topic Name | Mirror Status | Status Time Ms | Last Source Fetch Offset 242 | ------------+-------------------+-----------+----------------------+-------------------+---------------+----------------+--------------------------- 243 | test-link | test2 | 1 | 0 | test2 | ACTIVE | 1646336492618 | 9 244 | test-link | test2 | 0 | 0 | test2 | ACTIVE | 1646336492618 | 2 245 | test-link | test2 | 3 | 0 | test2 | ACTIVE | 1646336492618 | 16 246 | test-link | test2 | 2 | 0 | test2 | ACTIVE | 1646336492618 | 11 247 | test-link | test2 | 5 | 0 | test2 | ACTIVE | 1646336492618 | 5 248 | test-link | test2 | 4 | 0 | test2 | ACTIVE | 1646336492618 | 5 249 | 250 | 251 | ============================================================================ 252 | ========================= Promoting mirror topics ========================== 253 | 254 | Command: confluent kafka mirror promote test test2 --link test-link --cluster lkc-12rq15 --environment env-z8xj3 255 | 256 | Mirror Topic Name | Partition | Partition Mirror Lag | Error Message | Error Code | Last Source Fetch Offset 257 | --------------------+-----------+----------------------+---------------+------------+--------------------------- 258 | test | 1 | 0 | | | 7 259 | test | 0 | 0 | | | 3 260 | test | 3 | 0 | | | 1 261 | test | 2 | 0 | | | 3 262 | test | 5 | 0 | | | 1 263 | test | 4 | 0 | | | 6 264 | test2 | 1 | 0 | | | 9 265 | test2 | 0 | 0 | | | 2 266 | test2 | 3 | 0 | | | 16 267 | test2 | 2 | 0 | | | 11 268 | test2 | 5 | 0 | | | 5 269 | test2 | 4 | 0 | | | 5 270 | 271 | Wait while the topics are promoted 272 | 273 | =========== Displaying mirror topic current status post-promotion ========== 274 | 275 | Mirror Topic: test 276 | Command: confluent kafka mirror describe test --link test-link --cluster lkc-12rq15 --environment env-z8xj3 277 | 278 | Link Name | Mirror Topic Name | Partition | Partition Mirror Lag | Source Topic Name | Mirror Status | Status Time Ms | Last Source Fetch Offset 279 | ------------+-------------------+-----------+----------------------+-------------------+---------------+----------------+--------------------------- 280 | test-link | test | 1 | 0 | test | STOPPED | 1646430749830 | 7 281 | test-link | test | 0 | 0 | test | STOPPED | 1646430749830 | 3 282 | test-link | test | 3 | 0 | test | STOPPED | 1646430749830 | 1 283 | test-link | test | 2 | 0 | test | STOPPED | 1646430749830 | 3 284 | test-link | test | 5 | 0 | test | STOPPED | 1646430749830 | 1 285 | test-link | test | 4 | 0 | test | STOPPED | 1646430749830 | 6 286 | 287 | Mirror Topic: test2 288 | Command: confluent kafka mirror describe test2 --link test-link --cluster lkc-12rq15 --environment env-z8xj3 289 | 290 | Link Name | Mirror Topic Name | Partition | Partition Mirror Lag | Source Topic Name | Mirror Status | Status Time Ms | Last Source Fetch Offset 291 | ------------+-------------------+-----------+----------------------+-------------------+---------------+----------------+--------------------------- 292 | test-link | test2 | 1 | 0 | test2 | STOPPED | 1646430749751 | 9 293 | test-link | test2 | 0 | 0 | test2 | STOPPED | 1646430749751 | 2 294 | test-link | test2 | 3 | 0 | test2 | STOPPED | 1646430749751 | 16 295 | test-link | test2 | 2 | 0 | test2 | STOPPED | 1646430749751 | 11 296 | test-link | test2 | 5 | 0 | test2 | STOPPED | 1646430749751 | 5 297 | test-link | test2 | 4 | 0 | test2 | STOPPED | 1646430749751 | 5 298 | 299 | =========== Displaying target mirror topics still pending promotion ========== 300 | 301 | TOPIC PARTITION STATUS 302 | 303 | -------------------------------------------------------------------------------- /cluster-link-update-offset-sync.config: -------------------------------------------------------------------------------- 1 | consumer.offset.group.filters={"groupFilters":[ {"name": "*", "patternType": "LITERAL", "filterType": "INCLUDE"},{"name": "cg-test-app1", "patternType": "LITERAL", "filterType": "EXCLUDE"},{"name": "cg-test2-app1", "patternType": "LITERAL", "filterType": "EXCLUDE"},{"name": "cg-test2-app2", "patternType": "LITERAL", "filterType": "EXCLUDE"}]} 2 | -------------------------------------------------------------------------------- /consumer-offset-filter.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #Ensure the confluent CLI is installed and referenced in your PATH or set the CONFLUENT_HOME_BIN environment variable to the location of the binary 3 | #Cluster link must be running 4 | 5 | show_usage () { 6 | echo "Usage: $0 [options [parameters]]" 7 | echo "" 8 | echo "Options:" 9 | echo " --help -- Prints this page" 10 | echo " --input-file [input-file]" 11 | echo " --link-id [link-name]" 12 | echo " --cluster [ccloud-cluster-id]" 13 | echo " --environment [ccloud-environment-id]" 14 | echo " --exclude-all [use this flag to disable consumer offset sync]" 15 | 16 | return 0 17 | } 18 | 19 | if [[ $# -eq 0 ]];then 20 | echo "No input arguments provided." 21 | show_usage 22 | exit 1 23 | fi 24 | 25 | echo "============================================================================" 26 | echo "" 27 | echo "This script accepts a list of consumer groups to exclude from the offset sync functionality of the cluster link" 28 | echo "" 29 | 30 | echo "============================================================================" 31 | echo "=============================== PARAMETERS =================================" 32 | echo "" 33 | 34 | while [ ! -z "$1" ] 35 | do 36 | if [[ "$1" == "--help" ]] 37 | then 38 | show_usage 39 | exit 0 40 | elif [[ "$1" == "--input-file" ]] 41 | then 42 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 43 | then 44 | echo "No Value provided for "$1". Please ensure proper values are provided" 45 | show_usage 46 | exit 1 47 | fi 48 | inputFile="$2" 49 | shift 50 | elif [[ "$1" == "--link-id" ]] 51 | then 52 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 53 | then 54 | echo "No Value provided for "$1". Please ensure proper values are provided" 55 | show_usage 56 | exit 1 57 | fi 58 | linkId="$2" 59 | echo "Cluster Link Name: ${linkId}" 60 | shift 61 | elif [[ "$1" == "--cluster" ]] 62 | then 63 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 64 | then 65 | echo "No Value provided for "$1". Please ensure proper values are provided" 66 | show_usage 67 | exit 1 68 | fi 69 | clusterId="$2" 70 | echo "CCloud Kafka Cluster ID: ${clusterId}" 71 | shift 72 | elif [[ "$1" == "--environment" ]] 73 | then 74 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 75 | then 76 | echo "No Value provided for "$1". Please ensure proper values are provided" 77 | show_usage 78 | exit 1 79 | fi 80 | environmentId="$2" 81 | echo "CCloud Environment ID: ${environmentId}" 82 | shift 83 | elif [[ "$1" == "--exclude-all" ]] 84 | then 85 | filterAllConsumers=true 86 | echo "Unused source consumers will be filtered from the consumer group data" 87 | fi 88 | shift 89 | done 90 | 91 | 92 | if [[ -z "$linkId" ]] || [[ -z "$clusterId" ]] || [[ -z "$environmentId" ]] || [[ (-z "$inputFile" && "$filterAllConsumers" != "true") ]] 93 | then 94 | echo "--input-file or --exclude-all, --link-id, --cluster, and --environment are required for execution." 95 | show_usage 96 | exit 1 97 | fi 98 | 99 | 100 | if [[ ! -z "$inputFile" ]] 101 | then 102 | command="consumer.offset.group.filters={\"groupFilters\":[ {\"name\": \"*\", \"patternType\": \"LITERAL\", \"filterType\": \"INCLUDE\"}," 103 | 104 | while IFS= read -r line || [[ "$line" ]]; 105 | do 106 | command="${command}{\"name\": \"$line\", \"patternType\": \"LITERAL\", \"filterType\": \"EXCLUDE\"}," 107 | done < ${inputFile} 108 | command="${command%?}]}" 109 | shift 110 | elif [[ "$filterAllConsumers" = true ]] 111 | then 112 | command="consumer.offset.sync.enable=false" 113 | 114 | else 115 | echo "No input file or exclude all consumer group flag found" 116 | show_usage 117 | exit 1 118 | fi 119 | 120 | echo "" 121 | echo "============================================================================" 122 | echo "=============== Create New Configuration File for Link Update ==============" 123 | echo "" 124 | 125 | echo "${command}" > cluster-link-update-offset-sync-${clusterId}.config 126 | 127 | cat cluster-link-update-offset-sync-${clusterId}.config 128 | 129 | echo "" 130 | echo "============================================================================" 131 | echo "========== Update the Cluster Link Consumer Group Offset Syncing ===========" 132 | echo "" 133 | 134 | confluent kafka link update ${linkId} --config-file cluster-link-update-offset-sync-${clusterId}.config --cluster ${clusterId} --environment ${environmentId} 135 | 136 | echo "" 137 | echo "============================================================================" 138 | echo "================= Validate the Updated Link Configuration ==================" 139 | echo "" 140 | 141 | confluent kafka link describe ${linkId} --cluster ${clusterId} --environment ${environmentId} 142 | 143 | echo "" 144 | -------------------------------------------------------------------------------- /consumer-offset-validation-old.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #This script validates that consumer offsets are successfully synced over the cluster link from source to destination 3 | #The Kafka CLI commands must be installed in a location set in your PATH to run kafka-consumer-groups 4 | 5 | show_usage () { 6 | echo "Usage: $0 [options [parameters]]" 7 | echo "" 8 | echo "Options:" 9 | echo " --help -- Prints this page" 10 | echo " --dest-bootstrap-server [dest-bootstrap-server-list]" 11 | echo " --src-bootstrap-server [src-bootstrap-server-list]" 12 | echo " --command-config-dest [dest-command-config-file]" 13 | echo " --command-config-src [src-command-config-file]" 14 | echo " --remove-unused-consumers [use this flag if there are a number of unused consumers on the source cluster]" 15 | echo " --input-file [file containing a list of target consumer groups]" 16 | 17 | return 0 18 | } 19 | 20 | if [[ $# -eq 0 ]];then 21 | echo "No input arguments provided." 22 | show_usage 23 | exit 1 24 | fi 25 | 26 | echo "============================================================================" 27 | echo "" 28 | echo "This script validates that consumer offsets are successfully synced over the cluster link from the source to destination" 29 | echo "" 30 | 31 | echo "============================================================================" 32 | echo "=============================== PARAMETERS =================================" 33 | echo "" 34 | 35 | while [ ! -z "$1" ] 36 | do 37 | if [[ "$1" == "--help" ]] 38 | then 39 | show_usage 40 | exit 0 41 | elif [[ "$1" == "--dest-bootstrap-server" ]] 42 | then 43 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 44 | then 45 | echo "No Value provided for "$1". Please ensure proper values are provided" 46 | show_usage 47 | exit 1 48 | fi 49 | DEST_BOOTSTRAP_SERVERS="$2" 50 | echo "Test Env Bootstrap Servers are: ${DEST_BOOTSTRAP_SERVERS}" 51 | shift 52 | elif [[ "$1" == "--src-bootstrap-server" ]] 53 | then 54 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 55 | then 56 | echo "No Value provided for "$1". Please ensure proper values are provided" 57 | show_usage 58 | exit 1 59 | fi 60 | SRC_BOOTSTRAP_SERVERS="$2" 61 | echo "Default Env Bootstrap Servers are: ${SRC_BOOTSTRAP_SERVERS}" 62 | shift 63 | elif [[ "$1" == "--command-config-dest" ]] 64 | then 65 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 66 | then 67 | echo "No Value provided for "$1". Please ensure proper values are provided" 68 | show_usage 69 | exit 1 70 | fi 71 | COMMAND_CONFIG_DEST="$2" 72 | echo "Destination Command Config File path is: ${COMMAND_CONFIG_DEST}" 73 | shift 74 | elif [[ "$1" == "--command-config-src" ]] 75 | then 76 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 77 | then 78 | echo "No Value provided for "$1". Please ensure proper values are provided" 79 | show_usage 80 | exit 1 81 | fi 82 | COMMAND_CONFIG_SRC="$2" 83 | echo "Source Command Config File path is: ${COMMAND_CONFIG_SRC}" 84 | shift 85 | elif [[ "$1" == "--input-file" ]] 86 | then 87 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 88 | then 89 | echo "No Value provided for "$1". Please ensure proper values are provided" 90 | show_usage 91 | exit 1 92 | fi 93 | INPUT_FILE="$2" 94 | echo "File with the list of target consumer groups: ${INPUT_FILE}" 95 | elif [[ "$1" == "--remove-unused-consumers" ]] 96 | then 97 | REMOVE_UNUSED_CONSUMERS=true 98 | echo "Filtering unused source consumer group data: true" 99 | fi 100 | shift 101 | done 102 | 103 | echo "" 104 | 105 | CONSUMER_GROUPS=(${PWD}/`date +"%d-%m-%y-%T"`_consumer_groups.txt) 106 | DIFF=(${PWD}/`date +"%d-%m-%y-%T"`_diff.txt) 107 | DEST_GROUP_DATA=(${PWD}/dest_group_data.txt) 108 | SRC_GROUP_DATA=(${PWD}/src_group_data.txt) 109 | PARSED_DEST_GROUP_DATA=(${PWD}/parsed_dest_group_data.txt) 110 | PARSED_SRC_GROUP_DATA=(${PWD}/parsed_src_group_data.txt) 111 | 112 | 113 | if [[ -z "$DEST_BOOTSTRAP_SERVERS" ]] || [[ -z "$SRC_BOOTSTRAP_SERVERS" ]] || [[ -z "$COMMAND_CONFIG_DEST" ]] || [[ -z "$COMMAND_CONFIG_SRC" ]] 114 | then 115 | echo "--dest-bootstrap-server, --src-bootstrap-server, --command-config-dest, and --command-config-src are required for execution." 116 | show_usage 117 | exit 1 118 | fi 119 | 120 | if [[ -z "$INPUT_FILE" ]] 121 | then 122 | echo "============================================================================" 123 | echo "=============================== SRC CONSUMERS ==============================" 124 | echo "" 125 | echo "Gathering list of consumer groups from SRC cluster" 126 | echo "Output written to" `date +"%d-%m-%y-%T"`"_consumer_groups.txt" 127 | echo "" 128 | 129 | kafka-consumer-groups --bootstrap-server ${SRC_BOOTSTRAP_SERVERS} --command-config ${COMMAND_CONFIG_SRC} --list > ${CONSUMER_GROUPS} 130 | 131 | else 132 | echo "============================================================================" 133 | echo "=============================== SRC CONSUMERS ==============================" 134 | echo "" 135 | echo "Displaying list of target consumer groups from the input file" 136 | echo "" 137 | 138 | cat "$INPUT_FILE" 139 | echo "" 140 | cat "$INPUT_FILE" > ${CONSUMER_GROUPS} 141 | fi 142 | 143 | echo "============================================================================" 144 | echo "=================================== DIFF ===================================" 145 | echo "" 146 | echo "Comparing consumer group data between the SRC and DEST clusters to validate the syncing" 147 | echo "Output written to" `date +"%d-%m-%y-%T"`"_diff.txt" 148 | echo "" 149 | 150 | if [[ "$REMOVE_UNUSED_CONSUMERS" = true ]] 151 | then 152 | echo "================================================" 153 | echo "||Removing inactive consumer group information||" 154 | echo "================================================" 155 | cat ${CONSUMER_GROUPS} | while read -r group_name 156 | do 157 | if [[ -z "${group_name}" ]] 158 | then 159 | : 160 | else 161 | # Describe the Consumer Groups in SRC and DEST 162 | kafka-consumer-groups --bootstrap-server ${DEST_BOOTSTRAP_SERVERS} --command-config ${COMMAND_CONFIG_DEST} --group ${group_name} --describe > ${DEST_GROUP_DATA} 163 | kafka-consumer-groups --bootstrap-server ${SRC_BOOTSTRAP_SERVERS} --command-config ${COMMAND_CONFIG_SRC} --group ${group_name} --describe > ${SRC_GROUP_DATA} 164 | 165 | # Parse the Consumer Group data to only include columns expected to match, remove unneeded consumer group data, remove the offset data for unused consumers that is not synced over the link, and sort the data on the partition number to prepare for the diff 166 | printf "======================= SRC =======================\n" > ${PARSED_SRC_GROUP_DATA} 167 | printf "======================= DEST =======================\n" > ${PARSED_DEST_GROUP_DATA} 168 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '{print $col1, $col2, $col3, $col4, $col6}' "${SRC_GROUP_DATA}" | sort -t, -nk3 | column -t >> ${PARSED_SRC_GROUP_DATA} 169 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '{print $col1, $col2, $col3, $col4, $col6}' "${DEST_GROUP_DATA}" | sort -t, -nk3 | column -t >> ${PARSED_DEST_GROUP_DATA} 170 | 171 | # Diff the two sorted files 172 | printf "=========================== Group Name: %s ============================\n" "${group_name}" >> ${DIFF} 173 | diff ${PARSED_SRC_GROUP_DATA} ${PARSED_DEST_GROUP_DATA} >> ${DIFF} 174 | printf "\n" >> ${DIFF} 175 | fi 176 | done 177 | else 178 | cat ${CONSUMER_GROUPS} | while read -r group_name 179 | do 180 | if [[ -z "${group_name}" ]] 181 | then 182 | : 183 | else 184 | # Describe the Consumer Groups in SRC and DEST 185 | kafka-consumer-groups --bootstrap-server ${DEST_BOOTSTRAP_SERVERS} --command-config ${COMMAND_CONFIG_DEST} --group ${group_name} --describe > ${DEST_GROUP_DATA} 186 | kafka-consumer-groups --bootstrap-server ${SRC_BOOTSTRAP_SERVERS} --command-config ${COMMAND_CONFIG_SRC} --group ${group_name} --describe > ${SRC_GROUP_DATA} 187 | 188 | # Parse the Consumer Group data to only include columns expected to match, remove unneeded consumer group data, and sort the data on the partition number to prepare for the diff 189 | printf "======================= SRC =======================\n" > ${PARSED_SRC_GROUP_DATA} 190 | printf "======================= DEST =======================\n" > ${PARSED_DEST_GROUP_DATA} 191 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '{print $col1, $col2, $col3, $col4, $col6}' "${SRC_GROUP_DATA}" | sort -t, -nk3 | column -t >> ${PARSED_SRC_GROUP_DATA} 192 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '{print $col1, $col2, $col3, $col4, $col6}' "${DEST_GROUP_DATA}" | sort -t, -nk3 | column -t >> ${PARSED_DEST_GROUP_DATA} 193 | 194 | # Diff the two sorted files 195 | printf "=========================== Group Name: %s ============================\n" "${group_name}" >> ${DIFF} 196 | diff ${PARSED_SRC_GROUP_DATA} ${PARSED_DEST_GROUP_DATA} >> ${DIFF} 197 | printf "\n" >> ${DIFF} 198 | fi 199 | done 200 | fi 201 | 202 | # Cleanup temp files 203 | rm ${CONSUMER_GROUPS} 204 | rm ${DEST_GROUP_DATA} 205 | rm ${SRC_GROUP_DATA} 206 | rm ${PARSED_DEST_GROUP_DATA} 207 | rm ${PARSED_SRC_GROUP_DATA} 208 | 209 | echo "" -------------------------------------------------------------------------------- /consumer-offset-validation-src-noauth-old.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #This script validates that consumer offsets are successfully synced over the cluster link from source to destination 3 | #The Kafka CLI commands must be installed in a location set in your PATH to run kafka-consumer-groups 4 | 5 | show_usage () { 6 | echo "Usage: $0 [options [parameters]]" 7 | echo "" 8 | echo "Options:" 9 | echo " --help -- Prints this page" 10 | echo " --osk-bootstrap-server [osk-bootstrap-server-list]" 11 | echo " --ccloud-bootstrap-server [ccloud-bootstrap-server-list]" 12 | echo " --command-config [command-config-file]" 13 | echo " --remove-unused-consumers [use this flag if there are a number of unused consumers on the source cluster]" 14 | echo " --input-file [file containing a list of target consumer groups]" 15 | 16 | return 0 17 | } 18 | 19 | if [[ $# -eq 0 ]];then 20 | echo "No input arguments provided." 21 | show_usage 22 | exit 1 23 | fi 24 | 25 | echo "============================================================================" 26 | echo "" 27 | echo "This script validates that consumer offsets are successfully synced over the cluster link from the source to destination" 28 | echo "" 29 | 30 | echo "============================================================================" 31 | echo "=============================== PARAMETERS =================================" 32 | echo "" 33 | 34 | while [ ! -z "$1" ] 35 | do 36 | if [[ "$1" == "--help" ]] 37 | then 38 | show_usage 39 | exit 0 40 | elif [[ "$1" == "--osk-bootstrap-server" ]] 41 | then 42 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 43 | then 44 | echo "No Value provided for "$1". Please ensure proper values are provided" 45 | show_usage 46 | exit 1 47 | fi 48 | OSK_BOOTSTRAP_SERVERS="$2" 49 | echo "OSK Bootstrap Servers are: ${OSK_BOOTSTRAP_SERVERS}" 50 | shift 51 | elif [[ "$1" == "--ccloud-bootstrap-server" ]] 52 | then 53 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 54 | then 55 | echo "No Value provided for "$1". Please ensure proper values are provided" 56 | show_usage 57 | exit 1 58 | fi 59 | CCLOUD_BOOTSTRAP_SERVERS="$2" 60 | echo "CCloud Bootstrap Servers are: ${CCLOUD_BOOTSTRAP_SERVERS}" 61 | shift 62 | elif [[ "$1" == "--command-config" ]] 63 | then 64 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 65 | then 66 | echo "No Value provided for "$1". Please ensure proper values are provided" 67 | show_usage 68 | exit 1 69 | fi 70 | CCLOUD_COMMAND_CONFIG="$2" 71 | echo "CCloud Command Config File path is: ${CCLOUD_COMMAND_CONFIG}" 72 | shift 73 | elif [[ "$1" == "--input-file" ]] 74 | then 75 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 76 | then 77 | echo "No Value provided for "$1". Please ensure proper values are provided" 78 | show_usage 79 | exit 1 80 | fi 81 | INPUT_FILE="$2" 82 | echo "File with the list of target consumer groups: ${INPUT_FILE}" 83 | elif [[ "$1" == "--remove-unused-consumers" ]] 84 | then 85 | REMOVE_UNUSED_CONSUMERS=true 86 | echo "Filtering unused source consumer group data: true" 87 | fi 88 | shift 89 | done 90 | 91 | CONSUMER_GROUPS=(${PWD}/`date +"%d-%m-%y-%T"`osk_consumer_groups.txt) 92 | DIFF=(${PWD}/`date +"%d-%m-%y-%T"`_diff.txt) 93 | OSK_GROUP_DATA=(${PWD}/osk_group_data.txt) 94 | CCLOUD_GROUP_DATA=(${PWD}/ccloud_group_data.txt) 95 | PARSED_OSK_GROUP_DATA=(${PWD}/parsed_osk_group_data.txt) 96 | PARSED_CCLOUD_GROUP_DATA=(${PWD}/parsed_ccloud_group_data.txt) 97 | 98 | if [[ -z "$OSK_BOOTSTRAP_SERVERS" ]] || [[ -z "$CCLOUD_BOOTSTRAP_SERVERS" ]] || [[ -z "$CCLOUD_COMMAND_CONFIG" ]] 99 | then 100 | echo "--osk-bootstrap-server, --ccloud-bootstrap-server, and --command-config are required for execution." 101 | show_usage 102 | exit 1 103 | fi 104 | 105 | if [[ -z "$INPUT_FILE" ]] 106 | then 107 | echo "============================================================================" 108 | echo "=============================== OSK CONSUMERS ==============================" 109 | echo "" 110 | echo "Gathering list of consumer groups from OSK cluster" 111 | echo "Output written to" `date +"%d-%m-%y-%T"`"osk_consumer_groups.txt" 112 | echo "" 113 | 114 | kafka-consumer-groups --bootstrap-server ${OSK_BOOTSTRAP_SERVERS} --list > ${CONSUMER_GROUPS} 115 | 116 | else 117 | echo "============================================================================" 118 | echo "=============================== SRC CONSUMERS ==============================" 119 | echo "" 120 | echo "Displaying list of target consumer groups from the input file" 121 | echo "" 122 | 123 | cat "$INPUT_FILE" 124 | cat "$INPUT_FILE" > ${CONSUMER_GROUPS} 125 | fi 126 | 127 | echo "============================================================================" 128 | echo "=================================== DIFF ===================================" 129 | echo "" 130 | echo "Comparing consumer group data between the OSK and CCloud clusters to validate the syncing" 131 | echo "Output written to" `date +"%d-%m-%y-%T"`"_diff.txt" 132 | echo "" 133 | 134 | if [[ "$REMOVE_UNUSED_CONSUMERS" = true ]] 135 | then 136 | echo "================================================" 137 | echo "||Removing inactive consumer group information||" 138 | echo "================================================" 139 | cat ${CONSUMER_GROUPS} | while read -r group_name 140 | do 141 | if [[ -z "${group_name}" ]] 142 | then 143 | : 144 | else 145 | # Describe the Consumer Groups in SRC and DEST 146 | kafka-consumer-groups --bootstrap-server ${OSK_BOOTSTRAP_SERVERS} --group ${group_name} --describe > ${OSK_GROUP_DATA} 147 | kafka-consumer-groups --bootstrap-server ${CCLOUD_BOOTSTRAP_SERVERS} --command-config ${CCLOUD_COMMAND_CONFIG} --group ${group_name} --describe > ${CCLOUD_GROUP_DATA} 148 | 149 | # Parse the Consumer Group data to only include columns expected to match, remove unneeded consumer group data, remove the offset data for unused consumers that is not synced over the link, and sort the data on the partition number to prepare for the diff 150 | printf "======================= SRC =======================\n" > ${PARSED_OSK_GROUP_DATA} 151 | printf "======================= DEST =======================\n" > ${PARSED_CCLOUD_GROUP_DATA} 152 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '$col4 == "-" { next } {print $col1, $col2, $col3, $col4, $col6}' ${OSK_GROUP_DATA} | sort -t, -nk3 | column -t >> ${PARSED_OSK_GROUP_DATA} 153 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '$col4 == "-" { next } {print $col1, $col2, $col3, $col4, $col6}' ${CCLOUD_GROUP_DATA} | sort -t, -nk3 | column -t >> ${PARSED_CCLOUD_GROUP_DATA} 154 | 155 | # Diff the two sorted files 156 | printf "=========================== Group Name: %s ============================\n" "${group_name}" >> ${DIFF} 157 | diff ${PARSED_OSK_GROUP_DATA} ${PARSED_CCLOUD_GROUP_DATA} >> ${DIFF} 158 | printf "\n" >> ${DIFF} 159 | fi 160 | done 161 | else 162 | cat ${CONSUMER_GROUPS} | while read -r group_name 163 | do 164 | if [[ -z "${group_name}" ]] 165 | then 166 | : 167 | else 168 | # Describe the Consumer Groups in SRC and DEST 169 | kafka-consumer-groups --bootstrap-server ${OSK_BOOTSTRAP_SERVERS} --group ${group_name} --describe > ${OSK_GROUP_DATA} 170 | kafka-consumer-groups --bootstrap-server ${CCLOUD_BOOTSTRAP_SERVERS} --command-config ${CCLOUD_COMMAND_CONFIG} --group ${group_name} --describe > ${CCLOUD_GROUP_DATA} 171 | 172 | # Parse the Consumer Group data to only include columns expected to match, remove unneeded consumer group data, and sort the data on the partition number to prepare for the diff 173 | printf "======================= SRC =======================\n" > ${PARSED_OSK_GROUP_DATA} 174 | printf "======================= DEST =======================\n" > ${PARSED_CCLOUD_GROUP_DATA} 175 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '{print $col1, $col2, $col3, $col4, $col6}' ${OSK_GROUP_DATA} | sort -t, -nk3 | column -t >> ${PARSED_OSK_GROUP_DATA} 176 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '{print $col1, $col2, $col3, $col4, $col6}' ${CCLOUD_GROUP_DATA} | sort -t, -nk3 | column -t >> ${PARSED_CCLOUD_GROUP_DATA} 177 | 178 | # Diff the two sorted files 179 | printf "=========================== Group Name: %s ============================\n" "${group_name}" >> ${DIFF} 180 | diff ${PARSED_OSK_GROUP_DATA} ${PARSED_CCLOUD_GROUP_DATA} >> ${DIFF} 181 | printf "\n" >> ${DIFF} 182 | fi 183 | done 184 | fi 185 | 186 | # Cleanup temp files 187 | rm ${CONSUMER_GROUPS} 188 | rm ${OSK_GROUP_DATA} 189 | rm ${CCLOUD_GROUP_DATA} 190 | rm ${PARSED_OSK_GROUP_DATA} 191 | rm ${PARSED_CCLOUD_GROUP_DATA} 192 | 193 | echo "" -------------------------------------------------------------------------------- /consumer-offset-validation-src-noauth.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #This script validates that consumer offsets are successfully synced over the cluster link from source to destination 3 | #The Kafka CLI commands must be installed in a location set in your PATH to run kafka-consumer-groups 4 | 5 | show_usage () { 6 | echo "Usage: $0 [options [parameters]]" 7 | echo "" 8 | echo "Options:" 9 | echo " --help -- Prints this page" 10 | echo " --src-bootstrap-server [src-bootstrap-server-list]" 11 | echo " --ccloud-bootstrap-server [ccloud-bootstrap-server-list]" 12 | echo " --command-config [command-config-file]" 13 | echo " --remove-unused-consumers [use this flag if there are a number of unused consumers on the source cluster]" 14 | echo " --input-file [file containing a list of target consumer groups, if this isn't provided all consumers will be compared]" 15 | 16 | return 0 17 | } 18 | 19 | if [[ $# -eq 0 ]];then 20 | echo "No input arguments provided." 21 | show_usage 22 | exit 1 23 | fi 24 | 25 | echo "============================================================================" 26 | echo "" 27 | echo "This script validates that consumer offsets are successfully synced over the cluster link from the source to destination" 28 | echo "" 29 | 30 | echo "============================================================================" 31 | echo "=============================== PARAMETERS =================================" 32 | echo "" 33 | 34 | while [ ! -z "$1" ] 35 | do 36 | if [[ "$1" == "--help" ]] 37 | then 38 | show_usage 39 | exit 0 40 | elif [[ "$1" == "--src-bootstrap-server" ]] 41 | then 42 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 43 | then 44 | echo "No Value provided for "$1". Please ensure proper values are provided" 45 | show_usage 46 | exit 1 47 | fi 48 | SRC_BOOTSTRAP_SERVERS="$2" 49 | echo "Source Cluster Bootstrap Servers are: ${SRC_BOOTSTRAP_SERVERS}" 50 | shift 51 | elif [[ "$1" == "--ccloud-bootstrap-server" ]] 52 | then 53 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 54 | then 55 | echo "No Value provided for "$1". Please ensure proper values are provided" 56 | show_usage 57 | exit 1 58 | fi 59 | CCLOUD_BOOTSTRAP_SERVERS="$2" 60 | echo "CCloud Bootstrap Servers are: ${CCLOUD_BOOTSTRAP_SERVERS}" 61 | shift 62 | elif [[ "$1" == "--command-config" ]] 63 | then 64 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 65 | then 66 | echo "No Value provided for "$1". Please ensure proper values are provided" 67 | show_usage 68 | exit 1 69 | fi 70 | CCLOUD_COMMAND_CONFIG="$2" 71 | echo "CCloud Command Config File path is: ${CCLOUD_COMMAND_CONFIG}" 72 | shift 73 | elif [[ "$1" == "--input-file" ]] 74 | then 75 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 76 | then 77 | echo "No Value provided for "$1". Please ensure proper values are provided" 78 | show_usage 79 | exit 1 80 | fi 81 | INPUT_FILE="$2" 82 | echo "File with the list of target consumer groups: ${INPUT_FILE}" 83 | elif [[ "$1" == "--remove-unused-consumers" ]] 84 | then 85 | REMOVE_UNUSED_CONSUMERS=true 86 | echo "Filtering unused source consumer group data: true" 87 | fi 88 | shift 89 | done 90 | 91 | if [[ -z "$SRC_BOOTSTRAP_SERVERS" ]] || [[ -z "$CCLOUD_BOOTSTRAP_SERVERS" ]] || [[ -z "$CCLOUD_COMMAND_CONFIG" ]] 92 | then 93 | echo "--src-bootstrap-server, --ccloud-bootstrap-server, and --command-config are required for execution." 94 | show_usage 95 | exit 1 96 | fi 97 | 98 | if [[ -n "$INPUT_FILE" ]] 99 | then 100 | CONSUMER_GROUPS=(${PWD}/`date +"%d-%m-%y-%T"`_target_consumer_groups.txt) 101 | DIFF=(${PWD}/`date +"%d-%m-%y-%T"`_diff.txt) 102 | SRC_GROUP_DATA=(${PWD}/src_group_data.txt) 103 | CCLOUD_GROUP_DATA=(${PWD}/ccloud_group_data.txt) 104 | PARSED_SRC_GROUP_DATA=(${PWD}/parsed_src_group_data.txt) 105 | PARSED_CCLOUD_GROUP_DATA=(${PWD}/parsed_ccloud_group_data.txt) 106 | 107 | echo "============================================================================" 108 | echo "=============================== SRC CONSUMERS ==============================" 109 | echo "" 110 | echo "Displaying list of target consumer groups from the input file" 111 | echo "" 112 | cat "$INPUT_FILE" 113 | cat "$INPUT_FILE" > ${CONSUMER_GROUPS} 114 | 115 | echo "" 116 | echo "============================================================================" 117 | echo "=================================== DIFF ===================================" 118 | echo "" 119 | echo "Comparing consumer group data between the OSK and CCloud clusters to validate the syncing" 120 | echo "Output written to" `date +"%d-%m-%y-%T"`"_diff.txt" 121 | echo "" 122 | 123 | if [[ "$REMOVE_UNUSED_CONSUMERS" = true ]] 124 | then 125 | echo "================================================" 126 | echo "||Removing inactive consumer group information||" 127 | echo "================================================" 128 | cat ${CONSUMER_GROUPS} | while read -r group_name 129 | do 130 | if [[ -z "${group_name}" ]] 131 | then 132 | : 133 | else 134 | # Describe the Consumer Groups in SRC and DEST 135 | kafka-consumer-groups --bootstrap-server ${SRC_BOOTSTRAP_SERVERS} --group ${group_name} --describe > ${SRC_GROUP_DATA} 136 | kafka-consumer-groups --bootstrap-server ${CCLOUD_BOOTSTRAP_SERVERS} --command-config ${CCLOUD_COMMAND_CONFIG} --group ${group_name} --describe > ${CCLOUD_GROUP_DATA} 137 | 138 | # Parse the Consumer Group data to only include columns expected to match, remove unneeded consumer group data, remove the offset data for unused consumers that is not synced over the link, and sort the data on the partition number to prepare for the diff 139 | printf "======================= SRC =======================\n" > ${PARSED_SRC_GROUP_DATA} 140 | printf "======================= DEST =======================\n" > ${PARSED_CCLOUD_GROUP_DATA} 141 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '$col4 == "-" { next } {print $col1, $col2, $col3, $col4, $col6}' ${SRC_GROUP_DATA} | sort -t, -nk3 | column -t >> ${PARSED_SRC_GROUP_DATA} 142 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '$col4 == "-" { next } {print $col1, $col2, $col3, $col4, $col6}' ${CCLOUD_GROUP_DATA} | sort -t, -nk3 | column -t >> ${PARSED_CCLOUD_GROUP_DATA} 143 | 144 | # Diff the two sorted files 145 | printf "=========================== Group Name: %s ============================\n" "${group_name}" >> ${DIFF} 146 | diff -w ${PARSED_SRC_GROUP_DATA} ${PARSED_CCLOUD_GROUP_DATA} >> ${DIFF} 147 | printf "\n" >> ${DIFF} 148 | fi 149 | done 150 | else 151 | cat ${CONSUMER_GROUPS} | while read -r group_name 152 | do 153 | if [[ -z "${group_name}" ]] 154 | then 155 | : 156 | else 157 | # Describe the Consumer Groups in SRC and DEST 158 | kafka-consumer-groups --bootstrap-server ${SRC_BOOTSTRAP_SERVERS} --group ${group_name} --describe > ${SRC_GROUP_DATA} 159 | kafka-consumer-groups --bootstrap-server ${CCLOUD_BOOTSTRAP_SERVERS} --command-config ${CCLOUD_COMMAND_CONFIG} --group ${group_name} --describe > ${CCLOUD_GROUP_DATA} 160 | 161 | # Parse the Consumer Group data to only include columns expected to match, remove unneeded consumer group data, and sort the data on the partition number to prepare for the diff 162 | printf "======================= SRC =======================\n" > ${PARSED_SRC_GROUP_DATA} 163 | printf "======================= DEST =======================\n" > ${PARSED_CCLOUD_GROUP_DATA} 164 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '{print $col1, $col2, $col3, $col4, $col6}' ${SRC_GROUP_DATA} | sort -t, -nk3 | column -t >> ${PARSED_SRC_GROUP_DATA} 165 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '{print $col1, $col2, $col3, $col4, $col6}' ${CCLOUD_GROUP_DATA} | sort -t, -nk3 | column -t >> ${PARSED_CCLOUD_GROUP_DATA} 166 | 167 | # Diff the two sorted files 168 | printf "=========================== Group Name: %s ============================\n" "${group_name}" >> ${DIFF} 169 | diff -w ${PARSED_SRC_GROUP_DATA} ${PARSED_CCLOUD_GROUP_DATA} >> ${DIFF} 170 | printf "\n" >> ${DIFF} 171 | fi 172 | done 173 | fi 174 | 175 | # Cleanup temp files 176 | rm ${CONSUMER_GROUPS} 177 | rm ${SRC_GROUP_DATA} 178 | rm ${CCLOUD_GROUP_DATA} 179 | rm ${PARSED_SRC_GROUP_DATA} 180 | rm ${PARSED_CCLOUD_GROUP_DATA} 181 | 182 | else 183 | DIFF=(${PWD}/`date +"%d-%m-%y-%T"`_diff.txt) 184 | SRC_GROUP_DATA=(${PWD}/src_group_data.txt) 185 | CCLOUD_GROUP_DATA=(${PWD}/ccloud_group_data.txt) 186 | PARSED_SRC_GROUP_DATA=(${PWD}/parsed_src_group_data.txt) 187 | PARSED_CCLOUD_GROUP_DATA=(${PWD}/parsed_ccloud_group_data.txt) 188 | 189 | echo "" 190 | echo "============================================================================" 191 | echo "=================================== DIFF ===================================" 192 | echo "" 193 | echo "Comparing consumer group data between the OSK and CCloud clusters to validate the syncing" 194 | echo "Output written to" `date +"%d-%m-%y-%T"`"_diff.txt" 195 | echo "" 196 | 197 | if [[ "$REMOVE_UNUSED_CONSUMERS" = true ]] 198 | then 199 | echo "================================================" 200 | echo "||Removing inactive consumer group information||" 201 | echo "================================================" 202 | 203 | # Describe the Consumer Groups in SRC and DEST 204 | kafka-consumer-groups --bootstrap-server ${SRC_BOOTSTRAP_SERVERS} --all-groups --describe > ${SRC_GROUP_DATA} 205 | kafka-consumer-groups --bootstrap-server ${CCLOUD_BOOTSTRAP_SERVERS} --command-config ${CCLOUD_COMMAND_CONFIG} --all-groups --describe > ${CCLOUD_GROUP_DATA} 206 | 207 | # Parse the Consumer Group data to only include columns expected to match, remove unneeded consumer group data, remove the offset data for unused consumers that is not synced over the link, and sort the data on the partition number to prepare for the diff 208 | printf "======================= SRC =======================\n" > ${PARSED_SRC_GROUP_DATA} 209 | printf "======================= DEST =======================\n" > ${PARSED_CCLOUD_GROUP_DATA} 210 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '$col4 == "-" { next } {print $col1, $col2, $col3, $col4, $col6}' ${SRC_GROUP_DATA} | sort -t, -nk3 | column -t >> ${PARSED_SRC_GROUP_DATA} 211 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '$col4 == "-" { next } {print $col1, $col2, $col3, $col4, $col6}' ${CCLOUD_GROUP_DATA} | sort -t, -nk3 | column -t >> ${PARSED_CCLOUD_GROUP_DATA} 212 | 213 | # Diff the two sorted files 214 | diff -w ${PARSED_SRC_GROUP_DATA} ${PARSED_CCLOUD_GROUP_DATA} >> ${DIFF} 215 | printf "\n" >> ${DIFF} 216 | 217 | else 218 | 219 | # Describe the Consumer Groups in SRC and DEST 220 | kafka-consumer-groups --bootstrap-server ${SRC_BOOTSTRAP_SERVERS} --all-groups --describe > ${SRC_GROUP_DATA} 221 | kafka-consumer-groups --bootstrap-server ${CCLOUD_BOOTSTRAP_SERVERS} --command-config ${CCLOUD_COMMAND_CONFIG} --all-groups --describe > ${CCLOUD_GROUP_DATA} 222 | 223 | # Parse the Consumer Group data to only include columns expected to match, remove unneeded consumer group data, and sort the data on the partition number to prepare for the diff 224 | printf "======================= SRC =======================\n" > ${PARSED_SRC_GROUP_DATA} 225 | printf "======================= DEST =======================\n" > ${PARSED_CCLOUD_GROUP_DATA} 226 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '{print $col1, $col2, $col3, $col4, $col6}' ${SRC_GROUP_DATA} | sort -t, -nk3 | column -t >> ${PARSED_SRC_GROUP_DATA} 227 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '{print $col1, $col2, $col3, $col4, $col6}' ${CCLOUD_GROUP_DATA} | sort -t, -nk3 | column -t >> ${PARSED_CCLOUD_GROUP_DATA} 228 | 229 | # Diff the two sorted files 230 | diff -w ${PARSED_SRC_GROUP_DATA} ${PARSED_CCLOUD_GROUP_DATA} >> ${DIFF} 231 | printf "\n" >> ${DIFF} 232 | fi 233 | 234 | # Cleanup temp files 235 | rm ${SRC_GROUP_DATA} 236 | rm ${CCLOUD_GROUP_DATA} 237 | rm ${PARSED_SRC_GROUP_DATA} 238 | rm ${PARSED_CCLOUD_GROUP_DATA} 239 | fi 240 | 241 | echo "" -------------------------------------------------------------------------------- /consumer-offset-validation.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #This script validates that consumer offsets are successfully synced over the cluster link from source to destination 3 | #The Kafka CLI commands must be installed in a location set in your PATH to run kafka-consumer-groups 4 | 5 | show_usage () { 6 | echo "Usage: $0 [options [parameters]]" 7 | echo "" 8 | echo "Options:" 9 | echo " --help -- Prints this page" 10 | echo " --ccloud-bootstrap-server [ccloud-bootstrap-server-list]" 11 | echo " --src-bootstrap-server [src-bootstrap-server-list]" 12 | echo " --command-config-ccloud [ccloud-command-config-file]" 13 | echo " --command-config-src [src-command-config-file]" 14 | echo " --remove-unused-consumers [use this flag if there are a number of unused consumers on the source cluster]" 15 | echo " --input-file [file containing a list of target consumer groups, if this isn't provided all consumers will be compared]" 16 | 17 | return 0 18 | } 19 | 20 | if [[ $# -eq 0 ]];then 21 | echo "No input arguments provided." 22 | show_usage 23 | exit 1 24 | fi 25 | 26 | echo "============================================================================" 27 | echo "" 28 | echo "This script validates that consumer offsets are successfully synced over the cluster link from the source to destination" 29 | echo "" 30 | 31 | echo "============================================================================" 32 | echo "=============================== PARAMETERS =================================" 33 | echo "" 34 | 35 | while [ ! -z "$1" ] 36 | do 37 | if [[ "$1" == "--help" ]] 38 | then 39 | show_usage 40 | exit 0 41 | elif [[ "$1" == "--ccloud-bootstrap-server" ]] 42 | then 43 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 44 | then 45 | echo "No Value provided for "$1". Please ensure proper values are provided" 46 | show_usage 47 | exit 1 48 | fi 49 | CCLOUD_BOOTSTRAP_SERVERS="$2" 50 | echo "CCloud Bootstrap Servers are: ${CCLOUD_BOOTSTRAP_SERVERS}" 51 | shift 52 | elif [[ "$1" == "--src-bootstrap-server" ]] 53 | then 54 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 55 | then 56 | echo "No Value provided for "$1". Please ensure proper values are provided" 57 | show_usage 58 | exit 1 59 | fi 60 | SRC_BOOTSTRAP_SERVERS="$2" 61 | echo "Source Cluster Bootstrap Servers are: ${SRC_BOOTSTRAP_SERVERS}" 62 | shift 63 | elif [[ "$1" == "--command-config-ccloud" ]] 64 | then 65 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 66 | then 67 | echo "No Value provided for "$1". Please ensure proper values are provided" 68 | show_usage 69 | exit 1 70 | fi 71 | COMMAND_CONFIG_CCLOUD="$2" 72 | echo "Destination Command Config File path is: ${COMMAND_CONFIG_CCLOUD}" 73 | shift 74 | elif [[ "$1" == "--command-config-src" ]] 75 | then 76 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 77 | then 78 | echo "No Value provided for "$1". Please ensure proper values are provided" 79 | show_usage 80 | exit 1 81 | fi 82 | COMMAND_CONFIG_SRC="$2" 83 | echo "Source Command Config File path is: ${COMMAND_CONFIG_SRC}" 84 | shift 85 | elif [[ "$1" == "--input-file" ]] 86 | then 87 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 88 | then 89 | echo "No Value provided for "$1". Please ensure proper values are provided" 90 | show_usage 91 | exit 1 92 | fi 93 | INPUT_FILE="$2" 94 | echo "File with the list of target consumer groups: ${INPUT_FILE}" 95 | elif [[ "$1" == "--remove-unused-consumers" ]] 96 | then 97 | REMOVE_UNUSED_CONSUMERS=true 98 | echo "Filtering unused source consumer group data: true" 99 | fi 100 | shift 101 | done 102 | 103 | echo "" 104 | 105 | if [[ -z "$CCLOUD_BOOTSTRAP_SERVERS" ]] || [[ -z "$SRC_BOOTSTRAP_SERVERS" ]] || [[ -z "$COMMAND_CONFIG_CCLOUD" ]] || [[ -z "$COMMAND_CONFIG_SRC" ]] 106 | then 107 | echo "--ccloud-bootstrap-server, --src-bootstrap-server, --command-config-ccloud, and --command-config-src are required for execution." 108 | show_usage 109 | exit 1 110 | fi 111 | 112 | if [[ -n "$INPUT_FILE" ]] 113 | then 114 | CONSUMER_GROUPS=(${PWD}/`date +"%d-%m-%y-%T"`_target_consumer_groups.txt) 115 | DIFF=(${PWD}/`date +"%d-%m-%y-%T"`_diff.txt) 116 | SRC_GROUP_DATA=(${PWD}/src_group_data.txt) 117 | CCLOUD_GROUP_DATA=(${PWD}/ccloud_group_data.txt) 118 | PARSED_SRC_GROUP_DATA=(${PWD}/parsed_src_group_data.txt) 119 | PARSED_CCLOUD_GROUP_DATA=(${PWD}/parsed_ccloud_group_data.txt) 120 | 121 | echo "============================================================================" 122 | echo "=============================== SRC CONSUMERS ==============================" 123 | echo "" 124 | echo "Displaying list of target consumer groups from the input file:" 125 | echo "" 126 | cat "$INPUT_FILE" 127 | cat "$INPUT_FILE" > ${CONSUMER_GROUPS} 128 | 129 | echo "" 130 | echo "============================================================================" 131 | echo "=================================== DIFF ===================================" 132 | echo "" 133 | echo "Comparing consumer group data between the Source and CCloud clusters to validate the syncing" 134 | echo "Output written to" `date +"%d-%m-%y-%T"`"_diff.txt" 135 | echo "" 136 | 137 | if [[ "$REMOVE_UNUSED_CONSUMERS" = true ]] 138 | then 139 | echo "================================================" 140 | echo "||Removing inactive consumer group information||" 141 | echo "================================================" 142 | cat ${CONSUMER_GROUPS} | while read -r group_name 143 | do 144 | if [[ -z "${group_name}" ]] 145 | then 146 | : 147 | else 148 | # Describe the Consumer Groups in SRC and DEST 149 | kafka-consumer-groups --bootstrap-server ${SRC_BOOTSTRAP_SERVERS} --command-config ${COMMAND_CONFIG_SRC} --group ${group_name} --describe > ${SRC_GROUP_DATA} 150 | kafka-consumer-groups --bootstrap-server ${CCLOUD_BOOTSTRAP_SERVERS} --command-config ${COMMAND_CONFIG_CCLOUD} --group ${group_name} --describe > ${CCLOUD_GROUP_DATA} 151 | 152 | # Parse the Consumer Group data to only include columns expected to match, remove unneeded consumer group data, remove the offset data for unused consumers that is not synced over the link, and sort the data on the partition number to prepare for the diff 153 | printf "======================= SRC =======================\n" > ${PARSED_SRC_GROUP_DATA} 154 | printf "======================= DEST =======================\n" > ${PARSED_CCLOUD_GROUP_DATA} 155 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '$col4 == "-" { next } {print $col1, $col2, $col3, $col4, $col6}' ${SRC_GROUP_DATA} | sort -t, -nk3 | column -t >> ${PARSED_SRC_GROUP_DATA} 156 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '$col4 == "-" { next } {print $col1, $col2, $col3, $col4, $col6}' ${CCLOUD_GROUP_DATA} | sort -t, -nk3 | column -t >> ${PARSED_CCLOUD_GROUP_DATA} 157 | 158 | # Diff the two sorted files 159 | printf "=========================== Group Name: %s ============================\n" "${group_name}" >> ${DIFF} 160 | diff -w ${PARSED_SRC_GROUP_DATA} ${PARSED_CCLOUD_GROUP_DATA} >> ${DIFF} 161 | printf "\n" >> ${DIFF} 162 | fi 163 | done 164 | else 165 | cat ${CONSUMER_GROUPS} | while read -r group_name 166 | do 167 | if [[ -z "${group_name}" ]] 168 | then 169 | : 170 | else 171 | # Describe the Consumer Groups in SRC and DEST 172 | kafka-consumer-groups --bootstrap-server ${SRC_BOOTSTRAP_SERVERS} --command-config ${COMMAND_CONFIG_SRC} --group ${group_name} --describe > ${SRC_GROUP_DATA} 173 | kafka-consumer-groups --bootstrap-server ${CCLOUD_BOOTSTRAP_SERVERS} --command-config ${COMMAND_CONFIG_CCLOUD} --group ${group_name} --describe > ${CCLOUD_GROUP_DATA} 174 | 175 | # Parse the Consumer Group data to only include columns expected to match, remove unneeded consumer group data, and sort the data on the partition number to prepare for the diff 176 | printf "======================= SRC =======================\n" > ${PARSED_SRC_GROUP_DATA} 177 | printf "======================= DEST =======================\n" > ${PARSED_CCLOUD_GROUP_DATA} 178 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '{print $col1, $col2, $col3, $col4, $col6}' ${SRC_GROUP_DATA} | sort -t, -nk3 | column -t >> ${PARSED_SRC_GROUP_DATA} 179 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '{print $col1, $col2, $col3, $col4, $col6}' ${CCLOUD_GROUP_DATA} | sort -t, -nk3 | column -t >> ${PARSED_CCLOUD_GROUP_DATA} 180 | 181 | # Diff the two sorted files 182 | printf "=========================== Group Name: %s ============================\n" "${group_name}" >> ${DIFF} 183 | diff -w ${PARSED_SRC_GROUP_DATA} ${PARSED_CCLOUD_GROUP_DATA} >> ${DIFF} 184 | printf "\n" >> ${DIFF} 185 | fi 186 | done 187 | fi 188 | 189 | # Cleanup temp files 190 | rm ${CONSUMER_GROUPS} 191 | rm ${SRC_GROUP_DATA} 192 | rm ${CCLOUD_GROUP_DATA} 193 | rm ${PARSED_SRC_GROUP_DATA} 194 | rm ${PARSED_CCLOUD_GROUP_DATA} 195 | 196 | else 197 | DIFF=(${PWD}/`date +"%d-%m-%y-%T"`_diff.txt) 198 | SRC_GROUP_DATA=(${PWD}/src_group_data.txt) 199 | CCLOUD_GROUP_DATA=(${PWD}/ccloud_group_data.txt) 200 | PARSED_SRC_GROUP_DATA=(${PWD}/parsed_src_group_data.txt) 201 | PARSED_CCLOUD_GROUP_DATA=(${PWD}/parsed_ccloud_group_data.txt) 202 | 203 | echo "" 204 | echo "============================================================================" 205 | echo "=================================== DIFF ===================================" 206 | echo "" 207 | echo "Comparing consumer group data between the Source and CCloud clusters to validate the syncing" 208 | echo "Output written to" `date +"%d-%m-%y-%T"`"_diff.txt" 209 | echo "" 210 | 211 | if [[ "$REMOVE_UNUSED_CONSUMERS" = true ]] 212 | then 213 | echo "================================================" 214 | echo "||Removing inactive consumer group information||" 215 | echo "================================================" 216 | 217 | # Describe the Consumer Groups in SRC and DEST 218 | kafka-consumer-groups --bootstrap-server ${SRC_BOOTSTRAP_SERVERS} --command-config ${COMMAND_CONFIG_SRC} --all-groups --describe > ${SRC_GROUP_DATA} 219 | kafka-consumer-groups --bootstrap-server ${CCLOUD_BOOTSTRAP_SERVERS} --command-config ${COMMAND_CONFIG_CCLOUD} --all-groups --describe > ${CCLOUD_GROUP_DATA} 220 | 221 | # Parse the Consumer Group data to only include columns expected to match, remove unneeded consumer group data, remove the offset data for unused consumers that is not synced over the link, and sort the data on the partition number to prepare for the diff 222 | printf "======================= SRC =======================\n" > ${PARSED_SRC_GROUP_DATA} 223 | printf "======================= DEST =======================\n" > ${PARSED_CCLOUD_GROUP_DATA} 224 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '$col4 == "-" { next } {print $col1, $col2, $col3, $col4, $col6}' ${SRC_GROUP_DATA} | sort -t, -nk3 | column -t >> ${PARSED_SRC_GROUP_DATA} 225 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '$col4 == "-" { next } {print $col1, $col2, $col3, $col4, $col6}' ${CCLOUD_GROUP_DATA} | sort -t, -nk3 | column -t >> ${PARSED_CCLOUD_GROUP_DATA} 226 | 227 | # Diff the two sorted files 228 | diff -w ${PARSED_SRC_GROUP_DATA} ${PARSED_CCLOUD_GROUP_DATA} >> ${DIFF} 229 | printf "\n" >> ${DIFF} 230 | 231 | else 232 | 233 | # Describe the Consumer Groups in SRC and DEST 234 | kafka-consumer-groups --bootstrap-server ${SRC_BOOTSTRAP_SERVERS} --command-config ${COMMAND_CONFIG_SRC} --all-groups --describe > ${SRC_GROUP_DATA} 235 | kafka-consumer-groups --bootstrap-server ${CCLOUD_BOOTSTRAP_SERVERS} --command-config ${COMMAND_CONFIG_CCLOUD} --all-groups --describe > ${CCLOUD_GROUP_DATA} 236 | 237 | # Parse the Consumer Group data to only include columns expected to match, remove unneeded consumer group data, and sort the data on the partition number to prepare for the diff 238 | printf "======================= SRC =======================\n" > ${PARSED_SRC_GROUP_DATA} 239 | printf "======================= DEST =======================\n" > ${PARSED_CCLOUD_GROUP_DATA} 240 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '{print $col1, $col2, $col3, $col4, $col6}' ${SRC_GROUP_DATA} | sort -t, -nk3 | column -t >> ${PARSED_SRC_GROUP_DATA} 241 | awk -v col1=1 -v col2=2 -v col3=3 -v col4=4 -v col5=5 -v col6=6 '{print $col1, $col2, $col3, $col4, $col6}' ${CCLOUD_GROUP_DATA} | sort -t, -nk3 | column -t >> ${PARSED_CCLOUD_GROUP_DATA} 242 | 243 | # Diff the two sorted files 244 | diff -w ${PARSED_SRC_GROUP_DATA} ${PARSED_CCLOUD_GROUP_DATA} >> ${DIFF} 245 | printf "\n" >> ${DIFF} 246 | fi 247 | 248 | # Cleanup temp files 249 | rm ${SRC_GROUP_DATA} 250 | rm ${CCLOUD_GROUP_DATA} 251 | rm ${PARSED_SRC_GROUP_DATA} 252 | rm ${PARSED_CCLOUD_GROUP_DATA} 253 | fi 254 | 255 | echo "" -------------------------------------------------------------------------------- /example-consumer-filter.txt: -------------------------------------------------------------------------------- 1 | cg-test-app1 2 | cg-test2-app1 3 | cg-test2-app2 4 | -------------------------------------------------------------------------------- /example-promote.txt: -------------------------------------------------------------------------------- 1 | test 2 | test2 3 | test3 4 | test4 5 | -------------------------------------------------------------------------------- /failover-mirror-topic.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #Ensure the confluent CLI is installed and referenced in your PATH or set the CONFLUENT_HOME_BIN environment variable to the location of the binary 3 | #Cluster link must be running and mirror topics must exist on the destination cluster 4 | 5 | show_usage () { 6 | echo "Usage: $0 [options [parameters]]" 7 | echo "" 8 | echo "Options:" 9 | echo " --help -- Prints this page" 10 | echo " --input-file [input-file]" 11 | echo " --link-id [link-name]" 12 | echo " --cluster [ccloud-cluster-id]" 13 | echo " --environment [ccloud-environment-id]" 14 | echo " --all-active-mirror-topics [use this flag to failover all remaining active mirror topics]" 15 | return 0 16 | } 17 | 18 | if [[ $# -eq 0 ]];then 19 | echo "No input arguments provided." 20 | show_usage 21 | exit 1 22 | fi 23 | 24 | echo "============================================================================" 25 | echo "" 26 | echo "This script accepts a list of mirror topics to fail over or fails over all mirror topics, and helps validate successful fail over" 27 | echo "" 28 | 29 | MIRROR_TOPIC_STATUS_FILE=(${PWD}/`date +"%d-%m-%y-%T"`_mirror_topic_status.txt) 30 | mirrorTopics=() 31 | 32 | while [ ! -z "$1" ] 33 | do 34 | if [[ "$1" == "--help" ]] 35 | then 36 | show_usage 37 | exit 0 38 | elif [[ "$1" == "--input-file" ]] 39 | then 40 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 41 | then 42 | echo "No Value provided for "$1". Please ensure proper values are provided" 43 | show_usage 44 | exit 1 45 | fi 46 | inputFile="$2" 47 | shift 48 | elif [[ "$1" == "--link-id" ]] 49 | then 50 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 51 | then 52 | echo "No Value provided for "$1". Please ensure proper values are provided" 53 | show_usage 54 | exit 1 55 | fi 56 | linkId="$2" 57 | echo "Cluster Link Name: ${linkId}" 58 | shift 59 | elif [[ "$1" == "--cluster" ]] 60 | then 61 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 62 | then 63 | echo "No Value provided for "$1". Please ensure proper values are provided" 64 | show_usage 65 | exit 1 66 | fi 67 | clusterId="$2" 68 | echo "CCloud Kafka Cluster ID: ${clusterId}" 69 | shift 70 | elif [[ "$1" == "--environment" ]] 71 | then 72 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 73 | then 74 | echo "No Value provided for "$1". Please ensure proper values are provided" 75 | show_usage 76 | exit 1 77 | fi 78 | environmentId="$2" 79 | echo "CCloud Environment ID: ${environmentId}" 80 | shift 81 | elif [[ "$1" == "--all-active-mirror-topics" ]] 82 | then 83 | ACTIVE_MIRROR_TOPICS_FLAG=true 84 | echo "Failing over all active mirror topics: true" 85 | fi 86 | shift 87 | done 88 | 89 | if [[ -z "$linkId" ]] || [[ -z "$clusterId" ]] || [[ -z "$environmentId" ]] || [[ (! -z "$inputFile" || "$ACTIVE_MIRROR_TOPICS_FLAG" != "true") ]] 90 | then 91 | echo "" 92 | echo "Invalid usage:" 93 | echo "--input-file or --all-active-mirror-topics, --link-id, --cluster, and --environment are required for execution." 94 | echo "" 95 | show_usage 96 | exit 1 97 | fi 98 | 99 | describe_mirror_topic () { 100 | 101 | for index in "${!mirrorTopics[@]}"; 102 | do 103 | echo "Mirror Topic: ${mirrorTopics[$index]}" 104 | echo "Command: confluent kafka mirror describe ${mirrorTopics[$index]} --link ${linkId} --cluster ${clusterId} --environment ${environmentId}" 105 | echo "" 106 | confluent kafka mirror describe ${mirrorTopics[$index]} --link ${linkId} --cluster ${clusterId} --environment ${environmentId} 107 | echo "" 108 | done 109 | return 0 110 | } 111 | 112 | describe_mirror_topic_write () { 113 | 114 | for index in "${!mirrorTopics[@]}"; 115 | do 116 | confluent kafka mirror describe ${mirrorTopics[$index]} --link ${linkId} --cluster ${clusterId} --environment ${environmentId} | tee --append ${MIRROR_TOPIC_STATUS_FILE} 117 | done 118 | return 0 119 | } 120 | 121 | #describe_mirror_topic_write_test () { 122 | # #TEST BLOCK 123 | # confluent kafka mirror describe test6 --link ${linkId} --cluster ${clusterId} --environment ${environmentId} >> ${MIRROR_TOPIC_STATUS_FILE} 124 | # #TEST BLOCK 125 | #} 126 | 127 | if [[ ! -z "$inputFile" ]] 128 | then 129 | if [[ ! -s ${inputFile} ]] 130 | then 131 | echo "No topics to failover!" 132 | exit 1 133 | fi 134 | while IFS= read -r line || [[ "$line" ]]; 135 | do 136 | mirrorTopics+=("$line") 137 | done < ${inputFile} 138 | 139 | ##TEST BLOCK 140 | # confluent kafka mirror list > tmp-list-mirror.txt 141 | # 142 | # #query all the active mirrors 143 | # awk -v topic=3 -v status=11 '{if ($status != "STOPPED" && $topic != "|" && $topic != "") {print $topic}}' tmp-list-mirror.txt > tmp-input.txt 144 | ##TEST BLOCK 145 | 146 | elif [[ "$ACTIVE_MIRROR_TOPICS_FLAG" = true ]] 147 | then 148 | #retrieve mirror topic list 149 | echo "================================================================" 150 | echo "" 151 | echo "Retrieving list of mirror topics that are not in a Stopped state" 152 | 153 | confluent kafka mirror list > tmp-list-mirror.txt 154 | 155 | #query all the active mirrors 156 | awk -v topic=3 -v status=11 '{if ($status != "STOPPED" && $topic != "|" && $topic != "") {print $topic}}' tmp-list-mirror.txt > tmp-input.txt 157 | 158 | if [[ ! -s tmp-input.txt ]] 159 | then 160 | echo "" 161 | echo "No topics to failover!" 162 | echo "" 163 | #clean up the temp files 164 | rm tmp-input.txt 165 | rm tmp-list-mirror.txt 166 | exit 1 167 | fi 168 | 169 | while IFS= read -r line || [[ "$line" ]]; 170 | do 171 | mirrorTopics+=("$line") 172 | done < tmp-input.txt 173 | 174 | #clean up the temp files 175 | rm tmp-input.txt 176 | rm tmp-list-mirror.txt 177 | else 178 | echo "" 179 | echo "No input file or active mirror topics flags found" 180 | echo "" 181 | show_usage 182 | exit 1 183 | fi 184 | 185 | #Describe takes in one topic at a time 186 | echo "" 187 | echo "============================================================================" 188 | echo "=========== Displaying mirror topic current status pre-failover ===========" 189 | echo "" 190 | 191 | #describe_mirror_topic 192 | 193 | #Promoting mirror topics 194 | echo "" 195 | echo "============================================================================" 196 | echo "========================= Failing over mirror topics ==========================" 197 | echo "" 198 | echo "Command: confluent kafka mirror promote ${mirrorTopics[*]} --link ${linkId} --cluster ${clusterId} --environment ${environmentId}" 199 | echo "" 200 | 201 | confluent kafka mirror failover ${mirrorTopics[*]} --link ${linkId} --cluster ${clusterId} --environment ${environmentId} 202 | 203 | #Describe topics again after they've been promoted 204 | echo "" 205 | sleep 15 206 | echo "Wait while the topics are promoted" 207 | echo "" 208 | echo "=========== Displaying mirror topic current status post-failover ==========" 209 | echo "" 210 | 211 | #describe_mirror_topic 212 | describe_mirror_topic_write 213 | #describe_mirror_topic_write_test 214 | 215 | echo "=========== Displaying target mirror topics still pending failover ==========" 216 | echo "" 217 | awk -v topic=3 -v partition=5 -v status=11 'NR==1 {print "TOPIC","PARTITION","STATUS"} {if ($status != "STOPPED" && $topic != "|" && $topic != "") {gsub(/"/,""); print $topic, $partition, $status}}' ${MIRROR_TOPIC_STATUS_FILE} | column -t > tmp-input-mirror-state.txt 218 | cat tmp-input-mirror-state.txt 219 | rm tmp-input-mirror-state.txt 220 | 221 | echo "" 222 | -------------------------------------------------------------------------------- /mirroring-validation.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #This script can be used to validate that all topic partitions are being mirrored over the cluster link 3 | #The Confluent CLI commands must be installed in a location set in your PATH to run confluent kafka mirror describe 4 | 5 | show_usage () { 6 | echo "Usage: $0 [options [parameters]]" 7 | echo "" 8 | echo "Options:" 9 | echo " --help -- Prints this page" 10 | echo " --input-file [input-file]" 11 | echo " --link-id [link-name]" 12 | echo " --cluster [ccloud-cluster-id]" 13 | echo " --environment [ccloud-environment-id]" 14 | echo " --all-active-mirror-topics [use this flag to describe all mirror topics]" 15 | 16 | return 0 17 | } 18 | 19 | if [[ $# -eq 0 ]];then 20 | echo "No input arguments provided." 21 | show_usage 22 | exit 1 23 | fi 24 | 25 | echo "============================================================================" 26 | echo "" 27 | echo "This script accepts a list of mirror topics to promote and help validate successful promotion" 28 | echo "" 29 | 30 | MIRROR_TOPIC_STATUS_FILE=(${PWD}/`date +"%d-%m-%y-%T"`_mirror_topic_describe.txt) 31 | mirrorTopics=() 32 | 33 | while [ ! -z "$1" ] 34 | do 35 | if [[ "$1" == "--help" ]] 36 | then 37 | show_usage 38 | exit 0 39 | elif [[ "$1" == "--input-file" ]] 40 | then 41 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 42 | then 43 | echo "No Value provided for "$1". Please ensure proper values are provided" 44 | show_usage 45 | exit 1 46 | fi 47 | inputFile="$2" 48 | shift 49 | elif [[ "$1" == "--link-id" ]] 50 | then 51 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 52 | then 53 | echo "No Value provided for "$1". Please ensure proper values are provided" 54 | show_usage 55 | exit 1 56 | fi 57 | linkId="$2" 58 | echo "Cluster Link Name: ${linkId}" 59 | shift 60 | elif [[ "$1" == "--cluster" ]] 61 | then 62 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 63 | then 64 | echo "No Value provided for "$1". Please ensure proper values are provided" 65 | show_usage 66 | exit 1 67 | fi 68 | clusterId="$2" 69 | echo "CCloud Kafka Cluster ID: ${clusterId}" 70 | shift 71 | elif [[ "$1" == "--environment" ]] 72 | then 73 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 74 | then 75 | echo "No Value provided for "$1". Please ensure proper values are provided" 76 | show_usage 77 | exit 1 78 | fi 79 | environmentId="$2" 80 | echo "CCloud Environment ID: ${environmentId}" 81 | elif [[ "$1" == "--all-active-mirror-topics" ]] 82 | then 83 | activeMirrorTopicsFlag=true 84 | echo "Promoting all active mirror topics: true" 85 | fi 86 | shift 87 | done 88 | 89 | if [[ -z "$linkId" ]] || [[ -z "$clusterId" ]] || [[ -z "$environmentId" ]] || [[ (-z "$inputFile" && "$activeMirrorTopicsFlag" != "true") ]] 90 | then 91 | echo "--input-file or --all-active-mirror-topics, --link-id, --cluster, and --environment are required for execution." 92 | show_usage 93 | exit 1 94 | fi 95 | 96 | describe_mirror_topic () { 97 | 98 | for index in "${!mirrorTopics[@]}"; 99 | do 100 | confluent kafka mirror describe ${mirrorTopics[$index]} --link ${linkId} --cluster ${clusterId} --environment ${environmentId} | tee --append ${MIRROR_TOPIC_STATUS_FILE} 101 | done 102 | return 0 103 | } 104 | 105 | if [[ ! -z "$inputFile" ]] 106 | then 107 | if [[ ! -s ${inputFile} ]] 108 | then 109 | echo "No topics to promote!" 110 | exit 1 111 | fi 112 | while IFS= read -r line || [[ "$line" ]]; 113 | do 114 | mirrorTopics+=("$line") 115 | done < ${inputFile} 116 | 117 | elif [[ "$activeMirrorTopicsFlag" = true ]] 118 | then 119 | #retrieve mirror topic list 120 | echo "================================================================" 121 | echo "" 122 | echo "Retrieving list of mirror topics that are not in a Stopped state" 123 | 124 | confluent kafka mirror list --link ${linkId} --cluster ${clusterId} --environment ${environmentId} > tmp-list-mirror.txt 125 | 126 | #query all the active mirrors 127 | awk -v topic=3 -v status=11 '{if ($status != "STOPPED" && $topic != "|" && $topic != "") {print $topic}}' tmp-list-mirror.txt > tmp-input.txt 128 | 129 | if [[ ! -s tmp-input.txt ]] 130 | then 131 | echo "" 132 | echo "No active mirror topics!" 133 | echo "" 134 | #clean up the temp files 135 | rm tmp-input.txt 136 | rm tmp-list-mirror.txt 137 | exit 1 138 | fi 139 | 140 | while IFS= read -r line || [[ "$line" ]]; 141 | do 142 | mirrorTopics+=("$line") 143 | done < tmp-input.txt 144 | 145 | #clean up the temp files 146 | rm tmp-input.txt 147 | rm tmp-list-mirror.txt 148 | else 149 | echo "" 150 | echo "No input file or all active mirror topics flags found" 151 | echo "" 152 | show_usage 153 | exit 1 154 | fi 155 | 156 | echo "" 157 | echo "============================================================================" 158 | echo "============== Describing mirror topics to validate mirroring ==============" 159 | echo "" 160 | 161 | describe_mirror_topic 162 | 163 | echo "" 164 | echo "============================================================================" 165 | echo "======== Parsing output file to validate all partitions are active =========" 166 | echo "" 167 | 168 | awk -F '|' -v topic=2 -v partition=3 -v mirror_lag=4 -v status=6 -v last_fetch_offset=8 'NR==1 {print "TOPIC","PARTITION","MIRROR_LAG","STATUS","LAST_FETCH_OFFSET"} {if ($status != "STOPPED" && $mirror_lag==$last_fetch_offset) {print $topic, $partition, $mirror_lag, $status, $last_fetch_offset}}' ${MIRROR_TOPIC_STATUS_FILE} | column -t > `date +"%d-%m-%y-%T"`-input-mirror-state.txt 169 | #cat tmp-input-mirror-state.txt 170 | #rm tmp-input-mirror-state.txt 171 | 172 | echo "" -------------------------------------------------------------------------------- /promote-mirror-topic.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #Ensure the confluent CLI is installed and referenced in your PATH or set the CONFLUENT_HOME_BIN environment variable to the location of the binary 3 | #Cluster link must be running and mirror topics must exist on the destination cluster 4 | 5 | show_usage () { 6 | echo "Usage: $0 [options [parameters]]" 7 | echo "" 8 | echo "Options:" 9 | echo " --help -- Prints this page" 10 | echo " --input-file [input-file]" 11 | echo " --link-id [link-name]" 12 | echo " --cluster [ccloud-cluster-id]" 13 | echo " --environment [ccloud-environment-id]" 14 | echo " --all-active-mirror-topics [use this flag to promote all remaining active mirror topics]" 15 | return 0 16 | } 17 | 18 | if [[ $# -eq 0 ]];then 19 | echo "No input arguments provided." 20 | show_usage 21 | exit 1 22 | fi 23 | 24 | echo "============================================================================" 25 | echo "" 26 | echo "This script accepts a list of mirror topics to promote and help validate successful promotion" 27 | echo "" 28 | 29 | MIRROR_TOPIC_STATUS_FILE=(${PWD}/`date +"%d-%m-%y-%T"`_mirror_topic_status.txt) 30 | mirrorTopics=() 31 | 32 | while [ ! -z "$1" ] 33 | do 34 | if [[ "$1" == "--help" ]] 35 | then 36 | show_usage 37 | exit 0 38 | elif [[ "$1" == "--input-file" ]] 39 | then 40 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 41 | then 42 | echo "No Value provided for "$1". Please ensure proper values are provided" 43 | show_usage 44 | exit 1 45 | fi 46 | inputFile="$2" 47 | shift 48 | elif [[ "$1" == "--link-id" ]] 49 | then 50 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 51 | then 52 | echo "No Value provided for "$1". Please ensure proper values are provided" 53 | show_usage 54 | exit 1 55 | fi 56 | linkId="$2" 57 | echo "Cluster Link Name: ${linkId}" 58 | shift 59 | elif [[ "$1" == "--cluster" ]] 60 | then 61 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 62 | then 63 | echo "No Value provided for "$1". Please ensure proper values are provided" 64 | show_usage 65 | exit 1 66 | fi 67 | clusterId="$2" 68 | echo "CCloud Kafka Cluster ID: ${clusterId}" 69 | shift 70 | elif [[ "$1" == "--environment" ]] 71 | then 72 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 73 | then 74 | echo "No Value provided for "$1". Please ensure proper values are provided" 75 | show_usage 76 | exit 1 77 | fi 78 | environmentId="$2" 79 | echo "CCloud Environment ID: ${environmentId}" 80 | elif [[ "$1" == "--all-active-mirror-topics" ]] 81 | then 82 | activeMirrorTopicsFlag=true 83 | echo "Promoting all active mirror topics: true" 84 | fi 85 | shift 86 | done 87 | 88 | if [[ -z "$linkId" ]] || [[ -z "$clusterId" ]] || [[ -z "$environmentId" ]] || [[ (-z "$inputFile" && "$activeMirrorTopicsFlag" != "true") ]] 89 | then 90 | echo "--input-file or --all-active-mirror-topics, --link-id, --cluster, and --environment are required for execution." 91 | show_usage 92 | exit 1 93 | fi 94 | 95 | describe_mirror_topic () { 96 | 97 | for index in "${!mirrorTopics[@]}"; 98 | do 99 | echo "Mirror Topic: ${mirrorTopics[$index]}" 100 | echo "Command: confluent kafka mirror describe ${mirrorTopics[$index]} --link ${linkId} --cluster ${clusterId} --environment ${environmentId}" 101 | echo "" 102 | confluent kafka mirror describe ${mirrorTopics[$index]} --link ${linkId} --cluster ${clusterId} --environment ${environmentId} 103 | echo "" 104 | done 105 | return 0 106 | } 107 | 108 | describe_mirror_topic_write () { 109 | 110 | for index in "${!mirrorTopics[@]}"; 111 | do 112 | confluent kafka mirror describe ${mirrorTopics[$index]} --link ${linkId} --cluster ${clusterId} --environment ${environmentId} | tee --append ${MIRROR_TOPIC_STATUS_FILE} 113 | done 114 | return 0 115 | } 116 | 117 | #describe_mirror_topic_write_test () { 118 | # #TEST BLOCK 119 | # confluent kafka mirror describe test6 --link ${linkId} --cluster ${clusterId} --environment ${environmentId} >> ${MIRROR_TOPIC_STATUS_FILE} 120 | # #TEST BLOCK 121 | #} 122 | 123 | if [[ ! -z "$inputFile" ]] 124 | then 125 | if [[ ! -s ${inputFile} ]] 126 | then 127 | echo "No topics to promote!" 128 | exit 1 129 | fi 130 | while IFS= read -r line || [[ "$line" ]]; 131 | do 132 | mirrorTopics+=("$line") 133 | done < ${inputFile} 134 | 135 | ##TEST BLOCK 136 | # confluent kafka mirror list > tmp-list-mirror.txt 137 | # 138 | # #query all the active mirrors 139 | # awk -v topic=3 -v status=11 '{if ($status != "STOPPED" && $topic != "|" && $topic != "") {print $topic}}' tmp-list-mirror.txt > tmp-input.txt 140 | ##TEST BLOCK 141 | 142 | elif [[ "$activeMirrorTopicsFlag" = true ]] 143 | then 144 | #retrieve mirror topic list 145 | echo "================================================================" 146 | echo "" 147 | echo "Retrieving list of mirror topics that are not in a Stopped state" 148 | 149 | confluent kafka mirror list --link ${linkId} --cluster ${clusterId} --environment ${environmentId} > tmp-list-mirror.txt 150 | 151 | #query all the active mirrors 152 | awk -v topic=3 -v status=11 '{if ($status != "STOPPED" && $topic != "|" && $topic != "") {print $topic}}' tmp-list-mirror.txt > tmp-input.txt 153 | 154 | if [[ ! -s tmp-input.txt ]] 155 | then 156 | echo "" 157 | echo "No topics to promote!" 158 | echo "" 159 | #clean up the temp files 160 | rm tmp-input.txt 161 | rm tmp-list-mirror.txt 162 | exit 1 163 | fi 164 | 165 | while IFS= read -r line || [[ "$line" ]]; 166 | do 167 | mirrorTopics+=("$line") 168 | done < tmp-input.txt 169 | 170 | #clean up the temp files 171 | rm tmp-input.txt 172 | rm tmp-list-mirror.txt 173 | else 174 | echo "" 175 | echo "No input file or active mirror topics flags found" 176 | echo "" 177 | show_usage 178 | exit 1 179 | fi 180 | 181 | #Describe takes in one topic at a time 182 | echo "" 183 | echo "============================================================================" 184 | echo "=========== Displaying mirror topic current status pre-promotion ===========" 185 | echo "" 186 | 187 | #describe_mirror_topic 188 | 189 | #Promoting mirror topics 190 | echo "" 191 | echo "============================================================================" 192 | echo "========================= Promoting mirror topics ==========================" 193 | echo "" 194 | echo "Command: confluent kafka mirror promote ${mirrorTopics[*]} --link ${linkId} --cluster ${clusterId} --environment ${environmentId}" 195 | echo "" 196 | 197 | confluent kafka mirror promote ${mirrorTopics[*]} --link ${linkId} --cluster ${clusterId} --environment ${environmentId} 198 | 199 | #Describe topics again after they've been promoted 200 | echo "" 201 | sleep 15 202 | echo "Wait while the topics are promoted" 203 | echo "" 204 | echo "=========== Displaying mirror topic current status post-promotion ==========" 205 | echo "" 206 | 207 | #describe_mirror_topic 208 | describe_mirror_topic_write 209 | #describe_mirror_topic_write_test 210 | 211 | echo "=========== Displaying target mirror topics still pending promotion ==========" 212 | echo "" 213 | awk -v topic=3 -v partition=5 -v status=11 'NR==1 {print "TOPIC","PARTITION","STATUS"} {if ($status != "STOPPED" && $topic != "|" && $topic != "") {gsub(/"/,""); print $topic, $partition, $status}}' ${MIRROR_TOPIC_STATUS_FILE} | column -t > tmp-input-mirror-state.txt 214 | cat tmp-input-mirror-state.txt 215 | rm tmp-input-mirror-state.txt 216 | 217 | echo "" 218 | -------------------------------------------------------------------------------- /topic-delete.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #This script deletes all topics in a Confluent Cloud cluster 3 | #Ensure the confluent CLI is installed and referenced in your PATH or set the CONFLUENT_HOME_BIN environment variable to the location of the binary 4 | 5 | show_usage () { 6 | echo "Usage: $0 [options [parameters]]" 7 | echo "" 8 | echo "Options:" 9 | echo " --help -- Prints this page" 10 | echo " --input-file [input-file]" 11 | echo " --cluster [ccloud-cluster-id]" 12 | echo " --environment [ccloud-environment-id]" 13 | echo " --all-topics [use this flag to delete all topics]" 14 | return 0 15 | } 16 | 17 | if [[ $# -eq 0 ]];then 18 | echo "No input arguments provided." 19 | show_usage 20 | exit 1 21 | fi 22 | 23 | echo "======================================================================" 24 | echo "" 25 | echo "This script accepts a list of topics to delete, or deletes all topics" 26 | echo "" 27 | 28 | while [ ! -z "$1" ] 29 | do 30 | if [[ "$1" == "--help" ]] 31 | then 32 | show_usage 33 | exit 0 34 | elif [[ "$1" == "--input-file" ]] 35 | then 36 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 37 | then 38 | echo "No Value provided for "$1". Please ensure proper values are provided" 39 | show_usage 40 | exit 1 41 | fi 42 | inputFile="$2" 43 | shift 44 | elif [[ "$1" == "--cluster" ]] 45 | then 46 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 47 | then 48 | echo "No Value provided for "$1". Please ensure proper values are provided" 49 | show_usage 50 | exit 1 51 | fi 52 | clusterId="$2" 53 | echo "CCloud Kafka Cluster ID: ${clusterId}" 54 | shift 55 | elif [[ "$1" == "--environment" ]] 56 | then 57 | if [[ "$2" == --* ]] || [[ -z "$2" ]] 58 | then 59 | echo "No Value provided for "$1". Please ensure proper values are provided" 60 | show_usage 61 | exit 1 62 | fi 63 | environmentId="$2" 64 | echo "CCloud Environment ID: ${environmentId}" 65 | elif [[ "$1" == "--all-topics" ]] 66 | then 67 | ALL_TOPICS_FLAG=true 68 | echo "Delete all topics: true" 69 | fi 70 | shift 71 | done 72 | 73 | if [[ -z "$clusterId" ]] || [[ -z "$environmentId" ]] || [[ (! -z "$inputFile" || "$ALL_TOPICS_FLAG" != "true") ]] 74 | then 75 | echo "--input-file or --all-topics, --cluster, and --environment are required for execution." 76 | show_usage 77 | exit 1 78 | fi 79 | 80 | TOPICS=(${PWD}/`date +"%d-%m-%y-%T"`_ccloud_deleted_topics.txt) 81 | 82 | if [[ -z "$inputFile" ]] 83 | then 84 | echo "=====================================================================" 85 | echo "=============================== TOPICS ==============================" 86 | echo "" 87 | echo "Gathering list of topics to delete" 88 | echo "" 89 | 90 | confluent kafka topic list --cluster ${clusterId} --environment ${environmentId} > ${TOPICS} 91 | 92 | else 93 | echo "=====================================================================" 94 | echo "=============================== TOPICS ==============================" 95 | echo "" 96 | echo "Displaying list of target topics from the input file" 97 | echo "" 98 | 99 | cat "$inputFile" 100 | cat "$inputFile" > ${TOPICS} 101 | fi 102 | 103 | cat ${TOPICS} | while read -r topic_name 104 | do 105 | if [[ -z "${topic_name}" ]] 106 | then 107 | : 108 | else 109 | # Delete topics 110 | echo "=====================================================================" 111 | echo "======================== DELETING TOPICS ============================" 112 | confluent kafka topic delete ${topic_name} --cluster ${clusterId} --environment ${environmentId} 113 | fi 114 | done 115 | --------------------------------------------------------------------------------