├── README.md └── mysqlchk_iptables /README.md: -------------------------------------------------------------------------------- 1 | # clustercheck-iptables 2 | 3 | A background script that checks the availability of a Galera node, and adds a redirection port using iptables if the Galera node is healthy (instead of returning HTTP response). This allows other TCP-load balancers with limited health check capabilities to monitor the backend Galera nodes correctly. 4 | 5 | Other than HAProxy, you can now use your favorite reverse proxy to load balance requests across Galera nodes, namely: 6 | - nginx 1.9 (--with-stream) 7 | - keepalived 8 | - IPVS 9 | - distributor 10 | - balance 11 | - pen 12 | 13 | Watch it in action in [this asciinema recording](https://asciinema.org/a/29799). Example deployment at [Severalnines' blog post](http://severalnines.com/blog/nginx-database-load-balancer-mysql-or-mariadb-galera-cluster). 14 | 15 | # How does it work? 16 | 17 | 0) Requires iptables 18 | 19 | 1) Monitors Galera node 20 | 21 | 2) If healthy (Synced + read_only=OFF) or (Donor + xtrabackup[-v2]), a port redirection will be setup using iptables (default: 3308 redirects to 3306) 22 | 23 | 3) Else, the port redirection will be ruled out from the iptables PREROUTING chain 24 | 25 | On the load balancer, define the designated redirection port (3308) instead. For example on nginx 1.9 (configured with --with-stream): 26 | ```bash 27 | upstream stream_backend { 28 | zone tcp_servers 64k; 29 | server 192.168.0.201:3308; 30 | server 192.168.0.202:3308; 31 | server 192.168.0.203:3308; 32 | } 33 | ``` 34 | If the backend node is "unhealthy", 3308 will be unreachable because the corresponding iptables rule is removed on the database node and nginx will exclude it from the load balancing set accordingly. 35 | 36 | # Install 37 | 38 | 1) On the Galera node, install the script into /usr/local/sbin: 39 | ```bash 40 | $ git clone https://github.com/ashraf-s9s/clustercheck-iptables 41 | $ cp clustercheck-iptables/mysqlchk_iptables /usr/local/sbin/ 42 | $ chmod 755 /usr/local/sbin/mysqlchk_iptables 43 | ``` 44 | 45 | 2) Configure DB user/password (default as per below): 46 | ```mysql 47 | mysql> GRANT PROCESS ON *.* TO 'mysqlchk_user'@'localhost' IDENTIFIED BY 'mysqlchk_password'; 48 | ``` 49 | ** If you choose not to use the default user, you can use -u and -p or -e in the command line. Example in the Run section. 50 | 51 | 3) Make sure iptables is running and ensure we setup the firewall rules for Galera services: 52 | ```bash 53 | chkconfig iptables on # or systemctl enable iptables.service 54 | service iptables start # or systemctl start iptables.service 55 | iptables -I INPUT -m tcp -p tcp --dport 3306 -j ACCEPT 56 | iptables -I INPUT -m tcp -p tcp --dport 3308 -j ACCEPT 57 | iptables -I INPUT -m tcp -p tcp --dport 4444 -j ACCEPT 58 | iptables -I INPUT -m tcp -p tcp --dport 4567:4568 -j ACCEPT 59 | service iptables save 60 | service iptables restart 61 | ``` 62 | ** CentOS 7 comes with firewalld by default. You probably have to install iptables services beforehand, use ``yum install iptables-services`` 63 | 64 | # Run 65 | 66 | The script must run as root/sudo to allow iptables changes. Append 'sudo' at the beginning of the command line if you want to run it as non-root user. 67 | 68 | Test the script and ensure it detects Galera node healthiness correctly: 69 | ```bash 70 | mysqlchk_iptables -t 71 | ``` 72 | 73 | Once satisfied, run the script as daemon: 74 | ```bash 75 | mysqlchk_iptables -d 76 | ``` 77 | 78 | If you use non-default user/password, specify them as per below (replace -d with -t if you just want to test): 79 | ```bash 80 | mysqlchk_iptables -d --user=check --password=checkpassword 81 | ``` 82 | 83 | To stop it: 84 | ```bash 85 | mysqlchk_iptables -x 86 | ``` 87 | 88 | To check the process status: 89 | ```bash 90 | mysqlchk_iptables -s 91 | ``` 92 | 93 | To make it starts on boot, add the command into ``/etc/rc.local``: 94 | ```bash 95 | echo '/usr/local/sbin/mysqlchk_iptables -d' >> /etc/rc.local 96 | ``` 97 | 98 | ** Make sure /etc/rc.local has permission to run on startup. Verify with: 99 | ```bash 100 | chmod +x /etc/rc.local 101 | ``` 102 | Other parameters are available with ``--help``. 103 | 104 | You can also use [supervisord](http://supervisord.org/) or [monit](https://mmonit.com/monit/) to automate and monitor the process. 105 | 106 | # Caveats 107 | 108 | ### Credentials exposure 109 | 110 | The script defaults to fork a background process which expose full parameters containing sensitive information e.g user/password. Example of the ps output: 111 | ```bash 112 | $ ps aux | grep mysqlchk_iptables 113 | root 26768 0.2 0.0 113248 1612 pts/4 S 07:08 0:01 /bin/bash /usr/local/sbin/mysqlchk_iptables --username=mysqlchk_user --password=mysqlchk_password --mirror-port=3308 --real-port=3306 --log-file=/var/log/mysqlchk_iptables --source-address=0.0.0.0/0 --source-address-ipv6=0::0 --check-interval=1 --defaults-extra-file=/etc/my.cnf -R 114 | ``` 115 | 116 | If you don't want user/password values to be exposed in the command line, specify the user credentials under [client] directive inside MySQL default extra file. In the command line, pass empty values for username and password (-u and -p) and use -e to include the extra file: 117 | ```bash 118 | mysqlchk_iptables -d -u "" -p "" -e /root/.my.cnf 119 | ``` 120 | 121 | The example content of ``/root/.my.cnf``: 122 | ```bash 123 | [client] 124 | user='root' 125 | password='password!@#$' 126 | ``` 127 | 128 | ### Log rotation 129 | 130 | By default, the script will log all activities into ``/var/log/mysqlchk_iptables``. It's recommended to setup a log rotation so it won't fill up your disk space. Create a new file at ``/etc/logrotate.d/mysqlchk_iptables`` and add following lines: 131 | 132 | ```bash 133 | /var/log/mysqlchk_iptables { 134 | size 50M 135 | create 0664 root root 136 | rotate 3 137 | compress 138 | } 139 | ``` 140 | 141 | To disable logging, redirect the output to ``/dev/null`` in the command line: 142 | ```bash 143 | mysqlchk_iptables -d --log-file=/dev/null 144 | ``` 145 | 146 | ### Tested environment 147 | 148 | This script is built and tested on: 149 | 150 | * Percona XtraDB Cluster 5.6, MySQL Galera Cluster 5.6 and MariaDB Galera 10.0, galera 3.x 151 | * CentOS 7.1 152 | * iptables v1.4.21 153 | * nginx/1.9.6 154 | * nginx/1.9.4 (nginx-plus-r7-p1) 155 | -------------------------------------------------------------------------------- /mysqlchk_iptables: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # A background script use to check Galera node availability and add a port redirection using 4 | # iptables if Galera node is healthy. Derived from percona-clustercheck, but utilizing iptables 5 | # port redirection instead of returning HTTP response. This allows TCP load balancer to perform 6 | # health checks without custom monitoring port (percona-clustercheck runs on 9200 through xinetd). 7 | # 8 | # Author: Ashraf Sharif 9 | # 10 | # Documentation and download: https://github.com/ashraf-s9s/clustercheck-iptables/ 11 | # 12 | # Based on the original script from percona-clustercheck 13 | # 14 | 15 | ## Defaults 16 | MIRROR_PORT=3308 17 | REAL_PORT=3306 18 | SOURCE_ADDRESS="0.0.0.0/0" 19 | SOURCE_ADDRESS_IPV6="0::0/0" 20 | CHECK_INTERVAL=1 21 | MYSQL_USERNAME="mysqlchk_user" 22 | MYSQL_PASSWORD="mysqlchk_password" 23 | AVAILABLE_WHEN_DONOR=0 24 | LOG_FILE="/var/log/mysqlchk_iptables" 25 | AVAILABLE_WHEN_READONLY=0 26 | DEFAULTS_EXTRA_FILE="/etc/my.cnf" 27 | VERBOSE=0 28 | ERR_FILE="/dev/null" 29 | PROG_NAME="mysqlchk_iptables" 30 | 31 | TEMP=`getopt -o h,v,d,x,s,u:,p:,l:,e:,a:,a6:,m:,r:,i:,t,R,D --long help,verbose,daemon,stop,status,user:,password:,log-file:,defaults-extra-file:,source-address:,source-address-ipv6:,mirror-port:,real-port:,check-interval:,test,run,debug -n 'mysqlchk_iptables' -- "$@"` 32 | eval set -- "$TEMP" 33 | 34 | while true ; do 35 | case "$1" in 36 | -h|--help) HELP="1" ; shift ;; 37 | -v|--verbose) VERBOSE="1" ; shift ;; 38 | -d|--daemon) DAEMON="1" ; shift ;; 39 | -x|--stop) STOP="1" ; shift ;; 40 | -s|--status) STATUS="1"; shift ;; 41 | -u|--user) MYSQL_USERNAME="$2" ; shift 2 ;; 42 | -p|--password) MYSQL_PASSWORD="$2" ; shift 2 ;; 43 | -l|--log-file) LOG_FILE="$2" ; shift 2 ;; 44 | -e|--defaults-extra-file) DEFAULTS_EXTRA_FILE="$2"; shift 2 ;; 45 | -a|--source-address) SOURCE_ADDRESS="$2"; shift 2 ;; 46 | -a6|--source-address-ipv6) SOURCE_ADDRESS_IPV6="$2"; shift 2 ;; 47 | -m|--mirror-port) MIRROR_PORT="$2"; shift 2 ;; 48 | -r|--real-port) REAL_PORT="$2" ; shift 2 ;; 49 | -i|--check-interval) CHECK_INTERVAL="$2" ; shift 2 ;; 50 | -t|--test) TEST="1"; shift ;; 51 | -R|--run) RUN="1"; shift ;; 52 | -D|--debug) DEBUG="1"; shift ;; 53 | --) shift ; break ;; 54 | *) echo "$0: error - unrecognized option -$OPTARG" 1>&2; exit 1;; 55 | esac 56 | done 57 | 58 | help () 59 | { 60 | echo "" 61 | echo "Options:" 62 | echo "-v, --verbose : Run this script in foreground. CTRL+C to stop." 63 | echo "-d, --daemon : Run this script as daemon." 64 | echo "-x, --stop : Stop this script if it is running as daemon." 65 | echo "-s, --status : Report the status of this script." 66 | echo "-t, --test : Test the script." 67 | echo "-u, --user : MySQL user to perform health check. (Default: mysqlchk_user)" 68 | echo '-p, --password : Password for the MySQL user (Default: mysqlchk_password)' 69 | echo "-l, --log-file : Log file (Default: /var/log/mysqlchk_iptables)" 70 | echo "-e, --defaults-extra-file : This file (if exists) will be passed to the mysql-command with the command line option --defaults-extra-file (Default: /etc/my.cnf)" 71 | echo "-a, --source-address : The source IP address to allow accessing the mirror port by iptables. (Default: 0.0.0.0/0)" 72 | echo "-a6, --source-address-ipv6 : The source IPv6 address to allow accessing the mirror port by ip6tables. (Default: 0::0)" 73 | echo "-m, --mirror-port : This port will be created by iptables which redirects to the real port. (Default: 3308)" 74 | echo "-r, --real-port : MySQL port listening on this node. (Default: 3306)" 75 | echo "-i, --check-interval : Health check interval in seconds. (Default: 1)" 76 | echo "-h, --help : Print help" 77 | echo "" 78 | echo "Details at https://github.com/ashraf-s9s/clustercheck-iptables" 79 | echo "" 80 | exit 0 81 | } 82 | 83 | # Timeout exists for instances where mysqld may be hung 84 | TIMEOUT=10 85 | 86 | [ "$HELP" == "1" ] && help 87 | 88 | # if the disabled file is present, delete iptables rule. This allows 89 | # admins to manually remove a node from a cluster easily. 90 | if [ -e "/var/tmp/clustercheck.disabled" ]; then 91 | iptables_rules delete 92 | logging "Galera Cluster Node is manually disabled." 93 | fi 94 | 95 | logging() 96 | { 97 | if [[ $DEBUG == "1" ]]; then 98 | echo "[$(date +"%d-%m-%y %T.%N")] [INFO] $1" 99 | else 100 | echo "[$(date +"%d-%m-%y %T.%N")] [INFO] $1" >> $LOG_FILE 101 | fi 102 | } 103 | 104 | iptables_rules() 105 | { 106 | OPT="-A" 107 | [ $1 == "delete" ] && OPT="-D" 108 | 109 | # check if the rule exists 110 | iptables -t nat -C PREROUTING -s $SOURCE_ADDRESS -p tcp --dport $MIRROR_PORT -j REDIRECT --to-ports $REAL_PORT &> /dev/null 111 | proc=$? 112 | 113 | if [[ $proc == 0 && "$OPT" == "-D" ]]; then 114 | iptables -t nat $OPT PREROUTING -s $SOURCE_ADDRESS -p tcp --dport $MIRROR_PORT -j REDIRECT --to-ports $REAL_PORT 115 | [ $? -eq 0 ] && logging "iptables rule deleted. $MIRROR_PORT removed." || logging 'Failed to delete iptables rule.' 116 | fi 117 | 118 | if [[ $proc == 1 && "$OPT" == "-A" ]]; then 119 | if [[ "$DRYRUN" != "1" ]]; then 120 | iptables -t nat $OPT PREROUTING -s $SOURCE_ADDRESS -p tcp --dport $MIRROR_PORT -j REDIRECT --to-ports $REAL_PORT 121 | [ $? -eq 0 ] && logging "iptables rule added. $MIRROR_PORT is redirected to $REAL_PORT". || logging 'Failed to add iptables rule.' 122 | else 123 | logging "(TEST) iptables rule added. $MIRROR_PORT is redirected to $REAL_PORT". 124 | fi 125 | fi 126 | 127 | ip6tables -t nat -C PREROUTING -s $SOURCE_ADDRESS_IPV6 -p tcp --dport $MIRROR_PORT -j REDIRECT --to-ports $REAL_PORT &> /dev/null 128 | proc=$? 129 | 130 | if [[ $proc == 0 && "$OPT" == "-D" ]]; then 131 | ip6tables -t nat $OPT PREROUTING -s $SOURCE_ADDRESS_IPV6 -p tcp --dport $MIRROR_PORT -j REDIRECT --to-ports $REAL_PORT 132 | [ $? -eq 0 ] && logging "ip6tables rule deleted. $MIRROR_PORT removed." || logging 'Failed to delete ip6tables rule.' 133 | fi 134 | 135 | if [[ $proc == 1 && "$OPT" == "-A" ]]; then 136 | if [[ "$DRYRUN" != "1" ]]; then 137 | ip6tables -t nat $OPT PREROUTING -s $SOURCE_ADDRESS_IPV6 -p tcp --dport $MIRROR_PORT -j REDIRECT --to-ports $REAL_PORT 138 | [ $? -eq 0 ] && logging "ip6tables rule added. $MIRROR_PORT is redirected to $REAL_PORT". || logging 'Failed to add ip6tables rule.' 139 | else 140 | logging "(TEST) ip6tables rule added. $MIRROR_PORT is redirected to $REAL_PORT". 141 | fi 142 | fi 143 | } 144 | 145 | main() 146 | { 147 | EXTRA_ARGS="" 148 | if [[ -n "$MYSQL_USERNAME" ]]; then 149 | EXTRA_ARGS="$EXTRA_ARGS --user=${MYSQL_USERNAME}" 150 | fi 151 | if [[ -n "$MYSQL_PASSWORD" ]]; then 152 | EXTRA_ARGS="$EXTRA_ARGS --password=${MYSQL_PASSWORD}" 153 | fi 154 | if [[ -r $DEFAULTS_EXTRA_FILE ]];then 155 | MYSQL_CMDLINE="mysql --defaults-extra-file=$DEFAULTS_EXTRA_FILE -nNE --connect-timeout=$TIMEOUT \ 156 | ${EXTRA_ARGS}" 157 | else 158 | MYSQL_CMDLINE="mysql -nNE --connect-timeout=$TIMEOUT ${EXTRA_ARGS}" 159 | fi 160 | # 161 | # Perform the query to check the wsrep_local_state 162 | # 163 | WSREP_STATUS=$($MYSQL_CMDLINE -e "SHOW STATUS LIKE 'wsrep_local_state';" \ 164 | 2>${ERR_FILE} | tail -1 2>>${ERR_FILE}) 165 | WSREP_SST=$($MYSQL_CMDLINE -e "SHOW VARIABLES LIKE 'wsrep_sst_method';" \ 166 | 2>${ERR_FILE} | tail -1 2>>${ERR_FILE}) 167 | 168 | if [[ "$DRYRUN" == "1" ]]; then 169 | if [[ -z "$WSREP_STATUS" && -z "$WSREP_SST" ]]; then 170 | echo "Unable to detect wsrep_local_state and wsrep_sst_method. Something wasn't right." 171 | echo "Please check user/password. Ensure those are correct. Aborting." 172 | exit 1 173 | else 174 | echo "Detected variables/status:" 175 | echo "wsrep_local_state: $WSREP_STATUS" 176 | echo "wsrep_sst_method: $WSREP_SST" 177 | fi 178 | fi 179 | 180 | if [[ "${WSREP_STATUS}" == "4" ]] || [[ "${WSREP_STATUS}" == "2" && "${WSREP_SST}" =~ "xtrabackup" ]] 181 | then 182 | # Check only when set to 0 to avoid latency in response. 183 | if [[ $AVAILABLE_WHEN_READONLY -eq 0 ]];then 184 | READ_ONLY=$($MYSQL_CMDLINE -e "SHOW GLOBAL VARIABLES LIKE 'read_only';" \ 185 | 2>${ERR_FILE} | tail -1 2>>${ERR_FILE}) 186 | [[ "$DRYRUN" == "1" ]] && echo "read_only: $READ_ONLY" && echo "" 187 | if [[ "${READ_ONLY}" == "ON" ]];then 188 | # Galera Cluster node local state is 'Synced', but it is in 189 | # read-only mode. 190 | # => delete iptables rule 191 | iptables_rules delete 192 | logging "Galera Cluster Node is read-only." 193 | else 194 | # Galera Cluster node local state is 'Synced' without read_only = ON 195 | # => add iptables rule 196 | iptables_rules add 197 | logging "Galera Cluster Node is synced." 198 | fi 199 | fi 200 | else 201 | # Galera Cluster node local state is not 'Synced' 202 | # => delete iptables rule 203 | iptables_rules delete 204 | logging "Galera Cluster Node is not synced." 205 | fi 206 | } 207 | 208 | REQ_ARGUMENTS="--user=${MYSQL_USERNAME} --password=${MYSQL_PASSWORD} --mirror-port=$MIRROR_PORT --real-port=$REAL_PORT --log-file=$LOG_FILE --source-address=$SOURCE_ADDRESS --source-address-ipv6=$SOURCE_ADDRESS_IPV6 --check-interval=$CHECK_INTERVAL --defaults-extra-file=$DEFAULTS_EXTRA_FILE" 209 | 210 | ABSOLUTE_PATH=$(cd `dirname "${BASH_SOURCE[0]}"` && pwd)/$PROG_NAME 211 | PID_PATH=/var/run 212 | PID_FILE=$PID_PATH/$PROG_NAME 213 | 214 | ctrl_c() { 215 | iptables_rules delete 216 | echo "" 217 | echo "$ABSOLUTE_PATH stopped. Bye." 218 | exit 1 219 | } 220 | 221 | if [[ $DAEMON == "1" ]]; then 222 | [ -e $PID_FILE ] && echo "PID file exists at $PID_FILE. Refusing to fork another process." && exit 1 223 | $ABSOLUTE_PATH $REQ_ARGUMENTS -R & 224 | PID=$! 225 | if [ $? -ne 0 ]; then 226 | logging "Unable to run as daemon. Exiting." 227 | exit 1 228 | else 229 | echo $PID > $PID_FILE 230 | logging "Script started with PID $PID." 231 | logging "Command used to start: $ABSOLUTE_PATH $REQ_ARGUMENTS" 232 | echo "$ABSOLUTE_PATH started with PID $PID." 233 | exit 0 234 | fi 235 | elif [[ $STOP == "1" ]]; then 236 | echo "Stopping $PROG_NAME.." 237 | if [ ! -e $PID_FILE ]; then 238 | echo "PID file does not exist. Nothing to kill. Exiting." 239 | exit 1 240 | else 241 | EPID=$(cat $PID_FILE) 242 | kill $EPID 243 | [ $? -ne 0 ] && logging "Unable to kill $EPID. Exiting." && exit 1 244 | iptables_rules delete 245 | rm -f $PID_FILE 246 | logging "PID $EPID killed. $PID_FILE removed. Bye." 247 | echo "$ABSOLUTE_PATH stopped. Bye." 248 | exit 0 249 | fi 250 | elif [[ $STATUS == "1" ]]; then 251 | if [ ! -e $PID_FILE ]; then 252 | echo "PID file does not exist. Script is not running." 253 | exit 1 254 | else 255 | EPID=$(cat $PID_FILE) 256 | echo "PID file exists. Script is running with PID $EPID." 257 | exit 0 258 | fi 259 | elif [[ $VERBOSE == "1" ]]; then 260 | [ -e $PID_FILE ] && echo "PID file exists at $PID_FILE. Refusing to fork another process." && exit 1 261 | echo "Running in foreground.. CTRL+C to stop" 262 | trap ctrl_c INT 263 | $ABSOLUTE_PATH $REQ_ARGUMENTS -R -D 264 | fi 265 | 266 | if [[ $RUN == "1" ]]; then 267 | while true; do 268 | main 269 | sleep $CHECK_INTERVAL 270 | done 271 | elif [[ $TEST == "1" ]]; then 272 | DEBUG="1" 273 | DRYRUN=1 274 | main 275 | sleep $CHECK_INTERVAL 276 | else 277 | help 278 | fi 279 | --------------------------------------------------------------------------------