├── LICENSE ├── README.md ├── clustercheck └── systemd ├── mysqlchk.socket └── mysqlchk@.service /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2012-2014, Olaf van Zandwijk 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without modification, 5 | are permitted provided that the following conditions are met: 6 | 7 | 1. Redistributions of source code must retain the above copyright notice, 8 | this list of conditions and the following disclaimer. 9 | 10 | 2. Redistributions in binary form must reproduce the above copyright notice, 11 | this list of conditions and the following disclaimer in the documentation 12 | and/or other materials provided with the distribution. 13 | 14 | 3. Neither the name of the copyright holder nor the names of its contributors 15 | may be used to endorse or promote products derived from this software without 16 | specific prior written permission. 17 | 18 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 19 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 20 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 21 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR 22 | ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 23 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 24 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON 25 | ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 26 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 27 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 28 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Percona Clustercheck ## 2 | 3 | Script to make a proxy (ie HAProxy) capable of monitoring Percona XtraDB Cluster nodes properly. 4 | 5 | ## Usage ## 6 | Below is a sample configuration for HAProxy on the client. The point of this is that the application will be able to connect to localhost port 3307, so although we are using Percona XtraDB Cluster with several nodes, the application will see this as a single MySQL server running on localhost. 7 | 8 | `/etc/haproxy/haproxy.cfg` 9 | 10 | ... 11 | listen percona-cluster 0.0.0.0:3307 12 | balance leastconn 13 | option httpchk 14 | mode tcp 15 | server node1 1.2.3.4:3306 check port 9200 inter 5000 fastinter 2000 rise 2 fall 2 16 | server node2 1.2.3.5:3306 check port 9200 inter 5000 fastinter 2000 rise 2 fall 2 17 | server node3 1.2.3.6:3306 check port 9200 inter 5000 fastinter 2000 rise 2 fall 2 backup 18 | 19 | MySQL connectivity is checked via HTTP on port 9200. The clustercheck script is a simple shell script which accepts HTTP requests and checks MySQL on an incoming request. If the Percona XtraDB Cluster node is ready to accept requests, it will respond with HTTP code 200 (OK), otherwise a HTTP error 503 (Service Unavailable) is returned. 20 | 21 | ## Setup with xinetd ## 22 | This setup will create a process that listens on TCP port 9200 using xinetd. This process uses the clustercheck script from this repository to report the status of the node. 23 | 24 | First, create a clustercheckuser that will be doing the checks. 25 | 26 | mysql> GRANT PROCESS ON *.* TO 'clustercheckuser'@'localhost' IDENTIFIED BY 'clustercheckpassword!' 27 | 28 | Copy the clustercheck from the repository to a location (`/usr/bin` in the example below) and make it executable. Then add the following service to xinetd (make sure to match your location of the script with the 'server'-entry). 29 | 30 | `/etc/xinetd.d/mysqlchk`: 31 | 32 | # default: on 33 | # description: mysqlchk 34 | service mysqlchk 35 | { 36 | disable = no 37 | flags = REUSE 38 | socket_type = stream 39 | port = 9200 40 | wait = no 41 | user = nobody 42 | server = /usr/bin/clustercheck 43 | log_on_failure += USERID 44 | only_from = 0.0.0.0/0 45 | per_source = UNLIMITED 46 | } 47 | 48 | Also, you should add the mysqlchk service to `/etc/services` before restarting xinetd. 49 | 50 | xinetd 9098/tcp # ... 51 | mysqlchk 9200/tcp # MySQL check <--- Add this line 52 | git 9418/tcp # Git Version Control System 53 | zope 9673/tcp # ... 54 | 55 | Clustercheck will now listen on port 9200 after xinetd restart, and HAproxy is ready to check MySQL via HTTP poort 9200. 56 | 57 | ## Setup with systemd ## 58 | This setup will register a socket on TCP port 9200 and an associated instantiated service with systemd. Connections to the socket will be forwarded to the clustercheck script from this repository to report the status of the node. 59 | 60 | First, create a clustercheckuser that will be doing the checks. 61 | 62 | mysql> GRANT PROCESS ON *.* TO 'clustercheckuser'@'localhost' IDENTIFIED BY 'clustercheckpassword!' 63 | 64 | Copy the clustercheck from the repository to `/usr/bin` and make it executable. Then copy the `mysqlck.socket` and `mysqlchk@.service` files (under `systemd`) in the repository to `/usr/lib/systemd/system` and make them world-readable. 65 | 66 | To activate the socket, run the following commands: 67 | 68 | # systemctl enable mysqlchk.socket 69 | # systemctl start mysqlchk.socket 70 | 71 | Systemd will now service requests to port 9200 with the clustercheck script, and HAproxy is ready to check MySQL via HTTP port 9200. 72 | 73 | ## Setup with shell return values ## 74 | If you do not want to use the setup with xinetd, you can also execute `clustercheck` on the commandline and check for the return value. 75 | 76 | First, create a clustercheckuser that will be doing the checks. 77 | 78 | mysql> GRANT PROCESS ON *.* TO 'clustercheckuser'@'localhost' IDENTIFIED BY 'clustercheckpassword!' 79 | 80 | Then, you can execute the script. In case of a synced node: 81 | 82 | # /usr/bin/clustercheck > /dev/null 83 | # echo $? 84 | 0 85 | 86 | In case of an un-synced node: 87 | 88 | # /usr/bin/clustercheck > /dev/null 89 | # echo $? 90 | 1 91 | 92 | You can use this return value with monitoring tools like Zabbix or Zenoss. 93 | 94 | ## Configuration options ## 95 | The clustercheck script accepts several arguments: 96 | 97 | clustercheck 98 | 99 | - **user** and **pass** (default clustercheckuser and clustercheckpassword!): defines the username and password for the check. You can pass an empty username and/or password by supplying "" 100 | - **available_when_donor** (default 0): By default, the node is reported unavailable if it’s a donor for SST. If you want to allow queries on a node which is a donor for SST, you can set this variable to 1. Note: when you set this variable to 1, you will also need to use a non-blocking SST method like xtrabackup 101 | - **log_file** (default "/dev/null"): defines where logs and errors from the checks should go 102 | - **available_when_readonly** (default 1): Depending on this setting and the MySQL status variable 'read_only', the node is reported available 103 | - **defaults_extra_file** (default /etc/my.cnf): This file (if exists) will be passed to the mysql-command with the commandline option --defaults-extra-file 104 | 105 | Note: You can also specify the username and password for the check in the **defaults_extra_file** and pass empty values for **user** and **pass**. That way, nobody can see the username and password (which can otherwise be seen, since they are passed to the MySQL CLI) 106 | 107 | ## Manually removing a node from the cluster ## 108 | 109 | By touching /var/tmp/clustercheck.disabled, an admin may force clustercheck to return 503, regardless as to the actual state of the node. This is useful when the node is being put into maintenance mode. 110 | -------------------------------------------------------------------------------- /clustercheck: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # Script to make a proxy (ie HAProxy) capable of monitoring Percona XtraDB Cluster nodes properly 4 | # 5 | # Author: Olaf van Zandwijk 6 | # Author: Raghavendra Prabhu 7 | # 8 | # Documentation and download: https://github.com/olafz/percona-clustercheck 9 | # 10 | # Based on the original script from Unai Rodriguez 11 | # 12 | 13 | if [[ $1 == '-h' || $1 == '--help' ]];then 14 | echo "Usage: $0 " 15 | exit 16 | fi 17 | 18 | # if the disabled file is present, return 503. This allows 19 | # admins to manually remove a node from a cluster easily. 20 | if [ -e "/var/tmp/clustercheck.disabled" ]; then 21 | # Shell return-code is 1 22 | echo -en "HTTP/1.1 503 Service Unavailable\r\n" 23 | echo -en "Content-Type: text/plain\r\n" 24 | echo -en "Connection: close\r\n" 25 | echo -en "Content-Length: 51\r\n" 26 | echo -en "\r\n" 27 | echo -en "Percona XtraDB Cluster Node is manually disabled.\r\n" 28 | sleep 0.1 29 | exit 1 30 | fi 31 | 32 | set -e 33 | 34 | if [ -f /etc/sysconfig/clustercheck ]; then 35 | . /etc/sysconfig/clustercheck 36 | fi 37 | 38 | MYSQL_USERNAME="${MYSQL_USERNAME:=-clustercheckuser}" 39 | MYSQL_PASSWORD="${MYSQL_PASSWORD-clustercheckpassword!}" 40 | AVAILABLE_WHEN_DONOR=${AVAILABLE_WHEN_DONOR:-0} 41 | ERR_FILE="${ERR_FILE:-/dev/null}" 42 | AVAILABLE_WHEN_READONLY=${AVAILABLE_WHEN_READONLY:-1} 43 | DEFAULTS_EXTRA_FILE=${DEFAULTS_EXTRA_FILE:-/etc/my.cnf} 44 | 45 | #Timeout exists for instances where mysqld may be hung 46 | TIMEOUT=10 47 | 48 | EXTRA_ARGS="" 49 | if [[ -n "$MYSQL_USERNAME" ]]; then 50 | EXTRA_ARGS="$EXTRA_ARGS --user=${MYSQL_USERNAME}" 51 | fi 52 | if [[ -n "$MYSQL_PASSWORD" ]]; then 53 | EXTRA_ARGS="$EXTRA_ARGS --password=${MYSQL_PASSWORD}" 54 | fi 55 | if [[ -r $DEFAULTS_EXTRA_FILE ]];then 56 | MYSQL_CMDLINE="mysql --defaults-extra-file=$DEFAULTS_EXTRA_FILE -nNE --connect-timeout=$TIMEOUT \ 57 | ${EXTRA_ARGS}" 58 | else 59 | MYSQL_CMDLINE="mysql -nNE --connect-timeout=$TIMEOUT ${EXTRA_ARGS}" 60 | fi 61 | # 62 | # Perform the query to check the wsrep_local_state 63 | # 64 | WSREP_STATUS=$($MYSQL_CMDLINE -e "SHOW STATUS LIKE 'wsrep_local_state';" \ 65 | 2>${ERR_FILE} | tail -1 2>>${ERR_FILE}) 66 | 67 | if [[ "${WSREP_STATUS}" == "4" ]] || [[ "${WSREP_STATUS}" == "2" && ${AVAILABLE_WHEN_DONOR} == 1 ]] 68 | then 69 | # Check only when set to 0 to avoid latency in response. 70 | if [[ $AVAILABLE_WHEN_READONLY -eq 0 ]];then 71 | READ_ONLY=$($MYSQL_CMDLINE -e "SHOW GLOBAL VARIABLES LIKE 'read_only';" \ 72 | 2>${ERR_FILE} | tail -1 2>>${ERR_FILE}) 73 | 74 | if [[ "${READ_ONLY}" == "ON" ]];then 75 | # Percona XtraDB Cluster node local state is 'Synced', but it is in 76 | # read-only mode. The variable AVAILABLE_WHEN_READONLY is set to 0. 77 | # => return HTTP 503 78 | # Shell return-code is 1 79 | echo -en "HTTP/1.1 503 Service Unavailable\r\n" 80 | echo -en "Content-Type: text/plain\r\n" 81 | echo -en "Connection: close\r\n" 82 | echo -en "Content-Length: 43\r\n" 83 | echo -en "\r\n" 84 | echo -en "Percona XtraDB Cluster Node is read-only.\r\n" 85 | sleep 0.1 86 | exit 1 87 | fi 88 | fi 89 | # Percona XtraDB Cluster node local state is 'Synced' => return HTTP 200 90 | # Shell return-code is 0 91 | echo -en "HTTP/1.1 200 OK\r\n" 92 | echo -en "Content-Type: text/plain\r\n" 93 | echo -en "Connection: close\r\n" 94 | echo -en "Content-Length: 40\r\n" 95 | echo -en "\r\n" 96 | echo -en "Percona XtraDB Cluster Node is synced.\r\n" 97 | sleep 0.1 98 | exit 0 99 | else 100 | # Percona XtraDB Cluster node local state is not 'Synced' => return HTTP 503 101 | # Shell return-code is 1 102 | echo -en "HTTP/1.1 503 Service Unavailable\r\n" 103 | echo -en "Content-Type: text/plain\r\n" 104 | echo -en "Connection: close\r\n" 105 | echo -en "Content-Length: 44\r\n" 106 | echo -en "\r\n" 107 | echo -en "Percona XtraDB Cluster Node is not synced.\r\n" 108 | sleep 0.1 109 | exit 1 110 | fi 111 | -------------------------------------------------------------------------------- /systemd/mysqlchk.socket: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=Percona XtraDB Cluster node check socket 3 | 4 | [Socket] 5 | ListenStream=9200 6 | Accept=yes 7 | 8 | [Install] 9 | WantedBy=sockets.target -------------------------------------------------------------------------------- /systemd/mysqlchk@.service: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=Percona XtraDB Cluster node check service 3 | 4 | [Service] 5 | ExecStart=-/usr/bin/clustercheck 6 | StandardInput=socket --------------------------------------------------------------------------------