├── .gitignore ├── NOTES.md ├── LICENSE ├── README.md └── syncbinlog.sh /.gitignore: -------------------------------------------------------------------------------- 1 | .idea 2 | .my.cnf 3 | NOTES.md 4 | -------------------------------------------------------------------------------- /NOTES.md: -------------------------------------------------------------------------------- 1 | # TODO 2 | 3 | - encryption can be already handled by mysql binlog encryption 4 | - write about supervisor -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 Arda Beyazoğlu 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # About 2 | 3 | This is a simple bash script that encapsulates `mysqlbinlog` utility to backup and compress binlog files. See the [blog post](https://blog.ardabeyazoglu.com/mysql-live-binlog-backup-and-restore-cjxj1kbhf001ijhs188drhofi) . 4 | 5 | # Features 6 | 7 | - Live backup of binlogs (handled by `mysqlbinlog` already) 8 | - Compression of backup files 9 | - Rotation of backup files 10 | 11 | # Usage 12 | 13 | Clone the repository and run: 14 | 15 | ``` 16 | chmod +x syncbinlog.sh 17 | ./syncbinlog.sh --help 18 | ``` 19 | 20 | This will output: 21 | 22 | ``` 23 | Usage: syncbinlog.sh [options] 24 | Starts live binlog sync using mysqlbinlog utility 25 | 26 | --backup-dir= Backup destination directory (required) 27 | --log-dir= Log directory (defaults to '/var/log/syncbinlog') 28 | --prefix= Backup file prefix (defaults to 'backup-') 29 | --mysql-conf= Mysql defaults file for client auth (defaults to './.my.cnf') 30 | --compress Compress backuped binlog files 31 | --compress-app= Compression app (defaults to 'pigz -p{number-of-cores - 1}'). Compression parameters can be given as well (e.g. pigz -p6 for 6 threaded compression) 32 | --rotate=X Rotate backup files for X days (defaults to 30) 33 | --verbose= Write logs to stdout as well 34 | ``` 35 | 36 | Example: Backup binlog files of last 10 days and compress them 37 | 38 | `./syncbinlog.sh --backup-dir=/mnt/backup --prefix="mybackup-" --compress --rotate=10` 39 | 40 | # Notes 41 | 42 | - In a production database server, it should be controlled by a process manager such as `systemd` or `supervisord` to have more reliable start/restart behaviour. 43 | - `mysqlbinlog` utility copies the binlog files in real-time, however compression is only applied for files older than the one being written at the time. This happens when: 44 | - Mysql flushes the log files after a certain time or certain file size. (See `expire_logs_days` and `max_binlog_size`) 45 | - `FLUSH LOGS` is executed manually or by mysqldump etc. 46 | - `mysqlbinlog` utility requires the user to have `REPLICATION SLAVE` privilege 47 | 48 | # Resources 49 | 50 | Some useful resources about binlog backup and point-in-time-recovery: 51 | 52 | - https://www.percona.com/blog/2012/01/18/backing-up-binary-log-files-with-mysqlbinlog/ 53 | - https://www.percona.com/blog/2017/10/23/mysql-point-in-time-recovery-right-way/ 54 | - http://mysqlnoob.blogspot.com/2016/12/in-place-transparent-compression-of-mysql-binary.logs.html 55 | - https://lefred.be/content/howto-make-mysql-point-in-time-recovery-faster/ 56 | 57 | # License 58 | 59 | MIT 60 | -------------------------------------------------------------------------------- /syncbinlog.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ########### syncbinlog.sh ############# 4 | # Copyright 2019 Arda Beyazoglu 5 | # MIT License 6 | # 7 | # A bash script that uses mysqlbinlog 8 | # utility to syncronize binlog files 9 | ####################################### 10 | 11 | # Write usage 12 | usage() { 13 | echo -e "Usage: $(basename $0) [options]" 14 | echo -e "\tStarts live binlog sync using mysqlbinlog utility\n" 15 | echo -e " --backup-dir= Backup destination directory (required)" 16 | echo -e " --log-dir= Log directory (defaults to '/var/log/syncbinlog')" 17 | echo -e " --prefix= Backup file prefix (defaults to 'backup-')" 18 | echo -e " --mysql-conf= Mysql defaults file for client auth (defaults to './.my.cnf')" 19 | echo -e " --compress Compress backuped binlog files" 20 | echo -e " --compress-app= Compression app (defaults to 'pigz'). Compression parameters can be given as well (e.g. pigz -p6 for 6 threaded compression)" 21 | echo -e " --rotate=X Rotate backup files for X days (defaults to 30)" 22 | echo -e " --verbose= Write logs to stdout as well" 23 | exit 1 24 | } 25 | 26 | # Write log 27 | log () { 28 | local level="INFO" 29 | if [[ -n $2 ]]; then 30 | level=$2 31 | fi 32 | local msg="[$(date +'%Y-%m-%d %H:%M:%S')][${level}] $1" 33 | echo "${msg}" >> "${LOG_DIR}/status.log" 34 | 35 | if [[ ${VERBOSE} == true ]]; then 36 | echo "${msg}" 37 | fi 38 | } 39 | 40 | # Parse configuration parameters 41 | parse_config() { 42 | for arg in ${ARGS} 43 | do 44 | case ${arg} in 45 | --prefix=*) 46 | BACKUP_PREFIX="${arg#*=}" 47 | ;; 48 | --log-dir=*) 49 | LOG_DIR="${arg#*=}" 50 | ;; 51 | --backup-dir=*) 52 | BACKUP_DIR="${arg#*=}" 53 | ;; 54 | --mysql-conf=*) 55 | MYSQL_CONFIG_FILE="${arg#*=}" 56 | ;; 57 | --compress) 58 | COMPRESS=true 59 | ;; 60 | --compress-app=*) 61 | COMPRESS_APP="${arg#*=}" 62 | ;; 63 | --rotate=*) 64 | ROTATE_DAYS="${arg#*=}" 65 | ;; 66 | --verbose) 67 | VERBOSE=true 68 | ;; 69 | --help) 70 | usage 71 | ;; 72 | *) 73 | # unknown option 74 | usage 75 | ;; 76 | esac 77 | done 78 | } 79 | 80 | # Compress backup files that are currently open 81 | compress_files() { 82 | # find last modified binlog backup file (except the *.original ones) 83 | LAST_MODIFIED_BINLOG_FILE=$(find ${BACKUP_DIR} -type f -name "${BACKUP_PREFIX}${BINLOG_BASENAME}*" -printf "%T@ %p\n" | sort -n | tail -1 | awk '{print $2}' | grep -P ".+\.[0-9]+$") 84 | LAST_MODIFIED_BINLOG_FILE=$(basename ${LAST_MODIFIED_BINLOG_FILE}) 85 | 86 | # find all binlog backup files sorted by modification date 87 | SORTED_BINLOG_FILES=$(find ${BACKUP_DIR} -type f -name "${BACKUP_PREFIX}${BINLOG_BASENAME}*" -printf "%T@ %p\n" | sort -n | awk '{print $2}' | grep -P ".+\.[0-9]+(|\.original)$") 88 | 89 | for filename in ${SORTED_BINLOG_FILES} 90 | do 91 | # check if file exists 92 | [[ -f "${filename}" ]] || break 93 | 94 | # break on last modified backup file, because its not completely written yet 95 | [[ `basename ${filename}` == "${LAST_MODIFIED_BINLOG_FILE}" ]] && break 96 | 97 | log "Compressing ${filename}" 98 | ${COMPRESS_APP} --force ${filename} > "${LOG_DIR}/status.log" 99 | log "Compressed ${filename}" 100 | done 101 | } 102 | 103 | # Rotate older backups 104 | rotate_files() { 105 | # find binlog backup files older than rotation period 106 | ROTATED_FILES=$(find ${BACKUP_DIR} -type f -name "${BACKUP_PREFIX}${BINLOG_BASENAME}*" -mtime +${ROTATE_DAYS} | grep -P ".+\.[0-9]+(|\.original)$") 107 | for filename in ${ROTATED_FILES} 108 | do 109 | log "Rotation: deleting ${filename}" 110 | rm ${filename} 111 | done 112 | } 113 | 114 | # Exit safely on signal 115 | die() { 116 | log "Exit signal caught!" 117 | log "Stopping child processes before exit" 118 | trap - SIGINT SIGTERM # clear the listener 119 | kill -- -$$ # Sends SIGTERM to child/sub processes 120 | if [[ ! -z ${APP_PID} ]]; then 121 | log "Killing mysqlbinlog process" 122 | kill ${APP_PID} 123 | fi 124 | } 125 | 126 | # listen to the process signals 127 | trap die SIGINT SIGTERM 128 | 129 | # Default configuration parameters 130 | MYSQL_CONFIG_FILE=./.my.cnf 131 | BACKUP_DIR="" 132 | LOG_DIR=/var/log/syncbinlog 133 | BACKUP_PREFIX="backup-" 134 | COMPRESS=false 135 | COMPRESS_APP="pigz -p$(($(nproc) - 1))" 136 | ROTATE_DAYS=30 137 | VERBOSE=false 138 | 139 | ARGS="$@" 140 | parse_config 141 | 142 | if [[ -z ${BACKUP_DIR} ]]; then 143 | echo "ERROR: Please, specify a destination directory for backups using --backup-dir parameter." 144 | usage 145 | exit 1 146 | fi 147 | 148 | if [[ ! -f ${MYSQL_CONFIG_FILE} ]]; then 149 | echo "ERROR: Mysql client config file ${MYSQL_CONFIG_FILE} does not exist." 150 | exit 1 151 | fi 152 | 153 | APP_PID=0 154 | MYSQL_CONFIG_FILE=$(realpath ${MYSQL_CONFIG_FILE}) 155 | BACKUP_DIR=$(realpath ${BACKUP_DIR}) 156 | LOG_DIR=$(realpath ${LOG_DIR}) 157 | 158 | mkdir -p ${LOG_DIR} || exit 1 159 | mkdir -p ${BACKUP_DIR} || exit 1 160 | cd ${BACKUP_DIR} || exit 1 161 | 162 | log "Initializing binlog sync" 163 | log "Backup destination: $BACKUP_DIR" 164 | log "Log destination: $LOG_DIR" 165 | log "Reading mysql client configuration from $MYSQL_CONFIG_FILE" 166 | 167 | BINLOG_BASENAME=$(mysql --defaults-extra-file=${MYSQL_CONFIG_FILE} -Bse "SHOW GLOBAL VARIABLES LIKE 'log_bin_basename'") 168 | if [[ $? -eq "1" ]]; then 169 | log "Please, check your mysql credentials" "ERROR" 170 | exit 1 171 | fi 172 | 173 | ${COMPRESS} == true && log "Compression enabled" 174 | 175 | BINLOG_BASENAME=$(basename `echo ${BINLOG_BASENAME} | tail -1 | awk '{ print $2 }'`) 176 | log "Binlog file basename is $BINLOG_BASENAME" 177 | 178 | BINLOG_INDEX_FILE=`mysql --defaults-extra-file=${MYSQL_CONFIG_FILE} -Bse "SHOW GLOBAL VARIABLES LIKE 'log_bin_index'" | tail -1 | awk '{ print $2 }'` 179 | log "Binlog index file is $BINLOG_BASENAME" 180 | 181 | BINLOG_LAST_FILE=`tail -1 "$BINLOG_INDEX_FILE"` 182 | log "Most recent binlog file is $BINLOG_LAST_FILE" 183 | 184 | while : 185 | do 186 | RUNNING=false 187 | 188 | # check pid to see if mysqlbinlog is running 189 | if [[ "$APP_PID" -gt "0" ]]; then 190 | # check process name to ensure it is mysqlbinlog pid 191 | APP_NAME=$(ps -p ${APP_PID} -o cmd= | awk '{ print $1 }') 192 | if [[ ${APP_NAME} == "mysqlbinlog" ]]; then 193 | RUNNING=true 194 | fi 195 | fi 196 | 197 | if [[ ${RUNNING} == true ]]; then 198 | # check older backups to compress 199 | ${COMPRESS} == true && compress_files 200 | 201 | # check file timestamps to apply rotation 202 | rotate_files 203 | 204 | # sleep and continue 205 | sleep 10 206 | continue 207 | fi 208 | 209 | # Check last backup file to continue from (2> /dev/null suppresses error output) 210 | LAST_BACKUP_FILE=`ls -1 ${BACKUP_DIR}/${BACKUP_PREFIX}* 2> /dev/null | grep -v ".original" | tail -n 1` 211 | 212 | BINLOG_SYNC_FILE_NAME="" 213 | 214 | if [[ -z ${LAST_BACKUP_FILE} ]]; then 215 | log "No backup file found, starting from oldest binary log in the server" 216 | 217 | # If there is no backup yet, find the first binlog file to start copying 218 | BINLOG_START_FILE=`head -n 1 "$BINLOG_INDEX_FILE"` 219 | log "The oldest binlog file is ${BINLOG_START_FILE}" 220 | 221 | BINLOG_SYNC_FILE_NAME=`basename "${BINLOG_START_FILE}"` 222 | else 223 | # If mysqlbinlog crashes/exits in the middle of execution, we cant know the last position reliably. 224 | # Thats why restart syncing from the beginning of the same binlog file 225 | LAST_BACKUP_FILE=$(basename ${LAST_BACKUP_FILE}) 226 | log "Last used backup file is $LAST_BACKUP_FILE" 227 | 228 | # CAUTION: 229 | # If the last backup file is too old, the relevant binlog file might not exist anymore 230 | # In this case, there will be a gap in binlog backups 231 | 232 | # Storing a backup of the latest binlog backup file before exit/crash 233 | FILE_SIZE=$(stat -c%s ${BACKUP_DIR}/${LAST_BACKUP_FILE}) 234 | if [[ ${FILE_SIZE} -gt 0 ]]; then 235 | log "Backing up last binlog file ${LAST_BACKUP_FILE}" 236 | mv "${BACKUP_DIR}/${LAST_BACKUP_FILE}" "${BACKUP_DIR}/${LAST_BACKUP_FILE}.original" 237 | fi 238 | 239 | # strip backup file prefix to get real binlog name 240 | LAST_BACKUP_FILE=${LAST_BACKUP_FILE/$BACKUP_PREFIX/} 241 | BINLOG_SYNC_FILE_NAME=`basename "${LAST_BACKUP_FILE}"` 242 | fi 243 | 244 | log "Starting live binlog backup from ${BINLOG_SYNC_FILE_NAME}" 245 | 246 | mysqlbinlog --defaults-extra-file=${MYSQL_CONFIG_FILE} \ 247 | --raw --read-from-remote-server --stop-never \ 248 | --verify-binlog-checksum \ 249 | --result-file=${BACKUP_PREFIX} \ 250 | ${BINLOG_SYNC_FILE_NAME} >> "${LOG_DIR}/status.log" & APP_PID=$! 251 | 252 | log "mysqlbinlog PID=$APP_PID" 253 | 254 | done 255 | --------------------------------------------------------------------------------