├── README.md
├── backup-mysql.sh
├── extract-mysql.sh
└── prepare-mysql.sh
/README.md:
--------------------------------------------------------------------------------
1 | # Debian / Ubuntu 16.04 MariaDB backup using mariabackup
2 | *Currently tested with Debian 8 (jessie)*
3 |
4 | This repository contains a few scripts for automating backups with mariabackup (a fork of Percona Xtrabackup) by MariaDB.
5 |
6 | Please check and follow the instructions below. The instructions are taken from here
7 |
8 | ## Create a MySQL User with Appropriate Privileges
9 |
10 | The first thing we need to do is create a new MySQL user configured to handle backup tasks. We will only give this user the privileges it needs to copy the data safely while the system is running.
11 |
12 | To be explicit about the account's purpose, we will call the new user backup. We will be placing the user's credentials in a secure file, so feel free to choose a complex password:
13 |
14 | ```
15 | mysql> CREATE USER 'backup'@'localhost' IDENTIFIED BY 'password';
16 | ```
17 |
18 | Next we need to grant the new **backup** user the permissions it needs to perform all backup actions on the database system. Grant the required privileges and apply them to the current session by typing:
19 |
20 | ```
21 | mysql> GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT, CREATE TABLESPACE, PROCESS, SUPER, CREATE, INSERT, SELECT ON *.* TO 'backup'@'localhost';
22 | mysql> FLUSH PRIVILEGES;
23 | ```
24 |
25 | Our MySQL backup user is configured and has the access it requires.
26 |
27 | ## Display the value of the datadir variable
28 |
29 | ```
30 | mysql> SELECT @@datadir;
31 |
32 | +-----------------+
33 | | @@datadir |
34 | +-----------------+
35 | | /var/lib/mysql/ |
36 | +-----------------+
37 | 1 row in set (0.01 sec)
38 | ```
39 |
40 | Take a note of the location you find.
41 |
42 | ## Configuring a Systems Backup User and Assigning Permissions
43 |
44 | Now that we have a MySQL user to perform backups, we will ensure that a corresponding Linux user exists with similar limited privileges.
45 |
46 | On Ubuntu 16.04 / Debian 8, a backup user and corresponding backup group is already available. Confirm this by checking the /etc/passwd and /etc/group files with the following command:
47 |
48 | ```
49 | $ grep backup /etc/passwd /etc/group
50 |
51 | /etc/passwd:backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
52 | /etc/group:backup:x:34:
53 |
54 | ```
55 | The first line from the /etc/passwd file describes the backup user, while the second line from the /etc/group file defines the backup group.
56 |
57 | The /var/lib/mysql directory where the MySQL data is kept is owned by the mysql user and group. We can add the backup user to the mysql group to safely allow access to the database files and directories. We should also add our sudo user to the backup group so that we can access the files we will back up.
58 |
59 | Type the following commands to add the backup user to the mysql group and your sudo user to the backup group:
60 |
61 | ```
62 | $ sudo usermod -aG mysql backup
63 | $ sudo usermod -aG backup ${USER}
64 | ```
65 |
66 | If we check the /etc/group files again, you will see that your current user is added to the backup group and that the backup user is added to the mysql group:
67 |
68 | ```
69 | $ grep backup /etc/group
70 |
71 | backup:x:34:sammy
72 | mysql:x:116:backup
73 | ```
74 |
75 | The new group isn't available in our current session automatically. To re-evaluate the groups available to our sudo user, either log out and log back in, or type:
76 |
77 | ```
78 | $ exec su - ${USER}
79 | ```
80 |
81 | You will be prompted for your sudo user's password to continue. Confirm that your current session now has access to the backup group by checking our user's groups again:
82 |
83 | ```
84 | $ id -nG
85 |
86 | sammy sudo backup
87 | ```
88 |
89 | Our sudo user will now be able to take advantage of its membership in the backup group.
90 |
91 | Next, we need to make the /var/lib/mysql directory and its subdirectories accessible to the mysql group by adding group execute permissions. Otherwise, the backup user will be unable to enter those directories, even though it is a member of the mysql group.
92 |
93 | > **Note**: If the value of datadir was not **/var/lib/mysql** when you checked inside of MySQL earlier, substitute the directory you discovered in the commands that follow.
94 |
95 | To give the mysql group access to the MySQL data directories, type:
96 |
97 | ```
98 | $ sudo find /var/lib/mysql -type d -exec chmod 750 {} \;
99 | ```
100 |
101 | Our backup user now has the access it needs to the MySQL directory.
102 |
103 | ## Creating the Backup Assets
104 |
105 | Now that MySQL and system backup users are available, we can begin to set up the configuration files, encryption keys, and other assets that we need to successfully create and secure our backups.
106 |
107 | ### Create a MySQL Configuration File with the Backup Parameters
108 |
109 | Begin by creating a minimal MySQL configuration file that the backup script will use. This will contain the MySQL credentials for the MySQL user.
110 |
111 | Open a file at **/etc/mysql/backup.cnf** in your text editor:
112 |
113 | ```
114 | $ sudo nano /etc/mysql/backup.cnf
115 | ```
116 |
117 | Inside, start a ```[client]``` section and set the MySQL backup user and password user you defined within MySQL:
118 |
119 | ```
120 | [client]
121 | user=backup
122 | password=password
123 | ```
124 |
125 | Save and close the file when you are finished.
126 |
127 | Give ownership of the file to the backup user and then restrict the permissions so that no other users can access the file:
128 |
129 | ```
130 | $ sudo chown backup /etc/mysql/backup.cnf
131 | $ sudo chmod 600 /etc/mysql/backup.cnf
132 | ```
133 |
134 | The backup user will be able to access this file to get the proper credentials but other users will be restricted.
135 |
136 | ### Create a Backup Root Directory
137 |
138 | Next, create a directory for the backup content. We will use ```/backups/mysql``` as the base directory for our backups:
139 |
140 | ```
141 | $ sudo mkdir -p /backups/mysql
142 | ```
143 |
144 | Next, assign ownership of the ```/backups/mysql``` directory to the backup user and group ownership to the mysql group:
145 |
146 | ```
147 | $ sudo chown backup:mysql /backups/mysql
148 | ```
149 |
150 | The ```backup``` user should now be able to write backup data to this location.
151 |
152 | ## Downloading the Backup and Restore Scripts
153 |
154 | You can download the scripts directly from GitHub by typing:
155 |
156 | ```
157 | $ cd /tmp
158 | $ curl -LO https://raw.githubusercontent.com/nullart/debian-ubuntu-mariadb-backup/master/backup-mysql.sh
159 | $ curl -LO https://raw.githubusercontent.com/nullart/debian-ubuntu-mariadb-backup/master/extract-mysql.sh
160 | $ curl -LO https://raw.githubusercontent.com/nullart/debian-ubuntu-mariadb-backup/master/prepare-mysql.sh
161 | ```
162 |
163 | Be sure to inspect the scripts after downloading to make sure they were retrieved successfully and that you approve of the actions they will perform. If you are satisfied, mark the scripts as executable and then move them into the /usr/local/bin directory by typing:
164 |
165 | ```
166 | $ chmod +x /tmp/{backup,extract,prepare}-mysql.sh
167 | $ sudo mv /tmp/{backup,extract,prepare}-mysql.sh /usr/local/bin
168 | ```
169 |
170 | ## Using the Backup and Restore Scripts
171 |
172 | In order to make our backup and restore steps repeatable, we will script the entire process. We will use the following scripts:
173 |
174 | * backup-mysql.sh: This script backs up the MySQL databases, encrypting and compressing the files in the process. It creates full and incremental backups and automatically organizes content by day. By default, the script maintains 3 days worth of backups.
175 | * extract-mysql.sh: This script decompresses and decrypts the backup files to create directories with the backed up content.
176 | * prepare-mysql.sh: This script "prepares" the back up directories by processing the files and applying logs. Any incremental backups are applied to the full backup. Once the prepare script finishes, the files are ready to be moved back to the data directory.
177 |
178 | Be sure to inspect the scripts after downloading to make sure they were retrieved successfully and that you approve of the actions they will perform. If you are satisfied, mark the scripts as executable and then move them into the ```/usr/local/bin``` directory by typing:
179 |
180 | ```
181 | $ chmod +x /tmp/{backup,extract,prepare}-mysql.sh
182 | $ sudo mv /tmp/{backup,extract,prepare}-mysql.sh /usr/local/bin
183 | ```
184 |
185 | ### The backup-mysql.sh Script
186 |
187 | The script has the following functionality:
188 |
189 | * Creates a compressed full backup the first time it is run each day.
190 | * Generates compressed incremental backups based on the daily full backup when called again on the same day.
191 | * Maintains backups organized by day. By default, three days of backups are kept. This can be changed by adjusting the days_of_backups parameter within the script.
192 |
193 | When the script is run, a daily directory is created where timestamped files representing individual backups will be written. The first timestamped file will be a full backup, prefixed by full-. Subsequent backups for the day will be incremental backups, indicated by an incremental- prefix, representing the changes since the last full or incremental backup.
194 |
195 | Backups will generate a file called backup-progress.log in the daily directory with the output from the most recent backup operation. A file called xtrabackup_checkpoints containing the most recent backup metadata will be created there as well. This file is needed to produce future incremental backups, so it is important not to remove it. A file called xtrabackup_info, which contains additional metadata, is also produced but the script does not reference this file.
196 |
197 | ### The extract-mysql.sh Script
198 |
199 | Unlike the backup-mysql.sh script, which is designed to be automated, this script is designed to be used intentionally when you plan to restore from a backup. Because of this, the script expects you to pass in the .xbstream files that you wish to extract.
200 |
201 | The script creates a restore directory within the current directory and then creates individual directories within for each of the backups passed in as arguments. It will process the provided .xbstream files by extracting directory structure from the archive, decrypting the individual files within, and then decompressing the decrypted files.
202 |
203 | After this process has completed, the restore directory should contain directories for each of the provided backups. This allows you to inspect the directories, examine the contents of the backups, and decide which backups you wish to prepare and restore.
204 |
205 | ### The prepare-mysql.sh Script
206 |
207 | This script will apply the logs to each backup to create a consistent database snapshot. It will apply any incremental backups to the full backup to incorporate the later changes.
208 |
209 | The script looks in the current directory for directories beginning with full- or incremental-. It uses the MySQL logs to apply the committed transactions to the full backup. Afterwards, it applies any incremental backups to the full backup to update the data with the more recent information, again applying the committed transactions.
210 |
211 | Once all of the backups have been combined, the uncommitted transactions are rolled back. At this point, the full- backup will represent a consistent set of data that can be moved into MySQL's data directory.
212 |
213 | In order to minimize chance of data loss, the script stops short of copying the files into the data directory. This way, the user can manually verify the backup contents and the log file created during this process, and decide what to do with the current contents of the MySQL data directory. The commands needed to restore the files completely are displayed when the command exits.
214 |
215 | ## Testing the MySQL Backup and Restore Scripts
216 |
217 | ### Perform a Full Backup
218 |
219 | ```
220 | $ sudo -u backup backup-mysql.sh
221 |
222 | Backup successful!
223 | Backup created at /backups/mysql/Thu/full-04-20-2017_14-55-17.xbstream
224 |
225 | ```
226 |
227 | If everything went as planned, the script will execute correctly, indicate success, and output the location of the new backup file. As the above output indicates, a daily directory ("Thu" in this case) has been created to house the day's backups. The backup file itself begins with full- to express that this is a full backup.
228 |
229 | Let's move into the daily backup directory and view the contents:
230 |
231 | ```
232 | $ cd /backups/mysql/"$(date +%a)"
233 | $ ls
234 |
235 | backup-progress.log full-04-20-2017_14-55-17.xbstream xtrabackup_checkpoints xtrabackup_info
236 |
237 | ```
238 |
239 | Here, we see the actual backup file (full-04-20-2017_14-55-17.xbstream in this case), the log of the backup event (backup-progress.log), the xtrabackup_checkpoints file, which includes metadata about the backed up content, and the xtrabackup_info file, which contains additional metadata.
240 |
241 | If we tail the backup-progress.log, we can confirm that the backup completed successfully.
242 |
243 | ```
244 | $ tail backup-progress.log
245 |
246 | 170420 14:55:19 All tables unlocked
247 | 170420 14:55:19 [00] Compressing, encrypting and streaming ib_buffer_pool to
248 | 170420 14:55:19 [00] ...done
249 | 170420 14:55:19 Backup created in directory '/backups/mysql/Thu/'
250 | 170420 14:55:19 [00] Compressing, encrypting and streaming backup-my.cnf
251 | 170420 14:55:19 [00] ...done
252 | 170420 14:55:19 [00] Compressing, encrypting and streaming xtrabackup_info
253 | 170420 14:55:19 [00] ...done
254 | xtrabackup: Transaction log of lsn (2549956) to (2549965) was copied.
255 | 170420 14:55:19 completed OK!
256 | ```
257 |
258 | If we look at the xtrabackup_checkpoints file, we can view information about the backup. While this file provides some information that is useful for administrators, it's mainly used by subsequent backup jobs so that they know what data has already been processed.
259 |
260 | This is a copy of a file that's included in each archive. Even though this copy is overwritten with each backup to represent the latest information, each original will still be available inside the backup archive.
261 |
262 | ```
263 | $ cat xtrabackup_checkpoints
264 |
265 | backup_type = full-backuped
266 | from_lsn = 0
267 | to_lsn = 2549956
268 | last_lsn = 2549965
269 | compact = 0
270 | recover_binlog_info = 0
271 | ```
272 |
273 | The example above tells us that a full backup was taken and that the backup covers log sequence number (LSN) 0 to log sequence number 2549956. The last_lsn number indicates that some operations occurred during the backup process.
274 |
275 | ### Perform an Incremental Backup
276 |
277 | Now that we have a full backup, we can take additional incremental backups. Incremental backups record the changes that have been made since the last backup was performed. The first incremental backup is based on a full backup and subsequent incremental backups are based on the previous incremental backup.
278 |
279 | We should add some data to our database before taking another backup so that we can tell which backups have been applied.
280 |
281 | Insert another record into the equipment table of our playground database representing 10 yellow swings. You will be prompted for the MySQL administrative password during this process.
282 |
283 | Now that there is more current data than our most recent backup, we can take an incremental backup to capture the changes. The backup-mysql.sh script will take an incremental backup if a full backup for the same day exists:
284 |
285 | ```
286 | $ sudo -u backup backup-mysql.sh
287 |
288 | Backup successful!
289 | Backup created at /backups/mysql/Thu/incremental-04-20-2017_17-15-03.xbstream
290 | ```
291 |
292 | Check the daily backup directory again to find the incremental backup archive:
293 |
294 | ```
295 | $ cd /backups/mysql/"$(date +%a)"
296 | $ ls
297 |
298 | backup-progress.log incremental-04-20-2017_17-15-03.xbstream xtrabackup_info
299 | full-04-20-2017_14-55-17.xbstream xtrabackup_checkpoints
300 | ```
301 |
302 | The contents of the xtrabackup_checkpoints file now refer to the most recent incremental backup:
303 |
304 | ```
305 | $ cat xtrabackup_checkpoints
306 |
307 | backup_type = incremental
308 | from_lsn = 2549956
309 | to_lsn = 2550159
310 | last_lsn = 2550168
311 | compact = 0
312 | recover_binlog_info = 0
313 | ```
314 |
315 | The backup type is listed as "incremental" and instead of starting from LSN 0 like our full backup, it starts at the LSN where our last backup ended.
316 |
317 | ### Extract the Backups
318 |
319 | Next, let's extract the backup files to create backup directories. Due to space and security considerations, this should normally only be done when you are ready to restore the data.
320 |
321 | We can extract the backups by passing the .xbstream backup files to the extract-mysql.sh script. Again, this must be run by the backup user:
322 |
323 | ```
324 | $ sudo -u backup extract-mysql.sh *.xbstream
325 |
326 | Extraction complete! Backup directories have been extracted to the "restore" directory.
327 |
328 | ```
329 |
330 | The above output indicates that the process was completed successfully. If we check the contents of the daily backup directory again, an extract-progress.log file and a restore directory have been created.
331 |
332 | If we tail the extraction log, we can confirm that the latest backup was extracted successfully. The other backup success messages are displayed earlier in the file.
333 |
334 |
335 |
336 | ```
337 | $ tail extract-progress.log
338 |
339 | 170420 17:23:32 [01] decrypting and decompressing ./performance_schema/socket_instances.frm.qp.xbcrypt
340 | 170420 17:23:32 [01] decrypting and decompressing ./performance_schema/events_waits_summary_by_user_by_event_name.frm.qp.xbcrypt
341 | 170420 17:23:32 [01] decrypting and decompressing ./performance_schema/status_by_user.frm.qp.xbcrypt
342 | 170420 17:23:32 [01] decrypting and decompressing ./performance_schema/replication_group_members.frm.qp.xbcrypt
343 | 170420 17:23:32 [01] decrypting and decompressing ./xtrabackup_logfile.qp.xbcrypt
344 | 170420 17:23:33 completed OK!
345 |
346 |
347 | Finished work on incremental-04-20-2017_17-15-03.xbstream
348 | ```
349 |
350 | If we move into the restore directory, directories corresponding with the backup files we extracted are now available:
351 |
352 | ```
353 | $ cd restore
354 | $ ls -F
355 |
356 | full-04-20-2017_14-55-17/ incremental-04-20-2017_17-15-03/
357 | ```
358 |
359 | The backup directories contains the raw backup files, but they are not yet in a state that MySQL can use though. To fix that, we need to prepare the files.
360 |
361 | ### Prepare the Final Backup
362 |
363 | Next, we will prepare the backup files. To do so, you must be in the restore directory that contains the full- and any incremental- backups. The script will apply the changes from any incremental- directories onto the full- backup directory. Afterwards, it will apply the logs to create a consistent dataset that MySQL can use.
364 |
365 | If for any reason you don't want to restore some of the changes, now is your last chance to remove those incremental backup directories from the restore directory (the incremental backup files will still be available in the parent directory). Any remaining incremental- directories within the current directory will be applied to the full- backup directory.
366 |
367 | When you are ready, call the prepare-mysql.sh script. Again, make sure you are in the restore directory where your individual backup directories are located:
368 |
369 | ```
370 | $ sudo -u backup prepare-mysql.sh
371 |
372 | Backup looks to be fully prepared. Please check the "prepare-progress.log" file
373 | to verify before continuing.
374 |
375 | If everything looks correct, you can apply the restored files.
376 |
377 | First, stop MySQL and move or remove the contents of the MySQL data directory:
378 |
379 | sudo systemctl stop mysql
380 | sudo mv /var/lib/mysql/ /tmp/
381 |
382 | Then, recreate the data directory and copy the backup files:
383 |
384 | sudo mkdir /var/lib/mysql
385 | sudo xtrabackup --copy-back --target-dir=/backups/mysql/Thu/restore/full-04-20-2017_14-55-17
386 |
387 | Afterward the files are copied, adjust the permissions and restart the service:
388 |
389 | sudo chown -R mysql:mysql /var/lib/mysql
390 | sudo find /var/lib/mysql -type d -exec chmod 750 {} \;
391 | sudo systemctl start mysql
392 | ```
393 |
394 | The output above indicates that the script thinks that the backup is fully prepared and that the full- backup now represents a fully consistent dataset. As the output states, you should check the prepare-progress.log file to confirm that no errors were reported during the process.
395 |
396 | The script stops short of actually copying the files into MySQL's data directory so that you can verify that everything looks correct.
397 |
398 | ### Restore the Backup Data to the MySQL Data Directory
399 |
400 | If you are satisfied that everything is in order after reviewing the logs, you can follow the instructions outlined in the prepare-mysql.sh output.
401 |
402 | First, stop the running MySQL process:
403 |
404 | ```
405 | $ sudo systemctl stop mysql
406 | ```
407 |
408 | Since the backup data may conflict with the current contents of the MySQL data directory, we should remove or move the /var/lib/mysql directory. If you have space on your filesystem, the best option is to move the current contents to the /tmp directory or elsewhere in case something goes wrong:
409 |
410 | ```
411 | $ sudo mv /var/lib/mysql/ /tmp
412 | ```
413 |
414 | Recreate an empty /var/lib/mysql directory. We will need to fix permissions in a moment, so we do not need to worry about that yet:
415 |
416 | ```
417 | $ sudo mkdir /var/lib/mysql
418 | ```
419 |
420 | Now, we can copy the full backup to the MySQL data directory using the xtrabackup utility. Substitute the path to your prepared full backup in the command below:
421 |
422 | ```
423 | sudo mariabackup --copy-back --target-dir=/backups/mysql/Thu/restore/full-04-20-2017_14-55-17
424 | ```
425 |
426 | A running log of the files being copied will display throughout the process. Once the files are in place, we need to fix the ownership and permissions again so that the MySQL user and group own and can access the restored structure:
427 |
428 | ```
429 | $ sudo chown -R mysql:mysql /var/lib/mysql
430 | $ sudo find /var/lib/mysql -type d -exec chmod 750 {} \;
431 | ```
432 |
433 | Our restored files are now in the MySQL data directory.
434 |
435 | Start up MySQL again to complete the process:
436 |
437 | ```
438 | $ sudo systemctl start mysql
439 | ```
440 |
441 | After restoring your data, it is important to go back and delete the restore directory. Future incremental backups cannot be applied to the full backup once it has been prepared, so we should remove it. Furthermore, the backup directories should not be left unencrypted on disk for security reasons:
442 |
443 | ```
444 | $ cd ~
445 | $ sudo rm -rf /backups/mysql/"$(date +%a)"/restore
446 | ```
447 |
448 | The next time we need a clean copies of the backup directories, we can extract them again from the backup files.
449 |
450 | ## Creating a Cron Job to Run Backups Hourly
451 |
452 | Now that we've verified that the backup and restore process are working smoothly, we should set up a cron job to automatically take regular backups.
453 |
454 | We will create a small script within the /etc/cron.hourly directory to automatically run our backup script and log the results. The cron process will automatically run this every hour:
455 |
456 | ```
457 | $ sudo nano /etc/cron.hourly/backup-mysql
458 | ```
459 |
460 | Inside, we will call the backup script with the systemd-cat utility so that the output will be available in the journal. We'll mark them with a backup-mysql identifier so we can easily filter the logs:
461 |
462 | ```
463 | #!/bin/bash
464 | sudo -u backup systemd-cat --identifier=backup-mysql /usr/local/bin/backup-mysql.sh
465 | ```
466 |
467 | Save and close the file when you are finished. Make the script executable by typing:
468 |
469 | ```
470 | $ sudo chmod +x /etc/cron.hourly/backup-mysql
471 | ```
472 |
473 | The backup script will now run hourly. The script itself will take care of cleaning up backups older than three days ago.
474 |
475 | We can test the cron script by running it manually:
476 |
477 | ```
478 | sudo /etc/cron.hourly/backup-mysql
479 | ```
480 |
481 | After it completes, check the journal for the log messages by typing:
482 |
483 | ```
484 | $ sudo journalctl -t backup-mysql
485 |
486 | -- Logs begin at Wed 2017-04-19 18:59:23 UTC, end at Thu 2017-04-20 18:54:49 UTC. --
487 | Apr 20 18:35:07 myserver backup-mysql[2302]: Backup successful!
488 | Apr 20 18:35:07 myserver backup-mysql[2302]: Backup created at /backups/mysql/Thu/incremental-04-20-2017_18-35-05.xbstream
489 | ```
490 |
491 | Check back in a few hours to make sure that additional backups are being taken.
492 |
493 | ## Hope this helps! Cheers!
494 |
495 |
--------------------------------------------------------------------------------
/backup-mysql.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | export LC_ALL=C
4 |
5 | days_of_backups=3 # Must be less than 7
6 | backup_owner="backup"
7 | parent_dir="/backups/mysql"
8 | defaults_file="/etc/mysql/backup.cnf"
9 | todays_dir="${parent_dir}/$(date +%a)"
10 | log_file="${todays_dir}/backup-progress.log"
11 | #encryption_key_file="${parent_dir}/encryption_key"
12 | now="$(date +%m-%d-%Y_%H-%M-%S)"
13 | processors="$(nproc --all)"
14 |
15 | # Use this to echo to standard error
16 | error () {
17 | printf "%s: %s\n" "$(basename "${BASH_SOURCE}")" "${1}" >&2
18 | exit 1
19 | }
20 |
21 | trap 'error "An unexpected error occurred."' ERR
22 |
23 | sanity_check () {
24 | # Check user running the script
25 | if [ "$USER" != "$backup_owner" ]; then
26 | error "Script can only be run as the \"$backup_owner\" user"
27 | fi
28 |
29 | # Check whether the encryption key file is available
30 | #if [ ! -r "${encryption_key_file}" ]; then
31 | # error "Cannot read encryption key at ${encryption_key_file}"
32 | #fi
33 | }
34 |
35 | set_options () {
36 | # List the innobackupex arguments
37 | #declare -ga innobackupex_args=(
38 | #"--encrypt=AES256"
39 | #"--encrypt-key-file=${encryption_key_file}"
40 | #"--encrypt-threads=${processors}"
41 | #"--slave-info"
42 | #"--incremental"
43 |
44 | innobackupex_args=(
45 | "--defaults-file=${defaults_file}"
46 | "--extra-lsndir=${todays_dir}"
47 | "--backup"
48 | "--compress"
49 | "--stream=xbstream"
50 | "--parallel=${processors}"
51 | "--compress-threads=${processors}"
52 | )
53 |
54 | backup_type="full"
55 |
56 | # Add option to read LSN (log sequence number) if a full backup has been
57 | # taken today.
58 | if grep -q -s "to_lsn" "${todays_dir}/xtrabackup_checkpoints"; then
59 | backup_type="incremental"
60 | lsn=$(awk '/to_lsn/ {print $3;}' "${todays_dir}/xtrabackup_checkpoints")
61 | innobackupex_args+=( "--incremental-lsn=${lsn}" )
62 | fi
63 | }
64 |
65 | rotate_old () {
66 | # Remove the oldest backup in rotation
67 | day_dir_to_remove="${parent_dir}/$(date --date="${days_of_backups} days ago" +%a)"
68 |
69 | if [ -d "${day_dir_to_remove}" ]; then
70 | rm -rf "${day_dir_to_remove}"
71 | fi
72 | }
73 |
74 | take_backup () {
75 | # Make sure today's backup directory is available and take the actual backup
76 | mkdir -p "${todays_dir}"
77 | find "${todays_dir}" -type f -name "*.incomplete" -delete
78 | #innobackupex "${innobackupex_args[@]}" "${todays_dir}" > "${todays_dir}/${backup_type}-${now}.xbstream.incomplete" 2> "${log_file}"
79 | mariabackup "${innobackupex_args[@]}" "--target-dir=${todays_dir}" > "${todays_dir}/${backup_type}-${now}.xbstream.incomplete" 2> "${log_file}"
80 |
81 | mv "${todays_dir}/${backup_type}-${now}.xbstream.incomplete" "${todays_dir}/${backup_type}-${now}.xbstream"
82 | }
83 |
84 | sanity_check && set_options && rotate_old && take_backup
85 |
86 | # Check success and print message
87 | if tail -1 "${log_file}" | grep -q "completed OK"; then
88 | printf "Backup successful!\n"
89 | printf "Backup created at %s/%s-%s.xbstream\n" "${todays_dir}" "${backup_type}" "${now}"
90 | else
91 | error "Backup failure! Check ${log_file} for more information"
92 | fi
93 |
--------------------------------------------------------------------------------
/extract-mysql.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | export LC_ALL=C
4 |
5 | backup_owner="backup"
6 | #encryption_key_file="/backups/mysql/encryption_key"
7 | log_file="extract-progress.log"
8 | number_of_args="${#}"
9 | processors="$(nproc --all)"
10 |
11 | # Use this to echo to standard error
12 | error () {
13 | printf "%s: %s\n" "$(basename "${BASH_SOURCE}")" "${1}" >&2
14 | exit 1
15 | }
16 |
17 | trap 'error "An unexpected error occurred. Try checking the \"${log_file}\" file for more information."' ERR
18 |
19 | sanity_check () {
20 | # Check user running the script
21 | if [ "${USER}" != "${backup_owner}" ]; then
22 | error "Script can only be run as the \"${backup_owner}\" user"
23 | fi
24 |
25 | # Check whether the qpress binary is installed
26 | if ! command -v qpress >/dev/null 2>&1; then
27 | error "Could not find the \"qpress\" command. Please install it and try again."
28 | fi
29 |
30 | # Check whether any arguments were passed
31 | if [ "${number_of_args}" -lt 1 ]; then
32 | error "Script requires at least one \".xbstream\" file as an argument."
33 | fi
34 |
35 | # Check whether the encryption key file is available
36 | #if [ ! -r "${encryption_key_file}" ]; then
37 | # error "Cannot read encryption key at ${encryption_key_file}"
38 | #fi
39 | }
40 |
41 | do_extraction () {
42 | for file in "${@}"; do
43 | base_filename="$(basename "${file%.xbstream}")"
44 | restore_dir="./restore/${base_filename}"
45 |
46 | printf "\n\nExtracting file %s\n\n" "${file}"
47 |
48 | # Extract the directory structure from the backup file
49 | mkdir --verbose -p "${restore_dir}"
50 | mbstream -x -C "${restore_dir}" < "${file}"
51 | #"--decrypt=AES256"
52 | #"--encrypt-key-file=${encryption_key_file}"
53 | innobackupex_args=(
54 | "--parallel=${processors}"
55 | "--decompress"
56 | )
57 |
58 | #innobackupex "${innobackupex_args[@]}" "${restore_dir}"
59 | mariabackup "${innobackupex_args[@]}" --target-dir="${restore_dir}"
60 | #find "${restore_dir}" -name "*.xbcrypt" -exec rm {} \;
61 | find "${restore_dir}" -name "*.qp" -exec rm {} \;
62 |
63 | printf "\n\nFinished work on %s\n\n" "${file}"
64 |
65 | done > "${log_file}" 2>&1
66 | }
67 |
68 | sanity_check && do_extraction "$@"
69 |
70 | ok_count="$(grep -c 'completed OK' "${log_file}")"
71 |
72 | # Check the number of reported completions. For each file, there is an
73 | # informational "completed OK". If the processing was successful, an
74 | # additional "completed OK" is printed. Together, this means there should be 2
75 | # notices per backup file if the process was successful.
76 | if (( $ok_count != 2 * $# )); then
77 | error "It looks like something went wrong. Please check the \"${log_file}\" file for additional information"
78 | else
79 | printf "Extraction complete! Backup directories have been extracted to the \"restore\" directory.\n"
80 | fi
81 |
--------------------------------------------------------------------------------
/prepare-mysql.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | export LC_ALL=C
4 |
5 | shopt -s nullglob
6 | incremental_dirs=( ./incremental-*/ )
7 | full_dirs=( ./full-*/ )
8 | shopt -u nullglob
9 |
10 | backup_owner="backup"
11 | log_file="prepare-progress.log"
12 | full_backup_dir="${full_dirs[0]}"
13 |
14 | # Use this to echo to standard error
15 | error() {
16 | printf "%s: %s\n" "$(basename "${BASH_SOURCE}")" "${1}" >&2
17 | exit 1
18 | }
19 |
20 | trap 'error "An unexpected error occurred. Try checking the \"${log_file}\" file for more information."' ERR
21 |
22 | sanity_check () {
23 | # Check user running the script
24 | if [ "${USER}" != "${backup_owner}" ]; then
25 | error "Script can only be run as the \"${backup_owner}\" user."
26 | fi
27 |
28 | # Check whether a single full backup directory are available
29 | if (( ${#full_dirs[@]} != 1 )); then
30 | error "Exactly one full backup directory is required."
31 | fi
32 | }
33 |
34 | do_backup () {
35 | # Apply the logs to each of the backups
36 | printf "Initial prep of full backup %s\n" "${full_backup_dir}"
37 | #innobackupex --redo-only --apply-log "${full_backup_dir}"
38 | mariabackup --apply-log "${full_backup_dir}"
39 |
40 | for increment in "${incremental_dirs[@]}"; do
41 | printf "Applying incremental backup %s to %s\n" "${increment}" "${full_backup_dir}"
42 | #innobackupex --redo-only --apply-log --incremental-dir="${increment}" "${full_backup_dir}"
43 | mariabackup --apply-log --incremental-dir="${increment}" "${full_backup_dir}"
44 | done
45 |
46 | printf "Applying final logs to full backup %s\n" "${full_backup_dir}"
47 | #innobackupex --apply-log "${full_backup_dir}"
48 | mariabackup --apply-log "${full_backup_dir}"
49 | }
50 |
51 | sanity_check && do_backup > "${log_file}" 2>&1
52 |
53 | # Check the number of reported completions. Each time a backup is processed,
54 | # an informational "completed OK" and a real version is printed. At the end of
55 | # the process, a final full apply is performed, generating another 2 messages.
56 | ok_count="$(grep -c 'completed OK' "${log_file}")"
57 |
58 | if (( ${ok_count} == 2 * (${#full_dirs[@]} + ${#incremental_dirs[@]} + 1) )); then
59 | cat << EOF
60 | Backup looks to be fully prepared. Please check the "prepare-progress.log" file
61 | to verify before continuing.
62 |
63 | If everything looks correct, you can apply the restored files.
64 |
65 | First, stop MySQL and move or remove the contents of the MySQL data directory:
66 |
67 | sudo systemctl stop mysql
68 | sudo mv /var/lib/mysql/ /tmp/
69 |
70 | Then, recreate the data directory and copy the backup files:
71 |
72 | sudo mkdir /var/lib/mysql
73 | sudo mariabackup --copy-back ${PWD}/$(basename "${full_backup_dir}")
74 |
75 | Afterward the files are copied, adjust the permissions and restart the service:
76 |
77 | sudo chown -R mysql:mysql /var/lib/mysql
78 | sudo find /var/lib/mysql -type d -exec chmod 750 {} \\;
79 | sudo systemctl start mysql
80 | EOF
81 | else
82 | error "It looks like something went wrong. Check the \"${log_file}\" file for more information."
83 | fi
84 |
--------------------------------------------------------------------------------