├── LICENSE ├── README.md ├── pyxbackup ├── scripts ├── monitor-backup-age.sh └── monitor-binlog-stream.sh ├── tests ├── all_test.py ├── pyxbackup-binlog.py └── pyxbackup.py └── vagrant ├── Vagrantfile └── ansible ├── files ├── binaries-mysql ├── binaries-xtrabackup ├── commands-pyxbackup ├── make-sandboxes.sh ├── make-xtrabackups.sh ├── mysql-sandbox.sh ├── run-sysbench.sh └── run-tests.sh ├── pyxbackup.yml ├── tasks ├── linux-selinux.yml ├── linux-ssh.yml ├── repo-epel.yml ├── repo-percona.yml └── repo-twindb.yml └── templates └── pyxbackup.cnf /LICENSE: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 2, June 1991 3 | 4 | Copyright (C) 1989, 1991 Free Software Foundation, Inc., 5 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA 6 | Everyone is permitted to copy and distribute verbatim copies 7 | of this license document, but changing it is not allowed. 8 | 9 | Preamble 10 | 11 | The licenses for most software are designed to take away your 12 | freedom to share and change it. By contrast, the GNU General Public 13 | License is intended to guarantee your freedom to share and change free 14 | software--to make sure the software is free for all its users. This 15 | General Public License applies to most of the Free Software 16 | Foundation's software and to any other program whose authors commit to 17 | using it. (Some other Free Software Foundation software is covered by 18 | the GNU Lesser General Public License instead.) You can apply it to 19 | your programs, too. 20 | 21 | When we speak of free software, we are referring to freedom, not 22 | price. Our General Public Licenses are designed to make sure that you 23 | have the freedom to distribute copies of free software (and charge for 24 | this service if you wish), that you receive source code or can get it 25 | if you want it, that you can change the software or use pieces of it 26 | in new free programs; and that you know you can do these things. 27 | 28 | To protect your rights, we need to make restrictions that forbid 29 | anyone to deny you these rights or to ask you to surrender the rights. 30 | These restrictions translate to certain responsibilities for you if you 31 | distribute copies of the software, or if you modify it. 32 | 33 | For example, if you distribute copies of such a program, whether 34 | gratis or for a fee, you must give the recipients all the rights that 35 | you have. You must make sure that they, too, receive or can get the 36 | source code. And you must show them these terms so they know their 37 | rights. 38 | 39 | We protect your rights with two steps: (1) copyright the software, and 40 | (2) offer you this license which gives you legal permission to copy, 41 | distribute and/or modify the software. 42 | 43 | Also, for each author's protection and ours, we want to make certain 44 | that everyone understands that there is no warranty for this free 45 | software. If the software is modified by someone else and passed on, we 46 | want its recipients to know that what they have is not the original, so 47 | that any problems introduced by others will not reflect on the original 48 | authors' reputations. 49 | 50 | Finally, any free program is threatened constantly by software 51 | patents. We wish to avoid the danger that redistributors of a free 52 | program will individually obtain patent licenses, in effect making the 53 | program proprietary. To prevent this, we have made it clear that any 54 | patent must be licensed for everyone's free use or not licensed at all. 55 | 56 | The precise terms and conditions for copying, distribution and 57 | modification follow. 58 | 59 | GNU GENERAL PUBLIC LICENSE 60 | TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 61 | 62 | 0. This License applies to any program or other work which contains 63 | a notice placed by the copyright holder saying it may be distributed 64 | under the terms of this General Public License. The "Program", below, 65 | refers to any such program or work, and a "work based on the Program" 66 | means either the Program or any derivative work under copyright law: 67 | that is to say, a work containing the Program or a portion of it, 68 | either verbatim or with modifications and/or translated into another 69 | language. (Hereinafter, translation is included without limitation in 70 | the term "modification".) Each licensee is addressed as "you". 71 | 72 | Activities other than copying, distribution and modification are not 73 | covered by this License; they are outside its scope. The act of 74 | running the Program is not restricted, and the output from the Program 75 | is covered only if its contents constitute a work based on the 76 | Program (independent of having been made by running the Program). 77 | Whether that is true depends on what the Program does. 78 | 79 | 1. You may copy and distribute verbatim copies of the Program's 80 | source code as you receive it, in any medium, provided that you 81 | conspicuously and appropriately publish on each copy an appropriate 82 | copyright notice and disclaimer of warranty; keep intact all the 83 | notices that refer to this License and to the absence of any warranty; 84 | and give any other recipients of the Program a copy of this License 85 | along with the Program. 86 | 87 | You may charge a fee for the physical act of transferring a copy, and 88 | you may at your option offer warranty protection in exchange for a fee. 89 | 90 | 2. You may modify your copy or copies of the Program or any portion 91 | of it, thus forming a work based on the Program, and copy and 92 | distribute such modifications or work under the terms of Section 1 93 | above, provided that you also meet all of these conditions: 94 | 95 | a) You must cause the modified files to carry prominent notices 96 | stating that you changed the files and the date of any change. 97 | 98 | b) You must cause any work that you distribute or publish, that in 99 | whole or in part contains or is derived from the Program or any 100 | part thereof, to be licensed as a whole at no charge to all third 101 | parties under the terms of this License. 102 | 103 | c) If the modified program normally reads commands interactively 104 | when run, you must cause it, when started running for such 105 | interactive use in the most ordinary way, to print or display an 106 | announcement including an appropriate copyright notice and a 107 | notice that there is no warranty (or else, saying that you provide 108 | a warranty) and that users may redistribute the program under 109 | these conditions, and telling the user how to view a copy of this 110 | License. (Exception: if the Program itself is interactive but 111 | does not normally print such an announcement, your work based on 112 | the Program is not required to print an announcement.) 113 | 114 | These requirements apply to the modified work as a whole. If 115 | identifiable sections of that work are not derived from the Program, 116 | and can be reasonably considered independent and separate works in 117 | themselves, then this License, and its terms, do not apply to those 118 | sections when you distribute them as separate works. But when you 119 | distribute the same sections as part of a whole which is a work based 120 | on the Program, the distribution of the whole must be on the terms of 121 | this License, whose permissions for other licensees extend to the 122 | entire whole, and thus to each and every part regardless of who wrote it. 123 | 124 | Thus, it is not the intent of this section to claim rights or contest 125 | your rights to work written entirely by you; rather, the intent is to 126 | exercise the right to control the distribution of derivative or 127 | collective works based on the Program. 128 | 129 | In addition, mere aggregation of another work not based on the Program 130 | with the Program (or with a work based on the Program) on a volume of 131 | a storage or distribution medium does not bring the other work under 132 | the scope of this License. 133 | 134 | 3. You may copy and distribute the Program (or a work based on it, 135 | under Section 2) in object code or executable form under the terms of 136 | Sections 1 and 2 above provided that you also do one of the following: 137 | 138 | a) Accompany it with the complete corresponding machine-readable 139 | source code, which must be distributed under the terms of Sections 140 | 1 and 2 above on a medium customarily used for software interchange; or, 141 | 142 | b) Accompany it with a written offer, valid for at least three 143 | years, to give any third party, for a charge no more than your 144 | cost of physically performing source distribution, a complete 145 | machine-readable copy of the corresponding source code, to be 146 | distributed under the terms of Sections 1 and 2 above on a medium 147 | customarily used for software interchange; or, 148 | 149 | c) Accompany it with the information you received as to the offer 150 | to distribute corresponding source code. (This alternative is 151 | allowed only for noncommercial distribution and only if you 152 | received the program in object code or executable form with such 153 | an offer, in accord with Subsection b above.) 154 | 155 | The source code for a work means the preferred form of the work for 156 | making modifications to it. For an executable work, complete source 157 | code means all the source code for all modules it contains, plus any 158 | associated interface definition files, plus the scripts used to 159 | control compilation and installation of the executable. However, as a 160 | special exception, the source code distributed need not include 161 | anything that is normally distributed (in either source or binary 162 | form) with the major components (compiler, kernel, and so on) of the 163 | operating system on which the executable runs, unless that component 164 | itself accompanies the executable. 165 | 166 | If distribution of executable or object code is made by offering 167 | access to copy from a designated place, then offering equivalent 168 | access to copy the source code from the same place counts as 169 | distribution of the source code, even though third parties are not 170 | compelled to copy the source along with the object code. 171 | 172 | 4. You may not copy, modify, sublicense, or distribute the Program 173 | except as expressly provided under this License. Any attempt 174 | otherwise to copy, modify, sublicense or distribute the Program is 175 | void, and will automatically terminate your rights under this License. 176 | However, parties who have received copies, or rights, from you under 177 | this License will not have their licenses terminated so long as such 178 | parties remain in full compliance. 179 | 180 | 5. You are not required to accept this License, since you have not 181 | signed it. However, nothing else grants you permission to modify or 182 | distribute the Program or its derivative works. These actions are 183 | prohibited by law if you do not accept this License. Therefore, by 184 | modifying or distributing the Program (or any work based on the 185 | Program), you indicate your acceptance of this License to do so, and 186 | all its terms and conditions for copying, distributing or modifying 187 | the Program or works based on it. 188 | 189 | 6. Each time you redistribute the Program (or any work based on the 190 | Program), the recipient automatically receives a license from the 191 | original licensor to copy, distribute or modify the Program subject to 192 | these terms and conditions. You may not impose any further 193 | restrictions on the recipients' exercise of the rights granted herein. 194 | You are not responsible for enforcing compliance by third parties to 195 | this License. 196 | 197 | 7. If, as a consequence of a court judgment or allegation of patent 198 | infringement or for any other reason (not limited to patent issues), 199 | conditions are imposed on you (whether by court order, agreement or 200 | otherwise) that contradict the conditions of this License, they do not 201 | excuse you from the conditions of this License. If you cannot 202 | distribute so as to satisfy simultaneously your obligations under this 203 | License and any other pertinent obligations, then as a consequence you 204 | may not distribute the Program at all. For example, if a patent 205 | license would not permit royalty-free redistribution of the Program by 206 | all those who receive copies directly or indirectly through you, then 207 | the only way you could satisfy both it and this License would be to 208 | refrain entirely from distribution of the Program. 209 | 210 | If any portion of this section is held invalid or unenforceable under 211 | any particular circumstance, the balance of the section is intended to 212 | apply and the section as a whole is intended to apply in other 213 | circumstances. 214 | 215 | It is not the purpose of this section to induce you to infringe any 216 | patents or other property right claims or to contest validity of any 217 | such claims; this section has the sole purpose of protecting the 218 | integrity of the free software distribution system, which is 219 | implemented by public license practices. Many people have made 220 | generous contributions to the wide range of software distributed 221 | through that system in reliance on consistent application of that 222 | system; it is up to the author/donor to decide if he or she is willing 223 | to distribute software through any other system and a licensee cannot 224 | impose that choice. 225 | 226 | This section is intended to make thoroughly clear what is believed to 227 | be a consequence of the rest of this License. 228 | 229 | 8. If the distribution and/or use of the Program is restricted in 230 | certain countries either by patents or by copyrighted interfaces, the 231 | original copyright holder who places the Program under this License 232 | may add an explicit geographical distribution limitation excluding 233 | those countries, so that distribution is permitted only in or among 234 | countries not thus excluded. In such case, this License incorporates 235 | the limitation as if written in the body of this License. 236 | 237 | 9. The Free Software Foundation may publish revised and/or new versions 238 | of the General Public License from time to time. Such new versions will 239 | be similar in spirit to the present version, but may differ in detail to 240 | address new problems or concerns. 241 | 242 | Each version is given a distinguishing version number. If the Program 243 | specifies a version number of this License which applies to it and "any 244 | later version", you have the option of following the terms and conditions 245 | either of that version or of any later version published by the Free 246 | Software Foundation. If the Program does not specify a version number of 247 | this License, you may choose any version ever published by the Free Software 248 | Foundation. 249 | 250 | 10. If you wish to incorporate parts of the Program into other free 251 | programs whose distribution conditions are different, write to the author 252 | to ask for permission. For software which is copyrighted by the Free 253 | Software Foundation, write to the Free Software Foundation; we sometimes 254 | make exceptions for this. Our decision will be guided by the two goals 255 | of preserving the free status of all derivatives of our free software and 256 | of promoting the sharing and reuse of software generally. 257 | 258 | NO WARRANTY 259 | 260 | 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY 261 | FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN 262 | OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES 263 | PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED 264 | OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF 265 | MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS 266 | TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE 267 | PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, 268 | REPAIR OR CORRECTION. 269 | 270 | 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 271 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR 272 | REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, 273 | INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING 274 | OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED 275 | TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY 276 | YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER 277 | PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE 278 | POSSIBILITY OF SUCH DAMAGES. 279 | 280 | END OF TERMS AND CONDITIONS 281 | 282 | How to Apply These Terms to Your New Programs 283 | 284 | If you develop a new program, and you want it to be of the greatest 285 | possible use to the public, the best way to achieve this is to make it 286 | free software which everyone can redistribute and change under these terms. 287 | 288 | To do so, attach the following notices to the program. It is safest 289 | to attach them to the start of each source file to most effectively 290 | convey the exclusion of warranty; and each file should have at least 291 | the "copyright" line and a pointer to where the full notice is found. 292 | 293 | {description} 294 | Copyright (C) {year} {fullname} 295 | 296 | This program is free software; you can redistribute it and/or modify 297 | it under the terms of the GNU General Public License as published by 298 | the Free Software Foundation; either version 2 of the License, or 299 | (at your option) any later version. 300 | 301 | This program is distributed in the hope that it will be useful, 302 | but WITHOUT ANY WARRANTY; without even the implied warranty of 303 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 304 | GNU General Public License for more details. 305 | 306 | You should have received a copy of the GNU General Public License along 307 | with this program; if not, write to the Free Software Foundation, Inc., 308 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 309 | 310 | Also add information on how to contact you by electronic and paper mail. 311 | 312 | If the program is interactive, make it output a short notice like this 313 | when it starts in an interactive mode: 314 | 315 | Gnomovision version 69, Copyright (C) year name of author 316 | Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 317 | This is free software, and you are welcome to redistribute it 318 | under certain conditions; type `show c' for details. 319 | 320 | The hypothetical commands `show w' and `show c' should show the appropriate 321 | parts of the General Public License. Of course, the commands you use may 322 | be called something other than `show w' and `show c'; they could even be 323 | mouse-clicks or menu items--whatever suits your program. 324 | 325 | You should also get your employer (if you work as a programmer) or your 326 | school, if any, to sign a "copyright disclaimer" for the program, if 327 | necessary. Here is a sample; alter the names: 328 | 329 | Yoyodyne, Inc., hereby disclaims all copyright interest in the program 330 | `Gnomovision' (which makes passes at compilers) written by James Hacker. 331 | 332 | {signature of Ty Coon}, 1 April 1989 333 | Ty Coon, President of Vice 334 | 335 | This General Public License does not permit incorporating your program into 336 | proprietary programs. If your program is a subroutine library, you may 337 | consider it more useful to permit linking proprietary applications with the 338 | library. If this is what you want to do, use the GNU Lesser General 339 | Public License instead of this License. 340 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | pyxbackup 2 | ********* 3 | 4 | Summary 5 | ======= 6 | 7 | This backup script is somewhat a rewrite of https://github.com/dotmanila/mootools/blob/master/xbackup.sh. 8 | 9 | Features 10 | ======== 11 | 12 | - Can prepare a full + incrementals set with one command 13 | - Keep backups (full and/or incrementals) prepared on source or remote server 14 | - Compression with xbstream+gzip, tar+gzip, xbstream+qpress 15 | - Support for encryption on top of compression via Xtrabackup encryption 16 | - Stream backups directly to remote servers via scp or netcat, can also keep local copies 17 | - Binary log streaming support with mysqlbinlog 5.6+ 18 | 19 | Dependencies 20 | ============ 21 | 22 | The script is initially tested only with Python 2.6 on CentOS 6.5 and Python 2.7 on Ubuntu 14.04 - running it on newer versions i.e. 3.x may lead to incompatibility issues. Will appreciate pointers/pull requests on making it compatible with Python 3.x! 23 | 24 | Also it requires that the xtrabackup binaries i.e. innobackupex, xtrabackup*, xbstream are found in your PATH environment. 25 | 26 | Configuration 27 | ============= 28 | 29 | A file called ``pyxbackup.cnf`` can store configuration values. By default, the script looks for this file from ``/etc/pyxbackup.cnf`` first, if not found, on the same directory where the script is installed. It can also be specified from a manual location with the ``--config`` CLI option. Some configuration options are exclusive to the command line, they are marked with ``(cli)`` when executing ``pyxbackup.py --help``. 30 | 31 | You can also use multiple configuration sections akin to MySQL's popular ``--defaults-group`` option in your ``pyxbackup.cnf``. 32 | 33 | Below are some valid options recognized from the configuration file: 34 | 35 | [pyxbackup] 36 | # MySQL credentials that can be used to the instance 37 | # being backed up, only user and pass are used at the 38 | # time of this writing 39 | mysql_host = 127.0.0.1 40 | mysql_user = msandbox 41 | mysql_pass = msandbox 42 | mysql_port = 56190 43 | mysql_sock = /tmp/mysql.sock 44 | 45 | # Instructs the script to run prepare on a copy of the backup 46 | # with redo-only. The script will maintain a copy of every 47 | # backup inside work_dir and keep applying with redo-only 48 | # until the next full backup is executed 49 | apply_log = 1 50 | 51 | # Whether to compress backups 52 | compress = 1 53 | # What compression tool, supports gzip and qpress 54 | compress_with = gzip 55 | 56 | # Send abckup failure notifications to these addresses, separated by comma 57 | notify_by_email = myemail@example.com 58 | # Send backup completion notifications to these adresses, 59 | # separated by comma 60 | notify_on_success = myemail@example.com 61 | 62 | # Where to stor raw (compressed) backups on the local directory 63 | # If --remote-push-only is specified, this is still needed but 64 | # they will not contain the actual backups, only meta information 65 | # and logs will remain to keep the backup workflow going 66 | stor_dir = /sbx/msb/msb_5_6_190/bkp/stor 67 | # When apply-log is enabled, this is where the "prepared-full" 68 | # backup will be kept and also stage as temp work dir if backups 69 | # compression is enabled 70 | work_dir = /sbx/msb/msb_5_6_190/bkp/work 71 | 72 | # When specified, this value will be passed as --defaults-file to 73 | # innobackupex 74 | mysql_cnf = /sbx/msb/msb_5_6_190/my.sandbox.cnf 75 | 76 | # When streaming/copying backups to remote site 77 | # this is the destination. It should have the same structure as 78 | # stor_dir with full, incr, weekly, monthly folders within 79 | remote_stor_dir = /sbx/msb/msb_5_6_190/bkp/stor_remote 80 | # Remote host to stream to 81 | remote_host = 127.0.0.1 82 | # Optional SSH options when streaming with rsync 83 | # "-o PasswordAuthentication=no -q" is already specified by default 84 | ssh_opts = "-i /home/revin/.ssh/id_rsa" 85 | # The SSH user to use when streaming to remote 86 | ssh_user = root 87 | 88 | # When apply_log is enabled, this is how much memory in MB 89 | # will be used for --use-memory option with innobackupex 90 | prepare_memory = 128 91 | 92 | # How many sets of full + incrementals to keep in stor 93 | retention_sets = 2 94 | # How many archived weekly backups are kept, unused for now 95 | retention_weeks = 0 96 | # How many archived monthly backups are kept, unused for now 97 | retention_months = 0 98 | # When using binary log streaming, by default, the script will maintain 99 | # the oldest binary log based on the oldest backup. This can be overridden 100 | # by setting a customer retention period for binary logs for special 101 | # cases 102 | retention_binlogs = 365 103 | 104 | # Same functions as innobackupex --encrypt --encrypt-key-file options 105 | # to support for encrypted backups at rest 106 | encrypt = AES256 107 | encrypt_key_file = /path/to/backups/key 108 | 109 | # innobackupex has a lot of options not covered by this wrapper 110 | # therefore to support additional options, you can pass additional 111 | # parameters to innobackupex using this option. Enclose them in single or 112 | # double quotes and specify them as you would when running innobackupex 113 | # manually. Take into account to not conflict with options like 114 | # --compress, --encrypt*, --remote* as these are used in extended 115 | # fashion by pyxbackup. 116 | # 117 | # Note that anything after the equal sign is included, quotes are not 118 | # stripped since innobackupex can have options that will require quotes 119 | # i.e. --include=REGEXP 120 | extra_ibx_options = --slave-info --galera-info 121 | 122 | # When using Percona Server with Changed Page Tracking enabled, the 123 | # script can also purge the bitmaps automatically provided that it is 124 | # configured with valid credentials with SUPER privileges 125 | purge_bitmaps = 1 126 | 127 | # By default, when storing backups to a remote storage, scp/ssh streaming is 128 | # used. If you want to use netcat, simply specify using this option the 129 | # netcat port to open on the remote server. The script will use SSH to connect 130 | # to the remote server and open the nc port, make sure that the SSH user 131 | # has the privilege to open the port i.e. try to use unprivileged port 132 | # instead. 133 | # 134 | # If you have multiple backups running at the same time and storing to the 135 | # same server, make sure to assign unique ports to each. 136 | remote_nc_port=9999 137 | 138 | # When pushing backups to remote Linux servers, you can specify 139 | # the path to the pyxbackup script on the remote server and 140 | # other config/options 141 | # file if they are not in default locations ($PATH and /etc/pyxbackup.cnf) 142 | remote_script=/usr/local/bin/pyxbackup --config=/path/to/custom/pyxbackup.cnf 143 | 144 | 145 | Minimum Configuration 146 | ===================== 147 | 148 | At the very least, you should have the ``stor_dir`` and ``work_dir`` directories created. Inside ``stor_dir``, the folders **full**, **incr**, **weekly** and **monthly** will be created if they do not exist yet. When running the backup, you should specify these options on the command line or via the configuration file above. 149 | 150 | If you are streaming files to remote server, you should also have, aside from the 2 directories previously mentioned, the ``remote_stor_dir`` precreated withe the full, incr, weekly and monthly folders created as well. 151 | 152 | Quick Install 153 | ============= 154 | 155 | First, create your local backup folders and install a single dependency: 156 | 157 | mkdir /backups/folder/stor 158 | mkdir /backups/folder/work 159 | yum install MySQL-python # apt-get install python-mysqldb 160 | wget https://raw.githubusercontent.com/dotmanila/pyxbackup/master/pyxbackup 161 | chmod 0755 pyxbackup 162 | 163 | Run you first backup! 164 | 165 | ./pyxbackup full 166 | 167 | See more `Configuration`_ options above. 168 | 169 | Compressed Backups 170 | ================== 171 | 172 | There are several types of compressed backups when the ``compress`` option is enabled and each can be decompressed manuall if needed in different ways: 173 | 174 | tar + gzip (*.tar.gz) 175 | --------------------- 176 | 177 | This backup is a result when ``compress`` is enabled with combined with ``apply_log`` and ``compress_with=gzip``. Decompressing is fairly straighforward using the tar utility: 178 | 179 | tar xzvf /path/to/backup.tar.gz -C /path/to/destination/folder 180 | 181 | 182 | Streamed + gzip (*.xbs.gz) 183 | -------------------------- 184 | 185 | Same as tar+gz but without the ``apply-log`` option, because we can stream the backup directly, we use xbstream format for potential optimizations like ``rsync`` for local copies and ``parallel`` options. 186 | 187 | gzip -cd /path/to/backup.xbs.gz | xbstream -x -C /path/to/destination/folder 188 | 189 | 190 | Non-Streamed qpress (*.qp) 191 | -------------------------- 192 | 193 | Similar to tar+gz, but using qpress as compression binary for when ``apply-log`` is enabled. 194 | 195 | qpress -d /path/to/backup.qp /path/to/destination/folder 196 | 197 | Streamed qpress (*.xbs.qp) 198 | -------------------------- 199 | 200 | When ``apply-log`` is not used, and ``compress_with=qpress``, this will be the format. It takes 2 steps to prepare the backup before being used. 201 | 202 | cat /path/to/backup.xbs.qp | xbstream -x -C /path/to/destination/folder 203 | 204 | innobackupex --decompress /path/to/destination/folder 205 | 206 | 207 | Encrypted Backups (*.qp.xbcrypt) 208 | -------------------------------- 209 | 210 | When ``apply-log`` is enabled with encryption, compression is implicitly set to qpress. To decompress and decrypt, you can use a command like below: 211 | 212 | xbcrypt --decrypt --encrypt-algo=ENCRYPT_ALGO \ 213 | --encrypt-key-file=/path/to/encryption/key \ 214 | --input=/path/to/backup.qp.xbcrypt \ 215 | | qpress -di /path/to/destination/folder 216 | 217 | 218 | Streamed Encrypted Backups (*.xbs.qp.xbcrypt) 219 | --------------------------------------------- 220 | 221 | Similar to the previous format, except this is streamed with xbstream i.e. ``apply-log`` is disabled or ``remote_push_only`` is enabled. 222 | 223 | xbcrypt --decrypt --encrypt-algo=ENCRYPT_ALGO \ 224 | --encrypt-key-file=/path/to/encryption/key \ 225 | --input=/path/to/backup.qp.xbcrypt \ 226 | | xbstream -x -C /path/to/destination/folder 227 | 228 | innobackupex --decompress /path/to/destination/folder 229 | 230 | 231 | Binary Log Streaming 232 | ==================== 233 | 234 | Streaming binary logs can be done with the script via the ``binlog-stream`` command. The advantage of doing it via the script and the same configuration file as your backups is that it can keep track of your backups and automatically prune binary logs. For example, when your oldest full backup was taken 2 weeks ago, then your oldest binary log file on archive will correspond to that backup as well. 235 | 236 | Binary log streaming requires that you configure the ``mysql_host``, ``mysql_user``, ``mysql_pass`` options or on the command line. Additionally aside from ``REPLICATION SLAVE`` privilege, you also need ``REPLICATION CLIENT`` as te script uses ``SHOW BINARY LOGS`` command using the MySQL account. 237 | 238 | A simple invocation would look like: 239 | 240 | pyxbackup binlog-stream 241 | 242 | In some cases, if you are backing up data from a slave but want to stream the binary logs from the master, the script needs to know this is what you want as the master and slave will have a different set of binary logs. For this, you can specify the option ``--binlog-from-master`` or set ``binlog_from_master=1`` on the configuration file. 243 | 244 | As mentioned above, binary log streaming relies on the availability of your oldest full backup. If you do not have this, or simply want to override, you can specify the ``--first-binlog`` option with the name of the binary log from the server you want to stream from. 245 | 246 | Additionally, if you want a custom retention period i.e. longer than your oldest backup, ``--retention-binlogs`` can help. This is specified in the number of days and can be as far back as you want. This feature relies on the ``timestamp`` header of each binary log when pruning older copies and not on filesystem metadata. 247 | 248 | Examples 249 | ======== 250 | 251 | Assuming I have a very minimal ``pyxbackup.cnf`` below: 252 | 253 | [pyxbackup] 254 | stor_dir = /sbx/msb/msb_5_6_190/bkp/stor 255 | work_dir = /sbx/msb/msb_5_6_190/bkp/work 256 | retention_sets = 2 257 | 258 | Running a Full Backup 259 | --------------------- 260 | 261 | Taking a full backup: 262 | 263 | pyxbackup full 264 | 265 | Running an Incremental Backup 266 | ----------------------------- 267 | 268 | Taking an incremental backup: 269 | 270 | pyxbackup incr 271 | 272 | Listing Existing Backups 273 | ------------------------ 274 | 275 | Listing existing backups - also will help identify incomplete/failed backups that may be consuming disk space: 276 | 277 | pyxbackup list 278 | 279 | Checking Status of Last Backup 280 | ------------------------------ 281 | 282 | Support for Zabbix/Nagios tests for monitoring: 283 | 284 | pyxbackup --status-format=[nagios|zabbix] status 285 | 286 | Keeping a Running "prepared-full" Backup 287 | ---------------------------------------- 288 | 289 | When enabled, a special folder inside the ``work_dir`` will be maintained. This is prefixed with **P_** and the timestamp will correspond to the last full backup that has been taken. When the full backup is taken, a ``--redo-only`` will be applied to it, any succeeding incrementals will be prepared to the same. When in need of a recent snapshot, this special folder can be a quick source. 290 | 291 | pyxbackup --apply-log full 292 | 293 | One Touch Prepare of Specific Backup 294 | ------------------------------------ 295 | 296 | For example, I have these 2 backup sets with 2 incrementals each: 297 | 298 | [revin@forge ~]$ pyxbackup list 299 | # Full backup: 2014_10_15-11_32_32, incrementals: ['2014_10_15-11_34_17', '2014_10_15-11_32_41'] 300 | # Full backup: 2014_10_15-11_32_04, incrementals: ['2014_10_15-11_32_23', '2014_10_15-11_32_14'] 301 | 302 | If I want to prepare the backup ``2014_10_15-11_32_41`` and make it ready for use, I will use the following command: 303 | 304 | pyxbackup --restore-backup=2014_10_15-11_32_41 \ 305 | --restore-dir=/sbx/msb/msb_5_6_190/bkp/tmp restore-set 306 | 307 | After this command, I will have a folder ``/sbx/msb/msb_5_6_190/bkp/tmp/P_2014_10_15-11_32_41`` ready for use i.e. to provision a slave or staging server. 308 | 309 | -------------------------------------------------------------------------------- /scripts/monitor-backup-age.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | EMAIL=email@example.com 4 | 5 | last_backup_dt=$(ssh mysql@10.200.0.7 \ 6 | '/backups/primary/mysql/xbackup/xbackup -q --meta-item=xb_last_backup meta') 7 | last_backup_dt=$(echo $last_backup_dt|sed 's/[-_]/ /g'\ 8 | |awk '{printf "%04d-%02d-%02d %02d:%02d:%02d", $1, $2, $3, $4, $5, $6}') 9 | last_backup_dt=$(date -d "$last_backup_dt" +%s) 10 | 11 | now_dt=$(date +%s) 12 | last_backup_was_n_seconds_ago=$(($now_dt-$last_backup_dt)) 13 | 14 | if [ $last_backup_was_n_seconds_ago -ge 39600 ]; then 15 | ( 16 | echo "Subject: MySQL backup from $(hostname) has problems!"; 17 | echo "The last backup was more than 6hrs ago!"; 18 | ) | mail -s "MySQL backup from $(hostname) has problems!" ${EMAIL} 19 | fi -------------------------------------------------------------------------------- /scripts/monitor-binlog-stream.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | PIDFILE=/tmp/pyxbackup-binlog-stream.pid 4 | EMAIL=email@example.com 5 | PID=0 6 | ERROR="" 7 | 8 | while true; do 9 | if [ ! -f $PIDFILE ]; then 10 | ERROR="PID file $PIDFILE does not exist!" 11 | break 12 | fi 13 | 14 | PID=$(cat $PIDFILE) 15 | if [ "$PID" -le "0" ]; then 16 | ERROR="PID file $PIDFILE has invalid value!" 17 | break 18 | fi 19 | 20 | PROC=$(ps ax|grep mysqlbinlog|grep $PID) 21 | PID_B=$(echo $PROC|awk '{print $1}') 22 | if [ "$PID" != "$PID_B" ]; then 23 | ERROR="PID file $PIDFILE value is different from mysqlbinlog process!" 24 | break 25 | fi 26 | 27 | DEFUNCT=$(echo $PROC|grep defunct) 28 | if [ "$?" -eq 0 ]; then 29 | ERROR="mysqlbinlog process is marked as defunct!" 30 | break 31 | fi 32 | 33 | break 34 | done 35 | 36 | if [ "$ERROR" != "" ]; then 37 | ( 38 | echo "Subject: MySQL binlog streaming from $(hostname) has problems!"; 39 | echo "The error returned was: $ERROR"; 40 | ) | mail -s "MySQL binlog streaming from $(hostname) has problems!" ${EMAIL} 41 | fi -------------------------------------------------------------------------------- /tests/all_test.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | import sys 4 | import pyxbackup as pxb 5 | import pytest 6 | 7 | def test__parse_port_param(): 8 | assert(pxb._parse_port_param('27017,27019')) == True 9 | assert(pxb.xb_opt_remote_nc_port_min) == 27017 10 | assert(pxb.xb_opt_remote_nc_port_max) == 27019 11 | assert(pxb._parse_port_param('27017, 27019')) == True 12 | assert(pxb._parse_port_param('abcde, 27019')) == False 13 | assert(pxb._parse_port_param('abcde, ')) == False 14 | assert(pxb._parse_port_param('9999, ')) == False 15 | assert(pxb._parse_port_param('9999 ')) == False 16 | assert(pxb._parse_port_param('9999')) == True 17 | assert(pxb.xb_opt_remote_nc_port_min) == 9999 18 | assert(pxb.xb_opt_remote_nc_port_max) == 9999 19 | 20 | def test__xb_version(): 21 | assert(pxb._xb_version(verstr = '2.2.13')) == [2, 2, 13] 22 | assert(pxb._xb_version(verstr = '2.2.13', tof = True)) == 2.2 -------------------------------------------------------------------------------- /tests/pyxbackup-binlog.py: -------------------------------------------------------------------------------- 1 | import sys, traceback, os, errno, signal 2 | import time, calendar, shutil, re, pwd 3 | from datetime import datetime, timedelta 4 | from struct import unpack 5 | 6 | xb_binlogs_list = [['b1', 1485564648], ['b2', 1485764648], ['b3', 1485903391], ['b4', 1486103391]] 7 | xb_opt_retention_binlogs = 3 8 | 9 | def date(unixtime, format = '%m/%d/%Y %H:%M:%S'): 10 | d = datetime.fromtimestamp(unixtime) 11 | return d.strftime(format) 12 | 13 | def _out(tag, *msgs): 14 | s = '' 15 | 16 | if not msgs: 17 | return 18 | 19 | for msg in msgs: 20 | s += str(msg) 21 | 22 | out = "[%s] %s: %s" % (date(time.time()), tag, s) 23 | 24 | print out 25 | 26 | def _say(*msgs): 27 | _out('INFO', *msgs) 28 | 29 | def _purge_binlogs_to(old_binlog): 30 | if xb_binlogs_list is None: return 31 | 32 | if xb_opt_retention_binlogs is None: 33 | for l in xb_binlogs_list: 34 | if l < old_binlog: 35 | _say("Deleting old binary log %s" % l) 36 | os.remove(os.path.join(xb_stor_binlogs, l)) 37 | else: 38 | x = int(time.time())-(xb_opt_retention_binlogs*24*60*60) 39 | prev = None 40 | prev_ts = None 41 | _say("Binlog retention start %s" % str(datetime.fromtimestamp(x).strftime('%Y-%m-%d %H:%M:%S'))) 42 | _say("Current timestamp %s" % str(datetime.fromtimestamp(time.time()).strftime('%Y-%m-%d %H:%M:%S'))) 43 | for l in xb_binlogs_list: 44 | ts = l[1] 45 | ts_out = str(datetime.fromtimestamp(l[1]).strftime('%Y-%m-%d %H:%M:%S')) 46 | _say("%s created at %s" % (l[0], ts_out)) 47 | 48 | if prev is not None: 49 | if ts < x: 50 | _say("Pruning %s" % prev) 51 | prev = l[0] 52 | # Current binlog creation ts is later than start of retention period 53 | # We keep from this binlog and keep the previous one as well 54 | else: 55 | _say("%s matches binary log retention period, stopping" % l[0]) 56 | break 57 | elif prev is None: 58 | prev = l[0] 59 | 60 | _purge_binlogs_to(None) 61 | 62 | -------------------------------------------------------------------------------- /tests/pyxbackup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | # pyxbackup - Robust Xtrabackup based MySQL Backups Manager 4 | # 5 | # @author Jervin Real 6 | 7 | import sys, traceback, os, errno, signal 8 | import time, calendar, shutil, re, pwd 9 | import smtplib, MySQLdb, base64 10 | from datetime import datetime, timedelta 11 | from ConfigParser import ConfigParser, NoOptionError 12 | from optparse import OptionParser 13 | from subprocess import Popen, PIPE, STDOUT, CalledProcessError 14 | from struct import unpack 15 | 16 | XB_BIN_NAME = 'pyxbackup' 17 | 18 | xb_opt_config = None 19 | xb_opt_config_section = None 20 | xb_opt_stor_dir = '' 21 | xb_opt_work_dir = '' 22 | xb_opt_mysql_user = None 23 | xb_opt_mysql_pass = None 24 | xb_opt_mysql_host = 'localhost' 25 | xb_opt_mysql_port = 3306 26 | xb_opt_mysql_sock = '/tmp/mysql.sock' 27 | xb_opt_mysql_cnf = None 28 | xb_opt_retention_binlogs = False 29 | xb_opt_compress = False 30 | xb_opt_compress_with = 'gzip' 31 | xb_opt_apply_log = False 32 | xb_opt_prepare_memory = 128 33 | xb_opt_retention_sets = 2 34 | xb_opt_retention_months = 0 35 | xb_opt_retention_weeks = 0 36 | xb_opt_debug = False 37 | xb_opt_quiet = False 38 | xb_opt_status_format = None 39 | xb_opt_command = 'status' 40 | xb_opt_restore_backup = None 41 | xb_opt_restore_dir = None 42 | xb_opt_remote_stor_dir = None 43 | xb_opt_remote_host = None 44 | xb_opt_remote_push_only = None 45 | xb_opt_remote_script = XB_BIN_NAME 46 | xb_opt_remote_nc_port = 0 47 | xb_opt_remote_nc_port_min = 0 48 | xb_opt_remote_nc_port_max = 0 49 | xb_opt_ssh_opts = '' 50 | xb_opt_ssh_user = None 51 | xb_opt_notify_by_email = None 52 | xb_opt_notify_on_success = None 53 | xb_opt_meta_item = None 54 | xb_opt_wipeout = False 55 | xb_opt_first_binlog = False 56 | xb_opt_binlog_binary = None 57 | xb_opt_binlog_from_master = False 58 | xb_opt_encrypt = False 59 | xb_opt_encrypt_key_file = None 60 | xb_opt_extra_ibx_options = None 61 | xb_opt_purge_bitmaps = None 62 | 63 | xb_hostname = None 64 | xb_user = None 65 | xb_stor_full = None 66 | xb_stor_incr = None 67 | xb_stor_weekly = None 68 | xb_stor_monthly = None 69 | xb_stor_binlogs = None 70 | 71 | xb_curdate = None 72 | xb_cfg = None 73 | xb_cwd = None 74 | xb_version = 0.4 75 | xb_ibx_opts = '' 76 | xb_ibx_bin = 'innobackupex' 77 | xb_zip_bin = 'gzip' 78 | xb_xbs_bin = 'xbstream' 79 | xb_this_backup = None 80 | xb_this_backup_remote = None 81 | xb_this_binlog = None 82 | xb_this_master_binlog = None 83 | xb_this_last_lsn = None 84 | xb_last_full = None 85 | xb_last_incr = None 86 | xb_full_list = None 87 | xb_incr_list = None 88 | xb_weekly_list = None 89 | xb_monthly_list = None 90 | xb_last_backup = None 91 | xb_last_backup_is = None 92 | xb_first_binlog = None 93 | xb_last_binlog = None 94 | xb_binlogs_list = None 95 | xb_binlog_name = None 96 | xb_exit_code = 0 97 | xb_prepared_backup = '' 98 | xb_backup_is_success = False 99 | xb_prepare_is_success = False 100 | xb_backup_in_progress = None 101 | xb_info_bkp_start = None 102 | xb_info_bkp_end = None 103 | xb_info_prep_start = None 104 | xb_info_prep_end = None 105 | xb_log_file = '' 106 | xb_log_fd = None 107 | xb_is_last_day_of_week = False 108 | xb_is_last_day_of_month = False 109 | xb_mysqldb = None 110 | xb_backup_summary = None 111 | 112 | XB_CMD_INCR = 'incr' 113 | XB_CMD_FULL = 'full' 114 | XB_CMD_LIST = 'list' 115 | XB_CMD_STAT = 'status' 116 | XB_CMD_PREP = 'restore-set' 117 | XB_CMD_APPL = 'apply-last' 118 | XB_CMD_PRUNE = 'prune' 119 | XB_CMD_META = 'meta' 120 | XB_CMD_BINLOGS = 'binlog-stream' 121 | XB_CMD_WIPE = 'wipeout' 122 | XB_TAG_FILE = 'xtrabackup_checkpoints' 123 | XB_CKP_FILE = 'xtrabackup_checkpoints' 124 | XB_LOG_FILE = 'xtrabackup_logfile' 125 | XB_LCK_FILE = '' 126 | XB_META_FILE = 'backup.meta' 127 | XB_BKP_LOG = 'innobackupex-backup.log' 128 | XB_APPLY_LOG = 'innobackupex-prepare.log' 129 | XB_LOG_NAME = XB_BIN_NAME + '.log' 130 | XB_SSH_TMPFILE = '/tmp/' + XB_BIN_NAME + '-ssh-result' 131 | XB_SIGTERM_CAUGHT = False 132 | XB_VERSION_MAJOR = 0 133 | XB_VERSION_MINOR = 0 134 | XB_VERSION_REV = 0 135 | XB_VERSION = None 136 | 137 | XB_EXIT_COMPRESS_FAIL = 1 138 | XB_EXIT_REMOTE_PUSH_FAIL = 2 139 | XB_EXIT_EXTRACT_FAIL = 4 140 | XB_EXIT_BITMAP_PURGE_FAIL = 8 141 | XB_EXIT_NO_FULL = 16 142 | XB_EXIT_DECRYPT_FAIL = 32 143 | XB_EXIT_APPLY_FAIL = 64 144 | XB_EXIT_INNOBACKUP_FAIL = 65 145 | XB_EXIT_BINLOG_STREAM_FAIL = 66 146 | XB_EXIT_REMOTE_CMD_FAIL = 96 147 | XB_EXIT_BY_DEATH = 128 148 | XB_EXIT_EXCEPTION = 255 149 | 150 | # What commands does not need a log or lock file 151 | cmd_no_log = [ 152 | XB_CMD_LIST, XB_CMD_STAT, XB_CMD_META, XB_CMD_BINLOGS, 153 | XB_CMD_PRUNE] 154 | cmd_no_lock = cmd_no_log 155 | cmd_backups = [XB_CMD_FULL, XB_CMD_INCR] 156 | 157 | def date(unixtime, format = '%m/%d/%Y %H:%M:%S'): 158 | d = datetime.fromtimestamp(unixtime) 159 | return d.strftime(format) 160 | 161 | def _xb_version(verstr = None, tof = False): 162 | global XB_VERSION_MAJOR 163 | global XB_VERSION_MINOR 164 | global XB_VERSION_REV 165 | global XB_VERSION 166 | global xb_ibx_bin 167 | 168 | if verstr is None: 169 | if XB_VERSION is not None: 170 | if tof: return float("%d.%d" % (XB_VERSION_MAJOR, XB_VERSION_MINOR)) 171 | else: return True 172 | 173 | p = Popen(["xtrabackup", "--version"], stdout=PIPE, stderr=PIPE) 174 | 175 | # weird, xtrabackup outputs version 176 | # string on STDERR instead of STDOUT 177 | out, err = p.communicate() 178 | ver = re.search('version ([\d\.]+)', err) 179 | major, minor, rev = ver.group(1).split('.') 180 | 181 | XB_VERSION_MAJOR = int(major) if major else 0 182 | XB_VERSION_MINOR = int(minor) if minor else 0 183 | XB_VERSION_REV = int(rev) if rev else 0 184 | XB_VERSION = "%d.%d.%d" % ( 185 | XB_VERSION_MAJOR, XB_VERSION_MINOR, XB_VERSION_REV) 186 | 187 | if XB_VERSION_MAJOR == 0: 188 | _error( 189 | "Invalid xtrabackup version or unable to determine valid " 190 | "version string or binary version non-GA release") 191 | _error("Version string was \"%s\"" % err) 192 | _die("Exiting") 193 | 194 | if XB_VERSION_MINOR >= 3: xb_ibx_bin = 'xtrabackup' 195 | 196 | _debug("Found xtrabackup version %d.%d.%d" % ( 197 | XB_VERSION_MAJOR, XB_VERSION_MINOR, XB_VERSION_REV)) 198 | else: 199 | major, minor, rev = verstr.split('.') 200 | major = int(major) if major else 0 201 | minor = int(minor) if minor else 0 202 | rev = int(rev) if rev else 0 203 | 204 | if tof: 205 | return float("%d.%d" % (major, minor)) 206 | else: return [major, minor, rev] 207 | 208 | return True 209 | 210 | def _out(tag, *msgs): 211 | s = '' 212 | 213 | if not msgs: 214 | return 215 | 216 | for msg in msgs: 217 | s += str(msg) 218 | 219 | out = "[%s] %s: %s" % (date(time.time()), tag, s) 220 | 221 | if xb_log_fd is not None: 222 | os.write(xb_log_fd, "%s\n" % out) 223 | 224 | if not xb_opt_quiet: print out 225 | 226 | def _say(*msgs): 227 | _out('INFO', *msgs) 228 | 229 | def _warn(*msgs): 230 | _out('WARN', *msgs) 231 | 232 | def _error(*msgs): 233 | _out('ERROR', *msgs) 234 | 235 | def _die(*msgs): 236 | _out('FATAL', *msgs) 237 | if not xb_exit_code: _exit_code(XB_EXIT_BY_DEATH) 238 | raise Exception(str(msgs)) 239 | 240 | def _debug(*msgs): 241 | if xb_opt_debug: _out("** DEBUG **", *msgs) 242 | 243 | def _which(file): 244 | for path in os.environ["PATH"].split(os.pathsep): 245 | if os.path.exists(path + os.path.sep + file): 246 | return path + os.path.sep + file 247 | 248 | return None 249 | 250 | def _parse_port_param(param): 251 | """ 252 | Parses and assign given port range values 253 | i.e. 254 | remote_nc_port = 9999 255 | remote_nc_port = 9999,1000 256 | """ 257 | 258 | global xb_opt_remote_nc_port_min 259 | global xb_opt_remote_nc_port_max 260 | 261 | if not param: return False 262 | if param.isdigit(): 263 | xb_opt_remote_nc_port_min = int(param) 264 | xb_opt_remote_nc_port_max = xb_opt_remote_nc_port_min 265 | return True 266 | elif param.count(',') == 1: 267 | pmin, pmax = param.split(',') 268 | pmin = pmin.strip() 269 | pmax = pmax.strip() 270 | if not pmin.isdigit() or not pmax.isdigit(): return False 271 | xb_opt_remote_nc_port_min = int(pmin) 272 | xb_opt_remote_nc_port_max = int(pmax) 273 | if xb_opt_remote_nc_port_min > xb_opt_remote_nc_port_max: 274 | pmin = xb_opt_remote_nc_port_max 275 | xb_opt_remote_nc_port_max = xb_opt_remote_nc_port_min 276 | xb_opt_remote_nc_port_min = pmin 277 | return True 278 | 279 | return False 280 | 281 | def _read_magic_chunk(bfile, size): 282 | """ 283 | This is a more reliable way of reading some files format 284 | 285 | XBCRYP for xbcrypt files 286 | XBSTCK for xbstream files 287 | """ 288 | 289 | if not os.path.isfile(bfile): 290 | return None 291 | 292 | return open(bfile, 'rb').read(size) 293 | 294 | def _check_binary(name): 295 | bin = _which(name) 296 | if bin is None: 297 | _die("%s script is not found in $PATH" % name) 298 | 299 | return bin 300 | 301 | def _exit_code(code): 302 | global xb_exit_code 303 | 304 | c = int(code) 305 | if c > xb_exit_code: xb_exit_code = c 306 | 307 | def _destroy_lock_file(): 308 | if (xb_opt_command == XB_CMD_FULL or xb_opt_command == XB_CMD_INCR) \ 309 | and os.path.isfile(XB_LCK_FILE): 310 | if xb_backup_in_progress is None: 311 | os.remove(XB_LCK_FILE) 312 | 313 | def _create_lock_file(): 314 | if (xb_opt_command == XB_CMD_FULL or xb_opt_command == XB_CMD_INCR): 315 | lck = open(XB_LCK_FILE, 'w') 316 | lck.write("backup = %s\n" % xb_curdate) 317 | lck.write("type = %s\n" % xb_opt_command) 318 | 319 | if xb_opt_command == XB_CMD_INCR: 320 | lck.write("full = %s\n" % xb_last_full) 321 | 322 | lck.write("pid = %s\n" % str(os.getpid())) 323 | 324 | lck.close() 325 | 326 | def _xb_logfile_copy(bkp): 327 | # When backup is not compressed we need to preserve the 328 | # xtrabackup_logfile since preparing directly from the 329 | # stor_dir will touch the logfile and we cannot use it 330 | # again 331 | # We do this to make the process faster instead of copying 332 | # the whole incremental backup 333 | _say("Preserving %s from %s" % (XB_LOG_FILE, bkp)) 334 | xb_from = "%s/%s" % (bkp, XB_LOG_FILE) 335 | xb_to = "%s/%s.101" % (bkp, XB_LOG_FILE) 336 | shutil.copy(xb_from, xb_to) 337 | 338 | def _xb_logfile_restore(bkp): 339 | _say("Restoring %s from %s" % (XB_LOG_FILE, bkp)) 340 | xb_from = "%s/%s.101" % (bkp, XB_LOG_FILE) 341 | xb_to = "%s/%s" % (bkp, XB_LOG_FILE) 342 | if os.path.isfile(xb_to): os.remove(xb_to) 343 | shutil.move(xb_from, xb_to) 344 | 345 | def _sigterm_handler(signal, frame): 346 | global XB_SIGTERM_CAUGHT 347 | 348 | _say("Got TERM signal, cleaning up ...") 349 | XB_SIGTERM_CAUGHT = True 350 | 351 | def _check_in_progress(): 352 | global xb_backup_in_progress 353 | 354 | ret = False 355 | is_backup = False 356 | 357 | if xb_opt_command in [XB_CMD_FULL, XB_CMD_INCR]: 358 | is_backup = True 359 | 360 | if os.path.isfile(XB_LCK_FILE): 361 | _debug("%s lock file exists and is_backup is %s" % (XB_LCK_FILE, str(is_backup))) 362 | 363 | cfp = _parse_raw_config(XB_LCK_FILE) 364 | pid = int(cfp.get(XB_BIN_NAME, 'pid')) 365 | xb_backup_in_progress = cfp 366 | ret = True 367 | 368 | if is_backup: 369 | try: 370 | os.kill(pid, 0) 371 | except OSError, e: 372 | if e.errno == errno.ESRCH: 373 | _die("%s lock file exists but process is not running" % XB_LCK_FILE) 374 | elif e.errno == errno.EPERM: 375 | _die('Permission denied while checking backup process') 376 | else: 377 | _warn('Could not determine backup process state') 378 | else: 379 | _die("Another backup process in progress with PID %d" % pid) 380 | 381 | return ret 382 | 383 | def _write_backup_info(): 384 | global xb_backup_summary 385 | 386 | if (xb_opt_command == XB_CMD_FULL or xb_opt_command == XB_CMD_INCR): 387 | inf = open("%s/%s" % (xb_this_backup, XB_META_FILE), 'w') 388 | inf.write("backup = %s\n" % xb_curdate) 389 | inf.write("type = %s\n" % xb_opt_command) 390 | 391 | if xb_opt_command == XB_CMD_INCR: 392 | inf.write("full = %s\n" % xb_last_full) 393 | 394 | inf.write("start_backup = %s\n" % xb_curdate) 395 | inf.write("end_backup = %s\n" % xb_info_bkp_end) 396 | inf.write("start_prepare = %s\n" % xb_info_prep_start) 397 | inf.write("end_prepare = %s\n" % xb_info_prep_end) 398 | inf.write("compress = %d\n" % int(xb_opt_compress)) 399 | inf.write("compress_with = %s\n" % xb_opt_compress_with) 400 | inf.write("log_bin = %s\n" % xb_this_binlog) 401 | inf.write("master_log_bin = %s\n" % xb_this_master_binlog) 402 | inf.write("last_lsn = %s\n" % xb_this_last_lsn) 403 | inf.write("source_version = %d.%d.%d\n" % ( 404 | XB_VERSION_MAJOR, XB_VERSION_MINOR, XB_VERSION_REV)) 405 | 406 | inf.close() 407 | 408 | if xb_opt_notify_on_success: 409 | xb_backup_summary = "Backup summary: \n\n" 410 | xb_backup_summary += "Backup: %s\n" % xb_curdate 411 | xb_backup_summary += "Type: %s\n" % xb_opt_command 412 | 413 | if xb_opt_command == XB_CMD_INCR: 414 | xb_backup_summary += "Full: %s\n" % xb_last_full 415 | 416 | xb_backup_summary += "Backup started: %s\n" % xb_curdate 417 | xb_backup_summary += "Backup ended: %s\n" % xb_info_bkp_end 418 | xb_backup_summary += "Prepare started: %s\n" % xb_info_prep_start 419 | xb_backup_summary += "Prepare ended: %s\n" % xb_info_prep_end 420 | xb_backup_summary += "Compressed: %s\n" % bool(xb_opt_compress) 421 | xb_backup_summary += "Compressed with: %s\n" % xb_opt_compress_with 422 | xb_backup_summary += "Binary log name: %s\n" % xb_this_binlog 423 | xb_backup_summary += "Master binary log name: %s\n" % xb_this_master_binlog 424 | 425 | def _parse_raw_config(ckpnt_f): 426 | if not os.path.isfile(ckpnt_f): 427 | _warn("Config file not found, ", ckpnt_f, "!") 428 | return False 429 | 430 | with open(ckpnt_f) as ckp: 431 | defaults = dict([line.replace(' ','').rstrip("\n").split('=') for line in ckp]) 432 | 433 | cfp = ConfigParser(defaults) 434 | cfp.add_section(XB_BIN_NAME) 435 | 436 | return cfp 437 | 438 | def _read_backup_metadata(bkp): 439 | meta_path = os.path.join(bkp, XB_META_FILE) 440 | 441 | # For backwards compatibility 442 | if not os.path.isfile(meta_path): 443 | meta_path = os.path.join(bkp, 'xbackup.meta') 444 | 445 | meta = _parse_raw_config(meta_path) 446 | 447 | if not meta: 448 | _die("Unable to read backup meta information, ", 449 | "%s corrupt?" % meta_path) 450 | 451 | return meta 452 | 453 | def _apply_log(bkp, incrdir=None, final=False): 454 | if not os.path.isdir(bkp): 455 | _warn("Directory not found, ", bkp, " will not prepare") 456 | return False 457 | 458 | cfp = _parse_raw_config("%s/xtrabackup_checkpoints" % bkp) 459 | if not cfp: 460 | _die('Could not parse xtrabackup_checkpoints file') 461 | 462 | ibx_cmd = '' 463 | ibx_log = "%s/%s-innobackupex-prepare.log" % (xb_opt_work_dir, xb_curdate) 464 | tee_cmd = "tee %s" % ibx_log 465 | 466 | ibx_opts = "" 467 | if XB_VERSION_MINOR >= 3: 468 | ibx_opts = '--prepare ' 469 | else: ibx_opts = '--apply-log ' 470 | 471 | ibx_opts += "--use-memory=%dM" % xb_opt_prepare_memory 472 | log_fd = None 473 | p_tee = None 474 | 475 | if not final: 476 | if XB_VERSION_MINOR >= 3: ibx_opts += " --apply-log-only" 477 | else: ibx_opts += " --redo-only" 478 | 479 | if cfp.get(XB_BIN_NAME,'backup_type') == 'incremental': 480 | _say('Preparing incremental backup: ', bkp) 481 | if XB_VERSION_MINOR >= 3: 482 | ibx_opts += " --incremental-dir %s --target-dir %s" % (bkp, incrdir) 483 | else: ibx_opts += " --incremental-dir %s %s" % (bkp, incrdir) 484 | else: 485 | _say('Preparing full backup: ', bkp) 486 | if XB_VERSION_MINOR >= 3: ibx_opts += " --target-dir %s" % bkp 487 | else: ibx_opts += " %s" % bkp 488 | 489 | ibx_cmd = "%s %s" % (xb_ibx_bin, ibx_opts) 490 | _say("Running prepare command: ", ibx_cmd) 491 | 492 | try: 493 | if not xb_opt_debug: 494 | log_fd = os.open(ibx_log, os.O_WRONLY|os.O_CREAT) 495 | p_ibx = Popen(ibx_cmd, shell=True, stdout=PIPE, stderr=log_fd) 496 | else: 497 | p_ibx = Popen(ibx_cmd, shell=True, stdout=PIPE, stderr=PIPE) 498 | p_tee = Popen(tee_cmd, shell=True, stdin=p_ibx.stderr) 499 | 500 | r = p_ibx.poll() 501 | while r is None: 502 | time.sleep(2) 503 | r = p_ibx.poll() 504 | 505 | if p_tee is not None: p_tee.wait() 506 | 507 | if log_fd is not None: 508 | os.close(log_fd) 509 | 510 | if r != 0: raise Exception("Non-zero exit of innobackupex command!") 511 | 512 | if cfp.get(XB_BIN_NAME,'backup_type') == 'incremental': 513 | shutil.move(ibx_log, 514 | "%s/%s-innobackupex-prepare.log" % (incrdir, xb_curdate)) 515 | else: 516 | shutil.move(ibx_log, 517 | "%s/%s-innobackupex-prepare.log" % (bkp, xb_curdate)) 518 | 519 | return True 520 | 521 | except Exception, e: 522 | _error("Command was: ", ibx_cmd) 523 | _error("Error: process exited with status %s" % str(e)) 524 | _error("Please check innobackupex log file at %s" % ibx_log) 525 | _exit_code(XB_EXIT_APPLY_FAIL) 526 | 527 | return False 528 | 529 | def _prepare_backup(bkp, prep, final=False): 530 | prepare_success = False 531 | meta = _read_backup_metadata(bkp) 532 | 533 | if not meta: 534 | _die("Unable to read backup meta information, ", 535 | "%s corrupt?" % meta_f) 536 | 537 | is_cmp = bool(int(meta.get(XB_BIN_NAME, 'compress'))) 538 | is_of_type = meta.get(XB_BIN_NAME, 'type') 539 | this_bkp = meta.get(XB_BIN_NAME, 'backup') 540 | 541 | # If the backup is compressed, we extract to the prepare path 542 | if is_cmp: 543 | prep_tmp = os.path.join(os.path.dirname(prep), this_bkp) 544 | if is_of_type == XB_CMD_FULL: 545 | if not os.path.isdir(prep): os.mkdir(prep, 0755) 546 | cmp_to = prep 547 | else: 548 | if not os.path.isdir(prep_tmp): os.mkdir(prep_tmp, 0755) 549 | cmp_to = prep_tmp 550 | 551 | for fmt in ['xbs.gz', 'tar.gz', 'xbs.qp', 'xbs.qp.xbcrypt', 'qp', 'qp.xbcrypt']: 552 | bkp_file = "%s/backup.%s" % (bkp, fmt) 553 | if os.path.isfile(bkp_file): 554 | break 555 | 556 | _say("Decompressing %s" % bkp_file) 557 | if not _decompress(bkp_file, cmp_to, meta): 558 | _die("An error occurred while extracting %s to %s" % (bkp_file, cmp_to)) 559 | 560 | if is_of_type == XB_CMD_FULL: 561 | _say("Applying log on %s" % prep) 562 | prepare_success = _apply_log(prep, prep, final) 563 | else: 564 | _say("Applying log on %s with %s" % (prep, prep_tmp)) 565 | prepare_success = _apply_log(prep_tmp, prep, final) 566 | shutil.rmtree(prep_tmp) 567 | else: 568 | if is_of_type == XB_CMD_FULL: 569 | _say("Copying %s to %s" % (bkp, prep)) 570 | shutil.copytree(bkp, prep) 571 | _say("Applying log to %s" % prep) 572 | prepare_success = _apply_log(prep, prep, final) 573 | else: 574 | _xb_logfile_copy(bkp) 575 | _say("Applying log on %s with %s" % (prep, bkp)) 576 | prepare_success = _apply_log(bkp, prep, final) 577 | _xb_logfile_restore(bkp) 578 | 579 | return prepare_success 580 | 581 | def _compress(bkp, archive): 582 | global xb_exit_code 583 | 584 | if not os.path.isdir(bkp): 585 | _warn("Directory not found, ", bkp, " cannot compress") 586 | return False 587 | 588 | if xb_opt_compress_with == 'gzip': 589 | return _compress_tgz(bkp, archive) 590 | elif xb_opt_compress_with == 'qpress': 591 | return _compress_qp(bkp, archive) 592 | 593 | def _compress_qp(bkp, xbs): 594 | global xb_exit_code 595 | 596 | cwd = os.getcwd() 597 | os.chdir(bkp) 598 | 599 | # *.tar.gz tar+gzip compress either via innobackupex --stream=tar or 600 | # tar czvf . - 601 | # *.qp cmopressed with qpress i.e. qpress -rvT4 . 602 | # *.xbs.qp for streamed qpress i.e. innobackupex --stream --compress 603 | # *.xbs.qp.xbcrypt for streamed qpress, encrypted 604 | # i.e. innobackupex --stream --compress --encrypt 605 | 606 | xbc_cmd = None 607 | qp = None 608 | xbc = None 609 | FNULL = None 610 | 611 | if xb_opt_debug: 612 | qp_cmd = 'qpress -rvT4' 613 | else: 614 | qp_cmd = 'qpress -rT4' 615 | 616 | if xb_opt_encrypt: 617 | qp_cmd += 'o .' 618 | xbc_cmd = 'xbcrypt --encrypt-algo=%s --encrypt-key-file=%s --output=%s.qp.xbcrypt' % ( 619 | xb_opt_encrypt, xb_opt_encrypt_key_file, xbs) 620 | _debug("Encrypting with command: %s" % xbc_cmd) 621 | else: 622 | qp_cmd += ' . %s.qp' % xbs 623 | 624 | _debug("Compressing with command: %s" % qp_cmd) 625 | 626 | if not xb_opt_debug: 627 | FNULL = open(os.devnull, 'w') 628 | if xb_opt_encrypt: 629 | qp = Popen(qp_cmd, shell=True, stdout=PIPE, stderr=FNULL) 630 | xbc = Popen(xbc_cmd, shell=True, stdin=qp.stdout, stdout=FNULL, stderr=FNULL) 631 | else: 632 | qp = Popen(qp_cmd, shell=True, stdout=FNULL, stderr=STDOUT) 633 | else: 634 | if xb_opt_encrypt: 635 | qp = Popen(qp_cmd, shell=True, stdout=PIPE) 636 | xbc = Popen(xbc_cmd, shell=True, stdin=qp.stdout) 637 | else: 638 | qp = Popen(qp_cmd, shell=True) 639 | 640 | r = qp.poll() 641 | while r is None: 642 | time.sleep(5) 643 | r = qp.poll() 644 | 645 | if xbc is not None: 646 | x = xbc.poll() 647 | if x is None: xbc.wait() 648 | x = xbc.poll() 649 | 650 | if FNULL is not None: 651 | FNULL.close() 652 | 653 | if r != 0: 654 | _error("Compressing ", bkp, " to ", xbs, " failed.") 655 | _error("qpress command was: ", qp_cmd) 656 | _error("qpress returned exit code was: ", str(r)) 657 | 658 | if xb_opt_encrypt: 659 | _error("xbcrypt command was: ", xbc_cmd) 660 | _error("xbcrypt returned exit code was: ", str(x)) 661 | 662 | _exit_code(XB_EXIT_COMPRESS_FAIL) 663 | return False 664 | 665 | os.chdir(cwd) 666 | 667 | return True 668 | 669 | def _compress_tgz(bkp, tgz): 670 | global xb_exit_code 671 | 672 | tgz = "%s.tar.gz" % tgz 673 | 674 | if os.path.isfile(tgz): 675 | _warn("Destination archive already exists, ", tgz, " aborting compression") 676 | return False 677 | 678 | cwd = os.getcwd() 679 | os.chdir(bkp) 680 | 681 | run_cmd = "tar c" 682 | run_cmd += 'z' 683 | if xb_opt_debug: 684 | run_cmd += 'v' 685 | 686 | run_cmd += "f %s %s" % (tgz, './') 687 | FNULL = None 688 | 689 | _debug("Running compress command: %s" % run_cmd) 690 | 691 | if not xb_opt_debug: 692 | FNULL = open(os.devnull, 'w') 693 | p1 = Popen(run_cmd, shell=True, stdout=FNULL, stderr=STDOUT) 694 | else: 695 | p1 = Popen(run_cmd, shell=True) 696 | 697 | r = p1.poll() 698 | while r is None: 699 | time.sleep(5) 700 | r = p1.poll() 701 | 702 | if FNULL is not None: 703 | FNULL.close() 704 | 705 | os.chdir(cwd) 706 | 707 | if r != 0: 708 | _error("Compressing ", bkp, " to ", tgz, " failed.") 709 | _error("tar command was: ", run_cmd) 710 | _error("tar returned exit code was: ", str(r)) 711 | _exit_code(XB_EXIT_COMPRESS_FAIL) 712 | return False 713 | 714 | return True 715 | 716 | def _extract_tgz(tgz, dest): 717 | run_cmd = "tar xi" 718 | if xb_opt_compress_with == 'gzip': 719 | run_cmd += 'z' 720 | if xb_opt_debug: 721 | run_cmd += 'v' 722 | 723 | run_cmd += "f %s -C %s" % (tgz, dest) 724 | FNULL = None 725 | 726 | if not xb_opt_debug: 727 | FNULL = open(os.devnull, 'w') 728 | p1 = Popen(run_cmd, shell=True, stdout=FNULL, stderr=STDOUT) 729 | else: 730 | p1 = Popen(run_cmd, shell=True) 731 | 732 | r = p1.poll() 733 | while r is None: 734 | time.sleep(5) 735 | r = p1.poll() 736 | 737 | if FNULL is not None: 738 | FNULL.close() 739 | 740 | if r != 0: 741 | _error("Extracting ", tgz, " to ", dest, " failed.") 742 | _error("tar command was: ", run_cmd) 743 | _error("tar returned exit code was: ", str(r)) 744 | _exit_code(XB_EXIT_EXTRACT_FAIL) 745 | return False 746 | 747 | return True 748 | 749 | def _extract_xgz(xgz, dest): 750 | gz_cmd = "gzip -cd" 751 | #if xb_opt_debug: 752 | # gz_cmd += ' -v' 753 | 754 | gz_cmd += " %s" % xgz 755 | FNULL = None 756 | 757 | xbs_cmd = "xbstream -x -C %s" % dest 758 | 759 | _debug("Running gzip command: %s" % gz_cmd) 760 | _debug("Running xbstream command: %s" % xbs_cmd) 761 | 762 | if not os.path.isdir(dest): os.mkdir(dest, 0755) 763 | 764 | if not xb_opt_debug: 765 | FNULL = open(os.devnull, 'w') 766 | gz = Popen(gz_cmd, shell=True, stdout=PIPE, stderr=FNULL) 767 | xbs = Popen(xbs_cmd, shell=True, stderr=FNULL, stdin=gz.stdout) 768 | else: 769 | gz = Popen(gz_cmd, shell=True, stdout=PIPE) 770 | xbs = Popen(xbs_cmd, shell=True, stdin=gz.stdout) 771 | 772 | r = gz.poll() 773 | while r is None: 774 | time.sleep(5) 775 | r = gz.poll() 776 | 777 | x = xbs.poll() 778 | if x is None: xbs.wait() 779 | x = xbs.poll() 780 | 781 | if FNULL is not None: 782 | FNULL.close() 783 | 784 | if r != 0: 785 | _error("Extracting ", xgz, " to ", dest, " failed.") 786 | _error("Extract command was: %s | %s" % (gz_cmd, xbs_cmd)) 787 | _error("Extract returned exit codes were: %s and %s" % (str(r), str(x))) 788 | _exit_code(XB_EXIT_EXTRACT_FAIL) 789 | return False 790 | 791 | return True 792 | 793 | def _extract_xbs(xbs, dest, meta = None): 794 | xbs_cmd = "xbstream -x -C %s" % dest 795 | xbc_cmd = 'cat %s' % xbs 796 | 797 | if not os.path.isdir(dest): os.mkdir(dest, 0755) 798 | 799 | _say("Extracting from xbstream format: %s" % xbs) 800 | FNULL = None 801 | 802 | if not xb_opt_debug: 803 | FNULL = open(os.devnull, 'w') 804 | xbc = Popen(xbc_cmd, shell=True, stdout=PIPE, stderr=FNULL) 805 | xbs = Popen(xbs_cmd, shell=True, stderr=FNULL, stdin=xbc.stdout) 806 | else: 807 | xbc = Popen(xbc_cmd, shell=True, stdout=PIPE) 808 | xbs = Popen(xbs_cmd, shell=True, stdin=xbc.stdout) 809 | 810 | r = xbc.poll() 811 | while r is None: 812 | time.sleep(5) 813 | r = xbc.poll() 814 | 815 | x = xbs.poll() 816 | if x is None: xbs.wait() 817 | x = xbs.poll() 818 | 819 | if FNULL is not None: 820 | FNULL.close() 821 | 822 | if r != 0: 823 | _error("Extracting ", xbs, " to ", dest, " failed.") 824 | _error("Extract command was: %s | %s" % (xbc_cmd, xbs_cmd)) 825 | _error("Extract returned exit codes were: %s and %s" % (str(r), str(x))) 826 | _exit_code(XB_EXIT_EXTRACT_FAIL) 827 | _die("Decompress of xbstream file %s failed." % xbs) 828 | 829 | return True 830 | 831 | def _extract_xbcrypt(dest, meta = None): 832 | """ Decrypt a backup set encrypted with xbcrypt 833 | if xrabackup version is < 2.3 we use xtrabackup --decrypt 834 | via _extract_xbcrypt_file which decrypts the files one 835 | at a timedelta 836 | """ 837 | 838 | if _xb_version(tof = True) < 2.3: 839 | _say( 840 | "You are running an older xtrabackup version " 841 | "that do not have --decrypt support, " 842 | "switching manual decompresssion") 843 | return _extract_xbcrypt_file(dest) 844 | 845 | # Now we decompress *.xbcrypt files 846 | ibx_cmd = '%s --decrypt=%s --encrypt-key-file=%s --target-dir=%s' % ( 847 | xb_ibx_bin, xb_opt_encrypt, xb_opt_encrypt_key_file, dest) 848 | 849 | FNULL = None 850 | 851 | if not xb_opt_debug: 852 | FNULL = open(os.devnull, 'w') 853 | ibx = Popen(ibx_cmd, shell=True, stdout=FNULL, stderr=FNULL) 854 | else: 855 | ibx = Popen(ibx_cmd, shell=True) 856 | 857 | r = ibx.poll() 858 | while r is None: 859 | time.sleep(5) 860 | r = ibx.poll() 861 | 862 | if FNULL is not None: 863 | FNULL.close() 864 | 865 | if r != 0: 866 | _error("Decrypt of backup failed.") 867 | _error("Decrypt command was: %s" % ibx_cmd) 868 | _error("Decrypt returned exit code was: %s" % str(r)) 869 | _exit_code(XB_EXIT_DECRYPT_FAIL) 870 | _die("Decrypt of backup %s failed." % dest) 871 | 872 | _cleanup_files_by_ext(dest, 'xbcrypt') 873 | 874 | return True 875 | 876 | def _extract_xbcrypt_file(cfile): 877 | """ cfile is encrypted file or folder, this method 878 | traverses individual xbcrypt files if --decrypt option is 879 | not available 880 | """ 881 | 882 | if os.path.isdir(cfile): 883 | _debug("Decrypting from directory: %s" % cfile) 884 | ls = os.listdir(cfile) 885 | for f in ls: 886 | _extract_xbcrypt_file(os.path.join(cfile, f)) 887 | 888 | elif os.path.isfile(cfile) and '.xbcrypt' == cfile[-8:]: 889 | xbc_cmd = 'xbcrypt --decrypt --encrypt-algo=%s ' % xb_opt_encrypt 890 | xbc_cmd += '--encrypt-key-file=%s --input=%s --output=%s' % ( 891 | xb_opt_encrypt_key_file, cfile, cfile[:-8]) 892 | 893 | FNULL = None 894 | 895 | _debug("Decrypting from xbcrypt file: %s" % cfile) 896 | 897 | if not xb_opt_debug: 898 | FNULL = open(os.devnull, 'w') 899 | xbc = Popen(xbc_cmd, shell=True, stdout=FNULL, stderr=FNULL) 900 | else: 901 | xbc = Popen(xbc_cmd, shell=True) 902 | 903 | r = xbc.poll() 904 | t = 0.1 905 | while r is None: 906 | time.sleep(t) 907 | # Increase time per loop 908 | if t <= 5: t = t*1.5 909 | r = xbc.poll() 910 | 911 | if FNULL is not None: 912 | FNULL.close() 913 | 914 | if r != 0: 915 | _error("Extracting ", cfile, " failed.") 916 | _error("Extract command was: %s" % xbc_cmd) 917 | _error("Extract returned exit codes were: %s" % str(r)) 918 | _exit_code(XB_EXIT_DECRYPT_FAIL) 919 | _die("Decrypt of file %s failed." % cfile) 920 | 921 | os.remove(cfile) 922 | 923 | else: 924 | _warn("File %s is not a valid xxbcrypt file." % cfile) 925 | 926 | return True 927 | 928 | def _extract_stream_qpress(xbs, dest, meta = None): 929 | xbc_cmd = None 930 | is_encrypted = False 931 | 932 | xbs_cmd = "xbstream -x -C %s" % dest 933 | 934 | if xbs[-14:] == 'xbs.qp.xbcrypt': 935 | if not xb_opt_encrypt: 936 | _die("Backup is %s is encrypted, no encryption options provided" % qp) 937 | else: 938 | xbc_cmd = 'xbcrypt --decrypt --encrypt-algo=%s ' % xb_opt_encrypt 939 | xbc_cmd += '--encrypt-key-file=%s --input=%s' % (xb_opt_encrypt_key_file, xbs) 940 | 941 | is_encrypted = True 942 | 943 | """ xtrabackup 2.3+ made a change with streamed encrypted 944 | backups 945 | """ 946 | if _read_magic_chunk(xbs, 6) == 'XBSTCK': 947 | _extract_xbs(xbs, dest, meta) 948 | _extract_xbcrypt(dest, meta) 949 | return _extract_ibx_decompress(dest, meta) 950 | else: 951 | xbc_cmd = 'cat %s' % xbs 952 | 953 | FNULL = None 954 | 955 | _debug("Running xbcrypt command: %s" % xbc_cmd) 956 | _debug("Running xbstream command: %s" % xbs_cmd) 957 | 958 | if not os.path.isdir(dest): os.mkdir(dest, 0755) 959 | 960 | if not xb_opt_debug: 961 | FNULL = open(os.devnull, 'w') 962 | xbc = Popen(xbc_cmd, shell=True, stdout=PIPE, stderr=FNULL) 963 | xbs = Popen(xbs_cmd, shell=True, stderr=FNULL, stdin=xbc.stdout) 964 | else: 965 | xbc = Popen(xbc_cmd, shell=True, stdout=PIPE) 966 | xbs = Popen(xbs_cmd, shell=True, stdin=xbc.stdout) 967 | 968 | r = xbc.poll() 969 | while r is None: 970 | time.sleep(5) 971 | r = xbc.poll() 972 | 973 | x = xbs.poll() 974 | if x is None: xbs.wait() 975 | x = xbs.poll() 976 | 977 | if FNULL is not None: 978 | FNULL.close() 979 | 980 | if r != 0: 981 | _error("Extracting ", xbs, " to ", dest, " failed.") 982 | _error("Extract command was: %s | %s" % (xbc_cmd, xbs_cmd)) 983 | _error("Extract returned exit codes were: %s and %s" % (str(r), str(x))) 984 | _exit_code(XB_EXIT_EXTRACT_FAIL) 985 | return False 986 | 987 | return _extract_ibx_decompress(dest) 988 | 989 | def _extract_nostream_qpress(qp, dest, meta = None): 990 | xbc_cmd = None 991 | is_encrypted = False 992 | 993 | if qp[-10:] == 'qp.xbcrypt': 994 | if not xb_opt_encrypt: 995 | _die("Backup is %s is encrypted, no encryption options provided" % qp) 996 | else: 997 | xbc_cmd = 'xbcrypt --decrypt --encrypt-algo=%s ' % xb_opt_encrypt 998 | xbc_cmd += '--encrypt-key-file=%s --input=%s' % (xb_opt_encrypt_key_file, qp) 999 | 1000 | is_encrypted = True 1001 | qp_cmd = 'qpress -di %s' % dest 1002 | else: 1003 | qp_cmd = 'qpress -d %s %s' % (qp, dest) 1004 | 1005 | FNULL = None 1006 | 1007 | _debug("Running qpress command: %s" % qp_cmd) 1008 | if xbc_cmd is not None: 1009 | _debug("Running xbcrypt command: %s" % xbc_cmd) 1010 | 1011 | if not os.path.isdir(dest): os.mkdir(dest, 0755) 1012 | 1013 | if is_encrypted: 1014 | if not xb_opt_debug: 1015 | FNULL = open(os.devnull, 'w') 1016 | xbc = Popen(xbc_cmd, shell=True, stdout=PIPE, stderr=FNULL) 1017 | qp = Popen(qp_cmd, shell=True, stderr=FNULL, stdin=xbc.stdout) 1018 | else: 1019 | xbc = Popen(xbc_cmd, shell=True, stdout=PIPE) 1020 | qp = Popen(qp_cmd, shell=True, stdin=xbc.stdout) 1021 | else: 1022 | if not xb_opt_debug: 1023 | FNULL = open(os.devnull, 'w') 1024 | qp = Popen(qp_cmd, shell=True, stderr=FNULL, stdout=FNULL) 1025 | else: 1026 | qp = Popen(qp_cmd, shell=True) 1027 | 1028 | 1029 | if is_encrypted: 1030 | r = xbc.poll() 1031 | while r is None: 1032 | time.sleep(5) 1033 | r = xbc.poll() 1034 | 1035 | x = qp.poll() 1036 | if x is None: qp.wait() 1037 | x = qp.poll() 1038 | 1039 | if FNULL is not None: 1040 | FNULL.close() 1041 | 1042 | if r != 0: 1043 | _error("Extracting ", qp, " to ", dest, " failed.") 1044 | _error("Extract command was: %s | %s" % (xbc_cmd, qp_cmd)) 1045 | _error("Extract returned exit codes were: %s and %s" % (str(r), str(x))) 1046 | _exit_code(XB_EXIT_EXTRACT_FAIL) 1047 | return False 1048 | else: 1049 | r = qp.poll() 1050 | while r is None: 1051 | time.sleep(5) 1052 | r = qp.poll() 1053 | 1054 | if FNULL is not None: 1055 | FNULL.close() 1056 | 1057 | if r != 0: 1058 | _error("Extracting ", qp, " to ", dest, " failed.") 1059 | _error("Extract command was: %s" % qp_cmd, qp_cmd) 1060 | _error("Extract returned exit codes was: %s" % str(r)) 1061 | _exit_code(XB_EXIT_EXTRACT_FAIL) 1062 | return False 1063 | 1064 | return True 1065 | 1066 | def _extract_qp_decompress(dest): 1067 | """ this may not be efficient with millions of tables 1068 | unlike how _extract_xbcrypt does it with recursive which 1069 | avoids extra stat() 1070 | """ 1071 | 1072 | for root, dirs, files in os.walk(dest): 1073 | for f in files: 1074 | if f.endswith('.qp'): 1075 | _debug("Found %s to decompress" % os.path.join(root, f)) 1076 | _extract_qp_file(os.path.join(root, f)) 1077 | os.remove(os.path.join(root, f)) 1078 | 1079 | return True 1080 | 1081 | def _extract_qp_file(file): 1082 | qp_cmd = "qpress -d %s %s" % (file, os.path.dirname(file).rstrip('/')) 1083 | qp = Popen(qp_cmd, shell=True) 1084 | 1085 | r = qp.poll() 1086 | while r is None: 1087 | time.sleep(0.5) 1088 | r = qp.poll() 1089 | 1090 | if r != 0: 1091 | _error("Decompressing %s failed." % file) 1092 | _error("Extract command was: %s" % qp_cmd) 1093 | _error("Extract returned exit code was: %s" % str(r)) 1094 | _exit_code(XB_EXIT_EXTRACT_FAIL) 1095 | _die("Aborting") 1096 | 1097 | return True 1098 | 1099 | def _extract_ibx_decompress(dest, meta = None): 1100 | if (XB_VERSION_MAJOR == 2 and XB_VERSION_MINOR == 1 and XB_VERSION_REV < 4) or \ 1101 | XB_VERSION_MAJOR < 2 or (XB_VERSION_MAJOR == 2 and XB_VERSION_MINOR <= 0): 1102 | _say( 1103 | "You are running an older xtrabackup version " 1104 | "that do not have --decompress support, " 1105 | "switching manual decompresssion") 1106 | return _extract_qp_decompress(dest) 1107 | 1108 | # Now we decompress *.qp files 1109 | if xb_ibx_bin == 'xtrabackup': 1110 | ibx_cmd = xb_ibx_bin + ' --decompress --target-dir=%s' % dest 1111 | else: 1112 | ibx_cmd = xb_ibx_bin + ' --decompress %s' % dest 1113 | FNULL = None 1114 | 1115 | if not xb_opt_debug: 1116 | FNULL = open(os.devnull, 'w') 1117 | ibx = Popen(ibx_cmd, shell=True, stdout=FNULL, stderr=FNULL) 1118 | else: 1119 | ibx = Popen(ibx_cmd, shell=True) 1120 | 1121 | r = ibx.poll() 1122 | while r is None: 1123 | time.sleep(5) 1124 | r = ibx.poll() 1125 | 1126 | if FNULL is not None: 1127 | FNULL.close() 1128 | 1129 | if r != 0: 1130 | _error("Decompressing *.qp files failed.") 1131 | _error("Extract command was: %s" % ibx_cmd) 1132 | _error("Extract returned exit code was: %s" % str(r)) 1133 | _exit_code(XB_EXIT_EXTRACT_FAIL) 1134 | return False 1135 | 1136 | _cleanup_files_by_ext(dest, 'qp') 1137 | 1138 | return True 1139 | 1140 | def _decompress(archive, dest, meta = None): 1141 | if not os.path.isdir(dest): 1142 | _warn("Destination directory not found, ", dest, " cannot compress") 1143 | return False 1144 | 1145 | if not os.path.isfile(archive): 1146 | _warn("Source archive does not exists, ", archive) 1147 | return False 1148 | 1149 | if archive[-6:] == 'tar.gz': 1150 | return _extract_tgz(archive, dest, meta) 1151 | elif archive[-6:] == 'xbs.gz': 1152 | return _extract_xgz(archive, dest, meta) 1153 | elif archive[-6:] == 'xbs.qp': 1154 | return _extract_stream_qpress(archive, dest, meta) 1155 | elif archive[-14:] == 'xbs.qp.xbcrypt': 1156 | return _extract_stream_qpress(archive, dest, meta) 1157 | elif archive[-2:] == 'qp': 1158 | return _extract_nostream_qpress(archive, dest, meta) 1159 | elif archive[-10:] == 'qp.xbcrypt': 1160 | return _extract_nostream_qpress(archive, dest, meta) 1161 | else: 1162 | _warn("Unknown archive format %s" % archive) 1163 | return False 1164 | 1165 | def _init_log_file(path, close=False, create=True): 1166 | global xb_log_file 1167 | global xb_log_fd 1168 | 1169 | _debug("Attempting init log from \"%s\" to \"%s\"" % (xb_log_file, path)) 1170 | 1171 | if xb_opt_command is None or xb_opt_command in cmd_no_log: 1172 | return True 1173 | 1174 | d = path.rstrip('/') 1175 | if os.path.isdir(d): 1176 | _warn("New log file path must be a full path to file name, ", 1177 | "directory is given %s, aborting log init" % d) 1178 | return False 1179 | elif os.path.isfile(d): 1180 | if d == xb_log_file: 1181 | _debug("New log file is the same is current") 1182 | if not close and xb_log_fd is None: 1183 | _debug("Log file is not open, opening ...") 1184 | xb_log_fd = os.open(d, os.O_WRONLY|os.O_APPEND) 1185 | 1186 | return True 1187 | else: 1188 | _debug("New log file exists, %s, aborting log init" % d) 1189 | return False 1190 | 1191 | if os.path.isfile(xb_log_file): 1192 | _debug("Renaming log file from %s to %s" % (xb_log_file, d)) 1193 | 1194 | if close and xb_log_fd is not None: os.close(xb_log_fd) 1195 | return True 1196 | 1197 | if xb_log_fd is not None: os.close(xb_log_fd) 1198 | 1199 | shutil.move(xb_log_file, d) 1200 | xb_log_file = d 1201 | 1202 | if not close: xb_log_fd = os.open(d, os.O_WRONLY|os.O_APPEND) 1203 | else: 1204 | if create: 1205 | xb_log_file = path 1206 | _debug("Opening new log file %s" % path) 1207 | xb_log_fd = os.open(path, os.O_WRONLY|os.O_CREAT) 1208 | _debug("Logging to %s" % xb_log_file) 1209 | _debug("Log fd: %s" % str(xb_log_fd)) 1210 | else: 1211 | _warn("Log file %s does not exist, aborting rename" % xb_log_file) 1212 | 1213 | def _cleanup_files_by_ext(fpath, ext): 1214 | if os.path.isdir(fpath): 1215 | ls = os.listdir(fpath) 1216 | for f in ls: 1217 | _cleanup_files_by_ext(os.path.join(fpath, f), ext) 1218 | elif os.path.isfile(fpath) and fpath.endswith(".%s" % ext): 1219 | os.remove(fpath) 1220 | #else: 1221 | # _debug("Path does not match extension %s" % fpath) 1222 | 1223 | def _cleanup_dir(folder, excludes = []): 1224 | _say("Cleaning up %s excluding %s" % (folder, str(excludes))) 1225 | if not os.path.isdir(folder): 1226 | _warn("Cannot cleanup %s, directory does not exist" % folder) 1227 | return False 1228 | 1229 | l = os.listdir(folder) 1230 | if len(l) <= 0: return True 1231 | 1232 | for d in l: 1233 | if d in excludes: continue 1234 | 1235 | f = os.path.join(folder, d) 1236 | _debug("Deleting %s" % f) 1237 | if os.path.isfile(f): os.remove(f) 1238 | else: shutil.rmtree(f) 1239 | 1240 | def _get_binlog_info_from_log(logfile): 1241 | global xb_this_binlog 1242 | global xb_this_master_binlog 1243 | 1244 | if not os.path.isfile(logfile): return False 1245 | 1246 | with open(logfile, "r") as f: 1247 | f.seek (0, 2) 1248 | fsize = f.tell() 1249 | f.seek (max (fsize-1024, 0), 0) 1250 | lines = f.readlines() 1251 | 1252 | lines = lines[-20:] 1253 | m = None 1254 | s = None 1255 | 1256 | for l in lines: 1257 | if 'MySQL binlog position' in l: 1258 | m = re.search('filename \'(.*)\',', l) 1259 | 1260 | if 'MySQL slave binlog position' in l: 1261 | s = re.search('filename \'(.*)\'', l) 1262 | 1263 | if m is not None: 1264 | _say("Found binary log name from log %s" % m.group(1)) 1265 | xb_this_binlog = m.group(1) 1266 | 1267 | if s is not None: 1268 | _say("Found master binary log name from log %s" % s.group(1)) 1269 | xb_this_master_binlog = s.group(1) 1270 | 1271 | def _notify_by_email(subject, msg="", to=None): 1272 | try: 1273 | if to is not None: 1274 | recpt = to 1275 | else: 1276 | recpt = xb_opt_notify_by_email 1277 | 1278 | if os.path.isfile(xb_log_file): 1279 | fp = open(xb_log_file, 'rb') 1280 | log = fp.read() 1281 | fp.close() 1282 | msg = "%s\n\n%s" % (msg, log) 1283 | 1284 | fr = "%s@%s" % (xb_user, xb_hostname) 1285 | hdr = "From: %s\n" % fr 1286 | hdr += "To: %s\n" % recpt 1287 | hdr += "Subject: %s\n\n" % subject 1288 | s = smtplib.SMTP('127.0.0.1') 1289 | s.sendmail(fr, recpt.split(','), hdr + msg) 1290 | s.quit() 1291 | except Exception, e: 1292 | if xb_opt_debug: traceback.print_exc() 1293 | _die("Could not send mail ({0}): {1}".format(e.errno, e.strerror)) 1294 | 1295 | return True 1296 | 1297 | def _ssh_execute(cmd, out=False, nowait=False): 1298 | r_cmd = "(%s) || echo 'CMD_FAIL'" % cmd 1299 | 1300 | if xb_opt_debug: 1301 | ssh_cmd = "ssh -o PasswordAuthentication=no" 1302 | else: ssh_cmd = "ssh -o PasswordAuthentication=no -q" 1303 | 1304 | ssh_cmd = "%s %s %s@%s \"%s\"" % ( 1305 | ssh_cmd, xb_opt_ssh_opts, xb_opt_ssh_user, xb_opt_remote_host, r_cmd) 1306 | _debug("Executing remote command %s" % ssh_cmd) 1307 | 1308 | tee_cmd = "tee %s" % XB_SSH_TMPFILE 1309 | p_tee = None 1310 | 1311 | try: 1312 | if nowait: 1313 | Popen(ssh_cmd, shell=True, close_fds=True) 1314 | return True 1315 | 1316 | tmp_fd = os.open(XB_SSH_TMPFILE, os.O_WRONLY|os.O_CREAT|os.O_TRUNC) 1317 | 1318 | if out and not xb_opt_quiet: 1319 | p = Popen(ssh_cmd, shell=True, stdout=PIPE) 1320 | p_tee = Popen(tee_cmd, shell=True, stdin=p.stdout) 1321 | else: 1322 | p = Popen(ssh_cmd, shell=True, stdout=tmp_fd, stderr=tmp_fd) 1323 | 1324 | r = p.poll() 1325 | while r is None: 1326 | time.sleep(1) 1327 | r = p.poll() 1328 | 1329 | os.close(tmp_fd) 1330 | if p_tee is not None: 1331 | p_tee.wait() 1332 | 1333 | if r != 0: 1334 | _error("SSH command failed with error code %s" % str(r)) 1335 | _exit_code(XB_EXIT_REMOTE_CMD_FAIL) 1336 | return False 1337 | 1338 | f = open(XB_SSH_TMPFILE) 1339 | l = f.readlines() 1340 | if len(l) > 0: 1341 | err = l[-1:][0].rstrip('\n') 1342 | if 'CMD_FAIL' == err: 1343 | err = ', '.join(i.rstrip('\n') for i in l) 1344 | _error("Remote command failed \"%s\"" % err) 1345 | _exit_code(XB_EXIT_REMOTE_CMD_FAIL) 1346 | return False 1347 | else: 1348 | return l[0:][0].rstrip('\n') 1349 | 1350 | return True 1351 | 1352 | except Exception, e: 1353 | _error("Command was: ", ssh_cmd) 1354 | _error("Error: process exited with status %s" % str(e)) 1355 | _exit_code(XB_EXIT_REMOTE_CMD_FAIL) 1356 | raise 1357 | 1358 | def _pre_run_xb(): 1359 | global xb_last_backup 1360 | global xb_last_backup_is 1361 | global xb_last_full 1362 | 1363 | if xb_opt_remote_push_only: 1364 | t = _ssh_execute( 1365 | "%s -q --meta-item=xb_last_backup,xb_last_backup_is,xb_last_full meta" % xb_opt_remote_script 1366 | ) 1367 | if t: t = t.split(' ') 1368 | if len(t) < 3: 1369 | _die("Unable to determine previous backup information from remote") 1370 | 1371 | xb_last_backup, xb_last_backup_is, xb_last_full = t 1372 | 1373 | def _oldest_binlog_from_backup(): 1374 | old_binlog = False 1375 | 1376 | if xb_opt_binlog_from_master: 1377 | field_name = 'master_log_bin' 1378 | else: 1379 | field_name = 'log_bin' 1380 | 1381 | if xb_full_list is None or len(xb_full_list) <= 0: 1382 | return False 1383 | 1384 | # Backward compatibility with 'xbackup.meta' 1385 | meta_file_dir = os.path.join(xb_stor_full, xb_full_list[-1:][0]) 1386 | meta = _read_backup_metadata(meta_file_dir) 1387 | 1388 | try: 1389 | old_binlog = meta.get(XB_BIN_NAME, field_name) 1390 | _debug("Found binlog from oldest full backup, %s" % old_binlog) 1391 | 1392 | if old_binlog == 'None': 1393 | _warn("Invalid old binlog record from full backup, found '%s'" % old_binlog) 1394 | old_binlog = False 1395 | except NoOptionError, e: 1396 | _warn("No binlog information from oldest full backup!") 1397 | 1398 | return old_binlog 1399 | 1400 | def _purge_binlogs_to(old_binlog): 1401 | if xb_binlogs_list is None: return 1402 | 1403 | if xb_opt_retention_binlogs is None: 1404 | for l in xb_binlogs_list: 1405 | if l < old_binlog: 1406 | _say("Deleting old binary log %s" % l) 1407 | os.remove(os.path.join(xb_stor_binlogs, l)) 1408 | else: 1409 | x = int(time.time())-(xb_opt_retention_binlogs*24*60*60) 1410 | prev = None 1411 | for l in xb_binlogs_list: 1412 | f = os.path.join(xb_stor_binlogs, l) 1413 | b = open(f, 'rb') 1414 | b.seek(4) 1415 | ts = unpack('I', b.read(4))[0] 1416 | ts_out = str(datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M:%S')) 1417 | _debug("%s created at %s" % (l, ts_out)) 1418 | b.close() 1419 | 1420 | if prev is not None: 1421 | if ts < x: 1422 | _debug("Pruning %s" % prev) 1423 | os.remove(prev) 1424 | prev = f 1425 | # Current binlog creation ts is later than start of retention period 1426 | # We keep from this binlog and keep the previous one as well 1427 | else: 1428 | _say("%s matches binary log retention period, stopping" % f) 1429 | break 1430 | else: prev = f 1431 | 1432 | def _purge_bitmaps_to(lsn): 1433 | _say("Purging bitmap files to LSN: %s" % lsn) 1434 | 1435 | if not db_connect(): 1436 | _error("Failed to connect to server, unable to purge bitmaps.") 1437 | _exit_code(XB_EXIT_BITMAP_PURGE_FAIL) 1438 | return False 1439 | 1440 | try: 1441 | cur = xb_mysqldb.cursor(MySQLdb.cursors.DictCursor) 1442 | cur.execute("PURGE CHANGED_PAGE_BITMAPS BEFORE %s" % lsn) 1443 | except MySQLdb.OperationalError, e: 1444 | _error("Got MySQL error %d, \"%s\" at execute" % (e.args[0], e.args[1])) 1445 | _error("Failed to purge bitmaps!") 1446 | _exit_code(XB_EXIT_BITMAP_PURGE_FAIL) 1447 | return False 1448 | 1449 | return True 1450 | 1451 | def _find_open_port(): 1452 | """ 1453 | Check for an open port via netstat 1454 | """ 1455 | 1456 | netstat = _which('netstat') 1457 | nc = _which('nc') 1458 | awk = _which('awk') 1459 | grep = _which('grep') 1460 | port = False 1461 | 1462 | for p in range(xb_opt_remote_nc_port_min, xb_opt_remote_nc_port_max+1): 1463 | netstat_cmd = Popen([netstat, '-plnt'], stdout=PIPE) 1464 | awk_cmd = Popen([awk, ' {print $4}'], stdin=netstat_cmd.stdout, stdout=PIPE) 1465 | grep_cmd = Popen([grep, "\:%d" % p], stdin=awk_cmd.stdout, stdout=PIPE) 1466 | grep_cmd.communicate() 1467 | 1468 | if int(grep_cmd.returncode) > 0: 1469 | port = p 1470 | break 1471 | else: continue 1472 | 1473 | return port 1474 | 1475 | def _is_remote_nc_port_open(port): 1476 | # We only look for on specific line on the process tree 1477 | # this can be improved in the future 1478 | is_open = _ssh_execute( 1479 | "ps -f -u %s | egrep 'nc -l %d$'|wc -l" % (xb_opt_ssh_user, port)) 1480 | 1481 | return int(is_open) 1482 | 1483 | def _open_remote_nc_port(port, pipe_cmd): 1484 | is_open = _is_remote_nc_port_open(port) 1485 | 1486 | if is_open == 1: 1487 | _die("The requested port is already open on the remote server") 1488 | 1489 | for i in range(1, 4): 1490 | _ssh_execute( 1491 | "nc -l %d | %s" % (port, pipe_cmd), 1492 | nowait=True) 1493 | time.sleep(5) 1494 | 1495 | is_open = _is_remote_nc_port_open(port) 1496 | _debug("Got %d from remote port check" % is_open) 1497 | 1498 | if is_open == 1: break 1499 | if is_open == 0 and xb_opt_debug: 1500 | _debug("Failed to open remote nc port after %d attempt" % i) 1501 | 1502 | if is_open == 0: 1503 | _die("Could not open netcat port on remote server after 3 attempts") 1504 | 1505 | return True 1506 | 1507 | def _close_remote_nc_port(port): 1508 | is_open = 0 1509 | 1510 | for i in range(1, 4): 1511 | _ssh_execute( 1512 | "echo 'FORCE_CLOSE' | nc localhost %d" % port, 1513 | nowait=True) 1514 | time.sleep(3) 1515 | 1516 | is_open = _is_remote_nc_port_open(port) 1517 | if is_open == 0: break 1518 | 1519 | if is_open == 1 and xb_opt_debug: 1520 | _debug("Failed to close remote nc port after %d attempt" % i) 1521 | 1522 | if is_open == 1: 1523 | _debug("Failed to close remote netcat port after multiple attempts") 1524 | return False 1525 | 1526 | def _push_to_remote_scp(src, dst): 1527 | """ 1528 | Push a backup folder to a remote location via scp or netcat 1529 | """ 1530 | 1531 | global xb_exit_code 1532 | 1533 | FNULL = None 1534 | p_scp = None 1535 | 1536 | # Unfortunately rsync will not work on all platforms with closed 1537 | # security policies, we will add this as an option in the future. 1538 | #run_cmd = "rsync -avz -e 'ssh -o PasswordAuthentication=no -q %s' %s %s@%s:%s" % ( 1539 | run_cmd = "scp -r -o PasswordAuthentication=no -q %s %s %s@%s:%s" % ( 1540 | xb_opt_ssh_opts, src, xb_opt_ssh_user, xb_opt_remote_host, dst 1541 | ) 1542 | 1543 | _say("Pushing %s to remote host %s:%s" % (src, xb_opt_remote_host, dst)) 1544 | _debug("Push command is: %s" % str(run_cmd)) 1545 | 1546 | if not xb_opt_debug: 1547 | FNULL = open(os.devnull, 'w') 1548 | p_scp = Popen(run_cmd, shell=True, stdout=FNULL, stderr=STDOUT) 1549 | else: 1550 | p_scp = Popen(run_cmd, shell=True) 1551 | 1552 | r = p_scp.poll() 1553 | while r is None: 1554 | time.sleep(3) 1555 | r = p_scp.poll() 1556 | 1557 | if FNULL is not None: 1558 | FNULL.close() 1559 | 1560 | if r != 0: 1561 | _error("Pushing ", src, " to remote ", dst, " failed.") 1562 | _error("Push command was: ", run_cmd) 1563 | _error("scp returned exit code was: ", str(r)) 1564 | _exit_code(XB_EXIT_REMOTE_PUSH_FAIL) 1565 | return False 1566 | 1567 | return True 1568 | 1569 | def _push_to_remote_netcat(src, dst): 1570 | """ 1571 | Push a file or a directory to a remote destination. 1572 | """ 1573 | 1574 | global xb_exit_code 1575 | 1576 | FNULL = None 1577 | p_nc = None 1578 | 1579 | nc_cmd = "nc %s %d" % (xb_opt_remote_host, xb_opt_remote_nc_port_min) 1580 | 1581 | if os.path.isdir(src): 1582 | os.chdir(src) 1583 | tar_cmd = "tar -czf - ." 1584 | else: 1585 | os.chdir(os.path.dirname(src)) 1586 | tar_cmd = "tar -czf - %s" % os.path.basename(src) 1587 | 1588 | _say("Pushing %s to remote host %s:%s" % (src, xb_opt_remote_host, dst)) 1589 | _debug("Push command is: %s" % str(nc_cmd)) 1590 | 1591 | _open_remote_nc_port(xb_opt_remote_nc_port_min, "tar -C %s -xzf -" % dst) 1592 | 1593 | if not xb_opt_debug: 1594 | FNULL = open(os.devnull, 'w') 1595 | p_tar = Popen(tar_cmd, shell=True, stdout=PIPE, stderr=FNULL) 1596 | p_nc = Popen(nc_cmd, shell=True, stdin=p_tar.stdout, stderr=FNULL) 1597 | else: 1598 | p_tar = Popen(tar_cmd, shell=True, stdout=PIPE) 1599 | p_nc = Popen(nc_cmd, shell=True, stdin=p_tar.stdout) 1600 | 1601 | r = p_tar.poll() 1602 | while r is None: 1603 | time.sleep(3) 1604 | r = p_tar.poll() 1605 | 1606 | x = p_nc.poll() 1607 | if x is None: p_nc.wait() 1608 | x = p_nc.poll() 1609 | 1610 | if FNULL is not None: 1611 | FNULL.close() 1612 | 1613 | os.chdir(xb_cwd) 1614 | 1615 | if r != 0: 1616 | _error("Pushing ", src, " to remote ", dst, " failed.") 1617 | _error("Push command was: ", nc_cmd) 1618 | _error("scp returned exit code was: ", str(r)) 1619 | _exit_code(XB_EXIT_REMOTE_PUSH_FAIL) 1620 | return False 1621 | 1622 | return True 1623 | 1624 | def run_wipeout(): 1625 | if not xb_opt_wipeout: 1626 | _warn("*************************************************") 1627 | _warn("Warning! This is a dangerous option and it will wipe out ", 1628 | "all traces of any backups. If you are sure, specify ", 1629 | "--i-am-absolutely-sure-wipeout here too!") 1630 | _warn("*************************************************") 1631 | return True 1632 | 1633 | _warn("**WIPEOUT** executing!") 1634 | 1635 | dirs = [xb_stor_full, xb_stor_incr, xb_stor_weekly, xb_stor_monthly, 1636 | xb_stor_binlogs, os.path.join(xb_opt_stor_dir, 'tmp'), 1637 | xb_opt_work_dir] 1638 | 1639 | for d in dirs: 1640 | _say("Wiping out items from %s" % d) 1641 | _cleanup_dir(d) 1642 | 1643 | _say("Done!") 1644 | return True 1645 | 1646 | def run_meta_query(): 1647 | v = [] 1648 | if xb_opt_meta_item: 1649 | x = xb_opt_meta_item.split(',') 1650 | for k in x: 1651 | if k in globals() and globals()[k] is not None: 1652 | v.append(globals()[k]) 1653 | else: v.append('NULL') 1654 | 1655 | if len(v) > 0: 1656 | print ' '.join([str(i) for i in v]) 1657 | else: print 'NULL' 1658 | 1659 | return True 1660 | 1661 | def run_xb(): 1662 | global xb_ibx_opts 1663 | global xb_ibx_bin 1664 | global xb_prepared_backup 1665 | global xb_backup_is_success 1666 | global xb_prepare_is_success 1667 | global xb_this_backup 1668 | global xb_this_backup_remote 1669 | global xb_info_bkp_end 1670 | global xb_info_prep_start 1671 | global xb_info_prep_end 1672 | global xb_this_last_lsn 1673 | 1674 | backup_fname = 'backup' 1675 | backup_archive = None 1676 | 1677 | if xb_last_full: 1678 | xb_prepared_backup = "%s/P_%s" % (xb_opt_work_dir, xb_last_full) 1679 | 1680 | 1681 | if xb_opt_mysql_cnf: 1682 | xb_ibx_opts = ' --defaults-file=' + xb_opt_mysql_cnf + ' ' + xb_ibx_opts 1683 | 1684 | if XB_VERSION_MINOR >= 3: 1685 | xb_ibx_opts = ' --backup' + xb_ibx_opts 1686 | 1687 | xb_ibx_opts = ' --no-timestamp' + xb_ibx_opts 1688 | if xb_opt_mysql_user: 1689 | xb_ibx_opts = (' --user=%s ' % xb_opt_mysql_user) + xb_ibx_opts 1690 | 1691 | if xb_opt_mysql_pass: 1692 | xb_ibx_opts = (' --password=%s ' % xb_opt_mysql_pass) + xb_ibx_opts 1693 | 1694 | if xb_opt_mysql_host: 1695 | xb_ibx_opts = (' --host=%s ' % xb_opt_mysql_host) + xb_ibx_opts 1696 | 1697 | if xb_opt_mysql_sock: 1698 | xb_ibx_opts = (' --socket=%s ' % xb_opt_mysql_sock) + xb_ibx_opts 1699 | 1700 | if xb_opt_remote_push_only \ 1701 | and not _ssh_execute("mkdir -p %s" % xb_this_backup_remote): 1702 | _die("Could not create remote directory to push backup to!") 1703 | 1704 | if xb_opt_compress and not xb_opt_apply_log: 1705 | if xb_opt_compress_with == 'qpress': 1706 | xb_ibx_opts += ' --compress --compress-threads=4' 1707 | 1708 | xb_ibx_opts += ' --stream=xbstream --parallel=4' 1709 | xb_ibx_opts += ' --extra-lsndir=' + xb_this_backup 1710 | os.mkdir(xb_this_backup) 1711 | else: 1712 | xb_ibx_opts += ' --parallel=4' 1713 | 1714 | if xb_opt_encrypt and not xb_opt_apply_log: 1715 | xb_ibx_opts += ' --encrypt=%s --encrypt-threads=4 --encrypt-key-file=%s' % ( 1716 | xb_opt_encrypt, xb_opt_encrypt_key_file) 1717 | 1718 | # Check if rsync binary exists, if so let's use it for uncompressed 1719 | # on-streaming backups 1720 | xb_rsync_bin = _which('rsync') 1721 | if xb_rsync_bin is not None and \ 1722 | ((not xb_opt_compress and not xb_opt_remote_push_only) or \ 1723 | (xb_opt_apply_log)): 1724 | xb_ibx_opts += ' --rsync' 1725 | 1726 | if xb_opt_extra_ibx_options is not None: 1727 | xb_ibx_opts += ' ' + xb_opt_extra_ibx_options 1728 | 1729 | if XB_VERSION_MINOR >= 3: 1730 | # --binlog-info on lp152764811 1731 | xb_ibx_opts += ' --binlog-info=on --target-dir ' + xb_this_backup 1732 | else: 1733 | xb_ibx_opts += ' ' + xb_this_backup 1734 | 1735 | try: 1736 | run_cmd = xb_ibx_bin + xb_ibx_opts 1737 | 1738 | pipe_cmd = '' 1739 | p_cmp = None 1740 | p_tee = None 1741 | log_fd = None 1742 | 1743 | #if not xb_opt_debug: 1744 | # run_cmd += " 2> %s-xbackup.log" % xb_this_backup 1745 | ibx_log = "%s/%s-innobackupex-backup.log" % (xb_opt_work_dir, xb_curdate) 1746 | tee_cmd = "tee %s" % ibx_log 1747 | 1748 | if xb_opt_compress and not xb_opt_apply_log: 1749 | if xb_opt_compress_with == 'qpress': 1750 | backup_fname = 'backup.xbs.qp' 1751 | pipe_cmd = "cat - >" 1752 | else: 1753 | backup_fname = 'backup.xbs.gz' 1754 | pipe_cmd = "gzip - >" 1755 | 1756 | if xb_opt_encrypt: 1757 | backup_fname += '.xbcrypt' 1758 | 1759 | if xb_opt_remote_push_only: 1760 | backup_archive = "%s/%s" % (xb_this_backup_remote, backup_fname) 1761 | else: 1762 | backup_archive = "%s/%s" % (xb_this_backup, backup_fname) 1763 | 1764 | pipe_cmd = "%s %s" % (pipe_cmd, backup_archive) 1765 | 1766 | # We open the netcat port before opening the innobackupex process 1767 | if xb_opt_remote_push_only and xb_opt_remote_nc_port_min: 1768 | _open_remote_nc_port(xb_opt_remote_nc_port_min, pipe_cmd) 1769 | 1770 | if not xb_opt_debug: 1771 | log_fd = os.open(ibx_log, os.O_WRONLY|os.O_CREAT) 1772 | p_ibx = Popen(run_cmd, shell=True, stdout=PIPE, stderr=log_fd) 1773 | else: 1774 | p_ibx = Popen(run_cmd, shell=True, stdout=PIPE, stderr=PIPE) 1775 | p_tee = Popen(tee_cmd, shell=True, stdin=p_ibx.stderr) 1776 | 1777 | if xb_opt_remote_push_only: 1778 | if xb_opt_remote_nc_port_min: 1779 | pipe_cmd = "nc %s %d" % ( 1780 | xb_opt_remote_host, xb_opt_remote_nc_port_min) 1781 | else: 1782 | pipe_cmd = "ssh -o PasswordAuthentication=no -q %s %s@%s '%s'" % ( 1783 | xb_opt_ssh_opts, xb_opt_ssh_user, xb_opt_remote_host, pipe_cmd) 1784 | 1785 | _debug('Piping backup to remote with "%s"' % pipe_cmd) 1786 | else: 1787 | _debug('Compressing backup with "%s"' % pipe_cmd) 1788 | 1789 | p_cmp = Popen(pipe_cmd, stdin=p_ibx.stdout, shell=True) 1790 | elif xb_opt_remote_push_only: 1791 | pipe_cmd = "xbstream -x -C %s" % xb_this_backup_remote 1792 | 1793 | # We open the netcat port before opening the innobackupex process 1794 | if xb_opt_remote_nc_port_min: 1795 | _open_remote_nc_port(xb_opt_remote_nc_port_min, pipe_cmd) 1796 | 1797 | if not xb_opt_debug: 1798 | log_fd = os.open(ibx_log, os.O_WRONLY|os.O_CREAT) 1799 | p_ibx = Popen(run_cmd, shell=True, stdout=PIPE, stderr=log_fd) 1800 | else: 1801 | p_ibx = Popen(run_cmd, shell=True, stdout=PIPE, stderr=PIPE) 1802 | p_tee = Popen(tee_cmd, shell=True, stdin=p_ibx.stderr) 1803 | 1804 | if xb_opt_remote_nc_port_min: 1805 | pipe_cmd = "nc %s %d" % ( 1806 | xb_opt_remote_host, xb_opt_remote_nc_port_min) 1807 | else: 1808 | pipe_cmd = "ssh -o PasswordAuthentication=no -q %s %s@%s '%s'" % ( 1809 | xb_opt_ssh_opts, xb_opt_ssh_user, xb_opt_remote_host, pipe_cmd) 1810 | 1811 | _debug('Piping backup to remote with "%s"' % pipe_cmd) 1812 | p_cmp = Popen(pipe_cmd, stdin=p_ibx.stdout, shell=True) 1813 | else: 1814 | if not xb_opt_debug: 1815 | log_fd = os.open(ibx_log, os.O_WRONLY|os.O_CREAT) 1816 | p_ibx = Popen(run_cmd, shell=True, stderr=log_fd) 1817 | else: 1818 | p_ibx = Popen(run_cmd, shell=True, stderr=PIPE) 1819 | p_tee = Popen(tee_cmd, shell=True, stdin=p_ibx.stderr) 1820 | 1821 | _say("Running xtrabackup with command: ", 1822 | re.sub('\s--password=([^\s]+)', ' --password=*******', run_cmd)) 1823 | 1824 | r = p_ibx.poll() 1825 | while r is None: 1826 | time.sleep(2) 1827 | r = p_ibx.poll() 1828 | 1829 | if p_cmp is not None: p_cmp.wait() 1830 | if p_tee is not None: p_tee.wait() 1831 | 1832 | if log_fd is not None: 1833 | os.close(log_fd) 1834 | 1835 | if r != 0: raise Exception("Non-zero exit of innobackupex command!") 1836 | xb_backup_is_success = True 1837 | xb_info_bkp_end = date(time.time(), '%Y_%m_%d-%H_%M_%S') 1838 | 1839 | except Exception, e: 1840 | _error("Command was: ", run_cmd) 1841 | _error("Error: process exited with status %s" % str(e)) 1842 | _error("Please check innobackupex log file at %s" % ibx_log) 1843 | _exit_code(XB_EXIT_INNOBACKUP_FAIL) 1844 | raise 1845 | 1846 | if xb_opt_command == XB_CMD_FULL and xb_backup_is_success: 1847 | xb_full_list.insert(0, xb_curdate) 1848 | 1849 | if xb_backup_is_success: 1850 | full_ckp = _parse_raw_config(os.path.join(xb_this_backup, XB_CKP_FILE)) 1851 | xb_this_last_lsn = full_ckp.get(XB_BIN_NAME, 'last_lsn') 1852 | 1853 | # Cleanup work directory 1854 | # First, move the innobackupex logfile to the actual backup directory 1855 | if xb_backup_is_success and not xb_opt_apply_log: 1856 | shutil.move(ibx_log, "%s/innobackupex-backup.log" % xb_this_backup) 1857 | ibx_log = "%s/innobackupex-backup.log" % xb_this_backup 1858 | 1859 | if xb_opt_apply_log and xb_backup_is_success: 1860 | if xb_opt_command == XB_CMD_FULL and xb_prepared_backup \ 1861 | and os.path.isdir(xb_prepared_backup): 1862 | _say("Removing previous prepared backup ", xb_prepared_backup) 1863 | shutil.rmtree(xb_prepared_backup) 1864 | 1865 | xb_info_prep_start = date(time.time(), '%Y_%m_%d-%H_%M_%S') 1866 | 1867 | if xb_opt_command == XB_CMD_FULL: 1868 | t = "%s/P_%s" % (xb_opt_work_dir, xb_curdate) 1869 | shutil.copytree(xb_this_backup, t) 1870 | xb_prepare_is_success = _apply_log(t, xb_this_backup) 1871 | else: 1872 | _xb_logfile_copy(xb_this_backup) 1873 | xb_prepare_is_success = _apply_log(xb_this_backup, xb_prepared_backup) 1874 | _xb_logfile_restore(xb_this_backup) 1875 | 1876 | xb_info_prep_end = date(time.time(), '%Y_%m_%d-%H_%M_%S') 1877 | 1878 | if xb_prepare_is_success: 1879 | if xb_opt_command == XB_CMD_FULL: 1880 | t = "%s/%s" % (xb_stor_full, xb_curdate) 1881 | else: 1882 | t = "%s/%s/%s" % (xb_stor_incr, xb_last_full, xb_curdate) 1883 | 1884 | if xb_opt_compress: 1885 | _say("Post apply-log, compressing ", xb_this_backup) 1886 | 1887 | # Create base incremental folder if it does not exist yet 1888 | if xb_opt_command == XB_CMD_INCR: 1889 | ib = os.path.join(xb_stor_incr, xb_last_full) 1890 | if not os.path.isdir(ib): os.mkdir(ib) 1891 | 1892 | if not os.path.isdir(t): os.mkdir(t) 1893 | shutil.copy("%s/xtrabackup_checkpoints" % xb_this_backup, 1894 | "%s/xtrabackup_checkpoints" % t) 1895 | _compress(xb_this_backup, "%s/backup" % t) 1896 | else: 1897 | _say("Post apply-log, copying ", xb_this_backup) 1898 | if xb_opt_command == XB_CMD_FULL: 1899 | shutil.copytree(xb_this_backup, t) 1900 | else: 1901 | shutil.copytree(xb_this_backup, t) 1902 | 1903 | # Update path to this backup to reflect movement to stor dir 1904 | xb_this_backup = t 1905 | 1906 | shutil.move(ibx_log, "%s/innobackupex-backup.log" % t) 1907 | ibx_log = "%s/innobackupex-backup.log" % t 1908 | _say("Backup log has been moved to ", ibx_log) 1909 | else: 1910 | _die('Apply log failed, aborting!') 1911 | else: xb_prepare_is_success = True 1912 | 1913 | # Let's grab our binary log information 1914 | _get_binlog_info_from_log(ibx_log) 1915 | # Let's write the backupe metadata info 1916 | _write_backup_info() 1917 | # Let's preserve our xbackup.log first 1918 | #_init_log_file("%s/%s" % (xb_this_backup, XB_LOG_NAME)) 1919 | 1920 | if xb_opt_remote_host: 1921 | _ssh_execute("mkdir -p %s" % xb_this_backup_remote) 1922 | 1923 | if xb_opt_remote_nc_port_min: 1924 | _push_to_remote_netcat(xb_this_backup, xb_this_backup_remote) 1925 | else: 1926 | _push_to_remote_scp( 1927 | xb_this_backup, 1928 | "%s/" % os.path.dirname(xb_this_backup_remote) 1929 | ) 1930 | 1931 | # Cleanup from our work directory to free up disk space. 1932 | l = os.listdir(xb_opt_work_dir) 1933 | # If xb_opt_apply_log is not enabled, we cleanup the whole work dir 1934 | if not xb_opt_apply_log: 1935 | excludes = [os.path.basename(xb_log_file)] 1936 | else: 1937 | excludes = [os.path.basename(ibx_log), "P_%s" % xb_curdate, 1938 | os.path.basename(XB_LCK_FILE), 1939 | os.path.basename(xb_log_file)] 1940 | 1941 | if xb_prepared_backup: 1942 | excludes.append(os.path.basename(xb_prepared_backup)) 1943 | 1944 | _cleanup_dir(xb_opt_work_dir, excludes) 1945 | 1946 | if xb_prepare_is_success and xb_backup_is_success: 1947 | if xb_opt_remote_host: 1948 | _ssh_execute("%s prune" % xb_opt_remote_script, True) 1949 | 1950 | prune_full_incr() 1951 | prune_weekly() 1952 | prune_monthly() 1953 | 1954 | def run_xb_full(): 1955 | """Execute a full backup""" 1956 | 1957 | global xb_this_backup 1958 | global xb_this_backup_remote 1959 | 1960 | _say("Running FULL backup, started at ", 1961 | date(time.time(), '%Y-%m-%d %H:%M:%S')) 1962 | 1963 | if xb_opt_apply_log is False: 1964 | xb_this_backup = os.path.join(xb_stor_full, xb_curdate) 1965 | 1966 | if os.path.isdir(xb_this_backup): 1967 | _die(xb_this_backup, " backup directory already exists!") 1968 | else: 1969 | xb_this_backup = os.path.join(xb_opt_work_dir, xb_curdate) 1970 | 1971 | if xb_opt_remote_push_only: 1972 | xb_this_backup = os.path.join(xb_opt_work_dir, xb_curdate) 1973 | 1974 | if xb_opt_remote_host: 1975 | xb_this_backup_remote = os.path.join(xb_opt_remote_stor_dir, 'full', xb_curdate) 1976 | 1977 | run_xb() 1978 | 1979 | def run_xb_incr(): 1980 | """Execute an incremental backup""" 1981 | 1982 | global xb_ibx_opts 1983 | global xb_this_backup 1984 | global xb_this_backup_remote 1985 | 1986 | _say("Running INCREMENTAL backup, started at ", 1987 | date(time.time(), '%Y-%m-%d %H:%M:%S')) 1988 | 1989 | _pre_run_xb() 1990 | if xb_last_full == 'NULL' or xb_last_full is None: 1991 | _exit_code(XB_EXIT_NO_FULL) 1992 | _die('Incremental backup requested, ' 1993 | 'but there is no existing base full backup') 1994 | 1995 | if xb_opt_apply_log is False: 1996 | xb_this_backup = os.path.join(xb_stor_incr, xb_last_full, xb_curdate) 1997 | 1998 | if os.path.isdir(xb_this_backup): 1999 | _die(xb_this_backup, " backup directory already exists!") 2000 | 2001 | # Create base incremental folder if it does not exist yet 2002 | ib = os.path.join(xb_stor_incr, xb_last_full) 2003 | if not os.path.isdir(ib): os.mkdir(ib) 2004 | 2005 | else: 2006 | xb_this_backup = os.path.join(xb_opt_work_dir, xb_curdate) 2007 | 2008 | if XB_VERSION_MINOR >= 3: 2009 | xb_ibx_opts = ' --incremental-basedir=' 2010 | else: 2011 | xb_ibx_opts = ' --incremental --incremental-basedir=' 2012 | 2013 | if xb_opt_remote_push_only: 2014 | 2015 | ib = os.path.join(xb_opt_work_dir, xb_last_backup) 2016 | rb = os.path.join(xb_opt_remote_stor_dir, xb_last_backup_is) 2017 | if xb_last_backup_is == XB_CMD_FULL: 2018 | rb = os.path.join(rb, xb_last_backup) 2019 | else: 2020 | rb = os.path.join(rb, xb_last_full, xb_last_backup) 2021 | 2022 | if not os.path.isdir(ib): os.mkdir(ib) 2023 | pull_from_remote(os.path.join(rb, XB_CKP_FILE), os.path.join(ib, XB_CKP_FILE)) 2024 | 2025 | xb_this_backup = os.path.join(xb_opt_work_dir, xb_curdate) 2026 | else: 2027 | if xb_last_backup_is == XB_CMD_FULL: 2028 | ib = os.path.join(xb_stor_full, xb_last_backup) 2029 | else: 2030 | ib = os.path.join(xb_stor_incr, xb_last_full, xb_last_backup) 2031 | 2032 | if xb_opt_remote_host: 2033 | xb_this_backup_remote = os.path.join( 2034 | xb_opt_remote_stor_dir, 'incr', xb_last_full, xb_curdate) 2035 | 2036 | xb_ibx_opts += ib 2037 | 2038 | run_xb() 2039 | 2040 | if xb_opt_purge_bitmaps: 2041 | _purge_bitmaps_to(xb_this_last_lsn) 2042 | 2043 | def run_xb_list(): 2044 | """List existing "valid backups""" 2045 | 2046 | if xb_opt_remote_push_only: 2047 | return _ssh_execute("%s list" % xb_opt_remote_script, True) 2048 | 2049 | if len(xb_full_list) <= 0: 2050 | _say("No backups currently available.") 2051 | 2052 | for f in xb_full_list: 2053 | s = "# Full backup: " + f 2054 | if f in xb_incr_list and xb_incr_list[f] and len(xb_incr_list[f]) > 0: 2055 | s += ", incrementals: " + str(xb_incr_list[f]) 2056 | 2057 | print s 2058 | 2059 | if xb_weekly_list is not None and len(xb_weekly_list) > 0: 2060 | print "# Weekly list: %s" % str(xb_weekly_list) 2061 | 2062 | if xb_monthly_list is not None and len(xb_monthly_list) > 0: 2063 | print "# Monthly list: %s" % str(xb_monthly_list) 2064 | 2065 | if xb_binlogs_list is not None and len(xb_binlogs_list) > 0: 2066 | print "# Binary logs from %s to %s, %d total" % ( 2067 | xb_binlogs_list[0], xb_binlogs_list[-1], len(xb_binlogs_list)) 2068 | 2069 | def run_status(): 2070 | """Display status of last backup - excludes any currently running backup""" 2071 | 2072 | ret = 0 2073 | txt = '' 2074 | 2075 | if xb_opt_remote_push_only: _pre_run_xb() 2076 | 2077 | if xb_backup_in_progress is not None: 2078 | pid = int(xb_backup_in_progress.get(XB_BIN_NAME, 'pid')) 2079 | bkp = xb_backup_in_progress.get(XB_BIN_NAME, 'backup') 2080 | if pid <= 0: 2081 | ret = 2 2082 | txt = 'Invalid PID file found!' 2083 | else: 2084 | bkp_threshold = 24 2085 | last_dt = datetime.strptime(bkp, '%Y_%m_%d-%H_%M_%S') 2086 | old_dt = datetime.now() - timedelta(hours=bkp_threshold) 2087 | 2088 | try: 2089 | os.kill(pid, 0) 2090 | except OSError, e: 2091 | if e.errno == errno.ESRCH: 2092 | ret = 2 2093 | txt = 'PID/lock file exists but process is not running' 2094 | elif e.errno == errno.EPERM: 2095 | ret = 1 2096 | txt = 'Permission denied while checking backup process' 2097 | else: 2098 | ret = 2 2099 | txt = 'Unknown backup process state' 2100 | else: 2101 | if last_dt < old_dt: 2102 | ret = 1 2103 | txt = "Backup has been running for more than %d hours" \ 2104 | % bkp_threshold 2105 | else: txt = "Backup process in progress with PID %d" % pid 2106 | elif not xb_last_backup or xb_last_backup == 'NULL': 2107 | ret = 2 2108 | txt = 'No recent backup identified!' 2109 | else: 2110 | # We check how old our last backup is, for now itis hardcoded to a 2111 | # threshold of 36 hours. If a full backup takes longer than 12hrs 2112 | # then we should adjust this threshold 2113 | dt_threshold = 36 2114 | last_dt = datetime.strptime(xb_last_backup, '%Y_%m_%d-%H_%M_%S') 2115 | old_dt = datetime.now() - timedelta(hours=dt_threshold) 2116 | if last_dt < old_dt: 2117 | ret = 1 2118 | txt = "Last backup %s from %s is more than %d hours old" \ 2119 | % (xb_last_backup_is, xb_last_backup, dt_threshold) 2120 | else: 2121 | txt = "Last backup %s from %s" % (xb_last_backup_is, xb_last_backup) 2122 | 2123 | if ret == 0: txt = "OK - %s" % txt 2124 | elif ret == 1: txt = "WARN - %s" % txt 2125 | else: txt = "CRITICAL - %s" % txt 2126 | 2127 | if xb_opt_status_format == 'nagios': print txt 2128 | elif xb_opt_status_format == 'zabbix': print ret 2129 | sys.exit(ret) 2130 | 2131 | def run_xb_restore_set(prepare_path=None, finalize=True): 2132 | global xb_opt_restore_backup 2133 | 2134 | if xb_opt_restore_dir is None: 2135 | _die('No prepare directory was specified, please specify a folder ', 2136 | 'where you want to stage the prepare.') 2137 | if not os.path.isdir(xb_opt_restore_dir): 2138 | _die("The specified prepare directory is not a valid directory") 2139 | 2140 | if xb_opt_restore_backup is None: 2141 | xb_opt_restore_backup = xb_last_backup 2142 | 2143 | backup_is = XB_CMD_FULL 2144 | the_backup = None 2145 | the_backup_path = '' 2146 | full_backup = None 2147 | prepare_success = False 2148 | 2149 | # If xb_opt_restore_backup is specified, let's determine what kind of backup 2150 | # it is 2151 | if len(xb_full_list) <= 0: 2152 | _die("No backups currently available.") 2153 | 2154 | for f in xb_full_list: 2155 | # Break if we have already found the backup 2156 | if the_backup is not None: break 2157 | 2158 | if f == xb_opt_restore_backup: 2159 | the_backup = f 2160 | break 2161 | 2162 | if f in xb_incr_list and len(xb_incr_list[f]) > 0: 2163 | for i in xb_incr_list[f]: 2164 | if i == xb_opt_restore_backup: 2165 | the_backup = i 2166 | full_backup = f 2167 | backup_is = XB_CMD_INCR 2168 | break 2169 | 2170 | if the_backup is None: 2171 | _die("The specified backup to prepare was not found, check list") 2172 | 2173 | if not prepare_path: 2174 | prepare_path = os.path.join(xb_opt_restore_dir, "P_%s" % the_backup) 2175 | _say("Found backup %s to prepare of type %s" % (the_backup, backup_is)) 2176 | 2177 | if os.path.isdir(prepare_path): 2178 | _die("Cannot prepare from %s, directory already exists" % prepare_path) 2179 | 2180 | # If we are restoring the most recent backup and apply-log is enabled 2181 | # for the backups, we can use the existing prepared backup in the 2182 | # work directory 2183 | #if the_backup == xb_last_backup and xb_opt_apply_log: 2184 | # the_backup_path = os.path.join(xb_opt_work_dir, "P_%s" % xb_last_full) 2185 | # if os.path.isdir(the_backup_path): 2186 | # _say("Using existing prepared backup for the restore") 2187 | # _say("Copying %s to %s" % (the_backup_path, prepare_path)) 2188 | # 2189 | # shutil.copytree(the_backup_path, prepare_path) 2190 | 2191 | if backup_is == XB_CMD_FULL: 2192 | the_backup_path = os.path.join(xb_stor_full, the_backup) 2193 | 2194 | if finalize: 2195 | prepare_success = _prepare_backup(the_backup_path, prepare_path, finalize) 2196 | else: 2197 | prepare_success = _prepare_backup(the_backup_path, prepare_path) 2198 | 2199 | if not prepare_success: 2200 | _die("There was a problem preparing full backup %s" % the_backup_path) 2201 | 2202 | elif backup_is == XB_CMD_INCR: 2203 | # First let's work on the full backup for this incremental 2204 | the_backup_path = os.path.join(xb_stor_incr, full_backup, the_backup) 2205 | bkp_info = _read_backup_metadata(the_backup_path) 2206 | 2207 | bkp_full = bkp_info.get(XB_BIN_NAME, 'full') 2208 | current_bkp = os.path.join(xb_stor_full, bkp_full) 2209 | 2210 | if not _prepare_backup(current_bkp, prepare_path): 2211 | _die("There was a problem preparing base backup %s" % current_bkp) 2212 | 2213 | current_incr_list = xb_incr_list[bkp_full] 2214 | current_incr_list.reverse() 2215 | 2216 | for current_bkp in current_incr_list: 2217 | current_bkp_path = os.path.join(xb_stor_incr, bkp_full, current_bkp) 2218 | 2219 | if current_bkp == the_backup and finalize: 2220 | prepare_success = _prepare_backup(current_bkp_path, prepare_path, finalize) 2221 | else: 2222 | prepare_success = _prepare_backup(current_bkp_path, prepare_path) 2223 | 2224 | if not prepare_success: 2225 | _die("There was a problem applying incremental backup ", 2226 | "%s to %s" % (current_bkp_path, prepare_path)) 2227 | 2228 | if current_bkp == the_backup: 2229 | # We call a final apply-log on the resulting 2230 | # set if finalize == True 2231 | if finalize: _apply_log(prepare_path, final = finalize) 2232 | break 2233 | 2234 | _say("Prepare of backup %s successfully completed" % the_backup) 2235 | 2236 | def run_xb_apply_last(): 2237 | global xb_opt_restore_dir 2238 | 2239 | # Get list of backups 2240 | # Determine last backup 2241 | # Check from work dir if P_ dir matches last full 2242 | # if not, cleanup and copy full or extract 2243 | 2244 | prepare_path = os.path.join(xb_opt_work_dir, "P_%s" % xb_last_full) 2245 | if not os.path.isdir(prepare_path) \ 2246 | or not os.path.isfile(os.path.join(prepare_path, XB_CKP_FILE)): 2247 | _cleanup_dir(xb_opt_work_dir) 2248 | xb_opt_restore_dir = xb_opt_work_dir 2249 | run_xb_restore_set(prepare_path, False) 2250 | return True 2251 | 2252 | full_ckp = _parse_raw_config(os.path.join(prepare_path, XB_CKP_FILE)) 2253 | to_lsn = full_ckp.get(XB_BIN_NAME, 'to_lsn') 2254 | 2255 | if xb_last_full not in xb_incr_list or xb_incr_list[xb_last_full] is None: 2256 | _say("No incremental backups taken for %s, we're done for now" % xb_last_full) 2257 | return True 2258 | 2259 | i = xb_incr_list[xb_last_full] 2260 | i.reverse() 2261 | are_we_skippping = True 2262 | 2263 | for d in i: 2264 | bkp_path = "%s/incr/%s/%s" % (xb_opt_stor_dir, xb_last_full, d) 2265 | 2266 | if are_we_skippping: 2267 | ckp = _parse_raw_config("%s/%s" % (bkp_path, XB_CKP_FILE)) 2268 | _debug("Incremental %s, from_lsn: %s, to_lsn: %s, last_lsn: %s" % ( 2269 | bkp_path, ckp.get(XB_BIN_NAME, 'from_lsn'), 2270 | ckp.get(XB_BIN_NAME, 'to_lsn'), to_lsn)) 2271 | if to_lsn > ckp.get(XB_BIN_NAME, 'from_lsn'): 2272 | continue 2273 | else: 2274 | are_we_skippping = False 2275 | 2276 | if not _prepare_backup(bkp_path, prepare_path): 2277 | _die("Apply log of incremental backup %s to %s failed" % 2278 | (bkp_path, prepare_path)) 2279 | 2280 | _say("Apply-last-log completed OK") 2281 | 2282 | def run_binlog_stream(): 2283 | global xb_last_binlog 2284 | global xb_first_binlog 2285 | 2286 | if xb_opt_binlog_binary is not None: 2287 | if not os.path.isfile(xb_opt_binlog_binary): 2288 | _die("The specified mysqlbinlog binary", 2289 | "%s does not exist" % xb_opt_binlog_binary) 2290 | else: mysqlbinlog = xb_opt_binlog_binary 2291 | else: mysqlbinlog = 'mysqlbinlog' 2292 | 2293 | 2294 | # Determine oldest binlog we should get based on xb_last_full log file 2295 | # Determine latest binlog we have 2296 | old_binlog = _oldest_binlog_from_backup() 2297 | 2298 | if not xb_first_binlog and not old_binlog and not xb_opt_first_binlog: 2299 | _die("Cannot proceed, no binlog information from oldest backup nor ", 2300 | "there are any existing binlogs yet. Please try with --first-binlog ", 2301 | "option to specify the first binlog to start copying") 2302 | 2303 | if not old_binlog: old_binlog = xb_first_binlog 2304 | 2305 | # An explicit first-binlog takes precedence 2306 | if xb_opt_first_binlog: old_binlog = xb_opt_first_binlog 2307 | 2308 | _say("Maintaing binary logs from %s" % old_binlog) 2309 | 2310 | if not db_connect(): 2311 | _die("Failed to connect to remote host, ", 2312 | "unable to check list of binary logs.") 2313 | 2314 | cur = xb_mysqldb.cursor(MySQLdb.cursors.DictCursor) 2315 | cur.execute('SHOW BINARY LOGS') 2316 | logs = [] 2317 | low = None 2318 | high = None 2319 | 2320 | while True: 2321 | row = cur.fetchone() 2322 | if row is None: break 2323 | logs.append(row['Log_name']) 2324 | if low is None: low = row['Log_name'] 2325 | high = row['Log_name'] 2326 | 2327 | db_close() 2328 | if xb_last_binlog is None: xb_last_binlog = old_binlog 2329 | 2330 | _debug("old_binlog: %s, xb_last_binlog: %s, found_binlogs: %s" % ( 2331 | old_binlog, xb_last_binlog, str(logs))) 2332 | 2333 | if old_binlog not in logs and not int(old_binlog[-6:]) <= int(xb_last_binlog[-6:]): 2334 | _die("I cannot find our oldest binlog from the available binary logs ", 2335 | "on the server, aborting! Try again with --first-binlog") 2336 | 2337 | if xb_last_binlog not in logs: 2338 | _die("I cannot find our newest binlog from the available binary logs ", 2339 | "on the server, aborting! Try again with --first-binlog") 2340 | 2341 | run_cmd_s = mysqlbinlog 2342 | 2343 | if xb_opt_mysql_cnf is not None: 2344 | run_cmd_s += " --defaults-file=%s" % xb_opt_mysql_cnf 2345 | 2346 | run_cmd_s += " --read-from-remote-server --raw --stop-never" 2347 | 2348 | if xb_opt_mysql_user is not None: 2349 | run_cmd_s += " --user=%s" % xb_opt_mysql_user 2350 | 2351 | if xb_opt_mysql_pass is not None: 2352 | run_cmd_s += " --password=%s" % xb_opt_mysql_pass 2353 | 2354 | if xb_opt_mysql_host is not None: 2355 | run_cmd_s += " --host=%s" % xb_opt_mysql_host 2356 | 2357 | if xb_opt_mysql_port is not None: 2358 | run_cmd_s += " --port=%s" % xb_opt_mysql_port 2359 | 2360 | run_failures = 0 2361 | sleeps = 0 2362 | poll = 15 2363 | 2364 | try: 2365 | os.chdir(xb_stor_binlogs) 2366 | 2367 | while True: 2368 | FNULL = None 2369 | run_cmd = ("%s %s" % (run_cmd_s, xb_last_binlog)) 2370 | _debug("Running mysqlbinlog with: %s" % run_cmd) 2371 | 2372 | if xb_opt_debug: 2373 | p = Popen(run_cmd, shell=True) 2374 | else: 2375 | FNULL = open(os.devnull, 'w') 2376 | p = Popen(run_cmd, shell=True, stdout=FNULL, stderr=FNULL) 2377 | 2378 | pid_file = os.path.join('/tmp', "%s-binlog-stream.pid" % XB_BIN_NAME) 2379 | pid_file_h = open(pid_file, 'w') 2380 | pid_file_h.write(str(p.pid)) 2381 | pid_file_h.close() 2382 | 2383 | r = p.poll() 2384 | while r is None: 2385 | time.sleep(poll) 2386 | sleeps += poll 2387 | 2388 | if sleeps >= 1800: 2389 | # We re-evaluate our oldest binlog to keep and purge older ones 2390 | list_backups() 2391 | old_binlog = _oldest_binlog_from_backup() 2392 | if not old_binlog: old_binlog = xb_first_binlog 2393 | _purge_binlogs_to(old_binlog) 2394 | _say("Maintaing binary logs from %s" % old_binlog) 2395 | sleeps = 0 2396 | 2397 | 2398 | if XB_SIGTERM_CAUGHT: 2399 | p.kill() 2400 | sys.exit(0) 2401 | 2402 | r = p.poll() 2403 | 2404 | if FNULL is not None: 2405 | FNULL.close() 2406 | 2407 | if r != 0: 2408 | _error("mysqlbinlog command failed with error code %s" % str(r)) 2409 | _exit_code(XB_EXIT_BINLOG_STREAM_FAIL) 2410 | run_failures += 1 2411 | if run_failures == 10: return False 2412 | else: 2413 | _say("Reconnecting mysqlbinlog ...") 2414 | list_binlogs() 2415 | 2416 | os.chdir(xb_cwd) 2417 | except Exception, e: 2418 | _error("Command was: ", run_cmd) 2419 | _error("Error: process exited with status %s" % str(e)) 2420 | _exit_code(XB_EXIT_BINLOG_STREAM_FAIL) 2421 | raise 2422 | 2423 | return True 2424 | 2425 | def prune_full_incr(): 2426 | """Prune full/incremental sets from the store directory""" 2427 | 2428 | if len(xb_full_list) <= xb_opt_retention_sets: return True 2429 | 2430 | while len(xb_full_list) > xb_opt_retention_sets: 2431 | d = xb_full_list.pop() 2432 | 2433 | if xb_is_last_day_of_week and xb_opt_retention_weeks > 0 \ 2434 | and len(xb_full_list) == xb_opt_retention_sets: 2435 | # If today is the last day of the week, i.e Sunday 2436 | # we will take our xb_opt_retention_sets + 1 backup 2437 | # set and copy to our weekly folder 2438 | _say("Rotating backup %s to weekly" % d) 2439 | if d in xb_incr_list is not None: 2440 | w = xb_incr_list[d][0] 2441 | else: w = d 2442 | 2443 | w_dir = os.path.join(xb_stor_weekly, w) 2444 | os.mkdir(w_dir, 0755) 2445 | shutil.copytree( 2446 | os.path.join(xb_stor_full, d), os.path.join(w_dir, 'full')) 2447 | 2448 | # If we have incremental backups, we copy them too 2449 | if w != d: 2450 | shutil.copytree( 2451 | os.path.join(xb_stor_incr, d), os.path.join(w_dir, 'incr')) 2452 | 2453 | if os.path.isdir(os.path.join(xb_stor_incr, d)): 2454 | _say("Pruning incremental backup ", os.path.join(xb_stor_incr, d)) 2455 | shutil.rmtree(os.path.join(xb_stor_incr, d)) 2456 | 2457 | if os.path.isdir(os.path.join(xb_stor_full, d)): 2458 | shutil.rmtree(os.path.join(xb_stor_full, d)) 2459 | _say("Pruning full backup ", os.path.join(xb_stor_full, d)) 2460 | 2461 | def prune_weekly(): 2462 | """Prune weekly sets from the weekly store directory""" 2463 | 2464 | if xb_weekly_list is None \ 2465 | or len(xb_weekly_list) <= xb_opt_retention_weeks: 2466 | return True 2467 | 2468 | while len(xb_weekly_list) > xb_opt_retention_weeks: 2469 | d = xb_weekly_list.pop() 2470 | w_dir = os.path.join(xb_stor_weekly, d) 2471 | 2472 | # If this weekly set has our end of the month backup 2473 | # we rotate it first to monthly before deleting 2474 | dt = datetime.strptime(d, '%Y_%m_%d-%H_%M_%S') 2475 | m = dt - timedelta(days=6) 2476 | 2477 | if m.month < dt.month: 2478 | _say("Rotating backup %s to monthly" % d) 2479 | shutil.copytree(w_dir, os.path.join(xb_stor_monthly, d)) 2480 | 2481 | shutil.rmtree(w_dir) 2482 | 2483 | def prune_monthly(): 2484 | """Prune monthly monthly sets from monthly store directory""" 2485 | 2486 | if xb_monthly_list is None \ 2487 | or len(xb_monthly_list) <= xb_opt_retention_months: 2488 | return True 2489 | 2490 | while len(xb_monthly_list) > xb_opt_retention_months: 2491 | d = xb_monthly_list.pop() 2492 | m_dir = os.path.join(xb_stor_monthly, d) 2493 | 2494 | if os.path.isdir(m_dir): 2495 | _say("Pruning old monthly backup %s" % m_dir) 2496 | shutil.rmtree(m_dir) 2497 | 2498 | def pull_from_remote(src, dst): 2499 | """Pull a file from remote via scp""" 2500 | global xb_exit_code 2501 | 2502 | FNULL = None 2503 | p_scp = None 2504 | 2505 | run_cmd = "scp -r -o PasswordAuthentication=no -q %s %s@%s:%s %s" % ( 2506 | xb_opt_ssh_opts, xb_opt_ssh_user, xb_opt_remote_host, src, dst 2507 | ) 2508 | 2509 | _say("Pulling %s from remote host %s:%s" % (src, xb_opt_remote_host, dst)) 2510 | 2511 | if not xb_opt_debug: 2512 | FNULL = open(os.devnull, 'w') 2513 | p_scp = Popen(run_cmd, shell=True, stdout=FNULL, stderr=STDOUT) 2514 | else: 2515 | p_scp = Popen(run_cmd, shell=True) 2516 | 2517 | r = p_scp.poll() 2518 | while r is None: 2519 | time.sleep(5) 2520 | r = p_scp.poll() 2521 | 2522 | if FNULL is not None: 2523 | FNULL.close() 2524 | 2525 | if r != 0: 2526 | _error("Pulling ", src, " from remote ", dst, " failed.") 2527 | _error("Push command was: ", run_cmd) 2528 | _error("rsync returned exit code was: ", str(r)) 2529 | _exit_code(XB_EXIT_EXTRACT_FAIL) 2530 | return False 2531 | 2532 | return True 2533 | 2534 | def db_connect(): 2535 | global xb_mysqldb 2536 | 2537 | params = dict() 2538 | 2539 | if xb_opt_mysql_user is not None: 2540 | params['user'] = xb_opt_mysql_user 2541 | 2542 | if xb_opt_mysql_pass is not None: 2543 | params['passwd'] = xb_opt_mysql_pass 2544 | 2545 | params['db'] = '' 2546 | params['port'] = xb_opt_mysql_port 2547 | 2548 | if xb_opt_mysql_cnf is not None: 2549 | params['read_default_file'] = xb_opt_mysql_cnf 2550 | params['read_default_group'] = 'client' 2551 | 2552 | try: 2553 | xb_mysqldb = MySQLdb.connect(xb_opt_mysql_host, **params) 2554 | 2555 | # MySQLdb for some reason has autoccommit off by default 2556 | xb_mysqldb.autocommit(True) 2557 | except MySQLdb.Error, e: 2558 | _error("Error ", e.args[0], ": ", e.args[1]) 2559 | return False 2560 | 2561 | return xb_mysqldb 2562 | 2563 | def db_close(): 2564 | global xb_mysqldb 2565 | 2566 | if xb_mysqldb is not None: 2567 | xb_mysqldb.close() 2568 | xb_mysqldb = None 2569 | 2570 | def init(): 2571 | """Validate and populate all options/configuration values""" 2572 | 2573 | global xb_opt_config 2574 | global xb_opt_config_section 2575 | global xb_opt_mysql_host 2576 | global xb_opt_mysql_user 2577 | global xb_opt_mysql_pass 2578 | global xb_opt_mysql_port 2579 | global xb_opt_mysql_sock 2580 | global xb_opt_mysql_cnf 2581 | global xb_opt_stor_dir 2582 | global xb_opt_work_dir 2583 | global xb_opt_retention_binlogs 2584 | global xb_opt_compress 2585 | global xb_opt_compress_with 2586 | global xb_opt_apply_log 2587 | global xb_opt_prepare_memory 2588 | global xb_opt_retention_sets 2589 | global xb_opt_retention_months 2590 | global xb_opt_retention_weeks 2591 | global xb_opt_debug 2592 | global xb_opt_quiet 2593 | global xb_opt_status_format 2594 | global xb_opt_command 2595 | global xb_opt_restore_backup 2596 | global xb_opt_restore_dir 2597 | global xb_opt_remote_stor_dir 2598 | global xb_opt_remote_host 2599 | global xb_opt_remote_script 2600 | global xb_opt_remote_push_only 2601 | global xb_opt_remote_nc_port 2602 | global xb_opt_ssh_opts 2603 | global xb_opt_ssh_user 2604 | global xb_opt_notify_by_email 2605 | global xb_opt_notify_on_success 2606 | global xb_opt_meta_item 2607 | global xb_opt_wipeout 2608 | global xb_opt_first_binlog 2609 | global xb_opt_binlog_from_master 2610 | global xb_opt_binlog_binary 2611 | global xb_opt_encrypt 2612 | global xb_opt_encrypt_key_file 2613 | global xb_opt_extra_ibx_options 2614 | global xb_opt_purge_bitmaps 2615 | 2616 | xb_opt_config = "/etc/%s.cnf" % XB_BIN_NAME 2617 | 2618 | if not os.path.isfile(xb_opt_config): 2619 | xb_opt_config = "%s/%s.cnf" % (xb_cwd, XB_BIN_NAME) 2620 | 2621 | xb_opt_config_section = XB_BIN_NAME 2622 | 2623 | xb_cfg = None 2624 | 2625 | _init_log_file("/tmp/%s-%s" % (xb_curdate, XB_LOG_NAME)) 2626 | 2627 | p_usage = "Usage: %prog [options] COMMAND" 2628 | p_desc = "Managed xtrabackup based backups." 2629 | p_epilog = """ 2630 | 2631 | Options here can also be specified on a filed called %s.cnf which will be 2632 | checked in this order: 2633 | 2634 | - on a file specified via the --config option 2635 | - /etc/pyxbackup.cnf 2636 | - on the same directory of the script 2637 | 2638 | Valid commands are: 2639 | 2640 | full: Execute full backups 2641 | incr: Execute incremental backups 2642 | list: List existing backups and additional information 2643 | status: Check status of last backup 2644 | apply-last: Prepare to the most recent backup 2645 | restore-set: Restore to a specific backup set 2646 | last-lsn: Print out to_lsn value of last backup for incremental use 2647 | wipeout: Cleanup all existing backups 2648 | 2649 | """ 2650 | p_epilog = p_epilog % XB_BIN_NAME 2651 | 2652 | parser = PyxOptParser(p_usage, version="%prog " + str(xb_version), 2653 | description=p_desc, epilog=p_epilog) 2654 | parser.add_option('-f', '--config', dest='config', type='string', 2655 | help='Path to config file to use, useful for multiple back locations') 2656 | parser.add_option('', '--config-section', dest='config_section', type='string', 2657 | help=('By default, config options are read from the %s section. ' 2658 | 'If you have multiple sections/profile in the configuration file ' 2659 | 'you can specify the section name, similar to mysql --defaults-group. ' 2660 | '(cli)') % XB_BIN_NAME) 2661 | parser.add_option('-u', '--mysql-user', dest='mysql_user', type='string', 2662 | help='MySQL server username') 2663 | parser.add_option('-p', '--mysql-pass', dest='mysql_pass', type='string', 2664 | help='MySQL server password') 2665 | parser.add_option('-H', '--mysql-host', dest='mysql_host', type='string', 2666 | help='MySQL server hostname/IP address') 2667 | parser.add_option('-P', '--mysql-port', dest='mysql_port', type='int', 2668 | help='MySQL server port, socket has precendence') 2669 | parser.add_option('-S', '--mysql-socket', dest='mysql_sock', type='string', 2670 | help='MySQL server path to socket file') 2671 | parser.add_option('-c', '--mysql-cnf', dest='mysql_cnf', type='string', 2672 | help=('Path to custom my.cnf, in case you want to pass this value to ' 2673 | 'innobackupex --defaults-file')) 2674 | parser.add_option('-s', '--stor-dir', dest='stor_dir', type='string', 2675 | help='Path to directory where backups are stored.') 2676 | parser.add_option('-w', '--work-dir', dest='work_dir', type='string', 2677 | help='Path to temporary backup work directory') 2678 | parser.add_option('-b', '--retention-binlogs', dest='retention_binlogs', type="int", 2679 | help='Binary log period retention, in days') 2680 | parser.add_option('', '--extra-ibx-options', dest='extra_ibx_options', type='string', 2681 | help=('Specify additional innobackupex options, make sure to ' 2682 | 'mind your quotes and avoid conflicts with --encrypt*, ' 2683 | '--compress, --remote-host - will think of better way to ' 2684 | 'handle this in the future!')) 2685 | parser.add_option('-z', '--compress', dest='compress', action="store_true", 2686 | help='Compress backups, by default with gzip, see -Z') 2687 | parser.add_option('-Z', '--compress-with', dest='compress_with', 2688 | help='Compress backup with binary, default gzip, options (gzip, qpress)') 2689 | parser.add_option('-M', '--notify-by-email', dest='notify_by_email', 2690 | help='Send failed backup notifications to this address(es)') 2691 | parser.add_option('', '--notify-on-success', dest='notify_on_success', 2692 | help='Send success backup notifications to this address(es)') 2693 | parser.add_option('-R', '--remote-stor-dir', dest='remote_stor_dir', 2694 | help=('When --remote-host is not empty, backups to that host will be ' 2695 | 'streamed to this directory, similar to --stor-dir')) 2696 | parser.add_option('-T', '--remote-host', dest='remote_host', 2697 | help='Stream backups to this remote host') 2698 | parser.add_option('-L', '--remote-push-only', dest='remote_push_only', action="store_true", 2699 | help=('Instructs xtrabackup that all backups will be pushed to ' 2700 | 'remote only, no local post processing')) 2701 | parser.add_option('-B', '--remote-script', dest='remote_script', 2702 | help=('When --remote-push-only is enabled, we need to specify the ' 2703 | 'path to this script on the remote server, default is xbackup.py')) 2704 | parser.add_option('', '--remote-nc-port', dest='remote_nc_port', 2705 | help=('When requesting to open a netcat port, this is the port number ' 2706 | 'to try with, can be a range separated with comma')) 2707 | parser.add_option('-C', '--ssh-opts', dest='ssh_opts', 2708 | help=('SSH options when streaming backups to remote host ' 2709 | 'i.e. -i /path/to/identity file')) 2710 | parser.add_option('-U', '--ssh-user', dest='ssh_user', 2711 | help='SSH account to user when streaming backups to remote host, default is root') 2712 | parser.add_option('-x', '--apply-log', dest='apply_log', action="store_true", 2713 | help='Verify backups with --apply-log, requires enough disk space on --workdir') 2714 | parser.add_option('-m', '--prepare-memory', dest='prepare_memory', type="int", 2715 | help='How much memory to use with innobackupex --use-memory in MB, default 128M') 2716 | parser.add_option('-o', '--status-format', dest='status_format', type="string", 2717 | help=('For status command, what output format, default=none, ' 2718 | 'possible values: none, nagios, zabbix (cli)')) 2719 | parser.add_option('-r', '--restore-backup', dest='restore_backup', type="string", 2720 | help=('With command restore-set, specify which backup to restore, ' 2721 | 'choose any from output of list command. Default is restore to last ' 2722 | 'successful backup. (cli)')) 2723 | parser.add_option('-e', '--restore-dir', dest='restore_dir', type="string", 2724 | help='With command restore, specify where to restore selected backup (cli)') 2725 | parser.add_option('-i', '--retention-sets', dest='retention_sets', 2726 | help='How many sets of combined full + incr to keep on storage, default 2') 2727 | parser.add_option('-j', '--retention-months', dest='retention_months', type="int", 2728 | help='How many rotated monthly backups to keep, default 0', 2729 | default=0) 2730 | parser.add_option('-k', '--retention-weeks', dest='retention_weeks', type="int", 2731 | help='How many rotated weekly backups to keep, default 0', 2732 | default=0) 2733 | parser.add_option('-t', '--meta-item', dest='meta_item', type="string", 2734 | help=('Query meta information about backups, used when backups ' 2735 | 'are push to remote location. Allows the script to query information ' 2736 | 'about backups stored remotely')) 2737 | parser.add_option('-n', '--first-binlog', dest='first_binlog', type="string", 2738 | help=('For binlog-stream, if the script cannot determine the oldest ' 2739 | 'binary log filename from the backups to maintain the list of files ' 2740 | 'to keep, we can specify it manually here')) 2741 | parser.add_option('', '--binlog-from-master', dest='binlog_from_master', action="store_true", 2742 | help=('For binlog-stream, when --slave-info is enabled on the backups ' 2743 | 'and you want to stream binary logs from the master instead ' 2744 | 'this tells the script to determine the correct binary log file name')) 2745 | parser.add_option('-l', '--binlog-binary', dest='binlog_binary', type="string", 2746 | help=('For binlog-stream, specify where the 5.6+ mysqlbinlog utility ' 2747 | 'is located')) 2748 | parser.add_option('-d', '--debug', dest='debug', action="store_true", 2749 | help='Enable debugging, more verbose output (cli)', 2750 | default=False) 2751 | parser.add_option('-q', '--quiet', dest='quiet', action="store_true", 2752 | help='Supress all messages errors except intended output i.e. list command (cli)', 2753 | default=False) 2754 | parser.add_option('-X', '--i-am-absolutely-sure-wipeout', dest='wipeout', action="store_true", 2755 | help='Confirm to **WIPEOUT** all backups with wipeout command! (cli)', 2756 | default=False) 2757 | parser.add_option('', '--encrypt', dest='encrypt', type="string", 2758 | help='Whether to encrypt backups on storage') 2759 | parser.add_option('', '--encrypt-key-file', dest='encrypt_key_file', type="string", 2760 | help=('Key file for encrypting/decrypting backups')) 2761 | parser.add_option('', '--purge-bitmaps', dest='purge_bitmaps', action="store_true", 2762 | help=('If Changed Page Tracking is enabled, should we automatically ' 2763 | 'purge bitmaps? Requires that a valid mysql-user and mysql-pass ' 2764 | 'with SUPER privieleges is specified.')) 2765 | 2766 | (options, args) = parser.parse_args() 2767 | 2768 | if options.debug: xb_opt_debug = True 2769 | if options.quiet: xb_opt_quiet = True 2770 | if options.wipeout: xb_opt_wipeout = True 2771 | 2772 | if xb_opt_quiet and xb_opt_debug: 2773 | _die("--debug and --quiet are mutually exclusive") 2774 | 2775 | if options.config: 2776 | xb_opt_config = os.path.realpath(options.config) 2777 | if not os.path.isfile(xb_opt_config): 2778 | _die("The specified configuration file %s " % options.config, 2779 | "does not exist or is not readable!") 2780 | else: 2781 | _say("Using config file %s" % xb_opt_config) 2782 | 2783 | if options.config_section: xb_opt_config_section = options.config_section 2784 | 2785 | if os.path.isfile(xb_opt_config): 2786 | xb_cfg = ConfigParser() 2787 | xb_cfg.read(xb_opt_config) 2788 | 2789 | if xb_cfg.has_option(xb_opt_config_section, 'mysql_host'): 2790 | xb_opt_mysql_host = xb_cfg.get(xb_opt_config_section, 'mysql_host') 2791 | 2792 | if xb_cfg.has_option(xb_opt_config_section, 'mysql_user'): 2793 | xb_opt_mysql_user = xb_cfg.get(xb_opt_config_section, 'mysql_user') 2794 | 2795 | if xb_cfg.has_option(xb_opt_config_section, 'mysql_pass'): 2796 | xb_opt_mysql_pass = xb_cfg.get(xb_opt_config_section, 'mysql_pass') 2797 | 2798 | if xb_cfg.has_option(xb_opt_config_section, 'mysql_port'): 2799 | xb_opt_mysql_port = int(xb_cfg.get(xb_opt_config_section, 'mysql_port')) 2800 | 2801 | if xb_cfg.has_option(xb_opt_config_section, 'mysql_sock'): 2802 | xb_opt_mysql_sock = xb_cfg.get(xb_opt_config_section, 'mysql_sock') 2803 | 2804 | if xb_cfg.has_option(xb_opt_config_section, 'mysql_cnf'): 2805 | xb_opt_mysql_cnf = xb_cfg.get(xb_opt_config_section, 'mysql_cnf') 2806 | 2807 | if xb_cfg.has_option(xb_opt_config_section, 'stor_dir'): 2808 | xb_opt_stor_dir = xb_cfg.get(xb_opt_config_section, 'stor_dir').rstrip('/') 2809 | 2810 | if xb_cfg.has_option(xb_opt_config_section, 'work_dir'): 2811 | xb_opt_work_dir = xb_cfg.get(xb_opt_config_section, 'work_dir').rstrip('/') 2812 | 2813 | if xb_cfg.has_option(xb_opt_config_section, 'ssh_opts'): 2814 | xb_opt_ssh_opts = xb_cfg.get(xb_opt_config_section, 'ssh_opts') 2815 | 2816 | if xb_cfg.has_option(xb_opt_config_section, 'ssh_user'): 2817 | xb_opt_ssh_user = xb_cfg.get(xb_opt_config_section, 'ssh_user') 2818 | 2819 | if xb_cfg.has_option(xb_opt_config_section, 'remote_stor_dir'): 2820 | xb_opt_remote_stor_dir = xb_cfg.get(xb_opt_config_section, 'remote_stor_dir').rstrip('/') 2821 | 2822 | if xb_cfg.has_option(xb_opt_config_section, 'remote_host'): 2823 | xb_opt_remote_host = xb_cfg.get(xb_opt_config_section, 'remote_host') 2824 | 2825 | if xb_cfg.has_option(xb_opt_config_section, 'remote_script'): 2826 | xb_opt_remote_script = xb_cfg.get(xb_opt_config_section, 'remote_script') 2827 | 2828 | if xb_cfg.has_option(xb_opt_config_section, 'remote_push_only'): 2829 | xb_opt_remote_push_only = bool(int(xb_cfg.get(xb_opt_config_section, 'remote_push_only'))) 2830 | 2831 | if xb_cfg.has_option(xb_opt_config_section, 'remote_nc_port'): 2832 | if not _parse_port_param(xb_cfg.get(xb_opt_config_section, 'remote_nc_port')): 2833 | parser.error("The specified port (range) is not valid") 2834 | else: 2835 | xb_opt_remote_nc_port = xb_cfg.get(xb_opt_config_section, 'remote_nc_port') 2836 | 2837 | if xb_cfg.has_option(xb_opt_config_section, 'retention_binlogs'): 2838 | xb_opt_retention_binlogs = int(xb_cfg.get(xb_opt_config_section, 'retention_binlogs')) 2839 | 2840 | if xb_cfg.has_option(xb_opt_config_section, 'binlog_binary'): 2841 | xb_opt_binlog_binary = xb_cfg.get(xb_opt_config_section, 'binlog_binary') 2842 | 2843 | if xb_cfg.has_option(xb_opt_config_section, 'binlog_from_master'): 2844 | xb_opt_binlog_from_master = xb_cfg.get(xb_opt_config_section, 'binlog_from_master') 2845 | 2846 | if xb_cfg.has_option(xb_opt_config_section, 'compress'): 2847 | xb_opt_compress = bool(int(xb_cfg.get(xb_opt_config_section, 'compress'))) 2848 | 2849 | if xb_cfg.has_option(xb_opt_config_section, 'compress_with'): 2850 | xb_opt_compress_with = xb_cfg.get(xb_opt_config_section, 'compress_with') 2851 | 2852 | if xb_cfg.has_option(xb_opt_config_section, 'notify_by_email'): 2853 | xb_opt_notify_by_email = xb_cfg.get(xb_opt_config_section, 'notify_by_email') 2854 | 2855 | if xb_cfg.has_option(xb_opt_config_section, 'notify_on_success'): 2856 | xb_opt_notify_on_success = xb_cfg.get(xb_opt_config_section, 'notify_on_success') 2857 | 2858 | if xb_cfg.has_option(xb_opt_config_section, 'apply_log'): 2859 | xb_opt_apply_log = bool(int(xb_cfg.get(xb_opt_config_section, 'apply_log'))) 2860 | 2861 | if xb_cfg.has_option(xb_opt_config_section, 'prepare_memory'): 2862 | xb_opt_prepare_memory = int(xb_cfg.get(xb_opt_config_section, 'prepare_memory')) 2863 | 2864 | if xb_cfg.has_option(xb_opt_config_section, 'retention_sets'): 2865 | if int(xb_cfg.get(xb_opt_config_section, 'retention_sets')) > 0: 2866 | xb_opt_retention_sets = int(xb_cfg.get(xb_opt_config_section, 'retention_sets')) 2867 | 2868 | if xb_cfg.has_option(xb_opt_config_section, 'retention_months'): 2869 | if int(xb_cfg.get(xb_opt_config_section, 'retention_months')) > 0: 2870 | xb_opt_retention_months = int(xb_cfg.get(xb_opt_config_section, 'retention_months')) 2871 | 2872 | if xb_cfg.has_option(xb_opt_config_section, 'retention_weeks'): 2873 | if int(xb_cfg.get(xb_opt_config_section, 'retention_weeks')) > 0: 2874 | xb_opt_retention_weeks = int(xb_cfg.get(xb_opt_config_section, 'retention_weeks')) 2875 | 2876 | if xb_cfg.has_option(xb_opt_config_section, 'encrypt_key_file'): 2877 | xb_opt_encrypt_key_file = xb_cfg.get(xb_opt_config_section, 'encrypt_key_file') 2878 | 2879 | if xb_cfg.has_option(xb_opt_config_section, 'encrypt'): 2880 | xb_opt_encrypt = xb_cfg.get(xb_opt_config_section, 'encrypt') 2881 | 2882 | if xb_cfg.has_option(xb_opt_config_section, 'extra_ibx_options'): 2883 | xb_opt_extra_ibx_options = xb_cfg.get(xb_opt_config_section, 'extra_ibx_options') 2884 | 2885 | if xb_cfg.has_option(xb_opt_config_section, 'purge_bitmaps'): 2886 | xb_opt_purge_bitmaps = xb_cfg.get(xb_opt_config_section, 'purge_bitmaps') 2887 | 2888 | if options.mysql_user: xb_opt_mysql_user = options.mysql_user 2889 | if options.mysql_pass: xb_opt_mysql_pass = options.mysql_pass 2890 | if options.mysql_host: xb_opt_mysql_host = options.mysql_host 2891 | if options.mysql_port: xb_opt_mysql_port = options.mysql_port 2892 | if options.mysql_sock: xb_opt_mysql_sock = options.mysql_sock 2893 | if options.mysql_cnf: xb_opt_mysql_cnf = options.mysql_cnf 2894 | if options.stor_dir: xb_opt_stor_dir = options.stor_dir.rstrip('/') 2895 | if options.work_dir: xb_opt_work_dir = options.work_dir.rstrip('/') 2896 | if options.retention_binlogs: xb_opt_retention_binlogs = options.retention_binlogs 2897 | if options.compress: xb_opt_compress = options.compress 2898 | if options.compress_with: xb_opt_compress_with = options.compress_with 2899 | if options.notify_by_email: xb_opt_notify_by_email = options.notify_by_email 2900 | if options.notify_on_success: xb_opt_notify_on_success = options.notify_on_success 2901 | if options.first_binlog: xb_opt_first_binlog = options.first_binlog 2902 | if options.binlog_binary: xb_opt_binlog_binary = options.binlog_binary 2903 | if options.binlog_from_master: xb_opt_binlog_from_master = options.binlog_from_master 2904 | 2905 | if options.remote_stor_dir: xb_opt_remote_stor_dir = options.remote_stor_dir 2906 | if options.remote_host: xb_opt_remote_host = options.remote_host 2907 | if options.remote_script: xb_opt_remote_script = options.remote_script 2908 | if options.remote_push_only is not None: 2909 | xb_opt_remote_push_only = options.remote_push_only 2910 | 2911 | if options.remote_nc_port is not None and \ 2912 | not _parse_port_param(options.remote_nc_port): 2913 | parser.error("The specified port (range) is not valid") 2914 | else: 2915 | xb_opt_remote_nc_port = options.remote_nc_port 2916 | 2917 | if options.ssh_opts: xb_opt_ssh_opts = options.ssh_opts 2918 | if options.ssh_user: xb_opt_ssh_user = options.ssh_user 2919 | if options.meta_item: xb_opt_meta_item = options.meta_item 2920 | 2921 | if xb_opt_remote_host is not None and xb_opt_remote_stor_dir is None: 2922 | parser.error("Remote host specified but, remote store directory is empty") 2923 | 2924 | if options.apply_log: xb_opt_apply_log = options.apply_log 2925 | if options.prepare_memory: xb_opt_prepare_memory = options.prepare_memory 2926 | if options.status_format: xb_opt_status_format = options.status_format 2927 | if options.restore_backup is not None: 2928 | xb_opt_restore_backup = options.restore_backup 2929 | if options.restore_dir is not None: 2930 | xb_opt_restore_dir = options.restore_dir 2931 | if options.retention_sets and int(options.retention_sets) > 0: 2932 | xb_opt_retention_sets = int(options.retention_sets) 2933 | if options.retention_months > 0: 2934 | xb_opt_retention_months = int(options.retention_months) 2935 | if options.retention_weeks > 0: 2936 | xb_opt_retention_weeks = int(options.retention_weeks) 2937 | 2938 | if options.encrypt: xb_opt_encrypt = options.encrypt 2939 | if options.encrypt_key_file: xb_opt_encrypt_key_file = options.encrypt_key_file 2940 | if options.extra_ibx_options: xb_opt_extra_ibx_options = options.extra_ibx_options 2941 | if options.purge_bitmaps: xb_opt_purge_bitmaps = options.purge_bitmaps 2942 | 2943 | if xb_cfg: _debug('Found config file: ', xb_opt_config) 2944 | 2945 | cmds = [XB_CMD_FULL, XB_CMD_INCR, XB_CMD_LIST, XB_CMD_STAT, XB_CMD_PREP, 2946 | XB_CMD_APPL, XB_CMD_PRUNE, XB_CMD_META, XB_CMD_BINLOGS, XB_CMD_WIPE] 2947 | if len(args) >= 1 and args[0] not in cmds: 2948 | parser.error("Command not recognized, got '%s'. See more with --help" % args[0]) 2949 | elif len(args) <= 0: 2950 | parser.error("Command not specified. See more with --help") 2951 | else: 2952 | xb_opt_command = args[0] 2953 | 2954 | if xb_opt_remote_push_only and xb_opt_apply_log: 2955 | _die("--remote-push-only and --apply-log are mutually exclusive") 2956 | 2957 | if options.retention_sets is not None and options.retention_sets <= 0: 2958 | _die("Invalid value for retention sets, ", 2959 | "you should keep one or more backup sets!") 2960 | 2961 | if xb_opt_encrypt and not os.path.isfile(xb_opt_encrypt_key_file): 2962 | _die("The specified key file does not exist!") 2963 | 2964 | if xb_opt_encrypt and xb_opt_compress and xb_opt_compress_with == 'gzip': 2965 | _die("GZIP compression + encryption is not supported ", 2966 | "at the moment. Please use --compress-with=qpress instead.") 2967 | 2968 | if xb_opt_encrypt and not xb_opt_compress: 2969 | _die("Encryption requires compression for now, support for ", 2970 | "uncompressed encrypted backup will be added in the future") 2971 | 2972 | if xb_opt_command in [XB_CMD_FULL, XB_CMD_INCR, XB_CMD_PREP, XB_CMD_APPL]: 2973 | _check_binary('innobackupex') 2974 | _check_binary('xtrabackup') 2975 | 2976 | if xb_opt_remote_nc_port_min: 2977 | _check_binary('nc') 2978 | _check_binary('netstat') 2979 | 2980 | if xb_opt_encrypt or xb_opt_encrypt_key_file: 2981 | _check_binary('xbcrypt') 2982 | 2983 | if xb_opt_compress: 2984 | _check_binary('xbstream') 2985 | 2986 | if xb_opt_compress_with == 'qpress': 2987 | _check_binary('qpress') 2988 | 2989 | # store xtrabackup version numbers 2990 | _xb_version() 2991 | 2992 | # we test email delivery beforehand to make sure it works 2993 | # this will happen only once as long as the sentinel file exists 2994 | # i.e. STOR_DIR/pyxbackup_mail_ok 2995 | mail_status_file = "%s/%s_mail_ok" % (xb_opt_stor_dir, XB_BIN_NAME) 2996 | if (xb_opt_notify_by_email or xb_opt_notify_on_success) and \ 2997 | not os.path.isfile(mail_status_file): 2998 | mail_message = "This is a test message from %s@%s, please ignore." % ( 2999 | xb_user, xb_hostname) 3000 | mail_subject = "pyxbackup Test Mail" 3001 | mail_to = xb_opt_notify_by_email \ 3002 | if xb_opt_notify_by_email else xb_opt_notify_on_success 3003 | 3004 | _say("Mail has not been tested, sending initial test mail.") 3005 | 3006 | if _notify_by_email(mail_subject, mail_message, mail_to): 3007 | open(mail_status_file, 'a').close() 3008 | 3009 | if xb_opt_debug: 3010 | _debug("Supplied options:") 3011 | for x, v in options.__dict__.items(): 3012 | _debug(("\t%s: %s" % (x, globals()['xb_opt_' + str(x)]))) 3013 | _debug("\tcommand: %s" % xb_opt_command) 3014 | 3015 | def check_dirs(): 3016 | """Check and create required directories if they do not exist yet""" 3017 | 3018 | global xb_stor_full 3019 | global xb_stor_incr 3020 | global xb_stor_weekly 3021 | global xb_stor_monthly 3022 | global xb_stor_binlogs 3023 | 3024 | if not os.path.isdir(xb_opt_stor_dir): 3025 | _die("The store directory \"%s\" is not a valid directory" % xb_opt_stor_dir) 3026 | 3027 | if not os.path.isdir(xb_opt_work_dir): 3028 | _die("The work directory \"%s\" is not a valid directory" % xb_opt_work_dir) 3029 | 3030 | xb_stor_full = xb_opt_stor_dir + '/full' 3031 | xb_stor_incr = xb_opt_stor_dir + '/incr' 3032 | xb_stor_weekly = xb_opt_stor_dir + '/weekly' 3033 | xb_stor_monthly = xb_opt_stor_dir + '/monthly' 3034 | xb_stor_binlogs = xb_opt_stor_dir + '/binlogs' 3035 | 3036 | if not os.path.isdir(xb_stor_full): os.mkdir(xb_stor_full, 0755) 3037 | if not os.path.isdir(xb_stor_incr): os.mkdir(xb_stor_incr, 0755) 3038 | if not os.path.isdir(xb_stor_weekly): os.mkdir(xb_stor_weekly, 0755) 3039 | if not os.path.isdir(xb_stor_monthly): os.mkdir(xb_stor_monthly, 0755) 3040 | if not os.path.isdir(xb_stor_binlogs): os.mkdir(xb_stor_binlogs, 0755) 3041 | 3042 | def list_backups(): 3043 | """List all valid backups inside the store directory""" 3044 | 3045 | global xb_last_full 3046 | global xb_last_incr 3047 | global xb_full_list 3048 | global xb_incr_list 3049 | global xb_weekly_list 3050 | global xb_monthly_list 3051 | global xb_last_backup 3052 | global xb_last_backup_is 3053 | 3054 | l = os.listdir(xb_stor_full) 3055 | if len(l) <= 0 and xb_opt_command == XB_CMD_INCR and not xb_opt_remote_push_only: 3056 | _exit_code(XB_EXIT_NO_FULL) 3057 | _die("There is no available full backup for incremental from ", 3058 | xb_stor_full) 3059 | 3060 | l.sort() 3061 | l.reverse() 3062 | xb_full_list = [] 3063 | unrecognized_backups = False 3064 | 3065 | for d in l: 3066 | _debug("Checking full directory ", os.path.join(xb_stor_full, d)) 3067 | if os.path.isfile(os.path.join(xb_stor_full, d)): 3068 | _say(os.path.join(xb_stor_full, d), " is not recognized as backup") 3069 | unrecognized_backups = True 3070 | continue 3071 | 3072 | if not os.path.isfile(os.path.join(xb_stor_full, d, XB_TAG_FILE)): 3073 | _debug("Full backup ", os.path.join(xb_stor_full, d), 3074 | " is not recognized as full") 3075 | unrecognized_backups = True 3076 | continue 3077 | 3078 | if not xb_last_full: xb_last_full = d 3079 | xb_full_list.append(d) 3080 | 3081 | xb_incr_list = dict() 3082 | 3083 | l = os.listdir(xb_stor_incr) 3084 | if len(l) > 0: 3085 | for d in xb_full_list: 3086 | if not os.path.isdir(os.path.join(xb_stor_incr, d)): 3087 | continue 3088 | 3089 | i = os.listdir(os.path.join(xb_stor_incr, d)) 3090 | if len(i) <= 0: 3091 | xb_incr_list[d] = None 3092 | continue 3093 | else: 3094 | i.sort() 3095 | i.reverse() 3096 | 3097 | if d not in xb_full_list: 3098 | _debug("A group of incremental backup from the folder ", 3099 | os.path.join(xb_stor_incr, d), " has no parent backup from ", 3100 | xb_stor_full) 3101 | 3102 | # We iterate over a copy of the list, otherwise we lose reference 3103 | # to the list in case the first condition is hit i.e. invalid backup 3104 | for r in i[:]: 3105 | if not os.path.isfile(os.path.join(xb_stor_incr, d, r, XB_TAG_FILE)): 3106 | _debug("Incremental backup ", os.path.join(xb_stor_incr, d, r), 3107 | " is not recognized as incremental") 3108 | unrecognized_backups = True 3109 | i.remove(r) 3110 | elif d == xb_last_full and xb_last_incr is None: 3111 | _debug('I never hit this one!') 3112 | xb_last_incr = r 3113 | xb_last_backup = xb_last_incr 3114 | xb_last_backup_is = XB_CMD_INCR 3115 | 3116 | xb_incr_list[d] = i 3117 | 3118 | 3119 | if xb_last_backup is None: 3120 | xb_last_backup = xb_last_full 3121 | xb_last_backup_is = XB_CMD_FULL 3122 | 3123 | _debug("Full list: ", str(xb_full_list)) 3124 | _debug("Last full: ", xb_last_full) 3125 | if xb_last_incr: 3126 | _debug("Incr list: ", str(xb_incr_list)) 3127 | _debug("Last incr: ", xb_last_incr) 3128 | 3129 | l = os.listdir(xb_stor_weekly) 3130 | if len(l) > 0: 3131 | for d in l: 3132 | if not os.path.isdir(os.path.join(xb_stor_weekly, d)): 3133 | _debug("%s is not recognized as backup" % d) 3134 | unrecognized_backups = True 3135 | continue 3136 | 3137 | if not os.path.isdir(os.path.join(xb_stor_weekly, d, 'full')): 3138 | _debug("%s is not recognized as weekly backup" % d) 3139 | unrecognized_backups = True 3140 | continue 3141 | 3142 | if xb_weekly_list is None: xb_weekly_list = [] 3143 | xb_weekly_list.append(d) 3144 | 3145 | _debug("Weekly list: %s" % str(xb_weekly_list)) 3146 | 3147 | l = os.listdir(xb_stor_monthly) 3148 | if len(l) > 0: 3149 | for d in l: 3150 | if not os.path.isdir(os.path.join(xb_stor_monthly, d)): 3151 | _debug("%s is not recognized as backup" % d) 3152 | unrecognized_backups = True 3153 | continue 3154 | 3155 | if not os.path.isdir(os.path.join(xb_stor_monthly, d, 'full')): 3156 | _debug("%s is not recognized as monthly backup" % d) 3157 | unrecognized_backups = True 3158 | continue 3159 | 3160 | if xb_monthly_list is None: xb_monthly_list = [] 3161 | xb_monthly_list.append(d) 3162 | 3163 | _debug("Monthly list: %s" % str(xb_monthly_list)) 3164 | 3165 | list_binlogs() 3166 | 3167 | if unrecognized_backups == True: 3168 | _warn("Some files inside %s were not recognized " % xb_opt_stor_dir, 3169 | "as either complete or an actual backup directory.") 3170 | if xb_opt_debug: 3171 | _warn("Please review the files/folders above that are marked ", 3172 | "**not recognized**") 3173 | else: 3174 | _warn("To get a list of these files, please run the list command ", 3175 | "with --debug option specified.") 3176 | _warn("If these files are not needed, you can remove them from the ", 3177 | "filesystem to free up some disk space safely.") 3178 | 3179 | def list_binlogs(): 3180 | global xb_first_binlog 3181 | global xb_last_binlog 3182 | global xb_binlogs_list 3183 | global xb_binlog_name 3184 | 3185 | xb_binlogs_list = None 3186 | 3187 | l = os.listdir(xb_stor_binlogs) 3188 | if len(l) > 0: 3189 | for d in l: 3190 | f = os.path.join(xb_stor_binlogs, d) 3191 | if not os.path.isfile(f): 3192 | _debug("%s is not a file, skipping" % d) 3193 | continue 3194 | 3195 | # skip the magic number check if the name matches 3196 | # sort of an optimization to skip opening each file 3197 | # if you have thousands of binary logs 3198 | if xb_binlog_name and xb_binlog_name == d[0:-7]: 3199 | _debug("%s matches binary log name, appending" % d) 3200 | # we check the magic number for the binary log to validate 3201 | elif open(f, 'rb').read(4) != '\xfebin': 3202 | _debug("%s is not a valid binary log, skipping" % d) 3203 | continue 3204 | elif xb_binlog_name is None: 3205 | xb_binlog_name = d[0:-7] 3206 | 3207 | if xb_binlogs_list is None: xb_binlogs_list = [] 3208 | xb_binlogs_list.append(d) 3209 | 3210 | if xb_binlogs_list is not None: 3211 | xb_binlogs_list.sort() 3212 | _debug("Binary logs list: %s" % str(xb_binlogs_list)) 3213 | xb_first_binlog = xb_binlogs_list[0] 3214 | xb_last_binlog = xb_binlogs_list[len(xb_binlogs_list)-1] 3215 | 3216 | # http://stackoverflow.com/questions/1857346/\ 3217 | # python-optparse-how-to-include-additional-info-in-usage-output 3218 | class PyxOptParser(OptionParser): 3219 | def format_epilog(self, formatter): 3220 | return self.epilog 3221 | 3222 | if __name__ == "__main__": 3223 | try: 3224 | signal.signal(signal.SIGTERM, _sigterm_handler) 3225 | xb_curdate = date(time.time(), '%Y_%m_%d-%H_%M_%S') 3226 | xb_cwd = os.path.dirname(os.path.realpath(__file__)) 3227 | xb_hostname = os.uname()[1] 3228 | xb_user = pwd.getpwuid(os.getuid())[0] 3229 | 3230 | dt = datetime.strptime(xb_curdate, '%Y_%m_%d-%H_%M_%S') 3231 | if dt.weekday() == 6: 3232 | xb_is_last_day_of_week = True 3233 | 3234 | if calendar.monthrange(dt.year, dt.month)[1] == dt.day: 3235 | xb_is_last_day_of_month = True 3236 | 3237 | init() 3238 | check_dirs() 3239 | if xb_opt_command not in cmd_no_log: 3240 | # Initially our log file is created in /tmp/ until we can validate and 3241 | # make sure we can write to xb_opt_work_dir 3242 | _init_log_file("%s/%s-%s" % (xb_opt_work_dir, xb_curdate, XB_LOG_NAME)) 3243 | 3244 | list_backups() 3245 | os.chdir(xb_opt_work_dir) 3246 | 3247 | XB_LCK_FILE = os.path.join(xb_opt_work_dir, "%s.lock" % XB_BIN_NAME) 3248 | if not _check_in_progress(): 3249 | _create_lock_file() 3250 | 3251 | if xb_opt_command == XB_CMD_FULL: 3252 | run_xb_full() 3253 | elif xb_opt_command == XB_CMD_INCR: 3254 | run_xb_incr() 3255 | elif xb_opt_command == XB_CMD_LIST: 3256 | run_xb_list() 3257 | elif xb_opt_command == XB_CMD_PREP: 3258 | run_xb_restore_set() 3259 | elif xb_opt_command == XB_CMD_APPL: 3260 | run_xb_apply_last() 3261 | elif xb_opt_command == XB_CMD_PRUNE: 3262 | prune_full_incr() 3263 | prune_weekly() 3264 | prune_monthly() 3265 | elif xb_opt_command == XB_CMD_META: 3266 | run_meta_query() 3267 | elif xb_opt_command == XB_CMD_BINLOGS: 3268 | run_binlog_stream() 3269 | elif xb_opt_command == XB_CMD_WIPE: 3270 | run_wipeout() 3271 | else: run_status() 3272 | 3273 | _destroy_lock_file() 3274 | 3275 | if os.path.isfile(xb_log_file): 3276 | if xb_opt_remote_host and xb_opt_command not in [XB_CMD_PREP, XB_CMD_APPL]: 3277 | _push_to_remote_scp(xb_log_file, "%s/" % xb_this_backup_remote.rstrip('/')) 3278 | _init_log_file(xb_log_file, True) 3279 | 3280 | if xb_log_fd is not None: 3281 | os.close(xb_log_fd) 3282 | 3283 | if xb_exit_code > 0 and xb_opt_notify_by_email: 3284 | _notify_by_email("MySQL backup script at %s has errors!" % xb_hostname) 3285 | elif xb_opt_notify_on_success and xb_opt_command in [XB_CMD_FULL, XB_CMD_INCR]: 3286 | _notify_by_email( 3287 | "MySQL backup script at %s completed successfully!" % xb_hostname, 3288 | xb_backup_summary, xb_opt_notify_on_success) 3289 | 3290 | sys.exit(xb_exit_code) 3291 | except Exception, e: 3292 | if xb_opt_notify_by_email: 3293 | _notify_by_email( 3294 | "MySQL backup script at %s exception!" % xb_hostname, 3295 | traceback.format_exc()) 3296 | 3297 | if xb_exit_code > 0: 3298 | sys.exit(xb_exit_code) 3299 | if xb_opt_debug: traceback.print_exc() 3300 | else: 3301 | _error("An uncaught exception error has occurred!") 3302 | traceback.print_exc() 3303 | 3304 | sys.exit(255) 3305 | 3306 | class PyxOptions(object): 3307 | config = None 3308 | config_section = None 3309 | stor_dir = '' 3310 | work_dir = '' 3311 | mysql_user = None 3312 | mysql_pass = None 3313 | mysql_host = 'localhost' 3314 | mysql_port = 3306 3315 | mysql_sock = '/tmp/mysql.sock' 3316 | mysql_cnf = None 3317 | retention_binlogs = False 3318 | compress = False 3319 | compress_with = 'gzip' 3320 | apply_log = False 3321 | prepare_memory = 128 3322 | retention_sets = 2 3323 | retention_months = 0 3324 | retention_weeks = 0 3325 | debug = False 3326 | quiet = False 3327 | status_format = None 3328 | command = 'status' 3329 | restore_backup = None 3330 | restore_dir = None 3331 | remote_stor_dir = None 3332 | remote_host = None 3333 | remote_push_only = None 3334 | remote_script = XB_BIN_NAME 3335 | remote_nc_port = 0 3336 | remote_nc_port_min = 0 3337 | remote_nc_port_max = 0 3338 | ssh_opts = '' 3339 | ssh_user = None 3340 | notify_by_email = None 3341 | notify_on_success = None 3342 | meta_item = None 3343 | wipeout = False 3344 | first_binlog = False 3345 | binlog_binary = None 3346 | binlog_from_master = False 3347 | encrypt = False 3348 | encrypt_key_file = None 3349 | extra_ibx_options = None 3350 | purge_bitmaps = None 3351 | 3352 | def __init__(self): 3353 | 3354 | _init_log_file("/tmp/%s-%s" % (xb_curdate, XB_LOG_NAME)) 3355 | 3356 | p_usage = "Usage: %prog [options] COMMAND" 3357 | p_desc = "Managed xtrabackup based backups." 3358 | p_epilog = ["\n" 3359 | "Options here can also be specified on a filed called %s.cnf \n" 3360 | "which will be checked in this order: \n\n" 3361 | "- on a file specified via the --config option\n" 3362 | "- /etc/pyxbackup.cnf\n" 3363 | "- on the same directory of the script \n\n" 3364 | "Valid commands are:\n\n" 3365 | "\tfull: Execute full backups\n" 3366 | "\tincr: Execute incremental backups\n" 3367 | "\tlist: List existing backups and additional information\n" 3368 | "\tstatus: Check status of last backup\n" 3369 | "\tapply-last: Prepare to the most recent backup\n" 3370 | "\trestore-set: Restore to a specific backup set\n" 3371 | "\tlast-lsn: Print out to_lsn value of last backup for incremental use\n" 3372 | "\twipeout: Cleanup all existing backups\n"] 3373 | p_epilog = p_epilog % XB_BIN_NAME 3374 | 3375 | parser = PyxOptParser(p_usage, version="%prog " + str(xb_version), 3376 | description=p_desc, epilog=p_epilog) 3377 | parser.add_option('-f', '--config', dest='config', type='string', 3378 | help='Path to config file to use, useful for multiple back locations') 3379 | parser.add_option('', '--config-section', dest='config_section', type='string', 3380 | help=('By default, config options are read from the %s section. ' 3381 | 'If you have multiple sections/profile in the configuration file ' 3382 | 'you can specify the section name, similar to mysql --defaults-group. ' 3383 | '(cli)') % XB_BIN_NAME) 3384 | parser.add_option('-u', '--mysql-user', dest='mysql_user', type='string', 3385 | help='MySQL server username') 3386 | parser.add_option('-p', '--mysql-pass', dest='mysql_pass', type='string', 3387 | help='MySQL server password') 3388 | parser.add_option('-H', '--mysql-host', dest='mysql_host', type='string', 3389 | help='MySQL server hostname/IP address') 3390 | parser.add_option('-P', '--mysql-port', dest='mysql_port', type='int', 3391 | help='MySQL server port, socket has precendence') 3392 | parser.add_option('-S', '--mysql-socket', dest='mysql_sock', type='string', 3393 | help='MySQL server path to socket file') 3394 | parser.add_option('-c', '--mysql-cnf', dest='mysql_cnf', type='string', 3395 | help=('Path to custom my.cnf, in case you want to pass this value to ' 3396 | 'innobackupex --defaults-file')) 3397 | parser.add_option('-s', '--stor-dir', dest='stor_dir', type='string', 3398 | help='Path to directory where backups are stored.') 3399 | parser.add_option('-w', '--work-dir', dest='work_dir', type='string', 3400 | help='Path to temporary backup work directory') 3401 | parser.add_option('-b', '--retention-binlogs', dest='retention_binlogs', type="int", 3402 | help='Binary log period retention, in days') 3403 | parser.add_option('', '--extra-ibx-options', dest='extra_ibx_options', type='string', 3404 | help=('Specify additional innobackupex options, make sure to ' 3405 | 'mind your quotes and avoid conflicts with --encrypt*, ' 3406 | '--compress, --remote-host - will think of better way to ' 3407 | 'handle this in the future!')) 3408 | parser.add_option('-z', '--compress', dest='compress', action="store_true", 3409 | help='Compress backups, by default with gzip, see -Z') 3410 | parser.add_option('-Z', '--compress-with', dest='compress_with', 3411 | help='Compress backup with binary, default gzip, options (gzip, qpress)') 3412 | parser.add_option('-M', '--notify-by-email', dest='notify_by_email', 3413 | help='Send failed backup notifications to this address(es)') 3414 | parser.add_option('', '--notify-on-success', dest='notify_on_success', 3415 | help='Send success backup notifications to this address(es)') 3416 | parser.add_option('-R', '--remote-stor-dir', dest='remote_stor_dir', 3417 | help=('When --remote-host is not empty, backups to that host will be ' 3418 | 'streamed to this directory, similar to --stor-dir')) 3419 | parser.add_option('-T', '--remote-host', dest='remote_host', 3420 | help='Stream backups to this remote host') 3421 | parser.add_option('-L', '--remote-push-only', dest='remote_push_only', action="store_true", 3422 | help=('Instructs xtrabackup that all backups will be pushed to ' 3423 | 'remote only, no local post processing')) 3424 | parser.add_option('-B', '--remote-script', dest='remote_script', 3425 | help=('When --remote-push-only is enabled, we need to specify the ' 3426 | 'path to this script on the remote server, default is xbackup.py')) 3427 | parser.add_option('', '--remote-nc-port', dest='remote_nc_port', 3428 | help=('When requesting to open a netcat port, this is the port number ' 3429 | 'to try with, can be a range separated with comma')) 3430 | parser.add_option('-C', '--ssh-opts', dest='ssh_opts', 3431 | help=('SSH options when streaming backups to remote host ' 3432 | 'i.e. -i /path/to/identity file')) 3433 | parser.add_option('-U', '--ssh-user', dest='ssh_user', 3434 | help='SSH account to user when streaming backups to remote host, default is root') 3435 | parser.add_option('-x', '--apply-log', dest='apply_log', action="store_true", 3436 | help='Verify backups with --apply-log, requires enough disk space on --workdir') 3437 | parser.add_option('-m', '--prepare-memory', dest='prepare_memory', type="int", 3438 | help='How much memory to use with innobackupex --use-memory in MB, default 128M') 3439 | parser.add_option('-o', '--status-format', dest='status_format', type="string", 3440 | help=('For status command, what output format, default=none, ' 3441 | 'possible values: none, nagios, zabbix (cli)')) 3442 | parser.add_option('-r', '--restore-backup', dest='restore_backup', type="string", 3443 | help=('With command restore-set, specify which backup to restore, ' 3444 | 'choose any from output of list command. Default is restore to last ' 3445 | 'successful backup. (cli)')) 3446 | parser.add_option('-e', '--restore-dir', dest='restore_dir', type="string", 3447 | help='With command restore, specify where to restore selected backup (cli)') 3448 | parser.add_option('-i', '--retention-sets', dest='retention_sets', 3449 | help='How many sets of combined full + incr to keep on storage, default 2') 3450 | parser.add_option('-j', '--retention-months', dest='retention_months', type="int", 3451 | help='How many rotated monthly backups to keep, default 0', 3452 | default=0) 3453 | parser.add_option('-k', '--retention-weeks', dest='retention_weeks', type="int", 3454 | help='How many rotated weekly backups to keep, default 0', 3455 | default=0) 3456 | parser.add_option('-t', '--meta-item', dest='meta_item', type="string", 3457 | help=('Query meta information about backups, used when backups ' 3458 | 'are push to remote location. Allows the script to query information ' 3459 | 'about backups stored remotely')) 3460 | parser.add_option('-n', '--first-binlog', dest='first_binlog', type="string", 3461 | help=('For binlog-stream, if the script cannot determine the oldest ' 3462 | 'binary log filename from the backups to maintain the list of files ' 3463 | 'to keep, we can specify it manually here')) 3464 | parser.add_option('', '--binlog-from-master', dest='binlog_from_master', action="store_true", 3465 | help=('For binlog-stream, when --slave-info is enabled on the backups ' 3466 | 'and you want to stream binary logs from the master instead ' 3467 | 'this tells the script to determine the correct binary log file name')) 3468 | parser.add_option('-l', '--binlog-binary', dest='binlog_binary', type="string", 3469 | help=('For binlog-stream, specify where the 5.6+ mysqlbinlog utility ' 3470 | 'is located')) 3471 | parser.add_option('-d', '--debug', dest='debug', action="store_true", 3472 | help='Enable debugging, more verbose output (cli)', 3473 | default=False) 3474 | parser.add_option('-q', '--quiet', dest='quiet', action="store_true", 3475 | help='Supress all messages errors except intended output i.e. list command (cli)', 3476 | default=False) 3477 | parser.add_option('-X', '--i-am-absolutely-sure-wipeout', dest='wipeout', action="store_true", 3478 | help='Confirm to **WIPEOUT** all backups with wipeout command! (cli)', 3479 | default=False) 3480 | parser.add_option('', '--encrypt', dest='encrypt', type="string", 3481 | help='Whether to encrypt backups on storage') 3482 | parser.add_option('', '--encrypt-key-file', dest='encrypt_key_file', type="string", 3483 | help=('Key file for encrypting/decrypting backups')) 3484 | parser.add_option('', '--purge-bitmaps', dest='purge_bitmaps', action="store_true", 3485 | help=('If Changed Page Tracking is enabled, should we automatically ' 3486 | 'purge bitmaps? Requires that a valid mysql-user and mysql-pass ' 3487 | 'with SUPER privieleges is specified.')) 3488 | 3489 | (options, args) = parser.parse_args() 3490 | 3491 | if options.debug: debug = True 3492 | if options.quiet: quiet = True 3493 | if options.wipeout: wipeout = True 3494 | 3495 | if quiet and debug: 3496 | _die("--debug and --quiet are mutually exclusive") 3497 | 3498 | config = "/etc/%s.cnf" % XB_BIN_NAME 3499 | if not os.path.isfile(config): 3500 | config = "%s/%s.cnf" % (xb_cwd, XB_BIN_NAME) 3501 | config_section = XB_BIN_NAME 3502 | 3503 | if options.config: 3504 | config = os.path.realpath(options.config) 3505 | if not os.path.isfile(config): 3506 | _die("The specified configuration file %s " % options.config, 3507 | "does not exist or is not readable!") 3508 | else: 3509 | _say("Using config file %s" % config) 3510 | 3511 | if options.config_section: config_section = options.config_section 3512 | cfg = self.read_config_file(config, config_section) 3513 | 3514 | if options.mysql_user: mysql_user = options.mysql_user 3515 | if options.mysql_pass: mysql_pass = options.mysql_pass 3516 | if options.mysql_host: mysql_host = options.mysql_host 3517 | if options.mysql_port: mysql_port = options.mysql_port 3518 | if options.mysql_sock: mysql_sock = options.mysql_sock 3519 | if options.mysql_cnf: mysql_cnf = options.mysql_cnf 3520 | if options.stor_dir: stor_dir = options.stor_dir.rstrip('/') 3521 | if options.work_dir: work_dir = options.work_dir.rstrip('/') 3522 | if options.retention_binlogs: retention_binlogs = options.retention_binlogs 3523 | if options.compress: compress = options.compress 3524 | if options.compress_with: compress_with = options.compress_with 3525 | if options.notify_by_email: notify_by_email = options.notify_by_email 3526 | if options.notify_on_success: notify_on_success = options.notify_on_success 3527 | if options.first_binlog: first_binlog = options.first_binlog 3528 | if options.binlog_binary: binlog_binary = options.binlog_binary 3529 | if options.binlog_from_master: binlog_from_master = options.binlog_from_master 3530 | 3531 | if options.remote_stor_dir: remote_stor_dir = options.remote_stor_dir 3532 | if options.remote_host: remote_host = options.remote_host 3533 | if options.remote_script: remote_script = options.remote_script 3534 | if options.remote_push_only is not None: 3535 | remote_push_only = options.remote_push_only 3536 | 3537 | if options.remote_nc_port is not None and \ 3538 | not _parse_port_param(options.remote_nc_port): 3539 | parser.error("The specified port (range) is not valid") 3540 | else: 3541 | remote_nc_port = options.remote_nc_port 3542 | 3543 | if options.ssh_opts: ssh_opts = options.ssh_opts 3544 | if options.ssh_user: ssh_user = options.ssh_user 3545 | if options.meta_item: meta_item = options.meta_item 3546 | 3547 | if remote_host is not None and remote_stor_dir is None: 3548 | parser.error("Remote host specified but, remote store directory is empty") 3549 | 3550 | if options.apply_log: apply_log = options.apply_log 3551 | if options.prepare_memory: prepare_memory = options.prepare_memory 3552 | if options.status_format: status_format = options.status_format 3553 | if options.restore_backup is not None: 3554 | restore_backup = options.restore_backup 3555 | if options.restore_dir is not None: 3556 | restore_dir = options.restore_dir 3557 | if options.retention_sets and int(options.retention_sets) > 0: 3558 | retention_sets = int(options.retention_sets) 3559 | if options.retention_months > 0: 3560 | retention_months = int(options.retention_months) 3561 | if options.retention_weeks > 0: 3562 | retention_weeks = int(options.retention_weeks) 3563 | 3564 | if options.encrypt: encrypt = options.encrypt 3565 | if options.encrypt_key_file: encrypt_key_file = options.encrypt_key_file 3566 | if options.extra_ibx_options: extra_ibx_options = options.extra_ibx_options 3567 | if options.purge_bitmaps: purge_bitmaps = options.purge_bitmaps 3568 | 3569 | if cfg: _debug('Found config file: ', config) 3570 | 3571 | cmds = [XB_CMD_FULL, XB_CMD_INCR, XB_CMD_LIST, XB_CMD_STAT, 3572 | XB_CMD_PREP, XB_CMD_APPL, XB_CMD_PRUNE, XB_CMD_META, 3573 | XB_CMD_BINLOGS, XB_CMD_WIPE] 3574 | if len(args) >= 1 and args[0] not in cmds: 3575 | parser.error("Command not recognized, got '%s'. See more with --help" % args[0]) 3576 | elif len(args) <= 0: 3577 | parser.error("Command not specified. See more with --help") 3578 | else: 3579 | command = args[0] 3580 | 3581 | if remote_push_only and apply_log: 3582 | _die("--remote-push-only and --apply-log are mutually exclusive") 3583 | 3584 | if options.retention_sets is not None and options.retention_sets <= 0: 3585 | _die("Invalid value for retention sets, ", 3586 | "you should keep one or more backup sets!") 3587 | 3588 | if encrypt and not os.path.isfile(encrypt_key_file): 3589 | _die("The specified key file does not exist!") 3590 | 3591 | if encrypt and compress and compress_with == 'gzip': 3592 | _die("GZIP compression + encryption is not supported ", 3593 | "at the moment. Please use --compress-with=qpress instead.") 3594 | 3595 | if encrypt and not compress: 3596 | _die("Encryption requires compression for now, support for ", 3597 | "uncompressed encrypted backup will be added in the future") 3598 | 3599 | if command in [XB_CMD_FULL, XB_CMD_INCR, XB_CMD_PREP, XB_CMD_APPL]: 3600 | _check_binary('innobackupex') 3601 | _check_binary('xtrabackup') 3602 | 3603 | if remote_nc_port_min: 3604 | _check_binary('nc') 3605 | _check_binary('netstat') 3606 | 3607 | # store xtrabackup version numbers 3608 | _xb_version() 3609 | 3610 | # we test email delivery beforehand to make sure it works 3611 | # this will happen only once as long as the sentinel file exists 3612 | # i.e. STOR_DIR/pyxbackup_mail_ok 3613 | mail_status_file = "%s/%s_mail_ok" % (stor_dir, XB_BIN_NAME) 3614 | if (notify_by_email or notify_on_success) and \ 3615 | not os.path.isfile(mail_status_file): 3616 | mail_message = "This is a test message from %s@%s, please ignore." % ( 3617 | xb_user, xb_hostname) 3618 | mail_subject = "pyxbackup Test Mail" 3619 | mail_to = notify_by_email \ 3620 | if notify_by_email else notify_on_success 3621 | 3622 | _say("Mail has not been tested, sending initial test mail.") 3623 | 3624 | if _notify_by_email(mail_subject, mail_message, mail_to): 3625 | open(mail_status_file, 'a').close() 3626 | 3627 | if debug: 3628 | _debug("Supplied options:") 3629 | for x, v in options.__dict__.items(): 3630 | _debug(("\t%s: %s" % (x, globals()['' + str(x)]))) 3631 | _debug("\tcommand: %s" % command) 3632 | 3633 | 3634 | def read_config_file(cfg_file, config_section): 3635 | cfg = ConfigParser() 3636 | cfg.read(config) 3637 | 3638 | if cfg.has_option(config_section, 'mysql_host'): 3639 | mysql_host = cfg.get(config_section, 'mysql_host') 3640 | 3641 | if cfg.has_option(config_section, 'mysql_user'): 3642 | mysql_user = cfg.get(config_section, 'mysql_user') 3643 | 3644 | if cfg.has_option(config_section, 'mysql_pass'): 3645 | mysql_pass = cfg.get(config_section, 'mysql_pass') 3646 | 3647 | if cfg.has_option(config_section, 'mysql_port'): 3648 | mysql_port = int(cfg.get(config_section, 'mysql_port')) 3649 | 3650 | if cfg.has_option(config_section, 'mysql_sock'): 3651 | mysql_sock = cfg.get(config_section, 'mysql_sock') 3652 | 3653 | if cfg.has_option(config_section, 'mysql_cnf'): 3654 | mysql_cnf = cfg.get(config_section, 'mysql_cnf') 3655 | 3656 | if cfg.has_option(config_section, 'stor_dir'): 3657 | stor_dir = cfg.get(config_section, 'stor_dir').rstrip('/') 3658 | 3659 | if cfg.has_option(config_section, 'work_dir'): 3660 | work_dir = cfg.get(config_section, 'work_dir').rstrip('/') 3661 | 3662 | if cfg.has_option(config_section, 'ssh_opts'): 3663 | ssh_opts = cfg.get(config_section, 'ssh_opts') 3664 | 3665 | if cfg.has_option(config_section, 'ssh_user'): 3666 | ssh_user = cfg.get(config_section, 'ssh_user') 3667 | 3668 | if cfg.has_option(config_section, 'remote_stor_dir'): 3669 | remote_stor_dir = cfg.get(config_section, 'remote_stor_dir').rstrip('/') 3670 | 3671 | if cfg.has_option(config_section, 'remote_host'): 3672 | remote_host = cfg.get(config_section, 'remote_host') 3673 | 3674 | if cfg.has_option(config_section, 'remote_script'): 3675 | remote_script = cfg.get(config_section, 'remote_script') 3676 | 3677 | if cfg.has_option(config_section, 'remote_push_only'): 3678 | remote_push_only = bool(int(cfg.get(config_section, 'remote_push_only'))) 3679 | 3680 | if cfg.has_option(config_section, 'remote_nc_port'): 3681 | if not self.parse_port(cfg.get(config_section, 'remote_nc_port')): 3682 | _die("The specified port (range) is not valid") 3683 | else: 3684 | remote_nc_port = cfg.get(config_section, 'remote_nc_port') 3685 | 3686 | if cfg.has_option(config_section, 'retention_binlogs'): 3687 | retention_binlogs = int(cfg.get(config_section, 'retention_binlogs')) 3688 | 3689 | if cfg.has_option(config_section, 'binlog_binary'): 3690 | binlog_binary = cfg.get(config_section, 'binlog_binary') 3691 | 3692 | if cfg.has_option(config_section, 'binlog_from_master'): 3693 | binlog_from_master = cfg.get(config_section, 'binlog_from_master') 3694 | 3695 | if cfg.has_option(config_section, 'compress'): 3696 | compress = bool(int(cfg.get(config_section, 'compress'))) 3697 | 3698 | if cfg.has_option(config_section, 'compress_with'): 3699 | compress_with = cfg.get(config_section, 'compress_with') 3700 | 3701 | if cfg.has_option(config_section, 'notify_by_email'): 3702 | notify_by_email = cfg.get(config_section, 'notify_by_email') 3703 | 3704 | if cfg.has_option(config_section, 'notify_on_success'): 3705 | notify_on_success = cfg.get(config_section, 'notify_on_success') 3706 | 3707 | if cfg.has_option(config_section, 'apply_log'): 3708 | apply_log = bool(int(cfg.get(config_section, 'apply_log'))) 3709 | 3710 | if cfg.has_option(config_section, 'prepare_memory'): 3711 | prepare_memory = int(cfg.get(config_section, 'prepare_memory')) 3712 | 3713 | if cfg.has_option(config_section, 'retention_sets'): 3714 | if int(cfg.get(config_section, 'retention_sets')) > 0: 3715 | retention_sets = int(cfg.get(config_section, 'retention_sets')) 3716 | 3717 | if cfg.has_option(config_section, 'retention_months'): 3718 | if int(cfg.get(config_section, 'retention_months')) > 0: 3719 | retention_months = int(cfg.get(config_section, 'retention_months')) 3720 | 3721 | if cfg.has_option(config_section, 'retention_weeks'): 3722 | if int(cfg.get(config_section, 'retention_weeks')) > 0: 3723 | retention_weeks = int(cfg.get(config_section, 'retention_weeks')) 3724 | 3725 | if cfg.has_option(config_section, 'encrypt_key_file'): 3726 | encrypt_key_file = cfg.get(config_section, 'encrypt_key_file') 3727 | 3728 | if cfg.has_option(config_section, 'encrypt'): 3729 | encrypt = cfg.get(config_section, 'encrypt') 3730 | 3731 | if cfg.has_option(config_section, 'extra_ibx_options'): 3732 | extra_ibx_options = cfg.get(config_section, 'extra_ibx_options') 3733 | 3734 | if cfg.has_option(config_section, 'purge_bitmaps'): 3735 | purge_bitmaps = cfg.get(config_section, 'purge_bitmaps') 3736 | 3737 | return cfg 3738 | 3739 | def parse_port(param): 3740 | """ 3741 | Parses and assign given port range values 3742 | i.e. 3743 | remote_nc_port = 9999 3744 | remote_nc_port = 9999,1000 3745 | """ 3746 | 3747 | if not param: return False 3748 | if param.isdigit(): 3749 | self.remote_nc_port_min = int(param) 3750 | self.remote_nc_port_max = self.remote_nc_port_min 3751 | return True 3752 | elif param.count(',') == 1: 3753 | pmin, pmax = param.split(',') 3754 | pmin = pmin.strip() 3755 | pmax = pmax.strip() 3756 | if not pmin.isdigit() or not pmax.isdigit(): return False 3757 | self.remote_nc_port_min = int(pmin) 3758 | self.remote_nc_port_max = int(pmax) 3759 | if self.remote_nc_port_min > self.remote_nc_port_max: 3760 | pmin = self.remote_nc_port_max 3761 | self.remote_nc_port_max = self.remote_nc_port_min 3762 | self.remote_nc_port_min = pmin 3763 | return True 3764 | 3765 | return False 3766 | 3767 | class PyxMail(object): 3768 | pass 3769 | 3770 | class PyxLogger(object): 3771 | pass 3772 | 3773 | class PyxStorage(object): 3774 | pass 3775 | 3776 | class PyxBinlogs(object): 3777 | pass 3778 | 3779 | class PyxBackup(object): 3780 | pass 3781 | -------------------------------------------------------------------------------- /vagrant/Vagrantfile: -------------------------------------------------------------------------------- 1 | # -*- mode: ruby -*- 2 | # vi: set ft=ruby : 3 | NODE_NAMES = ["pxb"] 4 | START_IP = 0 5 | 6 | Vagrant.configure("2") do |config| 7 | #config.ssh.insert_key = false 8 | config.ssh.pty = true 9 | config.vm.box = "revin/pyxbackup" 10 | config.vm.synced_folder ".", "/vagrant", disabled: true 11 | config.vm.synced_folder "../", "/usr/local/pyxbackup" 12 | config.vm.provider "virtualbox" do |v| 13 | v.memory = 2048 14 | v.cpus = 4 15 | v.linked_clone = true 16 | end 17 | 18 | i = START_IP 19 | 20 | NODE_NAMES.each do |nn| 21 | ipaddr = "192.168.56.3" + i.to_s 22 | i = i+1 23 | config.vm.define nn do |box| 24 | box.vm.hostname = nn 25 | box.vm.network 'private_network', ip: ipaddr 26 | end 27 | end 28 | 29 | config.vm.provision "ansible" do |ansible| 30 | ansible.playbook = "ansible/pyxbackup.yml" 31 | ansible.sudo = true 32 | #ansible.verbose = 'vvv' 33 | ansible.extra_vars = { 34 | mysql_version: '5.7' 35 | } 36 | end 37 | end 38 | -------------------------------------------------------------------------------- /vagrant/ansible/files/binaries-mysql: -------------------------------------------------------------------------------- 1 | http://mysql.mirrors.hoobly.com/Downloads/MySQL-5.5/mysql-5.5.58-linux-glibc2.12-x86_64.tar.gz 2 | http://mysql.mirrors.hoobly.com/Downloads/MySQL-5.6/mysql-5.6.37-linux-glibc2.12-x86_64.tar.gz 3 | http://mysql.mirrors.hoobly.com/Downloads/MySQL-5.7/mysql-5.7.20-linux-glibc2.12-x86_64.tar.gz 4 | https://www.percona.com/downloads/Percona-Server-5.5/Percona-Server-5.5.55-38.8/binary/tarball/Percona-Server-5.5.55-rel38.8-Linux.x86_64.ssl100.tar.gz 5 | https://www.percona.com/downloads/Percona-Server-5.6/Percona-Server-5.6.36-82.0/binary/tarball/Percona-Server-5.6.36-rel82.0-Linux.x86_64.ssl100.tar.gz 6 | https://www.percona.com/downloads/Percona-Server-LATEST/Percona-Server-5.7.20-19/binary/tarball/Percona-Server-5.7.20-19-Linux.x86_64.ssl100.tar.gz 7 | -------------------------------------------------------------------------------- /vagrant/ansible/files/binaries-xtrabackup: -------------------------------------------------------------------------------- 1 | https://www.percona.com/downloads/XtraBackup/Percona-XtraBackup-2.4.7/binary/tarball/percona-xtrabackup-2.4.7-Linux-x86_64.tar.gz 2 | https://www.percona.com/downloads/XtraBackup/Percona-XtraBackup-2.3.9/binary/tarball/percona-xtrabackup-2.3.9-Linux-x86_64.tar.gz 3 | https://www.percona.com/downloads/XtraBackup/Percona-XtraBackup-2.4.9/binary/tarball/percona-xtrabackup-2.4.9-Linux-x86_64.tar.gz 4 | -------------------------------------------------------------------------------- /vagrant/ansible/files/commands-pyxbackup: -------------------------------------------------------------------------------- 1 | --compress -q full 2 | --compress -q incr 3 | --compress --compress-with=qpress -q full 4 | --compress --compress-with=qpress -q incr -------------------------------------------------------------------------------- /vagrant/ansible/files/make-sandboxes.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | export SANDBOX_BINARY=/home/ubuntu/mysql 4 | export SANDBOX_HOME=/home/ubuntu/sandboxes 5 | cd $SANDBOX_BINARY 6 | 7 | sudo ln -s /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libssl.so.10 8 | sudo ln -s /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.10 9 | sudo ln -s /lib/x86_64-linux-gnu/libssl.so.1.0.0 /usr/lib/x86_64-linux-gnu/libssl.so.10 10 | sudo ln -s /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /usr/lib/x86_64-linux-gnu/libcrypto.so.10 11 | 12 | # Download tarballs based on list 13 | while read url; do echo $url; wget $url; done < binaries 14 | 15 | # Extract tarballs 16 | for f in *.gz; do tar xzf $f; done 17 | 18 | # Prepare MySQL binaries 19 | for b in $(find . -mindepth 1 -maxdepth 1 -type d -name mysql-\*); do v=$(echo $b|cut -d'-' -f2); mv -f $b ./$v; done 20 | 21 | # Prepare Percona Server binaries 22 | for b in $(find . -mindepth 1 -maxdepth 1 -type d -name Percona-Server-\*); do v=$(echo $b|cut -d'-' -f3); mv -f $b "./${v}0"; done 23 | 24 | # Remove tarballs 25 | find . -mindepth 1 -maxdepth 1 -type f -name \*.tar.gz -exec rm -rf {} \; 26 | 27 | # Create sandboxes 28 | for v in $(find . -mindepth 1 -maxdepth 1 -type d); do 29 | NODE_OPTIONS="--my_clause=log_slave_updates=1 --my_clause=sync_binlog=0 --my_clause=innodb_flush_log_at_trx_commit=2" \ 30 | make_replication_sandbox --sandbox_base_port=$(basename $v|sed 's/\.//g') $(basename $v) \ 31 | --how_many_slaves=1 -- --no_confirm; 32 | done 33 | 34 | # Run sysbench scripts 35 | for b in $(find $SANDBOX_HOME/ -mindepth 1 -maxdepth 1 -type d -name rsandbox_\*); do ( run-sysbench $(basename $b|cut -d'_' -f2,3,4|sed 's/_//g') & ) ; done -------------------------------------------------------------------------------- /vagrant/ansible/files/make-xtrabackups.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ssh-keygen -t rsa 4 | cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 5 | chmod 0755 /usr/local/pyxbackup/pyxbackup 6 | 7 | cd /home/ubuntu/xb 8 | 9 | # Download tarballs based on list 10 | while read url; do echo $url; wget $url; done < binaries 11 | 12 | # Extract tarballs 13 | for f in *.gz; do tar xzf $f; done 14 | 15 | # Prepare xtrabackup binaries 16 | for b in $(find . -mindepth 1 -maxdepth 1 -type d -name percona-xtrabackup-\*); do v=$(echo $b|cut -d'-' -f3); mv -f $b ./$v; done 17 | 18 | # Remove tarballs 19 | find . -mindepth 1 -maxdepth 1 -type f -name \*.tar.gz -exec rm -rf {} \; 20 | -------------------------------------------------------------------------------- /vagrant/ansible/files/mysql-sandbox.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | cd ~ 4 | wget https://github.com/datacharmer/mysql-sandbox/releases/download/3.2.14/MySQL-Sandbox-3.2.14.tar.gz 5 | tar xzf MySQL-Sandbox-3.2.14.tar.gz 6 | cd MySQL-Sandbox-3.2.14/ 7 | perl Makefile.PL PREFIX=/usr 8 | make 9 | sudo make install -------------------------------------------------------------------------------- /vagrant/ansible/files/run-sysbench.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | PORT=$1 4 | 5 | sysbench --db-driver=mysql --mysql-user=msandbox --mysql-password=msandbox \ 6 | --mysql-db=test --mysql-host=127.0.0.1 --mysql-port=$PORT --tables=2 \ 7 | --table-size=100000 --auto-inc=off --threads=1 \ 8 | --time=0 --rate=2 --rand-type=pareto oltp_read_write cleanup 9 | 10 | sysbench --db-driver=mysql --mysql-user=msandbox --mysql-password=msandbox \ 11 | --mysql-db=test --mysql-host=127.0.0.1 --mysql-port=$PORT --tables=2 \ 12 | --table-size=100000 --auto-inc=off --threads=1 \ 13 | --time=0 --rate=2 --rand-type=pareto oltp_read_write prepare 14 | 15 | sysbench --db-driver=mysql --mysql-user=msandbox --mysql-password=msandbox \ 16 | --mysql-db=test --mysql-host=127.0.0.1 --mysql-port=$PORT --tables=2 \ 17 | --table-size=100000 --auto-inc=off --threads=1 \ 18 | --time=0 --rate=2 --rand-type=pareto oltp_read_write run -------------------------------------------------------------------------------- /vagrant/ansible/files/run-tests.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | export SANDBOX_BINARY=/home/ubuntu/mysql 4 | export SANDBOX_HOME=/home/ubuntu/sandboxes 5 | 6 | cd /home/ubuntu/ 7 | CPATH=/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/ubuntu/.local/bin:/home/ubuntu/bin 8 | PPATH="$CPATH:/usr/local/pyxbackup" 9 | export PATH=$PPATH 10 | pyxbackup -X -q wipeout 11 | 12 | for d in /p/bkp/stor /p/bkp/work /p/bkp/r/stor /p/bkp/r/work; do 13 | rm -rf $d/* 14 | done 15 | 16 | for v in $(find /home/ubuntu/xb -mindepth 1 -maxdepth 1 -type d); do 17 | export PATH=$PPATH:/home/ubuntu/xb/$(basename $v)/bin 18 | echo $PATH 19 | 20 | for b in $(find $SANDBOX_HOME/ -mindepth 1 -maxdepth 1 -type d -name rsandbox_\*); do 21 | sb=$(basename $b|cut -d'_' -f2,3,4) 22 | p=$(echo $sb|sed 's/_//g') 23 | 24 | rm -rf /p/bkp/work/pyxbackup.lock 25 | while read cmd; do 26 | xcmd="pyxbackup --mysql-cnf=$SANDBOX_HOME/rsandbox_${sb}/master/my.sandbox.cnf --mysql-sock=/tmp/mysql_sandbox${p}.sock ${cmd}" 27 | eval $xcmd 28 | echo "$? $p $(basename $v) $cmd" 29 | done < commands-pyxbackup 30 | done 31 | done 32 | 33 | export PATH=$CPATH 34 | -------------------------------------------------------------------------------- /vagrant/ansible/pyxbackup.yml: -------------------------------------------------------------------------------- 1 | --- 2 | - hosts: all 3 | tasks: 4 | 5 | - name: determine distrib 6 | command: lsb_release -sc 7 | register: distrib 8 | 9 | - name: determine kernel 10 | command: uname -r 11 | register: kernel 12 | 13 | - name: percona repo 14 | apt: 15 | deb: https://repo.percona.com/apt/percona-release_0.1-4.{{ distrib.stdout }}_all.deb 16 | state: present 17 | 18 | - name: install essential packages 19 | apt: 20 | name: make,qpress,netcat,socat,sysstat,mbuffer,libaio1,sysbench,lsof,percona-toolkit,linux-tools-common,linux-tools-{{ kernel.stdout }},linux-tools-{{ kernel.stdout }},python-mysqldb 21 | update_cache: true 22 | state: present 23 | 24 | - name: install build packages 25 | apt: 26 | name: build-essential,autoconf,libtool,gawk,alien,fakeroot,linux-headers-{{ kernel.stdout }} 27 | update_cache: true 28 | state: present 29 | 30 | - name: custom libgcrypt for xtrabackup binaries 31 | apt: 32 | deb: https://launchpadlibrarian.net/201289896/libgcrypt11_1.5.3-2ubuntu4.2_amd64.deb 33 | state: present 34 | 35 | - name: create directories 36 | file: 37 | path: "{{ item }}" 38 | owner: ubuntu 39 | group: ubuntu 40 | mode: 0755 41 | state: directory 42 | with_items: 43 | - /p/msb 44 | - /p/bkp 45 | - /p/bkp/stor 46 | - /p/bkp/work 47 | - /home/ubuntu/xb 48 | - /home/ubuntu/mysql 49 | - /p/bkp/r 50 | - /p/bkp/r/stor 51 | - /p/bkp/r/work 52 | 53 | - name: copy mysql binaries urls 54 | copy: 55 | src: files/binaries-mysql 56 | dest: /home/ubuntu/mysql/binaries 57 | mode: 0644 58 | owner: ubuntu 59 | group: ubuntu 60 | 61 | - name: copy xtrabackup binaries urls 62 | copy: 63 | src: files/binaries-xtrabackup 64 | dest: /home/ubuntu/xb/binaries 65 | mode: 0644 66 | owner: ubuntu 67 | group: ubuntu 68 | 69 | - name: upload mysql sandbox installer 70 | copy: 71 | src: files/mysql-sandbox.sh 72 | dest: /usr/bin/mysql-sandbox 73 | mode: 0755 74 | 75 | - name: upload sandboxes script 76 | copy: 77 | src: files/make-sandboxes.sh 78 | dest: /usr/bin/make-sandboxes 79 | mode: 0755 80 | 81 | - name: upload sysbench scripts 82 | copy: 83 | src: files/run-sysbench.sh 84 | dest: /usr/bin/run-sysbench 85 | mode: 0755 86 | 87 | - name: upload xtrabackups scripts 88 | copy: 89 | src: files/make-xtrabackups.sh 90 | dest: /usr/bin/make-xtrabackups 91 | mode: 0755 92 | 93 | - name: upload tests scripts 94 | copy: 95 | src: files/run-tests.sh 96 | dest: /usr/bin/run-tests 97 | mode: 0755 98 | 99 | - name: upload pyxbackup 100 | copy: 101 | src: ../../pyxbackup 102 | dest: /usr/bin/pyxbackup 103 | mode: 0755 104 | 105 | - name: upload pyxbackup test commands 106 | copy: 107 | src: files/commands-pyxbackup 108 | dest: /home/ubuntu/commands-pyxbackup 109 | mode: 0755 110 | 111 | - name: skeleton pyxbackup.cnf 112 | template: 113 | src: pyxbackup.cnf 114 | dest: /etc/pyxbackup.cnf 115 | mode: 0644 116 | owner: ubuntu 117 | group: ubuntu 118 | -------------------------------------------------------------------------------- /vagrant/ansible/tasks/linux-selinux.yml: -------------------------------------------------------------------------------- 1 | --- 2 | - selinux: 3 | policy: targeted 4 | state: permissive 5 | 6 | -------------------------------------------------------------------------------- /vagrant/ansible/tasks/linux-ssh.yml: -------------------------------------------------------------------------------- 1 | --- 2 | - copy: 3 | src: ~/.vagrant.d/insecure_private_key 4 | dest: /home/vagrant/.ssh/id_rsa 5 | owner: vagrant 6 | group: vagrant 7 | mode: 0600 8 | - name: update authorized_keys 9 | shell: > 10 | ssh-keygen -y -f id_rsa >> authorized_keys && 11 | touch /home/vagrant/.ansible/.state_authkeys_installed 12 | args: 13 | chdir: /home/vagrant/.ssh 14 | creates: /home/vagrant/.ansible/.state_authkeys_installed 15 | -------------------------------------------------------------------------------- /vagrant/ansible/tasks/repo-epel.yml: -------------------------------------------------------------------------------- 1 | --- 2 | - name: epel-repo 3 | yum: 4 | name: https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm 5 | state: present 6 | -------------------------------------------------------------------------------- /vagrant/ansible/tasks/repo-percona.yml: -------------------------------------------------------------------------------- 1 | --- 2 | - name: percona-repo 3 | yum: 4 | name: http://www.percona.com/downloads/percona-release/redhat/0.1-4/percona-release-0.1-4.noarch.rpm 5 | state: present 6 | -------------------------------------------------------------------------------- /vagrant/ansible/tasks/repo-twindb.yml: -------------------------------------------------------------------------------- 1 | --- 2 | - name: download twindb repo 3 | get_url: 4 | url: https://packagecloud.io/install/repositories/twindb/main/script.rpm.sh 5 | dest: /root/twindb_repo.sh 6 | - name: install twindb repo 7 | shell: bash /root/twindb_repo.sh 8 | args: 9 | creates: /etc/yum.repos.d/twindb_main.repo 10 | -------------------------------------------------------------------------------- /vagrant/ansible/templates/pyxbackup.cnf: -------------------------------------------------------------------------------- 1 | [pyxbackup] 2 | work_dir = /p/bkp/work 3 | stor_dir = /p/bkp/stor 4 | mysql_user = msandbox 5 | mysql_pass = msandbox --------------------------------------------------------------------------------