├── README.md └── rdiff-backup ├── CHANGELOG ├── COPYING ├── CVS-README ├── FAQ-body.html ├── README ├── TODO ├── Windows-README.txt ├── dist ├── makedist ├── makerpm ├── makeweb ├── rdiff-backup-rh7x.patch ├── rdiff-backup.rh7x.spec ├── rdiff-backup.spec.template ├── rdiff-backup.spec.template-fedora └── setup.py ├── examples-body.html ├── misc ├── compress-rdiff-backup-increments ├── find2dirs ├── init_files.py ├── librsync-many-files.py ├── make-many-data-files.py ├── myrm ├── rdiff-backup-delete.py ├── rdiff-many-files.py └── remove-comments.py ├── python-rdiff ├── rdiff-backup ├── rdiff-backup-statistics ├── rdiff-backup-statistics.1 ├── rdiff-backup.1 ├── rdiff_backup ├── FilenameMapping.py ├── Globals.py ├── Hardlink.py ├── Main.py ├── Rdiff.py ├── Security.py ├── SetConnections.py ├── TempFile.py ├── Time.py ├── __init__.py ├── _librsyncmodule.c ├── backup.py ├── cmodule.c ├── compare.py ├── compilec.py ├── connection.py ├── eas_acls.py ├── fs_abilities.py ├── hash.py ├── increment.py ├── iterfile.py ├── journal.py ├── lazy.py ├── librsync.py ├── librsync_memoryleak2.py ├── log.py ├── longname.py ├── manage.py ├── memoryleak.c ├── metadata.py ├── myrdiff.py ├── profiled_rdb.py ├── regress.py ├── restore.py ├── robust.py ├── rorpiter.py ├── rpath.py ├── selection.py ├── static.py ├── statistics.py ├── user_group.py └── win_acls.py └── testing ├── FilenameMappingtest.py ├── backuptest.py ├── benchmark.py ├── chdir-wrapper ├── chdir-wrapper2 ├── commontest.py ├── comparetest.py ├── connectiontest.py ├── ctest.py ├── destructive_steppingtest.py ├── eas_aclstest.py ├── filelisttest.py ├── finaltest.py ├── find-max-ram.py ├── fs_abilitiestest.py ├── hardlinktest.py ├── hashtest.py ├── incrementtest.py ├── iterfiletest.py ├── journaltest.py ├── killtest.py ├── lazytest.py ├── librsynctest.py ├── longnametest.py ├── makerestoretest3 ├── metadatatest.py ├── rdifftest.py ├── regressiontest.py ├── regresstest.py ├── resourceforktest.py ├── restoretest.py ├── rlisttest.py ├── robusttest.py ├── roottest.py ├── rorpitertest.py ├── rpathtest.py ├── securitytest.py ├── selectiontest.py ├── server.py ├── setconnectionstest.py ├── statictest.py ├── statisticstest.py ├── test1 └── tmp │ └── placeholder ├── test2 └── tmp │ └── foo ├── test_with_profiling.py ├── testall.py ├── timetest.py └── user_grouptest.py /README.md: -------------------------------------------------------------------------------- 1 | ## rdiff-backup is a reverse differential backup tool 2 | 3 | [rdiff-backup](http://www.nongnu.org/rdiff-backup/) backs up one directory to another, possibly over a network. The target directory ends up a copy of the source directory, but extra reverse diffs are stored in a special subdirectory of that target directory, so you can still recover files lost some time ago. The idea is to combine the best features of a mirror and an incremental backup. rdiff-backup also preserves subdirectories, hard links, dev files, permissions, uid/gid ownership (if it is running as root), modification times, acls, eas, resource forks, etc. Finally, rdiff-backup can operate in a bandwidth efficient manner over a pipe, like rsync. Thus you can use rdiff-backup and ssh to securely back a hard drive up to a remote location, and only the differences will be transmitted. 4 | 5 | Read more on [the rdiff-backup website](http://www.nongnu.org/rdiff-backup/). 6 | 7 | ## Installing 8 | 9 | rdiff-backup is available in package form across many operating systems. 10 | 11 | ### Linux 12 | 13 | * Debian/Ubuntu: `apt-get install rdiff-backup` 14 | * RHEL/CentOS: `yum install rdiff-backup` 15 | 16 | ### OS X 17 | 18 | * With [homebrew](http://brew.sh/): `brew install rdiff-backup` 19 | 20 | ### Windows 21 | 22 | * With the [Cygwin](https://cygwin.com/) setup tool, install the `rdiff-backup` package, or 23 | * With [Chocolatey](https://chocolatey.org/): `choco install rdiff-backup` (and probably `choco install win32-openssh` if you'd like to back up over a network) 24 | 25 | The [old README](rdiff-backup/README) contains information about building from source. 26 | 27 | ## Usage 28 | 29 | Here are some basic examples. For more detail and information about restoring files, see [these examples](http://www.nongnu.org/rdiff-backup/examples.html). 30 | 31 | #### Local to local backup 32 | 33 | `rdiff-backup /some/local-dir /some/other-local-dir` 34 | 35 | #### Local to remote backup 36 | 37 | `rdiff-backup /some/local-dir user@example.com::/some/remote-dir` 38 | 39 | #### Remote to local backup 40 | 41 | `rdiff-backup user@example.com::/some/remote-dir /some/local-dir` 42 | 43 | ## Support 44 | 45 | For help, try looking at the [documentation](http://www.nongnu.org/rdiff-backup/docs.html) and/or [the FAQ](http://www.nongnu.org/rdiff-backup/FAQ.html). If that doesn't help with your problem, try reading or posting a message to the [mailing list](http://www.nongnu.org/rdiff-backup/savannah.html#mailing_list). 46 | 47 | [Sol1](http://sol1.com.au) has taken over maintainership as of February 2016. We are currently in the process of migrating from the previous [Savannah bugs database](http://savannah.nongnu.org/bugs/?group=rdiff-backup) to the GitHub [issues list](https://github.com/sol1/rdiff-backup/issues) in this repository. Current bugs are being triaged and migrated as we can. 48 | 49 | If you think you've found a bug, please search both databases then, if you can't find anything, [create a new issue](https://github.com/sol1/rdiff-backup/issues/new). 50 | 51 | ## Contributing 52 | 53 | Contributions are welcome. Fork this repo on GitHub, commit and open a new pull request accordingly. 54 | 55 | ## License 56 | 57 | rdiff-backup is released under the [GNU General Public License](rdiff-backup/COPYING). 58 | 59 | ## Acknowledgements 60 | 61 | [Sol1](http://sol1.com.au) is the current maintainer of rdiff-backup. 62 | 63 | Previous project leads / maintainers were Edward Ned Harvey, Andrew Ferguson, Dean Gaudet and the original author Ben Escoto. 64 | 65 | [Other contributors](http://www.nongnu.org/rdiff-backup/acknowledgments.html) include Daniel Hazelbaker, Dean Gaudet, Andrew Ferguson, Josh Nisly and Fred Gansevles. -------------------------------------------------------------------------------- /rdiff-backup/CHANGELOG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Seravo/rdiff-backup/8ccc5a3b44c996ecd810f8d5d586d0da6435cc32/rdiff-backup/CHANGELOG -------------------------------------------------------------------------------- /rdiff-backup/CVS-README: -------------------------------------------------------------------------------- 1 | CVS README - Notes for people checking out of CVS 2 | ------------------------------------------------- 3 | 4 | Getting rdiff-backup to run: 5 | ---------------------------- 6 | 7 | If you want to run a version of rdiff-backup checked out of CVS into 8 | your $RDB_CVS directory, change to $RDB_CVS/rdiff_backup and run the 9 | ./compilec.py file: 10 | 11 | cd $RDB_CVS/rdiff_backup; python compilec.py 12 | 13 | With any luck, _librsync.so and C.so libraries will appear in that 14 | directory. Then run rdiff-backup, making sure that all the files are 15 | in your PYTHONPATH: 16 | 17 | PYTHONPATH=$RDB_CVS $RDB_CVS/rdiff-backup 18 | 19 | 20 | Running the unit tests: 21 | ----------------------- 22 | 23 | If you want to try some of tests, you first have to get the 24 | testfiles.tar.gz tarball. It is available at 25 | 26 | http://rdiff-backup.stanford.edu/testfiles.tar.gz 27 | 28 | To untar it, root is required because the tarball contains device 29 | files, files with various uid/gid, etc. If you don't have root, it's 30 | ok, all the tests except for roottest.py may still work. 31 | 32 | So, three steps: 33 | 34 | 1) Make sure the the C modules are compiled as explained above: 35 | 36 | cd $RDB_CVS/rdiff_backup; python compilec.py 37 | 38 | 2) Untar the testfiles tarball, as root if you have it: 39 | 40 | cd $RDB_CVS/testing; tar -xvzf testfiles.tar.gz 41 | 42 | 3) In the testing directory, run each of the *test.py files as 43 | desired. For instance, 44 | 45 | cd $RDB_CVS/testing; python rpathtest.py 46 | 47 | If python restoretest.py doesn't work, try running 48 | ./makerestoretest3 49 | 50 | Note: 51 | - some of the tests may require a normal user ben and group ben to exist. 52 | - you may need to install extra packages for acl/xattr... (such as 53 | python-pylibacl and python-pyxattr on debian) 54 | -------------------------------------------------------------------------------- /rdiff-backup/README: -------------------------------------------------------------------------------- 1 | INSTALLATION: 2 | 3 | Thank you for trying rdiff-backup. To install, run: 4 | 5 | python setup.py install 6 | 7 | The build process can be also be run separately: 8 | 9 | python setup.py build 10 | 11 | The default prefix is generally /usr, so files would be put in /usr/bin, 12 | /usr/share/man/, etc. An alternate prefix can be specified using the 13 | --prefix= option. For example: 14 | 15 | python setup.py install --prefix=/usr/local 16 | 17 | The default prefix depends on how you (or your distribution) installed and 18 | configured Python. Suggested reading is "How installation works" from the 19 | Python docs, which includes commands to determine your default prefix: 20 | http://docs.python.org/install/index.html#how-installation-works 21 | 22 | The setup script expects to find librsync headers and libraries in the 23 | default location, usually /usr/include and /usr/lib. If you want the 24 | setup script to check different locations, use the --librsync-dir 25 | switch or the LIBRSYNC_DIR environment variable. For instance, 26 | 27 | python setup.py --librsync-dir=/usr/local build 28 | 29 | instructs the setup program to look in /usr/local/include and 30 | /usr/local/lib for the librsync files. 31 | 32 | Finally, the --lflags and --libs options, and the LFLAGS and LIBS 33 | environment variables are also recognized. Running setup.py with no 34 | arguments will display some help. Additional help is displayed by the 35 | command: 36 | 37 | python setup.py install --help 38 | 39 | More information about using setup.py and how rdiff-backup is installed 40 | is available from the Python guide, Installing Python Modules for System 41 | Administrators, located at http://docs.python.org/install/index.html 42 | 43 | NB: There is no uninstall command provided by the Python distutils system. 44 | One strategy is to use the python setup.py install --record option 45 | to save a list of the files installed to . 46 | 47 | To build from source on Windows, you can use the command: 48 | 49 | python setup.py py2exe --single-file 50 | 51 | to build a single executable file which contains Python, librsync, and 52 | all required modules. 53 | 54 | REQUIREMENTS: 55 | 56 | Remember that you must have Python 2.2 or later and librsync 0.9.7 or 57 | later installed. For Python, see http://www.python.org. The 58 | rdiff-backup homepage at http://rdiff-backup.nongnu.org/ should 59 | have a recent version of librsync; otherwise see the librsync homepage 60 | at http://librsync.sourceforge.net/. On Windows, you must have the 61 | Python for Windows extensions installed if you are building from source. 62 | The extensions can be downloaded from: http://pywin32.sourceforge.net/ 63 | If you are not building from source on Windows, you do not need Python 64 | or librsync; they are bundled with the executable. 65 | 66 | For remote operation, rdiff-backup should be installed and in the 67 | PATH on remote system(s) (see man page for more information). On 68 | Windows, if you are using the provided .exe binary, you must have an 69 | SSH package installed for remote operation. 70 | 71 | The python modules pylibacl and pyxattr are optional. If they are 72 | installed and in the default pythonpath, rdiff-backup will support 73 | access control lists and user extended attributes, provided the file 74 | system(s) also support these features. Pylibacl and pyxattr can be 75 | downloaded from http://pylibacl.sourceforge.net/ and 76 | http://pyxattr.sourceforge.net/. Mac OS X users need a different 77 | pyxattr module, which can be downloaded from 78 | http://cheeseshop.python.org/pypi/xattr 79 | 80 | If you want to use rdiff-backup-statistics, you must have Python 2.4 81 | or later. 82 | 83 | TROUBLESHOOTING: 84 | 85 | If you have everything installed properly, and it still doesn't work, 86 | see the enclosed FAQ.html, the web page at http://rdiff-backup.nongnu.org 87 | and/or the mailing list. 88 | 89 | The FAQ in particular is an important reference, especially if you are 90 | using smbfs/CIFS, Windows, or have compiled by hand on Mac OS X. 91 | -------------------------------------------------------------------------------- /rdiff-backup/TODO: -------------------------------------------------------------------------------- 1 | Fix restore with --force over existing regular file. 2 | 3 | For comparing, check source filesystem's abilities 4 | 5 | Clean up compare reports 6 | 7 | Test comparing of single files, and files/directories specified by 8 | increment. Also test --include/--exclude with compare options. 9 | 10 | ---------[ Medium term ]--------------------------------------- 11 | 12 | Look into sparse file support (requested by Stelios K. Kyriacou) 13 | 14 | Look at Kent Borg's suggestion for restore options and digests. 15 | 16 | ---------[ Long term ]--------------------------------------- 17 | 18 | Think about adding Gaudet's idea for keeping track of renamed files. 19 | 20 | Look into different inode generation techniques (see treescan, Dean 21 | Gaudet's other post). 22 | 23 | -------------------------------------------------------------------------------- /rdiff-backup/Windows-README.txt: -------------------------------------------------------------------------------- 1 | INSTALLATION: 2 | 3 | Thank you for trying rdiff-backup on Windows. Native support for the Windows 4 | environment is quite new in rdiff-backup. Please read the manual page, FAQ and 5 | the Wiki thorougly. 6 | 7 | To install the provided binary, simply copy rdiff-backup.exe to someplace in 8 | your PATH. Everything is included in the binary (including Python) for local 9 | operation. For remote operation, you will need to install a Windows SSH 10 | program. You will also need to install rdiff-backup on the remote system(s). 11 | 12 | You will need the Microsoft Visual C++ 2008 redistributables. If these are 13 | not installed on your system, rdiff-backup will be unable to run and Windows 14 | will display a message such as "The system cannot execute the specified 15 | program". To install the redistributables for all users, install the package 16 | available from Microsoft.com (search for "visual c 2008 redistributable"). 17 | 18 | Alternatively, you can install the redistributable in a "side-by-side" 19 | configuration, which does not require administrator privelges. Simply 20 | download the DLL package from: 21 | http://download.savannah.gnu.org/releases/rdiff-backup/Microsoft.VC90.zip 22 | and copy the four enclosed files to the same directory as rdiff-backup.exe. 23 | 24 | You will need to follow either method only once. 25 | 26 | 27 | ADDITIONAL ISSUES: 28 | 29 | Currently, rdiff-backup's --include and --exclude options do not support 30 | Windows paths with \ as the directory separator. Instead, it is necessary to 31 | use / which is the Unix directory separator. 32 | 33 | Additionally, you may need to run rdiff-backup from the same directory as the 34 | source of your backup, eg: 35 | 36 | > cd c:\ 37 | > rdiff-backup.exe --include "c:/My Stuff" --exclude "c:/**" c:/ c:/Backup 38 | 39 | will work to backup "c:\My Stuff" to "c:\Backup", but: 40 | 41 | > cd "c:\My Stuff" 42 | > rdiff-backup.exe --include "c:/My Stuff" --exclude "c:/**" c:/ c:/Backup 43 | 44 | will not work. UPDATE: With appropriate escaping, it looks like it is 45 | possible for this to work. Follow this example: 46 | 47 | > mkdir c:\foo 48 | > cd "c:\Documents and Settings" 49 | > rdiff-backup.exe --include c:\\/foo --exclude c:\\/** c:\/ c:\bar 50 | 51 | The \\ is necessary in the --include and --exclude options because those 52 | options permit regular-expressions, and \ is the escape character in 53 | regular-expressions, and thus needs to be escaped itself. 54 | 55 | 56 | TROUBLESHOOTING: 57 | 58 | If you have everything installed properly, and it still doesn't work, 59 | see the enclosed manual, FAQ, the web page at http://rdiff-backup.nongnu.org, 60 | and/or the mailing list. You can subscribe to the mailing list at: 61 | http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users 62 | 63 | You can also try searching the mailing list archives: 64 | http://lists.nongnu.org/archive/html/rdiff-backup-users/ 65 | -------------------------------------------------------------------------------- /rdiff-backup/dist/makedist: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import os, re, shutil, time, sys, getopt 4 | 5 | SourceDir = "rdiff_backup" 6 | DistDir = "dist" 7 | 8 | # Various details about the files must also be specified by the rpm 9 | # spec template. 10 | spec_template = "dist/rdiff-backup.spec.template" 11 | # The Fedora distribution has its own spec template 12 | fedora_spec_template = "dist/rdiff-backup.spec.template-fedora" 13 | 14 | def CopyMan(destination, version): 15 | """Create updated man page at the specified location""" 16 | fp = open(destination, "w") 17 | date = time.strftime("%B %Y", time.localtime(time.time())) 18 | version = "Version "+version 19 | firstline = ('.TH RDIFF-BACKUP 1 "%s" "%s" "User Manuals"\n' % 20 | (date, version)) 21 | fp.write(firstline) 22 | infp = open(os.path.basename(destination), "r") 23 | infp.readline() 24 | fp.write(infp.read()) 25 | fp.close() 26 | infp.close() 27 | 28 | def MakeHTML(input_prefix, title): 29 | """Create html and wml versions from given html body 30 | 31 | Input expected in -body.html, and output will be put 32 | in .wml and .html. 33 | 34 | """ 35 | body_fp = open("%s-body.html" % (input_prefix,), "r") 36 | body_string = body_fp.read() 37 | body_fp.close() 38 | 39 | wml_fp = open("%s.wml" % (input_prefix,), "w") 40 | wml_fp.write( 41 | """#include 'template.wml' home=. curpage=none title="%s" 42 | 43 | 44 |

%s

45 | 46 | """ % (title, title)) 47 | wml_fp.write(body_string) 48 | wml_fp.write("\n\n") 49 | wml_fp.close() 50 | 51 | html_fp = open("%s.html" % (input_prefix,), "w") 52 | html_fp.write( 53 | """ 54 | 55 | 56 | 57 | %s 58 | 59 | 60 | 61 |

%s

62 | """ % (title, title)) 63 | html_fp.write(body_string) 64 | html_fp.write("\n") 65 | html_fp.close() 66 | 67 | def VersionedCopy(source, dest, munge_date = 0): 68 | """Copy source to dest, substituting $version with version 69 | 70 | If munge_date is true, also replace $date with string like "August 71 | 8, 2003". 72 | 73 | """ 74 | fin = open(source, "rb") 75 | inbuf = fin.read() 76 | assert not fin.close() 77 | 78 | outbuf = re.sub("\$version", Version, inbuf) 79 | if outbuf == inbuf: assert 0, "No $version string replaced" 80 | assert not re.search("\$version", outbuf), \ 81 | "Some $version strings not repleased in %s" % (source,) 82 | 83 | if munge_date: 84 | inbuf = outbuf 85 | outbuf = re.sub("\$date", time.strftime("%B %d, %Y"), inbuf, 1) 86 | if outbuf == inbuf: assert 0, "No $date string replaced" 87 | assert not re.search("\$date", outbuf), "Two $date strings found" 88 | 89 | fout = open(dest, "wb") 90 | fout.write(outbuf) 91 | assert not fout.close() 92 | 93 | def MakeTar(specfiles): 94 | """Create rdiff-backup tar file""" 95 | tardir = "rdiff-backup-%s" % Version 96 | tarfilename = "rdiff-backup-%s.tar.gz" % Version 97 | try: 98 | os.lstat(tardir) 99 | shutil.rmtree(tardir) 100 | except OSError: pass 101 | os.mkdir(tardir) 102 | for filename in ["CHANGELOG", "COPYING", "README", "Windows-README.txt", 103 | "FAQ.html", "examples.html", 104 | SourceDir + "/cmodule.c", 105 | SourceDir + "/_librsyncmodule.c", 106 | DistDir + "/setup.py"] + specfiles: 107 | shutil.copyfile(filename, 108 | os.path.join(tardir, os.path.basename(filename))) 109 | 110 | os.mkdir(tardir+"/rdiff_backup") 111 | for filename in ["eas_acls.py", "backup.py", "connection.py", "compare.py", 112 | "FilenameMapping.py", "fs_abilities.py", 113 | "Hardlink.py", "hash.py", "increment.py", "__init__.py", 114 | "iterfile.py", "lazy.py", "librsync.py", 115 | "log.py", "longname.py", "Main.py", "manage.py", 116 | "metadata.py", "Rdiff.py", "regress.py", "restore.py", 117 | "robust.py", "rorpiter.py", "rpath.py", 118 | "Security.py", "selection.py", 119 | "SetConnections.py", "static.py", 120 | "statistics.py", "TempFile.py", "Time.py", 121 | "user_group.py", "win_acls.py"]: 122 | shutil.copyfile(os.path.join(SourceDir, filename), 123 | os.path.join(tardir, "rdiff_backup", filename)) 124 | 125 | VersionedCopy("%s/Globals.py" % (SourceDir,), 126 | "%s/rdiff_backup/Globals.py" % (tardir,)) 127 | VersionedCopy("rdiff-backup", "%s/rdiff-backup" % (tardir,), 1) 128 | VersionedCopy("rdiff-backup-statistics", "%s/rdiff-backup-statistics" 129 | % (tardir,), 1) 130 | VersionedCopy(DistDir + "/setup.py", "%s/setup.py" % (tardir,)) 131 | 132 | os.chmod(os.path.join(tardir, "setup.py"), 0755) 133 | os.chmod(os.path.join(tardir, "rdiff-backup"), 0644) 134 | CopyMan(os.path.join(tardir, "rdiff-backup.1"), Version) 135 | CopyMan(os.path.join(tardir, "rdiff-backup-statistics.1"), Version) 136 | if os.name != 'nt': 137 | os.system("tar -cvzf %s %s" % (tarfilename, tardir)) 138 | else: 139 | import tarfile 140 | tar = tarfile.open(tarfilename, 'w:gz') 141 | for path in os.listdir(tardir): 142 | tar.add(os.path.join(tardir, path)) 143 | tar.close() 144 | 145 | shutil.rmtree(tardir) 146 | return tarfilename 147 | 148 | def MakeSpecFile(): 149 | """Create spec file using spec template""" 150 | specfile, fedora_specfile= "rdiff-backup.spec", "rdiff-backup.spec-fedora" 151 | VersionedCopy(spec_template, specfile) 152 | VersionedCopy(fedora_spec_template, fedora_specfile) 153 | return [specfile, fedora_specfile] 154 | 155 | def parse_cmdline(arglist): 156 | """Returns action""" 157 | global Version 158 | def error(): 159 | print "Syntax: makedist [--html-only] [version_number]" 160 | sys.exit(1) 161 | 162 | optlist, args = getopt.getopt(arglist, "", ["html-only"]) 163 | if len(args) != 1: error() 164 | else: Version = args[0] 165 | 166 | for opt, arg in optlist: 167 | if opt == "--html-only": return "HTML" 168 | else: assert 0, "Bad argument" 169 | return "All" 170 | 171 | def Main(): 172 | action = parse_cmdline(sys.argv[1:]) 173 | print "Making HTML" 174 | MakeHTML("FAQ", "rdiff-backup FAQ") 175 | MakeHTML("examples", "rdiff-backup examples") 176 | 177 | if action != "HTML": 178 | assert action == "All" 179 | print "Processing version " + Version 180 | specfiles = MakeSpecFile() 181 | print "Made specfiles ", specfiles 182 | tarfile = MakeTar(specfiles) 183 | print "Made tar file " + tarfile 184 | 185 | 186 | if __name__ == "__main__" and not globals().has_key('__no_execute__'): 187 | Main() 188 | 189 | -------------------------------------------------------------------------------- /rdiff-backup/dist/makerpm: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # 3 | # Builds two rpms, one for generic release, and the other for the 4 | # Fedora project. 5 | 6 | import os, sys 7 | 8 | rpmroot = os.path.join(os.environ['HOME'], 'rpm') 9 | 10 | if len(sys.argv) == 2: 11 | Version = sys.argv[1] 12 | specfile = "rdiff-backup.spec" 13 | fedora_specfile = "rdiff-backup.spec-fedora" 14 | print "Using specfiles %s, %s" % (specfile, fedora_specfile) 15 | else: 16 | print "Syntax: %s version_number" % sys.argv[0] 17 | sys.exit(1) 18 | 19 | base = "rdiff-backup-%s" % (Version,) 20 | tarfile = base + ".tar.gz" 21 | rpmbase = base + "-1" 22 | i386_rpm = rpmbase + ".i386.rpm" 23 | source_rpm = rpmbase + ".src.rpm" 24 | 25 | fedora_rpmbase = base + "-0.fdr.1" 26 | fedora_i386_rpm = fedora_rpmbase + ".i386.rpm" 27 | fedora_source_rpm = fedora_rpmbase + ".src.rpm" 28 | 29 | # These assume the rpm root directory $HOME/rpm. The 30 | # nonstandard location allows for building by non-root user. 31 | assert not os.system("cp %s %s/SOURCES" % (tarfile, rpmroot)) 32 | assert not os.system("rpmbuild -ba -v --sign " + specfile) 33 | assert not os.system("mv %s/RPMS/i386/%s ." % (rpmroot, i386_rpm)) 34 | assert not os.system("mv %s/SRPMS/%s ." % (rpmroot, source_rpm)) 35 | 36 | assert not os.system("rpmbuild -ba -v --sign " + fedora_specfile) 37 | assert not os.system("mv %s/RPMS/i386/%s ." % (rpmroot, fedora_i386_rpm)) 38 | assert not os.system("mv %s/SRPMS/%s ." % (rpmroot, fedora_source_rpm)) 39 | -------------------------------------------------------------------------------- /rdiff-backup/dist/makeweb: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import sys, os 4 | 5 | def RunCommand(cmd, ignore_error = 0): 6 | print cmd 7 | if ignore_error: os.system(cmd) 8 | else: assert not os.system(cmd) 9 | 10 | webprefix = "/home/ben/misc/html/mirror/rdiff-backup/" 11 | 12 | if not sys.argv[1:]: 13 | print 'Call with version number, as in "./makeweb 0.3.1"' 14 | print "to move new rpms and tarballs. Now just remaking FAQ and man page." 15 | print 16 | else: 17 | version = sys.argv[1] 18 | RunCommand("cp *%s* %s" % (version, webprefix)) 19 | 20 | RunCommand("rman -f html -r '' rdiff-backup.1 > %srdiff-backup.1.html" 21 | % webprefix) 22 | RunCommand("cp FAQ.wml CHANGELOG %s" % webprefix) 23 | 24 | 25 | if sys.argv[1:]: 26 | RunCommand("mkdir %s/OLD/%s" % (webprefix, version), ignore_error = 1) 27 | RunCommand("cp rdiff-backup-%s.tar.gz rdiff-backup-%s*rpm %s/OLD/%s" % 28 | (version, version, webprefix, version)) 29 | os.chdir(webprefix) 30 | print "cd ", webprefix 31 | if sys.argv[1:]: 32 | for filename in os.listdir('OLD/' + version): 33 | try: os.lstat(filename) 34 | except OSError: pass 35 | else: os.remove(filename) 36 | os.symlink('OLD/%s/%s' % (version, filename), filename) 37 | RunCommand("rm latest latest.src.rpm latest.tar.gz", ignore_error = 1) 38 | RunCommand("ln -s rdiff-backup-%s-1.src.rpm latest.src.rpm" % (version,)) 39 | os.symlink("rdiff-backup-%s.tar.gz" % (version,), 'latest.tar.gz') 40 | os.symlink('OLD/%s' % (version,), 'latest') 41 | RunCommand("./Make") 42 | -------------------------------------------------------------------------------- /rdiff-backup/dist/rdiff-backup-rh7x.patch: -------------------------------------------------------------------------------- 1 | --- rdiff-backup.old Sat Apr 6 10:05:18 2002 2 | +++ rdiff-backup Sat Apr 6 10:05:25 2002 3 | @@ -1,4 +1,4 @@ 4 | -#!/usr/bin/env python 5 | +#!/usr/bin/env python2 6 | # 7 | # rdiff-backup -- Mirror files while keeping incremental changes 8 | # Version 0.7.1 released March 25, 2002 9 | -------------------------------------------------------------------------------- /rdiff-backup/dist/rdiff-backup.rh7x.spec: -------------------------------------------------------------------------------- 1 | Summary: Convenient and transparent local/remote incremental mirror/backup 2 | Name: rdiff-backup 3 | Release: 1 4 | URL: http://www.stanford.edu/~bescoto/rdiff-backup/ 5 | Source: %{name}-%{version}.tar.gz 6 | Copyright: GPL 7 | Group: Applications/Archiving 8 | BuildRoot: %{_tmppath}/%{name}-root 9 | requires: librsync, python2 >= 2.2 10 | Patch: rdiff-backup-rh7x.patch 11 | 12 | %description 13 | rdiff-backup is a script, written in Python, that backs up one 14 | directory to another and is intended to be run periodically (nightly 15 | from cron for instance). The target directory ends up a copy of the 16 | source directory, but extra reverse diffs are stored in the target 17 | directory, so you can still recover files lost some time ago. The idea 18 | is to combine the best features of a mirror and an incremental 19 | backup. rdiff-backup can also operate in a bandwidth efficient manner 20 | over a pipe, like rsync. Thus you can use rdiff-backup and ssh to 21 | securely back a hard drive up to a remote location, and only the 22 | differences from the previous backup will be transmitted. 23 | 24 | %prep 25 | %setup 26 | %patch 27 | %build 28 | 29 | %install 30 | rm -rf $RPM_BUILD_ROOT 31 | mkdir -p $RPM_BUILD_ROOT/usr/bin 32 | mkdir -p $RPM_BUILD_ROOT/usr/share/man/man1 33 | 34 | install -m 755 rdiff-backup $RPM_BUILD_ROOT/usr/bin/rdiff-backup 35 | install -m 644 rdiff-backup.1 $RPM_BUILD_ROOT/usr/share/man/man1/rdiff-backup.1 36 | %clean 37 | 38 | %files 39 | %defattr(-,root,root) 40 | 41 | /usr/bin/rdiff-backup 42 | /usr/share/man/man1/rdiff-backup.1.gz 43 | %doc CHANGELOG COPYING README FAQ.html 44 | 45 | %changelog 46 | * Sat Apr 6 2002 Ben Escoto 47 | - Made new version for Redhat 7.x series 48 | * Sun Nov 4 2001 Ben Escoto 49 | - Initial RPM 50 | 51 | -------------------------------------------------------------------------------- /rdiff-backup/dist/rdiff-backup.spec.template: -------------------------------------------------------------------------------- 1 | %define PYTHON_NAME %((rpm -q --quiet python2 && echo python2) || echo python) 2 | %define PYTHON_VERSION %(%{PYTHON_NAME} -c 'import sys; print sys.version[:3],') 3 | %define NEXT_PYTHON_VERSION %(%{PYTHON_NAME} -c 'import sys; print "%d.%d" % (sys.version_info[0], sys.version_info[1]+1),') 4 | 5 | Version: $version 6 | Summary: Convenient and transparent local/remote incremental mirror/backup 7 | Name: rdiff-backup 8 | Release: 1 9 | Epoch: 0 10 | URL: http://www.nongnu.org/rdiff-backup/ 11 | Source: http://savannah.nongnu.org/download/rdiff-backup/rdiff-backup-%{version}.tar.gz 12 | License: GPL 13 | Group: Applications/Archiving 14 | BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) 15 | Requires: %{PYTHON_NAME} >= %{PYTHON_VERSION}, %{PYTHON_NAME} < %{NEXT_PYTHON_VERSION} 16 | BuildPrereq: %{PYTHON_NAME}-devel >= 2.2, librsync-devel >= 0.9.7 17 | 18 | %description 19 | rdiff-backup is a script, written in Python, that backs up one 20 | directory to another and is intended to be run periodically (nightly 21 | from cron for instance). The target directory ends up a copy of the 22 | source directory, but extra reverse diffs are stored in the target 23 | directory, so you can still recover files lost some time ago. The idea 24 | is to combine the best features of a mirror and an incremental 25 | backup. rdiff-backup can also operate in a bandwidth efficient manner 26 | over a pipe, like rsync. Thus you can use rdiff-backup and ssh to 27 | securely back a hard drive up to a remote location, and only the 28 | differences from the previous backup will be transmitted. 29 | 30 | %prep 31 | %setup -q 32 | 33 | %build 34 | %{PYTHON_NAME} setup.py build 35 | 36 | %install 37 | %{PYTHON_NAME} setup.py install --root $RPM_BUILD_ROOT 38 | # Produce .pyo files for %ghost directive later 39 | %{PYTHON_NAME} -Oc 'from compileall import *; compile_dir("'$RPM_BUILD_ROOT/%{_libdir}/python%{PYTHON_VERSION}/site-packages/rdiff_backup'")' 40 | rm -rf $RPM_BUILD_ROOT/usr/share/doc/* 41 | 42 | %clean 43 | rm -rf $RPM_BUILD_ROOT 44 | 45 | %files 46 | %defattr(-,root,root) 47 | %{_bindir}/rdiff-backup 48 | %{_bindir}/rdiff-backup-statistics 49 | %{_mandir}/man1/rdiff-backup* 50 | %dir %{_libdir}/python%{PYTHON_VERSION}/site-packages/rdiff_backup 51 | %{_libdir}/python%{PYTHON_VERSION}/site-packages/rdiff_backup/*.py 52 | %{_libdir}/python%{PYTHON_VERSION}/site-packages/rdiff_backup/*.pyc 53 | %{_libdir}/python%{PYTHON_VERSION}/site-packages/rdiff_backup/*.so 54 | %ghost %{_libdir}/python%{PYTHON_VERSION}/site-packages/rdiff_backup/*.pyo 55 | %doc CHANGELOG COPYING FAQ.html examples.html README 56 | 57 | %changelog 58 | * Wed Nov 15 2006 Gordon Rowell 1.1.7-1 59 | - Update URLs 60 | 61 | * Mon Aug 22 2005 Ben Escoto - 1.0.0-3 62 | - Matthijs van der Klip's patch to fix python2 support 63 | 64 | * Tue Aug 16 2005 Ben Escoto - 1.0.0-2 65 | - Removing /usr/share/doc in build for some obscure reason 66 | 67 | * Mon Mar 28 2005 Ben Escoto - 0.13.5-1 68 | - Set librsync >= 0.9.7 to encourage upgrade 69 | 70 | * Sat Dec 15 2003 Ben Escoto - 0.12.6-2 71 | - Readded python2/python code; turns out not everyone calls it python 72 | - A number of changes from Fedora rpm. 73 | 74 | * Thu Sep 11 2003 Ben Escoto - 0.12.4-1 75 | - Removed code that selected between python2 and python; I think 76 | everyone calls it python now. 77 | 78 | * Thu Aug 8 2003 Ben Escoto 79 | - Set librsync >= 0.9.6, because rsync.h renamed to librsync.h 80 | 81 | * Sun Jul 20 2003 Ben Escoto 82 | - Minor changes to comply with Fedora standards. 83 | 84 | * Sun Jan 19 2002 Troels Arvin 85 | - Builds, no matter if Python 2.2 is called python2-2.2 or python-2.2. 86 | 87 | * Sun Nov 4 2001 Ben Escoto 88 | - Initial RPM 89 | -------------------------------------------------------------------------------- /rdiff-backup/dist/rdiff-backup.spec.template-fedora: -------------------------------------------------------------------------------- 1 | %define PYTHON_VERSION %(python -c 'import sys; print sys.version[:3],') 2 | %define NEXT_PYTHON_VERSION %(python -c 'import sys; print "%d.%d" % (sys.version_info[0], sys.version_info[1]+1),') 3 | 4 | Version: $version 5 | Summary: Convenient and transparent local/remote incremental mirror/backup 6 | Name: rdiff-backup 7 | Release: 0.fdr.1 8 | Epoch: 0 9 | URL: http://www.nongnu.org/rdiff-backup/ 10 | Source: http://savannah.nongnu.org/download/rdiff-backup/rdiff-backup-%{version}.tar.gz 11 | License: GPL 12 | Group: Applications/Archiving 13 | BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) 14 | Requires: python >= 0:%{PYTHON_VERSION}, python < 0:%{NEXT_PYTHON_VERSION} 15 | BuildRequires: python-devel >= 0:2.2, librsync-devel >= 0:0.9.6 16 | 17 | %description 18 | rdiff-backup is a script, written in Python, that backs up one 19 | directory to another and is intended to be run periodically (nightly 20 | from cron for instance). The target directory ends up a copy of the 21 | source directory, but extra reverse diffs are stored in the target 22 | directory, so you can still recover files lost some time ago. The idea 23 | is to combine the best features of a mirror and an incremental 24 | backup. rdiff-backup can also operate in a bandwidth efficient manner 25 | over a pipe, like rsync. Thus you can use rdiff-backup and ssh to 26 | securely back a hard drive up to a remote location, and only the 27 | differences from the previous backup will be transmitted. 28 | 29 | %prep 30 | %setup -q 31 | 32 | %build 33 | python setup.py build 34 | 35 | %install 36 | python setup.py install --root $RPM_BUILD_ROOT 37 | # Produce .pyo files for %ghost directive later 38 | python -Oc 'from compileall import *; compile_dir("'$RPM_BUILD_ROOT/%{_libdir}/python%{PYTHON_VERSION}/site-packages/rdiff_backup'")' 39 | 40 | %clean 41 | rm -rf $RPM_BUILD_ROOT 42 | 43 | %files 44 | %defattr(-,root,root) 45 | %{_bindir}/rdiff-backup 46 | %{_bindir}/rdiff-backup-statistics 47 | %{_mandir}/man1/rdiff-backup* 48 | %dir %{_libdir}/python%{PYTHON_VERSION}/site-packages/rdiff_backup 49 | %{_libdir}/python%{PYTHON_VERSION}/site-packages/rdiff_backup/*.py 50 | %{_libdir}/python%{PYTHON_VERSION}/site-packages/rdiff_backup/*.pyc 51 | %{_libdir}/python%{PYTHON_VERSION}/site-packages/rdiff_backup/*.so 52 | %ghost %{_libdir}/python%{PYTHON_VERSION}/site-packages/rdiff_backup/*.pyo 53 | %doc CHANGELOG COPYING FAQ.html README 54 | 55 | %changelog 56 | * Wed Nov 15 2006 Gordon Rowell 0:1.1.7-0.fdr.1 57 | - update URLs 58 | 59 | * Sun Oct 05 2003 Ben Escoto - 0:0.12.5-0.fdr.1 60 | - Added epochs to python versions, more concise %%defines, %%ghost files 61 | 62 | * Thu Aug 16 2003 Ben Escoto - 0:0.12.3-0.fdr.4 63 | - Implemented various suggestions of Fedora QA 64 | 65 | * Sun Nov 4 2001 Ben Escoto 66 | - Initial RPM 67 | -------------------------------------------------------------------------------- /rdiff-backup/dist/setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import sys, os, getopt 4 | from distutils.core import setup, Extension 5 | 6 | version_string = "$version" 7 | 8 | if sys.version_info[:2] < (2,2): 9 | print "Sorry, rdiff-backup requires version 2.2 or later of python" 10 | sys.exit(1) 11 | 12 | # Defaults 13 | lflags_arg = [] 14 | libname = ['rsync'] 15 | incdir_list = libdir_list = None 16 | extra_options = {} 17 | 18 | if os.name == 'posix' or os.name == 'nt': 19 | LIBRSYNC_DIR = os.environ.get('LIBRSYNC_DIR', '') 20 | LFLAGS = os.environ.get('LFLAGS', []) 21 | LIBS = os.environ.get('LIBS', []) 22 | 23 | # Handle --librsync-dir=[PATH] and --lflags=[FLAGS] 24 | args = sys.argv[:] 25 | for arg in args: 26 | if arg.startswith('--librsync-dir='): 27 | LIBRSYNC_DIR = arg.split('=')[1] 28 | sys.argv.remove(arg) 29 | elif arg.startswith('--lflags='): 30 | LFLAGS = arg.split('=')[1].split() 31 | sys.argv.remove(arg) 32 | elif arg.startswith('--libs='): 33 | LIBS = arg.split('=')[1].split() 34 | sys.argv.remove(arg) 35 | 36 | if LFLAGS or LIBS: 37 | lflags_arg = LFLAGS + LIBS 38 | 39 | if LIBRSYNC_DIR: 40 | incdir_list = [os.path.join(LIBRSYNC_DIR, 'include')] 41 | libdir_list = [os.path.join(LIBRSYNC_DIR, 'lib')] 42 | if '-lrsync' in LIBS: 43 | libname = [] 44 | 45 | if os.name == 'nt': 46 | try: 47 | import py2exe 48 | except ImportError: 49 | pass 50 | else: 51 | extra_options = { 52 | 'console': ['rdiff-backup'], 53 | } 54 | if '--single-file' in sys.argv[1:]: 55 | sys.argv.remove('--single-file') 56 | extra_options.update({ 57 | 'options': {'py2exe': {'bundle_files': 1}}, 58 | 'zipfile': None 59 | }) 60 | 61 | setup(name="rdiff-backup", 62 | version=version_string, 63 | description="Local/remote mirroring+incremental backup", 64 | author="Ben Escoto", 65 | author_email="rdiff-backup@emerose.org", 66 | url="http://rdiff-backup.nongnu.org/", 67 | packages = ['rdiff_backup'], 68 | ext_modules = [Extension("rdiff_backup.C", ["cmodule.c"]), 69 | Extension("rdiff_backup._librsync", 70 | ["_librsyncmodule.c"], 71 | include_dirs=incdir_list, 72 | library_dirs=libdir_list, 73 | libraries=libname, 74 | extra_link_args=lflags_arg)], 75 | scripts = ['rdiff-backup', 'rdiff-backup-statistics'], 76 | data_files = [('share/man/man1', ['rdiff-backup.1', 77 | 'rdiff-backup-statistics.1']), 78 | ('share/doc/rdiff-backup-%s' % (version_string,), 79 | ['CHANGELOG', 'COPYING', 'README', 'FAQ.html'])], 80 | **extra_options) 81 | 82 | -------------------------------------------------------------------------------- /rdiff-backup/misc/compress-rdiff-backup-increments: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # 3 | # Compresses old rdiff-backup increments. See 4 | # http://www.stanford.edu/~bescoto/rdiff-backup for information on 5 | # rdiff-backup. 6 | 7 | from __future__ import nested_scopes, generators 8 | import os, sys, getopt 9 | 10 | rdiff_backup_location = "/usr/bin/rdiff-backup" 11 | no_compression_regexp_string = None 12 | __no_execute__ = 1 13 | 14 | def print_help(): 15 | """Print usage, exit""" 16 | print """ 17 | Usage: compress-rdiff-backup-increments [options] mirror_directory 18 | 19 | This script will compress the old rdiff-backup increments under 20 | mirror_directory, in a format compatible with rdiff-backup version 21 | 0.7.1 and later. So for instance if you were using an old version 22 | of rdiff-backup like this: 23 | 24 | rdiff-backup /foo /backup 25 | 26 | and now you want to take advantage of v0.7.1's space saving 27 | compression, you can run: 28 | 29 | compress-rdiff-backup-increments /backup 30 | 31 | 32 | Options: 33 | 34 | --rdiff-backup-location location 35 | This script reads your rdiff-backup executable. The default 36 | is "/usr/bin/rdiff-backup", so if your rdiff-backup is in a 37 | different location, you must use this switch. 38 | 39 | --no-compression-regexp regexp 40 | Any increments whose base name match this regular expression 41 | won't be compressed. This is generally used to avoid 42 | compressing already compressed files. See the rdiff-backup 43 | man page for the default. 44 | """ 45 | sys.exit(1) 46 | 47 | def parse_args(arglist): 48 | """Check and evaluate command line arguments, return dirname""" 49 | global rdiff_backup_location 50 | global no_compression_regexp_string 51 | try: optlist, args = getopt.getopt(arglist, "v:", 52 | ["rdiff-backup-location=", 53 | "no-compression-regexp="]) 54 | except getopt.error: print_help() 55 | 56 | for opt, arg in optlist: 57 | if opt == "--no-compression-regexp": 58 | no_compression_regexp_string = arg 59 | elif opt == "--rdiff-backup-location": rdiff_backup_location = arg 60 | else: 61 | print "Bad option: ", opt 62 | print_help() 63 | 64 | if len(args) != 1: 65 | print "Wrong number of arguments" 66 | print_help() 67 | return args[0] 68 | 69 | def exec_rdiff_backup(): 70 | """Execs rdiff-backup""" 71 | try: execfile(rdiff_backup_location, globals()) 72 | except IOError: 73 | print "Unable to read", rdiff_backup_location 74 | print "You may need to use the --rdiff-backup-location argument" 75 | sys.exit(1) 76 | 77 | if not map(int, Globals.version.split(".")) >= [0, 7, 1]: 78 | print "This script requires rdiff-backup version 0.7.1 or later,", 79 | print "found version", Globals.version 80 | sys.exit(1) 81 | 82 | def gzip_file(rp): 83 | """gzip rp, adding .gz to path and deleting original""" 84 | newrp = RPath(rp.conn, rp.base, rp.index[:-1] + (rp.index[-1]+".gz",)) 85 | if newrp.lstat(): 86 | print "Warning: %s already exists, skipping" % newrp.path 87 | return 88 | 89 | print "gzipping ", rp.path 90 | newrp.write_from_fileobj(rp.open("rb"), compress = 1) 91 | RPath.copy_attribs(rp, newrp) 92 | rp.delete() 93 | 94 | def Main(): 95 | dirname = parse_args(sys.argv[1:]) 96 | exec_rdiff_backup() 97 | if no_compression_regexp_string is not None: 98 | no_compression_regexp = re.compile(no_compression_regexp_string, re.I) 99 | else: no_compression_regexp = \ 100 | re.compile(Globals.no_compression_regexp_string, re.I) 101 | Globals.change_source_perms = 1 102 | Globals.change_ownership = (os.getuid() == 0) 103 | 104 | # Check to make sure rbdir exists 105 | root_rp = RPath(Globals.local_connection, dirname) 106 | rbdir = root_rp.append("rdiff-backup-data") 107 | if not rbdir.lstat(): 108 | print "Cannot find %s, exiting" % rbdir.path 109 | sys.exit(1) 110 | 111 | for dsrp in DestructiveStepping.Iterate_with_Finalizer(rbdir, 1): 112 | if (dsrp.isincfile() and dsrp.isreg() and 113 | not dsrp.isinccompressed() and 114 | (dsrp.getinctype() == "diff" or dsrp.getinctype() == "snapshot") 115 | and dsrp.getsize() != 0 and 116 | not no_compression_regexp.match(dsrp.getincbase_str())): 117 | gzip_file(dsrp) 118 | 119 | Main() 120 | 121 | -------------------------------------------------------------------------------- /rdiff-backup/misc/find2dirs: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | from __future__ import generators 4 | import sys, os, stat 5 | 6 | def usage(): 7 | print "Usage: find2dirs dir1 dir2" 8 | print 9 | print "Given the name of two directories, list all the files in both, one" 10 | print "per line, but don't repeat a file even if it is in both directories" 11 | sys.exit(1) 12 | 13 | def getlist(base, ext = ""): 14 | """Return iterator yielding filenames from directory""" 15 | if ext: yield ext 16 | else: yield "." 17 | 18 | fullname = os.path.join(base, ext) 19 | if stat.S_ISDIR(stat.S_IFMT(os.lstat(fullname)[stat.ST_MODE])): 20 | for subfile in os.listdir(fullname): 21 | for fn in getlist(base, os.path.join(ext, subfile)): yield fn 22 | 23 | def main(dir1, dir2): 24 | d = {} 25 | for fn in getlist(dir1): d[fn] = 1 26 | for fn in getlist(dir2): d[fn] = 1 27 | for fn in d.keys(): print fn 28 | 29 | if not len(sys.argv) == 3: usage() 30 | else: main(sys.argv[1], sys.argv[2]) 31 | -------------------------------------------------------------------------------- /rdiff-backup/misc/init_files.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """init_smallfiles.py 4 | 5 | This program makes a number of files of the given size in the 6 | specified directory. 7 | 8 | """ 9 | 10 | import os, stat, sys, math 11 | 12 | if len(sys.argv) > 5 or len(sys.argv) < 4: 13 | print "Usage: init_files [directory name] [file size] [file count] [base]" 14 | print 15 | print "Creates file_count files in directory_name of size file_size." 16 | print "The created directory has a tree type structure where each level" 17 | print "has at most base files or directories in it. Default is 50." 18 | sys.exit(1) 19 | 20 | dirname = sys.argv[1] 21 | filesize = int(sys.argv[2]) 22 | filecount = int(sys.argv[3]) 23 | block_size = 16384 24 | block = "." * block_size 25 | block_change = "." * (filesize % block_size) 26 | if len(sys.argv) == 4: base = 50 27 | else: base = int(sys.argv[4]) 28 | 29 | def make_file(path): 30 | """Make the file at path""" 31 | fp = open(path, "w") 32 | for i in xrange(int(math.floor(filesize/block_size))): fp.write(block) 33 | fp.write(block_change) 34 | fp.close() 35 | 36 | def find_sublevels(count): 37 | """Return number of sublevels required for count files""" 38 | return int(math.ceil(math.log(count)/math.log(base))) 39 | 40 | def make_dir(dir, count): 41 | """Make count files in the directory, making subdirectories if necessary""" 42 | print "Making directory %s with %d files" % (dir, count) 43 | os.mkdir(dir) 44 | level = find_sublevels(count) 45 | assert count <= pow(base, level) 46 | if level == 1: 47 | for i in range(count): make_file(os.path.join(dir, "file%d" %i)) 48 | else: 49 | files_per_subdir = pow(base, level-1) 50 | full_dirs = int(count/files_per_subdir) 51 | assert full_dirs <= base 52 | for i in range(full_dirs): 53 | make_dir(os.path.join(dir, "subdir%d" % i), files_per_subdir) 54 | 55 | change = count - full_dirs*files_per_subdir 56 | assert change >= 0 57 | if change > 0: 58 | make_dir(os.path.join(dir, "subdir%d" % full_dirs), change) 59 | 60 | def start(dir): 61 | try: os.stat(dir) 62 | except os.error: pass 63 | else: 64 | print "Directory %s already exists, exiting." % dir 65 | sys.exit(1) 66 | 67 | make_dir(dirname, filecount) 68 | 69 | start(dirname) 70 | -------------------------------------------------------------------------------- /rdiff-backup/misc/librsync-many-files.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """Use librsync to transform everything in one dir to another""" 4 | 5 | import sys, os, librsync 6 | 7 | dir1, dir2 = sys.argv[1:3] 8 | for i in xrange(1000): 9 | dir1fn = "%s/%s" % (dir1, i) 10 | dir2fn = "%s/%s" % (dir2, i) 11 | 12 | # Write signature file 13 | f1 = open(dir1fn, "rb") 14 | sigfile = open("sig", "wb") 15 | librsync.filesig(f1, sigfile, 2048) 16 | f1.close() 17 | sigfile.close() 18 | 19 | # Write delta file 20 | f2 = open(dir2fn, "r") 21 | sigfile = open("sig", "rb") 22 | deltafile = open("delta", "wb") 23 | librsync.filerdelta(sigfile, f2, deltafile) 24 | f2.close() 25 | sigfile.close() 26 | deltafile.close() 27 | 28 | # Write patched file 29 | f1 = open(dir1fn, "rb") 30 | newfile = open("%s/%s.out" % (dir1, i), "wb") 31 | deltafile = open("delta", "rb") 32 | librsync.filepatch(f1, deltafile, newfile) 33 | f1.close() 34 | deltafile.close() 35 | newfile.close() 36 | 37 | 38 | -------------------------------------------------------------------------------- /rdiff-backup/misc/make-many-data-files.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """Make 10000 files consisting of data 4 | 5 | Syntax: test.py directory_name number_of_files character filelength""" 6 | 7 | import sys, os 8 | 9 | dirname = sys.argv[1] 10 | num_files = int(sys.argv[2]) 11 | character = sys.argv[3] 12 | filelength = int(sys.argv[4]) 13 | 14 | os.mkdir(dirname) 15 | for i in xrange(num_files): 16 | fp = open("%s/%s" % (dirname, i), "w") 17 | fp.write(character * filelength) 18 | fp.close() 19 | 20 | fp = open("%s.big" % dirname, "w") 21 | fp.write(character * (filelength*num_files)) 22 | fp.close() 23 | -------------------------------------------------------------------------------- /rdiff-backup/misc/myrm: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | import sys, os 4 | #sys.path.insert(0, "../src") 5 | from rdiff_backup.rpath import * 6 | from rdiff_backup.connection import * 7 | from rdiff_backup import Globals 8 | 9 | lc = Globals.local_connection 10 | 11 | for filename in sys.argv[1:]: 12 | #print "Deleting %s" % filename 13 | rp = RPath(lc, filename) 14 | if rp.lstat(): rp.delete() 15 | #os.system("rm -rf " + rp.path) 16 | -------------------------------------------------------------------------------- /rdiff-backup/misc/rdiff-backup-delete.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """rdiff-backup-delete.py 4 | Script to delete a file or directory from rdiff-backup mirrors, including metadata and file statistics 5 | 6 | Copyright 2017 Sol1 Pty. Ltd. 7 | 8 | author(s): Wes Cilldhaire 9 | license: GPL v2 10 | """ 11 | 12 | import gzip, argparse, os, shutil, re, glob 13 | 14 | parser = argparse.ArgumentParser(description="Delete files or directories from rdiff-backup mirrors including metadata", epilog='') 15 | parser.add_argument('base', help='base directory of the backup (the one containing the rdiff-backup-data directory)') 16 | parser.add_argument('path', help='path of the file or directory to delete, relative to the base directory') 17 | parser.add_argument('-d','--dry', action='store_true', help='dry run - do not perform any action but print what would be done') 18 | 19 | base = parser.parse_args().base.rstrip('/') 20 | path = parser.parse_args().path.rstrip('/') 21 | dryrun = parser.parse_args().dry 22 | 23 | def check_type(base,path): 24 | """Determine whether specified path is a single file/link or directory that needs to be deleted recursively""" 25 | filetype = None 26 | p = base+'/'+path 27 | if os.path.lexists(p): 28 | if os.path.islink(p): 29 | filetype = 'sym' 30 | elif os.path.isdir(p): 31 | filetype = 'dir' 32 | else: 33 | filetype = 'file' 34 | return filetype 35 | 36 | def delete_entry(base,path): 37 | """Delete specified path, recursively if it is a directory. Fall through if it doesn't exist (ie if it was removed from the source or mirror prior to script execution)""" 38 | filetype = check_type(base,path) 39 | p = (base+'/'+path).replace('//','/') 40 | if filetype == None: 41 | # print "File/Directory '%s' doesn't appear to exist, skipping deletion" % p 42 | return 43 | if filetype == 'dir': 44 | print "dir:\t'%s' - deleting recursively" % p 45 | if not dryrun: 46 | shutil.rmtree(p) 47 | else: 48 | print "\t(dryrun, not deleting)" 49 | else: 50 | print "%s:\t'%s'" % (filetype,p) 51 | if not dryrun: 52 | os.remove(p) 53 | else: 54 | print "\t(dryrun, not deleted)" 55 | 56 | def remove_mirror(): 57 | """Delete specified file/dir from mirror""" 58 | global base 59 | global path 60 | global dryrun 61 | print "\nLooking for mirror in %s" % base 62 | delete_entry(base,path) 63 | 64 | def remove_increments(): 65 | """Delete all increments and dropfiles associated with specified file/dir""" 66 | global base 67 | global path 68 | global dryrun 69 | pattern = "^%s\\.[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.+?[0-9]{2}:[0-9]{2}\.(dir|missing|diff|diff\.gz)$" 70 | p = (base+'/rdiff-backup-data/increments' + '/' + '/'.join(path.split('/')[0:-1])).rstrip('/') 71 | print "\nLooking for increments in %s" % p 72 | increments = [] 73 | fname = path.split('/')[-1] 74 | list(map(lambda x: re.match(pattern % fname, x) and increments.append(x), os.listdir(p))) 75 | for inc in increments: 76 | delete_entry(p,inc) 77 | delete_entry(p,fname) 78 | 79 | def remove_metadata(): 80 | """Parse mirror metadata and remove references to file/dir""" 81 | global base 82 | global path 83 | global dryrun 84 | global filetype 85 | p = base + "/rdiff-backup-data" 86 | pattern_head = "^File\s+%s(/.*)?$" 87 | pattern_body = "^\s+.*$" 88 | print "\nLooking for mirror metadata in %s" % p 89 | metadata = [] 90 | list(map(lambda x: metadata.append(x.split('/')[-1]), glob.glob(base + '/rdiff-backup-data/mirror_metadata*'))) 91 | for meta in metadata: 92 | matchfound = 0 93 | print "file:\t%s" % meta 94 | fdin = gzip.GzipFile(p + '/' + meta, 'rb') 95 | fdout = gzip.GzipFile(p + '/' + 'temp_' + meta, 'wb', 9) 96 | switch = False 97 | for r in fdin: 98 | if re.match(pattern_head % path , r): 99 | switch = True 100 | matchfound += 1 101 | print "\t\t%s" % r.strip() 102 | elif switch: 103 | if not re.match(pattern_body, r): 104 | fdout.write(r) 105 | switch = False 106 | else: 107 | fdout.write(r) 108 | fdout.close() 109 | fdin.close() 110 | print "\t%s match%s found." % (matchfound, "" if matchfound == 1 else "es") 111 | if dryrun: 112 | print "\t(dryrun, not altered)\n" 113 | os.remove(p + '/' + 'temp_' + meta) 114 | else: 115 | print 116 | os.remove(p + '/' + meta) 117 | os.rename(p + '/' + 'temp_' + meta, p + '/' + meta) 118 | 119 | def remove_statistics(): 120 | """Parse file statistics and remove references to file/dir""" 121 | global base 122 | global path 123 | global dryrun 124 | global filetype 125 | p = base + "/rdiff-backup-data" 126 | pattern = "^%s(/[^\s]*)?(\s[^\s]+){4}$" 127 | print "\nLooking for statistics in %s" % p 128 | statistics = [] 129 | list(map(lambda x: statistics.append(x.split('/')[-1]), glob.glob(base + '/rdiff-backup-data/file_statistics*'))) 130 | for stats in statistics: 131 | matchfound = 0 132 | print "file:\t%s" % stats 133 | fdin = gzip.GzipFile(p + '/' + stats, 'rb') 134 | fdout = gzip.GzipFile(p + '/' + 'temp_' + stats, 'wb', 9) 135 | for r in fdin: 136 | if re.match(pattern % path, r): 137 | matchfound += 1 138 | print "\t\t%s" % r.strip() 139 | else: 140 | fdout.write(r) 141 | fdout.close() 142 | fdin.close() 143 | print "\t%s match%s found." % (matchfound, "" if matchfound == 1 else "es") 144 | if dryrun: 145 | print "\t(dryrun, not altered)\n" 146 | os.remove(p + '/' + 'temp_' + stats) 147 | else: 148 | print 149 | os.remove(p + '/' + stats) 150 | os.rename(p + '/' + 'temp_' + stats, p + '/' + stats) 151 | 152 | remove_mirror() 153 | remove_increments() 154 | remove_metadata() 155 | remove_statistics() 156 | -------------------------------------------------------------------------------- /rdiff-backup/misc/rdiff-many-files.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """Run rdiff to transform everything in one dir to another""" 4 | 5 | import sys, os 6 | 7 | dir1, dir2 = sys.argv[1:3] 8 | for i in xrange(1000): 9 | assert not os.system("rdiff signature %s/%s sig" % (dir1, i)) 10 | assert not os.system("rdiff delta sig %s/%s diff" % (dir2, i)) 11 | assert not os.system("rdiff patch %s/%s diff %s/%s.out" % 12 | (dir1, i, dir1, i)) 13 | 14 | 15 | -------------------------------------------------------------------------------- /rdiff-backup/misc/remove-comments.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """remove-comments.py 4 | 5 | Given a python program on standard input, spit one out on stdout that 6 | should work the same, but has blank and comment lines removed. 7 | 8 | """ 9 | 10 | import sys, re 11 | 12 | triple_regex = re.compile('"""') 13 | 14 | def eattriple(initial_line_stripped): 15 | """Keep reading until end of doc string""" 16 | assert initial_line_stripped.startswith('"""') 17 | if triple_regex.search(initial_line_stripped[3:]): return 18 | while 1: 19 | line = sys.stdin.readline() 20 | if not line or triple_regex.search(line): break 21 | 22 | while 1: 23 | line = sys.stdin.readline() 24 | if not line: break 25 | stripped = line.strip() 26 | if not stripped: continue 27 | if stripped[0] == "#": continue 28 | if stripped.startswith('"""'): 29 | eattriple(stripped) 30 | continue 31 | sys.stdout.write(line) 32 | 33 | -------------------------------------------------------------------------------- /rdiff-backup/python-rdiff: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import sys 4 | from rdiff_backup import librsync 5 | 6 | blocksize = 64*1024 # just used in copying 7 | librsync_blocksize = None # If not set, use defaults in _librsync module 8 | 9 | def usage(): 10 | """Print usage and then exit""" 11 | print """ 12 | Usage: %(cmd)s "signature" basis_file signature_file 13 | %(cmd)s "delta" signature_file new_file delta_file 14 | %(cmd)s "patch" basis_file delta_file new_file 15 | """ % {'cmd': sys.argv[0]} 16 | sys.exit(1) 17 | 18 | def copy_and_close(infp, outfp): 19 | """Copy file streams infp to outfp in blocks, closing when done""" 20 | while 1: 21 | buf = infp.read(blocksize) 22 | if not buf: break 23 | outfp.write(buf) 24 | assert not infp.close() and not outfp.close() 25 | 26 | def write_sig(input_path, output_path): 27 | """Open file at input_path, write signature to output_path""" 28 | infp = open(input_path, "rb") 29 | if librsync_blocksize: sigfp = librsync.SigFile(infp, librsync_blocksize) 30 | else: sigfp = librsync.SigFile(infp) 31 | copy_and_close(sigfp, open(output_path, "wb")) 32 | 33 | def write_delta(sig_path, new_path, output_path): 34 | """Read signature and new file, write to delta to delta_path""" 35 | deltafp = librsync.DeltaFile(open(sig_path, "rb"), open(new_path, "rb")) 36 | copy_and_close(deltafp, open(output_path, "wb")) 37 | 38 | def write_patch(basis_path, delta_path, out_path): 39 | """Patch file at basis_path with delta at delta_path, write to out_path""" 40 | patchfp = librsync.PatchedFile(open(basis_path, "rb"), 41 | open(delta_path, "rb")) 42 | copy_and_close(patchfp, open(out_path, "wb")) 43 | 44 | def check_different(filelist): 45 | """Make sure no files the same""" 46 | d = {} 47 | for file in filelist: d[file] = file 48 | assert len(d) == len(filelist), "Error, must use all different filenames" 49 | 50 | def Main(): 51 | """Run program""" 52 | if len(sys.argv) < 4: usage() 53 | mode = sys.argv[1] 54 | file_args = sys.argv[2:] 55 | check_different(file_args) 56 | if mode == "signature": 57 | if len(file_args) != 2: usage() 58 | write_sig(*file_args) 59 | elif mode == "delta": 60 | if len(file_args) != 3: usage() 61 | write_delta(*file_args) 62 | elif mode == "patch": 63 | if len(file_args) != 3: usage() 64 | write_patch(*file_args) 65 | else: usage() 66 | 67 | 68 | if __name__ == "__main__": Main() 69 | 70 | -------------------------------------------------------------------------------- /rdiff-backup/rdiff-backup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # rdiff-backup -- Mirror files while keeping incremental changes 3 | # Version $version released $date 4 | # Copyright (C) 2001-2005 Ben Escoto 5 | # 6 | # This program is licensed under the GNU General Public License (GPL). 7 | # you can redistribute it and/or modify it under the terms of the GNU 8 | # General Public License as published by the Free Software Foundation, 9 | # Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA; either 10 | # version 2 of the License, or (at your option) any later version. 11 | # Distributions of rdiff-backup should include a copy of the GPL in a 12 | # file called COPYING. The GPL is also available online at 13 | # http://www.gnu.org/copyleft/gpl.html. 14 | # 15 | # See http://rdiff-backup.nongnu.org/ for more information. Please 16 | # send mail to me or the mailing list if you find bugs or have any 17 | # suggestions. 18 | 19 | import sys 20 | import rdiff_backup.Main 21 | 22 | try: 23 | import msvcrt, os 24 | msvcrt.setmode(0, os.O_BINARY) 25 | msvcrt.setmode(1, os.O_BINARY) 26 | except ImportError: 27 | pass 28 | 29 | if __name__ == "__main__" and not globals().has_key('__no_execute__'): 30 | rdiff_backup.Main.error_check_Main(sys.argv[1:]) 31 | 32 | 33 | -------------------------------------------------------------------------------- /rdiff-backup/rdiff-backup-statistics.1: -------------------------------------------------------------------------------- 1 | .TH RDIFF-BACKUP-STATISTICS 1 "NOVEMBER 2006" "Version 1.1.6" "User Manuals" \" -*- nroff -*- 2 | .SH NAME 3 | rdiff-backup-statistics \- summarize rdiff-backup statistics files 4 | .SH SYNOPSIS 5 | .B rdiff-backup-statistics 6 | .BI [\-\-begin-time " time" ] 7 | .BI [\-\-end-time " time" ] 8 | .BI [\-\-minimum-ratio " ratio" ] 9 | .B [\-\-null-separator] 10 | .B [\-\-quiet] 11 | .I repository 12 | 13 | .SH DESCRIPTION 14 | .BI rdiff-backup-statistics 15 | reads the matching statistics files in a backup repository made by 16 | .B rdiff-backup 17 | and prints some summary statistics to the screen. It does not alter 18 | the repository in any way. 19 | 20 | The required argument is the pathname of the root of an rdiff-backup 21 | repository. For instance, if you ran "rdiff-backup in out", you could 22 | later run "rdiff-backup-statistics out". 23 | 24 | The output has two parts. The first is simply an average of the all 25 | matching session_statistics files. The meaning of these fields is 26 | explained in the FAQ included in the package, and also at 27 | .IR http://rdiff-backup.nongnu.org/FAQ.html#statistics . 28 | 29 | The second section lists some particularly significant files 30 | (including directories). These files are either contain a lot of 31 | data, take up increment space, or contain a lot of changed files. All 32 | the files that are above the minimum ratio (default 5%) will be 33 | listed. 34 | 35 | If a file or directory is listed, its contributions are subtracted 36 | from its parent. That is why the percentage listed after a directory 37 | can be larger than the percentage of its parent. Without this, the 38 | root directory would always be the largest, and the output would be 39 | boring. 40 | 41 | .SH OPTIONS 42 | .TP 43 | .BI \-\-begin-time " time" 44 | Do not read statistics files older than 45 | .IR time . 46 | By default, all statistics files will be read. 47 | .I time 48 | should be in the same format taken by \-\-restore-as-of. (See 49 | .B TIME FORMATS 50 | in the rdiff-backup man page for details.) 51 | .TP 52 | .BI \-\-end-time " time" 53 | Like 54 | .B \-\-begin-time 55 | but exclude statistics files later than 56 | .IR time . 57 | .TP 58 | .BI \-\-minimum-ratio " ratio" 59 | Print all directories contributing more than the given ratio to the 60 | total. The default value is .05, or 5 percent. 61 | .TP 62 | .B \-\-null-separator 63 | Specify that the lines of the file_statistics file are separated by 64 | nulls (\\0). The default is to assume that newlines separate. Use 65 | this switch if rdiff-backup was run with the \-\-null-separator when 66 | making the given repository. 67 | .TP 68 | .B \-\-quiet 69 | Suppress printing of the "Processing statistics from session..." 70 | output lines. 71 | 72 | .SH BUGS 73 | When aggregating multiple statistics files, some directories above 74 | (but close to) the minimum ratio may not be displayed. For this 75 | reason, you may want to set the minimum-ratio lower than need. 76 | 77 | .SH AUTHOR 78 | Ben Escoto , based on original script by Dean Gaudet. 79 | 80 | .SH SEE ALSO 81 | .BR rdiff-backup (1), 82 | .BR python (1). 83 | The rdiff-backup web page is at 84 | .IR http://rdiff-backup.nongnu.org/ . 85 | -------------------------------------------------------------------------------- /rdiff-backup/rdiff_backup/Hardlink.py: -------------------------------------------------------------------------------- 1 | # Copyright 2002 2005 Ben Escoto 2 | # 3 | # This file is part of rdiff-backup. 4 | # 5 | # rdiff-backup is free software; you can redistribute it and/or modify 6 | # under the terms of the GNU General Public License as published by the 7 | # Free Software Foundation; either version 2 of the License, or (at your 8 | # option) any later version. 9 | # 10 | # rdiff-backup is distributed in the hope that it will be useful, but 11 | # WITHOUT ANY WARRANTY; without even the implied warranty of 12 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 13 | # General Public License for more details. 14 | # 15 | # You should have received a copy of the GNU General Public License 16 | # along with rdiff-backup; if not, write to the Free Software 17 | # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 18 | # USA 19 | 20 | """Preserve and restore hard links 21 | 22 | If the preserve_hardlinks option is selected, linked files in the 23 | source directory will be linked in the mirror directory. Linked files 24 | are treated like any other with respect to incrementing, but their 25 | link status can be retrieved because their device location and inode # 26 | is written in the metadata file. 27 | 28 | All these functions are meant to be executed on the mirror side. The 29 | source side should only transmit inode information. 30 | 31 | """ 32 | 33 | from __future__ import generators 34 | import Globals, Time, log, robust, errno 35 | 36 | # The keys in this dictionary are (inode, devloc) pairs. The values 37 | # are a pair (index, remaining_links, dest_key, sha1sum) where index 38 | # is the rorp index of the first such linked file, remaining_links is 39 | # the number of files hard linked to this one we may see, and key is 40 | # either (dest_inode, dest_devloc) or None, and represents the 41 | # hardlink info of the existing file on the destination. Finally 42 | # sha1sum is the hash of the file if it exists, or None. 43 | _inode_index = None 44 | 45 | def initialize_dictionaries(): 46 | """Set all the hard link dictionaries to empty""" 47 | global _inode_index 48 | _inode_index = {} 49 | 50 | def clear_dictionaries(): 51 | """Delete all dictionaries""" 52 | global _inode_index 53 | _inode_index = None 54 | 55 | def get_inode_key(rorp): 56 | """Return rorp's key for _inode_ dictionaries""" 57 | return (rorp.getinode(), rorp.getdevloc()) 58 | 59 | def add_rorp(rorp, dest_rorp = None): 60 | """Process new rorp and update hard link dictionaries""" 61 | if not rorp.isreg() or rorp.getnumlinks() < 2: return None 62 | rp_inode_key = get_inode_key(rorp) 63 | if not _inode_index.has_key(rp_inode_key): 64 | if not dest_rorp: dest_key = None 65 | elif dest_rorp.getnumlinks() == 1: dest_key = "NA" 66 | else: dest_key = get_inode_key(dest_rorp) 67 | digest = rorp.has_sha1() and rorp.get_sha1() or None 68 | _inode_index[rp_inode_key] = (rorp.index, rorp.getnumlinks(), 69 | dest_key, digest) 70 | return rp_inode_key 71 | 72 | def del_rorp(rorp): 73 | """Remove rorp information from dictionary if seen all links""" 74 | if not rorp.isreg() or rorp.getnumlinks() < 2: return 75 | rp_inode_key = get_inode_key(rorp) 76 | val = _inode_index.get(rp_inode_key) 77 | if not val: return 78 | index, remaining, dest_key, digest = val 79 | if remaining == 1: 80 | del _inode_index[rp_inode_key] 81 | return 1 82 | else: 83 | _inode_index[rp_inode_key] = (index, remaining-1, dest_key, digest) 84 | return 0 85 | 86 | def rorp_eq(src_rorp, dest_rorp): 87 | """Compare hardlinked for equality 88 | 89 | Return false if dest_rorp is linked differently, which can happen 90 | if dest is linked more than source, or if it is represented by a 91 | different inode. 92 | 93 | """ 94 | if (not src_rorp.isreg() or not dest_rorp.isreg() or 95 | src_rorp.getnumlinks() == dest_rorp.getnumlinks() == 1): 96 | return 1 # Hard links don't apply 97 | 98 | if src_rorp.getnumlinks() < dest_rorp.getnumlinks(): return 0 99 | src_key = get_inode_key(src_rorp) 100 | index, remaining, dest_key, digest = _inode_index[src_key] 101 | if dest_key == "NA": 102 | # Allow this to be ok for first comparison, but not any 103 | # subsequent ones 104 | _inode_index[src_key] = (index, remaining, None, None) 105 | return 1 106 | try: 107 | return dest_key == get_inode_key(dest_rorp) 108 | except KeyError: 109 | return 0 # Inode key might be missing if the metadata file is corrupt 110 | 111 | def islinked(rorp): 112 | """True if rorp's index is already linked to something on src side""" 113 | if not rorp.getnumlinks() > 1: return 0 114 | dict_val = _inode_index.get(get_inode_key(rorp)) 115 | if not dict_val: return 0 116 | return dict_val[0] != rorp.index # If equal, then rorp is first 117 | 118 | def get_link_index(rorp): 119 | """Return first index on target side rorp is already linked to""" 120 | return _inode_index[get_inode_key(rorp)][0] 121 | 122 | def get_sha1(rorp): 123 | """Return sha1 digest of what rorp is linked to""" 124 | return _inode_index[get_inode_key(rorp)][3] 125 | 126 | def link_rp(diff_rorp, dest_rpath, dest_root = None): 127 | """Make dest_rpath into a link using link flag in diff_rorp""" 128 | if not dest_root: dest_root = dest_rpath # use base of dest_rpath 129 | dest_link_rpath = dest_root.new_index(diff_rorp.get_link_flag()) 130 | try: dest_rpath.hardlink(dest_link_rpath.path) 131 | except EnvironmentError, exc: 132 | # This can happen if the source of dest_link_rpath was deleted 133 | # after it's linking info was recorded but before 134 | # dest_link_rpath was written. 135 | if errno.errorcode[exc[0]] == 'ENOENT': 136 | dest_rpath.touch() # This will cause an UpdateError later 137 | else: raise Exception("EnvironmentError '%s' linking %s to %s" % 138 | (exc, dest_rpath.path, dest_link_rpath.path)) 139 | 140 | -------------------------------------------------------------------------------- /rdiff-backup/rdiff_backup/Rdiff.py: -------------------------------------------------------------------------------- 1 | # Copyright 2002 2005 Ben Escoto 2 | # 3 | # This file is part of rdiff-backup. 4 | # 5 | # rdiff-backup is free software; you can redistribute it and/or modify 6 | # under the terms of the GNU General Public License as published by the 7 | # Free Software Foundation; either version 2 of the License, or (at your 8 | # option) any later version. 9 | # 10 | # rdiff-backup is distributed in the hope that it will be useful, but 11 | # WITHOUT ANY WARRANTY; without even the implied warranty of 12 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 13 | # General Public License for more details. 14 | # 15 | # You should have received a copy of the GNU General Public License 16 | # along with rdiff-backup; if not, write to the Free Software 17 | # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 18 | # USA 19 | 20 | """Invoke rdiff utility to make signatures, deltas, or patch""" 21 | 22 | import os, librsync 23 | import Globals, log, static, TempFile, rpath, hash 24 | 25 | 26 | def get_signature(rp, blocksize = None): 27 | """Take signature of rpin file and return in file object""" 28 | if not blocksize: blocksize = find_blocksize(rp.getsize()) 29 | log.Log("Getting signature of %s with blocksize %s" % 30 | (rp.get_indexpath(), blocksize), 7) 31 | return librsync.SigFile(rp.open("rb"), blocksize) 32 | 33 | def find_blocksize(file_len): 34 | """Return a reasonable block size to use on files of length file_len 35 | 36 | If the block size is too big, deltas will be bigger than is 37 | necessary. If the block size is too small, making deltas and 38 | patching can take a really long time. 39 | 40 | """ 41 | if file_len < 4096: return 64 # set minimum of 64 bytes 42 | else: # Use square root, rounding to nearest 16 43 | return long(pow(file_len, 0.5)/16)*16 44 | 45 | def get_delta_sigfileobj(sig_fileobj, rp_new): 46 | """Like get_delta but signature is in a file object""" 47 | log.Log("Getting delta of %s with signature stream" % (rp_new.path,), 7) 48 | return librsync.DeltaFile(sig_fileobj, rp_new.open("rb")) 49 | 50 | def get_delta_sigrp(rp_signature, rp_new): 51 | """Take signature rp and new rp, return delta file object""" 52 | log.Log("Getting delta of %s with signature %s" % 53 | (rp_new.path, rp_signature.get_indexpath()), 7) 54 | return librsync.DeltaFile(rp_signature.open("rb"), rp_new.open("rb")) 55 | 56 | def get_delta_sigrp_hash(rp_signature, rp_new): 57 | """Like above but also calculate hash of new as close() value""" 58 | log.Log("Getting delta (with hash) of %s with signature %s" % 59 | (rp_new.path, rp_signature.get_indexpath()), 7) 60 | return librsync.DeltaFile(rp_signature.open("rb"), 61 | hash.FileWrapper(rp_new.open("rb"))) 62 | 63 | 64 | def write_delta(basis, new, delta, compress = None): 65 | """Write rdiff delta which brings basis to new""" 66 | log.Log("Writing delta %s from %s -> %s" % 67 | (basis.path, new.path, delta.path), 7) 68 | deltafile = librsync.DeltaFile(get_signature(basis), new.open("rb")) 69 | delta.write_from_fileobj(deltafile, compress) 70 | 71 | def write_patched_fp(basis_fp, delta_fp, out_fp): 72 | """Write patched file to out_fp given input fps. Closes input files""" 73 | rpath.copyfileobj(librsync.PatchedFile(basis_fp, delta_fp), out_fp) 74 | assert not basis_fp.close() and not delta_fp.close() 75 | 76 | def write_via_tempfile(fp, rp): 77 | """Write fileobj fp to rp by writing to tempfile and renaming""" 78 | tf = TempFile.new(rp) 79 | retval = tf.write_from_fileobj(fp) 80 | rpath.rename(tf, rp) 81 | return retval 82 | 83 | def patch_local(rp_basis, rp_delta, outrp = None, delta_compressed = None): 84 | """Patch routine that must be run locally, writes to outrp 85 | 86 | This should be run local to rp_basis because it needs to be a real 87 | file (librsync may need to seek around in it). If outrp is None, 88 | patch rp_basis instead. 89 | 90 | The return value is the close value of the delta, so it can be 91 | used to produce hashes. 92 | 93 | """ 94 | assert rp_basis.conn is Globals.local_connection 95 | if delta_compressed: deltafile = rp_delta.open("rb", 1) 96 | else: deltafile = rp_delta.open("rb") 97 | patchfile = librsync.PatchedFile(rp_basis.open("rb"), deltafile) 98 | if outrp: return outrp.write_from_fileobj(patchfile) 99 | else: return write_via_tempfile(patchfile, rp_basis) 100 | 101 | def copy_local(rpin, rpout, rpnew = None): 102 | """Write rpnew == rpin using rpout as basis. rpout and rpnew local""" 103 | assert rpout.conn is Globals.local_connection 104 | deltafile = rpin.conn.librsync.DeltaFile(get_signature(rpout), 105 | rpin.open("rb")) 106 | patched_file = librsync.PatchedFile(rpout.open("rb"), deltafile) 107 | 108 | if rpnew: rpnew.write_from_fileobj(patched_file) 109 | else: write_via_tempfile(patched_file, rpout) 110 | 111 | 112 | 113 | -------------------------------------------------------------------------------- /rdiff-backup/rdiff_backup/TempFile.py: -------------------------------------------------------------------------------- 1 | # Copyright 2002 Ben Escoto 2 | # 3 | # This file is part of rdiff-backup. 4 | # 5 | # rdiff-backup is free software; you can redistribute it and/or modify 6 | # under the terms of the GNU General Public License as published by the 7 | # Free Software Foundation; either version 2 of the License, or (at your 8 | # option) any later version. 9 | # 10 | # rdiff-backup is distributed in the hope that it will be useful, but 11 | # WITHOUT ANY WARRANTY; without even the implied warranty of 12 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 13 | # General Public License for more details. 14 | # 15 | # You should have received a copy of the GNU General Public License 16 | # along with rdiff-backup; if not, write to the Free Software 17 | # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 18 | # USA 19 | 20 | """Manage temp files 21 | 22 | Earlier this had routines for keeping track of existing tempfiles. 23 | Now we just use normal rpaths instead of the TempFile class. 24 | 25 | """ 26 | 27 | import os 28 | import Globals, rpath 29 | 30 | # To make collisions less likely, this gets put in the file name 31 | # and incremented whenever a new file is requested. 32 | _tfindex = 0 33 | 34 | def new(rp_base): 35 | """Return new tempfile that isn't in use in same dir as rp_base""" 36 | return new_in_dir(rp_base.get_parent_rp()) 37 | 38 | def new_in_dir(dir_rp): 39 | """Return new temp rpath in directory dir_rp""" 40 | global _tfindex 41 | assert dir_rp.conn is Globals.local_connection 42 | while 1: 43 | if _tfindex > 100000000: 44 | Log("Warning: Resetting tempfile index", 2) 45 | _tfindex = 0 46 | tf = dir_rp.append('rdiff-backup.tmp.%d' % _tfindex) 47 | _tfindex = _tfindex+1 48 | if not tf.lstat(): return tf 49 | 50 | 51 | 52 | 53 | class TempFile(rpath.RPath): 54 | """Like an RPath, but keep track of which ones are still here""" 55 | def rename(self, rp_dest): 56 | """Rename temp file to permanent location, possibly overwriting""" 57 | if not self.lstat(): # "Moving" empty file, so just delete 58 | if rp_dest.lstat(): rp_dest.delete() 59 | remove_listing(self) 60 | return 61 | 62 | if self.isdir() and not rp_dest.isdir(): 63 | # Cannot move a directory directly over another file 64 | rp_dest.delete() 65 | rpath.rename(self, rp_dest) 66 | 67 | # Sometimes this just seems to fail silently, as in one 68 | # hardlinked twin is moved over the other. So check to make 69 | # sure below. 70 | self.setdata() 71 | if self.lstat(): 72 | rp_dest.delete() 73 | rpath.rename(self, rp_dest) 74 | self.setdata() 75 | if self.lstat(): raise OSError("Cannot rename tmp file correctly") 76 | remove_listing(self) 77 | 78 | def delete(self): 79 | rpath.RPath.delete(self) 80 | remove_listing(self) 81 | 82 | -------------------------------------------------------------------------------- /rdiff-backup/rdiff_backup/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Seravo/rdiff-backup/8ccc5a3b44c996ecd810f8d5d586d0da6435cc32/rdiff-backup/rdiff_backup/__init__.py -------------------------------------------------------------------------------- /rdiff-backup/rdiff_backup/cmodule.c: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Seravo/rdiff-backup/8ccc5a3b44c996ecd810f8d5d586d0da6435cc32/rdiff-backup/rdiff_backup/cmodule.c -------------------------------------------------------------------------------- /rdiff-backup/rdiff_backup/compilec.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import sys, os 4 | from distutils.core import setup, Extension 5 | 6 | assert len(sys.argv) == 1 7 | sys.argv.append("build") 8 | 9 | setup(name="CModule", 10 | version="0.9.0", 11 | description="rdiff-backup's C component", 12 | ext_modules=[Extension("C", ["cmodule.c"]), 13 | Extension("_librsync", ["_librsyncmodule.c"], 14 | libraries=["rsync"])]) 15 | 16 | def get_libraries(): 17 | """Return filename of C.so and _librsync.so files""" 18 | build_files = os.listdir("build") 19 | lib_dirs = filter(lambda x: x.startswith("lib"), build_files) 20 | assert len(lib_dirs) == 1, "No library directory or too many" 21 | libdir = lib_dirs[0] 22 | if sys.platform == "cygwin" or os.name == "nt": libext = "dll" 23 | else: libext = "so" 24 | clib = os.path.join("build", libdir, "C." + libext) 25 | rsynclib = os.path.join("build", libdir, "_librsync." + libext) 26 | try: 27 | os.lstat(clib) 28 | os.lstat(rsynclib) 29 | except os.error: 30 | print "Library file missing" 31 | sys.exit(1) 32 | return clib, rsynclib 33 | 34 | for filename in get_libraries(): 35 | assert not os.system("mv '%s' ." % (filename,)) 36 | assert not os.system("rm -rf build") 37 | -------------------------------------------------------------------------------- /rdiff-backup/rdiff_backup/hash.py: -------------------------------------------------------------------------------- 1 | # Copyright 2005 Ben Escoto 2 | # 3 | # This file is part of rdiff-backup. 4 | # 5 | # rdiff-backup is free software; you can redistribute it and/or modify 6 | # under the terms of the GNU General Public License as published by the 7 | # Free Software Foundation; either version 2 of the License, or (at your 8 | # option) any later version. 9 | # 10 | # rdiff-backup is distributed in the hope that it will be useful, but 11 | # WITHOUT ANY WARRANTY; without even the implied warranty of 12 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 13 | # General Public License for more details. 14 | # 15 | # You should have received a copy of the GNU General Public License 16 | # along with rdiff-backup; if not, write to the Free Software 17 | # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 18 | # USA 19 | 20 | """Contains a file wrapper that returns a hash on close""" 21 | 22 | # Until rdiff-backup is ported to Python 3 (or abandons support for versions 23 | # below Python 2.5), we'll ignore the warning about the deprecated sha module 24 | import warnings 25 | warnings.filterwarnings("ignore", ".*sha module.*", DeprecationWarning) 26 | 27 | import sha 28 | import Globals 29 | 30 | class FileWrapper: 31 | """Wrapper around a file-like object 32 | 33 | Only use this with files that will be read through in a single 34 | pass and then closed. (There is no seek().) When you close it, 35 | return value will be a Report. 36 | 37 | Currently this just calculates a sha1sum of the datastream. 38 | 39 | """ 40 | def __init__(self, fileobj): 41 | self.fileobj = fileobj 42 | self.sha1 = sha.new() 43 | self.closed = 0 44 | 45 | def read(self, length = -1): 46 | assert not self.closed 47 | buf = self.fileobj.read(length) 48 | self.sha1.update(buf) 49 | return buf 50 | 51 | def close(self): 52 | return Report(self.fileobj.close(), self.sha1.hexdigest()) 53 | 54 | 55 | class Report: 56 | """Hold final information about a byte stream""" 57 | def __init__(self, close_val, sha1_digest): 58 | assert not close_val # For now just assume inner file closes correctly 59 | self.sha1_digest = sha1_digest 60 | 61 | 62 | def compute_sha1(rp, compressed = 0): 63 | """Return the hex sha1 hash of given rpath""" 64 | assert rp.conn is Globals.local_connection # inefficient not to do locally 65 | digest = compute_sha1_fp(rp.open("rb", compressed)) 66 | rp.set_sha1(digest) 67 | return digest 68 | 69 | def compute_sha1_fp(fp, compressed = 0): 70 | """Return hex sha1 hash of given file-like object""" 71 | blocksize = Globals.blocksize 72 | fw = FileWrapper(fp) 73 | while 1: 74 | if not fw.read(blocksize): break 75 | return fw.close().sha1_digest 76 | 77 | 78 | -------------------------------------------------------------------------------- /rdiff-backup/rdiff_backup/increment.py: -------------------------------------------------------------------------------- 1 | # Copyright 2002 Ben Escoto 2 | # 3 | # This file is part of rdiff-backup. 4 | # 5 | # rdiff-backup is free software; you can redistribute it and/or modify 6 | # under the terms of the GNU General Public License as published by the 7 | # Free Software Foundation; either version 2 of the License, or (at your 8 | # option) any later version. 9 | # 10 | # rdiff-backup is distributed in the hope that it will be useful, but 11 | # WITHOUT ANY WARRANTY; without even the implied warranty of 12 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 13 | # General Public License for more details. 14 | # 15 | # You should have received a copy of the GNU General Public License 16 | # along with rdiff-backup; if not, write to the Free Software 17 | # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 18 | # USA 19 | 20 | """Provides functions and *ITR classes, for writing increment files""" 21 | 22 | import Globals, Time, rpath, Rdiff, log, statistics, robust 23 | 24 | 25 | def Increment(new, mirror, incpref): 26 | """Main file incrementing function, returns inc file created 27 | 28 | new is the file on the active partition, 29 | mirror is the mirrored file from the last backup, 30 | incpref is the prefix of the increment file. 31 | 32 | This function basically moves the information about the mirror 33 | file to incpref. 34 | 35 | """ 36 | log.Log("Incrementing mirror file " + mirror.path, 5) 37 | if ((new and new.isdir()) or mirror.isdir()) and not incpref.lstat(): 38 | incpref.mkdir() 39 | 40 | if not mirror.lstat(): incrp = makemissing(incpref) 41 | elif mirror.isdir(): incrp = makedir(mirror, incpref) 42 | elif new.isreg() and mirror.isreg(): 43 | incrp = makediff(new, mirror, incpref) 44 | else: incrp = makesnapshot(mirror, incpref) 45 | statistics.process_increment(incrp) 46 | return incrp 47 | 48 | def makemissing(incpref): 49 | """Signify that mirror file was missing""" 50 | incrp = get_inc(incpref, "missing") 51 | incrp.touch() 52 | return incrp 53 | 54 | def iscompressed(mirror): 55 | """Return true if mirror's increments should be compressed""" 56 | return (Globals.compression and 57 | not Globals.no_compression_regexp.match(mirror.path)) 58 | 59 | def makesnapshot(mirror, incpref): 60 | """Copy mirror to incfile, since new is quite different""" 61 | compress = iscompressed(mirror) 62 | if compress and mirror.isreg(): 63 | snapshotrp = get_inc(incpref, "snapshot.gz") 64 | else: snapshotrp = get_inc(incpref, "snapshot") 65 | 66 | if mirror.isspecial(): # check for errors when creating special increments 67 | eh = robust.get_error_handler("SpecialFileError") 68 | if robust.check_common_error(eh, rpath.copy_with_attribs, 69 | (mirror, snapshotrp, compress)) == 0: 70 | snapshotrp.setdata() 71 | if snapshotrp.lstat(): snapshotrp.delete() 72 | snapshotrp.touch() 73 | else: rpath.copy_with_attribs(mirror, snapshotrp, compress) 74 | return snapshotrp 75 | 76 | def makediff(new, mirror, incpref): 77 | """Make incfile which is a diff new -> mirror""" 78 | compress = iscompressed(mirror) 79 | if compress: diff = get_inc(incpref, "diff.gz") 80 | else: diff = get_inc(incpref, "diff") 81 | 82 | old_new_perms, old_mirror_perms = (None, None) 83 | 84 | if Globals.process_uid != 0: 85 | # Check for unreadable files 86 | if not new.readable(): 87 | old_new_perms = new.getperms() 88 | new.chmod(0400 | old_new_perms) 89 | if not mirror.readable(): 90 | old_mirror_perms = mirror.getperms() 91 | mirror.chmod(0400 | old_mirror_perms) 92 | 93 | Rdiff.write_delta(new, mirror, diff, compress) 94 | 95 | if old_new_perms: new.chmod(old_new_perms) 96 | if old_mirror_perms: mirror.chmod(old_mirror_perms) 97 | 98 | rpath.copy_attribs_inc(mirror, diff) 99 | return diff 100 | 101 | def makedir(mirrordir, incpref): 102 | """Make file indicating directory mirrordir has changed""" 103 | dirsign = get_inc(incpref, "dir") 104 | dirsign.touch() 105 | rpath.copy_attribs_inc(mirrordir, dirsign) 106 | return dirsign 107 | 108 | def get_inc(rp, typestr, time = None): 109 | """Return increment like rp but with time and typestr suffixes 110 | 111 | To avoid any quoting, the returned rpath has empty index, and the 112 | whole filename is in the base (which is not quoted). 113 | 114 | """ 115 | if time is None: time = Time.prevtime 116 | addtostr = lambda s: "%s.%s.%s" % (s, Time.timetostring(time), typestr) 117 | if rp.index: 118 | incrp = rp.__class__(rp.conn, rp.base, rp.index[:-1] + 119 | (addtostr(rp.index[-1]),)) 120 | else: 121 | dirname, basename = rp.dirsplit() 122 | incrp = rp.__class__(rp.conn, dirname, (addtostr(basename),)) 123 | assert not incrp.lstat(), incrp 124 | return incrp 125 | 126 | 127 | -------------------------------------------------------------------------------- /rdiff-backup/rdiff_backup/lazy.py: -------------------------------------------------------------------------------- 1 | # Copyright 2002 Ben Escoto 2 | # 3 | # This file is part of rdiff-backup. 4 | # 5 | # rdiff-backup is free software; you can redistribute it and/or modify 6 | # under the terms of the GNU General Public License as published by the 7 | # Free Software Foundation; either version 2 of the License, or (at your 8 | # option) any later version. 9 | # 10 | # rdiff-backup is distributed in the hope that it will be useful, but 11 | # WITHOUT ANY WARRANTY; without even the implied warranty of 12 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 13 | # General Public License for more details. 14 | # 15 | # You should have received a copy of the GNU General Public License 16 | # along with rdiff-backup; if not, write to the Free Software 17 | # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 18 | # USA 19 | 20 | """Define some lazy data structures and functions acting on them""" 21 | 22 | from __future__ import generators 23 | import os, stat, types 24 | import static 25 | 26 | 27 | class Iter: 28 | """Hold static methods for the manipulation of lazy iterators""" 29 | 30 | def filter(predicate, iterator): 31 | """Like filter in a lazy functional programming language""" 32 | for i in iterator: 33 | if predicate(i): yield i 34 | 35 | def map(function, iterator): 36 | """Like map in a lazy functional programming language""" 37 | for i in iterator: yield function(i) 38 | 39 | def foreach(function, iterator): 40 | """Run function on each element in iterator""" 41 | for i in iterator: function(i) 42 | 43 | def cat(*iters): 44 | """Lazily concatenate iterators""" 45 | for iter in iters: 46 | for i in iter: yield i 47 | 48 | def cat2(iter_of_iters): 49 | """Lazily concatenate iterators, iterated by big iterator""" 50 | for iter in iter_of_iters: 51 | for i in iter: yield i 52 | 53 | def empty(iter): 54 | """True if iterator has length 0""" 55 | for i in iter: return None 56 | return 1 57 | 58 | def equal(iter1, iter2, verbose = None, operator = lambda x, y: x == y): 59 | """True if iterator 1 has same elements as iterator 2 60 | 61 | Use equality operator, or == if it is unspecified. 62 | 63 | """ 64 | for i1 in iter1: 65 | try: i2 = iter2.next() 66 | except StopIteration: 67 | if verbose: print "End when i1 = %s" % (i1,) 68 | return None 69 | if not operator(i1, i2): 70 | if verbose: print "%s not equal to %s" % (i1, i2) 71 | return None 72 | try: i2 = iter2.next() 73 | except StopIteration: return 1 74 | if verbose: print "End when i2 = %s" % (i2,) 75 | return None 76 | 77 | def Or(iter): 78 | """True if any element in iterator is true. Short circuiting""" 79 | i = None 80 | for i in iter: 81 | if i: return i 82 | return i 83 | 84 | def And(iter): 85 | """True if all elements in iterator are true. Short circuiting""" 86 | i = 1 87 | for i in iter: 88 | if not i: return i 89 | return i 90 | 91 | def len(iter): 92 | """Return length of iterator""" 93 | i = 0 94 | while 1: 95 | try: iter.next() 96 | except StopIteration: return i 97 | i = i+1 98 | 99 | def foldr(f, default, iter): 100 | """foldr the "fundamental list recursion operator"?""" 101 | try: next = iter.next() 102 | except StopIteration: return default 103 | return f(next, Iter.foldr(f, default, iter)) 104 | 105 | def foldl(f, default, iter): 106 | """the fundamental list iteration operator..""" 107 | while 1: 108 | try: next = iter.next() 109 | except StopIteration: return default 110 | default = f(default, next) 111 | 112 | def multiplex(iter, num_of_forks, final_func = None, closing_func = None): 113 | """Split a single iterater into a number of streams 114 | 115 | The return val will be a list with length num_of_forks, each 116 | of which will be an iterator like iter. final_func is the 117 | function that will be called on each element in iter just as 118 | it is being removed from the buffer. closing_func is called 119 | when all the streams are finished. 120 | 121 | """ 122 | if num_of_forks == 2 and not final_func and not closing_func: 123 | im2 = IterMultiplex2(iter) 124 | return (im2.yielda(), im2.yieldb()) 125 | if not final_func: final_func = lambda i: None 126 | if not closing_func: closing_func = lambda: None 127 | 128 | # buffer is a list of elements that some iterators need and others 129 | # don't 130 | buffer = [] 131 | 132 | # buffer[forkposition[i]] is the next element yieled by iterator 133 | # i. If it is -1, yield from the original iter 134 | starting_forkposition = [-1] * num_of_forks 135 | forkposition = starting_forkposition[:] 136 | called_closing_func = [None] 137 | 138 | def get_next(fork_num): 139 | """Return the next element requested by fork_num""" 140 | if forkposition[fork_num] == -1: 141 | try: buffer.insert(0, iter.next()) 142 | except StopIteration: 143 | # call closing_func if necessary 144 | if (forkposition == starting_forkposition and 145 | not called_closing_func[0]): 146 | closing_func() 147 | called_closing_func[0] = None 148 | raise StopIteration 149 | for i in range(num_of_forks): forkposition[i] += 1 150 | 151 | return_val = buffer[forkposition[fork_num]] 152 | forkposition[fork_num] -= 1 153 | 154 | blen = len(buffer) 155 | if not (blen-1) in forkposition: 156 | # Last position in buffer no longer needed 157 | assert forkposition[fork_num] == blen-2 158 | final_func(buffer[blen-1]) 159 | del buffer[blen-1] 160 | return return_val 161 | 162 | def make_iterator(fork_num): 163 | while(1): yield get_next(fork_num) 164 | 165 | return tuple(map(make_iterator, range(num_of_forks))) 166 | 167 | static.MakeStatic(Iter) 168 | 169 | 170 | class IterMultiplex2: 171 | """Multiplex an iterator into 2 parts 172 | 173 | This is a special optimized case of the Iter.multiplex function, 174 | used when there is no closing_func or final_func, and we only want 175 | to split it into 2. By profiling, this is a time sensitive class. 176 | 177 | """ 178 | def __init__(self, iter): 179 | self.a_leading_by = 0 # How many places a is ahead of b 180 | self.buffer = [] 181 | self.iter = iter 182 | 183 | def yielda(self): 184 | """Return first iterator""" 185 | buf, iter = self.buffer, self.iter 186 | while(1): 187 | if self.a_leading_by >= 0: # a is in front, add new element 188 | elem = iter.next() # exception will be passed 189 | buf.append(elem) 190 | else: elem = buf.pop(0) # b is in front, subtract an element 191 | self.a_leading_by += 1 192 | yield elem 193 | 194 | def yieldb(self): 195 | """Return second iterator""" 196 | buf, iter = self.buffer, self.iter 197 | while(1): 198 | if self.a_leading_by <= 0: # b is in front, add new element 199 | elem = iter.next() # exception will be passed 200 | buf.append(elem) 201 | else: elem = buf.pop(0) # a is in front, subtract an element 202 | self.a_leading_by -= 1 203 | yield elem 204 | -------------------------------------------------------------------------------- /rdiff-backup/rdiff_backup/librsync.py: -------------------------------------------------------------------------------- 1 | # Copyright 2002 2005 Ben Escoto 2 | # 3 | # This file is part of rdiff-backup. 4 | # 5 | # rdiff-backup is free software; you can redistribute it and/or modify 6 | # under the terms of the GNU General Public License as published by the 7 | # Free Software Foundation; either version 2 of the License, or (at your 8 | # option) any later version. 9 | # 10 | # rdiff-backup is distributed in the hope that it will be useful, but 11 | # WITHOUT ANY WARRANTY; without even the implied warranty of 12 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 13 | # General Public License for more details. 14 | # 15 | # You should have received a copy of the GNU General Public License 16 | # along with rdiff-backup; if not, write to the Free Software 17 | # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 18 | # USA 19 | 20 | """Provides a high-level interface to some librsync functions 21 | 22 | This is a python wrapper around the lower-level _librsync module, 23 | which is written in C. The goal was to use C as little as possible... 24 | 25 | """ 26 | 27 | import types, array 28 | import _librsync 29 | 30 | blocksize = _librsync.RS_JOB_BLOCKSIZE 31 | 32 | class librsyncError(Exception): 33 | """Signifies error in internal librsync processing (bad signature, etc.) 34 | 35 | underlying _librsync.librsyncError's are regenerated using this 36 | class because the C-created exceptions are by default 37 | unPickleable. There is probably a way to fix this in _librsync, 38 | but this scheme was easier. 39 | 40 | """ 41 | pass 42 | 43 | 44 | class LikeFile: 45 | """File-like object used by SigFile, DeltaFile, and PatchFile""" 46 | mode = "rb" 47 | 48 | # This will be replaced in subclasses by an object with 49 | # appropriate cycle() method 50 | maker = None 51 | 52 | def __init__(self, infile, need_seek = None): 53 | """LikeFile initializer - zero buffers, set eofs off""" 54 | self.check_file(infile, need_seek) 55 | self.infile = infile 56 | self.closed = self.infile_closed = None 57 | self.inbuf = "" 58 | self.outbuf = array.array('c') 59 | self.eof = self.infile_eof = None 60 | 61 | def check_file(self, file, need_seek = None): 62 | """Raise type error if file doesn't have necessary attributes""" 63 | if not hasattr(file, "read"): 64 | raise TypeError("Basis file must have a read() method") 65 | if not hasattr(file, "close"): 66 | raise TypeError("Basis file must have a close() method") 67 | if need_seek and not hasattr(file, "seek"): 68 | raise TypeError("Basis file must have a seek() method") 69 | 70 | def read(self, length = -1): 71 | """Build up self.outbuf, return first length bytes""" 72 | if length == -1: 73 | while not self.eof: self._add_to_outbuf_once() 74 | real_len = len(self.outbuf) 75 | else: 76 | while not self.eof and len(self.outbuf) < length: 77 | self._add_to_outbuf_once() 78 | real_len = min(length, len(self.outbuf)) 79 | 80 | return_val = self.outbuf[:real_len].tostring() 81 | del self.outbuf[:real_len] 82 | return return_val 83 | 84 | def _add_to_outbuf_once(self): 85 | """Add one cycle's worth of output to self.outbuf""" 86 | if not self.infile_eof: self._add_to_inbuf() 87 | try: self.eof, len_inbuf_read, cycle_out = self.maker.cycle(self.inbuf) 88 | except _librsync.librsyncError, e: raise librsyncError(str(e)) 89 | self.inbuf = self.inbuf[len_inbuf_read:] 90 | self.outbuf.fromstring(cycle_out) 91 | 92 | def _add_to_inbuf(self): 93 | """Make sure len(self.inbuf) >= blocksize""" 94 | assert not self.infile_eof 95 | while len(self.inbuf) < blocksize: 96 | new_in = self.infile.read(blocksize) 97 | if not new_in: 98 | self.infile_eof = 1 99 | self.infile_closeval = self.infile.close() 100 | self.infile_closed = 1 101 | break 102 | self.inbuf += new_in 103 | 104 | def close(self): 105 | """Close infile and pass on infile close value""" 106 | self.closed = 1 107 | if self.infile_closed: return self.infile_closeval 108 | else: return self.infile.close() 109 | 110 | 111 | class SigFile(LikeFile): 112 | """File-like object which incrementally generates a librsync signature""" 113 | def __init__(self, infile, blocksize = _librsync.RS_DEFAULT_BLOCK_LEN): 114 | """SigFile initializer - takes basis file 115 | 116 | basis file only needs to have read() and close() methods. It 117 | will be closed when we come to the end of the signature. 118 | 119 | """ 120 | LikeFile.__init__(self, infile) 121 | try: self.maker = _librsync.new_sigmaker(blocksize) 122 | except _librsync.librsyncError, e: raise librsyncError(str(e)) 123 | 124 | class DeltaFile(LikeFile): 125 | """File-like object which incrementally generates a librsync delta""" 126 | def __init__(self, signature, new_file): 127 | """DeltaFile initializer - call with signature and new file 128 | 129 | Signature can either be a string or a file with read() and 130 | close() methods. New_file also only needs to have read() and 131 | close() methods. It will be closed when self is closed. 132 | 133 | """ 134 | LikeFile.__init__(self, new_file) 135 | if type(signature) is types.StringType: sig_string = signature 136 | else: 137 | self.check_file(signature) 138 | sig_string = signature.read() 139 | assert not signature.close() 140 | try: self.maker = _librsync.new_deltamaker(sig_string) 141 | except _librsync.librsyncError, e: raise librsyncError(str(e)) 142 | 143 | 144 | class PatchedFile(LikeFile): 145 | """File-like object which applies a librsync delta incrementally""" 146 | def __init__(self, basis_file, delta_file): 147 | """PatchedFile initializer - call with basis delta 148 | 149 | Here basis_file must be a true Python file, because we may 150 | need to seek() around in it a lot, and this is done in C. 151 | delta_file only needs read() and close() methods. 152 | 153 | """ 154 | LikeFile.__init__(self, delta_file) 155 | if hasattr(basis_file, 'file'): 156 | basis_file = basis_file.file 157 | if type(basis_file) is not types.FileType: 158 | raise TypeError("basis_file must be a (true) file") 159 | try: self.maker = _librsync.new_patchmaker(basis_file) 160 | except _librsync.librsyncError, e: raise librsyncError(str(e)) 161 | 162 | 163 | class SigGenerator: 164 | """Calculate signature. 165 | 166 | Input and output is same as SigFile, but the interface is like md5 167 | module, not filelike object 168 | 169 | """ 170 | def __init__(self, blocksize = _librsync.RS_DEFAULT_BLOCK_LEN): 171 | """Return new signature instance""" 172 | try: self.sig_maker = _librsync.new_sigmaker(blocksize) 173 | except _librsync.librsyncError, e: raise librsyncError(str(e)) 174 | self.gotsig = None 175 | self.buffer = "" 176 | self.sig_string = "" 177 | 178 | def update(self, buf): 179 | """Add buf to data that signature will be calculated over""" 180 | if self.gotsig: 181 | raise librsyncError("SigGenerator already provided signature") 182 | self.buffer += buf 183 | while len(self.buffer) >= blocksize: 184 | if self.process_buffer(): 185 | raise librsyncError("Premature EOF received from sig_maker") 186 | 187 | def process_buffer(self): 188 | """Run self.buffer through sig_maker, add to self.sig_string""" 189 | try: eof, len_buf_read, cycle_out = self.sig_maker.cycle(self.buffer) 190 | except _librsync.librsyncError, e: raise librsyncError(str(e)) 191 | self.buffer = self.buffer[len_buf_read:] 192 | self.sig_string += cycle_out 193 | return eof 194 | 195 | def getsig(self): 196 | """Return signature over given data""" 197 | while not self.process_buffer(): pass # keep running until eof 198 | return self.sig_string 199 | 200 | -------------------------------------------------------------------------------- /rdiff-backup/rdiff_backup/librsync_memoryleak2.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """Demonstrate a memory leak in pysync/librsync""" 4 | 5 | import os, _librsync 6 | from librsync import * 7 | 8 | os.chdir("/tmp") 9 | 10 | # Write 2 1 byte files 11 | afile = open("a", "wb") 12 | afile.write("a") 13 | afile.close() 14 | 15 | efile = open("e", "wb") 16 | efile.write("e") 17 | efile.close() 18 | 19 | def copy(infileobj, outpath): 20 | outfile = open(outpath, "wb") 21 | while 1: 22 | buf = infileobj.read(32768) 23 | if not buf: break 24 | outfile.write(buf) 25 | assert not outfile.close() 26 | assert not infileobj.close() 27 | 28 | def test_cycle(): 29 | for i in xrange(100000): 30 | sm = _librsync.new_sigmaker() 31 | sm.cycle("a") 32 | 33 | def main_test(): 34 | for i in xrange(100000): 35 | # Write signature file 36 | afile = open("a", "rb") 37 | copy(SigFile(afile), "sig") 38 | 39 | # Write delta file 40 | efile = open("e", "r") 41 | sigfile = open("sig", "rb") 42 | copy(DeltaFile(sigfile, efile), "delta") 43 | 44 | # Write patched file 45 | afile = open("e", "rb") 46 | deltafile = open("delta", "rb") 47 | copy(PatchedFile(afile, deltafile), "a.out") 48 | 49 | main_test() 50 | -------------------------------------------------------------------------------- /rdiff-backup/rdiff_backup/memoryleak.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | 4 | main() 5 | { 6 | FILE *basis_file, *sig_file; 7 | char filename[50]; 8 | rs_stats_t stats; 9 | rs_result result; 10 | long i; 11 | 12 | for(i=0; i<=100000; i++) { 13 | basis_file = fopen("a", "r"); 14 | sig_file = fopen("sig", "w"); 15 | 16 | result = rs_sig_file(basis_file, sig_file, 17 | RS_DEFAULT_BLOCK_LEN, RS_DEFAULT_STRONG_LEN, 18 | &stats); 19 | if (result != RS_DONE) exit(result); 20 | fclose(basis_file); 21 | fclose(sig_file); 22 | } 23 | } 24 | -------------------------------------------------------------------------------- /rdiff-backup/rdiff_backup/myrdiff.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """Like rdiff, but written in python and uses librsync module. 4 | 5 | Useful for benchmarking and testing of librsync and _librsync. 6 | 7 | """ 8 | 9 | import librsync, sys 10 | blocksize = 32768 11 | 12 | def makesig(inpath, outpath): 13 | """Write a signature of inpath at outpath""" 14 | sf = librsync.SigFile(open(inpath, "rb")) 15 | fout = open(outpath, "wb") 16 | while 1: 17 | buf = sf.read(blocksize) 18 | if not buf: break 19 | fout.write(buf) 20 | assert not sf.close() 21 | assert not fout.close() 22 | 23 | def makedelta(sigpath, newpath, deltapath): 24 | """Write delta at deltapath using signature at sigpath""" 25 | df = librsync.DeltaFile(open(sigpath, "rb"), open(newpath, "rb")) 26 | fout = open(deltapath, "wb") 27 | while 1: 28 | buf = df.read(blocksize) 29 | if not buf: break 30 | fout.write(buf) 31 | assert not df.close() 32 | assert not fout.close() 33 | 34 | def makepatch(basis_path, delta_path, new_path): 35 | """Write new given basis and delta""" 36 | pf = librsync.PatchedFile(open(basis_path, "rb"), open(delta_path, "rb")) 37 | fout = open(new_path, "wb") 38 | while 1: 39 | buf = pf.read(blocksize) 40 | if not buf: break 41 | fout.write(buf) 42 | assert not pf.close() 43 | assert not fout.close() 44 | 45 | if sys.argv[1] == "signature": 46 | makesig(sys.argv[2], sys.argv[3]) 47 | elif sys.argv[1] == "delta": 48 | makedelta(sys.argv[2], sys.argv[3], sys.argv[4]) 49 | elif sys.argv[1] == "patch": 50 | makepatch(sys.argv[2], sys.argv[3], sys.argv[4]) 51 | else: assert 0, "Bad mode argument %s" % (sys.argv[1],) 52 | -------------------------------------------------------------------------------- /rdiff-backup/rdiff_backup/profiled_rdb.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """Run rdiff-backup with profiling on 4 | 5 | Same as rdiff-backup but runs profiler, and prints profiling 6 | statistics afterwards. 7 | 8 | """ 9 | 10 | __no_execute__ = 1 11 | import sys, rdiff_backup.Main, profile, pstats 12 | profile.run("rdiff_backup.Main.Main(%s)" % repr(sys.argv[1:]), 13 | "profile-output") 14 | p = pstats.Stats("profile-output") 15 | p.sort_stats('time') 16 | p.print_stats(40) 17 | #p.print_callers(20) 18 | 19 | 20 | -------------------------------------------------------------------------------- /rdiff-backup/rdiff_backup/robust.py: -------------------------------------------------------------------------------- 1 | # Copyright 2002 Ben Escoto 2 | # 3 | # This file is part of rdiff-backup. 4 | # 5 | # rdiff-backup is free software; you can redistribute it and/or modify 6 | # under the terms of the GNU General Public License as published by the 7 | # Free Software Foundation; either version 2 of the License, or (at your 8 | # option) any later version. 9 | # 10 | # rdiff-backup is distributed in the hope that it will be useful, but 11 | # WITHOUT ANY WARRANTY; without even the implied warranty of 12 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 13 | # General Public License for more details. 14 | # 15 | # You should have received a copy of the GNU General Public License 16 | # along with rdiff-backup; if not, write to the Free Software 17 | # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 18 | # USA 19 | 20 | """Catch various exceptions given system call""" 21 | 22 | import errno, signal, exceptions, zlib 23 | import librsync, C, static, rpath, Globals, log, statistics, connection 24 | 25 | def check_common_error(error_handler, function, args = []): 26 | """Apply function to args, if error, run error_handler on exception 27 | 28 | This uses the catch_error predicate below to only catch 29 | certain exceptions which seems innocent enough. 30 | 31 | """ 32 | try: return function(*args) 33 | except (Exception, KeyboardInterrupt, SystemExit), exc: 34 | TracebackArchive.add([function] + list(args)) 35 | if catch_error(exc): 36 | log.Log.exception() 37 | conn = Globals.backup_writer 38 | if conn is not None: conn.statistics.record_error() 39 | if error_handler: return error_handler(exc, *args) 40 | else: return None 41 | if is_routine_fatal(exc): log.Log.exception(1, 6) 42 | else: log.Log.exception(1, 2) 43 | raise 44 | 45 | def catch_error(exc): 46 | """Return true if exception exc should be caught""" 47 | for exception_class in (rpath.SkipFileException, rpath.RPathException, 48 | librsync.librsyncError, C.UnknownFileTypeError, 49 | zlib.error): 50 | if isinstance(exc, exception_class): return 1 51 | if (isinstance(exc, EnvironmentError) and 52 | # the invalid mode shows up in backups of /proc for some reason 53 | (exc[0] in ('invalid mode: rb', 'Not a gzipped file') or 54 | errno.errorcode.has_key(exc[0]) and 55 | errno.errorcode[exc[0]] in ('EPERM', 'ENOENT', 'EACCES', 'EBUSY', 56 | 'EEXIST', 'ENOTDIR', 'EILSEQ', 57 | 'ENAMETOOLONG', 'EINTR', 'ESTALE', 58 | 'ENOTEMPTY', 'EIO', 'ETXTBSY', 59 | 'ESRCH', 'EINVAL', 'EDEADLOCK', 60 | 'EDEADLK', 'EOPNOTSUPP', 'ETIMEDOUT'))): 61 | return 1 62 | return 0 63 | 64 | def is_routine_fatal(exc): 65 | """Return string if exception is non-error unrecoverable, None otherwise 66 | 67 | Used to suppress a stack trace for exceptions like keyboard 68 | interrupts or connection drops. Return value is string to use as 69 | an exit message. 70 | 71 | """ 72 | if isinstance(exc, exceptions.KeyboardInterrupt): 73 | return "User abort" 74 | elif isinstance(exc, connection.ConnectionError): 75 | return "Lost connection to the remote system" 76 | elif isinstance(exc, SignalException): 77 | return "Killed with signal %s" % (exc,) 78 | elif isinstance(exc, EnvironmentError) and exc.errno == errno.ENOTCONN: 79 | return ("Filesystem reports connection failure:\n%s" % exc) 80 | return None 81 | 82 | def get_error_handler(error_type): 83 | """Return error handler function that can be used above 84 | 85 | Function will just log error to the error_log and then return 86 | None. First two arguments must be the exception and then an rp 87 | (from which the filename will be extracted). 88 | 89 | """ 90 | def error_handler(exc, rp, *args): 91 | log.ErrorLog.write_if_open(error_type, rp, exc) 92 | return 0 93 | return error_handler 94 | 95 | def listrp(rp): 96 | """Like rp.listdir() but return [] if error, and sort results""" 97 | def error_handler(exc): 98 | log.Log("Error listing directory %s" % rp.path, 2) 99 | return [] 100 | dir_listing = check_common_error(error_handler, rp.listdir) 101 | dir_listing.sort() 102 | return dir_listing 103 | 104 | def signal_handler(signum, frame): 105 | """This is called when signal signum is caught""" 106 | raise SignalException(signum) 107 | 108 | def install_signal_handlers(): 109 | """Install signal handlers on current connection""" 110 | signals = [signal.SIGTERM, signal.SIGINT] 111 | try: 112 | signals.extend([signal.SIGHUP, signal.SIGQUIT]) 113 | except AttributeError: 114 | pass 115 | for signum in signals: 116 | signal.signal(signum, signal_handler) 117 | 118 | 119 | class SignalException(Exception): 120 | """SignalException(signum) means signal signum has been received""" 121 | pass 122 | 123 | 124 | class TracebackArchive: 125 | """Save last 10 caught exceptions, so they can be printed if fatal""" 126 | _traceback_strings = [] 127 | def add(cls, extra_args = []): 128 | """Add most recent exception to archived list 129 | 130 | If extra_args are present, convert to strings and add them as 131 | extra information to same traceback archive. 132 | 133 | """ 134 | cls._traceback_strings.append(log.Log.exception_to_string(extra_args)) 135 | if len(cls._traceback_strings) > 10: 136 | cls._traceback_strings = cls._traceback_strings[:10] 137 | 138 | def log(cls): 139 | """Print all exception information to log file""" 140 | if cls._traceback_strings: 141 | log.Log("------------ Old traceback info -----------\n%s\n" 142 | "-------------------------------------------" % 143 | ("\n".join(cls._traceback_strings),), 3) 144 | 145 | static.MakeClass(TracebackArchive) 146 | 147 | -------------------------------------------------------------------------------- /rdiff-backup/rdiff_backup/static.py: -------------------------------------------------------------------------------- 1 | # Copyright 2002 Ben Escoto 2 | # 3 | # This file is part of rdiff-backup. 4 | # 5 | # rdiff-backup is free software; you can redistribute it and/or modify 6 | # under the terms of the GNU General Public License as published by the 7 | # Free Software Foundation; either version 2 of the License, or (at your 8 | # option) any later version. 9 | # 10 | # rdiff-backup is distributed in the hope that it will be useful, but 11 | # WITHOUT ANY WARRANTY; without even the implied warranty of 12 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 13 | # General Public License for more details. 14 | # 15 | # You should have received a copy of the GNU General Public License 16 | # along with rdiff-backup; if not, write to the Free Software 17 | # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 18 | # USA 19 | 20 | """MakeStatic and MakeClass 21 | 22 | These functions are used to make all the instance methods in a class 23 | into static or class methods. 24 | 25 | """ 26 | 27 | class StaticMethodsError(Exception): pass 28 | 29 | def MakeStatic(cls): 30 | """turn instance methods into static ones 31 | 32 | The methods (that don't begin with _) of any class that 33 | subclasses this will be turned into static methods. 34 | 35 | """ 36 | for name in cls.__dict__: 37 | if name[0] != "_": 38 | cls.__dict__[name] = staticmethod(cls.__dict__[name]) 39 | 40 | def MakeClass(cls): 41 | """Turn instance methods into classmethods. Ignore _ like above""" 42 | for name in cls.__dict__: 43 | if name[0] != "_": 44 | cls.__dict__[name] = classmethod(cls.__dict__[name]) 45 | -------------------------------------------------------------------------------- /rdiff-backup/testing/FilenameMappingtest.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | from commontest import * 3 | from rdiff_backup import FilenameMapping, rpath, Globals 4 | 5 | class FilenameMappingTest(unittest.TestCase): 6 | """Test the FilenameMapping class, for quoting filenames""" 7 | def setUp(self): 8 | """Just initialize quoting""" 9 | Globals.chars_to_quote = 'A-Z' 10 | FilenameMapping.set_init_quote_vals() 11 | 12 | def testBasicQuote(self): 13 | """Test basic quoting and unquoting""" 14 | filenames = ["hello", "HeLLo", "EUOeu/EUOeu", ":", "::::EU", "/:/:"] 15 | for filename in filenames: 16 | quoted = FilenameMapping.quote(filename) 17 | assert FilenameMapping.unquote(quoted) == filename, filename 18 | 19 | def testQuotedRPath(self): 20 | """Test the QuotedRPath class""" 21 | 22 | def testQuotedSepBase(self): 23 | """Test get_quoted_sep_base function""" 24 | path = ("/usr/local/mirror_metadata" 25 | ".1969-12-31;08421;05833;05820-07;05800.data.gz") 26 | qrp = FilenameMapping.get_quoted_sep_base(path) 27 | assert qrp.base == "/usr/local", qrp.base 28 | assert len(qrp.index) == 1, qrp.index 29 | assert (qrp.index[0] == 30 | "mirror_metadata.1969-12-31T21:33:20-07:00.data.gz") 31 | 32 | def testLongFilenames(self): 33 | """See if long quoted filenames cause crash""" 34 | MakeOutputDir() 35 | outrp = rpath.RPath(Globals.local_connection, "testfiles/output") 36 | inrp = rpath.RPath(Globals.local_connection, "testfiles/quotetest") 37 | re_init_dir(inrp) 38 | long_filename = "A"*200 # when quoted should cause overflow 39 | longrp = inrp.append(long_filename) 40 | longrp.touch() 41 | shortrp = inrp.append("B") 42 | shortrp.touch() 43 | 44 | rdiff_backup(1, 1, inrp.path, outrp.path, 100000, 45 | extra_options = "--override-chars-to-quote A") 46 | 47 | longrp_out = outrp.append(long_filename) 48 | assert not longrp_out.lstat() 49 | shortrp_out = outrp.append('B') 50 | assert shortrp_out.lstat() 51 | 52 | rdiff_backup(1, 1, "testfiles/empty", outrp.path, 200000) 53 | shortrp_out.setdata() 54 | assert not shortrp_out.lstat() 55 | rdiff_backup(1, 1, inrp.path, outrp.path, 300000) 56 | shortrp_out.setdata() 57 | assert shortrp_out.lstat() 58 | 59 | if __name__ == "__main__": unittest.main() 60 | -------------------------------------------------------------------------------- /rdiff-backup/testing/backuptest.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | from commontest import * 3 | from rdiff_backup import Globals, SetConnections, user_group 4 | 5 | class RemoteMirrorTest(unittest.TestCase): 6 | """Test mirroring""" 7 | def setUp(self): 8 | """Start server""" 9 | Log.setverbosity(5) 10 | Globals.change_source_perms = 1 11 | SetConnections.UpdateGlobal('checkpoint_interval', 3) 12 | user_group.init_user_mapping() 13 | user_group.init_group_mapping() 14 | 15 | def testMirror(self): 16 | """Testing simple mirror""" 17 | MirrorTest(None, None, ["testfiles/increment1"]) 18 | 19 | def testMirror2(self): 20 | """Test mirror with larger data set""" 21 | MirrorTest(1, None, ['testfiles/increment1', 'testfiles/increment2', 22 | 'testfiles/increment3', 'testfiles/increment4']) 23 | 24 | def testMirror3(self): 25 | """Local version of testMirror2""" 26 | MirrorTest(1, 1, ['testfiles/increment1', 'testfiles/increment2', 27 | 'testfiles/increment3', 'testfiles/increment4']) 28 | 29 | 30 | if __name__ == "__main__": unittest.main() 31 | -------------------------------------------------------------------------------- /rdiff-backup/testing/benchmark.py: -------------------------------------------------------------------------------- 1 | import sys, time 2 | from commontest import * 3 | from rdiff_backup import rpath, Globals 4 | 5 | """benchmark.py 6 | 7 | When possible, use 'rdiff-backup' from the shell, which allows using 8 | different versions of rdiff-backup by altering the PYTHONPATH. We 9 | just use clock time, so this isn't exact at all. 10 | 11 | """ 12 | 13 | output_local = 1 14 | output_desc = "testfiles/output" 15 | new_pythonpath = None 16 | 17 | def run_cmd(cmd): 18 | """Run the given cmd, return the amount of time it took""" 19 | if new_pythonpath: full_cmd = "PYTHONPATH=%s %s" % (new_pythonpath, cmd) 20 | else: full_cmd = cmd 21 | print "Running command '%s'" % (full_cmd,) 22 | t = time.time() 23 | assert not os.system(full_cmd) 24 | return time.time() - t 25 | 26 | def create_many_files(dirname, s, count = 1000): 27 | """Create many short files in the dirname directory 28 | 29 | There will be count files in the directory, and each file will 30 | contain the string s. 31 | 32 | """ 33 | Myrm("testfiles/many_out") 34 | dir_rp = rpath.RPath(Globals.local_connection, dirname) 35 | dir_rp.mkdir() 36 | for i in xrange(count): 37 | rp = dir_rp.append(str(i)) 38 | fp = rp.open("wb") 39 | fp.write(s) 40 | assert not fp.close() 41 | 42 | def create_nested(dirname, s, depth, branch_factor = 10): 43 | """Create many short files in branching directory""" 44 | def write(rp): 45 | fp = rp.open("wb") 46 | fp.write(s) 47 | assert not fp.close() 48 | 49 | def helper(rp, depth): 50 | rp.mkdir() 51 | sub_rps = map(lambda i: rp.append(str(i)), range(branch_factor)) 52 | if depth == 1: map(write, sub_rps) 53 | else: map(lambda rp: helper(rp, depth-1), sub_rps) 54 | 55 | Myrm("testfiles/nested_out") 56 | helper(rpath.RPath(Globals.local_connection, dirname), depth) 57 | 58 | def benchmark(backup_cmd, restore_cmd, desc, update_func = None): 59 | """Print benchmark using backup_cmd and restore_cmd 60 | 61 | If update_func is given, run it and then do backup a third time. 62 | 63 | """ 64 | print "Initially backing up %s: %ss" % (desc, run_cmd(backup_cmd)) 65 | print "Updating %s, no change: %ss" % (desc, run_cmd(backup_cmd)) 66 | 67 | if update_func: 68 | update_func() 69 | print "Updating %s, all changed: %ss" % (desc, run_cmd(backup_cmd)) 70 | 71 | Myrm("testfiles/rest_out") 72 | print "Restoring %s to empty dir: %ss" % (desc, run_cmd(restore_cmd)) 73 | print "Restoring %s to unchanged dir: %ss" % (desc, run_cmd(restore_cmd)) 74 | 75 | def many_files(): 76 | """Time backup and restore of 2000 files""" 77 | count = 2000 78 | create_many_files("testfiles/many_out", "a", count) 79 | backup_cmd = "rdiff-backup testfiles/many_out " + output_desc 80 | restore_cmd = "rdiff-backup --force -r now %s testfiles/rest_out" % \ 81 | (output_desc,) 82 | update_func = lambda: create_many_files("testfiles/many_out", "e", count) 83 | benchmark(backup_cmd, restore_cmd, "2000 1-byte files", update_func) 84 | 85 | def many_files_rsync(): 86 | """Test rsync benchmark""" 87 | count = 2000 88 | create_many_files("testfiles/many_out", "a", count) 89 | rsync_command = ("rsync -e ssh -aH --delete testfiles/many_out " + 90 | output_desc) 91 | print "Initial rsync: %ss" % (run_cmd(rsync_command),) 92 | print "rsync update: %ss" % (run_cmd(rsync_command),) 93 | 94 | create_many_files("testfiles/many_out", "e", count) 95 | print "Update changed rsync: %ss" % (run_cmd(rsync_command),) 96 | 97 | def nested_files(): 98 | """Time backup and restore of 10000 nested files""" 99 | depth = 4 100 | create_nested("testfiles/nested_out", "a", depth) 101 | backup_cmd = "rdiff-backup testfiles/nested_out " + output_desc 102 | restore_cmd = "rdiff-backup --force -r now %s testfiles/rest_out" % \ 103 | (output_desc,) 104 | update_func = lambda: create_nested("testfiles/nested_out", "e", depth) 105 | benchmark(backup_cmd, restore_cmd, "10000 1-byte nested files", 106 | update_func) 107 | 108 | def nested_files_rsync(): 109 | """Test rsync on nested files""" 110 | depth = 4 111 | create_nested("testfiles/nested_out", "a", depth) 112 | rsync_command = ("rsync -e ssh -aH --delete testfiles/nested_out " + 113 | output_desc) 114 | print "Initial rsync: %ss" % (run_cmd(rsync_command),) 115 | print "rsync update: %ss" % (run_cmd(rsync_command),) 116 | 117 | create_nested("testfiles/nested_out", "e", depth) 118 | print "Update changed rsync: %ss" % (run_cmd(rsync_command),) 119 | 120 | if len(sys.argv) < 2 or len(sys.argv) > 3: 121 | print "Syntax: benchmark.py benchmark_func [output_description]" 122 | print 123 | print "Where output_description defaults to 'testfiles/output'." 124 | print "Currently benchmark_func includes:" 125 | print "'many_files', 'many_files_rsync', and, 'nested_files'." 126 | sys.exit(1) 127 | 128 | if len(sys.argv) == 3: 129 | output_desc = sys.argv[2] 130 | if ":" in output_desc: output_local = None 131 | 132 | if output_local: 133 | assert not rpath.RPath(Globals.local_connection, output_desc).lstat(), \ 134 | "Outfile file %s exists, try deleting it first" % (output_desc,) 135 | 136 | if os.environ.has_key('BENCHMARKPYPATH'): 137 | new_pythonpath = os.environ['BENCHMARKPYPATH'] 138 | 139 | function_name = sys.argv[1] 140 | print "Running ", function_name 141 | eval(sys.argv[1])() 142 | -------------------------------------------------------------------------------- /rdiff-backup/testing/chdir-wrapper: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """Used to emulate a remote connection by changing directories. 4 | 5 | If given an argument, will change to that directory, and then start 6 | the server. Otherwise will start the server without a chdir. 7 | 8 | """ 9 | 10 | import os, sys 11 | 12 | olddir = os.getcwd() 13 | if len(sys.argv) > 1: os.chdir(sys.argv[1]) 14 | #PipeConnection(sys.stdin, sys.stdout).Server() 15 | 16 | #os.system("/home/ben/prog/python/rdiff-backup/rdiff-backup --server") 17 | #os.system("/home/ben/prog/rdiff-backup/server.py /home/ben/prog/python/rdiff-backup/src") 18 | os.system("%s/server.py" % (olddir,)) 19 | -------------------------------------------------------------------------------- /rdiff-backup/testing/chdir-wrapper2: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """Used to emulate a remote connection by changing directories. 4 | 5 | Like chdir-wrapper, but this time run the 'rdiff-backup' script, not 6 | some other special thing. 7 | 8 | """ 9 | 10 | import os, sys 11 | 12 | if len(sys.argv) > 1: 13 | olddir = os.getcwd() 14 | os.chdir(sys.argv[1]) 15 | #PipeConnection(sys.stdin, sys.stdout).Server() 16 | 17 | os.system(os.path.join(olddir, "../rdiff-backup") + " --server") 18 | 19 | -------------------------------------------------------------------------------- /rdiff-backup/testing/comparetest.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | from commontest import * 3 | from rdiff_backup import compare 4 | 5 | """Test the compare.py module and overall compare functionality""" 6 | 7 | class CompareTest(unittest.TestCase): 8 | def setUp(self): 9 | Myrm("testfiles/output") 10 | rdiff_backup(1, 1, 'testfiles/increment2', 'testfiles/output', 11 | current_time = 10000) 12 | rdiff_backup(1, 1, 'testfiles/increment3', 'testfiles/output', 13 | current_time = 20000) 14 | 15 | def generic_test(self, local, compare_option): 16 | """Used for 6 tests below""" 17 | rdiff_backup(local, local, 'testfiles/increment3', 'testfiles/output', 18 | extra_options = compare_option) 19 | ret_val = rdiff_backup(local, local, 'testfiles/increment2', 20 | 'testfiles/output', extra_options = compare_option, 21 | check_return_val = 0) 22 | assert ret_val, ret_val 23 | rdiff_backup(local, local, 'testfiles/increment2', 'testfiles/output', 24 | extra_options = compare_option + "-at-time 10000") 25 | ret_val = rdiff_backup(local, local, 'testfiles/increment3', 26 | 'testfiles/output', 27 | extra_options = compare_option + "-at-time 10000", 28 | check_return_val = 0) 29 | assert ret_val, ret_val 30 | 31 | def testBasicLocal(self): 32 | """Test basic --compare and --compare-at-time modes""" 33 | self.generic_test(1, "--compare") 34 | 35 | def testBasicRemote(self): 36 | """Test basic --compare and --compare-at-time modes, both remote""" 37 | self.generic_test(0, "--compare") 38 | 39 | def testHashLocal(self): 40 | """Test --compare-hash and --compare-hash-at-time modes local""" 41 | self.generic_test(1, "--compare-hash") 42 | 43 | def testHashRemote(self): 44 | """Test --compare-hash and -at-time remotely""" 45 | self.generic_test(0, "--compare-hash") 46 | 47 | def testFullLocal(self): 48 | """Test --compare-full and --compare-full-at-time""" 49 | self.generic_test(1, "--compare-full") 50 | 51 | def testFullRemote(self): 52 | """Test full file compare remotely""" 53 | self.generic_test(0, "--compare-full") 54 | 55 | def generic_selective_test(self, local, compare_option): 56 | """Used for selective tests--just compare part of a backup""" 57 | rdiff_backup(local, local, 'testfiles/increment3/various_file_types', 58 | 'testfiles/output/various_file_types', 59 | extra_options = compare_option) 60 | ret_val = rdiff_backup(local, local, 61 | 'testfiles/increment2/increment1', 62 | 'testfiles/output/increment1', 63 | extra_options = compare_option, 64 | check_return_val = 0) 65 | assert ret_val, ret_val 66 | 67 | rdiff_backup(local, local, 'testfiles/increment2/newdir', 68 | 'testfiles/output/newdir', 69 | extra_options = compare_option + "-at-time 10000") 70 | ret_val = rdiff_backup(local, local, 71 | 'testfiles/increment3/newdir', 72 | 'testfiles/output/newdir', 73 | extra_options = compare_option + "-at-time 10000", 74 | check_return_val = 0) 75 | assert ret_val, ret_val 76 | 77 | def testSelLocal(self): 78 | """Test basic local compare of single subdirectory""" 79 | self.generic_selective_test(1, "--compare") 80 | 81 | def testSelRemote(self): 82 | """Test --compare of single directory, remote""" 83 | self.generic_selective_test(0, "--compare") 84 | 85 | def testSelHashLocal(self): 86 | """Test --compare-hash of subdirectory, local""" 87 | self.generic_selective_test(1, "--compare-hash") 88 | 89 | def testSelHashRemote(self): 90 | """Test --compare-hash of subdirectory, remote""" 91 | self.generic_selective_test(0, "--compare-hash") 92 | 93 | def testSelFullLocal(self): 94 | """Test --compare-full of subdirectory, local""" 95 | self.generic_selective_test(1, "--compare-full") 96 | 97 | def testSelFullRemote(self): 98 | """Test --compare-full of subdirectory, remote""" 99 | self.generic_selective_test(0, "--compare-full") 100 | 101 | def verify(self, local): 102 | """Used for the verify tests""" 103 | def change_file(rp): 104 | """Given rpath, open it, and change a byte in the middle""" 105 | fp = rp.open("rb") 106 | fp.seek(int(rp.getsize()/2)) 107 | char = fp.read(1) 108 | fp.close() 109 | 110 | fp = rp.open("wb") 111 | fp.seek(int(rp.getsize()/2)) 112 | if char == 'a': fp.write('b') 113 | else: fp.write('a') 114 | fp.close() 115 | 116 | def modify_diff(): 117 | """Write to the stph_icons.h diff""" 118 | l = [filename for filename in 119 | os.listdir('testfiles/output/rdiff-backup-data/increments') 120 | if filename.startswith('stph_icons.h')] 121 | assert len(l) == 1, l 122 | diff_rp = rpath.RPath(Globals.local_connection, 123 | 'testfiles/output/rdiff-backup-data/increments/' + l[0]) 124 | change_file(diff_rp) 125 | 126 | rdiff_backup(local, local, 'testfiles/output', None, 127 | extra_options = "--verify") 128 | rdiff_backup(local, local, 'testfiles/output', None, 129 | extra_options = "--verify-at-time 10000") 130 | modify_diff() 131 | ret_val = rdiff_backup(local, local, 'testfiles/output', None, 132 | extra_options = "--verify-at-time 10000", 133 | check_return_val = 0) 134 | assert ret_val, ret_val 135 | change_file(rpath.RPath(Globals.local_connection, 136 | 'testfiles/output/stph_icons.h')) 137 | ret_val = rdiff_backup(local, local, 'testfiles/output', None, 138 | extra_options = "--verify", check_return_val = 0) 139 | assert ret_val, ret_val 140 | 141 | def testVerifyLocal(self): 142 | """Test --verify of directory, local""" 143 | self.verify(1) 144 | 145 | def testVerifyRemote(self): 146 | """Test --verify remotely""" 147 | self.verify(0) 148 | 149 | if __name__ == "__main__": unittest.main() 150 | 151 | -------------------------------------------------------------------------------- /rdiff-backup/testing/ctest.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | from commontest import * 3 | from rdiff_backup import C 4 | from rdiff_backup.rpath import * 5 | 6 | class CTest(unittest.TestCase): 7 | """Test the C module by comparing results to python functions""" 8 | def test_make_dict(self): 9 | """Test making stat dictionaries""" 10 | rp1 = RPath(Globals.local_connection, "/dev/ttyS1") 11 | rp2 = RPath(Globals.local_connection, "./ctest.py") 12 | rp3 = RPath(Globals.local_connection, "aestu/aeutoheu/oeu") 13 | rp4 = RPath(Globals.local_connection, "testfiles/various_file_types/symbolic_link") 14 | rp5 = RPath(Globals.local_connection, "testfiles/various_file_types/fifo") 15 | 16 | for rp in [rp1, rp2, rp3, rp4, rp5]: 17 | dict1 = rp.make_file_dict_old() 18 | dict2 = C.make_file_dict(rp.path) 19 | if dict1 != dict2: 20 | print "Python dictionary: ", dict1 21 | print "not equal to C dictionary: ", dict2 22 | print "for path ", rp.path 23 | assert 0 24 | 25 | def test_strlong(self): 26 | """Test str2long and long2str""" 27 | self.assertRaises(TypeError, C.long2str, "hello") 28 | self.assertRaises(TypeError, C.str2long, 34) 29 | self.assertRaises(TypeError, C.str2long, "oeuo") 30 | self.assertRaises(TypeError, C.str2long, "oeuoaoeuaoeu") 31 | 32 | for s in ["\0\0\0\0\0\0\0", "helloto", 33 | "\xff\xff\xff\xff\xff\xff\xff", "randoms"]: 34 | assert len(s) == 7, repr(s) 35 | s_out = C.long2str(C.str2long(s)) 36 | assert s_out == s, (s_out, C.str2long(s), s) 37 | for l in 0L, 1L, 4000000000L, 34234L, 234234234L: 38 | assert C.str2long(C.long2str(l)) == l 39 | 40 | def test_sync(self): 41 | """Test running C.sync""" 42 | C.sync() 43 | 44 | def test_acl_quoting(self): 45 | """Test the acl_quote and acl_unquote functions""" 46 | assert C.acl_quote('foo') == 'foo', C.acl_quote('foo') 47 | assert C.acl_quote('\n') == '\\012', C.acl_quote('\n') 48 | assert C.acl_unquote('\\012') == '\n' 49 | s = '\\\n\t\145\n\01==' 50 | assert C.acl_unquote(C.acl_quote(s)) == s 51 | 52 | def test_acl_quoting2(self): 53 | """This string used to segfault the quoting code, try now""" 54 | s = '\xd8\xab\xb1Wb\xae\xc5]\x8a\xbb\x15v*\xf4\x0f!\xf9>\xe2Y\x86\xbb\xab\xdbp\xb0\x84\x13k\x1d\xc2\xf1\xf5e\xa5U\x82\x9aUV\xa0\xf4\xdf4\xba\xfdX\x03\x82\x07s\xce\x9e\x8b\xb34\x04\x9f\x17 \xf4\x8f\xa6\xfa\x97\xab\xd8\xac\xda\x85\xdcKvC\xfa#\x94\x92\x9e\xc9\xb7\xc3_\x0f\x84g\x9aB\x11<=^\xdbM\x13\x96c\x8b\xa7|*"\\\'^$@#!(){}?+ ~` ' 55 | quoted = C.acl_quote(s) 56 | assert C.acl_unquote(quoted) == s 57 | 58 | def test_acl_quoting_equals(self): 59 | """Make sure the equals character is quoted""" 60 | assert C.acl_quote('=') != '=' 61 | 62 | if __name__ == "__main__": unittest.main() 63 | -------------------------------------------------------------------------------- /rdiff-backup/testing/destructive_steppingtest.py: -------------------------------------------------------------------------------- 1 | from __future__ import generators 2 | import unittest 3 | from commontest import * 4 | from rdiff_backup import rpath, selection, Globals, destructive_stepping 5 | 6 | Log.setverbosity(4) 7 | 8 | class DSTest(unittest.TestCase): 9 | def setUp(self): 10 | self.lc = Globals.local_connection 11 | self.noperms = rpath.RPath(self.lc, "testfiles/noperms") 12 | Globals.change_source_perms = 1 13 | self.iteration_dir = rpath.RPath(self.lc, "testfiles/iteration-test") 14 | 15 | def testDSIter(self): 16 | """Testing destructive stepping iterator from baserp""" 17 | for i in range(2): 18 | sel = selection.Select(destructive_stepping. 19 | DSRPath(1, self.noperms)).set_iter() 20 | ds_iter = sel.iterate_with_finalizer() 21 | noperms = ds_iter.next() 22 | assert noperms.isdir() and noperms.getperms() == 0, \ 23 | (noperms.isdir(), noperms.getperms()) 24 | 25 | bar = ds_iter.next() 26 | assert bar.isreg() and bar.getperms() == 0, \ 27 | "%s %s" % (bar.isreg(), bar.getperms()) 28 | barbuf = bar.open("rb").read() 29 | assert len(barbuf) > 0 30 | 31 | foo = ds_iter.next() 32 | assert foo.isreg() and foo.getperms() == 0 33 | assert foo.getmtime() < 1000300000 34 | 35 | fuz = ds_iter.next() 36 | assert fuz.isreg() and fuz.getperms() == 0200 37 | fuzbuf = fuz.open("rb").read() 38 | assert len(fuzbuf) > 0 39 | 40 | self.assertRaises(StopIteration, ds_iter.next) 41 | 42 | if __name__ == "__main__": unittest.main() 43 | -------------------------------------------------------------------------------- /rdiff-backup/testing/filelisttest.py: -------------------------------------------------------------------------------- 1 | import unittest, StringIO 2 | from commontest import * 3 | from rdiff_backup.filelist import * 4 | 5 | 6 | class FilelistTest(unittest.TestCase): 7 | """Test Filelist class""" 8 | def testFile2Iter(self): 9 | """Test File2Iter function""" 10 | filelist = """ 11 | hello 12 | goodbye 13 | a/b/c 14 | 15 | test""" 16 | baserp = RPath(Globals.local_connection, "/base") 17 | i = Filelist.File2Iter(StringIO.StringIO(filelist), baserp) 18 | assert i.next().path == "/base/hello" 19 | assert i.next().path == "/base/goodbye" 20 | assert i.next().path == "/base/a/b/c" 21 | assert i.next().path == "/base/test" 22 | self.assertRaises(StopIteration, i.next) 23 | 24 | def testmake_subdirs(self): 25 | """Test Filelist.make_subdirs""" 26 | self.assertRaises(os.error, os.lstat, "foo_delete_me") 27 | Filelist.make_subdirs(RPath(Globals.local_connection, 28 | "foo_delete_me/a/b/c/d")) 29 | os.lstat("foo_delete_me") 30 | os.lstat("foo_delete_me/a") 31 | os.lstat("foo_delete_me/a/b") 32 | os.lstat("foo_delete_me/a/b/c") 33 | os.system("rm -rf foo_delete_me") 34 | 35 | if __name__ == "__main__": unittest.main() 36 | -------------------------------------------------------------------------------- /rdiff-backup/testing/find-max-ram.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """find-max-ram - Returns the maximum amount of memory used by a program. 4 | 5 | Every half second, run ps with the appropriate commands, getting the 6 | size of the program. Return max value. 7 | 8 | """ 9 | 10 | import os, sys, time 11 | 12 | def get_val(cmdstr): 13 | """Runs ps and gets sum rss for processes making cmdstr 14 | 15 | Returns None if process not found. 16 | 17 | """ 18 | cmd = ("ps -Ao cmd -o rss | grep '%s' | grep -v grep" % cmdstr) 19 | # print "Running ", cmd 20 | fp = os.popen(cmd) 21 | lines = fp.readlines() 22 | fp.close() 23 | 24 | if not lines: return None 25 | else: return reduce(lambda x,y: x+y, map(read_ps_line, lines)) 26 | 27 | def read_ps_line(psline): 28 | """Given a specially formatted line by ps, return rss value""" 29 | l = psline.split() 30 | assert len(l) >= 2 # first few are name, last one is rss 31 | return int(l[-1]) 32 | 33 | 34 | def main(cmdstr): 35 | while get_val(cmdstr) is None: time.sleep(0.5) 36 | 37 | current_max = 0 38 | while 1: 39 | rss = get_val(cmdstr) 40 | print rss 41 | if rss is None: break 42 | current_max = max(current_max, rss) 43 | time.sleep(0.5) 44 | 45 | print current_max 46 | 47 | 48 | if __name__=="__main__": 49 | 50 | if len(sys.argv) != 2: 51 | print """Usage: find-max-ram [command string] 52 | 53 | It will then run ps twice a second and keep totalling how much RSS 54 | (resident set size) the process(es) whose ps command name contain the 55 | given string use up. When there are no more processes found, it will 56 | print the number and exit. 57 | """ 58 | sys.exit(1) 59 | else: main(sys.argv[1]) 60 | 61 | -------------------------------------------------------------------------------- /rdiff-backup/testing/fs_abilitiestest.py: -------------------------------------------------------------------------------- 1 | import unittest, os, time 2 | from commontest import * 3 | from rdiff_backup import Globals, rpath, fs_abilities 4 | 5 | class FSAbilitiesTest(unittest.TestCase): 6 | """Test testing of file system abilities 7 | 8 | Some of these tests assume that the actual file system tested has 9 | the given abilities. If the file system this is run on differs 10 | from the original test system, this test may/should fail. Change 11 | the expected values below. 12 | 13 | """ 14 | # Describes standard linux file system without acls/eas 15 | dir_to_test = "testfiles" 16 | eas = acls = 1 17 | chars_to_quote = "" 18 | extended_filenames = 1 19 | case_sensitive = 1 20 | ownership = (os.getuid() == 0) 21 | hardlinks = fsync_dirs = 1 22 | dir_inc_perms = 1 23 | resource_forks = 0 24 | carbonfile = 0 25 | high_perms = 1 26 | 27 | # Describes MS-Windows style file system 28 | #dir_to_test = "/mnt/fat" 29 | #eas = acls = 0 30 | #extended_filenames = 0 31 | #chars_to_quote = "^a-z0-9_ -" 32 | #ownership = hardlinks = 0 33 | #fsync_dirs = 1 34 | #dir_inc_perms = 0 35 | #resource_forks = 0 36 | #carbonfile = 0 37 | 38 | # A case insensitive directory (a cdrom mount on my system!) 39 | case_insensitive_path = "/media/cdrecorder" 40 | 41 | def testReadOnly(self): 42 | """Test basic querying read only""" 43 | base_dir = rpath.RPath(Globals.local_connection, self.dir_to_test) 44 | fsa = fs_abilities.FSAbilities('read-only').init_readonly(base_dir) 45 | print fsa 46 | assert fsa.read_only == 1, fsa.read_only 47 | assert fsa.eas == self.eas, fsa.eas 48 | assert fsa.acls == self.acls, fsa.acls 49 | assert fsa.resource_forks == self.resource_forks, fsa.resource_forks 50 | assert fsa.carbonfile == self.carbonfile, fsa.carbonfile 51 | assert fsa.case_sensitive == self.case_sensitive, fsa.case_sensitive 52 | 53 | def testReadWrite(self): 54 | """Test basic querying read/write""" 55 | base_dir = rpath.RPath(Globals.local_connection, self.dir_to_test) 56 | new_dir = base_dir.append("fs_abilitiestest") 57 | if new_dir.lstat(): Myrm(new_dir.path) 58 | new_dir.setdata() 59 | new_dir.mkdir() 60 | t = time.time() 61 | fsa = fs_abilities.FSAbilities('read/write').init_readwrite(new_dir) 62 | print "Time elapsed = ", time.time() - t 63 | print fsa 64 | assert fsa.read_only == 0, fsa.read_only 65 | assert fsa.eas == self.eas, fsa.eas 66 | assert fsa.acls == self.acls, fsa.acls 67 | assert fsa.ownership == self.ownership, fsa.ownership 68 | assert fsa.hardlinks == self.hardlinks, fsa.hardlinks 69 | assert fsa.fsync_dirs == self.fsync_dirs, fsa.fsync_dirs 70 | assert fsa.dir_inc_perms == self.dir_inc_perms, fsa.dir_inc_perms 71 | assert fsa.resource_forks == self.resource_forks, fsa.resource_forks 72 | assert fsa.carbonfile == self.carbonfile, fsa.carbonfile 73 | assert fsa.high_perms == self.high_perms, fsa.high_perms 74 | assert fsa.extended_filenames == self.extended_filenames 75 | 76 | #ctq_rp = new_dir.append("chars_to_quote") 77 | #assert ctq_rp.lstat() 78 | #fp = ctq_rp.open('rb') 79 | #chars_to_quote = fp.read() 80 | #assert not fp.close() 81 | #assert chars_to_quote == self.chars_to_quote, chars_to_quote 82 | 83 | new_dir.delete() 84 | 85 | def test_case_sensitive(self): 86 | """Test a read-only case-INsensitive directory""" 87 | rp = rpath.RPath(Globals.local_connection, self.case_insensitive_path) 88 | fsa = fs_abilities.FSAbilities('read-only') 89 | fsa.set_case_sensitive_readonly(rp) 90 | assert fsa.case_sensitive == 0, fsa.case_sensitive 91 | 92 | if __name__ == "__main__": unittest.main() 93 | 94 | -------------------------------------------------------------------------------- /rdiff-backup/testing/hardlinktest.py: -------------------------------------------------------------------------------- 1 | import os, unittest, time 2 | from commontest import * 3 | from rdiff_backup import Globals, Hardlink, selection, rpath 4 | 5 | Log.setverbosity(6) 6 | 7 | class HardlinkTest(unittest.TestCase): 8 | """Test cases for Hard links""" 9 | outputrp = rpath.RPath(Globals.local_connection, "testfiles/output") 10 | hardlink_dir1 = rpath.RPath(Globals.local_connection, 11 | "testfiles/hardlinks/dir1") 12 | hardlink_dir1copy = rpath.RPath(Globals.local_connection, 13 | "testfiles/hardlinks/dir1copy") 14 | hardlink_dir2 = rpath.RPath(Globals.local_connection, 15 | "testfiles/hardlinks/dir2") 16 | hardlink_dir3 = rpath.RPath(Globals.local_connection, 17 | "testfiles/hardlinks/dir3") 18 | 19 | def reset_output(self): 20 | """Erase and recreate testfiles/output directory""" 21 | os.system(MiscDir+'/myrm testfiles/output') 22 | self.outputrp.mkdir() 23 | 24 | def testEquality(self): 25 | """Test rorp_eq function in conjunction with CompareRecursive""" 26 | assert CompareRecursive(self.hardlink_dir1, self.hardlink_dir1copy) 27 | assert CompareRecursive(self.hardlink_dir1, self.hardlink_dir2, 28 | compare_hardlinks = None) 29 | assert not CompareRecursive(self.hardlink_dir1, self.hardlink_dir2, 30 | compare_hardlinks = 1) 31 | 32 | def testBuildingDict(self): 33 | """See if the partial inode dictionary is correct""" 34 | Globals.preserve_hardlinks = 1 35 | reset_hardlink_dicts() 36 | for dsrp in selection.Select(self.hardlink_dir3).set_iter(): 37 | Hardlink.add_rorp(dsrp) 38 | 39 | assert len(Hardlink._inode_index.keys()) == 3, \ 40 | Hardlink._inode_index 41 | 42 | def testCompletedDict(self): 43 | """See if the hardlink dictionaries are built correctly""" 44 | reset_hardlink_dicts() 45 | for dsrp in selection.Select(self.hardlink_dir1).set_iter(): 46 | Hardlink.add_rorp(dsrp) 47 | Hardlink.del_rorp(dsrp) 48 | assert Hardlink._inode_index == {}, Hardlink._inode_index 49 | 50 | reset_hardlink_dicts() 51 | for dsrp in selection.Select(self.hardlink_dir2).set_iter(): 52 | Hardlink.add_rorp(dsrp) 53 | Hardlink.del_rorp(dsrp) 54 | assert Hardlink._inode_index == {}, Hardlink._inode_index 55 | 56 | def testSeries(self): 57 | """Test hardlink system by backing up and restoring a few dirs""" 58 | dirlist = ['testfiles/hardlinks/dir1', 59 | 'testfiles/hardlinks/dir2', 60 | 'testfiles/hardlinks/dir3', 61 | 'testfiles/various_file_types'] 62 | BackupRestoreSeries(None, None, dirlist, compare_hardlinks=1) 63 | BackupRestoreSeries(1, 1, dirlist, compare_hardlinks=1) 64 | 65 | def testInnerRestore(self): 66 | """Restore part of a dir, see if hard links preserved""" 67 | MakeOutputDir() 68 | output = rpath.RPath(Globals.local_connection, 69 | "testfiles/output") 70 | 71 | # Now set up directories out_hardlink1 and out_hardlink2 72 | hlout1 = rpath.RPath(Globals.local_connection, 73 | "testfiles/out_hardlink1") 74 | if hlout1.lstat(): hlout1.delete() 75 | hlout1.mkdir() 76 | hlout1_sub = hlout1.append("subdir") 77 | hlout1_sub.mkdir() 78 | hl1_1 = hlout1_sub.append("hardlink1") 79 | hl1_2 = hlout1_sub.append("hardlink2") 80 | hl1_3 = hlout1_sub.append("hardlink3") 81 | hl1_4 = hlout1_sub.append("hardlink4") 82 | # 1 and 2 are hard linked, as are 3 and 4 83 | hl1_1.touch() 84 | hl1_2.hardlink(hl1_1.path) 85 | hl1_3.touch() 86 | hl1_4.hardlink(hl1_3.path) 87 | 88 | hlout2 = rpath.RPath(Globals.local_connection, 89 | "testfiles/out_hardlink2") 90 | if hlout2.lstat(): hlout2.delete() 91 | assert not os.system("cp -a testfiles/out_hardlink1 " 92 | "testfiles/out_hardlink2") 93 | hlout2_sub = hlout2.append("subdir") 94 | hl2_1 = hlout2_sub.append("hardlink1") 95 | hl2_2 = hlout2_sub.append("hardlink2") 96 | hl2_3 = hlout2_sub.append("hardlink3") 97 | hl2_4 = hlout2_sub.append("hardlink4") 98 | # Now 2 and 3 are hard linked, also 1 and 4 99 | rpath.copy_with_attribs(hl1_1, hl2_1) 100 | rpath.copy_with_attribs(hl1_2, hl2_2) 101 | hl2_3.delete() 102 | hl2_3.hardlink(hl2_2.path) 103 | hl2_4.delete() 104 | hl2_4.hardlink(hl2_1.path) 105 | rpath.copy_attribs(hlout1_sub, hlout2_sub) 106 | 107 | # Now try backing up twice, making sure hard links are preserved 108 | InternalBackup(1, 1, hlout1.path, output.path) 109 | out_subdir = output.append("subdir") 110 | assert out_subdir.append("hardlink1").getinode() == \ 111 | out_subdir.append("hardlink2").getinode() 112 | assert out_subdir.append("hardlink3").getinode() == \ 113 | out_subdir.append("hardlink4").getinode() 114 | assert out_subdir.append("hardlink1").getinode() != \ 115 | out_subdir.append("hardlink3").getinode() 116 | 117 | time.sleep(1) 118 | InternalBackup(1, 1, hlout2.path, output.path) 119 | out_subdir.setdata() 120 | assert out_subdir.append("hardlink1").getinode() == \ 121 | out_subdir.append("hardlink4").getinode() 122 | assert out_subdir.append("hardlink2").getinode() == \ 123 | out_subdir.append("hardlink3").getinode() 124 | assert out_subdir.append("hardlink1").getinode() != \ 125 | out_subdir.append("hardlink2").getinode() 126 | 127 | # Now try restoring, still checking hard links. 128 | out2 = rpath.RPath(Globals.local_connection, "testfiles/out2") 129 | hlout1 = out2.append("hardlink1") 130 | hlout2 = out2.append("hardlink2") 131 | hlout3 = out2.append("hardlink3") 132 | hlout4 = out2.append("hardlink4") 133 | 134 | if out2.lstat(): out2.delete() 135 | InternalRestore(1, 1, "testfiles/output/subdir", "testfiles/out2", 1) 136 | out2.setdata() 137 | for rp in [hlout1, hlout2, hlout3, hlout4]: rp.setdata() 138 | assert hlout1.getinode() == hlout2.getinode() 139 | assert hlout3.getinode() == hlout4.getinode() 140 | assert hlout1.getinode() != hlout3.getinode() 141 | 142 | if out2.lstat(): out2.delete() 143 | InternalRestore(1, 1, "testfiles/output/subdir", "testfiles/out2", 144 | int(time.time())) 145 | out2.setdata() 146 | for rp in [hlout1, hlout2, hlout3, hlout4]: rp.setdata() 147 | assert hlout1.getinode() == hlout4.getinode(), \ 148 | "%s %s" % (hlout1.path, hlout4.path) 149 | assert hlout2.getinode() == hlout3.getinode() 150 | assert hlout1.getinode() != hlout2.getinode() 151 | 152 | 153 | if __name__ == "__main__": unittest.main() 154 | -------------------------------------------------------------------------------- /rdiff-backup/testing/hashtest.py: -------------------------------------------------------------------------------- 1 | import unittest, StringIO 2 | from rdiff_backup import hash 3 | from commontest import * 4 | 5 | class HashTest(unittest.TestCase): 6 | """Test the hash module""" 7 | s1 = "Hello, world!" 8 | s1_hash = "943a702d06f34599aee1f8da8ef9f7296031d699" 9 | s2 = "The quick brown dog jumped over the lazy fox" 10 | s2_hash = "eab21fb1a18b408909bae552b847f6b13f370f62" 11 | s3 = "foobar" 12 | s3_hash = "8843d7f92416211de9ebb963ff4ce28125932878" 13 | 14 | root_rp = rpath.RPath(Globals.local_connection, "testfiles") 15 | 16 | def test_basic(self): 17 | """Compare sha1sum of a few strings""" 18 | sfile = StringIO.StringIO(self.s1) 19 | fw = hash.FileWrapper(sfile) 20 | assert fw.read() == self.s1 21 | report = fw.close() 22 | assert report.sha1_digest == self.s1_hash, report.sha1_digest 23 | 24 | sfile2 = StringIO.StringIO(self.s1) 25 | fw2 = hash.FileWrapper(sfile2) 26 | assert fw2.read(5) == self.s1[:5] 27 | assert fw2.read() == self.s1[5:] 28 | report2 = fw2.close() 29 | assert report2.sha1_digest == self.s1_hash, report2.sha1_digest 30 | 31 | def make_dirs(self): 32 | """Make two input directories""" 33 | d1 = self.root_rp.append("hashtest1") 34 | re_init_dir(d1) 35 | d2 = self.root_rp.append("hashtest2") 36 | re_init_dir(d2) 37 | 38 | d1f1 = d1.append("file1") 39 | d1f1.write_string(self.s1) 40 | d1f1l = d1.append("file1_linked") 41 | d1f1l.hardlink(d1f1.path) 42 | 43 | d1f2 = d1.append("file2") 44 | d1f2.write_string(self.s2) 45 | d1f2l = d1.append("file2_linked") 46 | d1f2l.hardlink(d1f2.path) 47 | 48 | d1_hashlist = [None, self.s1_hash, self.s1_hash, 49 | self.s2_hash, self.s2_hash] 50 | 51 | d2f1 = d2.append("file1") 52 | rpath.copy_with_attribs(d1f1, d2f1) 53 | d2f1l = d2.append("file1_linked") 54 | d2f1l.write_string(self.s3) 55 | 56 | d1f2 = d2.append("file2") 57 | d1f2.mkdir() 58 | 59 | d2_hashlist = [None, self.s1_hash, self.s3_hash, None] 60 | 61 | return (d1, d1_hashlist, d2, d2_hashlist) 62 | 63 | def extract_hashs(self, metadata_rp): 64 | """Return list of hashes in the metadata_rp""" 65 | result = [] 66 | comp = metadata_rp.isinccompressed() 67 | extractor = metadata.RorpExtractor(metadata_rp.open("r", comp)) 68 | for rorp in extractor.iterate(): 69 | if rorp.has_sha1(): result.append(rorp.get_sha1()) 70 | else: result.append(None) 71 | return result 72 | 73 | def test_session(self): 74 | """Run actual sessions and make sure proper hashes recorded 75 | 76 | There are a few code paths here we need to test: creating 77 | ordinary files, updating ordinary files with diffs, hard 78 | linking, and keeping files the same. 79 | 80 | """ 81 | in_rp1, hashlist1, in_rp2, hashlist2 = self.make_dirs() 82 | Myrm("testfiles/output") 83 | 84 | rdiff_backup(1, 1, in_rp1.path, "testfiles/output", 10000, "-v3") 85 | meta_prefix = rpath.RPath(Globals.local_connection, 86 | "testfiles/output/rdiff-backup-data/mirror_metadata") 87 | incs = restore.get_inclist(meta_prefix) 88 | assert len(incs) == 1 89 | metadata_rp = incs[0] 90 | hashlist = self.extract_hashs(metadata_rp) 91 | assert hashlist == hashlist1, (hashlist1, hashlist) 92 | 93 | rdiff_backup(1, 1, in_rp2.path, "testfiles/output", 20000, "-v3") 94 | incs = restore.get_inclist(meta_prefix) 95 | assert len(incs) == 2 96 | if incs[0].getinctype() == 'snapshot': inc = incs[0] 97 | else: inc = incs[1] 98 | hashlist = self.extract_hashs(inc) 99 | assert hashlist == hashlist2, (hashlist2, hashlist) 100 | 101 | def test_rorpiter_xfer(self): 102 | """Test if hashes are transferred in files, rorpiter""" 103 | #log.Log.setverbosity(5) 104 | Globals.security_level = 'override' 105 | conn = SetConnections.init_connection('python ./server.py .') 106 | assert conn.reval("lambda x: x+1", 4) == 5 # connection sanity check 107 | 108 | fp = hash.FileWrapper(StringIO.StringIO(self.s1)) 109 | conn.Globals.set('tmp_file', fp) 110 | fp_remote = conn.Globals.get('tmp_file') 111 | assert fp_remote.read() == self.s1 112 | assert fp_remote.close().sha1_digest == self.s1_hash 113 | 114 | # Tested xfer of file, now test xfer of files in rorpiter 115 | root = MakeOutputDir() 116 | rp1 = root.append('s1') 117 | rp1.write_string(self.s1) 118 | rp2 = root.append('s2') 119 | rp2.write_string(self.s2) 120 | rp1.setfile(hash.FileWrapper(rp1.open('rb'))) 121 | rp2.setfile(hash.FileWrapper(rp2.open('rb'))) 122 | rpiter = iter([rp1, rp2]) 123 | 124 | conn.Globals.set('tmp_conn_iter', rpiter) 125 | remote_iter = conn.Globals.get('tmp_conn_iter') 126 | 127 | rorp1 = remote_iter.next() 128 | fp = rorp1.open('rb') 129 | assert fp.read() == self.s1, fp.read() 130 | ret_val = fp.close() 131 | assert isinstance(ret_val, hash.Report), ret_val 132 | assert ret_val.sha1_digest == self.s1_hash 133 | rorp2 = remote_iter.next() 134 | fp2 = rorp1.open('rb') 135 | assert fp2.close().sha1_digest == self.s2_hash 136 | 137 | conn.quit() 138 | 139 | 140 | from rdiff_backup import rpath, regress, restore, metadata, log, Globals 141 | 142 | if __name__ == "__main__": unittest.main() 143 | -------------------------------------------------------------------------------- /rdiff-backup/testing/incrementtest.py: -------------------------------------------------------------------------------- 1 | import unittest, os, re, time 2 | from commontest import * 3 | from rdiff_backup import log, rpath, increment, Time, Rdiff, statistics 4 | 5 | lc = Globals.local_connection 6 | Globals.change_source_perms = 1 7 | Log.setverbosity(3) 8 | 9 | def getrp(ending): 10 | return rpath.RPath(lc, "testfiles/various_file_types/" + ending) 11 | 12 | rf = getrp("regular_file") 13 | rf2 = getrp("two_hardlinked_files1") 14 | exec1 = getrp("executable") 15 | exec2 = getrp("executable2") 16 | sig = getrp("regular_file.sig") 17 | hl1, hl2 = map(getrp, ["two_hardlinked_files1", "two_hardlinked_files2"]) 18 | test = getrp("test") 19 | dir = getrp(".") 20 | sym = getrp("symbolic_link") 21 | nothing = getrp("nothing") 22 | 23 | target = rpath.RPath(lc, "testfiles/output/out") 24 | out2 = rpath.RPath(lc, "testfiles/output/out2") 25 | out_gz = rpath.RPath(lc, "testfiles/output/out.gz") 26 | 27 | Time.setcurtime(1000000000) 28 | Time.setprevtime(999424113) 29 | prevtimestr = "2001-09-02T02:48:33-07:00" 30 | t_pref = "testfiles/output/out.2001-09-02T02:48:33-07:00" 31 | t_diff = "testfiles/output/out.2001-09-02T02:48:33-07:00.diff" 32 | 33 | Globals.no_compression_regexp = \ 34 | re.compile(Globals.no_compression_regexp_string, re.I) 35 | 36 | class inctest(unittest.TestCase): 37 | """Test the incrementRP function""" 38 | def setUp(self): 39 | Globals.set('isbackup_writer',1) 40 | MakeOutputDir() 41 | 42 | def check_time(self, rp): 43 | """Make sure that rp is an inc file, and time is Time.prevtime""" 44 | assert rp.isincfile(), rp 45 | t = rp.getinctime() 46 | assert t == Time.prevtime, (t, Time.prevtime) 47 | 48 | def testreg(self): 49 | """Test increment of regular files""" 50 | Globals.compression = None 51 | target.setdata() 52 | if target.lstat(): target.delete() 53 | rpd = rpath.RPath(lc, t_diff) 54 | if rpd.lstat(): rpd.delete() 55 | 56 | diffrp = increment.Increment(rf, exec1, target) 57 | assert diffrp.isreg(), diffrp 58 | assert diffrp.equal_verbose(exec1, check_index = 0, compare_size = 0) 59 | self.check_time(diffrp) 60 | assert diffrp.getinctype() == 'diff', diffrp.getinctype() 61 | diffrp.delete() 62 | 63 | def testmissing(self): 64 | """Test creation of missing files""" 65 | missing_rp = increment.Increment(rf, nothing, target) 66 | self.check_time(missing_rp) 67 | assert missing_rp.getinctype() == 'missing' 68 | missing_rp.delete() 69 | 70 | def testsnapshot(self): 71 | """Test making of a snapshot""" 72 | Globals.compression = None 73 | snap_rp = increment.Increment(rf, sym, target) 74 | self.check_time(snap_rp) 75 | assert rpath.cmp_attribs(snap_rp, sym) 76 | assert rpath.cmp(snap_rp, sym) 77 | snap_rp.delete() 78 | 79 | snap_rp2 = increment.Increment(sym, rf, target) 80 | self.check_time(snap_rp2) 81 | assert snap_rp2.equal_verbose(rf, check_index = 0) 82 | assert rpath.cmp(snap_rp2, rf) 83 | snap_rp2.delete() 84 | 85 | def testGzipsnapshot(self): 86 | """Test making a compressed snapshot""" 87 | Globals.compression = 1 88 | rp = increment.Increment(rf, sym, target) 89 | self.check_time(rp) 90 | assert rp.equal_verbose(sym, check_index = 0, compare_size = 0) 91 | assert rpath.cmp(rp, sym) 92 | rp.delete() 93 | 94 | rp = increment.Increment(sym, rf, target) 95 | self.check_time(rp) 96 | assert rp.equal_verbose(rf, check_index = 0, compare_size = 0) 97 | assert rpath.cmpfileobj(rp.open("rb", 1), rf.open("rb")) 98 | assert rp.isinccompressed() 99 | rp.delete() 100 | 101 | def testdir(self): 102 | """Test increment on dir""" 103 | rp = increment.Increment(sym, dir, target) 104 | self.check_time(rp) 105 | assert rp.lstat() 106 | assert target.isdir() 107 | assert dir.equal_verbose(rp, check_index = 0, 108 | compare_size = 0, compare_type = 0) 109 | assert rp.isreg() 110 | rp.delete() 111 | target.delete() 112 | 113 | def testDiff(self): 114 | """Test making diffs""" 115 | Globals.compression = None 116 | rp = increment.Increment(rf, rf2, target) 117 | self.check_time(rp) 118 | assert rp.equal_verbose(rf2, check_index = 0, compare_size = 0) 119 | Rdiff.patch_local(rf, rp, out2) 120 | assert rpath.cmp(rf2, out2) 121 | rp.delete() 122 | out2.delete() 123 | 124 | def testGzipDiff(self): 125 | """Test making gzipped diffs""" 126 | Globals.compression = 1 127 | rp = increment.Increment(rf, rf2, target) 128 | self.check_time(rp) 129 | assert rp.equal_verbose(rf2, check_index = 0, compare_size = 0) 130 | Rdiff.patch_local(rf, rp, out2, delta_compressed = 1) 131 | assert rpath.cmp(rf2, out2) 132 | rp.delete() 133 | out2.delete() 134 | 135 | def testGzipRegexp(self): 136 | """Here a .gz file shouldn't be compressed""" 137 | Globals.compression = 1 138 | rpath.copy(rf, out_gz) 139 | assert out_gz.lstat() 140 | 141 | rp = increment.Increment(rf, out_gz, target) 142 | self.check_time(rp) 143 | assert rp.equal_verbose(out_gz, check_index = 0, compare_size = 0) 144 | Rdiff.patch_local(rf, rp, out2) 145 | assert rpath.cmp(out_gz, out2) 146 | rp.delete() 147 | out2.delete() 148 | out_gz.delete() 149 | 150 | if __name__ == '__main__': unittest.main() 151 | -------------------------------------------------------------------------------- /rdiff-backup/testing/iterfiletest.py: -------------------------------------------------------------------------------- 1 | import unittest, StringIO 2 | from commontest import * 3 | from rdiff_backup.iterfile import * 4 | from rdiff_backup import lazy 5 | 6 | class FileException: 7 | """Like a file, but raise exception after certain # bytes read""" 8 | def __init__(self, max): 9 | self.count = 0 10 | self.max = max 11 | def read(self, l): 12 | self.count += l 13 | if self.count > self.max: raise IOError(13, "Permission Denied") 14 | return "a"*l 15 | def close(self): return None 16 | 17 | 18 | class testIterFile(unittest.TestCase): 19 | def setUp(self): 20 | self.iter1maker = lambda: iter(range(50)) 21 | self.iter2maker = lambda: iter(map(str, range(50))) 22 | 23 | def testConversion(self): 24 | """Test iter to file conversion""" 25 | for itm in [self.iter1maker, self.iter2maker]: 26 | assert lazy.Iter.equal(itm(), 27 | IterWrappingFile(FileWrappingIter(itm()))) 28 | 29 | def testFile(self): 30 | """Test sending files through iters""" 31 | buf1 = "hello"*10000 32 | file1 = StringIO.StringIO(buf1) 33 | buf2 = "goodbye"*10000 34 | file2 = StringIO.StringIO(buf2) 35 | file_iter = FileWrappingIter(iter([file1, file2])) 36 | 37 | new_iter = IterWrappingFile(file_iter) 38 | assert new_iter.next().read() == buf1 39 | assert new_iter.next().read() == buf2 40 | self.assertRaises(StopIteration, new_iter.next) 41 | 42 | def testFileException(self): 43 | """Test encoding a file which raises an exception""" 44 | f = FileException(200*1024) # size depends on buffer size 45 | new_iter = IterWrappingFile(FileWrappingIter(iter([f, "foo"]))) 46 | f_out = new_iter.next() 47 | assert f_out.read(50000) == "a"*50000 48 | try: buf = f_out.read(190*1024) 49 | except IOError: pass 50 | else: assert 0, len(buf) 51 | 52 | assert new_iter.next() == "foo" 53 | self.assertRaises(StopIteration, new_iter.next) 54 | 55 | 56 | class testMiscIters(unittest.TestCase): 57 | """Test sending rorpiter back and forth""" 58 | def setUp(self): 59 | """Make testfiles/output directory and a few files""" 60 | Myrm("testfiles/output") 61 | self.outputrp = rpath.RPath(Globals.local_connection, 62 | "testfiles/output") 63 | self.regfile1 = self.outputrp.append("reg1") 64 | self.regfile2 = self.outputrp.append("reg2") 65 | self.regfile3 = self.outputrp.append("reg3") 66 | 67 | self.outputrp.mkdir() 68 | 69 | fp = self.regfile1.open("wb") 70 | fp.write("hello") 71 | fp.close() 72 | self.regfile1.setfile(self.regfile1.open("rb")) 73 | 74 | self.regfile2.touch() 75 | self.regfile2.setfile(self.regfile2.open("rb")) 76 | 77 | fp = self.regfile3.open("wb") 78 | fp.write("goodbye") 79 | fp.close() 80 | self.regfile3.setfile(self.regfile3.open("rb")) 81 | 82 | self.regfile1.setdata() 83 | self.regfile2.setdata() 84 | self.regfile3.setdata() 85 | 86 | def print_MiscIterFile(self, rpiter_file): 87 | """Print the given rorpiter file""" 88 | while 1: 89 | buf = rpiter_file.read() 90 | sys.stdout.write(buf) 91 | if buf[0] == "z": break 92 | 93 | def testBasic(self): 94 | """Test basic conversion""" 95 | l = [self.outputrp, self.regfile1, self.regfile2, self.regfile3] 96 | i_out = FileToMiscIter(MiscIterToFile(iter(l))) 97 | 98 | out1 = i_out.next() 99 | assert out1 == self.outputrp 100 | 101 | out2 = i_out.next() 102 | assert out2 == self.regfile1 103 | fp = out2.open("rb") 104 | assert fp.read() == "hello" 105 | assert not fp.close() 106 | 107 | out3 = i_out.next() 108 | assert out3 == self.regfile2 109 | fp = out3.open("rb") 110 | assert fp.read() == "" 111 | assert not fp.close() 112 | 113 | i_out.next() 114 | self.assertRaises(StopIteration, i_out.next) 115 | 116 | def testMix(self): 117 | """Test a mix of RPs and ordinary objects""" 118 | l = [5, self.regfile3, "hello"] 119 | s = MiscIterToFile(iter(l)).read() 120 | i_out = FileToMiscIter(StringIO.StringIO(s)) 121 | 122 | out1 = i_out.next() 123 | assert out1 == 5, out1 124 | 125 | out2 = i_out.next() 126 | assert out2 == self.regfile3 127 | fp = out2.open("rb") 128 | assert fp.read() == "goodbye" 129 | assert not fp.close() 130 | 131 | out3 = i_out.next() 132 | assert out3 == "hello", out3 133 | 134 | self.assertRaises(StopIteration, i_out.next) 135 | 136 | def testFlush(self): 137 | """Test flushing property of MiscIterToFile""" 138 | l = [self.outputrp, MiscIterFlush, self.outputrp] 139 | filelike = MiscIterToFile(iter(l)) 140 | new_filelike = StringIO.StringIO((filelike.read() + "z" + 141 | C.long2str(0L))) 142 | 143 | i_out = FileToMiscIter(new_filelike) 144 | assert i_out.next() == self.outputrp 145 | self.assertRaises(StopIteration, i_out.next) 146 | 147 | i_out2 = FileToMiscIter(filelike) 148 | assert i_out2.next() == self.outputrp 149 | self.assertRaises(StopIteration, i_out2.next) 150 | 151 | def testFlushRepeat(self): 152 | """Test flushing like above, but have Flush obj emerge from iter""" 153 | l = [self.outputrp, MiscIterFlushRepeat, self.outputrp] 154 | filelike = MiscIterToFile(iter(l)) 155 | new_filelike = StringIO.StringIO((filelike.read() + "z" + 156 | C.long2str(0L))) 157 | 158 | i_out = FileToMiscIter(new_filelike) 159 | assert i_out.next() == self.outputrp 160 | assert i_out.next() is MiscIterFlushRepeat 161 | self.assertRaises(StopIteration, i_out.next) 162 | 163 | i_out2 = FileToMiscIter(filelike) 164 | assert i_out2.next() == self.outputrp 165 | self.assertRaises(StopIteration, i_out2.next) 166 | 167 | 168 | 169 | if __name__ == "__main__": unittest.main() 170 | -------------------------------------------------------------------------------- /rdiff-backup/testing/journaltest.py: -------------------------------------------------------------------------------- 1 | from commontest import * 2 | import unittest 3 | from rdiff_backup import journal, Globals, rpath 4 | 5 | class JournalTest(unittest.TestCase): 6 | def testBasic(self): 7 | """Test opening a journal, then reading, writing, and deleting""" 8 | MakeOutputDir() 9 | Globals.dest_root = rpath.RPath(Globals.local_connection, 10 | "testfiles/output") 11 | Globals.rbdir = Globals.dest_root.append("rdiff-backup-data") 12 | 13 | Globals.rbdir.mkdir() 14 | journal.open_journal() 15 | assert len(journal.get_entries_from_journal()) == 0 16 | 17 | # It's important that none of these files really exist 18 | e1 = journal.write_entry(("Hello48",), ("temp_index", "foo"), 19 | 2, "reg") 20 | e2 = journal.write_entry(("2nd", "Entry", "now"), 21 | ("temp_index",), 1, None) 22 | assert e1.entry_rp and e2.entry_rp 23 | 24 | l = journal.get_entries_from_journal() 25 | assert len(l) == 2 26 | first_index = l[0].index 27 | assert (first_index == ("Hello48",) or 28 | first_index == ("2nd", "Entry", "now")) 29 | 30 | # Now test recovering journal, and make sure everything deleted 31 | journal.recover_journal() 32 | assert len(journal.get_entries_from_journal()) == 0 33 | 34 | 35 | if __name__ == "__main__": unittest.main() 36 | -------------------------------------------------------------------------------- /rdiff-backup/testing/lazytest.py: -------------------------------------------------------------------------------- 1 | from __future__ import generators 2 | import unittest, pickle 3 | from commontest import * 4 | from rdiff_backup.lazy import * 5 | 6 | class Iterators(unittest.TestCase): 7 | one_to_100 = lambda s: iter(range(1, 101)) 8 | evens = lambda s: iter(range(2, 101, 2)) 9 | odds = lambda s: iter(range(1, 100, 2)) 10 | empty = lambda s: iter([]) 11 | 12 | def __init__(self, *args): 13 | apply (unittest.TestCase.__init__, (self,) + args) 14 | self.falseerror = self.falseerror_maker() 15 | self.trueerror = self.trueerror_maker() 16 | self.emptygen = self.emptygen_maker() 17 | self.typeerror = self.typeerror_maker() 18 | self.nameerror = self.nameerror_maker() 19 | 20 | def falseerror_maker(self): 21 | yield None 22 | yield 0 23 | yield [] 24 | raise Exception 25 | 26 | def trueerror_maker(self): 27 | yield 1 28 | yield "hello" 29 | yield (2, 3) 30 | raise Exception 31 | 32 | def nameerror_maker(self): 33 | if 0: yield 1 34 | raise NameError 35 | 36 | def typeerror_maker(self): 37 | yield 1 38 | yield 2 39 | raise TypeError 40 | 41 | def alwayserror(self, x): 42 | raise Exception 43 | 44 | def emptygen_maker(self): 45 | if 0: yield 1 46 | 47 | 48 | class IterEqualTestCase(Iterators): 49 | """Tests for iter_equal function""" 50 | def testEmpty(self): 51 | """Empty iterators should be equal""" 52 | assert Iter.equal(self.empty(), iter([])) 53 | 54 | def testNormal(self): 55 | """See if normal iterators are equal""" 56 | assert Iter.equal(iter((1,2,3)), iter((1,2,3))) 57 | assert Iter.equal(self.odds(), iter(range(1, 100, 2))) 58 | assert Iter.equal(iter((1,2,3)), iter(range(1, 4))) 59 | 60 | def testNormalInequality(self): 61 | """See if normal unequals work""" 62 | assert not Iter.equal(iter((1,2,3)), iter((1,2,4))) 63 | assert not Iter.equal(self.odds(), iter(["hello", "there"])) 64 | 65 | def testGenerators(self): 66 | """equals works for generators""" 67 | def f(): 68 | yield 1 69 | yield "hello" 70 | def g(): 71 | yield 1 72 | yield "hello" 73 | assert Iter.equal(f(), g()) 74 | 75 | def testLength(self): 76 | """Differently sized iterators""" 77 | assert not Iter.equal(iter((1,2,3)), iter((1,2))) 78 | assert not Iter.equal(iter((1,2)), iter((1,2,3))) 79 | 80 | 81 | class FilterTestCase(Iterators): 82 | """Tests for lazy_filter function""" 83 | def testEmpty(self): 84 | """empty iterators -> empty iterators""" 85 | assert Iter.empty(Iter.filter(self.alwayserror, 86 | self.empty())), \ 87 | "Filtering an empty iterator should result in empty iterator" 88 | 89 | def testNum1(self): 90 | """Test numbers 1 - 100 #1""" 91 | assert Iter.equal(Iter.filter(lambda x: x % 2 == 0, 92 | self.one_to_100()), 93 | self.evens()) 94 | assert Iter.equal(Iter.filter(lambda x: x % 2, 95 | self.one_to_100()), 96 | self.odds()) 97 | 98 | def testError(self): 99 | """Should raise appropriate error""" 100 | i = Iter.filter(lambda x: x, self.falseerror_maker()) 101 | self.assertRaises(Exception, i.next) 102 | 103 | 104 | class MapTestCase(Iterators): 105 | """Test mapping of iterators""" 106 | def testNumbers(self): 107 | """1 to 100 * 2 = 2 to 200""" 108 | assert Iter.equal(Iter.map(lambda x: 2*x, self.one_to_100()), 109 | iter(range(2, 201, 2))) 110 | 111 | def testShortcut(self): 112 | """Map should go in order""" 113 | def f(x): 114 | if x == "hello": 115 | raise NameError 116 | i = Iter.map(f, self.trueerror_maker()) 117 | i.next() 118 | self.assertRaises(NameError, i.next) 119 | 120 | def testEmpty(self): 121 | """Map of an empty iterator is empty""" 122 | assert Iter.empty(Iter.map(lambda x: x, iter([]))) 123 | 124 | 125 | class CatTestCase(Iterators): 126 | """Test concatenation of iterators""" 127 | def testEmpty(self): 128 | """Empty + empty = empty""" 129 | assert Iter.empty(Iter.cat(iter([]), iter([]))) 130 | 131 | def testNumbers(self): 132 | """1 to 50 + 51 to 100 = 1 to 100""" 133 | assert Iter.equal(Iter.cat(iter(range(1, 51)), iter(range(51, 101))), 134 | self.one_to_100()) 135 | 136 | def testShortcut(self): 137 | """Process iterators in order""" 138 | i = Iter.cat(self.typeerror_maker(), self.nameerror_maker()) 139 | i.next() 140 | i.next() 141 | self.assertRaises(TypeError, i.next) 142 | 143 | 144 | class AndOrTestCase(Iterators): 145 | """Test And and Or""" 146 | def testEmpty(self): 147 | """And() -> true, Or() -> false""" 148 | assert Iter.And(self.empty()) 149 | assert not Iter.Or(self.empty()) 150 | 151 | def testAndShortcut(self): 152 | """And should return if any false""" 153 | assert Iter.And(self.falseerror_maker()) is None 154 | 155 | def testOrShortcut(self): 156 | """Or should return if any true""" 157 | assert Iter.Or(self.trueerror_maker()) == 1 158 | 159 | def testNormalAnd(self): 160 | """And should go through true iterators, picking last""" 161 | assert Iter.And(iter([1,2,3,4])) == 4 162 | self.assertRaises(Exception, Iter.And, self.trueerror_maker()) 163 | 164 | def testNormalOr(self): 165 | """Or goes through false iterators, picking last""" 166 | assert Iter.Or(iter([0, None, []])) == [] 167 | self.assertRaises(Exception, Iter.Or, self.falseerror_maker()) 168 | 169 | 170 | class FoldingTest(Iterators): 171 | """Test folding operations""" 172 | def f(self, x, y): return x + y 173 | 174 | def testEmpty(self): 175 | """Folds of empty iterators should produce defaults""" 176 | assert Iter.foldl(self.f, 23, self.empty()) == 23 177 | assert Iter.foldr(self.f, 32, self.empty()) == 32 178 | 179 | def testAddition(self): 180 | """Use folds to sum lists""" 181 | assert Iter.foldl(self.f, 0, self.one_to_100()) == 5050 182 | assert Iter.foldr(self.f, 0, self.one_to_100()) == 5050 183 | 184 | def testLargeAddition(self): 185 | """Folds on 10000 element iterators""" 186 | assert Iter.foldl(self.f, 0, iter(range(1, 10001))) == 50005000 187 | self.assertRaises(RuntimeError, 188 | Iter.foldr, self.f, 0, iter(range(1, 10001))) 189 | 190 | def testLen(self): 191 | """Use folds to calculate length of lists""" 192 | assert Iter.foldl(lambda x, y: x+1, 0, self.evens()) == 50 193 | assert Iter.foldr(lambda x, y: y+1, 0, self.odds()) == 50 194 | 195 | class MultiplexTest(Iterators): 196 | def testSingle(self): 197 | """Test multiplex single stream""" 198 | i_orig = self.one_to_100() 199 | i2_orig = self.one_to_100() 200 | i = Iter.multiplex(i_orig, 1)[0] 201 | assert Iter.equal(i, i2_orig) 202 | 203 | def testTrible(self): 204 | """Test splitting iterator into three""" 205 | counter = [0] 206 | def ff(x): counter[0] += 1 207 | i_orig = self.one_to_100() 208 | i2_orig = self.one_to_100() 209 | i1, i2, i3 = Iter.multiplex(i_orig, 3, ff) 210 | assert Iter.equal(i1, i2) 211 | assert Iter.equal(i3, i2_orig) 212 | assert counter[0] == 100, counter 213 | 214 | def testDouble(self): 215 | """Test splitting into two...""" 216 | i1, i2 = Iter.multiplex(self.one_to_100(), 2) 217 | assert Iter.equal(i1, self.one_to_100()) 218 | assert Iter.equal(i2, self.one_to_100()) 219 | 220 | 221 | if __name__ == "__main__": unittest.main() 222 | -------------------------------------------------------------------------------- /rdiff-backup/testing/librsynctest.py: -------------------------------------------------------------------------------- 1 | import unittest, random 2 | from commontest import * 3 | from rdiff_backup import librsync, log 4 | 5 | def MakeRandomFile(path, length = None): 6 | """Writes a random file of given length, or random len if unspecified""" 7 | if not length: length = random.randrange(5000, 100000) 8 | fp = open(path, "wb") 9 | fp_random = open('/dev/urandom', 'rb') 10 | 11 | # Old slow way, may still be of use on systems without /dev/urandom 12 | #randseq = [] 13 | #for i in xrange(random.randrange(5000, 30000)): 14 | # randseq.append(chr(random.randrange(256))) 15 | #fp.write("".join(randseq)) 16 | fp.write(fp_random.read(length)) 17 | 18 | fp.close() 19 | fp_random.close() 20 | 21 | class LibrsyncTest(unittest.TestCase): 22 | """Test various librsync wrapper functions""" 23 | basis = RPath(Globals.local_connection, "testfiles/basis") 24 | new = RPath(Globals.local_connection, "testfiles/new") 25 | new2 = RPath(Globals.local_connection, "testfiles/new2") 26 | sig = RPath(Globals.local_connection, "testfiles/signature") 27 | sig2 = RPath(Globals.local_connection, "testfiles/signature2") 28 | delta = RPath(Globals.local_connection, "testfiles/delta") 29 | def sig_file_test_helper(self, blocksize, iterations, file_len = None): 30 | """Compare SigFile output to rdiff output at given blocksize""" 31 | for i in range(iterations): 32 | MakeRandomFile(self.basis.path, file_len) 33 | if self.sig.lstat(): self.sig.delete() 34 | assert not os.system("rdiff -b %s signature %s %s" % 35 | (blocksize, self.basis.path, self.sig.path)) 36 | fp = self.sig.open("rb") 37 | rdiff_sig = fp.read() 38 | fp.close() 39 | 40 | sf = librsync.SigFile(self.basis.open("rb"), blocksize) 41 | librsync_sig = sf.read() 42 | sf.close() 43 | 44 | assert rdiff_sig == librsync_sig, \ 45 | (len(rdiff_sig), len(librsync_sig)) 46 | 47 | def testSigFile(self): 48 | """Make sure SigFile generates same data as rdiff, blocksize 512""" 49 | self.sig_file_test_helper(512, 5) 50 | 51 | def testSigFile2(self): 52 | """Test SigFile like above, but try various blocksize""" 53 | self.sig_file_test_helper(2048, 1, 60000) 54 | self.sig_file_test_helper(7168, 1, 6000) 55 | self.sig_file_test_helper(204800, 1, 40*1024*1024) 56 | 57 | def testSigGenerator(self): 58 | """Test SigGenerator, make sure it's same as SigFile""" 59 | for i in range(5): 60 | MakeRandomFile(self.basis.path) 61 | 62 | sf = librsync.SigFile(self.basis.open("rb")) 63 | sigfile_string = sf.read() 64 | sf.close() 65 | 66 | sig_gen = librsync.SigGenerator() 67 | infile = self.basis.open("rb") 68 | while 1: 69 | buf = infile.read(1000) 70 | if not buf: break 71 | sig_gen.update(buf) 72 | siggen_string = sig_gen.getsig() 73 | 74 | assert sigfile_string == siggen_string, \ 75 | (len(sigfile_string), len(siggen_string)) 76 | 77 | def OldtestDelta(self): 78 | """Test delta generation against Rdiff""" 79 | MakeRandomFile(self.basis.path) 80 | assert not os.system("rdiff signature %s %s" % 81 | (self.basis.path, self.sig.path)) 82 | for i in range(5): 83 | MakeRandomFile(self.new.path) 84 | assert not os.system("rdiff delta %s %s %s" % 85 | (self.sig.path, self.new.path, self.delta.path)) 86 | fp = self.delta.open("rb") 87 | rdiff_delta = fp.read() 88 | fp.close() 89 | 90 | df = librsync.DeltaFile(self.sig.open("rb"), self.new.open("rb")) 91 | librsync_delta = df.read() 92 | df.close() 93 | 94 | print len(rdiff_delta), len(librsync_delta) 95 | print repr(rdiff_delta[:100]) 96 | print repr(librsync_delta[:100]) 97 | assert rdiff_delta == librsync_delta 98 | 99 | def testDelta(self): 100 | """Test delta generation by making sure rdiff can process output 101 | 102 | There appears to be some undeterminism so we can't just 103 | byte-compare the deltas produced by rdiff and DeltaFile. 104 | 105 | """ 106 | MakeRandomFile(self.basis.path) 107 | assert not os.system("rdiff signature %s %s" % 108 | (self.basis.path, self.sig.path)) 109 | for i in range(5): 110 | MakeRandomFile(self.new.path) 111 | df = librsync.DeltaFile(self.sig.open("rb"), self.new.open("rb")) 112 | librsync_delta = df.read() 113 | df.close() 114 | fp = self.delta.open("wb") 115 | fp.write(librsync_delta) 116 | fp.close() 117 | 118 | assert not os.system("rdiff patch %s %s %s" % 119 | (self.basis.path, self.delta.path, 120 | self.new2.path)) 121 | new_fp = self.new.open("rb") 122 | new = new_fp.read() 123 | new_fp.close() 124 | 125 | new2_fp = self.new2.open("rb") 126 | new2 = new2_fp.read() 127 | new2_fp.close() 128 | 129 | assert new == new2, (len(new), len(new2)) 130 | 131 | def testPatch(self): 132 | """Test patching against Rdiff""" 133 | MakeRandomFile(self.basis.path) 134 | assert not os.system("rdiff signature %s %s" % 135 | (self.basis.path, self.sig.path)) 136 | for i in range(5): 137 | MakeRandomFile(self.new.path) 138 | assert not os.system("rdiff delta %s %s %s" % 139 | (self.sig.path, self.new.path, self.delta.path)) 140 | fp = self.new.open("rb") 141 | real_new = fp.read() 142 | fp.close() 143 | 144 | pf = librsync.PatchedFile(self.basis.open("rb"), 145 | self.delta.open("rb")) 146 | librsync_new = pf.read() 147 | pf.close() 148 | 149 | assert real_new == librsync_new, \ 150 | (len(real_new), len(librsync_new)) 151 | 152 | 153 | 154 | if __name__ == "__main__": unittest.main() 155 | 156 | -------------------------------------------------------------------------------- /rdiff-backup/testing/longnametest.py: -------------------------------------------------------------------------------- 1 | import unittest, errno 2 | from commontest import * 3 | from rdiff_backup import rpath, longname, Globals, regress 4 | 5 | max_len = 255 6 | 7 | class LongNameTest(unittest.TestCase): 8 | """Test the longname module""" 9 | root_rp = rpath.RPath(Globals.local_connection, "testfiles") 10 | out_rp = root_rp.append_path('output') 11 | 12 | def test_length_limit(self): 13 | """Confirm that length limit is max_len 14 | 15 | Some of these tests depend on the length being at most 16 | max_len, so check to make sure it's accurate. 17 | 18 | """ 19 | Myrm(self.out_rp.path) 20 | self.out_rp.mkdir() 21 | 22 | really_long = self.out_rp.append('a'*max_len) 23 | really_long.touch() 24 | 25 | try: too_long = self.out_rp.append("a"*(max_len+1)) 26 | except EnvironmentError, e: 27 | assert errno.errorcode[e[0]] == 'ENAMETOOLONG', e 28 | else: assert 0, "File made successfully with length " + str(max_len+1) 29 | 30 | def make_input_dirs(self): 31 | """Create two input directories with long filename(s) in them""" 32 | dir1 = self.root_rp.append('longname1') 33 | dir2 = self.root_rp.append('longname2') 34 | Myrm(dir1.path) 35 | Myrm(dir2.path) 36 | 37 | dir1.mkdir() 38 | rp11 = dir1.append('A'*max_len) 39 | rp11.write_string('foobar') 40 | rp12 = dir1.append('B'*max_len) 41 | rp12.mkdir() 42 | rp121 = rp12.append('C'*max_len) 43 | rp121.touch() 44 | 45 | dir2.mkdir() 46 | rp21 = dir2.append('A'*max_len) 47 | rp21.write_string('Hello, world') 48 | rp22 = dir2.append('D'*max_len) 49 | rp22.mkdir() 50 | rp221 = rp22.append('C'*max_len) 51 | rp221.touch() 52 | 53 | return dir1, dir2 54 | 55 | def check_dir1(self, dirrp): 56 | """Make sure dirrp looks like dir1""" 57 | rp1 = dirrp.append('A'*max_len) 58 | assert rp1.get_data() == 'foobar', "data doesn't match" 59 | rp2 = dirrp.append('B'*max_len) 60 | assert rp2.isdir(), rp2 61 | rp21 = rp2.append('C'*max_len) 62 | assert rp21.isreg(), rp21 63 | 64 | def check_dir2(self, dirrp): 65 | """Make sure dirrp looks like dir2""" 66 | rp1 = dirrp.append('A'*max_len) 67 | assert rp1.get_data() == 'Hello, world', "data doesn't match" 68 | rp2 = dirrp.append('D'*max_len) 69 | assert rp2.isdir(), rp2 70 | rp21 = rp2.append('C'*max_len) 71 | assert rp21.isreg(), rp21 72 | 73 | def generic_test(self, inlocal, outlocal, extra_args, compare_back): 74 | """Used for some of the tests below""" 75 | in1, in2 = self.make_input_dirs() 76 | Myrm(self.out_rp.path) 77 | restore_dir = self.root_rp.append('longname_out') 78 | 79 | # Test backing up 80 | rdiff_backup(inlocal, outlocal, in1.path, self.out_rp.path, 10000, 81 | extra_options = extra_args) 82 | if compare_back: self.check_dir1(self.out_rp) 83 | rdiff_backup(inlocal, outlocal, in2.path, self.out_rp.path, 20000, 84 | extra_options = extra_args) 85 | if compare_back: self.check_dir2(self.out_rp) 86 | 87 | # Now try restoring 88 | Myrm(restore_dir.path) 89 | rdiff_backup(inlocal, outlocal, self.out_rp.path, restore_dir.path, 90 | 30000, extra_options = "-r now " + extra_args) 91 | self.check_dir2(restore_dir) 92 | Myrm(restore_dir.path) 93 | rdiff_backup(1, 1, self.out_rp.path, restore_dir.path, 30000, 94 | extra_options = "-r 10000 " + extra_args) 95 | self.check_dir1(restore_dir) 96 | 97 | def test_basic_local(self): 98 | """Test backup session when increment would be too long""" 99 | self.generic_test(1, 1, "", 1) 100 | 101 | def test_quoting_local(self): 102 | """Test backup session with quoting, so reg files also too long""" 103 | self.generic_test(1, 1, "--override-chars-to-quote A-Z", 0) 104 | 105 | def generic_regress_test(self, extra_args): 106 | """Used for regress tests below""" 107 | in1, in2 = self.make_input_dirs() 108 | Myrm(self.out_rp.path) 109 | restore_dir = self.root_rp.append('longname_out') 110 | Myrm(restore_dir.path) 111 | 112 | rdiff_backup(1, 1, in1.path, self.out_rp.path, 10000, 113 | extra_options = extra_args) 114 | rdiff_backup(1, 1, in2.path, self.out_rp.path, 20000, 115 | extra_options = extra_args) 116 | 117 | # Regress repository back to in1 condition 118 | Globals.rbdir = self.out_rp.append_path('rdiff-backup-data') 119 | self.add_current_mirror(10000) 120 | self.out_rp.setdata() 121 | regress.Regress(self.out_rp) 122 | 123 | # Restore in1 and compare 124 | rdiff_backup(1, 1, self.out_rp.path, restore_dir.path, 30000, 125 | extra_options = '-r now ' + extra_args) 126 | self.check_dir1(restore_dir) 127 | 128 | def add_current_mirror(self, time): 129 | """Add current_mirror marker at given time""" 130 | cur_mirror_rp = Globals.rbdir.append( 131 | "current_mirror.%s.data" % (Time.timetostring(time),)) 132 | cur_mirror_rp.touch() 133 | 134 | def test_regress_basic(self): 135 | """Test regressing when increments would be too long""" 136 | self.generic_regress_test('') 137 | 138 | 139 | if __name__ == "__main__": unittest.main() 140 | 141 | 142 | -------------------------------------------------------------------------------- /rdiff-backup/testing/makerestoretest3: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | # This script will create the testing/restoretest3 directory as it 4 | # needs to be for one of the tests in restoretest.py to work. 5 | 6 | rm -rf testfiles/restoretest3 7 | ../rdiff-backup --current-time 10000 testfiles/increment1 testfiles/restoretest3 8 | ../rdiff-backup --current-time 20000 testfiles/increment2 testfiles/restoretest3 9 | ../rdiff-backup --current-time 30000 testfiles/increment3 testfiles/restoretest3 10 | ../rdiff-backup --current-time 40000 testfiles/increment4 testfiles/restoretest3 11 | -------------------------------------------------------------------------------- /rdiff-backup/testing/rdifftest.py: -------------------------------------------------------------------------------- 1 | import unittest, random 2 | from commontest import * 3 | from rdiff_backup import Globals, Rdiff, selection, log, rpath 4 | 5 | Log.setverbosity(7) 6 | 7 | def MakeRandomFile(path): 8 | """Writes a random file of length between 10000 and 100000""" 9 | fp = open(path, "w") 10 | randseq = [] 11 | for i in xrange(random.randrange(5000, 30000)): 12 | randseq.append(chr(random.randrange(256))) 13 | fp.write("".join(randseq)) 14 | fp.close() 15 | 16 | 17 | class RdiffTest(unittest.TestCase): 18 | """Test rdiff""" 19 | lc = Globals.local_connection 20 | basis = rpath.RPath(lc, "testfiles/basis") 21 | new = rpath.RPath(lc, "testfiles/new") 22 | output = rpath.RPath(lc, "testfiles/output") 23 | delta = rpath.RPath(lc, "testfiles/delta") 24 | signature = rpath.RPath(lc, "testfiles/signature") 25 | 26 | def testRdiffSig(self): 27 | """Test making rdiff signatures""" 28 | sig = rpath.RPath(self.lc, 29 | "testfiles/various_file_types/regular_file.sig") 30 | sigfp = sig.open("r") 31 | rfsig = Rdiff.get_signature(RPath(self.lc, "testfiles/various_file_types/regular_file"), 2048) 32 | assert rpath.cmpfileobj(sigfp, rfsig) 33 | sigfp.close() 34 | rfsig.close() 35 | 36 | def testRdiffDeltaPatch(self): 37 | """Test making deltas and patching files""" 38 | rplist = [self.basis, self.new, self.delta, 39 | self.signature, self.output] 40 | for rp in rplist: 41 | if rp.lstat(): rp.delete() 42 | 43 | for i in range(2): 44 | MakeRandomFile(self.basis.path) 45 | MakeRandomFile(self.new.path) 46 | map(rpath.RPath.setdata, [self.basis, self.new]) 47 | assert self.basis.lstat() and self.new.lstat() 48 | self.signature.write_from_fileobj(Rdiff.get_signature(self.basis)) 49 | assert self.signature.lstat() 50 | self.delta.write_from_fileobj(Rdiff.get_delta_sigrp(self.signature, 51 | self.new)) 52 | assert self.delta.lstat() 53 | Rdiff.patch_local(self.basis, self.delta, self.output) 54 | assert rpath.cmp(self.new, self.output) 55 | map(rpath.RPath.delete, rplist) 56 | 57 | def testRdiffDeltaPatchGzip(self): 58 | """Same as above by try gzipping patches""" 59 | rplist = [self.basis, self.new, self.delta, 60 | self.signature, self.output] 61 | for rp in rplist: 62 | if rp.lstat(): rp.delete() 63 | 64 | MakeRandomFile(self.basis.path) 65 | MakeRandomFile(self.new.path) 66 | map(rpath.RPath.setdata, [self.basis, self.new]) 67 | assert self.basis.lstat() and self.new.lstat() 68 | self.signature.write_from_fileobj(Rdiff.get_signature(self.basis)) 69 | assert self.signature.lstat() 70 | self.delta.write_from_fileobj(Rdiff.get_delta_sigrp(self.signature, 71 | self.new)) 72 | assert self.delta.lstat() 73 | os.system("gzip " + self.delta.path) 74 | os.system("mv %s %s" % (self.delta.path + ".gz", self.delta.path)) 75 | self.delta.setdata() 76 | 77 | Rdiff.patch_local(self.basis, self.delta, self.output, 78 | delta_compressed = 1) 79 | assert rpath.cmp(self.new, self.output) 80 | map(rpath.RPath.delete, rplist) 81 | 82 | def testWriteDelta(self): 83 | """Test write delta feature of rdiff""" 84 | if self.delta.lstat(): self.delta.delete() 85 | rplist = [self.basis, self.new, self.delta, self.output] 86 | MakeRandomFile(self.basis.path) 87 | MakeRandomFile(self.new.path) 88 | map(rpath.RPath.setdata, [self.basis, self.new]) 89 | assert self.basis.lstat() and self.new.lstat() 90 | 91 | Rdiff.write_delta(self.basis, self.new, self.delta) 92 | assert self.delta.lstat() 93 | Rdiff.patch_local(self.basis, self.delta, self.output) 94 | assert rpath.cmp(self.new, self.output) 95 | map(rpath.RPath.delete, rplist) 96 | 97 | def testWriteDeltaGzip(self): 98 | """Same as above but delta is written gzipped""" 99 | rplist = [self.basis, self.new, self.delta, self.output] 100 | MakeRandomFile(self.basis.path) 101 | MakeRandomFile(self.new.path) 102 | map(rpath.RPath.setdata, [self.basis, self.new]) 103 | assert self.basis.lstat() and self.new.lstat() 104 | delta_gz = rpath.RPath(self.delta.conn, self.delta.path + ".gz") 105 | if delta_gz.lstat(): delta_gz.delete() 106 | 107 | Rdiff.write_delta(self.basis, self.new, delta_gz, 1) 108 | assert delta_gz.lstat() 109 | os.system("gunzip " + delta_gz.path) 110 | delta_gz.setdata() 111 | self.delta.setdata() 112 | Rdiff.patch_local(self.basis, self.delta, self.output) 113 | assert rpath.cmp(self.new, self.output) 114 | map(rpath.RPath.delete, rplist) 115 | 116 | def testRdiffRename(self): 117 | """Rdiff replacing original file with patch outfile""" 118 | rplist = [self.basis, self.new, self.delta, self.signature] 119 | for rp in rplist: 120 | if rp.lstat(): rp.delete() 121 | 122 | MakeRandomFile(self.basis.path) 123 | MakeRandomFile(self.new.path) 124 | map(rpath.RPath.setdata, [self.basis, self.new]) 125 | assert self.basis.lstat() and self.new.lstat() 126 | self.signature.write_from_fileobj(Rdiff.get_signature(self.basis)) 127 | assert self.signature.lstat() 128 | self.delta.write_from_fileobj(Rdiff.get_delta_sigrp(self.signature, 129 | self.new)) 130 | assert self.delta.lstat() 131 | Rdiff.patch_local(self.basis, self.delta) 132 | assert rpath.cmp(self.basis, self.new) 133 | map(rpath.RPath.delete, rplist) 134 | 135 | def testCopy(self): 136 | """Using rdiff to copy two files""" 137 | rplist = [self.basis, self.new] 138 | for rp in rplist: 139 | if rp.lstat(): rp.delete() 140 | 141 | MakeRandomFile(self.basis.path) 142 | MakeRandomFile(self.new.path) 143 | map(rpath.RPath.setdata, rplist) 144 | Rdiff.copy_local(self.basis, self.new) 145 | assert rpath.cmp(self.basis, self.new) 146 | map(rpath.RPath.delete, rplist) 147 | 148 | 149 | if __name__ == '__main__': 150 | unittest.main() 151 | -------------------------------------------------------------------------------- /rdiff-backup/testing/regresstest.py: -------------------------------------------------------------------------------- 1 | """regresstest - test the regress module. 2 | 3 | Not to be confused with the regression tests. 4 | 5 | """ 6 | 7 | import unittest 8 | from commontest import * 9 | from rdiff_backup import regress, Time 10 | 11 | Log.setverbosity(3) 12 | 13 | class RegressTest(unittest.TestCase): 14 | output_rp = rpath.RPath(Globals.local_connection, "testfiles/output") 15 | output_rbdir_rp = output_rp.append_path("rdiff-backup-data") 16 | inc1_rp = rpath.RPath(Globals.local_connection, "testfiles/increment1") 17 | inc2_rp = rpath.RPath(Globals.local_connection, "testfiles/increment2") 18 | inc3_rp = rpath.RPath(Globals.local_connection, "testfiles/increment3") 19 | inc4_rp = rpath.RPath(Globals.local_connection, "testfiles/increment4") 20 | 21 | def runtest(self, regress_function): 22 | """Test regressing a full directory to older state 23 | 24 | Make two directories, one with one more backup in it. Then 25 | regress the bigger one, and then make sure they compare the 26 | same. 27 | 28 | Regress_function takes a time and should regress 29 | self.output_rp back to that time. 30 | 31 | """ 32 | self.output_rp.setdata() 33 | if self.output_rp.lstat(): Myrm(self.output_rp.path) 34 | 35 | rdiff_backup(1, 1, self.inc1_rp.path, self.output_rp.path, 36 | current_time = 10000) 37 | assert CompareRecursive(self.inc1_rp, self.output_rp) 38 | 39 | rdiff_backup(1, 1, self.inc2_rp.path, self.output_rp.path, 40 | current_time = 20000) 41 | assert CompareRecursive(self.inc2_rp, self.output_rp) 42 | 43 | rdiff_backup(1, 1, self.inc3_rp.path, self.output_rp.path, 44 | current_time = 30000) 45 | assert CompareRecursive(self.inc3_rp, self.output_rp) 46 | 47 | rdiff_backup(1, 1, self.inc4_rp.path, self.output_rp.path, 48 | current_time = 40000) 49 | assert CompareRecursive(self.inc4_rp, self.output_rp) 50 | 51 | Globals.rbdir = self.output_rbdir_rp 52 | 53 | regress_function(30000) 54 | assert CompareRecursive(self.inc3_rp, self.output_rp, 55 | compare_hardlinks = 0) 56 | regress_function(20000) 57 | assert CompareRecursive(self.inc2_rp, self.output_rp, 58 | compare_hardlinks = 0) 59 | regress_function(10000) 60 | assert CompareRecursive(self.inc1_rp, self.output_rp, 61 | compare_hardlinks = 0) 62 | 63 | def regress_to_time_local(self, time): 64 | """Regress self.output_rp to time by running regress locally""" 65 | self.output_rp.setdata() 66 | self.output_rbdir_rp.setdata() 67 | self.add_current_mirror(time) 68 | regress.Regress(self.output_rp) 69 | 70 | def add_current_mirror(self, time): 71 | """Add current_mirror marker at given time""" 72 | cur_mirror_rp = self.output_rbdir_rp.append( 73 | "current_mirror.%s.data" % (Time.timetostring(time),)) 74 | cur_mirror_rp.touch() 75 | 76 | def regress_to_time_remote(self, time): 77 | """Like test_full above, but run regress remotely""" 78 | self.output_rp.setdata() 79 | self.output_rbdir_rp.setdata() 80 | self.add_current_mirror(time) 81 | cmdline = (SourceDir + 82 | "/../rdiff-backup -v3 --check-destination-dir " 83 | "--remote-schema './chdir-wrapper2 %s' " 84 | "test1::../" + self.output_rp.path) 85 | print "Running:", cmdline 86 | assert not os.system(cmdline) 87 | 88 | def test_local(self): 89 | """Run regress test locally""" 90 | self.runtest(self.regress_to_time_local) 91 | 92 | def test_remote(self): 93 | """Run regress test remotely""" 94 | self.runtest(self.regress_to_time_remote) 95 | 96 | def test_unreadable(self): 97 | """Run regress test when regular file is unreadable""" 98 | self.output_rp.setdata() 99 | if self.output_rp.lstat(): Myrm(self.output_rp.path) 100 | unreadable_rp = self.make_unreadable() 101 | 102 | rdiff_backup(1, 1, unreadable_rp.path, self.output_rp.path, 103 | current_time = 1) 104 | rbdir = self.output_rp.append('rdiff-backup-data') 105 | marker = rbdir.append('current_mirror.2000-12-31T21:33:20-07:00.data') 106 | marker.touch() 107 | self.change_unreadable() 108 | 109 | cmd = "rdiff-backup --check-destination-dir " + self.output_rp.path 110 | print "Executing:", cmd 111 | assert not os.system(cmd) 112 | 113 | def make_unreadable(self): 114 | """Make unreadable input directory 115 | 116 | The directory needs to be readable initially (otherwise it 117 | just won't get backed up, and then later we will turn it 118 | unreadable. 119 | 120 | """ 121 | rp = rpath.RPath(Globals.local_connection, "testfiles/regress") 122 | if rp.lstat(): Myrm(rp.path) 123 | rp.setdata() 124 | rp.mkdir() 125 | rp1 = rp.append('unreadable_dir') 126 | rp1.mkdir() 127 | rp1_1 = rp1.append('to_be_unreadable') 128 | rp1_1.write_string('aensuthaoeustnahoeu') 129 | return rp 130 | 131 | def change_unreadable(self): 132 | """Change attributes in directory, so regress will request fp""" 133 | subdir = self.output_rp.append('unreadable_dir') 134 | assert subdir.lstat() 135 | rp1_1 = subdir.append('to_be_unreadable') 136 | rp1_1.chmod(0) 137 | subdir.chmod(0) 138 | 139 | 140 | if __name__ == "__main__": unittest.main() 141 | 142 | -------------------------------------------------------------------------------- /rdiff-backup/testing/resourceforktest.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | from commontest import * 3 | from rdiff_backup import rpath 4 | from rdiff_backup import metadata 5 | 6 | """***NOTE*** 7 | 8 | None of these tests should work unless your system supports resource 9 | forks. So basically these tests should only be run on Mac OS X. 10 | 11 | """ 12 | 13 | Globals.read_resource_forks = Globals.write_resource_forks = 1 14 | 15 | class ResourceForkTest(unittest.TestCase): 16 | """Test dealing with Mac OS X style resource forks""" 17 | tempdir = rpath.RPath(Globals.local_connection, 'testfiles/output') 18 | rf_testdir1 = rpath.RPath(Globals.local_connection, 19 | 'testfiles/resource_fork_test1') 20 | rf_testdir2 = rpath.RPath(Globals.local_connection, 21 | 'testfiles/resource_fork_test2') 22 | def make_temp(self): 23 | """Make temp directory testfiles/resource_fork_test""" 24 | if self.tempdir.lstat(): self.tempdir.delete() 25 | self.tempdir.mkdir() 26 | 27 | def testBasic(self): 28 | """Test basic reading and writing of resource forks""" 29 | self.make_temp() 30 | rp = self.tempdir.append('test') 31 | rp.touch() 32 | assert rp.get_resource_fork() == '', rp.get_resource_fork() 33 | 34 | s = 'new resource fork data' 35 | rp.write_resource_fork(s) 36 | assert rp.get_resource_fork() == s, rp.get_resource_fork() 37 | 38 | rp2 = self.tempdir.append('test') 39 | assert rp2.isreg() 40 | assert rp2.get_resource_fork() == s, rp2.get_resource_fork() 41 | 42 | def testRecord(self): 43 | """Test reading, writing, and comparing of records with rforks""" 44 | self.make_temp() 45 | rp = self.tempdir.append('test') 46 | rp.touch() 47 | rp.set_resource_fork('hello') 48 | 49 | record = metadata.RORP2Record(rp) 50 | #print record 51 | rorp_out = metadata.Record2RORP(record) 52 | assert rorp_out == rp, (rorp_out, rp) 53 | assert rorp_out.get_resource_fork() == 'hello' 54 | 55 | def make_backup_dirs(self): 56 | """Create testfiles/resource_fork_test[12] dirs for testing""" 57 | if self.rf_testdir1.lstat(): self.rf_testdir1.delete() 58 | if self.rf_testdir2.lstat(): self.rf_testdir2.delete() 59 | self.rf_testdir1.mkdir() 60 | rp1_1 = self.rf_testdir1.append('1') 61 | rp1_2 = self.rf_testdir1.append('2') 62 | rp1_3 = self.rf_testdir1.append('3') 63 | rp1_1.touch() 64 | rp1_2.touch() 65 | rp1_3.symlink('foo') 66 | rp1_1.write_resource_fork('This should appear in resource fork') 67 | rp1_1.chmod(0400) # test for bug changing resource forks after perms 68 | rp1_2.write_resource_fork('Data for the resource fork 2') 69 | 70 | 71 | self.rf_testdir2.mkdir() 72 | rp2_1 = self.rf_testdir2.append('1') 73 | rp2_2 = self.rf_testdir2.append('2') 74 | rp2_3 = self.rf_testdir2.append('3') 75 | rp2_1.touch() 76 | rp2_2.touch() 77 | rp2_3.touch() 78 | rp2_1.write_resource_fork('New data for resource fork 1') 79 | rp2_1.chmod(0400) 80 | rp2_3.write_resource_fork('New fork') 81 | 82 | def testSeriesLocal(self): 83 | """Test backing up and restoring directories with ACLs locally""" 84 | Globals.read_resource_forks = Globals.write_resource_forks = 1 85 | self.make_backup_dirs() 86 | dirlist = ['testfiles/resource_fork_test1', 'testfiles/empty', 87 | 'testfiles/resource_fork_test2', 88 | 'testfiles/resource_fork_test1'] 89 | # BackupRestoreSeries(1, 1, dirlist, compare_resource_forks = 1) 90 | BackupRestoreSeries(1, 1, dirlist) 91 | 92 | def testSeriesRemote(self): 93 | """Test backing up and restoring directories with ACLs locally""" 94 | Globals.read_resource_forks = Globals.write_resource_forks = 1 95 | self.make_backup_dirs() 96 | dirlist = ['testfiles/resource_fork_test1', 97 | 'testfiles/resource_fork_test2', 'testfiles/empty', 98 | 'testfiles/resource_fork_test1'] 99 | # BackupRestoreSeries(1, 1, dirlist, compare_resource_forks = 1) 100 | BackupRestoreSeries(1, 1, dirlist) 101 | 102 | 103 | if __name__ == "__main__": unittest.main() 104 | 105 | -------------------------------------------------------------------------------- /rdiff-backup/testing/restoretest.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | from commontest import * 3 | from rdiff_backup import log, restore, Globals, rpath, TempFile 4 | 5 | Log.setverbosity(3) 6 | lc = Globals.local_connection 7 | tempdir = rpath.RPath(Globals.local_connection, "testfiles/output") 8 | restore_base_rp = rpath.RPath(Globals.local_connection, 9 | "testfiles/restoretest") 10 | restore_base_filenames = restore_base_rp.listdir() 11 | mirror_time = 1041109438 # just some late time 12 | 13 | class RestoreFileComparer: 14 | """Holds a file to be restored and tests against it 15 | 16 | Each object has a restore file and a dictionary of times -> 17 | rpaths. When the restore file is restored to one of the given 18 | times, the resulting file should be the same as the related rpath. 19 | 20 | """ 21 | def __init__(self, rf): 22 | self.rf = rf 23 | self.time_rp_dict = {} 24 | 25 | def add_rpath(self, rp, t): 26 | """Add rp, which represents what rf should be at given time t""" 27 | assert not self.time_rp_dict.has_key(t) 28 | self.time_rp_dict[t] = rp 29 | 30 | def compare_at_time(self, t): 31 | """Restore file, make sure it is the same at time t""" 32 | log.Log("Checking result at time %s" % (t,), 7) 33 | tf = TempFile.new(tempdir.append("foo")) 34 | restore.MirrorStruct._mirror_time = mirror_time 35 | restore.MirrorStruct._rest_time = t 36 | self.rf.set_relevant_incs() 37 | out_rorpath = self.rf.get_attribs().getRORPath() 38 | correct_result = self.time_rp_dict[t] 39 | 40 | if out_rorpath.isreg(): 41 | out_rorpath.setfile(self.rf.get_restore_fp()) 42 | rpath.copy_with_attribs(out_rorpath, tf) 43 | assert tf.equal_verbose(correct_result, check_index = 0), \ 44 | "%s, %s" % (tf, correct_result) 45 | if tf.isreg(): 46 | assert rpath.cmpfileobj(tf.open("rb"), correct_result.open("rb")) 47 | if tf.lstat(): tf.delete() 48 | 49 | def compare_all(self): 50 | """Check restore results for all available times""" 51 | for t in self.time_rp_dict.keys(): self.compare_at_time(t) 52 | 53 | 54 | class RestoreTimeTest(unittest.TestCase): 55 | def test_time_from_session(self): 56 | """Test getting time from session number (as in Time.time_from_session) 57 | 58 | Test here instead of in timetest because it depends on an 59 | rdiff-backup-data directory already being laid out. 60 | 61 | """ 62 | restore.MirrorStruct._mirror_time = None # Reset 63 | Globals.rbdir = rpath.RPath(lc, 64 | "testfiles/restoretest3/rdiff-backup-data") 65 | assert Time.genstrtotime("0B") == Time.time_from_session(0) 66 | assert Time.genstrtotime("2B") == Time.time_from_session(2) 67 | assert Time.genstrtotime("23B") == Time.time_from_session(23) 68 | 69 | assert Time.time_from_session(0) == 40000, Time.time_from_session(0) 70 | assert Time.time_from_session(2) == 20000, Time.time_from_session(2) 71 | assert Time.time_from_session(5) == 10000, Time.time_from_session(5) 72 | 73 | 74 | class RestoreTest(unittest.TestCase): 75 | """Test Restore class""" 76 | def get_rfcs(self): 77 | """Return available RestoreFileCompararer objects""" 78 | base_rf = restore.RestoreFile(restore_base_rp, restore_base_rp, []) 79 | rfs = base_rf.yield_sub_rfs() 80 | rfcs = [] 81 | for rf in rfs: 82 | if rf.mirror_rp.dirsplit()[1] in ["dir"]: 83 | log.Log("skipping 'dir'", 5) 84 | continue 85 | 86 | rfc = RestoreFileComparer(rf) 87 | for inc in rf.inc_list: 88 | test_time = inc.getinctime() 89 | rfc.add_rpath(self.get_correct(rf.mirror_rp, test_time), 90 | test_time) 91 | rfc.add_rpath(rf.mirror_rp, mirror_time) 92 | rfcs.append(rfc) 93 | return rfcs 94 | 95 | def get_correct(self, mirror_rp, test_time): 96 | """Return correct version with base mirror_rp at time test_time""" 97 | assert -1 < test_time < 2000000000, test_time 98 | dirname, basename = mirror_rp.dirsplit() 99 | for filename in restore_base_filenames: 100 | comps = filename.split(".") 101 | base = ".".join(comps[:-1]) 102 | t = Time.stringtotime(comps[-1]) 103 | if t == test_time and basename == base: 104 | return restore_base_rp.append(filename) 105 | # Correct rp must be empty 106 | return restore_base_rp.append("%s.%s" % 107 | (basename, Time.timetostring(test_time))) 108 | 109 | def testRestoreSingle(self): 110 | """Test restoring files one at at a time""" 111 | MakeOutputDir() 112 | for rfc in self.get_rfcs(): 113 | if rfc.rf.inc_rp.isincfile(): continue 114 | log.Log("Comparing %s" % (rfc.rf.inc_rp.path,), 5) 115 | rfc.compare_all() 116 | 117 | def testBothLocal(self): 118 | """Test directory restore everything local""" 119 | self.restore_dir_test(1,1) 120 | 121 | def testMirrorRemote(self): 122 | """Test directory restore mirror is remote""" 123 | self.restore_dir_test(0, 1) 124 | 125 | def testDestRemote(self): 126 | """Test directory restore destination is remote""" 127 | self.restore_dir_test(1, 0) 128 | 129 | def testBothRemote(self): 130 | """Test directory restore everything is remote""" 131 | self.restore_dir_test(0, 0) 132 | 133 | def restore_dir_test(self, mirror_local, dest_local): 134 | """Run whole dir tests 135 | 136 | If any of the above tests don't work, try rerunning 137 | makerestoretest3. 138 | 139 | """ 140 | Myrm("testfiles/output") 141 | target_rp = rpath.RPath(Globals.local_connection, "testfiles/output") 142 | mirror_rp = rpath.RPath(Globals.local_connection, 143 | "testfiles/restoretest3") 144 | inc1_rp = rpath.RPath(Globals.local_connection, 145 | "testfiles/increment1") 146 | inc2_rp = rpath.RPath(Globals.local_connection, 147 | "testfiles/increment2") 148 | inc3_rp = rpath.RPath(Globals.local_connection, 149 | "testfiles/increment3") 150 | inc4_rp = rpath.RPath(Globals.local_connection, 151 | "testfiles/increment4") 152 | 153 | InternalRestore(mirror_local, dest_local, "testfiles/restoretest3", 154 | "testfiles/output", 45000) 155 | assert CompareRecursive(inc4_rp, target_rp) 156 | InternalRestore(mirror_local, dest_local, "testfiles/restoretest3", 157 | "testfiles/output", 35000) 158 | assert CompareRecursive(inc3_rp, target_rp, compare_hardlinks = 0) 159 | InternalRestore(mirror_local, dest_local, "testfiles/restoretest3", 160 | "testfiles/output", 25000) 161 | assert CompareRecursive(inc2_rp, target_rp, compare_hardlinks = 0) 162 | InternalRestore(mirror_local, dest_local, "testfiles/restoretest3", 163 | "testfiles/output", 5000) 164 | assert CompareRecursive(inc1_rp, target_rp, compare_hardlinks = 0) 165 | 166 | def testRestoreNoincs(self): 167 | """Test restoring a directory with no increments, just mirror""" 168 | Myrm("testfiles/output") 169 | InternalRestore(1, 1, 'testfiles/restoretest5/regular_file', 'testfiles/output', 170 | 10000) 171 | assert os.lstat("testfiles/output") 172 | 173 | if __name__ == "__main__": unittest.main() 174 | -------------------------------------------------------------------------------- /rdiff-backup/testing/rlisttest.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | from commontest import * 3 | from rdiff_backup.rlist import * 4 | 5 | class BasicObject: 6 | """The simplest object that can be used with RList""" 7 | def __init__(self, i): 8 | self.index = i 9 | self.data = "This is object # %d" % i 10 | 11 | def __eq__(self, other): 12 | return self.index == other.index and self.data == other.data 13 | 14 | l1_pre = filter(lambda x: x != 342 and not x in [650, 651, 652] and 15 | x != 911 and x != 987, 16 | range(1, 1001)) 17 | l2_pre = filter(lambda x: not x in [222, 223, 224, 225] and x != 950 18 | and x != 999 and x != 444, 19 | range(1, 1001)) 20 | 21 | l1 = map(BasicObject, l1_pre) 22 | l2 = map(BasicObject, l2_pre) 23 | combined = map(BasicObject, range(1, 1001)) 24 | 25 | def lmaphelper2((x, i)): 26 | """Return difference triple to say that index x only in list # i""" 27 | if i == 1: return (BasicObject(x), None) 28 | elif i == 2: return (None, BasicObject(x)) 29 | else: assert 0, "Invalid parameter %s for i" % i 30 | 31 | difference1 = map(lmaphelper2, [(222, 1), (223, 1), (224, 1), (225, 1), 32 | (342, 2), (444, 1), (650, 2), (651, 2), 33 | (652, 2), (911, 2), (950, 1), (987, 2), 34 | (999, 1)]) 35 | difference2 = map(lambda (a, b): (b, a), difference1) 36 | 37 | def comparelists(l1, l2): 38 | print len(l1), len(l2) 39 | for i in range(len(l1)): 40 | if l1[i] != l2[i]: print l1[i], l2[i] 41 | print l1 42 | print l2 43 | 44 | 45 | 46 | class RListTest(unittest.TestCase): 47 | def setUp(self): 48 | """Make signatures, deltas""" 49 | self.l1_sig = RList.Signatures(l1) 50 | self.l2_sig = RList.Signatures(l2) 51 | self.l1_to_l2_diff = RList.Deltas(self.l1_sig, l2) 52 | self.l2_to_l1_diff = RList.Deltas(self.l2_sig, l1) 53 | 54 | # for d in makedeltas(makesigs(l2ci(l1)), l2ci(l2)): 55 | # print d.min, d.max 56 | # print d.elemlist 57 | 58 | def testPatching(self): 59 | """Test to make sure each list can be reconstructed from other""" 60 | newlist = list(RList.Patch(l1, RList.Deltas(RList.Signatures(l1), 61 | l2))) 62 | assert l2 == newlist 63 | newlist = list(RList.Patch(l2, RList.Deltas(RList.Signatures(l2), 64 | l1))) 65 | assert l1 == newlist 66 | 67 | def testDifference(self): 68 | """Difference between lists correctly identified""" 69 | diff = list(RList.Dissimilar(l1, RList.Deltas(RList.Signatures(l1), 70 | l2))) 71 | assert diff == difference1 72 | diff = list(RList.Dissimilar(l2, RList.Deltas(RList.Signatures(l2), 73 | l1))) 74 | assert diff == difference2 75 | 76 | 77 | 78 | class CachingIterTest(unittest.TestCase): 79 | """Test the Caching Iter object""" 80 | def testNormalIter(self): 81 | """Make sure it can act like a normal iterator""" 82 | ci = CachingIter(iter(range(10))) 83 | for i in range(10): assert i == ci.next() 84 | self.assertRaises(StopIteration, ci.next) 85 | 86 | def testPushing(self): 87 | """Pushing extra objects onto the iterator""" 88 | ci = CachingIter(iter(range(10))) 89 | ci.push(12) 90 | ci.push(11) 91 | assert ci.next() == 11 92 | assert ci.next() == 12 93 | assert ci.next() == 0 94 | ci.push(10) 95 | assert ci.next() == 10 96 | 97 | 98 | if __name__ == "__main__": unittest.main() 99 | -------------------------------------------------------------------------------- /rdiff-backup/testing/robusttest.py: -------------------------------------------------------------------------------- 1 | 2 | import os, unittest 3 | from commontest import * 4 | from rdiff_backup import rpath, robust, TempFile, Globals 5 | 6 | 7 | class RobustTest(unittest.TestCase): 8 | """Test robust module""" 9 | def test_check_common_error(self): 10 | """Test capturing errors""" 11 | def cause_catchable_error(a): 12 | os.lstat("aoenuthaoeu/aosutnhcg.4fpr,38p") 13 | def cause_uncatchable_error(): 14 | ansoethusaotneuhsaotneuhsaontehuaou 15 | result = robust.check_common_error(None, cause_catchable_error, [1]) 16 | assert result is None, result 17 | try: robust.check_common_error(None, cause_uncatchable_error) 18 | except NameError: pass 19 | else: assert 0, "Key error not raised" 20 | 21 | 22 | if __name__ == '__main__': unittest.main() 23 | -------------------------------------------------------------------------------- /rdiff-backup/testing/server.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import sys, os 4 | 5 | __doc__ = """ 6 | 7 | This starts an rdiff-backup server using the existing source files. 8 | If not run from the source directory, the only argument should be 9 | the directory the source files are in. 10 | """ 11 | 12 | def Test_SetConnGlobals(conn, setting, value): 13 | """This is used in connectiontest.py""" 14 | conn.Globals.set(setting, value) 15 | 16 | def print_usage(): 17 | print "Usage: server.py [path to source files]", __doc__ 18 | 19 | if len(sys.argv) > 2: 20 | print_usage() 21 | sys.exit(1) 22 | 23 | try: 24 | if len(sys.argv) == 2: sys.path.insert(0, sys.argv[1]) 25 | import rdiff_backup.Globals 26 | import rdiff_backup.Security 27 | from rdiff_backup.connection import * 28 | except (OSError, IOError, ImportError): 29 | print_usage() 30 | raise 31 | 32 | #log.Log.setverbosity(5) 33 | rdiff_backup.Globals.security_level = "override" 34 | PipeConnection(sys.stdin, sys.stdout).Server() 35 | -------------------------------------------------------------------------------- /rdiff-backup/testing/setconnectionstest.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | from commontest import * 3 | from rdiff_backup import SetConnections 4 | 5 | class SetConnectionsTest(unittest.TestCase): 6 | """Set SetConnections Class""" 7 | def testParsing(self): 8 | """Test parsing of various file descriptors""" 9 | pfd = SetConnections.parse_file_desc 10 | assert pfd("bescoto@folly.stanford.edu::/usr/bin/ls") == \ 11 | ("bescoto@folly.stanford.edu", "/usr/bin/ls") 12 | assert pfd("hello there::/goodbye:euoeu") == \ 13 | ("hello there", "/goodbye:euoeu") 14 | assert pfd(r"test\\ing\::more::and more\\..") == \ 15 | (r"test\ing::more", r"and more\\.."), \ 16 | pfd(r"test\\ing\::more::and more\\..") 17 | assert pfd("a:b:c:d::e") == ("a:b:c:d", "e") 18 | assert pfd("foobar") == (None, "foobar") 19 | assert pfd(r"hello\::there") == (None, "hello\::there") 20 | 21 | self.assertRaises(SetConnections.SetConnectionsException, 22 | pfd, r"hello\:there::") 23 | self.assertRaises(SetConnections.SetConnectionsException, 24 | pfd, "foobar\\") 25 | 26 | 27 | if __name__ == "__main__": unittest.main() 28 | -------------------------------------------------------------------------------- /rdiff-backup/testing/statictest.py: -------------------------------------------------------------------------------- 1 | import unittest, types 2 | from commontest import * 3 | from rdiff_backup.static import * 4 | 5 | 6 | class D: 7 | def foo(x, y): 8 | return x, y 9 | def bar(self, x): 10 | return 3, x 11 | def _hello(self): 12 | return self 13 | 14 | MakeStatic(D) 15 | 16 | 17 | class C: 18 | _a = 0 19 | def get(cls): 20 | return cls._a 21 | def inc(cls): 22 | cls._a = cls._a + 1 23 | 24 | MakeClass(C) 25 | 26 | 27 | class StaticMethodsTest(unittest.TestCase): 28 | """Test StaticMethods module""" 29 | def testType(self): 30 | """Methods should have type StaticMethod""" 31 | assert type(D.foo) is types.FunctionType 32 | assert type(D.bar) is types.FunctionType 33 | 34 | def testStatic(self): 35 | """Methods should be callable without instance""" 36 | assert D.foo(1,2) == (1,2) 37 | assert D.bar(3,4) == (3,4) 38 | 39 | def testBound(self): 40 | """Methods should also work bound""" 41 | d = D() 42 | assert d.foo(1,2) == (1,2) 43 | assert d.bar(3,4) == (3,4) 44 | 45 | def testStatic_(self): 46 | """_ Methods should be untouched""" 47 | d = D() 48 | self.assertRaises(TypeError, d._hello, 4) 49 | assert d._hello() is d 50 | 51 | 52 | class ClassMethodsTest(unittest.TestCase): 53 | def test(self): 54 | """Test MakeClass function""" 55 | assert C.get() == 0 56 | C.inc() 57 | assert C.get() == 1 58 | C.inc() 59 | assert C.get() == 2 60 | 61 | 62 | if __name__ == "__main__": 63 | unittest.main() 64 | -------------------------------------------------------------------------------- /rdiff-backup/testing/statisticstest.py: -------------------------------------------------------------------------------- 1 | import unittest, time 2 | from commontest import * 3 | from rdiff_backup import statistics, rpath, restore 4 | 5 | class StatsObjTest(unittest.TestCase): 6 | """Test StatsObj class""" 7 | def set_obj(self, s): 8 | """Set values of s's statistics""" 9 | s.SourceFiles = 1 10 | s.SourceFileSize = 2 11 | s.MirrorFiles = 13 12 | s.MirrorFileSize = 14 13 | s.NewFiles = 3 14 | s.NewFileSize = 4 15 | s.DeletedFiles = 5 16 | s.DeletedFileSize = 6 17 | s.ChangedFiles = 7 18 | s.ChangedSourceSize = 8 19 | s.ChangedMirrorSize = 9 20 | s.IncrementFiles = 15 21 | s.IncrementFileSize = 10 22 | s.StartTime = 11 23 | s.EndTime = 12 24 | 25 | def test_get_stats(self): 26 | """Test reading and writing stat objects""" 27 | s = statistics.StatsObj() 28 | assert s.get_stat('SourceFiles') is None 29 | self.set_obj(s) 30 | assert s.get_stat('SourceFiles') == 1 31 | 32 | s1 = statistics.StatFileObj() 33 | assert s1.get_stat('SourceFiles') == 0 34 | 35 | def test_get_stats_string(self): 36 | """Test conversion of stat object into string""" 37 | s = statistics.StatsObj() 38 | stats_string = s.get_stats_string() 39 | assert stats_string == "", stats_string 40 | 41 | self.set_obj(s) 42 | stats_string = s.get_stats_string() 43 | ss_list = stats_string.split("\n") 44 | tail = "\n".join(ss_list[2:]) # Time varies by time zone, don't check 45 | #"""StartTime 11.00 (Wed Dec 31 16:00:11 1969) 46 | #EndTime 12.00 (Wed Dec 31 16:00:12 1969)" 47 | assert tail == \ 48 | """ElapsedTime 1.00 (1 second) 49 | SourceFiles 1 50 | SourceFileSize 2 (2 bytes) 51 | MirrorFiles 13 52 | MirrorFileSize 14 (14 bytes) 53 | NewFiles 3 54 | NewFileSize 4 (4 bytes) 55 | DeletedFiles 5 56 | DeletedFileSize 6 (6 bytes) 57 | ChangedFiles 7 58 | ChangedSourceSize 8 (8 bytes) 59 | ChangedMirrorSize 9 (9 bytes) 60 | IncrementFiles 15 61 | IncrementFileSize 10 (10 bytes) 62 | TotalDestinationSizeChange 7 (7 bytes) 63 | """, "'%s'" % stats_string 64 | 65 | def test_line_string(self): 66 | """Test conversion to a single line""" 67 | s = statistics.StatsObj() 68 | self.set_obj(s) 69 | statline = s.get_stats_line(("sample", "index", "w", "new\nline")) 70 | assert statline == "sample/index/w/new\\nline 1 2 13 14 " \ 71 | "3 4 5 6 7 8 9 15 10", repr(statline) 72 | 73 | statline = s.get_stats_line(()) 74 | assert statline == ". 1 2 13 14 3 4 5 6 7 8 9 15 10" 75 | 76 | statline = s.get_stats_line(("file name with spaces",)) 77 | assert statline == "file\\x20name\\x20with\\x20spaces 1 2 13 14 " \ 78 | "3 4 5 6 7 8 9 15 10", repr(statline) 79 | 80 | def test_byte_summary(self): 81 | """Test conversion of bytes to strings like 7.23MB""" 82 | s = statistics.StatsObj() 83 | f = s.get_byte_summary_string 84 | assert f(1) == "1 byte" 85 | assert f(234.34) == "234 bytes" 86 | assert f(2048) == "2.00 KB" 87 | assert f(3502243) == "3.34 MB" 88 | assert f(314992230) == "300 MB" 89 | assert f(36874871216) == "34.3 GB", f(36874871216) 90 | assert f(3775986812573450) == "3434 TB" 91 | 92 | def test_init_stats(self): 93 | """Test setting stat object from string""" 94 | s = statistics.StatsObj() 95 | s.set_stats_from_string("NewFiles 3 hello there") 96 | for attr in s.stat_attrs: 97 | if attr == 'NewFiles': assert s.get_stat(attr) == 3 98 | else: assert s.get_stat(attr) is None, (attr, s.__dict__[attr]) 99 | 100 | s1 = statistics.StatsObj() 101 | self.set_obj(s1) 102 | assert not s1.stats_equal(s) 103 | 104 | s2 = statistics.StatsObj() 105 | s2.set_stats_from_string(s1.get_stats_string()) 106 | assert s1.stats_equal(s2) 107 | 108 | def test_write_rp(self): 109 | """Test reading and writing of statistics object""" 110 | rp = rpath.RPath(Globals.local_connection, "testfiles/statstest") 111 | if rp.lstat(): rp.delete() 112 | s = statistics.StatsObj() 113 | self.set_obj(s) 114 | s.write_stats_to_rp(rp) 115 | 116 | s2 = statistics.StatsObj() 117 | assert not s2.stats_equal(s) 118 | s2.read_stats_from_rp(rp) 119 | assert s2.stats_equal(s) 120 | 121 | def testAverage(self): 122 | """Test making an average statsobj""" 123 | s1 = statistics.StatsObj() 124 | s1.StartTime = 5 125 | s1.EndTime = 10 126 | s1.ElapsedTime = 5 127 | s1.ChangedFiles = 2 128 | s1.SourceFiles = 100 129 | s1.NewFileSize = 4 130 | 131 | s2 = statistics.StatsObj() 132 | s2.StartTime = 25 133 | s2.EndTime = 35 134 | s2.ElapsedTime = 10 135 | s2.ChangedFiles = 1 136 | s2.SourceFiles = 50 137 | s2.DeletedFiles = 0 138 | 139 | s3 = statistics.StatsObj().set_to_average([s1, s2]) 140 | assert s3.StartTime is s3.EndTime is None 141 | assert s3.ElapsedTime == 7.5 142 | assert s3.DeletedFiles is s3.NewFileSize is None, (s3.DeletedFiles, 143 | s3.NewFileSize) 144 | assert s3.ChangedFiles == 1.5 145 | assert s3.SourceFiles == 75 146 | 147 | 148 | class IncStatTest(unittest.TestCase): 149 | """Test statistics as produced by actual backup""" 150 | def stats_check_initial(self, s): 151 | """Make sure stats object s compatible with initial mirroring 152 | 153 | A lot of the off by one stuff is because the root directory 154 | exists in the below examples. 155 | 156 | """ 157 | assert s.MirrorFiles == 1 or s.MirrorFiles == 0 158 | assert s.MirrorFileSize < 20000 159 | assert s.NewFiles <= s.SourceFiles <= s.NewFiles + 1 160 | assert s.NewFileSize <= s.SourceFileSize <= s.NewFileSize + 20000 161 | assert s.ChangedFiles == 1 or s.ChangedFiles == 0 162 | assert s.ChangedSourceSize < 20000 163 | assert s.ChangedMirrorSize < 20000 164 | assert s.DeletedFiles == s.DeletedFileSize == 0 165 | assert s.IncrementFileSize == 0 166 | 167 | def testStatistics(self): 168 | """Test the writing of statistics 169 | 170 | The file sizes are approximate because the size of directories 171 | could change with different file systems... 172 | 173 | """ 174 | def sorti(inclist): 175 | l = [(inc.getinctime(), inc) for inc in inclist] 176 | l.sort() 177 | return [inc for (t, inc) in l] 178 | 179 | Globals.compression = 1 180 | Myrm("testfiles/output") 181 | InternalBackup(1, 1, "testfiles/stattest1", "testfiles/output") 182 | InternalBackup(1, 1, "testfiles/stattest2", "testfiles/output", 183 | time.time()+1) 184 | 185 | rbdir = rpath.RPath(Globals.local_connection, 186 | "testfiles/output/rdiff-backup-data") 187 | 188 | incs = sorti(restore.get_inclist(rbdir.append("session_statistics"))) 189 | assert len(incs) == 2 190 | s2 = statistics.StatsObj().read_stats_from_rp(incs[0]) 191 | assert s2.SourceFiles == 7 192 | assert 700000 <= s2.SourceFileSize < 750000, s2.SourceFileSize 193 | self.stats_check_initial(s2) 194 | 195 | root_stats = statistics.StatsObj().read_stats_from_rp(incs[1]) 196 | assert root_stats.SourceFiles == 7, root_stats.SourceFiles 197 | assert 550000 <= root_stats.SourceFileSize < 570000 198 | assert root_stats.MirrorFiles == 7 199 | assert 700000 <= root_stats.MirrorFileSize < 750000 200 | assert root_stats.NewFiles == 1 201 | assert root_stats.NewFileSize == 0 202 | assert root_stats.DeletedFiles == 1, root_stats.DeletedFiles 203 | assert root_stats.DeletedFileSize == 200000 204 | assert 3 <= root_stats.ChangedFiles <= 4, root_stats.ChangedFiles 205 | assert 450000 <= root_stats.ChangedSourceSize < 470000 206 | assert 400000 <= root_stats.ChangedMirrorSize < 420000, \ 207 | root_stats.ChangedMirrorSize 208 | assert 10 < root_stats.IncrementFileSize < 30000 209 | 210 | if __name__ == "__main__": unittest.main() 211 | -------------------------------------------------------------------------------- /rdiff-backup/testing/test1/tmp/placeholder: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Seravo/rdiff-backup/8ccc5a3b44c996ecd810f8d5d586d0da6435cc32/rdiff-backup/testing/test1/tmp/placeholder -------------------------------------------------------------------------------- /rdiff-backup/testing/test2/tmp/foo: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Seravo/rdiff-backup/8ccc5a3b44c996ecd810f8d5d586d0da6435cc32/rdiff-backup/testing/test2/tmp/foo -------------------------------------------------------------------------------- /rdiff-backup/testing/test_with_profiling.py: -------------------------------------------------------------------------------- 1 | import profile, pstats 2 | from metadatatest import * 3 | 4 | profile.run("unittest.main()", "profile-output") 5 | p = pstats.Stats("profile-output") 6 | p.sort_stats('time') 7 | p.print_stats(40) 8 | -------------------------------------------------------------------------------- /rdiff-backup/testing/testall.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | 3 | """This probably doesn't work any more - just run the tests manually.""" 4 | 5 | from connectiontest import * 6 | #from destructive-steppingtest import * 7 | from dstest import * 8 | from highleveltest import * 9 | from incrementtest import * 10 | from iterfiletest import * 11 | from lazytest import * 12 | from rdifftest import * 13 | from regressiontest import * 14 | from restoretest import * 15 | from rlisttest import * 16 | from rorpitertest import * 17 | from rpathtest import * 18 | #from finaltest import * 19 | from statictest import * 20 | from timetest import * 21 | from filelisttest import * 22 | from setconnectionstest import * 23 | 24 | if __name__ == "__main__": 25 | unittest.main() 26 | 27 | -------------------------------------------------------------------------------- /rdiff-backup/testing/timetest.py: -------------------------------------------------------------------------------- 1 | import unittest, time, types 2 | from commontest import * 3 | from rdiff_backup import Globals, Time 4 | 5 | class TimeTest(unittest.TestCase): 6 | def testConversion(self): 7 | """test timetostring and stringtotime""" 8 | Time.setcurtime() 9 | assert type(Time.curtime) is types.FloatType or types.LongType 10 | assert type(Time.curtimestr) is types.StringType 11 | assert (Time.cmp(int(Time.curtime), Time.curtimestr) == 0 or 12 | Time.cmp(int(Time.curtime) + 1, Time.curtimestr) == 0) 13 | time.sleep(1.05) 14 | assert Time.cmp(time.time(), Time.curtime) == 1 15 | assert Time.cmp(Time.timetostring(time.time()), Time.curtimestr) == 1 16 | 17 | def testConversion_separator(self): 18 | """Same as testConversion, but change time Separator""" 19 | Globals.time_separator = "_" 20 | self.testConversion() 21 | Globals.time_separator = ":" 22 | 23 | def testCmp(self): 24 | """Test time comparisons""" 25 | cmp = Time.cmp 26 | assert cmp(1,2) == -1 27 | assert cmp(2,2) == 0 28 | assert cmp(5,1) == 1 29 | assert cmp("2001-09-01T21:49:04Z", "2001-08-01T21:49:04Z") == 1 30 | assert cmp("2001-09-01T04:49:04+03:23", "2001-09-01T21:49:04Z") == -1 31 | assert cmp("2001-09-01T12:00:00Z", "2001-09-01T04:00:00-08:00") == 0 32 | assert cmp("2001-09-01T12:00:00-08:00", 33 | "2001-09-01T12:00:00-07:00") == 1 34 | 35 | def testStringtotime(self): 36 | """Test converting string to time""" 37 | timesec = int(time.time()) 38 | assert timesec == int(Time.stringtotime(Time.timetostring(timesec))) 39 | assert not Time.stringtotime("2001-18-83T03:03:03Z") 40 | assert not Time.stringtotime("2001-01-23L03:03:03L") 41 | assert not Time.stringtotime("2001_01_23T03:03:03Z") 42 | 43 | def testIntervals(self): 44 | """Test converting strings to intervals""" 45 | i2s = Time.intstringtoseconds 46 | for s in ["32", "", "d", "231I", "MM", "s", "-2h"]: 47 | try: i2s(s) 48 | except Time.TimeException: pass 49 | else: assert 0, s 50 | assert i2s("7D") == 7*86400 51 | assert i2s("232s") == 232 52 | assert i2s("2M") == 2*30*86400 53 | assert i2s("400m") == 400*60 54 | assert i2s("1Y") == 365*86400 55 | assert i2s("30h") == 30*60*60 56 | assert i2s("3W") == 3*7*86400 57 | 58 | def testIntervalsComposite(self): 59 | """Like above, but allow composite intervals""" 60 | i2s = Time.intstringtoseconds 61 | assert i2s("7D2h") == 7*86400 + 2*3600 62 | assert i2s("2Y3s") == 2*365*86400 + 3 63 | assert i2s("1M2W4D2h5m20s") == (30*86400 + 2*7*86400 + 4*86400 + 64 | 2*3600 + 5*60 + 20) 65 | 66 | def testPrettyIntervals(self): 67 | """Test printable interval conversion""" 68 | assert Time.inttopretty(3600) == "1 hour" 69 | assert Time.inttopretty(7220) == "2 hours 20 seconds" 70 | assert Time.inttopretty(0) == "0 seconds" 71 | assert Time.inttopretty(353) == "5 minutes 53 seconds" 72 | assert Time.inttopretty(3661) == "1 hour 1 minute 1 second" 73 | assert Time.inttopretty(353.234234) == "5 minutes 53.23 seconds" 74 | 75 | def testPrettyTimes(self): 76 | """Convert seconds to pretty and back""" 77 | now = int(time.time()) 78 | for i in [1, 200000, now]: 79 | assert Time.prettytotime(Time.timetopretty(i)) == i, i 80 | assert Time.prettytotime("now") is None 81 | assert Time.prettytotime("12314") is None 82 | 83 | def testGenericString(self): 84 | """Test genstrtotime, conversion of arbitrary string to time""" 85 | g2t = Time.genstrtotime 86 | assert g2t('now', 1000) == 1000 87 | assert g2t('2h3s', 10000) == 10000 - 2*3600 - 3 88 | assert g2t('2001-09-01T21:49:04Z') == \ 89 | Time.stringtotime('2001-09-01T21:49:04Z') 90 | assert g2t('2002-04-26T04:22:01') == \ 91 | Time.stringtotime('2002-04-26T04:22:01' + Time.gettzd()) 92 | t = Time.stringtotime('2001-05-12T00:00:00' + Time.gettzd()) 93 | assert g2t('2001-05-12') == t 94 | assert g2t('2001/05/12') == t 95 | assert g2t('5/12/2001') == t 96 | assert g2t('123456') == 123456 97 | 98 | def testGenericStringErrors(self): 99 | """Test genstrtotime on some bad strings""" 100 | g2t = Time.genstrtotime 101 | self.assertRaises(Time.TimeException, g2t, "hello") 102 | self.assertRaises(Time.TimeException, g2t, "") 103 | self.assertRaises(Time.TimeException, g2t, "3q") 104 | 105 | def testTimeZone(self): 106 | """Test stringtotime on two strings straddling timezones""" 107 | f = Time.stringtotime 108 | invf = Time.timetostring 109 | s1 = "2005-04-03T03:45:03-03:00" 110 | s2 = "2005-04-03T02:45:03-03:00" 111 | diff = f(s1) - f(s2) 112 | assert diff == 3600, diff 113 | 114 | assert f(invf(f(s1))) == f(s1), (s1, invf(f(s1)), f(invf(f(s1))), f(s1)) 115 | assert f(invf(f(s2))) == f(s2), (s2, f(invf(f(s2))), f(s2)) 116 | 117 | 118 | if __name__ == '__main__': unittest.main() 119 | -------------------------------------------------------------------------------- /rdiff-backup/testing/user_grouptest.py: -------------------------------------------------------------------------------- 1 | import unittest, pwd, grp, code 2 | from commontest import * 3 | from rdiff_backup import user_group 4 | 5 | 6 | class UserGroupTest(unittest.TestCase): 7 | """Test user and group functionality""" 8 | def test_basic_conversion(self): 9 | """Test basic id2name. May need to modify for different systems""" 10 | user_group.uid2uname_dict = {}; user_group.gid2gname_dict = {} 11 | assert user_group.uid2uname(0) == "root" 12 | assert user_group.uid2uname(0) == "root" 13 | assert user_group.gid2gname(0) == "root" 14 | assert user_group.gid2gname(0) == "root" 15 | # Assume no user has uid 29378 16 | assert user_group.gid2gname(29378) is None 17 | assert user_group.gid2gname(29378) is None 18 | 19 | def test_basic_reverse(self): 20 | """Test basic name2id. Depends on systems users/groups""" 21 | user_group.uname2uid_dict = {}; user_group.gname2gid_dict = {} 22 | assert user_group.uname2uid("root") == 0 23 | assert user_group.uname2uid("root") == 0 24 | assert user_group.gname2gid("root") == 0 25 | assert user_group.gname2gid("root") == 0 26 | assert user_group.uname2uid("aoeuth3t2ug89") is None 27 | assert user_group.uname2uid("aoeuth3t2ug89") is None 28 | 29 | def test_default_mapping(self): 30 | """Test the default user mapping""" 31 | Globals.isdest = 1 32 | rootid = 0 33 | binid = pwd.getpwnam('bin')[2] 34 | syncid = pwd.getpwnam('sync')[2] 35 | user_group.init_user_mapping() 36 | assert user_group.UserMap(0) == 0 37 | assert user_group.UserMap(0, 'bin') == binid 38 | assert user_group.UserMap(0, 'sync') == syncid 39 | assert user_group.UserMap.map_acl(0, 'aoeuth3t2ug89') is None 40 | 41 | def test_user_mapping(self): 42 | """Test the user mapping file through the DefinedMap class""" 43 | mapping_string = """ 44 | root:bin 45 | bin:root 46 | 500:501 47 | 0:sync 48 | sync:0""" 49 | Globals.isdest = 1 50 | rootid = 0 51 | binid = pwd.getpwnam('bin')[2] 52 | syncid = pwd.getpwnam('sync')[2] 53 | daemonid = pwd.getpwnam('daemon')[2] 54 | user_group.init_user_mapping(mapping_string) 55 | 56 | assert user_group.UserMap(rootid, 'root') == binid 57 | assert user_group.UserMap(binid, 'bin') == rootid 58 | assert user_group.UserMap(0) == syncid 59 | assert user_group.UserMap(syncid, 'sync') == 0 60 | assert user_group.UserMap(500) == 501 61 | 62 | assert user_group.UserMap(501) == 501 63 | assert user_group.UserMap(123, 'daemon') == daemonid 64 | 65 | assert user_group.UserMap.map_acl(29378, 'aoeuth3t2ug89') is None 66 | assert user_group.UserMap.map_acl(0, 'aoeuth3t2ug89') is syncid 67 | 68 | if 0: code.InteractiveConsole(globals()).interact() 69 | 70 | def test_overflow(self): 71 | """Make sure querying large uids/gids doesn't raise exception""" 72 | large_num = 4000000000 73 | assert user_group.uid2uname(large_num) is None 74 | assert user_group.gid2gname(large_num) is None 75 | 76 | 77 | if __name__ == "__main__": unittest.main() 78 | --------------------------------------------------------------------------------