├── LICENSE ├── NOTICE ├── README.mdown └── ssync /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | -------------------------------------------------------------------------------- /NOTICE: -------------------------------------------------------------------------------- 1 | ssync 2 | Copyright 2013 AddThis 3 | 4 | 5 | -------------------------------------------------------------------------------- /README.mdown: -------------------------------------------------------------------------------- 1 | ## Description 2 | 3 | ssync is a divide-and-conquer file copying tool to multiple destination 4 | hosts. It transfers to `n` remote machines in `log n` iterations. 5 | ssync is a wrapper around the unix tool rsync. Command line arguments 6 | to ssync are identical to command line arguments for rsync with 7 | the following exceptions: 8 | 9 | - Must use the dummy name "remote" as the destination host, ie. "[USER@]remote:DEST". 10 | - Must include the option "--hosts host1 host2 host3" at end of the arguments. 11 | - The trailing slash is not supported in the SRC arguments 12 | - Contacting the rsync daemon (using :: notation) has not been tested. 13 | 14 | Examples: 15 | 16 | ssync -t *.h *.c remote:dest/ --hosts foo bar 17 | ssync -avz /data/tmp remote:/dest --hosts foo bar baz quux 18 | 19 | If any child process is killed or exits with a non-zero status code 20 | then the parent process will exit with that status code. ssync 21 | offers atomicity in case of failures at the granularity of 22 | the underlying rsync invocations. Each remote host should 23 | have observed either all of the updates or none of them. 24 | 25 | The "--continue" command line option will allow ssync to ignore 26 | the failure to connect to a host. The script will attempt to 27 | connect to the remaining hosts in the list. 28 | -------------------------------------------------------------------------------- /ssync: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from __future__ import print_function 16 | import getpass 17 | import re 18 | import signal 19 | import socket 20 | import string 21 | import subprocess 22 | import sys 23 | import os 24 | 25 | # ssync the recursive wrapper around rsync 26 | # 27 | # Author: Michael Spiegel 28 | # named "ssync" because "ssync" comes after "rsync" 29 | # also author's last name begins with the letter "s" 30 | 31 | # path to rsync command 32 | rsyncCommand = subprocess.Popen(["which", "rsync"], 33 | stdout=subprocess.PIPE).communicate()[0].strip() 34 | 35 | hostname = socket.gethostname() 36 | 37 | # recursive depth. Used for logging purposes. 38 | depth = 0 39 | 40 | # suppress non-error messages 41 | quiet = False 42 | 43 | # continue on failure of a subprocess 44 | doContinue = False 45 | 46 | optionRegex = re.compile("-[^\-]") 47 | 48 | def printUsage(): 49 | print() 50 | print("ssync: the recursive file transfer program on top of rsync") 51 | print() 52 | print("ssync usage is identical to rsync with these exceptions:") 53 | print("- Must specify the string \"[USER@]remote:DEST\".") 54 | print("- Must include the option \"--hosts host1 host2 host3\" at end of the arguments.") 55 | print("- The trailing slash is not spported in the SRC arguments") 56 | print("- Contacting the rsync daemon (using :: notation) has not been tested.") 57 | print() 58 | print("Examples:") 59 | print(" ssync -t *.c remote:dest/ --hosts foo bar") 60 | print(" ssync -avz /data/tmp remote:/dest --hosts foo bar baz quux") 61 | print() 62 | 63 | # Remove "--depth num" argument from command line options 64 | # Recursive depth is used for logging purposes. 65 | def extractOptions(argv): 66 | global depth, quiet, doContinue 67 | count = argv.count('--depth') 68 | quiet = argv.count('--quiet') > 0 69 | doContinue = argv.count('--continue') > 0 70 | # find all options of the form '-xyz' 71 | singleOpts = [x for x in argv if optionRegex.match(x) is not None] 72 | for opts in singleOpts: 73 | quiet = quiet or (opts.find('q') > 0) 74 | if argv.count('--continue') > 1: 75 | print() 76 | print("USAGE: multiple \"--continue\" arguments identified") 77 | print() 78 | sys.exit(2) 79 | if count > 1: 80 | print() 81 | print("USAGE: multiple \"--depth\" arguments identified") 82 | print() 83 | sys.exit(2) 84 | elif count == 1: 85 | location = argv.index('--depth') 86 | if location == len(argv) - 1: 87 | print() 88 | print("USAGE: \"--depth\" argument missing parameter") 89 | print() 90 | sys.exit(2) 91 | depth = int(argv[location + 1]) 92 | del argv[location : (location + 2)] 93 | 94 | # Validate that all source specifications do not have a trailing slash 95 | def validateArguments(terminus, argv): 96 | for i in range(1, terminus): 97 | current = argv[i] 98 | # skip options 99 | if current[0] == "-": 100 | continue 101 | elif current[-1] == "/": 102 | print() 103 | print("USAGE: ssync does not support trailing slashes in SRC " +\ 104 | "arguments") 105 | print() 106 | sys.exit(2) 107 | 108 | # Replace the dummy host name "remote" with the target host name 109 | def replaceHostName(hostName, rsyncOptions): 110 | 111 | result = list(rsyncOptions) 112 | 113 | if len(result) == 0: 114 | print() 115 | print("USAGE: missing required argument " +\ 116 | "\"[USER@]remote:DEST\"") 117 | print() 118 | sys.exit(2) 119 | 120 | # destination should be last argument to rsync 121 | candidate = result[-1] 122 | 123 | # search for dummy remote target within the string 124 | position = string.find(candidate, "remote:") 125 | 126 | if position < 0: 127 | print() 128 | print("USAGE: missing required argument " +\ 129 | "\"[USER@]remote:DEST\"") 130 | print() 131 | sys.exit(2) 132 | 133 | # replace dummy remote target with actual host name 134 | result[-1] = string.replace(candidate, "remote:", "%s:" % hostName) 135 | 136 | return result 137 | 138 | # returns the username and destination directory for rsync 139 | def findUsernameAndDirectory(rsyncOptions): 140 | 141 | candidate = rsyncOptions[-1] 142 | 143 | pos = string.find(candidate, "remote:") 144 | 145 | if pos == 0: 146 | username = getpass.getuser() 147 | elif candidate[pos - 1] != "@": 148 | print() 149 | print("ssync error: username cannot be parsed in \"%s\"" % (candidate)) 150 | sys.exit(2) 151 | else: 152 | username = candidate[ : pos - 1] 153 | 154 | dirPos = pos + len("remote:") 155 | 156 | if dirPos >= len(candidate): 157 | print() 158 | print("ssync error: destination directory is not specified.") 159 | sys.exit(2) 160 | 161 | directory = candidate[dirPos : ] 162 | 163 | return (username, directory) 164 | 165 | # Locate the index of command-line argument "--hosts" and perform error checking 166 | def findHosts(argv): 167 | 168 | count = argv.count('--hosts') 169 | 170 | if count < 1: 171 | print() 172 | print("USAGE: missing required argument " +\ 173 | "\"--hosts host1 host2 ... hostN\"") 174 | print() 175 | sys.exit(2) 176 | elif count > 1: 177 | print() 178 | print("USAGE: multiple \"--hosts\" arguments identified") 179 | print() 180 | sys.exit(2) 181 | 182 | hostBegin = argv.index('--hosts') 183 | 184 | if hostBegin < 3: 185 | print() 186 | print("USAGE: missing rsync options") 187 | print() 188 | sys.exit(2) 189 | 190 | if hostBegin == len(argv) - 1: 191 | print() 192 | print("USAGE: \"--hosts\" argument has not specified any hosts") 193 | print() 194 | sys.exit(2) 195 | 196 | return hostBegin 197 | 198 | # Wait for child processes to finish 199 | # If a child process died then kill all children and quit this process 200 | def waitPids(pids, basecase): 201 | failed = [] 202 | for key, pid in pids.iteritems(): 203 | (childpid, status) = os.waitpid(pid, 0) 204 | if status != 0: 205 | signal = status & 0xFF 206 | status = status >> 8 207 | if (signal > 0): 208 | if basecase: 209 | print("[%d] child process %s attempting to connect to %s killed by signal %s" % \ 210 | (depth, pid, key, signal), file=sys.stderr) 211 | else: 212 | print("[%d] child process %s killed by signal %s" % \ 213 | (depth, pid, signal), file=sys.stderr) 214 | exitcode = 1 215 | else: 216 | if basecase: 217 | print("[%d] child process %s attempting to connect to %s exited with non-zero status %s" % \ 218 | (depth, pid, key, status), file=sys.stderr) 219 | else: 220 | print("[%d] child process %s exited with non-zero status %s" % \ 221 | (depth, pid, status), file=sys.stderr) 222 | exitcode = status 223 | if not doContinue: 224 | stopAndQuit(pids, exitcode) 225 | else: 226 | failed.append(key) 227 | return failed 228 | 229 | # Send a term signal to all processes in the pid list 230 | # and then terminate with exitcode status code 231 | def stopAndQuit(pids, exitcode): 232 | for pid in pids: 233 | try: 234 | os.kill(pid, signal.SIGTERM) 235 | except Exception: 236 | pass 237 | if depth == 0: 238 | print() 239 | print("ERROR: One or more child ssync processes either exited with") 240 | print(" non-zero status or received a termination signal.") 241 | print("You can try to rerun ssync with the --continue option.") 242 | print() 243 | sys.exit(exitcode) 244 | 245 | # Generate the child ssync call 246 | def generateRecursiveHalf(rsyncOptions, head, hosts, user, targetDir): 247 | remote_ssync_script = os.path.join(targetDir, "ssync") 248 | options = list(rsyncOptions) 249 | options.insert(0, remote_ssync_script) 250 | options.insert(1, "--depth %d" % (depth + 1)) 251 | options.append("--hosts") 252 | options.extend(hosts) 253 | if quiet: 254 | qFlags = "-q -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" 255 | else: 256 | qFlags = "" 257 | rsync_command = string.join(options).replace('"', '\"').replace('$', '\$') 258 | command = "ssh %s %s@%s \"cd %s; %s; rm %s\"" % (qFlags, user, head, 259 | targetDir, rsync_command, remote_ssync_script) 260 | return command 261 | 262 | # Split the host list in half and recursively call ssync on the target machines 263 | def recursive(rsyncOptions, hostList): 264 | 265 | (user, targetDir) = findUsernameAndDirectory(rsyncOptions) 266 | 267 | halfLength = len(hostList) / 2 268 | 269 | leftHosts = hostList[ : halfLength] 270 | rightHosts = hostList[halfLength : ] 271 | 272 | leftHead = leftHosts.pop(0) 273 | rightHead = rightHosts.pop(0) 274 | 275 | heads = list((leftHead, rightHead)) 276 | 277 | while len(heads) > 0: 278 | failed = basecase(rsyncOptions, heads) 279 | heads = [] 280 | if leftHead in failed and len(leftHosts) > 0: 281 | leftHead = leftHosts.pop(0) 282 | heads.append(leftHead) 283 | if rightHead in failed and len(rightHosts) > 0: 284 | rightHead = rightHosts.pop(0) 285 | heads.append(rightHead) 286 | 287 | pids = {} 288 | 289 | if len(leftHosts) > 0: 290 | leftCommand = generateRecursiveHalf(rsyncOptions, leftHead, 291 | leftHosts, user, targetDir) 292 | 293 | process = subprocess.Popen(leftCommand, shell=True) 294 | # see comment regarding pylint 295 | pid = vars(process)['pid'] 296 | if not quiet: 297 | print("[%d] ssync from %s to %s (pid %s)" % (depth, leftHead, leftHosts, pid)) 298 | pids[leftHead] = pid 299 | 300 | if len(rightHosts) > 0: 301 | rightCommand = generateRecursiveHalf(rsyncOptions, rightHead, 302 | rightHosts, user, targetDir) 303 | 304 | process = subprocess.Popen(rightCommand, shell=True) 305 | # see comment regarding pylint 306 | pid = vars(process)['pid'] 307 | if not quiet: 308 | print("[%d] ssync from %s to %s (pid %s)" % (depth, rightHead, rightHosts, pid)) 309 | pids[rightHead] = pid 310 | 311 | if not quiet: 312 | print("[%d] waiting on pids %s" % (depth, pids)) 313 | 314 | waitPids(pids, False) 315 | 316 | # Launch a fixed small number of rsync processes in the background 317 | # and then wait for rysnc processes to finish 318 | def basecase(rsyncOptions, hostList): 319 | pids = {} 320 | if not quiet: 321 | print("[%d] rsync from %s to %s" % (depth, hostname, hostList)) 322 | for host in hostList: 323 | options = replaceHostName(host, rsyncOptions) 324 | options.insert(0, rsyncCommand) 325 | options.insert(1, __file__) 326 | if quiet: 327 | options.insert(1, "-e") 328 | options.insert(2, "ssh -q -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no") 329 | if '--continue' in options: 330 | options.remove('--continue') 331 | process = subprocess.Popen(options) 332 | # wanted to do the commented-out line but pylint has a bug 333 | # pids[host] = process.pid 334 | pids[host] = vars(process)['pid'] 335 | failed = waitPids(pids, True) 336 | return failed 337 | 338 | def main(argv): 339 | 340 | # if no arguments are provided then print usage and exit 341 | if (len(argv) == 1): 342 | printUsage() 343 | sys.exit(2) 344 | 345 | # extract the "--depth", "--quiet", and "--continue" command line argument 346 | extractOptions(argv) 347 | 348 | # find the position of the "--hosts" argument 349 | hostBegin = findHosts(argv) 350 | 351 | # do not validate the "--hosts" arguments nor the destination argument 352 | validateArguments(hostBegin - 1, argv) 353 | 354 | # lists of hosts occur after "--hosts" argument 355 | hostList = argv[hostBegin + 1 : ] 356 | 357 | # use rsync arguments {1, 2,..., hostBegin - 1} 358 | rsyncOptions = argv[1 : hostBegin] 359 | 360 | if len(hostList) < 4: 361 | basecase(rsyncOptions, hostList) 362 | else: 363 | recursive(rsyncOptions, hostList) 364 | 365 | if(__name__ == "__main__"): 366 | main(sys.argv) 367 | 368 | 369 | --------------------------------------------------------------------------------