├── .gitignore ├── LICENSE.txt ├── README.md ├── pytograph └── pytograph.cfg.dist /.gitignore: -------------------------------------------------------------------------------- 1 | # Vim swapfiles 2 | *.swp 3 | 4 | # Mac OS X Finder metadata 5 | *.DS_Store 6 | 7 | # Local pytograph configuration file(s) 8 | *.cfg 9 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Copyright (c) 2011-2012, Josh Dick 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are met: 6 | 7 | * Redistributions of source code must retain the above copyright notice, 8 | this list of conditions and the following disclaimer. 9 | * Redistributions in binary form must reproduce the above copyright 10 | notice, this list of conditions and the following disclaimer in the 11 | documentation and/or other materials provided with the distribution. 12 | 13 | THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND ANY 14 | EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 15 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 16 | DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR ANY 17 | DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 18 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 19 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 20 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 21 | LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 22 | OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH 23 | DAMAGE. 24 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # pytograph 2 | 3 | Reflect local filesystem changes on a remote system in real time, automatically. 4 | 5 | *by [Josh Dick][1]* 6 | 7 | *[https://github.com/joshdick/pytograph][2]* 8 | 9 | ## Introduction 10 | 11 | ### The Motivation 12 | 13 | Ever been frustrated by these situations? 14 | 15 | * You were editing code directly on a live system and had to remember which files to copy back to your own machine, in order to later get your changes into source control. 16 | * You develop code on your own machine, but had to remember to manually transfer files to a live system to test changes. 17 | 18 | Well, I sure have, and that's why I created pytograph. 19 | 20 | In a nutshell, you instruct pytograph that there are directory structures on your machine that match directory structures on a remote machine. It monitors the directories on your local machine for changes, then makes identical changes on the remote machine in real time via SFTP. So, if you add/update/delete a file locally, that same file will be automatically added/updated/deleted on the remote machine. If you delete a directory locally, that same directory will be deleted from the remote machine. You get the picture. 21 | 22 | ### But Why's It Called That? 23 | 24 | Pytograph is named after the [pantograph][3], an image duplication device. A pantograph uses the movements of a pen to simultaneously draw a duplicate image. Pytograph is like a pantograph for your files. Python + pantograph = pytograph. 'Nuff said. 25 | 26 | ### Can't I Just Use rsync? 27 | 28 | Yes, you can, but pytograph only needs to be invoked once and automatically keeps things in sync after that. So there's no need to run something like rsync after every change you make locally. Just set it and forget it. 29 | 30 | ## Installation and Usage 31 | 32 | Pytograph requires Python 2.7. 33 | 34 | It depends on several third-party Python packages: config, pysftp, and watchdog. The easiest way to install these is via [pip][4]: 35 | 36 | `$ pip install config pysftp watchdog` 37 | 38 | Once pytograph's dependencies are installed, view the included sample configuration file, pytograph.cfg.dist. This file contains all of pytograph's settings with accompanying explanations. 39 | 40 | Make a copy of pytograph.cfg.dist and edit it as appropriate for your environment, then save it as pytograph.cfg in the same directory as the script itself. 41 | 42 | Then simply run `pytograph.py` and you're good to go! 43 | 44 | *Note: By default, pytograph looks for a pytograph.cfg configuration file located in the same directory as the script itself, but you can specify a custom configuration file location using a command line argument. Run `pytograph.py -h` for details.* 45 | 46 | ## This Seems Far Too Awesome, What's The Catch? 47 | 48 | Pytograph has only been tested against Python 2.7.2 on Mac OS X, but it should theoretically work with all versions of Python 2.7. It should work on Linux or any *NIX. It may even work on Windows...anything's possible! :) 49 | 50 | **OBLIGATORY WARNING: Please back up any important data before using pytograph.** While pytograph has worked well for me, I take **no responsibility** if it causes data loss, strange storm patterns, indigestion, zombie attacks, global thermonuclear war, etc. **Use it at your own risk.** 51 | 52 | That said, I welcome questions, comments and GitHub pull requests! 53 | 54 | pytograph is released under the Simplified BSD License. 55 | 56 | ## Now Try It Out Already! :) 57 | 58 | [1]: http://joshdick.net 59 | [2]: https://github.com/joshdick/pytograph 60 | [3]: http://en.wikipedia.org/wiki/Pantograph 61 | [4]: http://pip-installer.org 62 | -------------------------------------------------------------------------------- /pytograph: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """ 4 | pytograph - Reflect local filesystem changes on a remote system in real time, automatically. 5 | 6 | 7 | 8 | Requires Python 2.7, and the third-party Python packages config, pysftp, and watchdog. 9 | """ 10 | 11 | __author__ = 'Josh Dick ' 12 | __email__ = 'josh@joshdick.net' 13 | __copyright__ = '(c) 2011-2012, Josh Dick' 14 | __license__ = 'Simplified BSD' 15 | 16 | from config import Config 17 | from watchdog.observers import Observer 18 | from watchdog.events import * 19 | import argparse, getpass, logging, paramiko, posixpath, pysftp, sys, time 20 | 21 | logFormat='%(levelname)s: %(message)s' 22 | logging.basicConfig(format=logFormat) 23 | logger = logging.getLogger('pytograph') 24 | logger.setLevel(logging.INFO) 25 | 26 | class PytoWatchdogHandler(PatternMatchingEventHandler): 27 | """ 28 | Watchdog event handler. 29 | Triggers appropriate actions on a remote server via a RemoteControl when 30 | specific Watchdog events are fired due to local filesystem changes. 31 | """ 32 | 33 | def __init__(self, remote_control, **kw): 34 | super(PytoWatchdogHandler, self).__init__(**kw) 35 | self._remote_control = remote_control 36 | 37 | def on_created(self, event): 38 | if isinstance(event, DirCreatedEvent): 39 | # Ignoring this event for now since directories will automatically 40 | # be created on the remote server by transfer_file() 41 | logger.debug('Ignoring DirCreatedEvent for %s' % event.src_path) 42 | else: 43 | self._remote_control.transfer_file(event.src_path) 44 | 45 | def on_deleted(self, event): 46 | self._remote_control.delete_resource(event.src_path) 47 | 48 | def on_modified(self, event): 49 | if isinstance(event, DirModifiedEvent): 50 | logger.debug('Ignoring DirModifiedEvent for %s' % event.src_path) 51 | else: 52 | self._remote_control.transfer_file(event.src_path) 53 | 54 | def on_moved(self, event): 55 | self._remote_control.move_resource(event.src_path, event.dest_path) 56 | 57 | 58 | class RemoteControl: 59 | """ 60 | Performs filesystem manipulations on a remote server, 61 | using data from the local machine's filesystem as necessary. 62 | """ 63 | 64 | def __init__(self, sftp_connection, local_base, remote_base): 65 | self._connection = sftp_connection.connection 66 | self._ssh_prefix = sftp_connection.ssh_prefix 67 | self._local_base = local_base 68 | self._remote_base = remote_base 69 | self._local_base_length = len(local_base) 70 | 71 | # Given a full canonical path on the local filesystem, returns an equivalent full 72 | # canonical path on the remote filesystem. 73 | def get_remote_path(self, local_path): 74 | # Strip the local base path from the local full canonical path to get the relative path 75 | # TODO: This will not work properly on Windows since slash directions will conflict 76 | remote_relative = local_path[self._local_base_length:] 77 | return self._remote_base + remote_relative 78 | 79 | def transfer_file(self, src_path): 80 | dest_path = self.get_remote_path(src_path) 81 | logger.info('Copying\n\t%s\nto\n\t%s:%s' % (src_path, self._ssh_prefix, dest_path)) 82 | try: 83 | # Make sure the intermediate destination path to this file actually exists on the remote machine 84 | self._connection.execute('mkdir -p "' + os.path.split(dest_path)[0] + '"') 85 | self._connection.put(src_path, dest_path) 86 | except Exception as e: 87 | logger.error('Caught exception while copying:') 88 | logger.exception(e) 89 | 90 | def delete_resource(self, src_path): 91 | dest_path = self.get_remote_path(src_path) 92 | logger.info('Deleting %s:%s' % (self._ssh_prefix, dest_path)) 93 | try: 94 | self._connection.execute('rm -rf "' + dest_path + '"') 95 | except Exception as e: 96 | logger.error('Caught exception while deleting:') 97 | logger.exception(e) 98 | 99 | def move_resource(self, src_path, dest_path): 100 | logger.info('Moving\n\t%s:%s\nto\n\t%s:%s' % 101 | (self._ssh_prefix, self.get_remote_path(src_path), self._ssh_prefix, self.get_remote_path(dest_path))) 102 | try: 103 | # Make sure the intermediate destination path to this file actually exists on the remote machine 104 | self._connection.execute('mkdir -p "' + os.path.split(dest_path)[0] + '"') 105 | self._connection.execute('mv "' + src_path + '" "' + dest_path + '"') 106 | except Exception as e: 107 | logger.error('Caught exception while moving:') 108 | logger.exception(e) 109 | 110 | 111 | class SFTPConnection: 112 | """ 113 | Maintains an SSH connection to a remote server via pysftp. 114 | """ 115 | 116 | def __init__(self, host, port = None, private_key_file = None, private_key_password = None, username = None, password = None): 117 | 118 | self._ssh_prefix = None 119 | self._connection = None 120 | 121 | if username == '': 122 | username = getpass.getuser() 123 | logger.debug('No username configured; assuming username %s' % username) 124 | else: 125 | logger.debug('Using configured username %s' % username) 126 | 127 | self._ssh_prefix = '%s@%s' % (username, host) 128 | 129 | # Attempt key authentication if no password was specified; prompt for a password if it fails 130 | if password == '': 131 | try: 132 | logger.debug('No password specified, attempting to use key authentication') 133 | self._connection = pysftp.Connection(host, port = port, username = username, private_key = private_key_file, private_key_pass = private_key_password) 134 | except Exception as e: 135 | logger.debug('Key authentication failed.\nCause: %s\nFalling back to password authentication...' % e) 136 | password = getpass.getpass('Password for %s: ' % self._ssh_prefix) 137 | else: 138 | logger.debug('Using configured password') 139 | 140 | # If we don't have a connection yet, attempt password authentication 141 | if self._connection is None: 142 | try: 143 | self._connection = pysftp.Connection(host, port = port, username = username, password = password) 144 | except Exception as e: 145 | logger.error('Could not successfully connect to %s\nCause: %s' % (self._ssh_prefix, e)) 146 | sys.exit(1) 147 | 148 | logger.debug('Successfully connected to %s' % self._ssh_prefix) 149 | 150 | @property 151 | def ssh_prefix(self): 152 | """ 153 | (Read-only) 154 | String containing the username and host information for the remote server. 155 | """ 156 | return self._ssh_prefix 157 | 158 | @property 159 | def connection(self): 160 | """ 161 | (Read-only) 162 | A pysftp Connection object representing the active connection to the remote server. 163 | """ 164 | return self._connection 165 | 166 | 167 | def _main(): 168 | 169 | # Cannot use argparse.FileType with a default value since the help message will not display if pytograph.cfg 170 | # doesn't appear in the default location. Could subclass argparse.FileType but the following seems more intuitive. 171 | # See http://stackoverflow.com/questions/8236954/specifying-default-filenames-with-argparse-but-not-opening-them-on-help 172 | parser = argparse.ArgumentParser(description='Reflect local filesystem changes on a remote system in real time, automatically.') 173 | parser.add_argument('-c', '--config-file', default='pytograph.cfg', help='location of a pytograph configuration file') 174 | args = parser.parse_args() 175 | 176 | try: 177 | config_file = file(args.config_file) 178 | except Exception as e: 179 | logger.error('Couldn\'t read pytograph configuration file!\n\ 180 | Either place a pytograph.cfg file in the same folder as pytograph.py, or specify an alternate location.\n\ 181 | Run \'%s -h\' for usage information.\nCause: %s' % (os.path.basename(__file__), e)) 182 | sys.exit(1) 183 | 184 | try: 185 | cfg = Config(config_file) 186 | except Exception as e: 187 | logger.error('Pytograph configuration file is invalid!\nCause: %s' % e) 188 | sys.exit(1) 189 | 190 | # Read configuration 191 | local_root_path = os.path.abspath(os.path.expanduser(cfg.local_root_path)) 192 | if not os.path.isdir(local_root_path): 193 | logger.error('Invalid local_root_path configured: %s is not a valid path on the local machine' % cfg.local_root_path) 194 | sys.exit(1) 195 | else: 196 | logger.debug('Using local root path: ' + local_root_path) 197 | 198 | # Create persistent SSH connection to remote server 199 | sftp_connection = SFTPConnection(cfg.remote_host, cfg.remote_port, cfg.private_key_file, cfg.private_key_password, cfg.remote_username, cfg.remote_password) 200 | 201 | logger.debug('Initializating path mappings...') 202 | 203 | # If this is still true when the loop below completes, no valid mappings are configured. 204 | no_valid_mappings = True 205 | 206 | observer = Observer() 207 | 208 | for mapping in cfg.path_mappings: 209 | 210 | # Create an absolute local path from the local root path and this mapping's local relative path 211 | local_base = os.path.join(local_root_path, mapping.local) 212 | if not os.path.isdir(local_base): 213 | logger.warn('Invalid path mapping configured: %s is not a valid path on the local machine' % local_base) 214 | continue 215 | 216 | # If we got this far, we have at least one valid mapping 217 | no_valid_mappings = False 218 | 219 | # Create an absolute remote path from the remote root path and this mapping's remote relative path 220 | # Use explicit posixpath.join since the remote server will always use UNIX-style paths for SFTP 221 | # TODO: Validate this, expand tilde notation, etc. 222 | remote_base = posixpath.join(cfg.remote_root_path, mapping.remote) 223 | 224 | logger.info('Path mapping initializing:\nChanges at local path\n\t%s\nwill be reflected at remote path\n\t%s:%s' 225 | % (local_base, sftp_connection.ssh_prefix, remote_base)) 226 | 227 | # Create necessary objects for this particular mapping and schedule this mapping on the Watchdog observer as appropriate 228 | remote_control = RemoteControl(sftp_connection = sftp_connection, local_base = local_base, remote_base = remote_base) 229 | event_handler = PytoWatchdogHandler(ignore_patterns = cfg.ignore_patterns, remote_control = remote_control) 230 | observer.schedule(event_handler, path=local_base, recursive=True) 231 | 232 | if no_valid_mappings: 233 | logger.error('No valid path mappings were configured, so there\'s nothing to do. Please check your pytograph configuration file.') 234 | sys.exit('Terminating.') 235 | 236 | # We have at least one valid mapping, so start the Watchdog observer - filesystem monitoring actually begins here 237 | observer.start() 238 | 239 | try: 240 | while True: 241 | time.sleep(1) 242 | except KeyboardInterrupt: 243 | observer.stop() 244 | observer.join() 245 | 246 | if __name__ == "__main__": 247 | 248 | _main() 249 | 250 | -------------------------------------------------------------------------------- /pytograph.cfg.dist: -------------------------------------------------------------------------------- 1 | # Configuration file for pytograph. 2 | # 3 | 4 | # Files matching these patterns will be completely ignored (Vim swapfiles, Mac .DS_Store files, etc.) 5 | ignore_patterns: ['*.swp', '*.DS_Store'] 6 | 7 | # Hostname or IP address of the remote server. 8 | remote_host: '' 9 | 10 | # TCP port for the SSH service on the remote server. 11 | # The default port is 22. 12 | remote_port: 22 13 | 14 | # SSH username for the remote server. 15 | # If blank, will use the username of the current user of the local machine. 16 | remote_username: '' 17 | 18 | # Path to an SSH private key file to use for key authentication. 19 | # If blank, will attempt to automatically determine the correct private key file to use. 20 | private_key_file: '' 21 | 22 | # Password for the SSH private key file specified above, if necessary. 23 | private_key_password: '' 24 | 25 | # SSH password for the remote server. 26 | # If blank, will attempt to use key authentication for the username as specified/determined above. 27 | # If key authentication fails, you will be interactively prompted for a password. 28 | remote_password: '' 29 | 30 | # ==== Path Configuration ==== 31 | # 32 | # Trailing slashes are optional for all paths. 33 | # 34 | # For each path mapping "pm" defined in path_mappings, pytograph will: 35 | # 36 | # 1) Monitor the following folder for changes: / 37 | # 2) Reflect those changes in the following remote folder: / 38 | # 39 | 40 | # Top level directory location on the remote server; should be an absolute path *without tilde notation* 41 | # (should start with a leading slash.) 42 | # Should use UNIX-style forward slashes, since the remote server will be accessed via SFTP. 43 | remote_root_path: '/var/www' 44 | 45 | # Top level directory location on the local machine. Should be an absolute path. Tilde notation is acceptable. 46 | local_root_path: '~/code' 47 | 48 | # A list of relative path mappings (local path => remote path.) 49 | # Each local path should be located relative to and underneath the defined above. 50 | # Each remote path will be manipulated relative to and underneath the defined above, 51 | # and should use forward slashes. 52 | # No path appearing below should contain leading slashes. 53 | path_mappings: 54 | [ 55 | #{ local: 'my/source/code/www/js', remote: 'js' }, # Would map /my/source/code/www/js => /js 56 | #{ local: 'my/source/code/www/css', remote: 'css' }, # Would map /my/source/code/www/css => /css 57 | ] 58 | --------------------------------------------------------------------------------