├── .gitignore ├── LICENSE.txt ├── MANIFEST.in ├── README.rst ├── gitobox ├── __init__.py ├── __main__.py ├── git.py ├── hooks │ └── update ├── main.py ├── server.py ├── sync.py ├── timer.py ├── utils.py └── watch.py ├── setup.cfg └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.py[co] 2 | 3 | # Packages 4 | *.egg 5 | *.egg-info 6 | dist 7 | build 8 | eggs 9 | parts 10 | bin 11 | var 12 | sdist 13 | develop-eggs 14 | .installed.cfg 15 | lib 16 | lib64 17 | 18 | # Installer logs 19 | pip-log.txt 20 | 21 | # Unit test / coverage reports 22 | .coverage 23 | .tox 24 | nosetests.xml 25 | 26 | # IDEs 27 | .idea 28 | .project 29 | .pydevproject 30 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Copyright (c) 2014, Remi Rampin 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are met: 6 | 7 | 1. Redistributions of source code must retain the above copyright notice, this 8 | list of conditions and the following disclaimer. 9 | 10 | 2. Redistributions in binary form must reproduce the above copyright notice, 11 | this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 12 | 13 | 3. Neither the name of the copyright holder nor the names of its contributors 14 | may be used to endorse or promote products derived from this software 15 | without specific prior written permission. 16 | 17 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 18 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 19 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 20 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 21 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 22 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 23 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 24 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 25 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 26 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 27 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include README.rst 2 | include LICENSE.txt 3 | 4 | global-exclude *.py[co] 5 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | Gitobox 2 | ======= 3 | 4 | This program synchronizes a DropBox directory (or any directory) with a Git repository. Any change in the directory will create a new commit on a specified branch, and a push to that branch will update the directory. 5 | 6 | Note that this is different from putting your .git folder inside your DropBox. Here, you don't have anything git-related in DropBox, Gitobox gives you a separate, 2-way-synced Git repository that you can use instead of DropBox (or just keep using DropBox and have the Git history for future reference). 7 | 8 | Deployment Guide 9 | ---------------- 10 | 11 | First, this is intended to be deployed on some kind of server or "always on" machine. While it will work correctly in other setups, Gitobox cannot detect and record versions that happen while it is offline, so you would get a single "big commit" when synchronization resumes, making the history less useful. 12 | 13 | Installing Gitobox is easy if you have Python and `pip `_ installed:: 14 | 15 | $ pip install gitobox 16 | 17 | Simply create a Git repository that will be synced:: 18 | 19 | $ git init dropbox-project 20 | 21 | The automatic commits will be created with the local identity, so you might want to set that as well:: 22 | 23 | $ pushd dropbox-project 24 | $ git config user.name "dropbox" 25 | $ git config user.email "gitobox@my-own-server" 26 | $ popd 27 | 28 | then start Gitobox:: 29 | 30 | $ gitobox ~/Dropbox/my-project dropbox-project/.git 31 | 32 | You can then clone that repository and tada! You get to work with Git instead of Dropbox:: 33 | 34 | $ git clone my-working-copy dropbox-project 35 | 36 | FAQ 37 | --- 38 | 39 | Why make Gitobox? 40 | ''''''''''''''''' 41 | 42 | Because we all have friends/colleagues who don't know how to use Git, and sometimes you have to work with them. 43 | 44 | Does it work? 45 | ''''''''''''' 46 | 47 | Yes! Although not very thoroughly (and automatically?) tested yet, it does work. 48 | 49 | Can I use something else than Linux? 50 | '''''''''''''''''''''''''''''''''''' 51 | 52 | Yes! Through the use of `watchdog `__, all platforms should now be supported. 53 | 54 | Can I use something else than Git? 55 | '''''''''''''''''''''''''''''''''' 56 | 57 | Any version control system that can notify Gitobox when new changes come in should work; you just have to write a replacement for the notification, check in, and check out code. I have no intention of doing that myself, but patches are welcome! 58 | 59 | Can I use something else than DropBox? 60 | '''''''''''''''''''''''''''''''''''''' 61 | 62 | Yes, Gitobox has no knowledge of what DropBox is, it just watches a directory. Whether this gets changed by rsync, FTP, DropBox, Google Drive or Bittorrent Sync does not matter. 63 | 64 | It would probably be cool to try and get metadata from the syncing system (i.e. the name of the user who made the change), but this is not written yet. 65 | 66 | Are conflicts possible? 67 | ''''''''''''''''''''''' 68 | 69 | Yes; this is because DropBox has no locking mechanism. The Git repository will not accept pushes while the directory is changing, and the usual "non fast-forward; pull first" behavior will happen for conflicts on the Git side. However, if somebody changes the DropBox while Git is writing to it, conflicts will happen and will be resolved by DropBox's own mechanisms. This will in turn create a new commit in Git, so you can fix that from Git (but conflicted files will be side-by-side, the DropBox way). Gitobox also tries to detect that and give you a warning when you push, so read these "remote:" messages! 70 | -------------------------------------------------------------------------------- /gitobox/__init__.py: -------------------------------------------------------------------------------- 1 | __version__ = '0.3' 2 | -------------------------------------------------------------------------------- /gitobox/__main__.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | try: 4 | from gitobox.main import main 5 | except ImportError: 6 | sys.path.append(os.path.join(os.path.dirname(__file__), '..')) 7 | from gitobox.main import main 8 | 9 | 10 | if __name__ == '__main__': 11 | main() 12 | -------------------------------------------------------------------------------- /gitobox/git.py: -------------------------------------------------------------------------------- 1 | """Logic for handling Git repositories. 2 | 3 | Contains the :class:`~gitobox.git.GitRepository` class. 4 | """ 5 | 6 | from __future__ import unicode_literals 7 | 8 | import logging 9 | import os 10 | import pkg_resources 11 | from rpaths import Path 12 | import subprocess 13 | import sys 14 | import tarfile 15 | 16 | 17 | def decode_utf8(s): 18 | if isinstance(s, bytes): 19 | return s.decode('utf-8', 'replace') 20 | else: 21 | return s 22 | 23 | 24 | def repr_cmdline(cmd): 25 | return ' '.join(decode_utf8(s) for s in cmd) 26 | 27 | 28 | def shell_quote(s): 29 | """Given bl"a, returns "bl\\"a". 30 | 31 | Returns bytes. 32 | """ 33 | if not isinstance(s, bytes): 34 | s = s.encode('utf-8') 35 | if any(c in s for c in b' \t\n\r\x0b\x0c*$\\"\''): 36 | return b'"' + (s.replace(b'\\', b'\\\\') 37 | .replace(b'"', b'\\"') 38 | .replace(b'$', b'\\$')) + b'"' 39 | else: 40 | return s 41 | 42 | 43 | class GitRepository(object): 44 | def __init__(self, repo, workdir, branch, password, port): 45 | if not (repo / 'objects').is_dir() or not (repo / 'refs').is_dir(): 46 | logging.critical("Not a Git repository: %s", repo) 47 | sys.exit(1) 48 | 49 | self.repo = repo.absolute() 50 | self.workdir = workdir.absolute() 51 | self.branch = branch 52 | self._git = ['git', '--git-dir', self.repo.path, 53 | '--work-tree', self.workdir.path] 54 | 55 | self._run(['config', 'receive.denyCurrentBranch', 'ignore']) 56 | 57 | # Installs hook 58 | update_hook = self.repo / 'hooks' / 'update' 59 | if update_hook.exists(): 60 | with update_hook.open('rb') as fp: 61 | line = fp.readline().rstrip() 62 | if line.startswith(b'#!'): 63 | line = fp.readline().rstrip() 64 | if line != b'# Gitobox hook: do not edit!': 65 | logging.critical("Repository at %s already has an update " 66 | "hook; not overriding!\n" 67 | "Please delete it and try again\n", 68 | self.repo) 69 | sys.exit(1) 70 | else: 71 | logging.debug("Replacing update hook") 72 | else: 73 | logging.debug("Installing update hook") 74 | template = pkg_resources.resource_stream('gitobox', 'hooks/update') 75 | with update_hook.open('wb') as fp: 76 | for line in template: 77 | if line.find(b'{{') != -1: 78 | line = (line 79 | .replace(b'{{PASSWORD}}', shell_quote(password)) 80 | .replace(b'{{PORT}}', shell_quote(str(port))) 81 | .replace(b'{{BRANCH}}', shell_quote(branch))) 82 | fp.write(line) 83 | template.close() 84 | update_hook.chmod(0o755) 85 | 86 | def _run(self, cmd, allow_fail=False, stdout=False): 87 | logging.debug("Running: %s", repr_cmdline(['git'] + cmd)) 88 | cmd = self._git + cmd 89 | 90 | if allow_fail: 91 | return subprocess.call(cmd) 92 | elif stdout: 93 | return subprocess.check_output(cmd) 94 | else: 95 | return subprocess.check_call(cmd) 96 | 97 | def has_changes(self, ref): 98 | """Determines whether the working copy has changes, compared to `ref`. 99 | """ 100 | self._run(['update-ref', '--no-deref', 'HEAD', ref]) 101 | self._run(['add', '--all', '.']) 102 | status = self._run(['status', '--porcelain'], stdout=True) 103 | self._run(['symbolic-ref', 'HEAD', 'refs/heads/%s' % self.branch]) 104 | return bool(status.strip()) 105 | 106 | def check_in(self, paths=None): 107 | """Commit changes to the given files (if there are differences). 108 | 109 | If `paths` is None, assumes that any file might have changed. 110 | """ 111 | self._run(['symbolic-ref', 'HEAD', 'refs/heads/%s' % self.branch]) 112 | 113 | self._run(['add', '--all', '.']) 114 | ret = self._run(['commit', '-m', '(gitobox automatic commit)'], 115 | allow_fail=True) 116 | 117 | if ret == 0: 118 | ref = self._run(['rev-parse', 'HEAD'], stdout=True) 119 | logging.info("Created revision %s", ref.decode('ascii')) 120 | else: 121 | logging.info("No revision created") 122 | 123 | def check_out(self, ref): 124 | """Check out the given revision. 125 | """ 126 | fd, temptar = Path.tempfile() 127 | os.close(fd) 128 | try: 129 | self._run(['symbolic-ref', 'HEAD', 'refs/heads/%s' % self.branch]) 130 | 131 | # Creates an archive from the tree 132 | self._run(['archive', '--format=tar', '-o', temptar.path, ref]) 133 | tar = tarfile.open(str(temptar), 'r') 134 | 135 | # List the files in the tree 136 | files = set(self.workdir / m.name 137 | for m in tar.getmembers()) 138 | 139 | # Remove from the directory all the files that don't exist 140 | removed_files = False 141 | for path in self.workdir.recursedir(top_down=False): 142 | if path.is_file() and path not in files: 143 | logging.info("Removing file %s", path) 144 | path.remove() 145 | removed_files = True 146 | elif path.is_dir(): 147 | if not path.listdir() and removed_files: 148 | logging.info("Removing empty directory %s", path) 149 | path.rmdir() 150 | removed_files = False 151 | 152 | # Replace all the files 153 | tar.extractall(str(self.workdir)) 154 | 155 | tar.close() 156 | finally: 157 | temptar.remove() 158 | -------------------------------------------------------------------------------- /gitobox/hooks/update: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Gitobox hook: do not edit! 3 | # This file is installed by Gitobox and will be overridden the next time it is 4 | # started 5 | 6 | GITOBOX_PASSWORD={{PASSWORD}} 7 | GITOBOX_PORT={{PORT}} 8 | GITOBOX_BRANCH={{BRANCH}} 9 | 10 | # --- Command line 11 | refname="$1" 12 | oldrev="$2" 13 | newrev="$3" 14 | 15 | # --- Safety check 16 | if [ -z "$GIT_DIR" ]; then 17 | echo "Don't run this script from the command line." >&2 18 | echo " (if you want, you could supply GIT_DIR then run" >&2 19 | echo " $0 )" >&2 20 | exit 1 21 | fi 22 | 23 | if [ -z "$refname" -o -z "$oldrev" -o -z "$newrev" ]; then 24 | echo "usage: $0 " >&2 25 | exit 1 26 | fi 27 | 28 | # --- Check types 29 | # if $newrev is 0000...0000, it's a commit to delete a ref. 30 | zero="0000000000000000000000000000000000000000" 31 | if [ "$newrev" = "$zero" ]; then 32 | newrev_type=delete 33 | else 34 | newrev_type=$(git cat-file -t $newrev) 35 | fi 36 | 37 | # --- Gitobox communication logic 38 | do_sync(){ 39 | (echo "$GITOBOX_PASSWORD"; echo "$1") | \ 40 | nc 127.0.0.1 $GITOBOX_PORT | \ 41 | (while read line; do 42 | if [ "$line" = "OK" ]; then 43 | exit 0 44 | elif [ "$line" = "ERROR" ]; then 45 | echo "gitobox: fail" 1>&2 46 | exit 1 47 | fi 48 | echo "gitobox: $line" 1>&2 49 | done) 50 | } 51 | 52 | # --- Dispatches according to event type 53 | case "$refname","$newrev_type" in 54 | refs/heads/$GITOBOX_BRANCH,commit) 55 | do_sync "$newrev" || exit 1 56 | ;; 57 | refs/heads/$GITOBOX_BRANCH,delete) 58 | echo "*** You can't delete branch 'gitobox'" >&2 59 | exit 1 60 | ;; 61 | esac 62 | 63 | # --- Finished 64 | exit 0 65 | -------------------------------------------------------------------------------- /gitobox/main.py: -------------------------------------------------------------------------------- 1 | """Entry point for the gitobox utility. 2 | 3 | This contains :func:`~gitobox.main.main`, which is the entry point declared to 4 | setuptools. It is also callable directly. 5 | """ 6 | 7 | from __future__ import unicode_literals 8 | 9 | import argparse 10 | import codecs 11 | import locale 12 | import logging 13 | from rpaths import Path 14 | import sys 15 | 16 | from gitobox import __version__ as gitobox_version 17 | from gitobox.sync import synchronize 18 | 19 | 20 | def setup_logging(verbosity): 21 | levels = [logging.CRITICAL, logging.WARNING, logging.INFO, logging.DEBUG] 22 | level = levels[min(verbosity, 3)] 23 | 24 | fmt = "%(asctime)s %(levelname)s: %(message)s" 25 | formatter = logging.Formatter(fmt) 26 | 27 | handler = logging.StreamHandler() 28 | handler.setFormatter(formatter) 29 | 30 | logger = logging.getLogger() 31 | logger.setLevel(level) 32 | logger.addHandler(handler) 33 | 34 | 35 | def main(): 36 | """Entry point when called on the command line. 37 | """ 38 | # Locale 39 | locale.setlocale(locale.LC_ALL, '') 40 | 41 | # Encoding for output streams 42 | if str == bytes: # PY2 43 | writer = codecs.getwriter(locale.getpreferredencoding()) 44 | o_stdout, o_stderr = sys.stdout, sys.stderr 45 | sys.stdout = writer(sys.stdout) 46 | sys.stdout.buffer = o_stdout 47 | sys.stderr = writer(sys.stderr) 48 | sys.stderr.buffer = o_stderr 49 | 50 | # Parses command-line 51 | 52 | # General options 53 | options = argparse.ArgumentParser(add_help=False) 54 | options.add_argument('--version', action='version', 55 | version="gitobox version %s" % gitobox_version) 56 | options.add_argument('-v', '--verbose', action='count', default=1, 57 | dest='verbosity', 58 | help="augments verbosity level") 59 | 60 | parser = argparse.ArgumentParser( 61 | description="gitobox synchronizes a directory with a Git " 62 | "repository; it is particularly useful to make a Git " 63 | "branch out of changes happening in DropBox or " 64 | "similar \"dump\" collaboration software", 65 | parents=[options]) 66 | parser.add_argument('folder', 67 | help="Folder to watch for changes") 68 | parser.add_argument('repository', 69 | help="Git repository to synchronize") 70 | parser.add_argument('-b', '--branch', action='store', default='master', 71 | help="Git branch to synchronize (default: master)") 72 | parser.add_argument('-t', '--timeout', action='store', type=int, 73 | default='5', 74 | help="Time to wait after last directory change before " 75 | "committing (in seconds)") 76 | 77 | args = parser.parse_args() 78 | setup_logging(args.verbosity) 79 | 80 | synchronize(Path(args.folder), Path(args.repository), args.branch, 81 | args.timeout) 82 | 83 | sys.exit(0) 84 | 85 | 86 | if __name__ == '__main__': 87 | main() 88 | -------------------------------------------------------------------------------- /gitobox/server.py: -------------------------------------------------------------------------------- 1 | """Hook server code. 2 | 3 | Contains the :class:`~gitobox.server.Server` class which is used to communicate 4 | with the Git hook. 5 | """ 6 | 7 | from __future__ import unicode_literals 8 | 9 | import logging 10 | import select 11 | import socket 12 | import sys 13 | import time 14 | 15 | from gitobox.utils import irange, iteritems, itervalues 16 | 17 | 18 | class Server(object): 19 | """A server, that receives a bunch of lines on a TCP socket. 20 | 21 | Listens on a random TCP port (`port` attribute) and calls back the given 22 | function when the specified number of lines have been received. 23 | 24 | The callback gets passed the data (list of bytes objects), the connection 25 | and the address, so more data can be exchanged. 26 | """ 27 | TIMEOUT = 5.0 28 | LENGTH = 1024 29 | 30 | def __init__(self, client_lines, callback): 31 | self._callback = callback 32 | self._client_lines = client_lines 33 | 34 | # Choose a port 35 | self._server = None 36 | for port in irange(15550, 15580): 37 | server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 38 | address = ('127.0.0.1', port) 39 | try: 40 | server.bind(address) 41 | server.listen(5) 42 | except socket.error: 43 | pass 44 | else: 45 | logging.debug("Server created on %s:%d", *address) 46 | self._server = server 47 | self.port = port 48 | break 49 | if self._server is None: 50 | logging.critical("Couldn't find a TCP port to listen on") 51 | sys.exit(1) 52 | 53 | def run(self): 54 | clients = {} 55 | 56 | next_timeout = None 57 | now = time.time() 58 | 59 | while True: 60 | sockets = [self._server] 61 | sockets.extend(clients) 62 | timeout = (None if next_timeout is None 63 | else next_timeout - now + 0.2) 64 | rlist, _, _ = select.select(sockets, [], [], 65 | timeout) 66 | now = time.time() 67 | 68 | # Timeouts 69 | for sock, (data, timeout, addr) in list(iteritems(clients)): 70 | if now > timeout: 71 | del clients[sock] 72 | sock.send(b"timed out\nERROR\n") 73 | sock.close() 74 | logging.debug("Connection from %s timed out", 75 | addr) 76 | next_timeout = -1 77 | 78 | for sock in rlist: 79 | if sock == self._server: 80 | conn, addr = self._server.accept() 81 | logging.debug("Connection from %s", addr) 82 | timeout = now + self.TIMEOUT 83 | clients[conn] = [b''], timeout, addr 84 | if next_timeout is None: 85 | next_timeout = timeout 86 | else: 87 | next_timeout = min(next_timeout, timeout) 88 | else: 89 | data, timeout, addr = clients[conn] 90 | res = conn.recv(self.LENGTH - len(data[-1])) 91 | done = not res 92 | if res: 93 | end = res.find(b'\n') 94 | while end != -1: 95 | data[-1] += res[:end] 96 | if len(data) == self._client_lines: 97 | done = True 98 | break 99 | data.append(b'') 100 | res = res[end+1:] 101 | end = res.find(b'\n') 102 | else: 103 | data[-1] += res 104 | if done or len(data[-1]) >= self.LENGTH: 105 | del clients[conn] 106 | try: 107 | if len(data) == self._client_lines: 108 | self._callback(data, conn, addr) 109 | except Exception: 110 | conn.send(b"internal server error\nERROR\n") 111 | raise 112 | finally: 113 | conn.close() 114 | next_timeout = -1 115 | 116 | if next_timeout == -1: 117 | if clients: 118 | next_timeout = min(t 119 | for _, t, _ in itervalues(clients)) 120 | else: 121 | next_timeout = None 122 | -------------------------------------------------------------------------------- /gitobox/sync.py: -------------------------------------------------------------------------------- 1 | """Main application logic: class :class:`~gitobox.sync.Synchronizer`. 2 | """ 3 | 4 | from __future__ import unicode_literals 5 | 6 | import logging 7 | from threading import Semaphore, Thread 8 | 9 | from gitobox.git import GitRepository 10 | from gitobox.server import Server 11 | from gitobox.utils import unicode_, make_unique_bytestring 12 | from gitobox.watch import DirectoryWatcher 13 | 14 | 15 | class Synchronizer(object): 16 | """Main application logic: synchronizes folder with a Git repository. 17 | """ 18 | def __init__(self, folder, repository, branchname, timeout): 19 | # The global lock for synchronization operations 20 | self._lock = Semaphore(1) 21 | 22 | # Watches the directory for file changes 23 | self._watcher = DirectoryWatcher(folder, 24 | self._directory_changed, 25 | self._lock, 26 | timeout) 27 | 28 | # Make up a random password 29 | self.password = make_unique_bytestring() 30 | 31 | # Listens for connections from the Git hook 32 | self._hook_server = Server(2, self._hook_triggered) 33 | 34 | # Sets up the directory (installs the hook) 35 | self._repository = GitRepository(repository, folder, branchname, 36 | self.password, 37 | self._hook_server.port) 38 | 39 | def run(self): 40 | watcher_thread = Thread(target=self._watcher.run) 41 | watcher_thread.setDaemon(True) 42 | self._watcher.assume_all_changed() 43 | watcher_thread.start() 44 | 45 | try: 46 | self._hook_server.run() 47 | except KeyboardInterrupt: 48 | logging.warning("Got KeyboardInterrupt, exiting...") 49 | except Exception: 50 | logging.critical("Exiting after unhandled exception!") 51 | raise 52 | 53 | def _directory_changed(self, paths=None): 54 | # We got called back though the ResettableTimer, so the lock is held 55 | if paths is None: 56 | logging.warning("Assuming all paths changed") 57 | else: 58 | logging.warning("Paths changed: %s", 59 | " ".join(unicode_(p) for p in paths)) 60 | self._repository.check_in(paths) 61 | 62 | def _hook_triggered(self, data, conn, addr): 63 | passwd, ref = data 64 | if passwd != self.password: 65 | logging.debug("Got invalid message on hook server from %s", 66 | addr) 67 | conn.send(b"hook auth failed\nERROR\n") 68 | return 69 | logging.info("Hook triggered from %s", addr, ) 70 | if not self._lock.acquire(blocking=False): 71 | logging.info("Lock is held, failing...") 72 | conn.send(b"update is in progress, try again later\nERROR\n") 73 | else: 74 | try: 75 | conn.send(b"updating directory to " + ref[:7] + b"...\n") 76 | self._repository.check_out(ref) 77 | conn.send(b"synced directory updated!\n") 78 | logging.info("Directory updated to %s", 79 | ref.decode('ascii')[:7]) 80 | finally: 81 | self._lock.release() 82 | 83 | if self._repository.has_changes(ref): 84 | # The lock has been released, changes now happening in DropBox 85 | # will start DirectoryWatcher's timer as usual. 86 | # However, while we were copying files from Git into the 87 | # directory, it might have been changed (i.e. DropBox might 88 | # have conflicted). In this case, a new commit will happen in a 89 | # few seconds 90 | logging.info("Conflict detected during directory update") 91 | self._watcher.assume_all_changed() 92 | # Tell the pusher about it, so he can fetch 93 | conn.send(b"WARNING: DROPBOX CONFLICT\n" 94 | b"the directory was updated while changes were " 95 | b"being written from Git to the directory; " 96 | b"leave DropBox time to sync then fetch again\n") 97 | 98 | conn.send(b"OK\n") 99 | 100 | 101 | def synchronize(folder, repository, branchname, timeout): 102 | sync = Synchronizer(folder, repository, branchname, timeout) 103 | sync.run() 104 | -------------------------------------------------------------------------------- /gitobox/timer.py: -------------------------------------------------------------------------------- 1 | """Generic timer used to wait directory changes to stop. 2 | 3 | Contains :class:`~gitobox.timer.ResettableTimer`, a timer that waits a given 4 | amount of time after the *last* call to 5 | :meth:`~gitobox.timer.ResettableTimer.start()`. This means that every call to 6 | `start()` makes the timer restart. 7 | 8 | Also acquires a lock while ticking, allowing Gitobox to not accept hooks while 9 | waiting for the tree to be stable. 10 | """ 11 | 12 | from __future__ import unicode_literals 13 | 14 | from threading import Condition, Thread 15 | import traceback 16 | 17 | from gitobox.utils import irange 18 | 19 | 20 | class ResettableTimer(object): 21 | """Calls a function a specified number of seconds after the last start(). 22 | 23 | If a lock is passed to the constructor, it will be acquired when calling 24 | start(), returning False immediately if that's impossible. It will be 25 | released when the timer triggers or is canceled. 26 | """ 27 | IDLE, RESET, PRIMED = irange(3) 28 | 29 | def __init__(self, timeout, function, args=[], kwargs={}, lock=None): 30 | self.thread = Thread(target=self._run) 31 | self.thread.setDaemon(True) 32 | self.timeout = timeout 33 | 34 | self.function = function 35 | self.args = args 36 | self.kwargs = kwargs 37 | self.lock = lock 38 | self.executing = False 39 | 40 | # Only start the thread on the first start() 41 | self.started = False 42 | 43 | # This protects self.status and is used to wake up self._run() 44 | self.cond = Condition() 45 | self.status = ResettableTimer.IDLE 46 | # The lock is held if and only if status != IDLE 47 | 48 | def start(self): 49 | """Starts or restarts the countdown. 50 | 51 | If the Timer has an associated lock, this method returns False if it 52 | can't be acquired. 53 | """ 54 | with self.cond: 55 | if (self.status == ResettableTimer.IDLE and 56 | not self.executing and 57 | self.lock is not None and 58 | not self.lock.acquire(blocking=False)): 59 | return False 60 | if self.started: 61 | self.status = ResettableTimer.RESET 62 | self.cond.notifyAll() 63 | else: 64 | # Go to PRIMED directly, saves self._run() a loop 65 | self.status = ResettableTimer.PRIMED 66 | self.started = True 67 | self.thread.start() 68 | return True 69 | 70 | def cancel(self): 71 | """Cancels the countdown without calling back. 72 | """ 73 | with self.cond: 74 | if self.status != ResettableTimer.IDLE: 75 | self.status = ResettableTimer.IDLE 76 | if not self.executing: 77 | self.cond.notifyAll() 78 | if self.lock is not None: 79 | self.lock.release() 80 | 81 | def _run(self): 82 | with self.cond: 83 | while True: 84 | if self.status == ResettableTimer.PRIMED: 85 | self.cond.wait(self.timeout) 86 | else: 87 | self.cond.wait() 88 | 89 | # RESET: go to prime and start counting again 90 | if self.status == ResettableTimer.RESET: 91 | self.status = ResettableTimer.PRIMED 92 | # Still PRIMED: we timed out without interruption, call back 93 | elif self.status == ResettableTimer.PRIMED: 94 | self.status = ResettableTimer.IDLE 95 | self.executing = True 96 | try: 97 | self.function(*self.args, **self.kwargs) 98 | except Exception: 99 | traceback.print_exc() 100 | self.executing = False 101 | if self.lock is not None: 102 | self.lock.release() 103 | -------------------------------------------------------------------------------- /gitobox/utils.py: -------------------------------------------------------------------------------- 1 | """Various utilities, mainly related to PY2/PY3 compatibility. 2 | """ 3 | 4 | from __future__ import unicode_literals 5 | 6 | import random 7 | import sys 8 | 9 | 10 | PY3 = sys.version_info[0] == 3 11 | 12 | 13 | if PY3: 14 | unicode_ = str 15 | izip = zip 16 | irange = range 17 | iteritems = dict.items 18 | itervalues = dict.values 19 | listvalues = lambda d: list(d.values()) 20 | else: 21 | unicode_ = unicode 22 | import itertools 23 | izip = itertools.izip 24 | irange = xrange 25 | iteritems = dict.iteritems 26 | itervalues = dict.itervalues 27 | listvalues = dict.values 28 | 29 | 30 | def unique_bytestring_gen(): 31 | """Generates unique sequences of bytes. 32 | """ 33 | characters = (b"abcdefghijklmnopqrstuvwxyz" 34 | b"0123456789") 35 | characters = [characters[i:i + 1] for i in irange(len(characters))] 36 | rng = random.Random() 37 | while True: 38 | letters = [rng.choice(characters) for i in irange(10)] 39 | yield b''.join(letters) 40 | unique_bytestring_gen = unique_bytestring_gen() 41 | 42 | 43 | def make_unique_bytestring(): 44 | """Makes a unique (random) bytestring. 45 | """ 46 | return next(unique_bytestring_gen) 47 | -------------------------------------------------------------------------------- /gitobox/watch.py: -------------------------------------------------------------------------------- 1 | """Directory-watching logic. 2 | 3 | Contains :class:`~gitobox.watch.DirectoryWatcher`, the class that monitors the 4 | directory for changes. Uses `pyinotify`, so it's only available on Linux. 5 | """ 6 | 7 | from __future__ import unicode_literals 8 | 9 | import logging 10 | from watchdog.observers import Observer 11 | from watchdog.events import FileSystemEventHandler 12 | 13 | from gitobox.timer import ResettableTimer 14 | 15 | 16 | class DirectoryWatcher(FileSystemEventHandler): 17 | ALL_CHANGED = None 18 | 19 | def __init__(self, folder, callback, lock, timeout): 20 | self._callback = callback 21 | 22 | self.observer = Observer() 23 | 24 | self._folder = folder 25 | self._changes = set() 26 | self.observer.schedule(self, str(folder), recursive=True) 27 | 28 | self._timer = ResettableTimer(timeout, self._timer_expired, 29 | lock=lock) 30 | 31 | def assume_all_changed(self): 32 | self._changes.add(DirectoryWatcher.ALL_CHANGED) 33 | self._timer.start() 34 | 35 | def run(self): 36 | self.observer.start() 37 | 38 | def _timer_expired(self): 39 | changes = self._changes 40 | self._changes = set() 41 | logging.info("Directory stable, syncing...") 42 | if DirectoryWatcher.ALL_CHANGED in changes: 43 | self._callback() 44 | else: 45 | self._callback(changes) 46 | 47 | def on_moved(self, event): 48 | what = 'directory' if event.is_directory else 'file' 49 | logging.info("Moved %s: from %s to %s", what, event.src_path, 50 | event.dest_path) 51 | self._changes.add(event.src_path) 52 | self._changes.add(event.dest_path) 53 | self._timer.start() 54 | 55 | def on_created(self, event): 56 | what = 'directory' if event.is_directory else 'file' 57 | logging.info("Created %s: %s", what, event.src_path) 58 | self._changes.add(event.src_path) 59 | self._timer.start() 60 | 61 | def on_deleted(self, event): 62 | what = 'directory' if event.is_directory else 'file' 63 | logging.info("Deleted %s: %s", what, event.src_path) 64 | self._changes.add(event.src_path) 65 | self._timer.start() 66 | 67 | def on_modified(self, event): 68 | what = 'directory' if event.is_directory else 'file' 69 | logging.info("Modified %s: %s", what, event.src_path) 70 | self._changes.add(event.src_path) 71 | self._timer.start() 72 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [bdist_wheel] 2 | universal = 1 3 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import os 2 | from setuptools import setup 3 | import sys 4 | 5 | 6 | # pip workaround 7 | os.chdir(os.path.abspath(os.path.dirname(__file__))) 8 | 9 | 10 | with open('README.rst') as fp: 11 | description = fp.read() 12 | req = ['watchdog', 13 | 'rpaths>=0.7'] 14 | if sys.version_info < (2, 7): 15 | req.append('argparse') 16 | setup(name='gitobox', 17 | version='0.3', 18 | packages=['gitobox'], 19 | package_data={'gitobox': ['hooks/*']}, 20 | entry_points={'console_scripts': [ 21 | 'gitobox = gitobox.main:main']}, 22 | install_requires=req, 23 | description= 24 | "Synchronizes a directory with a Git repository; particularly " 25 | "useful to track \"dumb\" collaboration software like DropBox", 26 | author="Remi Rampin", 27 | author_email='remirampin@gmail.com', 28 | maintainer="Remi Rampin", 29 | maintainer_email='remirampin@gmail.com', 30 | url='https://github.com/remram44/gitobox', 31 | long_description=description, 32 | license='BSD', 33 | keywords=['git', 'dropbox', 'drive', 'gdrive', 'cloud', 'dumb', 'sync', 34 | 'synchronization', 'collaboration'], 35 | classifiers=[ 36 | 'Development Status :: 4 - Beta', 37 | 'Environment :: Console', 38 | 'Environment :: No Input/Output (Daemon)', 39 | 'Intended Audience :: Developers', 40 | 'Intended Audience :: Information Technology', 41 | 'Intended Audience :: System Administrators', 42 | 'License :: OSI Approved :: BSD License', 43 | 'Operating System :: POSIX :: Linux', 44 | 'Programming Language :: Python :: 2', 45 | 'Programming Language :: Python :: 3', 46 | 'Topic :: Communications :: File Sharing', 47 | 'Topic :: Internet', 48 | 'Topic :: Software Development :: Version Control']) 49 | --------------------------------------------------------------------------------