├── .gitignore ├── README.rst ├── image ├── .gitignore ├── http │ ├── find-volume.sh │ ├── install-vagga.sh │ ├── upgrade-vagga.sh │ ├── vagga-ssh.sh │ └── vagga.settings.yaml ├── setup.sh ├── vagga.json └── vagga.yaml ├── setup.py ├── tests ├── expose_ports │ └── vagga.yaml ├── mixins1 │ ├── dir │ │ ├── nested1.yaml │ │ └── nested_mixins.yaml │ ├── first_mixin.yaml │ └── vagga.yaml ├── test_expose.py └── test_mixins1.py ├── vagga.yaml └── vagga_box ├── __init__.py ├── __main__.py ├── arguments.py ├── config.py ├── id_rsa ├── main.py ├── runtime.py ├── settings.py ├── ssh_config ├── unison.py ├── unox.py └── virtualbox.py /.gitignore: -------------------------------------------------------------------------------- 1 | /.vagga 2 | /.pytest_cache 3 | __pycache__ 4 | *.egg-info 5 | /venv 6 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | =================== 2 | Vagga in VirtualBox 3 | =================== 4 | 5 | :Status: PoC 6 | 7 | This is a prototype which brings vagga as the first-class tool to OS X and 8 | (possibly) windows using virtualbox 9 | 10 | 11 | Installation 12 | ============ 13 | 14 | First `download and install virtualbox`__. The project is tested on 15 | VirtualBox 5.1 but may work on earlier versions too. 16 | 17 | Then run the following commands (assuming you have brew_ installed):: 18 | 19 | $ brew install python3 unison wget 20 | $ pip3 install git+http://github.com/tailhook/vagga-box 21 | [ .. snip .. ] 22 | $ vagga 23 | Available commands: 24 | run 25 | 26 | Effectively it requires python >= 3.5 and unison 2.48.4 (unison is very picky 27 | on version numbers) 28 | 29 | __ https://www.virtualbox.org/wiki/Downloads 30 | .. _brew: http://brew.sh 31 | 32 | IDE support is enabled by the following command (and requires sudo access):: 33 | 34 | $ vagga _box mount 35 | Running sudo mount -t nfs -o vers=4,resvport,port=7049 127.0.0.1:/vagga /Users/myuser/.vagga/remote 36 | Password: 37 | Now you can add ~/.vagga/remote//.vagga//dir 38 | to the search paths of your IDE 39 | 40 | You need to run it each time your machine is rebooted, or if you restarted your 41 | virtualbox manually. 42 | 43 | 44 | Upgrading 45 | ========= 46 | 47 | Once you have installed vagga-box you can upgrade vagga inside the container 48 | using the following command-line:: 49 | 50 | vagga _box upgrade_vagga 51 | 52 | Changing the disk size 53 | ====================== 54 | 55 | By default, the disk size in Virtualbox is set to 20 GB. If necessary, it can be increased by the following steps: 56 | 57 | 1. Change VM's image size:: 58 | 59 | $ vagga _box down 60 | $ VBoxManage modifyhd ~/.vagga/vm/storage.vdi --resize 40860 61 | $ vagga _box up 62 | 63 | 2. Change partition size inside VM:: 64 | 65 | $ vagga _box ssh 66 | $ sudo apk add cfdisk e2fsprogs-extra 67 | $ sudo cfdisk /dev/sdb 68 | [ .. Delete /dev/sdb1 ..] 69 | [ .. New partition / default size / Primary .. ] 70 | [ .. Write changes / Quit .. ] 71 | $ sudo reboot 72 | 73 | 3. Final steps:: 74 | 75 | $ vagga _box ssh 76 | $ sudo resize2fs /dev/sdb1 77 | $ sudo df -h # Check size 78 | 79 | Short FAQ 80 | ========= 81 | 82 | **Why is it in python?** For a quick prototype. It will be integrated into 83 | vagga as soon as is proven to be useful. Or may be we leave it in python if 84 | it would be easier to install. 85 | 86 | **So should I try this version or wait it integrated in vagga?** Definitely you 87 | should try. The integrated version will work the same. 88 | 89 | **Is there any difference between this and vagga on linux?** There are two key 90 | differences: 91 | 92 | * you need to export ports that you want to be accessible from the 93 | host system 94 | * we keep all the container files (and a copy of the project) in the virtualbox 95 | * to view it from the host system mount nfs volume (``vagga _box mount``) 96 | * to make filesync fast you can add some dirs to the ignore list 97 | (``_ignore-dirs`` setting) 98 | 99 | .. code-block:: yaml 100 | 101 | _ignore-dirs: 102 | - .git 103 | - tmp 104 | - data 105 | 106 | containers: 107 | django: 108 | setup: 109 | - !Alpine v3.3 110 | - !Py3Install ['Django >=1.9,<1.10'] 111 | 112 | commands: 113 | run: !Command 114 | description: Start the django development server 115 | container: django 116 | _expose-ports: [8080] 117 | run: python3 manage.py runserver 118 | 119 | **Please report if you find any other differences using the tool**. Ah, but 120 | exact text of some error messages may differ, don't be too picky :) 121 | 122 | **Why `_expose-ports` are underscored?** This is a standard 123 | way to add extension metadata or user-defined things in vagga.yaml. We will 124 | remove the underscore as soon as integrate it into main code. Fixing 125 | underscores isn't going to be a big deal. 126 | 127 | **Will linux users add `_expose-ports` for me?** Frankly, 128 | currently probably now. But it's small change that probably no one will need 129 | to delete. In the future we want to apply ``seccomp`` filters to allow to bind 130 | only exposed ports on linux too. 131 | 132 | **What will be changed when we integrate this into vagga?** We will move more 133 | operations from virtualbox into host system. For example list of commands will 134 | be executed by mac os. Also ``vagga _list``, some parts of ``vagga _clean`` and 135 | so on. But we will do our best to keep semantics exactly the same. 136 | 137 | 138 | LICENSE 139 | ======= 140 | 141 | This project has been placed into the public domain. 142 | -------------------------------------------------------------------------------- /image/.gitignore: -------------------------------------------------------------------------------- 1 | /output-virtualbox-iso 2 | /packer_cache 3 | /http/unison 4 | /http/unison-fsmonitor 5 | -------------------------------------------------------------------------------- /image/http/find-volume.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | name=${VAGGA_PROJECT_NAME} 3 | base=/vagga 4 | 5 | if [ ! -e $base/$name ]; then 6 | mkdir $base/$name 7 | echo $name 8 | exit 0 9 | fi 10 | for i in $(seq 100); do 11 | dir=$base/$name-$i 12 | if [ ! -e $dir ]; then 13 | mkdir $dir 14 | echo $name-$i 15 | exit 0 16 | fi 17 | done 18 | echo Too many directories named '"'"$name"'"' >> /dev/stderr 19 | exit 1 20 | -------------------------------------------------------------------------------- /image/http/install-vagga.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | curl -sfS http://files.zerogw.com/vagga/vagga-install.sh | sh 3 | -------------------------------------------------------------------------------- /image/http/upgrade-vagga.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | curl -sfS http://files.zerogw.com/vagga/vagga-install-testing.sh | sh 3 | -------------------------------------------------------------------------------- /image/http/vagga-ssh.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh -e 2 | test -n "$VAGGA_RESOLV_CONF" 3 | test -n "$VAGGA_VOLUME" 4 | if [ "$(cat /etc/resolv.conf)" != "$VAGGA_RESOLV_CONF" ]; then 5 | echo "$VAGGA_RESOLV_CONF" | sudo tee /etc/resolv.conf > /dev/null 6 | fi 7 | cd "/vagga/$VAGGA_VOLUME" 8 | exec vagga "$@" 9 | -------------------------------------------------------------------------------- /image/http/vagga.settings.yaml: -------------------------------------------------------------------------------- 1 | cache-dir: /vagga/.cache 2 | -------------------------------------------------------------------------------- /image/setup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh -ex 2 | HTTP_PREFIX="$(cat /tmp/http)" 3 | export no_proxy="$(cat /tmp/no_proxy)" 4 | 5 | mkdir /home/user/.ssh 6 | echo ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4s++fCkUFUVJAGWv5St/V5CFsga0ElLxYtQGKEHy2HfPom8Im+PM28q3d8NBCXt7GDRLjQg8K/vBMge8VBJ68N76B0WDG9A/Nx6HID7LASOUAAig+YgnkJBQm8rTo3yqlKkDMx65OqtC09bG9FrsIpgDTaEt+mCl+lkvk7fkORZw77kNMx6W768cMXSEvaV6f3BfgAUVw6PjrUh4EPtnbvIRSFw9BLPS8LJHTW8zY2ctrn5rCoDLmtozn0FmTTi9h3Px+OIwgTx4k+PGLCBYich6VVSD2KyWPcM13feI5BjVc2yNSoWpYm7klsTMMANUjIqiR9rlNe/6esVS0bowl vagga insecure public key > /home/user/.ssh/authorized_keys 7 | chown -R user /home/user/.ssh 8 | chmod -R go-rwsx /home/user/.ssh 9 | chmod 0755 /home/user/.ssh 10 | chmod 0644 /home/user/.ssh/authorized_keys 11 | 12 | curl -sfS $HTTP_PREFIX/unison > /bin/unison 13 | chmod +x /bin/unison 14 | curl -sfS $HTTP_PREFIX/unison-fsmonitor > /bin/unison-fsmonitor 15 | chmod +x /bin/unison-fsmonitor 16 | 17 | apk add nfs-utils 18 | mkdir /vagga 19 | mkfs.ext4 /dev/sdb1 20 | echo "/dev/sdb1 /vagga ext4 rw,data=ordered,noatime,discard 0 2" >> /etc/fstab 21 | mount /vagga 22 | mkdir /vagga/.unison /vagga/.cache 23 | chown user /vagga /vagga/.unison /vagga/.cache 24 | curl -sfS $HTTP_PREFIX/vagga.settings.yaml > /home/user/.vagga.yaml 25 | curl -sfS $HTTP_PREFIX/vagga-ssh.sh > /usr/local/bin/vagga-ssh.sh 26 | chmod +x /usr/local/bin/vagga-ssh.sh 27 | curl -sfS $HTTP_PREFIX/find-volume.sh > /usr/local/bin/find-volume.sh 28 | chmod +x /usr/local/bin/find-volume.sh 29 | 30 | # install vagga stable version 31 | curl -sfS $HTTP_PREFIX/install-vagga.sh > /usr/local/bin/install-vagga 32 | chmod +x /usr/local/bin/install-vagga 33 | /usr/local/bin/install-vagga 34 | 35 | # but allow to upgrade to latest testing 36 | curl -sfS $HTTP_PREFIX/upgrade-vagga.sh > /usr/local/bin/upgrade-vagga 37 | chmod +x /usr/local/bin/upgrade-vagga 38 | 39 | cat <> /etc/exports 40 | /vagga *(rw,sync,no_subtree_check,all_squash,anonuid=1000,anongid=1000) 41 | NFS 42 | rc-update add nfs 43 | rc-update add nfsmount 44 | rc-update add netmount 45 | rc-update add ntpd 46 | 47 | apk add virtualbox-guest-additions --update-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/community/ 48 | apk add shadow-uidmap --update-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/community/ 49 | 50 | cat < /etc/sysctl.d/01-vagga.conf 51 | fs.inotify.max_user_watches=131072 52 | SYSCTL 53 | cat < /etc/subuid 54 | user:100000:165536 55 | SUBUID 56 | cat < /etc/subgid 57 | user:100000:165536 58 | SUBGID 59 | cat <> /etc/ssh/sshd_config 60 | PermitUserEnvironment yes 61 | AcceptEnv VAGGA_* 62 | SSHCONFIG 63 | 64 | rm -rf /var/run/* 65 | rm /etc/profile.d/proxy.sh 66 | -------------------------------------------------------------------------------- /image/vagga.json: -------------------------------------------------------------------------------- 1 | { 2 | "variables": { 3 | "http_proxy": "{{env `http_proxy`}}" 4 | }, 5 | "builders": [{ 6 | "type": "virtualbox-iso", 7 | "virtualbox_version_file": ".vbox_version", 8 | "headless": false, 9 | 10 | "guest_os_type": "Ubuntu_64", 11 | "disk_size": 1024, 12 | 13 | "iso_url": "http://dl-cdn.alpinelinux.org/alpine/v3.4/releases/x86_64/alpine-vanilla-3.4.3-x86_64.iso", 14 | "iso_checksum": "e2ac2c35da8c277d0acdd28880e6e006e91274fd", 15 | "iso_checksum_type": "sha1", 16 | 17 | "boot_wait": "40s", 18 | "boot_command": [ 19 | "root", 20 | "nc -lp 22&", 21 | "ifconfig eth0 up && udhcpc -i eth0", 22 | "export http_proxy={{user `http_proxy`}}", 23 | "setup-alpine", 24 | "us", 25 | "us", 26 | "vagga", 27 | "doneno", 28 | "vagga", 29 | "vagga", 30 | "UTC", 31 | "1", 32 | "openssh", 33 | "openntpd", 34 | "sda", 35 | "sys", 36 | "y", 37 | "echo ,,L | sfdisk /dev/sdb", 38 | "killall -9 nc", 39 | "service sshd stop", 40 | "mount /dev/sda3 /mnt", 41 | "mount -o bind /dev /mnt/dev", 42 | "mount -t tmpfs -o size=100m tmpfs /mnt/tmp", 43 | "mount -t tmpfs -o size=100m tmpfs /mnt/run", 44 | "chroot /mnt", 45 | "ln -sf /bin/ash /bin/bash", 46 | "mount -t proc none /proc", 47 | "mount -t devpts devpts /dev/pts", 48 | "apk add -U sudo curl bash", 49 | "adduser uservaggavagga", 50 | "killall -9 udhcpc", 51 | "ifconfig eth0 down", 52 | "setup-interfaces", 53 | "eth0", 54 | "dhcp", 55 | "doneno", 56 | "echo 'user ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers", 57 | "/usr/sbin/sshd", 58 | "ifup eth0", 59 | "echo http://{{ .HTTPIP }}:{{ .HTTPPort }}/ > /tmp/http", 60 | "echo {{ .HTTPIP }} > /tmp/no_proxy" 61 | ], 62 | 63 | "http_directory": "http", 64 | 65 | "ssh_username": "user", 66 | "ssh_password": "vagga", 67 | "ssh_port": 22, 68 | "ssh_wait_timeout": "10m", 69 | 70 | "vboxmanage": [ 71 | ["modifyvm", "{{.Name}}", "--audio", "none"], 72 | ["modifyvm", "{{.Name}}", "--memory", "2048"], 73 | ["modifyvm", "{{.Name}}", "--cpus", "2"], 74 | ["modifyvm", "{{.Name}}", "--natpf1", "guestssh,tcp,,7022,,22"], 75 | ["createhd", "--format", "VDI", "--filename", "storage.vdi", 76 | "--size", "20480"], 77 | ["storagectl", "{{.Name}}", 78 | "--name", "SATA Controller", "--add", "sata"], 79 | ["storageattach", "{{.Name}}", 80 | "--storagectl", "SATA Controller", 81 | "--device", "0", 82 | "--port", "1", 83 | "--type", "hdd", 84 | "--medium", "storage.vdi"] 85 | ], 86 | 87 | "shutdown_command": "sudo poweroff" 88 | }], 89 | "provisioners": [{ 90 | "type": "shell", 91 | "execute_command": "sudo bash -c 'export http_proxy={{user `http_proxy`}}; {{.Path}}'", 92 | "scripts": [ 93 | "setup.sh" 94 | ] 95 | }] 96 | } 97 | -------------------------------------------------------------------------------- /image/vagga.yaml: -------------------------------------------------------------------------------- 1 | _ignore-dirs: 2 | - output-virtualbox-iso 3 | - packer_cache 4 | 5 | containers: 6 | unison: 7 | setup: 8 | - !Ubuntu xenial 9 | - !UbuntuUniverse 10 | - !Install [ocaml, inotify-tools, curl, ca-certificates, build-essential] 11 | 12 | commands: 13 | build: !Command 14 | environ: 15 | # Note this version must match the one in brew 16 | UNISON_VERSION: 2.48.4 17 | HOME: /root 18 | container: unison 19 | run: | 20 | set -ex 21 | [ -d /work/http ] || mkdir /work/http 22 | cd /tmp 23 | curl -O http://www.seas.upenn.edu/~bcpierce/unison/download/releases/unison-$UNISON_VERSION/unison-$UNISON_VERSION.tar.gz 24 | tar -xzf unison-$UNISON_VERSION.tar.gz 25 | cd src 26 | make GLIBC_SUPPORT_INOTIFY=true UISTYLE=text NATIVE=true STATIC=true unison unison-fsmonitor 27 | cp unison unison-fsmonitor /work/http 28 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | from setuptools import setup 4 | 5 | setup(name='vagga-box', 6 | version='0.1', 7 | description='A wrapper to run vagga in virtualbox (easier on osx)', 8 | author='Paul Colomiets', 9 | author_email='paul@colomiets.name', 10 | url='http://github.com/tailhook/vagga-box', 11 | packages=['vagga_box'], 12 | install_requires=[ 13 | 'PyYaml', 14 | 'macfsevents', 15 | ], 16 | package_data={'vagga_box': ['id_rsa', 'ssh_config']}, 17 | entry_points = { 18 | 'console_scripts': [ 19 | 'vagga=vagga_box.main:main', 20 | 'unison-fsmonitor=vagga_box.unox:main', 21 | ], 22 | }, 23 | classifiers=[ 24 | 'Development Status :: 4 - Beta', 25 | 'Programming Language :: Python :: 3', 26 | 'Programming Language :: Python :: 3.4', 27 | ], 28 | ) 29 | -------------------------------------------------------------------------------- /tests/expose_ports/vagga.yaml: -------------------------------------------------------------------------------- 1 | commands: 2 | run: !Command 3 | _expose-ports: [10, 20] 4 | 5 | super: !Supervise 6 | _expose-ports: [110] 7 | children: 8 | one: !Supervise 9 | _expose-ports: [210, 220] 10 | two: !Supervise 11 | _expose-ports: [230] 12 | -------------------------------------------------------------------------------- /tests/mixins1/dir/nested1.yaml: -------------------------------------------------------------------------------- 1 | containers: 2 | nested1: 3 | -------------------------------------------------------------------------------- /tests/mixins1/dir/nested_mixins.yaml: -------------------------------------------------------------------------------- 1 | containers: 2 | nested: 3 | mixins: 4 | - nested1.yaml 5 | -------------------------------------------------------------------------------- /tests/mixins1/first_mixin.yaml: -------------------------------------------------------------------------------- 1 | containers: 2 | first_mixin: 3 | -------------------------------------------------------------------------------- /tests/mixins1/vagga.yaml: -------------------------------------------------------------------------------- 1 | mixins: 2 | - first_mixin.yaml 3 | - dir/nested_mixins.yaml 4 | 5 | commands: 6 | root: !Command 7 | container: x 8 | run: y 9 | -------------------------------------------------------------------------------- /tests/test_expose.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | from vagga_box.config import get_config 4 | from vagga_box.runtime import Vagga 5 | from vagga_box.arguments import parse_args 6 | 7 | 8 | def test_read_mixins(): 9 | cwd = os.getcwd() 10 | try: 11 | os.chdir('tests/expose_ports') 12 | path, cfg, _ = get_config() 13 | 14 | vagga = Vagga(path, cfg, parse_args(['run'])) 15 | print("VAGGA", vagga.run_commands) 16 | assert vagga.exposed_ports() \ 17 | == frozenset([10, 20]) 18 | 19 | vagga = Vagga(path, cfg, parse_args(['super'])) 20 | assert vagga.exposed_ports() \ 21 | == frozenset([110, 210, 220, 230]) 22 | finally: 23 | os.chdir(cwd) 24 | -------------------------------------------------------------------------------- /tests/test_mixins1.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | from vagga_box.config import get_config 4 | 5 | 6 | def test_read_mixins(): 7 | cwd = os.getcwd() 8 | try: 9 | os.chdir('tests/mixins1') 10 | assert get_config()[1] == { 11 | 'containers': { 12 | 'first_mixin': None, 13 | 'nested': None, 14 | 'nested1': None, 15 | }, 16 | 'commands': { 17 | 'root': {'container': 'x', 'run': 'y'}, 18 | }, 19 | 'mixins': ['first_mixin.yaml', 'dir/nested_mixins.yaml'], 20 | } 21 | finally: 22 | os.chdir(cwd) 23 | -------------------------------------------------------------------------------- /vagga.yaml: -------------------------------------------------------------------------------- 1 | containers: 2 | test: 3 | setup: 4 | - !Alpine v3.7 5 | - !PipConfig { dependencies: true } 6 | - !Py3Install [PyYaml, pytest] 7 | 8 | commands: 9 | test: !Command 10 | container: test 11 | environ: 12 | PYTHONPATH: /work 13 | run: [pytest] 14 | -------------------------------------------------------------------------------- /vagga_box/__init__.py: -------------------------------------------------------------------------------- 1 | import os 2 | import pathlib 3 | 4 | BASE = pathlib.Path().home() / '.vagga' 5 | KEY_PATH = BASE / 'id_rsa' 6 | BASE_SSH_COMMAND = [ 7 | 'ssh', 8 | '-i', str(KEY_PATH), 9 | '-F', os.path.join(os.path.dirname(__file__), 'ssh_config'), 10 | 'user@127.0.0.1', 11 | ] 12 | # checking stdout as it's the most useful thing to redirect 13 | BASE_SSH_TTY_COMMAND = BASE_SSH_COMMAND + (['-t'] if os.isatty(1) else []) 14 | BASE_SSH_COMMAND_QUIET = BASE_SSH_COMMAND + ['-o LogLevel=QUIET'] 15 | -------------------------------------------------------------------------------- /vagga_box/__main__.py: -------------------------------------------------------------------------------- 1 | if __name__ == '__main__': 2 | from .main import main 3 | main() 4 | -------------------------------------------------------------------------------- /vagga_box/arguments.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | 3 | 4 | def parse_args(args=None): 5 | ap = argparse.ArgumentParser( 6 | usage="vagga [options] COMMAND [ARGS...]", 7 | description=""" 8 | Runs a command in container, optionally builds container if that 9 | does not exists or outdated. Run `vagga` without arguments to see 10 | the list of commands. 11 | """) 12 | ap.add_argument("command", nargs=argparse.REMAINDER, 13 | help="A vagga command to run") 14 | ap.add_argument("-V", "--version", action='store_true', 15 | help="Show vagga version and exit") 16 | ap.add_argument("-E", "--env", "--environ", metavar="NAME=VALUE", 17 | help="Set environment variable for running command") 18 | ap.add_argument("-e", "--use-env", metavar="VAR", 19 | help="Propagate variable VAR into command environment") 20 | ap.add_argument("--ignore-owner-check", action="store_true", 21 | help="Ignore checking owner of the project directory") 22 | ap.add_argument("--no-prerequisites", action="store_true", 23 | help="Run only specified command(s), don't run prerequisites") 24 | ap.add_argument("--no-image-download", action="store_true", 25 | help="Do not download container image from image index.") 26 | ap.add_argument("--no-build", action="store_true", 27 | help="Do not build container even if it is out of date. \ 28 | Return error code 29 if it's out of date.") 29 | ap.add_argument("--no-version-check", action="store_true", 30 | help="Do not run versioning code, just pick whatever \ 31 | container version with the name was run last (or \ 32 | actually whatever is symlinked under \ 33 | `.vagga/container_name`). Implies `--no-build`") 34 | ap.add_argument("-m", "--run-multi", nargs="*", 35 | help="Run the following list of commands. Each without an \ 36 | arguments. When any of them fails, stop the chain. \ 37 | Basically it's the shortcut to `vagga cmd1 && vagga \ 38 | cmd2` except containers for `cmd2` are built \ 39 | beforehand, for your convenience. Also builtin commands \ 40 | (those starting with underscore) do not work with \ 41 | `vagga -m`") 42 | return ap.parse_args(args=args) 43 | -------------------------------------------------------------------------------- /vagga_box/config.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import pathlib 4 | import yaml 5 | 6 | 7 | LOCAL_MIXINS = ( 8 | 'vagga.local.yaml', 9 | '.vagga.local.yaml', 10 | '.vagga/local.yaml', 11 | ) 12 | 13 | 14 | class FancyLoader(yaml.Loader): 15 | pass 16 | 17 | 18 | def generic_object(loader, suffix, node): 19 | if isinstance(node, yaml.ScalarNode): 20 | constructor = loader.__class__.construct_scalar 21 | elif isinstance(node, yaml.SequenceNode): 22 | constructor = loader.__class__.construct_sequence 23 | elif isinstance(node, yaml.MappingNode): 24 | constructor = loader.__class__.construct_mapping 25 | else: 26 | raise ValueError(node) 27 | # TODO(tailhook) wrap into some object? 28 | return constructor(loader, node) 29 | 30 | 31 | yaml.add_multi_constructor('!', generic_object, Loader=FancyLoader) 32 | 33 | def load(f): 34 | return yaml.load(f, Loader=FancyLoader) 35 | 36 | 37 | def find_config(): 38 | path = pathlib.Path(os.getcwd()) 39 | suffix = pathlib.Path("") 40 | while str(path) != path.root: 41 | vagga = path / 'vagga.yaml' 42 | if vagga.exists(): 43 | return path, vagga, suffix 44 | for fname in LOCAL_MIXINS: 45 | vagga = path / fname 46 | if vagga.exists(): 47 | return path, vagga, suffix 48 | suffix = path.parts[-1] / suffix 49 | path = path.parent 50 | raise RuntimeError("No vagga.yaml found in path {!r}".format(path)) 51 | 52 | 53 | def read_mixins(base_filename, mixin_list, dest): 54 | for subpath in mixin_list: 55 | filename = base_filename.parent / subpath 56 | try: 57 | with filename.open('rb') as file: 58 | data = load(file) 59 | except Exception as e: 60 | print("Error reading mixin {}: {}".format(filename, e), 61 | file=sys.stderr) 62 | else: 63 | for name, val in data.get('containers', {}).items(): 64 | if name not in dest['containers']: 65 | dest['containers'][name] = val 66 | for name, val in data.get('commands', {}).items(): 67 | if name not in dest['commands']: 68 | dest['commands'][name] = val 69 | read_mixins(filename, data.get('mixins', ()), dest) 70 | 71 | 72 | def get_config(): 73 | dir, vagga, suffix = find_config() 74 | with vagga.open('rb') as file: 75 | data = load(file) 76 | mix = {'containers': {}, 'commands': {}} 77 | read_mixins(vagga, data.get('mixins', ()), mix) 78 | mix['containers'].update(data.get('containers', {})) 79 | data['containers'] = mix['containers'] 80 | mix['commands'].update(data.get('commands', {})) 81 | data['commands'] = mix['commands'] 82 | 83 | local_mixin_list = [ 84 | fname for fname in LOCAL_MIXINS if (dir / fname).exists() 85 | ] 86 | if local_mixin_list: 87 | local_mix = {'containers': {}, 'commands': {}} 88 | read_mixins(vagga, local_mixin_list[::-1], local_mix) 89 | data['containers'].update(local_mix['containers']) 90 | data['commands'].update(local_mix['commands']) 91 | 92 | return dir, data, suffix 93 | -------------------------------------------------------------------------------- /vagga_box/id_rsa: -------------------------------------------------------------------------------- 1 | -----BEGIN RSA PRIVATE KEY----- 2 | MIIEpAIBAAKCAQEAuLPvnwpFBVFSQBlr+Urf1eQhbIGtBJS8WLUBihB8th3z6JvC 3 | JvjzNvKt3fDQQl7exg0S40IPCv7wTIHvFQSevDe+gdFgxvQPzcehyA+ywEjlAAIo 4 | PmIJ5CQUJvK06N8qpSpAzMeuTqrQtPWxvRa7CKYA02hLfpgpfpZL5O35DkWcO+5D 5 | TMelu+vHDF0hL2len9wX4AFFcOj461IeBD7Z27yEUhcPQSz0vCyR01vM2NnLa5+a 6 | wqAy5raM59BZk04vYdz8fjiMIE8eJPjxiwgWInIelVUg9islj3DNd33iOQY1XNsj 7 | UqFqWJu5JbEzDADVIyKokfa5TXv+nrFUtG6MJQIDAQABAoIBAQCKvfcYY3F3Th/X 8 | sIDv8TN0ivokdMBPuZ5FkCoI2NulPZizORU9izG/K8o49iSqRnNXGAkBFuUP4HMH 9 | NW8vPZozTjhXcb0dlcWtUPEQw+IFGHyUZgpu2dwOOJ++pgAJEWIKUVP9v10LELrX 10 | w9twa3uxQmlKZISZIpnA0gtOewfjlFPHm/uolUzOGLxJxLKMzon+JfoRhKyDT3wX 11 | Oh3S9Fqh9uatykyGP5L7JZpECGkuCBB7pZhk01ZKg1OO2zFkL8kYh7yURLL9fRyp 12 | XNfA5sN97sn29yLmPgtPlxmWE8hyyMmTixKdUmObQG4u0vdfGIl2/6Gem9LvYXx7 13 | +kJ9V7RhAoGBAPNX9l/QrGcK2kqqFQiLOlQI32P1+4QMEUOzxKCtef2uJaA4f2ZP 14 | O8sP637KODqdk6CSJ9iZnBHSkFcGAXzIe1KFrgwcA02JNFpWCOYHsQmnWdJ9EHW4 15 | XVV3wvrITVQybS7+6jvpn5B6P/d7Vsaj3Z3FMX3v/bMkWYAZMfc9MYUJAoGBAMJP 16 | MTQkVn++c2Xv8fgXO3M2x1OynacaEweO6i1uuFm+zpuQbV18zqyIJXGRDCoAfBXf 17 | mTIHlznuYd3c3S1jVCH5YMmUbhDKjf66rNT64mq7CPJjPSJxfn3VJ/Qi3wCyhRiL 18 | ixVeDOgybKj5P+rGtWlShvMSme49H0AAnlRsulE9AoGAMlHOMKIGBIjJ+waQsuOX 19 | fCkZiKIlEHkuWMGjt1YoE70fKrKEJbPcuXDhUaafWf+bt2iBtNiO3WCdWGF1jUgn 20 | uDjMdNSWGkJ1APkpfee5RDXG7S/PZ4hoRHQvbYxd8Ts8OKud5CW7STT+ZT6sAwd/ 21 | nFBam6A05gZuO376RhXxV4ECgYAH+bwpSbyLLcQK0Rh7eGimR/9SfihebYGOc91E 22 | 1CCY/m34kKsMhUzuZAA1cyCusKpnM3BUT19zr0cxFhm1/Te81UGVxJPn+IhvhjYF 23 | 3+5fNyIc4NnnigUJITCsoqnIm4s/AKtKyzt4ZGl1XhWzi0hy9EI5w++xiay4sc3N 24 | 5VCYeQKBgQDGAgoUi1YRs0cYa/ZyGr7D62Qi7bwaCP0qBFlINB0LUIGdqU12W0eP 25 | avHLr6UguDRxHIPAOOpZQrilKJpPi+hMOeNb8gc6CKTPa3dtUcNqgKo6P31pqBai 26 | sn7wkP04dXeikO3uE2D22hA/cnV/yIV2iTpeCVTanequYP7FkX8a/A== 27 | -----END RSA PRIVATE KEY----- 28 | -------------------------------------------------------------------------------- /vagga_box/main.py: -------------------------------------------------------------------------------- 1 | import os 2 | import re 3 | import sys 4 | import json 5 | import time 6 | import shlex 7 | import shutil 8 | import logging 9 | import subprocess 10 | 11 | from . import BASE, KEY_PATH, BASE_SSH_COMMAND, BASE_SSH_COMMAND_QUIET 12 | from . import BASE_SSH_TTY_COMMAND 13 | from . import config 14 | from . import virtualbox 15 | from . import runtime 16 | from . import arguments 17 | from . import settings 18 | from . import unison 19 | 20 | 21 | log = logging.getLogger(__name__) 22 | KEY_SOURCE = os.path.join(os.path.dirname(__file__), 'id_rsa') 23 | VOLUME_RE = re.compile('^[a-zA-Z0-9_-]+$') 24 | DEFAULT_RESOLV_CONF = """ 25 | # No resolv.conf could be read from the host system 26 | nameserver 8.8.8.8 27 | nameserver 8.8.4.4 28 | """ 29 | 30 | def check_key(): 31 | if not KEY_PATH.exists(): 32 | if not BASE.exists(): 33 | BASE.mkdir() 34 | tmp = KEY_PATH.with_suffix('.tmp') 35 | shutil.copy(str(KEY_SOURCE), str(tmp)) 36 | os.chmod(str(tmp), 0o600) 37 | os.rename(str(tmp), str(KEY_PATH)) 38 | 39 | 40 | def ide_hint(): 41 | if os.path.exists('.vagga/.virtualbox-volume'): 42 | with open('.vagga/.virtualbox-volume') as f: 43 | volume = f.read().strip() 44 | print("Now you can add " 45 | "~/.vagga/remote/"+volume+ 46 | "/.vagga//dir " 47 | "to the search paths of your IDE") 48 | else: 49 | print("Now you can add " 50 | "~/.vagga/remote/" 51 | "/.vagga//dir " 52 | "to the search paths of your IDE") 53 | print(" will be in " 54 | "`.vagga/.virtualbox-volume` " 55 | "after you run vagga command for the first time") 56 | 57 | 58 | def find_volume(vagga): 59 | vol_file = vagga.vagga_dir / '.virtualbox-volume' 60 | if vol_file.exists(): 61 | name = vol_file.open('rt').read().strip() 62 | if VOLUME_RE.match(name): 63 | return name 64 | basename = vagga.base.stem 65 | if not VOLUME_RE.match(basename): 66 | basename = 'unknown' 67 | name = subprocess.check_output(BASE_SSH_COMMAND + 68 | ['/usr/local/bin/find-volume.sh'], env={ 69 | 'VAGGA_PROJECT_NAME': basename, 70 | }).decode('ascii').strip() 71 | if not VOLUME_RE.match(name): 72 | raise RuntimeError("Command returned bad volume name {!r}" 73 | .format(name)) 74 | with vol_file.open('w') as f: 75 | f.write(name) 76 | return name 77 | 78 | 79 | def read_resolv_conf(): 80 | try: 81 | with open('/etc/resolv.conf', 'rt') as f: 82 | data = f.read() 83 | except OSError as e: 84 | print("Warning: Error reading /etc/resolv.conf:", e, file=sys.stderr) 85 | data = None 86 | if not data: 87 | data = DEFAULT_RESOLV_CONF 88 | return data 89 | 90 | 91 | def get_vagga_version(): 92 | (stdout, stderr) = subprocess.Popen( 93 | BASE_SSH_COMMAND_QUIET + ['vagga --version'], stdout=subprocess.PIPE, 94 | ).communicate() 95 | return stdout.decode('utf-8').rstrip() 96 | 97 | 98 | def main(): 99 | 100 | logging.basicConfig( 101 | # TODO(tailhook) should we use RUST_LOG like in original vagga? 102 | level=os.environ.get('VAGGA_LOG', 'WARNING')) 103 | 104 | args = arguments.parse_args() 105 | check_key() 106 | 107 | if args.command[0:1] == ['_box']: 108 | # use real argparse here 109 | if args.command[1:2] == ['ssh']: 110 | returncode = subprocess.Popen( 111 | BASE_SSH_TTY_COMMAND + args.command[2:], 112 | ).wait() 113 | return sys.exit(returncode) 114 | elif args.command[1:2] == ['up']: 115 | virtualbox.init_vm(new_storage_callback=unison.clean_local_dir) 116 | return sys.exit(0) 117 | elif args.command[1:2] == ['down']: 118 | virtualbox.stop_vm() 119 | return sys.exit(0) 120 | elif args.command[1:2] == ['upgrade_vagga']: 121 | print('Starting upgrade vagga. Current version:', 122 | get_vagga_version()) 123 | returncode = subprocess.Popen( 124 | BASE_SSH_COMMAND_QUIET + ['/usr/local/bin/upgrade-vagga'], 125 | ).wait() 126 | if returncode == 0: 127 | print('Vagga successfully upgraded to', get_vagga_version()) 128 | print('All OK!') 129 | else: 130 | print('Failed to upgrade vagga. Exit with status:', returncode, 131 | file=sys.stderr) 132 | return sys.exit(returncode) 133 | elif args.command[1:2] == ['mount']: 134 | virtualbox.init_vm(new_storage_callback=unison.clean_local_dir) 135 | dir = BASE / 'remote' 136 | if not dir.exists(): 137 | dir.mkdir() 138 | cmd = ['sudo', 'mount', '-t', 'nfs', 139 | '-o', 'vers=4,resvport,port=7049', 140 | '127.0.0.1:/vagga', str(dir)] 141 | print("Running", ' '.join(cmd), file=sys.stderr) 142 | if (dir / 'lost+found').exists(): 143 | print("It looks like your volume is already mounted.", 144 | file=sys.stderr) 145 | print("You only need to mount the volume once.", 146 | file=sys.stderr) 147 | ide_hint() 148 | return sys.exit(1) 149 | returncode = subprocess.Popen(cmd).wait() 150 | if returncode == 0: 151 | ide_hint() 152 | return sys.exit(returncode) 153 | else: 154 | print("Unknown command", repr((args.command[1:2] or [''])[0]), 155 | file=sys.stderr) 156 | print("Specify one of " 157 | "`ssh`, `upgrade_vagga`, `mount`, `up`, 'down'", 158 | file=sys.stderr) 159 | return sys.exit(1) 160 | 161 | 162 | path, cfg, suffix = config.get_config() 163 | 164 | setting = settings.parse_all(path) 165 | 166 | vagga = runtime.Vagga(path, cfg, args) 167 | 168 | if not vagga.vagga_dir.exists(): 169 | vagga.vagga_dir.mkdir() 170 | 171 | vm = virtualbox.init_vm(new_storage_callback=unison.clean_local_dir) 172 | 173 | setting['auto-apply-sysctl'] = True 174 | 175 | vagga.storage_volume = find_volume(vagga) 176 | 177 | env = os.environ.copy() 178 | env.update({ 179 | 'VAGGA_VOLUME': vagga.storage_volume, 180 | 'VAGGA_RESOLV_CONF': read_resolv_conf(), 181 | 'VAGGA_SETTINGS': json.dumps(setting), 182 | }) 183 | 184 | with unison.start_sync(vagga): 185 | with virtualbox.expose_ports(vm, vagga.exposed_ports()): 186 | result = subprocess.Popen( 187 | BASE_SSH_TTY_COMMAND + [ 188 | '-q', 189 | '/usr/local/bin/vagga-ssh.sh', 190 | ] + list(map(shlex.quote, sys.argv[1:])), 191 | env=env, 192 | ).wait() 193 | 194 | sys.exit(result) 195 | 196 | -------------------------------------------------------------------------------- /vagga_box/runtime.py: -------------------------------------------------------------------------------- 1 | class Vagga(object): 2 | 3 | def __init__(self, path, config, arguments): 4 | self.base = path 5 | self.vagga_dir = path / '.vagga' 6 | self.containers = config.get('containers', {}) 7 | self.commands = config.get('commands', {}) 8 | self.ignore_list = config.get('_ignore-dirs', []) 9 | self.arguments = arguments 10 | 11 | if arguments.command: 12 | self.run_commands = [arguments.command[0]] 13 | else: 14 | self.run_commands = arguments.run_multi or [] 15 | 16 | def exposed_ports(self): 17 | return frozenset(_exposed_ports(self, self.run_commands)) 18 | 19 | 20 | 21 | def _exposed_ports(vagga, commands): 22 | for command in commands: 23 | cmd = vagga.commands.get(command, {}) 24 | # allow expose ports in the command 25 | yield from _get_ports(cmd) 26 | # and in children commands (for supervise) 27 | for child in cmd.get('children', {}).values(): 28 | yield from _get_ports(child) 29 | 30 | 31 | def _get_ports(cmd): 32 | ports = cmd.get('_expose-ports', []) 33 | if not isinstance(ports, list): 34 | raise RuntimeError("the `_expose-ports` setting must be a list" 35 | "of integers, got {!r} instead".format(ports)) 36 | yield from ports 37 | -------------------------------------------------------------------------------- /vagga_box/settings.py: -------------------------------------------------------------------------------- 1 | import os 2 | import yaml 3 | import pathlib 4 | import logging 5 | import warnings 6 | 7 | 8 | log = logging.getLogger(__name__) 9 | SETTING_FILES = [ 10 | '.vagga.yaml', 11 | '.vagga/settings.yaml', 12 | '.config/vagga/settings.yaml', 13 | ] 14 | 15 | 16 | def parse_all(vagga_base): 17 | base_str = str(vagga_base) 18 | settings = {} 19 | for filename in SETTING_FILES: 20 | path = pathlib.Path(os.path.expanduser('~')) / filename 21 | if path.exists(): 22 | with path.open('rb') as f: 23 | try: 24 | data = yaml.load(f) 25 | except ValueError: 26 | print("WARNING: error loading settings from {}: {}" 27 | .format(path, data)) 28 | continue 29 | site = data.pop('site-settings', {}).get(base_str, {}) 30 | settings.update(data) 31 | settings.update(site) 32 | if settings.pop('external-volumes', None): 33 | warnings.warn("External volumes are not supported " 34 | " (defined in {})".format(path)) 35 | if settings.pop('storage-dir', None): 36 | warnings.warn("storage-dir is not supported as we use docker" 37 | " volumes for storage" 38 | " (defined in {})".format(path)) 39 | if settings.pop('cache-dir', None): 40 | warnings.warn("cache-dir is not supported yet" 41 | " (defined in {})".format(path)) 42 | log.debug("Got settings %r", settings) 43 | return settings 44 | 45 | -------------------------------------------------------------------------------- /vagga_box/ssh_config: -------------------------------------------------------------------------------- 1 | Port 7022 2 | StrictHostKeyChecking no 3 | CheckHostIp no 4 | SendEnv VAGGA_* 5 | -------------------------------------------------------------------------------- /vagga_box/unison.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import time 4 | import errno 5 | import fcntl 6 | import shlex 7 | import shutil 8 | import signal 9 | import socket 10 | import hashlib 11 | import logging 12 | from contextlib import contextmanager 13 | import resource 14 | import subprocess 15 | import warnings 16 | 17 | from . import BASE, KEY_PATH, BASE_SSH_COMMAND 18 | 19 | log = logging.getLogger(__name__) 20 | CONFIG_FILE = os.path.join(os.path.dirname(__file__), 'ssh_config') 21 | 22 | 23 | OUTER_LIMIT = 10000 # Note: should be lower than kern.maxfilesperproc 24 | INNER_LIMIT = 20000 # TODO(tailhook) don't know if bigger than outer is useful 25 | 26 | 27 | def clean_local_dir(): 28 | dir = BASE / 'unison' 29 | if dir.exists(): 30 | shutil.rmtree(str(dir)) 31 | 32 | 33 | def _unison_cli(vagga): 34 | 35 | ignores = [] 36 | for v in vagga.ignore_list: 37 | ignores.append('-ignore') 38 | ignores.append('Path ' + v) 39 | 40 | sync_start = time.time() 41 | unison_dir = BASE / 'unison' 42 | if not unison_dir.exists(): 43 | unison_dir.mkdir() 44 | env = os.environ.copy() 45 | env.update({ 46 | "UNISON": str(unison_dir), 47 | }) 48 | cmdline = [ 49 | 'unison', '.', 50 | 'ssh://user@localhost//vagga/' + vagga.storage_volume, 51 | '-sshargs', 52 | ' -i ' + str(KEY_PATH) + 53 | ' -F ' + CONFIG_FILE + 54 | ' exec sudo sh -c "ulimit -n 20000; exec sudo -u user env UNISON=/vagga/.unison \\"\\$@\\"" --', 55 | '-batch', '-silent', 56 | '-prefer', '.', 57 | '-ignore', 'Path .vagga', 58 | ] + ignores 59 | return cmdline, env 60 | 61 | 62 | def openlock(path): 63 | # unfortunately this combination of flags doesn't have string 64 | # representation 65 | fd = os.open(str(path), os.O_CREAT|os.O_RDWR) 66 | return os.fdopen(fd, 'rb+') 67 | 68 | 69 | def background_run(cmdline, env, logfile): 70 | log = open(str(logfile), 'wb') 71 | pro = subprocess.Popen(cmdline, env=env, 72 | stdin=subprocess.DEVNULL, stdout=log, stderr=log, 73 | preexec_fn=lambda: signal.signal(signal.SIGHUP, signal.SIG_IGN)) 74 | return pro.pid 75 | 76 | 77 | def unison_archive_names(vagga): 78 | myhost = socket.gethostname() 79 | me = '//{}/{}'.format(myhost, vagga.base) 80 | other = '//vagga//vagga/' + vagga.storage_volume 81 | suffix = ';' + ', '.join(sorted([me, other])) + ';22' 82 | 83 | hash = hashlib.md5() 84 | hash.update(me.encode('ascii')) 85 | hash.update(suffix.encode('ascii')) 86 | my = hash.hexdigest() 87 | 88 | hash = hashlib.md5() 89 | hash.update(other.encode('ascii')) 90 | hash.update(suffix.encode('ascii')) 91 | other = hash.hexdigest() 92 | 93 | return my, other 94 | 95 | 96 | def _remove_unison_lock(vagga): 97 | """Removes unison lock on both sides 98 | 99 | Two notes: 100 | 101 | 1. We assume that there is no remote lock if there is no local one 102 | 2. We assume that unison is not running (this is ensured by start_sync) 103 | """ 104 | myhash, vmhash = unison_archive_names(vagga) 105 | locallock = BASE / 'unison' / ('lk' + myhash) 106 | if locallock.exists(): 107 | subprocess.check_call(BASE_SSH_COMMAND + 108 | ['sh', '-c', '"rm /vagga/.unison/lk{} || true"'.format(vmhash)]) 109 | locallock.unlink() 110 | 111 | 112 | def set_ulimit(): 113 | try: 114 | cur, max = resource.getrlimit(resource.RLIMIT_NOFILE) 115 | if cur < OUTER_LIMIT: 116 | resource.setrlimit(resource.RLIMIT_NOFILE, [OUTER_LIMIT, max]) 117 | except Exception as e: 118 | warnings.warn("Could not set file limit to {}: {}" 119 | .format(OUTER_LIMIT, e)) 120 | 121 | 122 | 123 | @contextmanager 124 | def start_sync(vagga): 125 | set_ulimit() 126 | lockfilename = vagga.vagga_dir / '.unison-lock' 127 | while True: 128 | lockfile = openlock(lockfilename) 129 | lock = fcntl.lockf(lockfile, fcntl.LOCK_SH) 130 | try: 131 | lockfile.seek(0) 132 | pid = int(lockfile.read().strip()) 133 | os.kill(pid, 0) 134 | except (ValueError, OSError) as e: 135 | fcntl.lockf(lockfile, fcntl.LOCK_UN) 136 | try: 137 | fcntl.lockf(lockfile, fcntl.LOCK_EX | fcntl.LOCK_NB) 138 | except IOError: 139 | if (isinstance(e, OSError) and e.errno == errno.ESRCH or 140 | isinstance(e, ValueError)): 141 | # there is a file and it's locked but no such process 142 | lockfilename.unlink() 143 | continue 144 | else: 145 | cmdline, env = _unison_cli(vagga) 146 | 147 | _remove_unison_lock(vagga) 148 | 149 | start_time = time.time() 150 | log.info("Syncing files...") 151 | subprocess.check_call(cmdline, env=env) 152 | log.info("Synced in %.1f sec", time.time() - start_time) 153 | 154 | pid = background_run(cmdline + ['-repeat=watch'], 155 | env=env, logfile=vagga.vagga_dir / '.unison-log') 156 | 157 | lockfile.seek(0) 158 | lockfile.write(str(pid).encode('ascii')) 159 | lockfile.flush() 160 | finally: 161 | fcntl.lockf(lockfile, fcntl.LOCK_UN) 162 | fcntl.lockf(lockfile, fcntl.LOCK_SH) 163 | break 164 | else: 165 | break 166 | 167 | try: 168 | yield 169 | finally: 170 | fcntl.lockf(lockfile, fcntl.LOCK_UN) 171 | lockfile = openlock(lockfilename) 172 | try: 173 | fcntl.lockf(lockfile, fcntl.LOCK_EX | fcntl.LOCK_NB) 174 | except IOError: 175 | pass # locked by someone else, don't do anything 176 | else: 177 | try: 178 | lockfile.seek(0) 179 | os.kill(int(lockfile.read()), signal.SIGINT) 180 | except (ValueError, OSError) as e: 181 | log.info("Error when killing unison %r", e) 182 | -------------------------------------------------------------------------------- /vagga_box/unox.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # 3 | # unox 4 | # 5 | # Author: Hannes Landeholm 6 | # 7 | # The Unison beta (2.48) comes with file system change monitoring (repeat = watch) 8 | # through an abstract "unison-fsmonitor" adapter that integrates with each respective 9 | # OS file update watch interface. This allows responsive dropbox like master-master sync 10 | # of files over SSH. The Unison beta comes with an adapter for Windows and Linux but 11 | # unfortunately lacks one for OS X. 12 | # 13 | # This script implements the Unison fswatch protocol (see /src/fswatch.ml) 14 | # and is intended to be installed as unison-fsmonitor in the PATH in OS X. This is the 15 | # missing puzzle piece for repeat = watch support for Unison in in OS X. 16 | # 17 | # Dependencies: pip install macfsevents 18 | # 19 | # Licence: MPLv2 (https://www.mozilla.org/MPL/2.0/) 20 | 21 | import sys 22 | import os 23 | import time 24 | import fsevents 25 | import urllib.request, urllib.parse, urllib.error 26 | import traceback 27 | 28 | my_log_prefix = "[unox]" 29 | 30 | _in_debug = "--debug" in sys.argv 31 | _in_debug_plus = False 32 | 33 | # Global MacFSEvents observer. 34 | observer = fsevents.Observer() 35 | observer.start() 36 | 37 | # Dict of monitored replicas. 38 | # Replica hash mapped to fsevents.Stream objects. 39 | replicas = {} 40 | 41 | # Dict of pending replicas that are beeing waited on. 42 | # Replica hash mapped to True if replica is pending. 43 | pending_reps = {} 44 | 45 | # Dict of triggered replicas. 46 | # Replica hash mapped to recursive dict where keys are path tokens or True for pending leaf. 47 | triggered_reps = {} 48 | 49 | def format_exception(e): 50 | # Thanks for not bundling this function in the Python library Guido. *facepalm* 51 | exception_list = traceback.format_stack() 52 | exception_list = exception_list[:-2] 53 | exception_list.extend(traceback.format_tb(sys.exc_info()[2])) 54 | exception_list.extend(traceback.format_exception_only(sys.exc_info()[0], sys.exc_info()[1])) 55 | exception_str = "Traceback (most recent call last):\n" 56 | exception_str += "".join(exception_list) 57 | exception_str = exception_str[:-1] 58 | return exception_str 59 | 60 | def _debug_triggers(): 61 | global pending_reps, triggered_reps 62 | if not _in_debug_plus: 63 | return 64 | wait_info = "" 65 | if len(pending_reps) > 0: 66 | wait_info = " | wait=" + str(pending_reps) 67 | sys.stderr.write(my_log_prefix + "[DEBUG+]: trig=" + str(triggered_reps) + wait_info + "\n") 68 | 69 | def _debug(msg): 70 | sys.stderr.write(my_log_prefix + "[DEBUG]: " + msg.strip() + "\n") 71 | 72 | def warn(msg): 73 | sys.stderr.write(my_log_prefix + "[WARN]: " + msg.strip() + "\n") 74 | 75 | def sendCmd(cmd, args): 76 | raw_cmd = cmd 77 | for arg in args: 78 | raw_cmd += " " + urllib.parse.quote(arg); 79 | if _in_debug: _debug("sendCmd: " + raw_cmd) 80 | sys.stdout.write(raw_cmd + "\n") 81 | 82 | # Safely injects a command to send from non-receive context. 83 | def injectCmd(cmd, args): 84 | sendCmd(cmd, args) 85 | sys.stdout.flush() 86 | 87 | def sendAck(): 88 | sendCmd("OK", []) 89 | 90 | def sendError(msg): 91 | sendCmd("ERROR", [msg]) 92 | os._exit(1) 93 | 94 | def recvCmd(): 95 | # We flush before stalling on read instead of 96 | # flushing every write for optimization purposes. 97 | sys.stdout.flush() 98 | line = sys.stdin.readline() 99 | if not line.endswith("\n"): 100 | # End of stream means we're done. 101 | if _in_debug: _debug("stdin closed, exiting") 102 | sys.exit(0) 103 | if _in_debug: _debug("recvCmd: " + line) 104 | # Parse cmd and args. Args are url encoded. 105 | words = line.strip().split(" ") 106 | args = [] 107 | for word in words[1:]: 108 | args.append(urllib.parse.unquote(word)) 109 | return [words[0], args] 110 | 111 | def pathTokenize(path): 112 | path_toks = [] 113 | for path_tok in path.split("/"): 114 | if len(path_tok) > 0: 115 | path_toks.append(path_tok) 116 | return path_toks 117 | 118 | def triggerReplica(replica, local_path_toks): 119 | global pending_reps, triggered_reps 120 | if replica in pending_reps: 121 | # Got event for pending replica, notify and reset wait. 122 | injectCmd("CHANGES", [replica]) 123 | pending_reps = {} 124 | # Handle root. 125 | if len(local_path_toks) == 0: 126 | triggered_reps[replica] = True 127 | return 128 | elif not replica in triggered_reps: 129 | cur_lvl = {} 130 | triggered_reps[replica] = cur_lvl 131 | else: 132 | cur_lvl = triggered_reps[replica] 133 | # Iterate through branches. 134 | for branch_path_tok in local_path_toks[:len(local_path_toks) - 1]: 135 | if cur_lvl == True: 136 | return 137 | if not branch_path_tok in cur_lvl: 138 | new_lvl = {} 139 | cur_lvl[branch_path_tok] = new_lvl 140 | else: 141 | new_lvl = cur_lvl[branch_path_tok] 142 | cur_lvl = new_lvl 143 | # Handle leaf. 144 | if cur_lvl == True: 145 | return 146 | leaf_path_tok = local_path_toks[len(local_path_toks) - 1] 147 | cur_lvl[leaf_path_tok] = True 148 | _debug_triggers() 149 | 150 | # Starts monitoring of a replica. 151 | def startReplicaMon(replica, fspath, path): 152 | global replicas, observer 153 | if not replica in replicas: 154 | # Ensure fspath has trailing slash. 155 | fspath = os.path.join(fspath, "") 156 | if _in_debug: _debug("start monitoring of replica [" + replica + "] [" + fspath + "]") 157 | def replicaFileEventCallback(path, mask): 158 | try: 159 | if not path.startswith(fspath): 160 | return warn("unexpected file event at path [" + path + "] for [" + fspath + "]") 161 | local_path = path[len(fspath):] 162 | local_path_toks = pathTokenize(local_path) 163 | if _in_debug: _debug("replica:[" + replica + "] file event @[" + local_path + "] (" + path + ")") 164 | triggerReplica(replica, local_path_toks) 165 | except Exception as e: 166 | # Because python is a horrible language it has a special behavior for non-main threads that 167 | # fails to catch an exception. Instead of crashing the process, only the thread is destroyed. 168 | # We fix this with this catch all exception handler. 169 | sys.stderr.write(format_exception(e)) 170 | sys.stderr.flush() 171 | os._exit(1) 172 | try: 173 | # OS X has no interface for "file level" events. You would have to implement this manually in userspace, 174 | # and compare against a snapshot. This means there's no point in us doing it, better leave it to Unison. 175 | if _in_debug: _debug("replica:[" + replica + "] watching path [" + fspath + "]") 176 | stream = fsevents.Stream(replicaFileEventCallback, fspath) 177 | observer.schedule(stream) 178 | except (FileNotFoundError, NotADirectoryError) as e: 179 | sendError(str(e)) 180 | replicas[replica] = { 181 | "stream": stream, 182 | "fspath": fspath 183 | } 184 | sendAck() 185 | while True: 186 | [cmd, args] = recvCmd(); 187 | if cmd == "DIR": 188 | sendAck() 189 | elif cmd == "LINK": 190 | sendError("link following is not supported by unison-watchdog, please disable this option (-links)") 191 | elif cmd == "DONE": 192 | return 193 | else: 194 | sendError("unexpected cmd in replica start: " + cmd) 195 | 196 | def reportRecursiveChanges(local_path, cur_lvl): 197 | if (cur_lvl == True): 198 | sendCmd("RECURSIVE", [local_path]) 199 | return 200 | for path_tok, new_lvl in list(cur_lvl.items()): 201 | reportRecursiveChanges(os.path.join(local_path, path_tok), new_lvl); 202 | 203 | def _main(): 204 | global replicas, pending_reps, triggered_reps 205 | # Version handshake. 206 | sendCmd("VERSION", ["1"]) 207 | [cmd, args] = recvCmd(); 208 | if cmd != "VERSION": 209 | sendError("unexpected version cmd: " + cmd) 210 | [v_no] = args 211 | if v_no != "1": 212 | warn("unexpected version: " + v_no) 213 | # Start watch operation. 214 | _debug_triggers() 215 | while True: 216 | [cmd, args] = recvCmd(); 217 | # Cancel pending waits when any other command is received. 218 | if cmd != "WAIT": 219 | pending_reps = {} 220 | # Check command. 221 | if cmd == "DEBUG": 222 | _in_debug = True 223 | elif cmd == "START": 224 | # Start observing replica. 225 | if len(args) >= 3: 226 | [replica, fspath, path] = args 227 | else: 228 | # No path, only monitoring fspath. 229 | [replica, fspath] = args 230 | path = "" 231 | startReplicaMon(replica, fspath, path) 232 | elif cmd == "WAIT": 233 | # Start waiting for another replica. 234 | [replica] = args 235 | if not replica in replicas: 236 | sendError("unknown replica: " + replica) 237 | if replica in triggered_reps: 238 | # Is pre-triggered replica. 239 | sendCmd("CHANGES", replica) 240 | pending_reps = {} 241 | else: 242 | pending_reps[replica] = True 243 | _debug_triggers() 244 | elif cmd == "CHANGES": 245 | # Get pending replicas. 246 | [replica] = args 247 | if not replica in replicas: 248 | sendError("unknown replica: " + replica) 249 | if replica in triggered_reps: 250 | reportRecursiveChanges("", triggered_reps[replica]) 251 | del triggered_reps[replica] 252 | sendCmd("DONE", []) 253 | _debug_triggers() 254 | elif cmd == "RESET": 255 | # Stop observing replica. 256 | [replica] = args 257 | if not replica in replicas: 258 | warn("unknown replica: " + replica) 259 | continue 260 | stream = replicas[replica]["stream"] 261 | if stream is not None: 262 | observer.unschedule(stream) 263 | del replicas[replica] 264 | if replica in triggered_reps: 265 | del triggered_reps[replica] 266 | _debug_triggers() 267 | else: 268 | sendError("unexpected root cmd: " + cmd) 269 | 270 | 271 | def main(): 272 | try: 273 | _main() 274 | finally: 275 | for replica in replicas: 276 | observer.unschedule(replicas[replica]["stream"]) 277 | observer.stop() 278 | observer.join() 279 | 280 | 281 | if __name__ == '__main__': 282 | main() 283 | 284 | -------------------------------------------------------------------------------- /vagga_box/virtualbox.py: -------------------------------------------------------------------------------- 1 | import re 2 | import time 3 | import pathlib 4 | import warnings 5 | import hashlib 6 | import subprocess 7 | from contextlib import contextmanager 8 | 9 | from . import BASE 10 | 11 | PORT_RE = re.compile('name\s*=\s*port_(\d+),') 12 | 13 | IMAGE_VERSION = '0.1' 14 | IMAGE_NAME = 'virtualbox-image-{}.vmdk'.format(IMAGE_VERSION) 15 | IMAGE_URL = 'http://files.zerogw.com/vagga/' + IMAGE_NAME 16 | IMAGE_SHA256 = 'e8d84cff0acb084dc4bbbe93769e0e40c410846ff7643e6eb4a3990a882d1785' 17 | 18 | STORAGE_VERSION = '0.1' 19 | STORAGE_NAME = 'virtualbox-storage-{}.vmdk'.format(IMAGE_VERSION) 20 | STORAGE_URL = 'http://files.zerogw.com/vagga/' + STORAGE_NAME 21 | STORAGE_SHA256 = 'a70c77816dc422b0695a34678493ab874128cbf010b695de9041d09609db870d' 22 | 23 | 24 | def check_sha256(filename, sum): 25 | with open(str(filename), 'rb') as f: 26 | sha = hashlib.sha256() 27 | while True: 28 | chunk = f.read() 29 | if not chunk: 30 | break 31 | sha.update(chunk) 32 | if sha.hexdigest() != sum: 33 | filename.unlink() 34 | raise ValueError("Sha256 sum mismatch") 35 | 36 | 37 | def stop_vm(): 38 | id, ver = find_vm() 39 | if id: 40 | subprocess.check_call(['VBoxManage', 'controlvm', id, 41 | 'acpipowerbutton']) 42 | 43 | 44 | def create_vm(): 45 | tmpname = 'vagga-tmp' 46 | 47 | if (pathlib.Path.home() / '.ssh/known_hosts').exists(): 48 | # remove old host key for this host:port in case VM was removed 49 | subprocess.check_call(['ssh-keygen', '-R', '[127.0.0.1]:7022']) 50 | 51 | subprocess.check_call(['VBoxManage', 'createvm', '--register', 52 | '--name', tmpname, '--ostype', 'Ubuntu_64']) 53 | subprocess.check_call(['VBoxManage', 'modifyvm', tmpname, 54 | '--audio', 'none', 55 | '--memory', '2048', 56 | '--cpus', '2', 57 | '--nic1', 'nat', 58 | '--nictype1', '82545EM', 59 | '--rtcuseutc', 'on', 60 | '--natpf1', 'SSH,tcp,,7022,,22', 61 | '--natpf1', 'NFSt1,tcp,,7049,,2049', 62 | '--natpf1', 'NFSu1,udp,,7049,,2049', 63 | '--natpf1', 'NFSt2,tcp,,7111,,111', 64 | '--natpf1', 'NFSu2,udp,,7111,,111', 65 | '--natpf1', 'unison,tcp,,7767,,7767', 66 | ]) 67 | subprocess.check_call(['VBoxManage', 'storagectl', tmpname, 68 | '--name', 'SATA Controller', 69 | '--add', 'sata']) 70 | subprocess.check_call(['VBoxManage', 'storageattach', tmpname, 71 | '--storagectl', 'SATA Controller', 72 | '--device', '0', '--port', '0', '--type', 'hdd', 73 | '--medium', str(BASE / 'vm/image.vdi')]) 74 | subprocess.check_call(['VBoxManage', 'storageattach', tmpname, 75 | '--storagectl', 'SATA Controller', 76 | '--device', '0', '--port', '1', '--type', 'hdd', 77 | '--medium', str(BASE / 'vm/storage.vdi')]) 78 | subprocess.check_call(['VBoxManage', 'modifyvm', tmpname, 79 | '--name', 'vagga-' + IMAGE_VERSION]) 80 | return find_vm() 81 | 82 | 83 | def find_vm(): 84 | vboxlist = subprocess.check_output([ 85 | 'VBoxManage', 'list', 'vms', 86 | ]).decode('ascii').splitlines() 87 | for line in vboxlist: 88 | if line.startswith('"vagga-'): 89 | return ( 90 | line.split('{')[1].rstrip('}'), # id 91 | line.split('-')[1].split('"')[0], # version 92 | ) 93 | return None, None 94 | 95 | 96 | def download_image(url, basename, hash, destination): 97 | 98 | if not destination.exists(): 99 | 100 | image_dir = BASE / 'downloads' 101 | image_path = image_dir / basename 102 | 103 | if not image_dir.exists(): 104 | image_dir.mkdir() 105 | 106 | if not image_path.exists(): 107 | tmp_path = image_path.with_suffix('.tmp') 108 | subprocess.check_call( 109 | ['wget', '--continue', '-O', str(tmp_path), url]) 110 | check_sha256(tmp_path, hash) 111 | tmp_path.rename(image_path) 112 | 113 | subprocess.check_call([ 114 | 'VBoxManage', 'clonehd', '--format', 'vdi', 115 | str(image_path), str(destination)]) 116 | 117 | return True 118 | 119 | 120 | def check_running(cur_id): 121 | info = subprocess.check_output(['VBoxManage', 'showvminfo', cur_id]) 122 | for line in info.decode('latin-1').splitlines(): 123 | if line.startswith('State:'): 124 | if line.split()[1] == 'running': 125 | return True 126 | 127 | 128 | def init_vm(new_storage_callback): 129 | if not BASE.exists(): 130 | BASE.mkdir() 131 | 132 | subprocess.run( 133 | ['VBoxManage', 'unregistervm', 'vagga-tmp', '--delete'], 134 | stderr=subprocess.DEVNULL) 135 | 136 | download_image(IMAGE_URL, IMAGE_NAME, IMAGE_SHA256, BASE / 'vm/image.vdi') 137 | new_storage = download_image(STORAGE_URL, STORAGE_NAME, 138 | STORAGE_SHA256, BASE / 'vm/storage.vdi') 139 | if new_storage: 140 | new_storage_callback() 141 | 142 | cur_id, cur_version = find_vm() 143 | if cur_id is None: 144 | cur_id, cur_version = create_vm() 145 | elif cur_version != IMAGE_VERSION: 146 | warnings.warn("Image version {} required but {} found. " 147 | "Please run vagga _box_upgrade" 148 | .format(IMAGE_VERSION, cur_version)) 149 | 150 | if not check_running(cur_id): 151 | subprocess.check_call(['VBoxManage', 'startvm', cur_id, 152 | '--type', 'headless']) 153 | 154 | return cur_id 155 | 156 | 157 | @contextmanager 158 | def expose_ports(vm, ports): 159 | 160 | info = subprocess.check_output(['VBoxManage', 'showvminfo', vm]) 161 | already = set() 162 | for line in info.decode('latin-1').splitlines(): 163 | if line.startswith('NIC 1 Rule'): 164 | m = PORT_RE.search(line) 165 | if m: 166 | already.add(int(m.group(1))) 167 | 168 | left = ports - already 169 | if left: 170 | cmdline = ['VBoxManage', 'controlvm', vm, 'natpf1'] 171 | for port in left: 172 | cmdline.append('port_{0},tcp,,{0},,{0}'.format(port)) 173 | subprocess.check_call(cmdline) 174 | try: 175 | yield 176 | finally: 177 | if ports: 178 | cmdline = ['VBoxManage', 'controlvm', vm, 'natpf1'] 179 | for port in ports: 180 | cmdline.append('delete') 181 | cmdline.append('port_{0}'.format(port)) 182 | subprocess.check_call(cmdline) 183 | --------------------------------------------------------------------------------