├── test ├── uberspaces │ ├── u6 │ │ ├── empty │ │ │ ├── home │ │ │ │ └── .gitkeep │ │ │ ├── var │ │ │ │ └── www │ │ │ │ │ └── virtual │ │ │ │ │ └── .gitkeep │ │ │ └── etc │ │ │ │ └── centos-release │ │ └── isabell │ │ │ ├── commands │ │ │ ├── crontab -l │ │ │ ├── uberspace-list-domains -m │ │ │ └── uberspace-list-domains -w │ │ │ ├── home │ │ │ └── isabell │ │ │ │ ├── html │ │ │ │ ├── Maildir │ │ │ │ ├── cur │ │ │ │ │ └── mail-888 │ │ │ │ └── new │ │ │ │ │ └── mail-999 │ │ │ │ └── .my.cnf │ │ │ ├── var │ │ │ └── www │ │ │ │ └── virtual │ │ │ │ └── isabell │ │ │ │ ├── unknown-domain.com │ │ │ │ └── .gitkeep │ │ │ │ └── html │ │ │ │ ├── blog │ │ │ │ └── index.html │ │ │ │ └── index.html │ │ │ └── etc │ │ │ └── centos-release │ └── u7 │ │ └── empty │ │ ├── var │ │ └── www │ │ │ └── virtual │ │ │ └── .gitkeep │ │ └── etc │ │ └── centos-release ├── test_cli_version.py ├── test_storage.py └── test_takeout.py ├── requirements.txt ├── uberspace_takeout ├── exc.py ├── items │ ├── __init__.py │ ├── base.py │ ├── u6.py │ ├── common.py │ └── u7.py ├── __main__.py ├── __init__.py └── storage.py ├── tox.ini ├── .pre-commit-config.yaml ├── setup.py ├── .gitignore └── README.md /test/uberspaces/u6/empty/home/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /test/uberspaces/u6/empty/var/www/virtual/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /test/uberspaces/u7/empty/var/www/virtual/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | pre-commit 2 | tox 3 | twine 4 | wheel 5 | -------------------------------------------------------------------------------- /uberspace_takeout/exc.py: -------------------------------------------------------------------------------- 1 | class TakeoutError(Exception): 2 | pass 3 | -------------------------------------------------------------------------------- /test/uberspaces/u6/isabell/commands/crontab -l: -------------------------------------------------------------------------------- 1 | @daily echo good morning 2 | -------------------------------------------------------------------------------- /test/uberspaces/u6/isabell/home/isabell/html: -------------------------------------------------------------------------------- 1 | ../../var/www/virtual/isabell/html -------------------------------------------------------------------------------- /test/uberspaces/u6/isabell/var/www/virtual/isabell/unknown-domain.com/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /test/uberspaces/u6/empty/etc/centos-release: -------------------------------------------------------------------------------- 1 | CentOS Linux release 6.11.1810 (Core) 2 | -------------------------------------------------------------------------------- /test/uberspaces/u6/isabell/etc/centos-release: -------------------------------------------------------------------------------- 1 | CentOS Linux release 6.11.1810 (Core) 2 | -------------------------------------------------------------------------------- /test/uberspaces/u6/isabell/home/isabell/Maildir/cur/mail-888: -------------------------------------------------------------------------------- 1 | From: you 2 | To: me 3 | -------------------------------------------------------------------------------- /test/uberspaces/u6/isabell/home/isabell/Maildir/new/mail-999: -------------------------------------------------------------------------------- 1 | From: me 2 | To: you 3 | -------------------------------------------------------------------------------- /test/uberspaces/u7/empty/etc/centos-release: -------------------------------------------------------------------------------- 1 | CentOS Linux release 7.5.1810 (Core) 2 | -------------------------------------------------------------------------------- /test/uberspaces/u6/isabell/var/www/virtual/isabell/html/blog/index.html: -------------------------------------------------------------------------------- 1 |

blog

2 | -------------------------------------------------------------------------------- /test/uberspaces/u6/isabell/var/www/virtual/isabell/html/index.html: -------------------------------------------------------------------------------- 1 |

landing page

2 | -------------------------------------------------------------------------------- /uberspace_takeout/items/__init__.py: -------------------------------------------------------------------------------- 1 | from . import common # NOQA 2 | from . import u6 # NOQA 3 | from . import u7 # NOQA 4 | -------------------------------------------------------------------------------- /test/uberspaces/u6/isabell/commands/uberspace-list-domains -m: -------------------------------------------------------------------------------- 1 | mail.example.com 2 | isabell.andromeda.uberspace.de 3 | *.isabell.andromeda.uberspace.de 4 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [pylama:pycodestyle] 2 | max_line_length = 120 3 | 4 | [tox] 5 | envlist = py36,py37,py38 6 | 7 | [testenv] 8 | deps = 9 | pytest 10 | -e.[test] 11 | commands = pytest {posargs} 12 | -------------------------------------------------------------------------------- /test/uberspaces/u6/isabell/commands/uberspace-list-domains -w: -------------------------------------------------------------------------------- 1 | *.example.com 2 | example.com 3 | foo.example.com 4 | isabell.andromeda.uberspace.de 5 | ep.isabell.andromeda.uberspace.de 6 | *.isabell.andromeda.uberspace.de 7 | -------------------------------------------------------------------------------- /test/test_cli_version.py: -------------------------------------------------------------------------------- 1 | import subprocess 2 | 3 | from uberspace_takeout import __version__ 4 | 5 | 6 | def test_cli_version(capfdbinary): 7 | command = ["uberspace-takeout", "--version"] 8 | res = subprocess.run(command) 9 | assert res.returncode == 0 10 | out, err = capfdbinary.readouterr() 11 | version = out.decode("utf-8").strip() 12 | assert version == __version__ 13 | -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | default_language_version: 3 | python: python3 4 | repos: 5 | - repo: https://github.com/pre-commit/pre-commit-hooks 6 | rev: master 7 | hooks: 8 | # Generall Stuff 9 | - id: trailing-whitespace 10 | - id: mixed-line-ending 11 | args: [--fix=lf] 12 | - id: end-of-file-fixer 13 | exclude: "^(.bumpversion.cfg|CHANGELOG.rst)$" 14 | # VCS 15 | - id: check-merge-conflict 16 | # Config / Data Files 17 | - id: check-yaml 18 | # Python 19 | - id: debug-statements 20 | # Python: flakes8 (syntax check with pyflakes only) 21 | - repo: https://gitlab.com/pycqa/flake8 22 | rev: master 23 | hooks: 24 | - id: flake8 25 | args: 26 | - "--select=F" 27 | # Python: reorder imports 28 | - repo: https://github.com/asottile/reorder_python_imports 29 | rev: master 30 | hooks: 31 | - id: reorder-python-imports 32 | args: 33 | - "--application-directories=." 34 | # Python: black 35 | - repo: https://github.com/psf/black 36 | rev: stable 37 | hooks: 38 | - id: black 39 | -------------------------------------------------------------------------------- /test/uberspaces/u6/isabell/home/isabell/.my.cnf: -------------------------------------------------------------------------------- 1 | # Ansible managed 2 | # 3 | # Moechtest du dein Passwort aendern, so kannst du das mit dem Befehl 4 | # 5 | # SET PASSWORD = PASSWORD("..."); 6 | # 7 | # auf der MySQL-Shell tun. Anschliessend kannst du es auch hier anpassen. 8 | # 9 | # Beachte, dass dies die Konfigurationsdatei des MySQL-Clients ist, nicht 10 | # die des MySQL-Servers - das Passwort wird hier gefuehrt, damit du dich 11 | # ohne manuelle Eingabe mit dem Server verbinden kannst. Du kannst es 12 | # hierueber aber nicht *setzen*; das muss eben mit SET PASSWORD geschehen. 13 | # 14 | # Mehr dazu findest du hier: 15 | # 16 | # https://uberspace.de/dokuwiki/database:mysql#passwort_aendern 17 | # 18 | # Einen alternativen Account ohne Schreibrechte kannst du wie folgt nutzen: 19 | # mysql --defaults-group-suffix=readonly 20 | # Damit kannst du Clients auf das Statement 'SELECT' einschrnken. 21 | # Dies Passwort kann nur vom Support gendert werden. 22 | # Bitte wende dich bei Bedarf an hallo@uberspace.de. 23 | 24 | [client] 25 | user=isabell 26 | password="Lei4e%ngekäe3iÖt4Ies" 27 | 28 | [clientreadonly] 29 | user=isabell_ro 30 | password=eeruaSooch6iereequoo 31 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import sys 2 | 3 | try: 4 | from setuptools import setup 5 | except ImportError: 6 | print("This project needs setuptools.", file=sys.stderr) 7 | print("Please install it using your package-manager or pip.", file=sys.stderr) 8 | sys.exit(1) 9 | 10 | from uberspace_takeout import __version__ as version 11 | 12 | 13 | setup( 14 | name="uberspace_takeout", 15 | version=version, 16 | description="", 17 | author="uberspace.de", 18 | author_email="hallo@uberspace.de", 19 | url="https://github.com/uberspace/takeout", 20 | packages=["uberspace_takeout", "uberspace_takeout.items",], 21 | extras_require={"test": ["pyfakefs>=3.6", "pytest-mock",],}, 22 | entry_points={ 23 | "console_scripts": ["uberspace-takeout=uberspace_takeout.__main__:main"], 24 | }, 25 | classifiers=[ 26 | "Development Status :: 5 - Production/Stable", 27 | "Intended Audience :: Developers", 28 | "Intended Audience :: Information Technology", 29 | "Intended Audience :: System Administrators", 30 | "Topic :: System :: Systems Administration", 31 | "Topic :: Security", 32 | "Topic :: Utilities", 33 | "Natural Language :: English", 34 | "Operating System :: POSIX :: Linux", 35 | "Programming Language :: Python :: 2.7", 36 | ], 37 | zip_safe=True, 38 | ) 39 | -------------------------------------------------------------------------------- /uberspace_takeout/__main__.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import datetime 3 | import getpass 4 | import sys 5 | 6 | from . import __version__ as version 7 | from . import Takeout 8 | 9 | 10 | def main(): 11 | username = getpass.getuser() 12 | timestamp = datetime.datetime.now().strftime("%Y-%m-%d_%H_%M_%S") 13 | default_tar_path = "takeout_{}_{}.tar.bz2".format(username, timestamp) 14 | 15 | p = argparse.ArgumentParser() 16 | p.add_argument("action", choices=["takeout", "takein", "items"]) 17 | p.add_argument("--username", default=username) 18 | p.add_argument("--skip-item", action="append", default=[]) 19 | p.add_argument("--tar-file", default=default_tar_path) 20 | p.add_argument("--version", action="version", version=version) 21 | args = p.parse_args() 22 | 23 | tar_path = args.tar_file 24 | t = Takeout() 25 | 26 | if args.action == "takeout": 27 | if tar_path == "-": 28 | tar_path = "/dev/stdout" 29 | sys.stdout = sys.stderr 30 | 31 | print("writing " + tar_path) 32 | t.takeout(tar_path, args.username, args.skip_item) 33 | 34 | elif args.action == "takein": 35 | if tar_path == "-": 36 | tar_path = "/dev/stdin" 37 | 38 | print("reading " + tar_path) 39 | t.takein(tar_path, args.username, args.skip_item) 40 | 41 | elif args.action == "items": 42 | print( 43 | "\n".join( 44 | i.__name__.ljust(25, " ") + i.description for i in Takeout.takeout_menu 45 | ) 46 | ) 47 | return 0 48 | 49 | else: 50 | raise NotImplementedError() 51 | 52 | if t.errors: 53 | print() 54 | for item_class, errors in t.errors.items(): 55 | print(f"[ERROR] {item_class}:") 56 | for error in errors: 57 | print(f"- {error}") 58 | print() 59 | return 1 60 | 61 | else: 62 | return 0 63 | 64 | 65 | if __name__ == "__main__": 66 | sys.exit(main()) 67 | -------------------------------------------------------------------------------- /uberspace_takeout/items/base.py: -------------------------------------------------------------------------------- 1 | import re 2 | import os 3 | import subprocess 4 | 5 | 6 | class TakeoutItem: 7 | description = None 8 | 9 | def __init__(self, username, hostname, storage): 10 | self.username = username 11 | self.hostname = hostname 12 | self.storage = storage 13 | 14 | def takeout(self): 15 | raise NotImplementedError() 16 | 17 | def takein(self): 18 | raise NotImplementedError() 19 | 20 | def is_active(self): 21 | return True 22 | 23 | def run_command(self, cmd, input_text=None): 24 | env = os.environ.copy() 25 | env["PATH"] = "/usr/local/bin/:" + env["PATH"] 26 | env.pop("SUDO_USER", None) 27 | 28 | p = subprocess.Popen( 29 | cmd, 30 | stdout=subprocess.PIPE, 31 | stderr=subprocess.STDOUT, 32 | stdin=subprocess.PIPE, 33 | env=env, 34 | universal_newlines=True, # get a string, not bytes 35 | ) 36 | out, _ = p.communicate(input_text) 37 | 38 | return [l for l in out.split("\n") if l] 39 | 40 | def run_uberspace(self, *cmd): 41 | return self.run_command(["uberspace"] + list(cmd)) 42 | 43 | 44 | class PathItem(TakeoutItem): 45 | storage_path = None 46 | 47 | def takeout(self): 48 | self.storage.store_directory(self.path(), self.storage_path) 49 | 50 | def takein(self): 51 | self.storage.unstore_directory(self.storage_path, self.path()) 52 | 53 | 54 | class UberspaceVersionMixin: 55 | uberspace_version = None 56 | 57 | @property 58 | def current_uberspace_version(self): 59 | with open("/etc/centos-release") as f: 60 | text = f.read() 61 | # looks like "CentOS release 6.10 (Final)" 62 | centos_release = re.search(r"release ([0-9])+\.", text).groups()[0] 63 | return int(centos_release) 64 | 65 | def is_active(self): 66 | return self.current_uberspace_version == int(self.uberspace_version) 67 | -------------------------------------------------------------------------------- /uberspace_takeout/__init__.py: -------------------------------------------------------------------------------- 1 | #!/opt/uberspace/python-venv/bin/python 2 | import socket 3 | 4 | import uberspace_takeout.items as items 5 | import uberspace_takeout.storage as storage 6 | from uberspace_takeout.exc import TakeoutError 7 | 8 | 9 | __version__ = "0.3.0" 10 | 11 | 12 | class Takeout: 13 | takeout_menu = [ 14 | items.common.TakeoutMarker, 15 | items.common.Homedir, 16 | items.common.Www, 17 | items.common.Cronjobs, 18 | items.common.MySQLPassword, 19 | items.u7.WebDomains, 20 | items.u7.MailDomains, 21 | items.u7.AccessLogItem, 22 | items.u7.ApacheErrorLogItem, 23 | items.u7.PhpErrorLogItem, 24 | items.u7.SpamfilterLogItem, 25 | items.u7.ToolVersions, 26 | items.u6.WebDomains, 27 | items.u6.MailDomains, 28 | ] 29 | 30 | def __init__(self, hostname=None): 31 | if not hostname: 32 | hostname = socket.getfqdn() 33 | 34 | self.hostname = hostname 35 | self.errors = {} 36 | 37 | def get_items(self, username, storage): 38 | for item in self.takeout_menu: 39 | instance = item(username, self.hostname, storage) 40 | if instance.is_active(): 41 | yield instance 42 | 43 | def takein(self, tar_path, username, skipped_items=None): 44 | if skipped_items is None: 45 | skipped_items = [] 46 | with storage.TarStorage(tar_path, "takein") as stor: 47 | for item in self.get_items(username, stor): 48 | if item.__class__.__name__ in skipped_items: 49 | print("skip: " + item.description) 50 | continue 51 | 52 | print("takein: " + item.description) 53 | try: 54 | item.takein() 55 | except TakeoutError as exc: 56 | self.errors[item.__class__.__name__] = exc.args 57 | 58 | def takeout(self, tar_path, username, skipped_items=None): 59 | if skipped_items is None: 60 | skipped_items = [] 61 | with storage.TarStorage(tar_path, "takeout") as stor: 62 | for item in self.get_items(username, stor): 63 | if item.__class__.__name__ in skipped_items: 64 | print("skip: " + item.description) 65 | continue 66 | 67 | print("takeout: " + item.description) 68 | try: 69 | item.takeout() 70 | except TakeoutError as exc: 71 | self.errors[item.__class__.__name__] = exc.args 72 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | takeout_*.tar.bz2 2 | 3 | # Created by https://www.gitignore.io/api/python 4 | # Edit at https://www.gitignore.io/?templates=python 5 | 6 | ### Python ### 7 | # Byte-compiled / optimized / DLL files 8 | __pycache__/ 9 | *.py[cod] 10 | *$py.class 11 | 12 | # C extensions 13 | *.so 14 | 15 | # Distribution / packaging 16 | .Python 17 | build/ 18 | develop-eggs/ 19 | dist/ 20 | downloads/ 21 | eggs/ 22 | .eggs/ 23 | lib/ 24 | lib64/ 25 | parts/ 26 | sdist/ 27 | var/ 28 | !/test/uberspaces/**/var 29 | wheels/ 30 | pip-wheel-metadata/ 31 | share/python-wheels/ 32 | *.egg-info/ 33 | .installed.cfg 34 | *.egg 35 | MANIFEST 36 | 37 | # PyInstaller 38 | # Usually these files are written by a python script from a template 39 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 40 | *.manifest 41 | *.spec 42 | 43 | # Installer logs 44 | pip-log.txt 45 | pip-delete-this-directory.txt 46 | 47 | # Unit test / coverage reports 48 | htmlcov/ 49 | .tox/ 50 | .nox/ 51 | .coverage 52 | .coverage.* 53 | .cache 54 | nosetests.xml 55 | coverage.xml 56 | *.cover 57 | .hypothesis/ 58 | .pytest_cache/ 59 | 60 | # Translations 61 | *.mo 62 | *.pot 63 | 64 | # Django stuff: 65 | *.log 66 | local_settings.py 67 | db.sqlite3 68 | 69 | # Flask stuff: 70 | instance/ 71 | .webassets-cache 72 | 73 | # Scrapy stuff: 74 | .scrapy 75 | 76 | # Sphinx documentation 77 | docs/_build/ 78 | 79 | # PyBuilder 80 | target/ 81 | 82 | # Jupyter Notebook 83 | .ipynb_checkpoints 84 | 85 | # IPython 86 | profile_default/ 87 | ipython_config.py 88 | 89 | # pyenv 90 | .python-version 91 | 92 | # pipenv 93 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 94 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 95 | # having no cross-platform support, pipenv may install dependencies that don’t work, or not 96 | # install all needed dependencies. 97 | #Pipfile.lock 98 | 99 | # celery beat schedule file 100 | celerybeat-schedule 101 | 102 | # SageMath parsed files 103 | *.sage.py 104 | 105 | # Environments 106 | .env 107 | .venv 108 | env/ 109 | venv/ 110 | ENV/ 111 | env.bak/ 112 | venv.bak/ 113 | 114 | # Spyder project settings 115 | .spyderproject 116 | .spyproject 117 | 118 | # Rope project settings 119 | .ropeproject 120 | 121 | # mkdocs documentation 122 | /site 123 | 124 | # mypy 125 | .mypy_cache/ 126 | .dmypy.json 127 | dmypy.json 128 | 129 | # Pyre type checker 130 | .pyre/ 131 | 132 | # End of https://www.gitignore.io/api/python 133 | -------------------------------------------------------------------------------- /uberspace_takeout/items/u6.py: -------------------------------------------------------------------------------- 1 | import os 2 | import re 3 | 4 | from .base import TakeoutItem 5 | from .base import UberspaceVersionMixin 6 | 7 | 8 | class U6Mixin(UberspaceVersionMixin): 9 | uberspace_version = 6 10 | 11 | 12 | class DomainItem(U6Mixin, TakeoutItem): 13 | flag = None 14 | storage_path = None 15 | 16 | def _find_domains(self): 17 | domains = self.run_command(["uberspace-list-domains", self.flag]) 18 | domains = set(domains) - { 19 | self.username + "." + self.hostname, 20 | "*." + self.username + "." + self.hostname, 21 | } 22 | return domains 23 | 24 | def takeout(self): 25 | text = "\n".join(self._find_domains()) 26 | self.storage.store_text(text, self.storage_path) 27 | 28 | def takein(self): 29 | text = self.storage.unstore_text(self.storage_path) 30 | 31 | for domain in (d for d in text.split("\n") if d): 32 | self.run_command(["uberspace-add-domain", self.flag, "-d", domain]) 33 | 34 | 35 | class WebDomains(DomainItem): 36 | description = "Web Domains" 37 | flag = "-w" 38 | storage_path = "conf/domains-web" 39 | 40 | def _find_domains(self): 41 | domains = super()._find_domains() 42 | 43 | try: 44 | for candidate in os.listdir('/var/www/virtual/' + self.username): 45 | if not re.search(r'\.[a-z0-9-]{2,}$', candidate): 46 | continue 47 | 48 | domains.add(candidate) 49 | except OSError: 50 | pass 51 | 52 | return domains 53 | 54 | 55 | class MailDomains(DomainItem): 56 | description = "Mail Domains" 57 | flag = "-m" 58 | storage_path = "conf/domains-mail" 59 | 60 | 61 | class ToolVersion(TakeoutItem): 62 | storage_path = None 63 | version_regex = None 64 | software_command = None 65 | 66 | def _clean_version(self, version): 67 | return version 68 | 69 | def takeout(self): 70 | output = self.run_command(self.software_command) 71 | version = re.search(self.version_regex, output).groups()[0] 72 | version = self._clean_version(version) 73 | self.storage.store_text(version, self.storage_path) 74 | 75 | def takein(self): 76 | pass 77 | # TODO: print warning 78 | 79 | 80 | class NodeVersion(ToolVersion): 81 | storage_path = "conf/tool-version/node" 82 | version_regex = r"^v([0-9]+)" 83 | software_command = "node --version" 84 | 85 | 86 | class RubyVersion(ToolVersion): 87 | storage_path = "conf/tool-version/ruby" 88 | version_regex = r"^ruby ([0-9]+\.[0-9]+)" 89 | software_command = "ruby --version" 90 | 91 | 92 | class PhpVersion(ToolVersion): 93 | storage_path = "conf/tool-version/php" 94 | version_regex = r"^PHP ([0-9]+\.[0-9]+)" 95 | software_command = "php --version" 96 | -------------------------------------------------------------------------------- /uberspace_takeout/items/common.py: -------------------------------------------------------------------------------- 1 | import configparser 2 | 3 | from .base import PathItem 4 | from .base import TakeoutItem 5 | from uberspace_takeout.exc import TakeoutError 6 | 7 | 8 | class TakeoutMarker(TakeoutItem): 9 | description = "Takeout Marker (internal)" 10 | 11 | def takeout(self): 12 | self.storage.store_text("uberspace_takeout", ".uberspace_takeout") 13 | 14 | def takein(self): 15 | content = self.storage.unstore_text(".uberspace_takeout") 16 | 17 | if content != "uberspace_takeout": 18 | raise Exception("input is not a takeout.") 19 | 20 | 21 | class Homedir(PathItem): 22 | description = "Homedirectory" 23 | storage_path = "home/" 24 | 25 | def path(self): 26 | return "/home/" + self.username 27 | 28 | 29 | class Www(PathItem): 30 | description = "Documentroot" 31 | storage_path = "www/" 32 | 33 | def path(self): 34 | return "/var/www/virtual/" + self.username 35 | 36 | 37 | class Cronjobs(TakeoutItem): 38 | description = "Cronjobs" 39 | 40 | def takeout(self): 41 | cronjobs = self.run_command(["crontab", "-l"]) 42 | 43 | if len(cronjobs) == 1 and cronjobs[0].startswith("no crontab for"): 44 | text = "" 45 | else: 46 | text = "\n".join(cronjobs) + "\n" 47 | 48 | self.storage.store_text(text, "conf/cronjobs") 49 | 50 | def takein(self): 51 | text = self.storage.unstore_text("conf/cronjobs") 52 | self.run_command(["crontab", "-"], input_text=text) 53 | 54 | 55 | class MySQLPassword(TakeoutItem): 56 | description = "MySQL password" 57 | 58 | @property 59 | def _my_cnf_path(self): 60 | return "/home/" + self.username + "/.my.cnf" 61 | 62 | def _open_my_cnf(self, section): 63 | config = configparser.ConfigParser(interpolation=None) 64 | config.read(self._my_cnf_path, encoding="utf-8") 65 | return config 66 | 67 | def _read_my_cnf_password(self, section): 68 | raw_pw = self._open_my_cnf(section)[section]["password"] 69 | return raw_pw.partition(" #")[0].strip().strip('"') 70 | 71 | def _check_password(self, password): 72 | if len(password) < 12: 73 | msg = ( 74 | "Your current MySQL password does not satisfy our policy requirements:" 75 | " it is shorter than 12 characters." 76 | " Please set a sufficiently long one - as described under" 77 | " https://wiki.uberspace.de/database:mysql#passwort_aendern" 78 | " - or run this script with '--skip-item MySQLPassword'." 79 | ) 80 | raise TakeoutError(msg) 81 | 82 | def _write_my_cnf_password(self, section, password): 83 | config = self._open_my_cnf(section) 84 | config[section]["password"] = password 85 | 86 | with open(self._my_cnf_path, "w") as f: 87 | config.write(f) 88 | 89 | def _set_password(self, suffix): 90 | password = self.storage.unstore_text("conf/mysql-password-client" + suffix) 91 | self.run_command( 92 | [ 93 | "mysql", 94 | "--defaults-group-suffix=" + suffix, 95 | "-e", 96 | "SET PASSWORD = PASSWORD('" + password + "')", 97 | ] 98 | ) 99 | self._write_my_cnf_password("client" + suffix, password) 100 | 101 | def takeout(self): 102 | password = self._read_my_cnf_password("client") 103 | self.storage.store_text(password, "conf/mysql-password-client") 104 | self._check_password(password) 105 | 106 | def takein(self): 107 | self._set_password("") 108 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Uberspace Takeout 2 | 3 | Import and export _uberspaces_ to and from special `.tar` files. 4 | 5 | ## Items 6 | 7 | Takeout is organized in **TakeoutItems**, which handle the import and export of 8 | a single topic. `Homedir` takes care of all files in the users home directory, 9 | `MySQLPassword` gets and sets the MySQL passwords. 10 | 11 | Implementations can be found in `uberspace_takeout/items/{common,u6,u7}.py`. 12 | 13 | ## Usage 14 | 15 | To get ready, install takeout using git and pip: 16 | 17 | ```console 18 | $ git clone https://github.com/Uberspace/takeout.git 19 | $ cd takeout 20 | $ pip install --user -e . 21 | ``` 22 | 23 | After pip is done, you can access the tool by calling `uberspace-takeout`: 24 | 25 | ```console 26 | $ uberspace-takeout --help 27 | ``` 28 | 29 | ### Tasks 30 | 31 | Use `uberspace-takeout items` to list all tasks, which can be executed. 32 | 33 | ```console 34 | $ uberspace-takeout items 35 | TakeoutMarker Takeout Marker (internal) 36 | Homedir Homedirectory 37 | Www Documentroot 38 | Cronjobs Cronjobs 39 | MailDomains Mail Domains 40 | (...) 41 | ``` 42 | 43 | Note that some items will be duplicated (once for U6 and once for U7) and that 44 | some might not run at all, because they aren't available for uberspace version 45 | takeout is running on. 46 | 47 | ### Exporting: Takeout 48 | 49 | Takeout creates an archive of everything that makes up an uberspace. This 50 | includes files, software versions, mysql passwords and much more. Exporting 51 | MySQL databases is not yet implemented. 52 | 53 | ```console 54 | $ uberspace-takeout takeout 55 | writing takeout_luto_2019-09-04_14_44_30.tar.bz2 56 | takeout: TakeoutMarker 57 | takeout: Homedir 58 | takeout: Www 59 | takeout: Cronjobs 60 | takeout: MySQLPassword 61 | takeout: AccessLogItem 62 | takeout: ApacheErrorLogItem 63 | takeout: PhpErrorLogItem 64 | takeout: SpamfilterLogItem 65 | takeout: ToolVersions 66 | takeout: WebDomains 67 | takeout: MailDomains 68 | $ ll *.tar.bz2 69 | -rw-r--r-- 1 luto luto 132 Sep 4 14:44 takeout_luto_2019-09-04_14_44_30.tar.bz2 70 | ``` 71 | 72 | ### Importing: Takein 73 | 74 | You can read an archive created by `uberspace-takeout takeout` using 75 | `... takein`. It will, to the best of its ability, restore all settings and 76 | files exported earlier. Importing MySQL databases is not yet implemented. 77 | 78 | ```console 79 | $ uberspace-takeout takein --tar-file takeout_luto_2019-09-04_14_44_30.tar.bz2 80 | reading takeout_luto_2019-09-04_14_44_30.tar.bz2 81 | takein: TakeoutMarker 82 | takein: Homedir 83 | takein: Www 84 | takein: Cronjobs 85 | takein: MySQLPassword 86 | takein: AccessLogItem 87 | takein: ApacheErrorLogItem 88 | takein: PhpErrorLogItem 89 | takein: SpamfilterLogItem 90 | takein: ToolVersions 91 | takein: WebDomains 92 | takein: MailDomains 93 | ``` 94 | 95 | ## Development 96 | 97 | After cloning, create a _virtual environment_: 98 | 99 | ```console 100 | python3 -m venv venv 101 | source venv/bin/activate 102 | pip install --upgrade pip 103 | ``` 104 | 105 | Install the development-requirements: 106 | 107 | ``` 108 | pip install -r requirements.txt 109 | ``` 110 | 111 | And run some setup: 112 | 113 | ``` 114 | pre-commit install 115 | ``` 116 | 117 | After that you can… 118 | 119 | ### Lint 120 | 121 | ```console 122 | pre-commit run --all-files 123 | ``` 124 | 125 | ### Test 126 | 127 | ```console 128 | tox 129 | ``` 130 | 131 | ### Release 132 | 133 | Assuming you have been handed the required credentials, a new version 134 | can be released as follows. 135 | 136 | 1. adapt the `__version__` in `uberspace_takeout/__init__.py`, according to [semver][]. 137 | 2. commit this change as `Version 1.2.3` 138 | 3. tag the resulting commit as `v1.2.3` 139 | 4. push the new tag as well as the `master` branch 140 | 5. update the package on PyPI: 141 | 142 | ```console 143 | $ rm dist/* 144 | $ python setup.py sdist bdist_wheel 145 | $ twine upload dist/* 146 | ``` 147 | 148 | [semver]: https://semver.org/ 149 | -------------------------------------------------------------------------------- /uberspace_takeout/items/u7.py: -------------------------------------------------------------------------------- 1 | import re 2 | 3 | from .base import TakeoutItem 4 | from .base import UberspaceVersionMixin 5 | 6 | 7 | class U7Mixin(UberspaceVersionMixin): 8 | uberspace_version = 7 9 | 10 | 11 | def convert_legacy_domain(domain): 12 | """ 13 | convert legacy domains to new .uber.space ones, e.g. 14 | 15 | luto.cygnus.uberspace.de => luto.uber.space 16 | ep.luto.cygnus.uberspace.de => ep.luto.uber.space 17 | """ 18 | 19 | if domain.endswith('.uberspace.de'): 20 | # strip suffix off a full domain, e.g. 21 | # luto.cygnus.uberspace.de => luto 22 | domain = re.sub(r'\.[a-z]+\.uberspace\.de$', '', domain) + '.uber.space' 23 | 24 | return domain 25 | 26 | class DomainItem(U7Mixin, TakeoutItem): 27 | area = None 28 | 29 | def takeout(self): 30 | domains = self.run_uberspace(self.area, "domain", "list") 31 | domains = set(domains) - { 32 | self.username + ".uber.space", 33 | self.username + "." + self.hostname, 34 | } 35 | self.storage.store_text("\n".join(domains), self.storage_path) 36 | 37 | def takein(self): 38 | text = self.storage.unstore_text(self.storage_path) 39 | for domain in (d for d in text.split("\n") if d): 40 | if " " in domain: 41 | domain, _, namespace = domain.partition(" ") 42 | print("namespaced domains are not supported, stripping namespace: " + namespace) 43 | if domain.startswith("*."): 44 | print("wildcard certs are not supported, adding subdomain www at least") 45 | domain = "www" + domain[1:] 46 | if domain.endswith('.uberspace.de'): 47 | print("user.host.uberspace.de domains are not supported, rewriting to .uber.space") 48 | domain = convert_legacy_domain(domain) 49 | self.run_uberspace(self.area, "domain", "add", domain) 50 | 51 | 52 | class WebDomains(DomainItem): 53 | description = "Web Domains" 54 | area = "web" 55 | storage_path = "conf/domains-web" 56 | 57 | 58 | class MailDomains(DomainItem): 59 | description = "Mail Domains" 60 | area = "mail" 61 | storage_path = "conf/domains-mail" 62 | 63 | 64 | class FlagItem(U7Mixin, TakeoutItem): 65 | """ 66 | A flag like "is the spamfilter enabled?". It provide a status/enable/disable 67 | interface via a uberspace sub-command. Provide the uberspace command without 68 | leading "uberspace" in `cmd` (e.g. ['web', 'log', 'access']). 69 | """ 70 | 71 | def takeout(self): 72 | out = self.run_uberspace(*(self.cmd + ["status"])) 73 | if "enabled" in out: 74 | status = "enable" 75 | else: 76 | status = "disable" 77 | 78 | self.storage.store_text(status, self.storage_path) 79 | 80 | def takein(self): 81 | try: 82 | data = self.storage.unstore_text(self.storage_path) 83 | except FileNotFoundError: 84 | return 85 | 86 | if data not in ("enable", "disable"): 87 | raise Exception( 88 | 'invalid "uberspace {}" value: {}, expected "enabled" or "disabled".'.format( 89 | " ".join(self.cmd), data, 90 | ) 91 | ) 92 | 93 | self.run_uberspace(*(self.cmd + [data])) 94 | 95 | 96 | class AccessLogItem(FlagItem): 97 | description = "Setting: Access-Log" 98 | cmd = ["web", "log", "access"] 99 | storage_path = "conf/log-access" 100 | 101 | 102 | class ApacheErrorLogItem(FlagItem): 103 | description = "Setting: Apache-Error-Log" 104 | cmd = ["web", "log", "apache_error"] 105 | storage_path = "conf/log-apache_error" 106 | 107 | 108 | class PhpErrorLogItem(FlagItem): 109 | description = "Setting: PHP-Error-Log" 110 | cmd = ["web", "log", "php_error"] 111 | storage_path = "conf/log-php_error" 112 | 113 | 114 | class SpamfilterLogItem(FlagItem): 115 | description = "Setting: Spamfilter" 116 | cmd = ["mail", "spamfilter"] 117 | storage_path = "conf/spamfilter-enabled" 118 | 119 | 120 | class ToolVersions(U7Mixin, TakeoutItem): 121 | description = "Setting: Tool Versions" 122 | 123 | def takeout(self): 124 | tools = self.run_uberspace("tools", "version", "list") 125 | 126 | for tool in tools: 127 | tool = tool.lstrip("- ") 128 | out = self.run_uberspace("tools", "version", "show", tool) 129 | version = re.search(r"'([0-9\.]+)'", out[0]).groups()[0] 130 | self.storage.store_text(version, "conf/tool-version/" + tool) 131 | 132 | def takein(self): 133 | try: 134 | tools = self.storage.list_files("conf/tool-versions") 135 | except FileNotFoundError: 136 | return 137 | 138 | for tool in tools: 139 | version = self.storage.unstore_text("conf/tool-versions/" + tool) 140 | self.run_uberspace("tools", "version", "use", tool, version) 141 | -------------------------------------------------------------------------------- /test/test_storage.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | from uberspace_takeout.storage import LocalMoveStorage 4 | from uberspace_takeout.storage import Storage 5 | from uberspace_takeout.storage import TarStorage 6 | 7 | 8 | @pytest.mark.parametrize("mode", ["takeout", "takein",]) 9 | def test_storage_ctor(mode): 10 | s = Storage("dest.tar.gz", mode) 11 | assert s.destination == "dest.tar.gz" 12 | assert s.mode == mode 13 | 14 | 15 | def test_storage_ctor_mode(): 16 | with pytest.raises(Exception) as ex: 17 | Storage("dest.tar.gz", "foo") 18 | 19 | assert "Invalid mode foo" in str(ex) 20 | 21 | 22 | @pytest.fixture 23 | def test_file(tmp_path): 24 | path = tmp_path / "some_file.txt" 25 | 26 | with path.open("w") as f: 27 | f.write(u"some file text") 28 | 29 | return path 30 | 31 | 32 | @pytest.fixture 33 | def test_file2(tmp_path): 34 | path = tmp_path / "some_file2.txt" 35 | 36 | with path.open("w") as f: 37 | f.write(u"some file text2") 38 | 39 | return path 40 | 41 | 42 | @pytest.fixture 43 | def test_dir(tmp_path): 44 | (tmp_path / "some_dir/some_subdir").mkdir(parents=True) 45 | 46 | with (tmp_path / "some_dir/file.txt").open("w") as f: 47 | f.write(u"some file text") 48 | 49 | with (tmp_path / "some_dir/some_subdir/file2.txt").open("w") as f: 50 | f.write(u"some file text 2") 51 | 52 | return tmp_path / "some_dir" 53 | 54 | 55 | storages = [ 56 | TarStorage, 57 | LocalMoveStorage, 58 | ] 59 | 60 | 61 | @pytest.mark.parametrize("storage", storages) 62 | def test_storage_text(storage, tmp_path): 63 | with storage(tmp_path / "test.tar.gz", "takeout") as s: 64 | s.store_text("some text 1", "simple_text.txt") 65 | s.store_text("some text 2", "subdir/bla/simple_text.txt") 66 | 67 | with storage(tmp_path / "test.tar.gz", "takein") as s: 68 | assert s.unstore_text("simple_text.txt") == "some text 1" 69 | assert s.unstore_text("subdir/bla/simple_text.txt") == "some text 2" 70 | 71 | 72 | @pytest.mark.parametrize("storage", storages) 73 | def test_storage_file(storage, tmp_path, test_file, test_file2): 74 | with storage(tmp_path / "test.tar.gz", "takeout") as s: 75 | s.store_file(test_file, "simple_file.txt") 76 | s.store_file(test_file2, "subdir/bla/simple_file.txt") 77 | 78 | with storage(tmp_path / "test.tar.gz", "takein") as s: 79 | assert s.unstore_text("simple_file.txt") == "some file text" 80 | # read twice to check hat we don't break the tarinfo objects 81 | assert s.unstore_text("subdir/bla/simple_file.txt") == "some file text2" 82 | assert s.unstore_text("subdir/bla/simple_file.txt") == "some file text2" 83 | 84 | s.unstore_file("simple_file.txt", tmp_path / "extracted.txt") 85 | 86 | with (tmp_path / "extracted.txt").open() as f: 87 | assert f.read() == "some file text" 88 | 89 | 90 | @pytest.mark.parametrize("storage", storages) 91 | def test_storage_file_eexists(storage, tmp_path, test_file): 92 | with storage(tmp_path / "test.tar.gz", "takeout") as s: 93 | s.store_file(test_file, "simple_file.txt") 94 | 95 | test_file.touch() 96 | 97 | with pytest.raises(FileExistsError): 98 | s.store_file(test_file, "simple_file.txt") 99 | 100 | 101 | @pytest.mark.parametrize("storage", storages) 102 | def test_storage_file_noent(storage, tmp_path, test_file): 103 | with storage(tmp_path / "test.tar.gz", "takeout") as s: 104 | pass 105 | 106 | with storage(tmp_path / "test.tar.gz", "takein") as s: 107 | with pytest.raises(FileNotFoundError): 108 | s.unstore_file("simple_file.txt", tmp_path / "somefile") 109 | 110 | 111 | @pytest.mark.parametrize("storage", storages) 112 | def test_storage_directory(storage, tmp_path, test_dir): 113 | with storage(tmp_path / "test.tar.gz", "takeout") as s: 114 | s.store_directory(test_dir, "dir") 115 | 116 | with storage(tmp_path / "test.tar.gz", "takein") as s: 117 | assert s.unstore_text("dir/some_subdir/file2.txt") == "some file text 2" 118 | 119 | s.unstore_directory("dir", tmp_path / "new_dir") 120 | 121 | with (tmp_path / "new_dir" / "file.txt").open() as f: 122 | assert f.read() == "some file text" 123 | 124 | with (tmp_path / "new_dir" / "some_subdir/file2.txt").open() as f: 125 | assert f.read() == "some file text 2" 126 | 127 | 128 | @pytest.mark.parametrize("storage", storages) 129 | def test_storage_directory_eexists(storage, tmp_path, test_dir): 130 | with storage(tmp_path / "test.tar.gz", "takeout") as s: 131 | s.store_directory(test_dir, "dir") 132 | 133 | test_dir.mkdir(exist_ok=True) 134 | 135 | with pytest.raises(FileExistsError): 136 | s.store_directory(test_dir, "dir") 137 | 138 | 139 | @pytest.mark.parametrize("storage", storages) 140 | def test_storage_directory_noent(storage, tmp_path): 141 | with storage(tmp_path / "test.tar.gz", "takeout") as s: 142 | pass 143 | 144 | with storage(tmp_path / "test.tar.gz", "takein") as s: 145 | with pytest.raises(FileNotFoundError): 146 | s.unstore_directory("dir", tmp_path / "new_dir") 147 | 148 | 149 | @pytest.mark.parametrize("storage", storages) 150 | def test_storage_list_files(storage, tmp_path, test_dir): 151 | with storage(tmp_path / "test.tar.gz", "takeout") as s: 152 | s.store_directory(test_dir, "dir") 153 | assert sorted(s.list_files("dir")) == ["file.txt", "some_subdir"] 154 | assert sorted(s.list_files("dir/some_subdir")) == ["file2.txt"] 155 | 156 | 157 | @pytest.mark.parametrize("storage", storages) 158 | def test_storage_list_files_noent(storage, tmp_path, test_dir): 159 | with storage(tmp_path / "test.tar.gz", "takeout") as s: 160 | pass 161 | 162 | with storage(tmp_path / "test.tar.gz", "takeout") as s: 163 | with pytest.raises(FileNotFoundError): 164 | s.list_files("dir") 165 | 166 | 167 | @pytest.mark.parametrize("storage", storages) 168 | def test_storage_slashes(storage, tmp_path, test_file): 169 | with storage(tmp_path / "test.tar.gz", "takeout") as s: 170 | s.store_text("some text 3", "/subdir/bla/simple_text2.txt") 171 | s.store_file(test_file, "/subdir/bla/simple_file2.txt") 172 | 173 | with storage(tmp_path / "test.tar.gz", "takein") as s: 174 | assert s.unstore_text("/subdir/bla/simple_text2.txt") == "some text 3" 175 | assert s.unstore_text("/subdir/bla/simple_file2.txt") == "some file text" 176 | assert s.unstore_text("subdir/bla/simple_text2.txt") == "some text 3" 177 | assert s.unstore_text("subdir/bla/simple_file2.txt") == "some file text" 178 | 179 | 180 | def test_localmovestorage_file_removed(tmp_path, test_file): 181 | with LocalMoveStorage(tmp_path / "test.tar.gz", "takeout") as s: 182 | assert test_file.exists() 183 | s.store_file(test_file, "simple_file.txt") 184 | assert not test_file.exists() 185 | 186 | 187 | def test_localmovestorage_directory_removed(tmp_path, test_dir): 188 | with LocalMoveStorage(tmp_path / "test.tar.gz", "takeout") as s: 189 | assert test_dir.exists() 190 | s.store_directory(test_dir, "dir") 191 | assert not test_dir.exists() 192 | -------------------------------------------------------------------------------- /test/test_takeout.py: -------------------------------------------------------------------------------- 1 | import os 2 | import shutil 3 | from pathlib import Path 4 | 5 | import pytest 6 | from pyfakefs.fake_filesystem_unittest import Pause 7 | 8 | from uberspace_takeout import Takeout 9 | 10 | 11 | def prefix_root(prefix): 12 | return Path(__file__).parent / "uberspaces" / prefix 13 | 14 | 15 | def populate_root(fs, prefix): 16 | outside_root = prefix_root(prefix) 17 | 18 | with Pause(fs): 19 | for dir in os.listdir(outside_root): 20 | if not os.path.isdir(outside_root / dir): 21 | raise NotImplementedError( 22 | "currently only directories are supported at root level" 23 | ) 24 | if dir == "commands": 25 | continue 26 | 27 | try: 28 | fs.remove_object("/" + dir) 29 | except FileNotFoundError: 30 | pass 31 | 32 | fs.add_real_directory( 33 | outside_root / dir, 34 | lazy_read=False, 35 | read_only=False, 36 | target_path="/" + dir, 37 | ) 38 | 39 | 40 | def clean_root(skip_dirs=["tmp", "etc"]): 41 | for path in os.listdir("/"): 42 | if path not in skip_dirs: 43 | shutil.rmtree("/" + path) 44 | 45 | Path("/home").mkdir() 46 | Path("/var/www/virtual").mkdir(parents=True) 47 | 48 | assert not os.listdir("/home") 49 | assert not os.listdir("/var/www/virtual") 50 | 51 | 52 | @pytest.fixture 53 | def mock_run_command(fs, mocker): 54 | class Commands: 55 | def __init__(self, *args, **kwargs): 56 | self.called = {} 57 | self.commands = {} 58 | 59 | self._mock() 60 | 61 | def clear(self): 62 | self.called.clear() 63 | self.commands.clear() 64 | 65 | def add_prefix_commands(self, prefix): 66 | commands = prefix_root(prefix) / "commands" 67 | 68 | with Pause(fs): 69 | for cmd in os.listdir(commands): 70 | with open(commands / cmd) as f: 71 | self.add_command(cmd, f.read()) 72 | 73 | def add_command(self, command, output=""): 74 | if command in self.commands: 75 | raise Exception("Command '{}' is already defined.".format(command)) 76 | if isinstance(command, list): 77 | command = " ".join(command) 78 | 79 | self.commands[command] = output 80 | 81 | def assert_called(self, command, input_text=None): 82 | if isinstance(command, list): 83 | command = " ".join(command) 84 | 85 | assert command in self.called 86 | 87 | if input_text: 88 | assert self.called[command] == input_text 89 | 90 | del self.called[command] 91 | 92 | def assert_no_unexpected(self): 93 | assert not self.called, "unexpected commands executed" 94 | 95 | def _mock(self): 96 | def _run_command(_, cmd, input_text=None, *args, **kwargs): 97 | cmd_str = " ".join(cmd) 98 | self.called[cmd_str] = input_text 99 | 100 | if cmd_str in self.commands: 101 | output = self.commands[cmd_str] 102 | return [l.rstrip() for l in output.split("\n") if l.rstrip()] 103 | else: 104 | return [] 105 | 106 | mocker.patch( 107 | "uberspace_takeout.items.base.TakeoutItem.run_command", _run_command 108 | ) 109 | 110 | return Commands() 111 | 112 | 113 | def content(path): 114 | with open(path) as f: 115 | return f.read() 116 | 117 | 118 | def assert_in_file(path, text): 119 | assert text in content(path) 120 | 121 | 122 | def assert_files_equal(path1, path2): 123 | assert content(path1) == content(path2) 124 | 125 | 126 | def assert_file_unchanged(path, fs, prefix): 127 | with Pause(fs): 128 | original = content(prefix_root(prefix) / path.lstrip("/")) 129 | 130 | assert original == content(path) 131 | 132 | 133 | def test_takeout_u6_to_u6(fs, mock_run_command): 134 | populate_root(fs, "u6/isabell") 135 | mock_run_command.add_prefix_commands("u6/isabell") 136 | 137 | takeout = Takeout(hostname="andromeda.uberspace.de") 138 | 139 | takeout.takeout("/tmp/test.tar.gz", "isabell") 140 | 141 | clean_root() 142 | 143 | mock_run_command.clear() 144 | 145 | takeout.takein("/tmp/test.tar.gz", "isabell") 146 | 147 | mock_run_command.assert_called( 148 | "mysql --defaults-group-suffix= -e SET PASSWORD = PASSWORD('Lei4e%ngekäe3iÖt4Ies')" 149 | ) 150 | assert_in_file("/home/isabell/.my.cnf", "Lei4e%ngekäe3iÖt4Ies") 151 | 152 | mock_run_command.assert_called("uberspace-add-domain -w -d *.example.com") 153 | mock_run_command.assert_called("uberspace-add-domain -w -d example.com") 154 | mock_run_command.assert_called("uberspace-add-domain -w -d foo.example.com") 155 | mock_run_command.assert_called("uberspace-add-domain -m -d mail.example.com") 156 | 157 | mock_run_command.assert_called("crontab -", "@daily echo good morning\n") 158 | 159 | mock_run_command.assert_no_unexpected() 160 | 161 | assert_file_unchanged("/var/www/virtual/isabell/html/index.html", fs, "u6/isabell") 162 | assert_file_unchanged( 163 | "/var/www/virtual/isabell/html/blog/index.html", fs, "u6/isabell" 164 | ) 165 | assert os.path.islink("/home/isabell/html") 166 | assert_file_unchanged("/home/isabell/html/index.html", fs, "u6/isabell") 167 | assert_file_unchanged("/home/isabell/Maildir/cur/mail-888", fs, "u6/isabell") 168 | 169 | 170 | def test_takeout_u6_to_u7(fs, mock_run_command): 171 | populate_root(fs, "u6/isabell") 172 | mock_run_command.add_prefix_commands("u6/isabell") 173 | 174 | takeout = Takeout(hostname="andromeda.uberspace.de") 175 | 176 | takeout.takeout("/tmp/test.tar.gz", "isabell") 177 | 178 | clean_root() 179 | mock_run_command.clear() 180 | populate_root(fs, "u7/empty") 181 | 182 | takeout.takein("/tmp/test.tar.gz", "isabell") 183 | 184 | mock_run_command.assert_called( 185 | "mysql --defaults-group-suffix= -e SET PASSWORD = PASSWORD('Lei4e%ngekäe3iÖt4Ies')" 186 | ) 187 | assert_in_file("/home/isabell/.my.cnf", "Lei4e%ngekäe3iÖt4Ies") 188 | 189 | mock_run_command.assert_called("uberspace web domain add example.com") 190 | mock_run_command.assert_called("uberspace web domain add foo.example.com") 191 | mock_run_command.assert_called("uberspace web domain add ep.isabell.uber.space") 192 | mock_run_command.assert_called("uberspace web domain add unknown-domain.com") 193 | mock_run_command.assert_called("uberspace mail domain add mail.example.com") 194 | 195 | mock_run_command.assert_called("crontab -", "@daily echo good morning\n") 196 | 197 | mock_run_command.assert_no_unexpected() 198 | 199 | assert_file_unchanged("/var/www/virtual/isabell/html/index.html", fs, "u6/isabell") 200 | assert_file_unchanged( 201 | "/var/www/virtual/isabell/html/blog/index.html", fs, "u6/isabell" 202 | ) 203 | assert os.path.islink("/home/isabell/html") 204 | assert_file_unchanged("/home/isabell/html/index.html", fs, "u6/isabell") 205 | assert_file_unchanged("/home/isabell/Maildir/cur/mail-888", fs, "u6/isabell") 206 | -------------------------------------------------------------------------------- /uberspace_takeout/storage.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | import errno 3 | import os 4 | import tarfile 5 | 6 | try: 7 | from BytesIO import BytesIO 8 | except ImportError: 9 | from io import BytesIO 10 | 11 | 12 | class Storage: 13 | def __init__(self, destination, mode): 14 | if mode not in ("takein", "takeout"): 15 | raise Exception( 16 | 'Invalid mode {}, expected "takein" or "takeout".'.format(mode) 17 | ) 18 | 19 | self.destination = str(destination) 20 | self.mode = mode 21 | 22 | def __enter__(self): 23 | raise NotImplementedError() 24 | 25 | def __exit__(self, exception_type, exception_value, traceback): 26 | raise NotImplementedError() 27 | 28 | def list_files(self, storage_path): 29 | raise NotImplementedError() 30 | 31 | def store_text(self, content, storage_path): 32 | raise NotImplementedError() 33 | 34 | def unstore_text(self, storage_path): 35 | raise NotImplementedError() 36 | 37 | def store_file(self, system_path, storage_path): 38 | raise NotImplementedError() 39 | 40 | def unstore_file(self, storage_path, system_path): 41 | raise NotImplementedError() 42 | 43 | def store_directory(self, system_path, storage_path): 44 | return self.store_file(system_path, storage_path) 45 | 46 | def unstore_directory(self, storage_path, system_path): 47 | return self.unstore_file(storage_path, system_path) 48 | 49 | 50 | class TarStorage(Storage): 51 | def __enter__(self): 52 | mode = "w:bz2" if self.mode == "takeout" else "r:bz2" 53 | self.tar = tarfile.open(self.destination, mode) 54 | return self 55 | 56 | def __exit__(self, exception_type, exception_value, traceback): 57 | self.tar.close() 58 | 59 | def _check_member_type(self, member): 60 | if member.type not in (tarfile.REGTYPE, tarfile.SYMTYPE, tarfile.DIRTYPE): 61 | raise Exception( 62 | "tar member has illegal type: {}. " 63 | "Must be tarfile.REGTYPE/file, SYMTYPE/symlink or DIRTYPE/directory, " 64 | "but is {}".format(member.name, member.type) 65 | ) 66 | 67 | def clone_tarinfo(self, tarinfo): 68 | # "clone" the object so we don't modify names inside the tar 69 | tarinfo2 = tarfile.TarInfo() 70 | for attr in (*tarinfo.get_info().keys(), "offset", "offset_data"): 71 | setattr(tarinfo2, attr, getattr(tarinfo, attr)) 72 | return tarinfo2 73 | 74 | def get_members_in(self, directory): 75 | directory = directory.rstrip("/") + "/" 76 | 77 | for m in self.tar.getmembers(): 78 | if ".." in m.name: 79 | raise Exception( 80 | 'tar member has illegal name (contains ".."): ' + m.name 81 | ) 82 | if m.name.startswith("/"): 83 | raise Exception( 84 | 'tar member has illegal name (starts with "/"): ' + m.name 85 | ) 86 | if m.name.startswith("./"): 87 | raise Exception( 88 | 'tar member has illegal name (starts with "./"): ' + m.name 89 | ) 90 | 91 | self._check_member_type(m) 92 | 93 | if m.name.startswith(directory): 94 | m = self.clone_tarinfo(m) 95 | # files might be stored as /www/domain.com/something.html, but need to be extracted 96 | # as domain.com/something.html. 97 | m.name = m.name[len(directory) :] 98 | yield m 99 | 100 | def has_member(self, path): 101 | for m in self.tar.getmembers(): 102 | self._check_member_type(m) 103 | 104 | if m.name == path: 105 | return True 106 | 107 | return False 108 | 109 | def get_member(self, path): 110 | matching = [] 111 | 112 | for m in self.tar.getmembers(): 113 | self._check_member_type(m) 114 | 115 | if m.name == path: 116 | matching.append(m) 117 | 118 | if len(matching) == 0: 119 | raise FileNotFoundError() 120 | if len(matching) > 1: 121 | raise Exception( 122 | "There are {} files matching the path {}. Expected only one.".format( 123 | len(matching), path 124 | ) 125 | ) 126 | 127 | return self.clone_tarinfo(matching[0]) 128 | 129 | @classmethod 130 | def _len(cls, f): 131 | old_position = f.tell() 132 | f.seek(0, os.SEEK_END) 133 | length = f.tell() 134 | f.seek(old_position) 135 | return length 136 | 137 | def list_files(self, storage_path): 138 | members = list(self.get_members_in(storage_path)) 139 | if not members: 140 | raise FileNotFoundError() 141 | return [m.name for m in members if "/" not in m.name] 142 | 143 | def store_text(self, content, storage_path): 144 | storage_path = str(storage_path).lstrip("/") 145 | content = BytesIO(content.encode("utf-8")) 146 | info = tarfile.TarInfo(storage_path) 147 | info.size = self._len(content) 148 | info.mtime = int(datetime.datetime.now().strftime("%s")) 149 | self.tar.addfile(info, content) 150 | 151 | def unstore_text(self, storage_path): 152 | storage_path = str(storage_path).lstrip("/") 153 | if not self.has_member(storage_path): 154 | raise FileNotFoundError() 155 | return self.tar.extractfile(storage_path).read().decode("utf-8") 156 | 157 | def store_file(self, system_path, storage_path): 158 | storage_path = str(storage_path).lstrip("/") 159 | if self.has_member(storage_path): 160 | raise FileExistsError() 161 | self.tar.add(str(system_path), storage_path) 162 | 163 | def unstore_directory(self, storage_path, system_path): 164 | storage_path = str(storage_path).lstrip("/") 165 | members = list(self.get_members_in(storage_path)) 166 | if not members: 167 | raise FileNotFoundError() 168 | self.tar.extractall(system_path, members) 169 | 170 | def unstore_file(self, storage_path, system_path): 171 | storage_path = str(storage_path).lstrip("/") 172 | member = self.get_member(storage_path) 173 | member.name = os.path.basename(system_path) 174 | self.tar.extractall(os.path.dirname(system_path), [member]) 175 | 176 | 177 | class LocalMoveStorage(Storage): 178 | def __enter__(self): 179 | return self 180 | 181 | def __exit__(self, exception_type, exception_value, traceback): 182 | pass 183 | 184 | def _storage_path(self, storage_path): 185 | storage_path = str(storage_path).lstrip("/") 186 | return self.destination + "/" + storage_path 187 | 188 | def _mkdir_p(self, path): 189 | if not path: 190 | return 191 | 192 | try: 193 | os.makedirs(path) 194 | except OSError as exc: 195 | if exc.errno == errno.EEXIST and os.path.isdir(path): 196 | pass 197 | else: 198 | raise 199 | 200 | def list_files(self, storage_path): 201 | storage_path = self._storage_path(storage_path) 202 | if not os.path.exists(storage_path): 203 | raise FileNotFoundError() 204 | return os.listdir(storage_path) 205 | 206 | def store_text(self, content, storage_path): 207 | storage_path = self._storage_path(storage_path) 208 | if os.path.exists(storage_path): 209 | raise FileExistsError() 210 | self._mkdir_p(os.path.dirname(storage_path)) 211 | with open(storage_path, "w") as f: 212 | f.write(content) 213 | 214 | def unstore_text(self, storage_path): 215 | with open(self._storage_path(storage_path)) as f: 216 | return f.read() 217 | 218 | def store_file(self, system_path, storage_path): 219 | storage_path = self._storage_path(storage_path) 220 | if os.path.exists(storage_path): 221 | raise FileExistsError() 222 | self._mkdir_p(os.path.dirname(storage_path)) 223 | os.rename(system_path, storage_path) 224 | 225 | def unstore_file(self, storage_path, system_path): 226 | storage_path = self._storage_path(storage_path) 227 | if not os.path.exists(storage_path): 228 | raise FileNotFoundError() 229 | self._mkdir_p(os.path.dirname(system_path)) 230 | os.rename(storage_path, system_path) 231 | --------------------------------------------------------------------------------