├── MANIFEST.in ├── migrate_anything ├── tests │ ├── __init__.py │ ├── migrations │ │ ├── __init__.py │ │ └── 01-test.py │ ├── common.py │ ├── test_storage.py │ └── test_migrator.py ├── __init__.py ├── log.py ├── main.py ├── migrator.py └── storage │ └── __init__.py ├── pyproject.toml ├── examples ├── mongodb │ ├── test-in-docker.sh │ ├── migrations │ │ ├── __init__.py │ │ └── 01-test.py │ ├── Dockerfile │ ├── docker-test.bat │ ├── docker-test.sh │ ├── README.md │ ├── test.sh │ └── test.bat ├── simple │ ├── migrations │ │ ├── __init__.py │ │ └── 01-test.py │ ├── README.md │ ├── test.sh │ └── test.bat ├── custom_storage │ ├── README.md │ ├── migrations │ │ ├── 01-test.py │ │ ├── 02-test.py │ │ ├── 03-test.py │ │ └── __init__.py │ ├── test.sh │ └── test.bat └── arangodb │ ├── README.md │ ├── migrations │ ├── __init__.py │ └── 01-test.py │ ├── test.sh │ └── test.bat ├── setup.cfg ├── dev-requirements.txt ├── .gitignore ├── .pre-commit-config.yaml ├── sonar-project.properties ├── setup.py ├── LICENSE ├── .github ├── CONTRIBUTING.md └── workflows │ └── test-and-release.yaml └── README.rst /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include README.rst 2 | -------------------------------------------------------------------------------- /migrate_anything/tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [tool.black] 2 | target_version = ["py36"] 3 | -------------------------------------------------------------------------------- /migrate_anything/__init__.py: -------------------------------------------------------------------------------- 1 | from migrate_anything.migrator import configure 2 | from migrate_anything.storage import * 3 | -------------------------------------------------------------------------------- /examples/mongodb/test-in-docker.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | cd /test 4 | pip install -r requirements.txt 5 | sh test.sh 6 | -------------------------------------------------------------------------------- /examples/simple/migrations/__init__.py: -------------------------------------------------------------------------------- 1 | from migrate_anything import configure, CSVStorage 2 | 3 | configure(storage=CSVStorage("test.csv")) 4 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [metadata] 2 | license_files = LICENSE 3 | 4 | [bdist_wheel] 5 | # One package to rule them all .. I mean run on Py2 and Py3 6 | universal=1 7 | -------------------------------------------------------------------------------- /dev-requirements.txt: -------------------------------------------------------------------------------- 1 | codecov==2.1.13 2 | pytest>=7.0.0 3 | twine>=1.15.0 4 | wheel==0.38.1 5 | mongomock==4.3.0 6 | pymongo==4.1.1 7 | python-arango==5.2.1 8 | -------------------------------------------------------------------------------- /examples/custom_storage/README.md: -------------------------------------------------------------------------------- 1 | # Migrations with custom storage 2 | 3 | Demonstrates use of custom storage. 4 | 5 | To test run `test.bat` (on Windows) / `test.sh` (*nix). 6 | -------------------------------------------------------------------------------- /examples/simple/README.md: -------------------------------------------------------------------------------- 1 | # Simple migration example 2 | 3 | Just demonstrates how you can basically run any code using 4 | migrate-anything. 5 | 6 | To test run `test.bat` (on Windows) / `test.sh` (*nix). 7 | -------------------------------------------------------------------------------- /migrate_anything/tests/migrations/__init__.py: -------------------------------------------------------------------------------- 1 | from migrate_anything import configure, CSVStorage 2 | from migrate_anything.tests.test_migrator import TEST_CSV 3 | 4 | configure(storage=CSVStorage(TEST_CSV)) 5 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .idea 2 | __pycache__ 3 | .pytest_cache 4 | *.pyc 5 | *.bak 6 | *.log 7 | examples/**/*.txt 8 | *.csv 9 | *.egg-info 10 | build 11 | dist 12 | MANIFEST 13 | .coverage 14 | coverage.xml 15 | .venv 16 | -------------------------------------------------------------------------------- /examples/mongodb/migrations/__init__.py: -------------------------------------------------------------------------------- 1 | from migrate_anything import configure, MongoDBStorage 2 | import pymongo 3 | 4 | db = pymongo.MongoClient().test_db 5 | 6 | configure(storage=MongoDBStorage(db.migrations)) 7 | -------------------------------------------------------------------------------- /examples/mongodb/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM mongo:3.4.23-xenial 2 | 3 | RUN set -ex \ 4 | && apt-get update \ 5 | && apt-get install -y python python-pip \ 6 | && pip install migrate-anything \ 7 | && apt-get clean 8 | 9 | COPY . /test 10 | -------------------------------------------------------------------------------- /examples/mongodb/docker-test.bat: -------------------------------------------------------------------------------- 1 | @echo off 2 | 3 | docker build . -t migrate-anything-docker-test 4 | docker run --rm --name ma-mongo -d migrate-anything-docker-test 5 | docker exec ma-mongo sh /test/test-in-docker.sh 6 | docker exec ma-mongo kill 1 7 | -------------------------------------------------------------------------------- /examples/mongodb/docker-test.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | docker build . -t migrate-anything-docker-test 4 | docker run --rm --name ma-mongo -d migrate-anything-docker-test 5 | docker exec ma-mongo sh /test/test-in-docker.sh 6 | docker exec ma-mongo kill 1 7 | -------------------------------------------------------------------------------- /migrate_anything/log.py: -------------------------------------------------------------------------------- 1 | from __future__ import absolute_import 2 | 3 | import logging 4 | 5 | LOG_FORMAT = "%(asctime)s (%(process)d) [%(levelname)8s]: %(message)s" 6 | 7 | logging.basicConfig(format=LOG_FORMAT, level="INFO") 8 | logger = logging.getLogger(__name__) 9 | -------------------------------------------------------------------------------- /examples/arangodb/README.md: -------------------------------------------------------------------------------- 1 | # ArangoDB example 2 | 3 | Just demonstrates how you can basically run any code using 4 | migrate-anything. 5 | 6 | To test run `test.bat` (on Windows) / `test.sh` (*nix). Make sure you've 7 | installed the dependencies: `pip install -r requirements.txt`. 8 | -------------------------------------------------------------------------------- /examples/arangodb/migrations/__init__.py: -------------------------------------------------------------------------------- 1 | from arango import ArangoClient 2 | from migrate_anything import configure, ArangoDBStorage 3 | 4 | client = ArangoClient(hosts="https://example.com:8529/") 5 | db = client.db(name="my_db", username="root", password="supersecret") 6 | 7 | configure(storage=ArangoDBStorage(collection="migrations", db=db)) 8 | -------------------------------------------------------------------------------- /examples/mongodb/README.md: -------------------------------------------------------------------------------- 1 | # MongoDB example 2 | 3 | Just demonstrates how you can basically run any code using 4 | migrate-anything. 5 | 6 | To test run `test.bat` (on Windows) / `test.sh` (*nix). Make sure you've 7 | installed the dependencies: `pip install -r requirements.txt`. 8 | 9 | To test in Docker try `docker-test.bat` or `docker-test.sh`. 10 | -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- 1 | # See https://pre-commit.com for more information 2 | # See https://pre-commit.com/hooks.html for more hooks 3 | repos: 4 | - repo: https://github.com/pre-commit/pre-commit-hooks 5 | rev: v4.3.0 6 | hooks: 7 | - id: trailing-whitespace 8 | - id: end-of-file-fixer 9 | - repo: https://github.com/psf/black 10 | rev: 22.3.0 11 | hooks: 12 | - id: black 13 | -------------------------------------------------------------------------------- /examples/mongodb/migrations/01-test.py: -------------------------------------------------------------------------------- 1 | from pymongo import MongoClient 2 | 3 | client = MongoClient() 4 | db = client.my_db 5 | 6 | 7 | def up(): 8 | db.posts.insert_one( 9 | { 10 | "id": "post-1", 11 | "title": "We're live!", 12 | "content": "This is our first post, yay.", 13 | } 14 | ) 15 | db.posts.create_index("id") 16 | 17 | 18 | def down(): 19 | db.posts.drop() 20 | -------------------------------------------------------------------------------- /examples/simple/migrations/01-test.py: -------------------------------------------------------------------------------- 1 | from os import remove 2 | from time import time 3 | 4 | file = "test-file.txt" 5 | 6 | 7 | def up(): 8 | with open(file, "w") as f: 9 | f.write(str(time())) 10 | 11 | 12 | def down(): 13 | with open(file) as f: 14 | old = float(f.read()) 15 | diff = abs(time() - old) 16 | if diff > 1: 17 | raise Exception("Something is wrong") 18 | remove(file) 19 | -------------------------------------------------------------------------------- /migrate_anything/tests/migrations/01-test.py: -------------------------------------------------------------------------------- 1 | from os import remove 2 | from time import time 3 | 4 | file = "test-file.txt" 5 | 6 | 7 | def up(): 8 | with open(file, "w") as f: 9 | f.write(str(time())) 10 | 11 | 12 | def down(): 13 | with open(file) as f: 14 | old = float(f.read()) 15 | diff = abs(time() - old) 16 | if diff > 0.5: 17 | raise Exception("Something is wrong") 18 | remove(file) 19 | -------------------------------------------------------------------------------- /examples/custom_storage/migrations/01-test.py: -------------------------------------------------------------------------------- 1 | from os import remove 2 | from time import time 3 | 4 | file = "test-file.txt" 5 | 6 | 7 | def up(): 8 | with open(file, "w") as f: 9 | f.write(str(time())) 10 | 11 | 12 | def down(): 13 | with open(file) as f: 14 | old = float(f.read()) 15 | diff = abs(time() - old) 16 | if diff > 0.5: 17 | raise Exception("Something is wrong") 18 | remove(file) 19 | -------------------------------------------------------------------------------- /examples/custom_storage/migrations/02-test.py: -------------------------------------------------------------------------------- 1 | from os import remove 2 | from time import time 3 | 4 | file = "test-file.txt" 5 | 6 | 7 | def up(): 8 | with open(file, "w") as f: 9 | f.write(str(time())) 10 | 11 | 12 | def down(): 13 | with open(file) as f: 14 | old = float(f.read()) 15 | diff = abs(time() - old) 16 | if diff > 0.5: 17 | raise Exception("Something is wrong") 18 | remove(file) 19 | -------------------------------------------------------------------------------- /examples/custom_storage/migrations/03-test.py: -------------------------------------------------------------------------------- 1 | from os import remove 2 | from time import time 3 | 4 | file = "test-file.txt" 5 | 6 | 7 | def up(): 8 | with open(file, "w") as f: 9 | f.write(str(time())) 10 | 11 | 12 | def down(): 13 | with open(file) as f: 14 | old = float(f.read()) 15 | diff = abs(time() - old) 16 | if diff > 0.5: 17 | raise Exception("Something is wrong") 18 | remove(file) 19 | -------------------------------------------------------------------------------- /examples/arangodb/test.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | set -e 3 | 4 | echo Running migrations 5 | echo 6 | 7 | migrate-anything migrations 8 | 9 | echo 10 | echo Applying new migration 11 | echo 12 | 13 | cp migrations/01-test.py migrations/02-test.py 14 | migrate-anything migrations 15 | 16 | echo 17 | echo Undoing old migration 18 | echo 19 | 20 | rm -f migrations/02-test.py 21 | migrate-anything migrations 22 | if [ -f "test-file.txt" ]; then 23 | echo Undo did not work. 24 | exit 1 25 | fi 26 | -------------------------------------------------------------------------------- /sonar-project.properties: -------------------------------------------------------------------------------- 1 | sonar.projectKey=cocreators-ee_migrate-anything 2 | sonar.organization=cocreators-ee 3 | 4 | # this is the name and version displayed in the SonarCloud UI. 5 | sonar.projectName=migrate-anything 6 | sonar.projectVersion=1.0 7 | 8 | # Path is relative to the sonar-project.properties file. Replace "\" by "/" on Windows. 9 | # This property is optional if sonar.modules is set. 10 | sonar.sources=. 11 | 12 | # Encoding of the source code. Default is default system encoding 13 | sonar.sourceEncoding=UTF-8 14 | -------------------------------------------------------------------------------- /examples/mongodb/test.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | set -e 3 | 4 | echo Running migrations 5 | echo 6 | 7 | migrate-anything migrations 8 | 9 | echo 10 | echo Applying new migration 11 | echo 12 | 13 | cp migrations/01-test.py migrations/02-test.py 14 | migrate-anything migrations 15 | 16 | echo 17 | echo Undoing old migration 18 | echo 19 | 20 | rm -f migrations/02-test.py 21 | find . -name "*.pyc" -delete 22 | migrate-anything migrations 23 | if [ -f "test-file.txt" ]; then 24 | echo Undo did not work. 25 | exit 1 26 | fi 27 | -------------------------------------------------------------------------------- /examples/simple/test.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | set -e 3 | 4 | echo Running migrations 5 | echo 6 | 7 | migrate-anything migrations 8 | 9 | echo 10 | echo Applying new migration 11 | echo 12 | 13 | cp migrations/01-test.py migrations/02-test.py 14 | migrate-anything migrations 15 | 16 | echo 17 | echo Undoing old migration 18 | echo 19 | 20 | rm -f migrations/02-test.py 21 | migrate-anything migrations 22 | if [ -f "test-file.txt" ]; then 23 | echo Undo did not work. 24 | exit 1 25 | fi 26 | 27 | echo 28 | echo Cleaning up 29 | rm -f test.csv test-file.txt 30 | -------------------------------------------------------------------------------- /examples/custom_storage/test.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | set -e 3 | 4 | echo Running migrations 5 | echo 6 | 7 | migrate-anything migrations 8 | 9 | echo 10 | echo Applying new migration 11 | echo 12 | 13 | cp migrations/01-test.py migrations/02-test.py 14 | migrate-anything migrations 15 | 16 | echo 17 | echo Undoing old migration 18 | echo 19 | 20 | rm -f migrations/02-test.py 21 | migrate-anything migrations 22 | if [ -f "test-file.txt" ]; then 23 | echo Undo did not work. 24 | exit 1 25 | fi 26 | 27 | echo 28 | echo Cleaning up 29 | rm -f test.csv test-file.txt 30 | -------------------------------------------------------------------------------- /examples/arangodb/migrations/01-test.py: -------------------------------------------------------------------------------- 1 | from arango import ArangoClient 2 | 3 | # Don't actually put your credentials in these files as they get stored in 4 | # the DB in readable format. You should rather just import the `db` from some 5 | # other location. 6 | client = ArangoClient(hosts="https://example.com:8529/") 7 | db = client.db(name="my_db", username="root", password="supersecret") 8 | 9 | 10 | def up(): 11 | collection = db.create_collection("test_collection") 12 | collection.insert({"foo": "bar"}) 13 | 14 | 15 | def down(): 16 | db.delete_collection("test_collection") 17 | -------------------------------------------------------------------------------- /examples/arangodb/test.bat: -------------------------------------------------------------------------------- 1 | @echo off 2 | 3 | echo Running migrations 4 | echo. 5 | 6 | migrate-anything migrations 7 | if ERRORLEVEL 1 GOTO ERROR 8 | 9 | 10 | echo. 11 | echo Adding new migration 12 | echo. 13 | 14 | copy migrations\01-test.py migrations\02-test.py 15 | migrate-anything migrations 16 | if ERRORLEVEL 1 GOTO ERROR 17 | 18 | echo. 19 | echo Undoing old migration 20 | echo. 21 | 22 | del migrations\02-test.py 23 | migrate-anything migrations 24 | if ERRORLEVEL 1 GOTO ERROR 25 | if exist test-file.txt ( 26 | echo Undo did not work. 27 | GOTO :ERROR 28 | ) 29 | 30 | GOTO :CLEANUP 31 | 32 | :ERROR 33 | echo Error during test 34 | goto :EOF 35 | 36 | :CLEANUP 37 | -------------------------------------------------------------------------------- /examples/mongodb/test.bat: -------------------------------------------------------------------------------- 1 | @echo off 2 | 3 | echo Running migrations 4 | echo. 5 | 6 | migrate-anything migrations 7 | if ERRORLEVEL 1 GOTO ERROR 8 | 9 | 10 | echo. 11 | echo Adding new migration 12 | echo. 13 | 14 | copy migrations\01-test.py migrations\02-test.py 15 | migrate-anything migrations 16 | if ERRORLEVEL 1 GOTO ERROR 17 | 18 | echo. 19 | echo Undoing old migration 20 | echo. 21 | 22 | del migrations\02-test.py 23 | del migrations\*.pyc 24 | 25 | migrate-anything migrations 26 | if ERRORLEVEL 1 GOTO ERROR 27 | if exist test-file.txt ( 28 | echo Undo did not work. 29 | GOTO :ERROR 30 | ) 31 | 32 | GOTO :CLEANUP 33 | 34 | :ERROR 35 | echo Error during test 36 | goto :EOF 37 | 38 | :CLEANUP 39 | -------------------------------------------------------------------------------- /examples/simple/test.bat: -------------------------------------------------------------------------------- 1 | @echo off 2 | 3 | echo Running migrations 4 | echo. 5 | 6 | migrate-anything migrations 7 | if ERRORLEVEL 1 GOTO ERROR 8 | 9 | 10 | echo. 11 | echo Adding new migration 12 | echo. 13 | 14 | copy migrations\01-test.py migrations\02-test.py 15 | migrate-anything migrations 16 | if ERRORLEVEL 1 GOTO ERROR 17 | 18 | echo. 19 | echo Undoing old migration 20 | echo. 21 | 22 | del migrations\02-test.py 23 | migrate-anything migrations 24 | if ERRORLEVEL 1 GOTO ERROR 25 | if exist test-file.txt ( 26 | echo Undo did not work. 27 | GOTO :ERROR 28 | ) 29 | 30 | GOTO :CLEANUP 31 | 32 | :ERROR 33 | echo Error during test 34 | goto :EOF 35 | 36 | :CLEANUP 37 | echo. 38 | echo Cleaning up 39 | del test.csv 40 | del test-file.txt 41 | -------------------------------------------------------------------------------- /examples/custom_storage/test.bat: -------------------------------------------------------------------------------- 1 | @echo off 2 | 3 | echo Running migrations 4 | echo. 5 | 6 | migrate-anything migrations 7 | if ERRORLEVEL 1 GOTO ERROR 8 | 9 | 10 | echo. 11 | echo Adding new migration 12 | echo. 13 | 14 | copy migrations\01-test.py migrations\02-test.py 15 | migrate-anything migrations 16 | if ERRORLEVEL 1 GOTO ERROR 17 | 18 | echo. 19 | echo Undoing old migration 20 | echo. 21 | 22 | del migrations\02-test.py 23 | migrate-anything migrations 24 | if ERRORLEVEL 1 GOTO ERROR 25 | if exist test-file.txt ( 26 | echo Undo did not work. 27 | GOTO :ERROR 28 | ) 29 | 30 | GOTO :CLEANUP 31 | 32 | :ERROR 33 | echo Error during test 34 | goto :EOF 35 | 36 | :CLEANUP 37 | echo. 38 | echo Cleaning up 39 | del test.csv 40 | del test-file.txt 41 | -------------------------------------------------------------------------------- /migrate_anything/main.py: -------------------------------------------------------------------------------- 1 | from argparse import ArgumentParser 2 | 3 | from migrate_anything.migrator import run 4 | 5 | 6 | def main(): 7 | ap = ArgumentParser( 8 | prog="migrate-anything", 9 | description="Helps manage migrations for databases and anything else", 10 | epilog="For more information check out https://github.com/cocreators-ee/migrate-anything", 11 | ) 12 | 13 | ap.add_argument( 14 | "package", help="The Python package where your migrations are stored" 15 | ) 16 | 17 | ap.add_argument( 18 | "--revert-latest", 19 | action="store_true", 20 | dest="revert", 21 | help="Reverts last migration applied using migration file rather than migration stored in DB", 22 | ) 23 | 24 | options = ap.parse_args() 25 | run(options.package, revert=options.revert) 26 | 27 | 28 | if __name__ == "__main__": 29 | main() 30 | -------------------------------------------------------------------------------- /examples/custom_storage/migrations/__init__.py: -------------------------------------------------------------------------------- 1 | from migrate_anything import configure 2 | 3 | 4 | class CustomStorage(object): 5 | def __init__(self, file): 6 | self.file = file 7 | 8 | def save_migration(self, name, code): 9 | with open(self.file, "a", encoding="utf-8") as file: 10 | file.write("{},{}\n".format(name, code)) 11 | 12 | def list_migrations(self): 13 | try: 14 | with open(self.file, encoding="utf-8") as file: 15 | return [ 16 | line.split(",") 17 | for line in file.readlines() 18 | if line.strip() # Skip empty lines 19 | ] 20 | except FileNotFoundError: 21 | return [] 22 | 23 | def remove_migration(self, name): 24 | migrations = [ 25 | migration for migration in self.list_migrations() if migration[0] != name 26 | ] 27 | 28 | with open(self.file, "w", encoding="utf-8") as file: 29 | for row in migrations: 30 | file.write("{},{}\n".format(*row)) 31 | 32 | 33 | configure(storage=CustomStorage("test.txt")) 34 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | from setuptools import setup 3 | from io import open 4 | from os import path 5 | 6 | here = path.abspath(path.dirname(__file__)) 7 | with open(path.join(here, "README.rst"), encoding="utf-8") as f: 8 | long_description = f.read() 9 | 10 | setup( 11 | name="migrate-anything", 12 | entry_points={"console_scripts": ["migrate-anything = migrate_anything.main:main"]}, 13 | version="0.2.0", 14 | description="Helps manage migrations for databases and anything else", 15 | long_description=long_description, 16 | long_description_content_type="text/x-rst", 17 | url="https://github.com/cocreators-ee/migrate-anything", 18 | author="Cocreators OÜ", 19 | author_email="janne@cocreators.ee", 20 | packages=["migrate_anything", "migrate_anything.storage"], 21 | keywords="migrate database db release", 22 | python_requires=">=3.6, <4", 23 | classifiers=[ 24 | "License :: OSI Approved :: BSD License", 25 | "Programming Language :: Python :: 3", 26 | "Programming Language :: Python :: 3.8", 27 | "Programming Language :: Python :: 3.9", 28 | "Programming Language :: Python :: 3.10", 29 | "Programming Language :: Python :: 3.11", 30 | "Programming Language :: Python :: 3.12", 31 | "Programming Language :: Python :: 3.13", 32 | ], 33 | project_urls={ 34 | "Bug Reports": "https://github.com/cocreators-ee/migrate-anything/issues", 35 | "Source": "https://github.com/cocreators-ee/migrate-anything/", 36 | }, 37 | ) 38 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | "New BSD" / "BSD 3-clause" -license 2 | ----------------------------------- 3 | 4 | 5 | Copyright 2019 Cocreators OÜ 6 | 7 | Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 8 | 9 | 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 10 | 11 | 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 12 | 13 | 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. 14 | 15 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 16 | -------------------------------------------------------------------------------- /.github/CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to this project 2 | 3 | Thank you for your interest in contributing! We welcome contributions of all kinds, from bug reports to new features. This document outlines our guidelines to make the process smooth and effective. 4 | 5 | ## Code of Conduct 6 | 7 | We are committed to providing a welcoming and respectful community for everyone. We expect all contributors to adhere to a high standard of conduct. **Harassment, abusive language, discrimination (including but not limited to racism, sexism, homophobia, transphobia, ableism, or any other form of prejudice) will not be tolerated.** Violations may result in removal from the project and/or reporting. 8 | 9 | ## How to contribute 10 | 11 | 1. **Discuss first:** Before tackling a significant change or feature, please open an issue to discuss your idea. This helps ensure it aligns with the project's goals and avoids wasted effort. 12 | 13 | 2. **Code style & formatting:** We aim for consistent code style throughout the project. 14 | * Please run any automated formatting tools included in this repository (e.g., `pre-commit`, `prettier`, `ruff`, `go fmt`). Check the `.editorconfig` or build scripts for details. 15 | * Try to follow existing coding conventions within the codebase. 16 | 17 | 3. **Submit a pull request:** When you're ready to contribute: 18 | * Create a pull request against the `main` branch. 19 | * In your PR description, *clearly explain*: 20 | * **What changed?** A concise summary of your changes. 21 | * **Why did you make this change?** What problem does it solve or what improvement does it offer? 22 | * **Output Changes (if applicable):** If your changes affect the output of the project (e.g., UI, logs), please include screenshots *before and after* to illustrate the impact. 23 | 24 | 4. **Be patient:** We are a small team with limited time dedicated to managing this repository. We will review pull requests as quickly as possible, but it may take some time. Your patience is greatly appreciated! 25 | 26 | 27 | 28 | Thanks again for your contribution! 29 | -------------------------------------------------------------------------------- /migrate_anything/tests/common.py: -------------------------------------------------------------------------------- 1 | import fnmatch 2 | import os 3 | from functools import wraps 4 | from shutil import rmtree 5 | 6 | GOOD_CODE = """ 7 | from time import time 8 | 9 | def up(): 10 | print(time()) 11 | 12 | def down(): 13 | print(time()) 14 | """ 15 | 16 | WITHOUT_DOWN = """ 17 | from time import time 18 | 19 | def up(): 20 | print(time()) 21 | """ 22 | 23 | WITHOUT_UP = """ 24 | from time import time 25 | 26 | def down(): 27 | print(time()) 28 | """ 29 | 30 | 31 | def find_cache_files(): 32 | """ 33 | Finds cache files, __pycache__ and *.pyc 34 | :return List[str]: 35 | """ 36 | files = [] 37 | 38 | for root, dirnames, filenames in os.walk("."): 39 | for filename in fnmatch.filter(filenames, "*.pyc"): 40 | files.append(os.path.join(root, filename)) 41 | 42 | for root, dirnames, filenames in os.walk("."): 43 | for filename in fnmatch.filter(filenames, "__pycache__"): 44 | files.append(os.path.join(root, filename)) 45 | 46 | return files 47 | 48 | 49 | def remove_files(files): 50 | """ 51 | Delete a bunch of files if they exist 52 | :param List[str] files: 53 | """ 54 | for file in files: 55 | if os.path.exists(file): 56 | if file.startswith("./") or file.startswith(".\\"): 57 | file = file[2:] 58 | if os.path.isdir(file): 59 | rmtree(file) 60 | else: 61 | os.unlink(file) 62 | 63 | 64 | def clean_filesystem(files=[]): 65 | """ 66 | Remove given files + python cache files 67 | :param List[str] files: 68 | """ 69 | remove_files(files + find_cache_files()) 70 | 71 | 72 | def clean_files(files): 73 | """ 74 | Remove test artifacts before and after running the test 75 | :param List[str] files: 76 | """ 77 | 78 | def _decorator(f): 79 | @wraps(f) 80 | def _wraps(*args, **kwargs): 81 | clean_filesystem(files) 82 | try: 83 | f(*args, **kwargs) 84 | finally: 85 | clean_filesystem(files) 86 | 87 | return _wraps 88 | 89 | return _decorator 90 | -------------------------------------------------------------------------------- /migrate_anything/tests/test_storage.py: -------------------------------------------------------------------------------- 1 | from collections import OrderedDict 2 | from os.path import dirname, join 3 | 4 | import mongomock 5 | import pytest 6 | 7 | from migrate_anything.migrator import _encode_code 8 | from migrate_anything.storage import CSVStorage, Storage, MongoDBStorage 9 | from migrate_anything.tests.common import GOOD_CODE, clean_files 10 | 11 | MIGRATIONS = OrderedDict([("01-test", _encode_code(GOOD_CODE))]) 12 | 13 | HERE = dirname(__file__) 14 | TEST_CSV = join(HERE, "storage_test.csv") 15 | 16 | 17 | def test_base_class(): 18 | s = Storage() 19 | with pytest.raises(NotImplementedError): 20 | for name in MIGRATIONS: 21 | s.save_migration(name, MIGRATIONS[name]) 22 | break 23 | 24 | with pytest.raises(NotImplementedError): 25 | s.list_migrations() 26 | 27 | with pytest.raises(NotImplementedError): 28 | for name in MIGRATIONS: 29 | s.remove_migration(name) 30 | break 31 | 32 | 33 | @clean_files([TEST_CSV]) 34 | def test_csv_storage(): 35 | s = CSVStorage(TEST_CSV) 36 | for name in MIGRATIONS: 37 | s.save_migration(name, MIGRATIONS[name]) 38 | 39 | received = s.list_migrations() 40 | assert len(s.list_migrations()) == len(MIGRATIONS) 41 | for name, code in received: 42 | assert name in MIGRATIONS 43 | assert code == MIGRATIONS[name] 44 | 45 | for name in MIGRATIONS: 46 | s.remove_migration(name) 47 | 48 | assert len(s.list_migrations()) == 0 49 | 50 | 51 | def test_mongodb_storage(): 52 | db = mongomock.MongoClient().test_db 53 | s = MongoDBStorage(db.migrations) 54 | 55 | for name in MIGRATIONS: 56 | s.save_migration(name, MIGRATIONS[name]) 57 | 58 | received = s.list_migrations() 59 | assert len(s.list_migrations()) == len(MIGRATIONS) 60 | for name, code in received: 61 | assert name in MIGRATIONS 62 | assert code == MIGRATIONS[name] 63 | assert db.migrations.count_documents({}) == len(MIGRATIONS) 64 | 65 | for name in MIGRATIONS: 66 | s.remove_migration(name) 67 | 68 | assert len(s.list_migrations()) == 0 69 | assert db.migrations.count_documents({}) == 0 70 | -------------------------------------------------------------------------------- /.github/workflows/test-and-release.yaml: -------------------------------------------------------------------------------- 1 | name: Test and release latest version 2 | 3 | on: 4 | push: 5 | branches: [ main, master ] 6 | pull_request: 7 | 8 | jobs: 9 | tests: 10 | runs-on: ubuntu-22.04 11 | environment: Build 12 | strategy: 13 | fail-fast: false 14 | matrix: 15 | python-version: [ 16 | "3.8", 17 | "3.9", 18 | "3.10", 19 | "3.11", 20 | "3.12", 21 | "3.13", 22 | "pypy3.9", 23 | "pypy3.10" 24 | ] 25 | 26 | steps: 27 | - uses: actions/checkout@v5 28 | 29 | - name: Set up Python ${{ matrix.python-version }} 30 | uses: actions/setup-python@v6 31 | with: 32 | python-version: ${{ matrix.python-version }} 33 | 34 | - name: Set up dev dependencies 35 | run: | 36 | python -m pip install -U pip 37 | pip install -r dev-requirements.txt 38 | 39 | - name: Run all the tests 40 | run: | 41 | export PYTHONDONTWRITEBYTECODE=1 # Hopefully prevents flaky tests 42 | coverage run --include "migrate_anything/*" -m pytest 43 | 44 | - name: Run codecov 45 | if: matrix.python-version == '3.13' 46 | run: | 47 | coverage xml -i 48 | mkdir coverage-reports 49 | mv coverage.xml coverage-reports/coverage-python.xml 50 | 51 | - name: Codecov report 52 | if: matrix.python-version == '3.13' 53 | uses: codecov/codecov-action@v5 54 | with: 55 | files: coverage-reports/coverage-python.xml 56 | flags: unittests 57 | 58 | # Sonar keeps insisting it's not getting a token, so bye Sonar 59 | # sonar: 60 | # runs-on: ubuntu-22.04 61 | # environment: Build 62 | # steps: 63 | # - uses: actions/checkout@v5 64 | # 65 | # - name: SonarCloud Scan 66 | # uses: SonarSource/sonarqube-scan-action@master 67 | # env: 68 | # GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} 69 | # SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} 70 | 71 | release: 72 | runs-on: ubuntu-22.04 73 | environment: Build 74 | needs: tests 75 | steps: 76 | - uses: actions/checkout@v5 77 | 78 | - name: Set up Python 79 | uses: actions/setup-python@v6 80 | with: 81 | python-version: "3.13" 82 | 83 | - name: Set up dev dependencies 84 | run: | 85 | python -m pip install -U pip setuptools 86 | pip install -r dev-requirements.txt 87 | 88 | - name: Build package 89 | run: | 90 | python setup.py sdist bdist_wheel 91 | twine check dist/* 92 | 93 | - name: Publish to PyPI 94 | if: github.ref == 'refs/heads/master' 95 | #if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags') 96 | uses: pypa/gh-action-pypi-publish@release/v1 97 | with: 98 | user: __token__ 99 | password: ${{ secrets.PYPI_API_TOKEN }} 100 | skip_existing: true 101 | -------------------------------------------------------------------------------- /migrate_anything/tests/test_migrator.py: -------------------------------------------------------------------------------- 1 | from os.path import join, dirname, sep, exists 2 | 3 | import pytest 4 | 5 | from migrate_anything import CSVStorage 6 | from migrate_anything.migrator import _check_module, run 7 | from migrate_anything.tests.common import ( 8 | GOOD_CODE, 9 | WITHOUT_DOWN, 10 | WITHOUT_UP, 11 | clean_files, 12 | clean_filesystem, 13 | ) 14 | 15 | try: 16 | from importlib import invalidate_caches, machinery, util 17 | except ImportError: 18 | 19 | def invalidate_caches(): 20 | pass 21 | 22 | 23 | MIGRATION_CODE = """ 24 | from os import remove 25 | from time import time 26 | 27 | file = "test-file2.txt" 28 | 29 | 30 | def up(): 31 | with open(file, "w") as f: 32 | f.write(str(time())) 33 | 34 | 35 | def down(): 36 | with open(file) as f: 37 | old = float(f.read()) 38 | diff = abs(time() - old) 39 | if diff > 0.5: 40 | raise Exception("Something is wrong") 41 | remove(file) 42 | """ 43 | 44 | HERE = dirname(__file__) 45 | TEST_CSV = join(HERE, "migrator_test.csv") 46 | MIGRATIONS_PKG = "migrate_anything.tests.migrations" 47 | MIGRATIONS_PATH = MIGRATIONS_PKG.replace(".", sep) 48 | NEW_MIGRATION = join(MIGRATIONS_PATH, "02-good-code.py") 49 | 50 | 51 | def test_check_module(): 52 | module_spec = machinery.ModuleSpec("test", None) 53 | module = util.module_from_spec(module_spec) 54 | exec(GOOD_CODE, module.__dict__) 55 | 56 | _check_module(module) 57 | 58 | module_spec = machinery.ModuleSpec("test2", None) 59 | module = util.module_from_spec(module_spec) 60 | exec(WITHOUT_DOWN, module.__dict__) 61 | with pytest.raises(Exception): 62 | _check_module(module) 63 | 64 | module_spec = machinery.ModuleSpec("test3", None) 65 | module = util.module_from_spec(module_spec) 66 | exec(WITHOUT_UP, module.__dict__) 67 | with pytest.raises(Exception): 68 | _check_module(module) 69 | 70 | 71 | @clean_files([TEST_CSV, "test-file.txt", "test-file2.txt", NEW_MIGRATION]) 72 | def test_run(): 73 | storage = CSVStorage(TEST_CSV) 74 | 75 | assert len(storage.list_migrations()) == 0 76 | 77 | run(MIGRATIONS_PKG) 78 | first = storage.list_migrations() 79 | 80 | assert len(first) > 0 81 | assert exists("test-file.txt") 82 | 83 | with open(NEW_MIGRATION, "w") as f: 84 | f.write(MIGRATION_CODE) 85 | 86 | clean_filesystem() 87 | invalidate_caches() # Reset import caches 88 | 89 | run(MIGRATIONS_PKG) 90 | second = storage.list_migrations() 91 | 92 | assert len(second) > len(first) 93 | assert exists("test-file2.txt") 94 | 95 | clean_filesystem([NEW_MIGRATION]) 96 | invalidate_caches() # Reset import caches 97 | 98 | run(MIGRATIONS_PKG) 99 | third = storage.list_migrations() 100 | 101 | assert third == first 102 | assert not exists("test-file2.txt") 103 | 104 | 105 | @clean_files([TEST_CSV, "test-file.txt", "test-file2.txt", NEW_MIGRATION]) 106 | def test_run_with_revert_mode(): 107 | storage = CSVStorage(TEST_CSV) 108 | 109 | assert len(storage.list_migrations()) == 0 110 | 111 | run(MIGRATIONS_PKG) 112 | first = storage.list_migrations() 113 | 114 | assert len(first) > 0 115 | assert exists("test-file.txt") 116 | 117 | with open(NEW_MIGRATION, "w") as f: 118 | f.write(MIGRATION_CODE) 119 | 120 | clean_filesystem() 121 | invalidate_caches() # Reset import caches 122 | 123 | run(MIGRATIONS_PKG) 124 | second = storage.list_migrations() 125 | 126 | assert len(second) > len(first) 127 | assert exists("test-file2.txt") 128 | 129 | invalidate_caches() # Reset import caches 130 | 131 | run(MIGRATIONS_PKG, revert=True) 132 | third = storage.list_migrations() 133 | 134 | assert third == first 135 | assert not exists("test-file2.txt") 136 | -------------------------------------------------------------------------------- /migrate_anything/migrator.py: -------------------------------------------------------------------------------- 1 | import base64 2 | import importlib 3 | import inspect 4 | import os 5 | import pkgutil 6 | import sys 7 | from collections import OrderedDict 8 | from io import open 9 | 10 | from migrate_anything.log import logger 11 | from migrate_anything.storage import Storage 12 | 13 | PY3 = sys.version_info[0] >= 3 14 | 15 | 16 | class _CONFIG: 17 | storage = None # type: Storage 18 | 19 | 20 | def configure(storage): 21 | """ 22 | Configure migrate-anything 23 | :param Storage storage: 24 | """ 25 | _CONFIG.storage = storage 26 | 27 | 28 | def _encode_code(code): 29 | """ 30 | Convert source code to encoded format 31 | :param str code: 32 | :return str: 33 | """ 34 | if PY3: 35 | code = code.encode("utf-8") 36 | 37 | code = base64.b64encode(code) 38 | 39 | if PY3: 40 | code = code.decode("utf-8") 41 | 42 | return code 43 | 44 | 45 | def _decode_code(encoded): 46 | """ 47 | Convert encoded code to readable format 48 | :param str encoded: 49 | :return str: 50 | """ 51 | return base64.b64decode(encoded) 52 | 53 | 54 | def _encode_module(module): 55 | """ 56 | Convert a Python module to encoded code 57 | :param types.Module module: 58 | :return str: 59 | """ 60 | src = inspect.getsourcefile(module) 61 | with open(src) as file: 62 | return _encode_code(file.read()) 63 | 64 | 65 | def _decode_module(name, b64): 66 | """ 67 | Convert encoded code to plain Python code 68 | :param str code: 69 | :return types.Module: 70 | """ 71 | module_spec = importlib.machinery.ModuleSpec(name, None) 72 | module = importlib.util.module_from_spec(module_spec) 73 | exec(_decode_code(b64), module.__dict__) 74 | return module 75 | 76 | 77 | def _load_package(package): 78 | """ 79 | Load the migrations in the package 80 | :param str package: Package name 81 | :return dict: 82 | """ 83 | logger.info("Loading migrations from {}".format(package)) 84 | sys.path.append(os.getcwd()) 85 | importlib.import_module(package) 86 | migrations = {} 87 | 88 | for _, name, _ in pkgutil.walk_packages(sys.modules[package].__path__): 89 | full_name = package + "." + name 90 | logger.info(" - {}".format(full_name)) 91 | 92 | module = importlib.import_module(full_name) 93 | _check_module(module) 94 | migrations[name] = module 95 | 96 | return migrations 97 | 98 | 99 | def _check_module(module): 100 | """ 101 | Quickly validate that module seems like a valid migration 102 | :param types.Module module: 103 | """ 104 | if not getattr(module, "up", None): 105 | raise Exception("Module {} does not define up()".format(module)) 106 | if not getattr(module, "down", None): 107 | raise Exception("Module {} does not define down()".format(module)) 108 | 109 | 110 | def _check_config(): 111 | """ 112 | Check that the configuration is sufficient 113 | """ 114 | errors = False 115 | if not _CONFIG.storage: 116 | logger.error( 117 | "No storage configured for migrate-anything. Did you run configure()?" 118 | ) 119 | errors = True 120 | 121 | if errors: 122 | sys.exit(1) 123 | 124 | 125 | def _undo_migrations(migrations): 126 | """ 127 | Undo any migrations that are no longer active 128 | :param dict[str, str] migrations: Old migrations to undo 129 | """ 130 | for name in migrations: 131 | logger.info("Undoing migration {}".format(name)) 132 | code = migrations[name] 133 | module = _decode_module(name, code) 134 | module.down() 135 | _CONFIG.storage.remove_migration(name) 136 | 137 | 138 | def _apply_migrations(migrations): 139 | """ 140 | Apply new migrations 141 | :param dict[str,types.Module] migrations: 142 | """ 143 | for name in migrations: 144 | logger.info("Applying migration {}".format(name)) 145 | module = migrations[name] 146 | module.up() 147 | _CONFIG.storage.save_migration(name, _encode_module(module)) 148 | 149 | 150 | def run(package, revert=False): 151 | if revert: 152 | run_one_down(package) 153 | else: 154 | run_auto_mode(package) 155 | 156 | 157 | def run_one_down(package): 158 | migrations = _load_package(package) 159 | _check_config() 160 | 161 | # Calculate diffs 162 | applied = OrderedDict(_CONFIG.storage.list_migrations()) 163 | applied_keys = sorted(set(applied.keys())) 164 | current_keys = sorted(set(migrations.keys())) 165 | 166 | # Select the latest applied key 167 | try: 168 | last_run_migration = applied_keys[-1] 169 | except IndexError: 170 | logger.info("No migrations to rollback found.") 171 | return 172 | 173 | # Check to make sure the migration exists 174 | migration_exists = last_run_migration in current_keys 175 | 176 | if migration_exists: 177 | module = migrations[last_run_migration] 178 | module.down() 179 | _CONFIG.storage.remove_migration(last_run_migration) 180 | logger.info("Migration reverted: {}".format(last_run_migration)) 181 | else: 182 | logger.info("Unable to find module for {}".format(last_run_migration)) 183 | 184 | 185 | def run_auto_mode(package): 186 | """ 187 | Run the complete process 188 | :param str package: Name of the package that defines the migrations 189 | """ 190 | 191 | # Package should define config, load it first, then check 192 | migrations = _load_package(package) 193 | _check_config() 194 | 195 | # Calculate diffs 196 | applied = OrderedDict(_CONFIG.storage.list_migrations()) 197 | applied_keys = set(applied.keys()) 198 | current = set(migrations.keys()) 199 | 200 | if current: 201 | logger.info("Found previously applied migrations:") 202 | for name in sorted(applied_keys): 203 | logger.info(" - {}".format(name)) 204 | 205 | undo_migrations = applied_keys - current 206 | new_migrations = current - applied_keys 207 | 208 | _undo_migrations( 209 | OrderedDict( 210 | [ 211 | (name, applied[name]) 212 | for name in reversed(applied) 213 | if name in undo_migrations 214 | ] 215 | ) 216 | ) 217 | 218 | _apply_migrations( 219 | OrderedDict( 220 | [(name, migrations[name]) for name in migrations if name in new_migrations] 221 | ) 222 | ) 223 | -------------------------------------------------------------------------------- /migrate_anything/storage/__init__.py: -------------------------------------------------------------------------------- 1 | import csv 2 | import sys 3 | import types 4 | from collections import namedtuple 5 | from io import open 6 | 7 | from migrate_anything.log import logger 8 | 9 | try: 10 | from itertools import imap 11 | except ImportError: 12 | imap = map 13 | 14 | try: 15 | import pymongo 16 | except ImportError: 17 | pymongo = None 18 | 19 | try: 20 | import arango 21 | import arango.collection 22 | except ImportError: 23 | arango = None 24 | 25 | PY3 = sys.version_info.major >= 3 26 | 27 | _CSVRow = namedtuple("Row", "name,code") 28 | 29 | 30 | def _fix_docs(cls): 31 | """ 32 | Used to copy function docstring from Storage baseclass to subclasses 33 | """ 34 | for name, func in vars(cls).items(): 35 | if isinstance(func, types.FunctionType) and not func.__doc__: 36 | for parent in cls.__bases__: 37 | parfunc = getattr(parent, name, None) 38 | if parfunc and getattr(parfunc, "__doc__", None): 39 | func.__doc__ = parfunc.__doc__ 40 | break 41 | return cls 42 | 43 | 44 | class Storage(object): 45 | def save_migration(self, name, code): 46 | """ 47 | Save a migration 48 | :param str name: The name of the migration 49 | :param str code: The source code (encoded) 50 | """ 51 | raise NotImplementedError("Storage class does not implement save_migration") 52 | 53 | def list_migrations(self): 54 | """ 55 | List applied migrations 56 | :return List[Tuple[str, str]]: 57 | """ 58 | raise NotImplementedError("Storage class does not implement list_migrations") 59 | 60 | def remove_migration(self, name): 61 | """ 62 | Remove migration after it's been undone 63 | :param str name: 64 | """ 65 | raise NotImplementedError("Storage class does not implement remove_migration") 66 | 67 | 68 | @_fix_docs 69 | class ArangoDBStorage(Storage): 70 | INDEX = ["name"] 71 | 72 | def __init__(self, collection, db=None, *args, **kwargs): 73 | """ 74 | :param Union[arango.collection.Collection, str] collection: Either the 75 | Collection to store the migrations in, or the name of the collection. 76 | If the name is given, then the database must also be given. 77 | :param Optional[arango.database.Database] db: The arango database, only 78 | needed if the collection name is given rather than the collection. 79 | :param args: Positional arguments used if creating the collection 80 | :param kwargs: Keyword arguments used if creating the collection 81 | """ 82 | if not arango: 83 | raise Exception("Cannot load arango, is it installed?") 84 | 85 | if not isinstance(collection, arango.collection.Collection): 86 | collection = self._get_collection(collection, db, *args, **kwargs) 87 | 88 | self.collection = collection 89 | 90 | @classmethod 91 | def _get_collection(cls, name, db, *args, **kwargs): 92 | """ 93 | Get the collection for storing migrations. Creates it if needed. 94 | 95 | :param arango.database.Database db: The database 96 | :param str name: The name of the collection 97 | :param args: Positional arguments used if creating the collection 98 | :param kwargs: Keyword arguments used if creating the collection 99 | :return arango.collection.Collection: The collection 100 | """ 101 | if not db: 102 | raise RuntimeError("Can not create collection without db.") 103 | 104 | if not db.has_collection(name): 105 | collection = db.create_collection(name, *args, **kwargs) 106 | collection.add_hash_index(cls.INDEX, unique=True) 107 | else: 108 | collection = db.collection(name) 109 | 110 | return collection 111 | 112 | def save_migration(self, name, code): 113 | self.collection.insert({"name": name, "code": code}) 114 | 115 | def list_migrations(self): 116 | return [(e["name"], e["code"]) for e in self.collection.all()] 117 | 118 | def remove_migration(self, name): 119 | self.collection.delete_match({"name": name}, limit=1) 120 | 121 | 122 | @_fix_docs 123 | class CSVStorage(Storage): 124 | def __init__(self, file): 125 | self.file = file 126 | logger.warning( 127 | "Using CSV storage - hopefully you're just testing " 128 | "or know what you're doing as this data can be easily lost." 129 | ) 130 | 131 | def save_migration(self, name, code): 132 | def _to_writable(value): 133 | return value if PY3 else value.encode("utf-8") 134 | 135 | mode = "a" if PY3 else "ab" 136 | with open(self.file, mode) as csvfile: 137 | writer = csv.writer(csvfile) 138 | writer.writerow([_to_writable(name), _to_writable(code)]) 139 | 140 | def list_migrations(self): 141 | migrations = [] 142 | 143 | try: 144 | with open(self.file) as csvfile: 145 | reader = csv.reader(csvfile) 146 | for row in reader: 147 | if not row: 148 | continue 149 | migrations.append(_CSVRow(*row)) 150 | except IOError: 151 | pass 152 | 153 | return migrations 154 | 155 | def remove_migration(self, name): 156 | migrations = [ 157 | migration for migration in self.list_migrations() if migration.name != name 158 | ] 159 | 160 | mode = "w" if PY3 else "wb" 161 | with open(self.file, mode) as csvfile: 162 | writer = csv.writer(csvfile) 163 | for row in migrations: 164 | writer.writerow(row) 165 | 166 | 167 | @_fix_docs 168 | class MongoDBStorage(Storage): 169 | INDEX = "name" 170 | 171 | def __init__(self, collection): 172 | """ 173 | :param pymongo.collection.Collection collection: 174 | """ 175 | if not pymongo: 176 | raise Exception("Cannot load pymongo, is it installed?") 177 | 178 | self.collection = collection 179 | 180 | if self.INDEX not in collection.index_information(): 181 | collection.create_index(self.INDEX, unique=True) 182 | 183 | def save_migration(self, name, code): 184 | self.collection.insert_one({"name": name, "code": code}) 185 | 186 | def list_migrations(self): 187 | return [(e["name"], e["code"]) for e in self.collection.find()] 188 | 189 | def remove_migration(self, name): 190 | self.collection.delete_one({"name": name}) 191 | 192 | 193 | __all__ = ["ArangoDBStorage", "CSVStorage", "MongoDBStorage"] 194 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | .. image:: https://travis-ci.org/cocreators-ee/migrate-anything.svg?branch=master 2 | :target: https://travis-ci.org/cocreators-ee/migrate-anything 3 | 4 | .. image:: https://img.shields.io/badge/code%20style-black-000000.svg 5 | :target: https://github.com/psf/black 6 | 7 | .. image:: https://codecov.io/gh/cocreators-ee/migrate-anything/branch/master/graph/badge.svg 8 | :target: https://codecov.io/gh/cocreators-ee/migrate-anything 9 | 10 | .. image:: https://sonarcloud.io/api/project_badges/measure?project=cocreators_migrate-anything&metric=alert_status 11 | :target: https://sonarcloud.io/dashboard?id=cocreators_migrate-anything 12 | 13 | .. image:: https://img.shields.io/github/issues/cocreators-ee/migrate-anything 14 | :target: https://github.com/cocreators-ee/migrate-anything/issues 15 | :alt: GitHub issues 16 | 17 | .. image:: https://img.shields.io/pypi/dm/migrate-anything 18 | :target: https://pypi.org/project/migrate-anything/ 19 | :alt: PyPI - Downloads 20 | 21 | .. image:: https://img.shields.io/pypi/v/migrate-anything 22 | :target: https://pypi.org/project/migrate-anything/ 23 | :alt: PyPI 24 | 25 | .. image:: https://img.shields.io/pypi/pyversions/migrate-anything 26 | :target: https://pypi.org/project/migrate-anything/ 27 | :alt: PyPI - Python Version 28 | 29 | .. image:: https://img.shields.io/badge/License-BSD%203--Clause-blue.svg 30 | :target: https://opensource.org/licenses/BSD-3-Clause 31 | 32 | Migrate anything - database (etc.) migration utility, especially for Python projects. 33 | 34 | 35 | What is this? 36 | ============= 37 | 38 | It's kinda annoying how often you run into the question of how to handle migrations in your project, and there hasn't seem to emerged any good, DB -agnostic, framework-agnostic, and storage-agnostic tool to manage them. 39 | 40 | This project is an attempt to change that. 41 | 42 | Basically what it does when you run :code:`migrate-anything migrations` is: 43 | 44 | 1. Find all the files :code:`migrations/*.py` and sort them 45 | 2. Any that are not yet registered in the DB will be loaded, their :code:`up()` is executed, and the file's contents stored in the DB 46 | 3. Any files that are missing from the fs but are in the DB will have their code loaded from the DB and their :code:`down()` is executed - in reverse order 47 | 48 | 49 | License 50 | ------- 51 | 52 | Licensing is important. This project uses BSD 3-clause license, and adds no other dependencies to your project (it does use a few things during build & testing) - that's about as simple, safe, and free to use as it gets. 53 | 54 | For more information check the `LICENSE `_ -file. 55 | 56 | 57 | Usage examples 58 | ============== 59 | 60 | Basic usage 61 | ----------- 62 | 63 | Firstly you'll need this package in your project. Pick one of these: 64 | 65 | .. code-block:: python 66 | 67 | pip install -U migrate-anything 68 | poetry add migrate-anything 69 | pipenv install migrate-anything 70 | 71 | Simply put, create a Python package, don't be too clever and call it e.g. ``migrations``. Then put files in that package: 72 | 73 | .. code-block:: python 74 | 75 | # migrations/__init__.py 76 | from migrate_anything import configure, CSVStorage 77 | 78 | configure(storage=CSVStorage("migration_status.csv")) 79 | 80 | .. code-block:: python 81 | 82 | # migrations/01-initialize-db.py 83 | # Please note that this is built for a completely hypothetical DB layer 84 | from my_db import get_db 85 | 86 | DB = get_db() 87 | 88 | def up(): 89 | DB.create_table("example") 90 | 91 | def down(): 92 | DB.delete_table("example") 93 | 94 | This would configure your migrations' status to be stored in a local file called ``migration_status.csv`` and set up your first migration script. If you have a ``my_db`` module that works like this you could just run this with a single command: 95 | 96 | .. code-block:: shell 97 | 98 | migrate-anything migrations 99 | poetry run migrate-anything migrations 100 | pipenv run migrate-anything migrations 101 | 102 | Now in the real world you might want something more durable and a realistic example, so here's e.g. what you'd do when using MongoDB: 103 | 104 | .. code-block:: python 105 | 106 | # __init__.py 107 | from migrate_anything import configure, MongoDBStorage 108 | import pymongo 109 | 110 | db = pymongo.MongoClient().my_db 111 | 112 | configure(storage=MongoDBStorage(db.migrations)) 113 | 114 | .. code-block:: python 115 | 116 | # 01-initialize-db.py 117 | from pymongo import MongoClient 118 | 119 | client = MongoClient() 120 | db = client.my_db 121 | 122 | def up(): 123 | db.posts.insert_one({ 124 | "id": "post-1", 125 | "title": "We're live!", 126 | "content": "This is our first post, yay." 127 | }) 128 | db.posts.create_index("id") 129 | 130 | def down(): 131 | db.posts.drop() 132 | 133 | This would configure storage to a ``my_db.migrations`` MongoDB collection. 134 | 135 | 136 | Command line flags 137 | ----------------------- 138 | 139 | .. code-block:: shell 140 | 141 | # Revert the last migration using migration code file. 142 | migrate-anything migrations --revert-latest 143 | 144 | 145 | Custom storage engines 146 | ----------------------- 147 | 148 | Writing your own custom storage engine is easy. 149 | 150 | .. code-block:: python 151 | 152 | # __init__.py 153 | from migrate_anything import configure 154 | 155 | 156 | class CustomStorage(object): 157 | def __init__(self, file): 158 | self.file = file 159 | 160 | def save_migration(self, name, code): 161 | with open(self.file, "a", encoding="utf-8") as file: 162 | file.write("{},{}\n".format(name, code)) 163 | 164 | def list_migrations(self): 165 | try: 166 | with open(self.file, encoding="utf-8") as file: 167 | return [ 168 | line.split(",") 169 | for line in file.readlines() 170 | if line.strip() # Skip empty lines 171 | ] 172 | except FileNotFoundError: 173 | return [] 174 | 175 | def remove_migration(self, name): 176 | migrations = [ 177 | migration for migration in self.list_migrations() if migration[0] != name 178 | ] 179 | 180 | with open(self.file, "w", encoding="utf-8") as file: 181 | for row in migrations: 182 | file.write("{},{}\n".format(*row)) 183 | 184 | 185 | configure(storage=CustomStorage("test.txt")) 186 | 187 | You can also check out the `examples `_. 188 | 189 | 190 | Contributing 191 | ============ 192 | 193 | This project is run on GitHub using the issue tracking and pull requests here. If you want to contribute, feel free to `submit issues `_ (incl. feature requests) or PRs here. 194 | 195 | You will need `pre-commit `_ set up to make contributions. 196 | 197 | To set up development tools for this, run: 198 | 199 | .. code-block:: shell 200 | 201 | pre-commit install 202 | virtualenv .venv 203 | 204 | .venv/bin/activate 205 | # OR 206 | .venv\Scripts\activate.bat 207 | 208 | pip install -r dev-requirements.txt 209 | pip install -e . 210 | 211 | And then to run the tests 212 | 213 | .. code-block:: shell 214 | 215 | pytest 216 | 217 | When you have improvements to make, commit (and include any cleanup pre-commit might do), push your changes to your own fork, make a PR. 218 | 219 | 220 | Future ideas 221 | ================= 222 | 223 | Future ideas include support for other DB engines (feel free to contribute), 224 | and Kubernetes ConfigMap. Annoyingly storage to Kubernetes from inside a pod 225 | and in code is not quite as simple as just running ``kubectl``. 226 | 227 | Oh and your Kubernetes pods will likely require the necessary RBAC rules to manage their ConfigMap. It's unfortunately kinda complex, but I'm sure you can figure it out e.g. with this `guide `_. 228 | 229 | 230 | Financial support 231 | ================= 232 | 233 | This project has been made possible thanks to `Cocreators `_ and `Lietu `_. You can help us continue our open source work by supporting us on `Buy me a coffee `_. 234 | 235 | .. image:: https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png 236 | :target: https://www.buymeacoffee.com/cocreators 237 | --------------------------------------------------------------------------------