├── tests ├── __init__.py ├── unittests │ ├── __init__.py │ ├── commands │ │ ├── __init__.py │ │ ├── in │ │ │ ├── .gitkeep │ │ │ ├── __init__.py │ │ │ └── data_migrations │ │ │ │ ├── __init__.py │ │ │ │ ├── trash_file.py │ │ │ │ ├── 0002_some_super_delicate_migration.py │ │ │ │ └── 0001_first.py │ │ ├── out │ │ │ └── .gitkeep │ │ ├── test_aliases.py │ │ ├── test_data_migrate.py │ │ ├── test_makemigrations.py │ │ ├── test_squashmigrations.py │ │ └── test_migrate.py │ ├── services │ │ ├── __init__.py │ │ ├── out │ │ │ └── .gitkeep │ │ ├── data_migrations │ │ │ ├── __init__.py │ │ │ ├── 0001_first.py │ │ │ └── 0002_auto.py │ │ ├── test_node.py │ │ ├── test_file_generator.py │ │ └── test_graph.py │ ├── test_app │ │ ├── migrations │ │ │ ├── __init__.py │ │ │ └── 0001_first.py │ │ ├── data_migrations │ │ │ ├── __init__.py │ │ │ └── 0001_first.py │ │ ├── models.py │ │ ├── __init__.py │ │ ├── apps.py │ │ └── helper.py │ ├── test_app_2 │ │ ├── migrations │ │ │ ├── __init__.py │ │ │ ├── 0007_remove_customer_address.py │ │ │ ├── 0005_customer_address.py │ │ │ ├── 0008_customer_is_business.py │ │ │ ├── 0001_initial.py │ │ │ ├── 0004_customer_is_active.py │ │ │ ├── 0003_mmodel.py │ │ │ ├── 0006_address_line_split.py │ │ │ └── 0002_split_name.py │ │ ├── data_migrations │ │ │ └── __init__.py │ │ ├── __init__.py │ │ ├── apps.py │ │ ├── helper.py │ │ └── models.py │ ├── test_helper.py │ ├── conftest.py │ └── test_settings.py ├── integrationtests │ ├── __init__.py │ └── test_node.py └── utils.py ├── data_migration ├── management │ ├── __init__.py │ └── commands │ │ ├── __init__.py │ │ ├── django_migrate.py │ │ ├── django_squashmigrations.py │ │ ├── django_makemigrations.py │ │ ├── data_migrate.py │ │ ├── makemigrations.py │ │ ├── migrate.py │ │ └── squashmigrations.py ├── services │ ├── __init__.py │ ├── templates │ │ └── migration.py.txt │ ├── node.py │ ├── file_generator.py │ ├── squasher.py │ └── graph.py ├── __init__.py ├── helper.py ├── apps.py └── settings.py ├── Makefile ├── MANIFEST.in ├── .flake8 ├── renovate.json ├── .pre-commit-config.yaml ├── .gitignore ├── dev-requirements.txt ├── .coveragerc ├── LICENSE ├── .github └── workflows │ ├── test-package.yml │ ├── test-dev.yml │ └── publish-package.yml ├── setup.py └── README.rst /tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unittests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /data_migration/management/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /data_migration/services/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/integrationtests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unittests/commands/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unittests/commands/in/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unittests/commands/out/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unittests/services/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unittests/services/out/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unittests/commands/in/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | test: 2 | pytest -v tests --cov 3 | -------------------------------------------------------------------------------- /tests/unittests/test_app/migrations/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unittests/services/data_migrations/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unittests/test_app/data_migrations/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unittests/test_app_2/migrations/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unittests/commands/in/data_migrations/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unittests/commands/in/data_migrations/trash_file.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unittests/test_app_2/data_migrations/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /data_migration/management/commands/__init__.py: -------------------------------------------------------------------------------- 1 | """Extended management commands.""" 2 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include LICENSE 2 | include README.md 3 | recursive-include data_migration * 4 | recursive-include docs * 5 | -------------------------------------------------------------------------------- /.flake8: -------------------------------------------------------------------------------- 1 | [flake8] 2 | ignore = W503 3 | exclude = .git,__pycache__,migrations,data_migrations,setup.py,000*.py 4 | max-complexity = 15 -------------------------------------------------------------------------------- /renovate.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://docs.renovatebot.com/renovate-schema.json", 3 | "extends": [ 4 | "config:base" 5 | ] 6 | } 7 | -------------------------------------------------------------------------------- /tests/unittests/test_app/models.py: -------------------------------------------------------------------------------- 1 | from django.db import models 2 | 3 | 4 | class MModel(models.Model): 5 | bit = models.BooleanField() 6 | -------------------------------------------------------------------------------- /data_migration/__init__.py: -------------------------------------------------------------------------------- 1 | import django 2 | 3 | 4 | if django.VERSION < (3, 2): 5 | default_app_config = 'data_migration.apps.DataMigrationsConfig' 6 | -------------------------------------------------------------------------------- /tests/unittests/test_app/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | Test app to simulate data migration process. 3 | """ 4 | default_app_config = 'tests.unittests.test_app.apps.TestAppConfig' 5 | -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- 1 | repos: 2 | - repo: https://github.com/pycqa/flake8 3 | rev: '3.9.2' # pick a git hash / tag to point to 4 | hooks: 5 | - id: flake8 6 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .idea/ 2 | .pytest_cache/ 3 | .venv/ 4 | data_migration.egg-info/ 5 | coverage_html_report/ 6 | __pycache__/ 7 | 8 | 9 | .coverage 10 | .pypirc 11 | 12 | *.pyc 13 | -------------------------------------------------------------------------------- /tests/unittests/test_app_2/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | Test app to simulate migration squashing and extraction process. 3 | """ 4 | default_app_config = 'tests.unittests.test_app_2.apps.TestApp2Config' 5 | -------------------------------------------------------------------------------- /tests/unittests/test_app/apps.py: -------------------------------------------------------------------------------- 1 | from django.apps import AppConfig 2 | 3 | 4 | class TestAppConfig(AppConfig): 5 | default_auto_field = 'django.db.models.BigAutoField' 6 | name = 'tests.unittests.test_app' 7 | label = 'test_app' 8 | -------------------------------------------------------------------------------- /tests/unittests/test_app_2/apps.py: -------------------------------------------------------------------------------- 1 | from django.apps import AppConfig 2 | 3 | 4 | class TestApp2Config(AppConfig): 5 | default_auto_field = 'django.db.models.BigAutoField' 6 | name = 'tests.unittests.test_app_2' 7 | label = 'test_app_2' 8 | -------------------------------------------------------------------------------- /data_migration/helper.py: -------------------------------------------------------------------------------- 1 | """public helpers of package.""" 2 | import os 3 | 4 | 5 | def get_package_version_string(): 6 | """:return: version string of package.""" 7 | name = 'data_migration' 8 | version = os.getenv('VERSION', '0.0.1a') 9 | return f'{name} {version}' 10 | -------------------------------------------------------------------------------- /dev-requirements.txt: -------------------------------------------------------------------------------- 1 | asgiref==3.7.2 2 | attrs==23.1.0 3 | coverage==7.3.2 4 | Django>=3.2.,<4.0 5 | iniconfig==2.0.0 6 | packaging==23.2 7 | pluggy==1.3.0 8 | py==1.11.0 9 | pyparsing==3.1.1 10 | pytest==7.4.3 11 | pytest-cov==4.1.0 12 | pytz==2023.3.post1 13 | sqlparse==0.5.0 14 | tomli==2.0.1 15 | -------------------------------------------------------------------------------- /tests/unittests/test_helper.py: -------------------------------------------------------------------------------- 1 | from unittest import TestCase 2 | 3 | from data_migration.helper import get_package_version_string 4 | 5 | 6 | class HelperTestCase(TestCase): 7 | def test_get_package_version_string(self): 8 | self.assertEqual(get_package_version_string(), 'data_migration 0.0.1a') 9 | -------------------------------------------------------------------------------- /tests/unittests/conftest.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | from tests.utils import setup_django, teardown_django 3 | 4 | 5 | @pytest.hookimpl(tryfirst=True) 6 | def pytest_configure(config): 7 | setup_django() 8 | 9 | 10 | @pytest.hookimpl(tryfirst=True) 11 | def pytest_unconfigure(config): 12 | teardown_django() 13 | -------------------------------------------------------------------------------- /tests/unittests/services/data_migrations/0001_first.py: -------------------------------------------------------------------------------- 1 | """ 2 | Generated by data_migration 0.0.1a 3 | 4 | File: 0001_first.py 5 | Created: 2021-05-22 11:23 6 | """ 7 | 8 | 9 | class Node: 10 | name = '0001_first' 11 | dependencies = () 12 | migration_dependencies = () 13 | routines = [ 14 | ] 15 | -------------------------------------------------------------------------------- /data_migration/management/commands/django_migrate.py: -------------------------------------------------------------------------------- 1 | from django.core.management.commands.migrate import Command as Migrate 2 | 3 | 4 | class Command(Migrate): 5 | """Alias for default `migrate` command provided by Django.""" 6 | 7 | def handle(self, *args, **options): # noqa D102 8 | return super().handle(*args, **options) 9 | -------------------------------------------------------------------------------- /data_migration/management/commands/django_squashmigrations.py: -------------------------------------------------------------------------------- 1 | from django.core.management.commands.squashmigrations import Command as Migrate 2 | 3 | 4 | class Command(Migrate): 5 | """Alias for default `squashmigrations` command provided by Django.""" 6 | 7 | def handle(self, **options): # noqa D102 8 | return super().handle(**options) 9 | -------------------------------------------------------------------------------- /data_migration/management/commands/django_makemigrations.py: -------------------------------------------------------------------------------- 1 | from django.core.management.commands.makemigrations import Command as Migrate 2 | 3 | 4 | class Command(Migrate): 5 | """Alias for default `makemigrations` command provided by Django.""" 6 | 7 | def handle(self, *args, **options): # noqa D102 8 | return super().handle(*args, **options) 9 | -------------------------------------------------------------------------------- /tests/unittests/commands/in/data_migrations/0002_some_super_delicate_migration.py: -------------------------------------------------------------------------------- 1 | """ 2 | Generated by data_migration 0.0.1a 3 | 4 | File: 0002_auto.py 5 | Created: 2021-05-22 11:23 6 | """ 7 | 8 | 9 | class Node: 10 | name = '0002_auto' 11 | dependencies = ('0001_first', ) 12 | migration_dependencies = () 13 | routines = [ 14 | ] 15 | -------------------------------------------------------------------------------- /data_migration/apps.py: -------------------------------------------------------------------------------- 1 | from django.conf import settings 2 | from django.apps import AppConfig 3 | 4 | from data_migration.settings import internal_settings 5 | 6 | 7 | class DataMigrationsConfig(AppConfig): 8 | name = 'data_migration' 9 | verbose_name = 'Django data migrations' 10 | 11 | def ready(self): 12 | internal_settings.update(settings) 13 | -------------------------------------------------------------------------------- /tests/unittests/test_app/helper.py: -------------------------------------------------------------------------------- 1 | import os.path 2 | 3 | from tests.utils import ResetDirectoryMixin 4 | 5 | this_dir = os.path.dirname(__file__) 6 | 7 | 8 | class ResetDirectoryContext(ResetDirectoryMixin): 9 | targets = ['migrations', 'data_migrations'] 10 | protected_files = ['__init__.py', '0001_first.py', '0002_add_name.py'] 11 | this_dir = this_dir 12 | -------------------------------------------------------------------------------- /tests/unittests/commands/in/data_migrations/0001_first.py: -------------------------------------------------------------------------------- 1 | """ 2 | Generated by data_migration 0.0.1a 3 | 4 | File: 0001_first.py 5 | Created: 2021-05-22 11:23 6 | """ 7 | from tests.unittests.commands.test_migrate import set_some_value 8 | 9 | 10 | class Node: 11 | name = '0001_first' 12 | dependencies = () 13 | migration_dependencies = () 14 | routines = [ 15 | set_some_value 16 | ] 17 | -------------------------------------------------------------------------------- /tests/unittests/services/data_migrations/0002_auto.py: -------------------------------------------------------------------------------- 1 | """ 2 | Generated by data_migration 0.0.1a 3 | 4 | File: 0002_auto.py 5 | Created: 2021-05-22 11:23 6 | """ 7 | from tests.unittests.services.test_graph import set_some_value 8 | 9 | 10 | class Node: 11 | name = '0002_auto' 12 | dependencies = ('0001_first', ) 13 | migration_dependencies = () 14 | routines = [ 15 | set_some_value 16 | ] 17 | -------------------------------------------------------------------------------- /tests/unittests/test_app/data_migrations/0001_first.py: -------------------------------------------------------------------------------- 1 | """ 2 | Generated by data_migration 0.0.1a 3 | 4 | File: 0001_first.py 5 | Created: 2021-05-22 11:23 6 | """ 7 | from tests.unittests.commands.test_migrate import set_some_value 8 | 9 | 10 | class Node: 11 | name = '0001_first' 12 | dependencies = () 13 | migration_dependencies = ('test_app.0001_first', ) 14 | routines = [ 15 | set_some_value 16 | ] 17 | -------------------------------------------------------------------------------- /tests/unittests/test_app_2/migrations/0007_remove_customer_address.py: -------------------------------------------------------------------------------- 1 | # Generated by Django 3.2 on 2021-05-24 14:51 2 | 3 | from django.db import migrations 4 | 5 | 6 | class Migration(migrations.Migration): 7 | 8 | dependencies = [ 9 | ('test_app_2', '0006_address_line_split'), 10 | ] 11 | 12 | operations = [ 13 | migrations.RemoveField( 14 | model_name='customer', 15 | name='address', 16 | ), 17 | ] 18 | -------------------------------------------------------------------------------- /tests/unittests/test_app_2/migrations/0005_customer_address.py: -------------------------------------------------------------------------------- 1 | # Generated by Django 3.2 on 2021-05-24 14:49 2 | 3 | from django.db import migrations, models 4 | 5 | 6 | class Migration(migrations.Migration): 7 | 8 | dependencies = [ 9 | ('test_app_2', '0004_customer_is_active'), 10 | ] 11 | 12 | operations = [ 13 | migrations.AddField( 14 | model_name='customer', 15 | name='address', 16 | field=models.CharField(max_length=255, null=True), 17 | ), 18 | ] 19 | -------------------------------------------------------------------------------- /tests/unittests/test_app_2/migrations/0008_customer_is_business.py: -------------------------------------------------------------------------------- 1 | # Generated by Django 3.2 on 2021-05-24 17:20 2 | 3 | from django.db import migrations, models 4 | 5 | 6 | class Migration(migrations.Migration): 7 | 8 | dependencies = [ 9 | ('test_app_2', '0007_remove_customer_address'), 10 | ] 11 | 12 | operations = [ 13 | migrations.AddField( 14 | model_name='customer', 15 | name='is_business', 16 | field=models.BooleanField(default=False), 17 | ), 18 | ] 19 | -------------------------------------------------------------------------------- /tests/unittests/test_app/migrations/0001_first.py: -------------------------------------------------------------------------------- 1 | # Generated by Django 3.2.3 on 2021-05-22 20:22 2 | 3 | from django.db import migrations, models 4 | 5 | 6 | class Migration(migrations.Migration): 7 | 8 | initial = True 9 | 10 | dependencies = [ 11 | ] 12 | 13 | operations = [ 14 | migrations.CreateModel( 15 | name='MModel', 16 | fields=[ 17 | ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), 18 | ('bit', models.BooleanField()), 19 | ], 20 | ), 21 | ] 22 | -------------------------------------------------------------------------------- /tests/unittests/test_app_2/helper.py: -------------------------------------------------------------------------------- 1 | import os.path 2 | from tests.utils import ResetDirectoryMixin 3 | 4 | this_dir = os.path.dirname(__file__) 5 | 6 | 7 | class ResetDirectory2Context(ResetDirectoryMixin): 8 | targets = ['migrations', 'data_migrations'] 9 | protected_files = [ 10 | '0001_initial.py', '0004_customer_is_active.py', 11 | '0007_remove_customer_address.py', 12 | '0002_split_name.py', '0005_customer_address.py', 13 | '0008_customer_is_business.py', '0003_mmodel.py', 14 | '0006_address_line_split.py', '__init__.py', 15 | ] 16 | this_dir = this_dir 17 | -------------------------------------------------------------------------------- /tests/unittests/test_app_2/migrations/0001_initial.py: -------------------------------------------------------------------------------- 1 | # Generated by Django 3.2 on 2021-05-24 09:19 2 | 3 | from django.db import migrations, models 4 | 5 | 6 | class Migration(migrations.Migration): 7 | 8 | initial = True 9 | 10 | dependencies = [ 11 | ] 12 | 13 | operations = [ 14 | migrations.CreateModel( 15 | name='SomeClass', 16 | fields=[ 17 | ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), 18 | ('name', models.CharField(max_length=2, null=True)), 19 | ], 20 | ), 21 | ] 22 | -------------------------------------------------------------------------------- /tests/unittests/test_app_2/migrations/0004_customer_is_active.py: -------------------------------------------------------------------------------- 1 | # Generated by Django 3.2 on 2021-05-24 14:48 2 | 3 | from django.db import migrations, models 4 | 5 | 6 | class Migration(migrations.Migration): 7 | 8 | dependencies = [ 9 | ('test_app_2', '0003_mmodel'), 10 | ] 11 | 12 | operations = [ 13 | migrations.RenameModel( 14 | old_name='SomeClass', 15 | new_name='Customer', 16 | ), 17 | migrations.AddField( 18 | model_name='customer', 19 | name='is_active', 20 | field=models.BooleanField(default=True), 21 | ), 22 | ] 23 | -------------------------------------------------------------------------------- /tests/unittests/test_app_2/migrations/0003_mmodel.py: -------------------------------------------------------------------------------- 1 | # Generated by Django 3.2 on 2021-05-25 10:12 2 | 3 | from django.db import migrations, models 4 | 5 | 6 | class Migration(migrations.Migration): 7 | 8 | dependencies = [ 9 | ('test_app_2', '0002_split_name'), 10 | ] 11 | 12 | operations = [ 13 | migrations.CreateModel( 14 | name='MModel', 15 | fields=[ 16 | ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), 17 | ('bit', models.BooleanField()), 18 | ('name', models.CharField(max_length=2, null=True)), 19 | ], 20 | ), 21 | ] 22 | -------------------------------------------------------------------------------- /tests/unittests/test_app_2/models.py: -------------------------------------------------------------------------------- 1 | from django.db import models 2 | 3 | 4 | class Customer(models.Model): 5 | first_name = models.CharField(max_length=1, null=True) 6 | last_name = models.CharField(max_length=1, null=True) 7 | address_line_1 = models.CharField(max_length=255, null=True) 8 | address_line_2 = models.CharField(max_length=255, null=True) 9 | is_business = models.BooleanField(default=False) 10 | 11 | is_active = models.BooleanField(default=True) 12 | 13 | def __str__(self): 14 | return f'{self.first_name} {self.last_name}' 15 | 16 | 17 | class MModel(models.Model): 18 | bit = models.BooleanField() 19 | name = models.CharField(max_length=2, null=True) 20 | -------------------------------------------------------------------------------- /.coveragerc: -------------------------------------------------------------------------------- 1 | # .coveragerc to control coverage.py 2 | [run] 3 | branch = True 4 | omit = 5 | .venv/* 6 | */__init__.py 7 | */tests/* 8 | */migrations/*.py 9 | include = 10 | data_migration/* 11 | 12 | [report] 13 | # Regexes for lines to exclude from consideration 14 | exclude_lines = 15 | # Have to re-enable the standard pragma 16 | pragma: no cover 17 | 18 | # Don't complain about missing debug-only code: 19 | def __repr__ 20 | def __str__ 21 | if self\.debug 22 | 23 | # Don't complain if tests don't hit defensive assertion code: 24 | raise AssertionError 25 | raise NotImplementedError 26 | 27 | # Don't complain if non-runnable code isn't run: 28 | if 0: 29 | if __name__ == .__main__.: 30 | 31 | ignore_errors = True 32 | show_missing = True 33 | skip_covered = True 34 | 35 | [html] 36 | directory = coverage_html_report -------------------------------------------------------------------------------- /data_migration/services/templates/migration.py.txt: -------------------------------------------------------------------------------- 1 | {% comment %} 2 | 3 | Required parameters: 4 | - set_header: bool 5 | - package: str 6 | - file_name: str 7 | - date: str 8 | - routines: Routine 9 | - dependencies: list[str] 10 | - migration_dependencies: list[str] 11 | 12 | {% endcomment %}{% if set_header %}""" 13 | Generated by {{ package }} 14 | 15 | File: {{ file_name }}.py 16 | Created: {{ date }} 17 | """{% endif %} 18 | {% if routines %} 19 | import importlib.util 20 | {% for fun in routines %}spec = importlib.util.spec_from_file_location("{{fun.module}}", "{{fun.file_path}}") 21 | {{fun.module_name}} = importlib.util.module_from_spec(spec) 22 | spec.loader.exec_module({{fun.module_name}}) 23 | {% endfor %}{% endif %} 24 | 25 | class Node: 26 | name = '{{ file_name }}' 27 | dependencies = ({% for d in dependencies %}'{{ d }}', {% endfor %}) 28 | migration_dependencies = ({% for d in migration_dependencies %}'{{ d }}', {% endfor %}) 29 | routines = [ 30 | {% for fun in routines %}{{fun.module_name}}.{{fun.method}}, 31 | {% endfor %}] 32 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2021 Philipp Zettl 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /.github/workflows/test-package.yml: -------------------------------------------------------------------------------- 1 | name: Test package 2 | 3 | on: 4 | pull_request 5 | 6 | jobs: 7 | build: 8 | 9 | runs-on: ubuntu-latest 10 | strategy: 11 | fail-fast: false 12 | matrix: 13 | python-version: [3.7, 3.8, 3.9, "3.10"] 14 | django-version: [2.2, 3.2, 4.0, 4.1, 4.2] 15 | exclude: 16 | - python-version: 3.7 17 | django-version: 4.0 18 | - python-version: 3.7 19 | django-version: 4.1 20 | - python-version: 3.7 21 | django-version: 4.2 22 | 23 | steps: 24 | - uses: actions/checkout@v4 25 | - name: Set up Python ${{ matrix.python-version }} 26 | uses: actions/setup-python@v4 27 | with: 28 | python-version: ${{ matrix.python-version }} 29 | - name: Install dependencies 30 | run: | 31 | python -m pip install --upgrade pip 32 | pip install pytest coverage 33 | pip install Django==${{ matrix.django-version }} 34 | - name: Test with pytest 35 | run: | 36 | coverage run -m pytest 37 | - name: Report coverage 38 | run: | 39 | coverage report 40 | -------------------------------------------------------------------------------- /.github/workflows/test-dev.yml: -------------------------------------------------------------------------------- 1 | name: Test dev branch 2 | 3 | on: 4 | push: 5 | branches: 6 | - dev 7 | 8 | jobs: 9 | build: 10 | 11 | runs-on: ubuntu-latest 12 | strategy: 13 | fail-fast: false 14 | matrix: 15 | python-version: [3.8, 3.9, "3.10"] 16 | django-version: [2.2, 3.2, 4.0, 4.1, 4.2] 17 | exclude: 18 | - python-version: 3.7 19 | django-version: 4.0 20 | - python-version: 3.7 21 | django-version: 4.1 22 | - python-version: 3.7 23 | django-version: 4.2 24 | 25 | steps: 26 | - uses: actions/checkout@v4 27 | - name: Set up Python ${{ matrix.python-version }} 28 | uses: actions/setup-python@v4 29 | with: 30 | python-version: ${{ matrix.python-version }} 31 | - name: Install dependencies 32 | run: | 33 | python -m pip install --upgrade pip 34 | pip install pytest coverage 35 | pip install Django==${{ matrix.django-version }} 36 | - name: Test with pytest 37 | run: | 38 | coverage run -m pytest 39 | - name: Report coverage 40 | run: | 41 | coverage report -------------------------------------------------------------------------------- /data_migration/management/commands/data_migrate.py: -------------------------------------------------------------------------------- 1 | from django.core.management import BaseCommand 2 | 3 | from data_migration.services.squasher import MigrationSquash 4 | from data_migration.settings import internal_settings as data_migration_settings 5 | 6 | 7 | class Command(BaseCommand): 8 | """ 9 | Extended migrate command. 10 | 11 | Allows forward and backward migration of data/regular migrations 12 | """ 13 | 14 | def add_arguments(self, parser): # noqa D102 15 | parser.add_argument( 16 | '--app_labels', nargs='?', dest='app_labels', 17 | help='App label of an application to synchronize the state.', 18 | ) 19 | parser.add_argument( 20 | '--all', '-a', action='store_true', dest='squash_all', 21 | help='Squash all apps.', 22 | ) 23 | super().add_arguments(parser) 24 | 25 | def handle(self, *args, **options): # noqa D102 26 | # extract parameters 27 | apps_to_squash = options.get('app_labels') 28 | if apps_to_squash is None and options['squash_all']: 29 | apps_to_squash = data_migration_settings.SQUASHABLE_APPS 30 | 31 | MigrationSquash(apps_to_squash).squash() 32 | -------------------------------------------------------------------------------- /tests/unittests/test_app_2/migrations/0006_address_line_split.py: -------------------------------------------------------------------------------- 1 | # Generated by Django 3.2 on 2021-05-24 14:49 2 | 3 | from django.db import migrations, models 4 | 5 | 6 | def move_data_forward(apps, schema_editor): 7 | some_class = apps.get_model('test_app_2', 'Customer') 8 | some_class.objects.filter(address__isnull=False).update(address_line_1=models.F('address')) 9 | 10 | 11 | def move_data_back(apps, schema_editor): 12 | some_class = apps.get_model('test_app_2', 'Customer') 13 | some_class.objects.filter(address_line_1__isnull=False).update(address=models.F('address_line_1')) 14 | 15 | 16 | class Migration(migrations.Migration): 17 | 18 | dependencies = [ 19 | ('test_app_2', '0005_customer_address'), 20 | ] 21 | 22 | operations = [ 23 | migrations.AddField( 24 | model_name='customer', 25 | name='address_line_1', 26 | field=models.CharField(max_length=255, null=True), 27 | ), 28 | migrations.AddField( 29 | model_name='customer', 30 | name='address_line_2', 31 | field=models.CharField(max_length=255, null=True), 32 | ), 33 | migrations.RunPython( 34 | move_data_forward, move_data_back 35 | ) 36 | ] 37 | -------------------------------------------------------------------------------- /tests/unittests/commands/test_aliases.py: -------------------------------------------------------------------------------- 1 | from unittest import mock 2 | 3 | from django.core.management import call_command 4 | from django.test import TestCase 5 | from tests.unittests.test_app.helper import ResetDirectoryContext 6 | 7 | 8 | class AliasCommandTestCase(TestCase): 9 | @mock.patch('django.core.management.commands.migrate.Command.handle') 10 | def test_django_migrate(self, migrate_command): 11 | migrate_command.return_value = '' 12 | call_command('django_migrate') 13 | migrate_command.assert_called_once() 14 | 15 | @mock.patch('django.core.management.commands.' 16 | 'makemigrations.Command.handle') 17 | def test_django_makemigrations(self, migrate_command): 18 | with ResetDirectoryContext(): 19 | migrate_command.return_value = '' 20 | call_command('django_makemigrations', ['test_app']) 21 | migrate_command.assert_called_once() 22 | 23 | @mock.patch('django.core.management.commands.' 24 | 'squashmigrations.Command.handle') 25 | def test_django_squashmigraitons(self, migrate_command): 26 | migrate_command.return_value = '' 27 | call_command('django_squashmigrations', 'test_app', '0001_initial') 28 | migrate_command.assert_called_once() 29 | -------------------------------------------------------------------------------- /data_migration/management/commands/makemigrations.py: -------------------------------------------------------------------------------- 1 | from django.core.management.commands.makemigrations import Command as Migrate 2 | 3 | from data_migration.services.file_generator import DataMigrationGenerator 4 | 5 | 6 | class Command(Migrate): 7 | """Extended makemigrations command.""" 8 | 9 | def add_arguments(self, parser): # noqa D102 10 | parser.add_argument( 11 | '--data-only', action='store_true', dest='data_migration', 12 | help='Creates a data migration file.', 13 | ) 14 | super().add_arguments(parser) 15 | 16 | def handle(self, *app_labels, **options): # noqa D102 17 | create_empty_data_migration = options.get('data_migration') 18 | if create_empty_data_migration: 19 | is_dry_run = options.get('dry_run') 20 | return 'Generated files: ' + ', '.join([ 21 | DataMigrationGenerator( 22 | app, 23 | readable_name=options.get('name'), 24 | set_header=options.get('include_header', True), 25 | empty=options.get('empty', False), 26 | dry_run=is_dry_run 27 | ).file_name for app in app_labels]) 28 | 29 | else: 30 | return super().handle(*app_labels, **options) 31 | -------------------------------------------------------------------------------- /data_migration/management/commands/migrate.py: -------------------------------------------------------------------------------- 1 | from django.apps import apps 2 | from django.core.management import CommandError 3 | from django.core.management.commands.migrate import Command as Migrate 4 | 5 | from data_migration.services.graph import Graph 6 | 7 | 8 | class Command(Migrate): 9 | """ 10 | Extended migrate command. 11 | 12 | Allows forward and backward migration of data/regular migrations 13 | """ 14 | 15 | def add_arguments(self, parser): # noqa D102 16 | parser.add_argument( 17 | '--data-only', action='store_true', dest='data_migration', 18 | help='Applies data migrations', 19 | ) 20 | super().add_arguments(parser) 21 | 22 | def handle(self, *args, **options): # noqa D102 23 | # extract parameters 24 | 25 | if options['app_label']: 26 | # Validate app_label. 27 | app_label = options['app_label'] 28 | try: 29 | apps.get_app_config(app_label) 30 | except LookupError as err: 31 | raise CommandError(str(err)) 32 | 33 | data_migrations = Graph.from_dir(app_label) 34 | data_migrations.apply(options.get('migration_name')) 35 | 36 | if not options['data_migration']: 37 | return super().handle(*args, **options) 38 | -------------------------------------------------------------------------------- /data_migration/settings.py: -------------------------------------------------------------------------------- 1 | from typing import Dict, Optional 2 | 3 | from django.core.signals import setting_changed 4 | 5 | DATA_MIGRATION_DEFAULTS = { 6 | "SQUASHABLE_APPS": [] 7 | } 8 | 9 | 10 | class DataMigrationSettings(object): 11 | def __init__(self, user_settings=None, defaults: Optional[Dict] = None): 12 | if defaults is None: 13 | defaults = DATA_MIGRATION_DEFAULTS 14 | 15 | self.settings = defaults.copy() 16 | if user_settings: 17 | self.update(user_settings) 18 | 19 | def update(self, settings): 20 | try: 21 | self.settings.update(getattr(settings, 'DATA_MIGRATION')) 22 | except AttributeError: 23 | self.settings.update(settings.get('DATA_MIGRATION', {})) 24 | 25 | def reload(self, settings): 26 | try: 27 | _user_settings = getattr(settings, 'DATA_MIGRATION') 28 | self.settings = _user_settings 29 | except AttributeError: 30 | pass 31 | 32 | def __getattr__(self, item): 33 | return self.settings[item] 34 | 35 | 36 | internal_settings = DataMigrationSettings(None) 37 | 38 | 39 | def reload(sender, setting, value, *args, **kwargs): 40 | if setting == 'DATA_MIGRATION': 41 | internal_settings.update(value) 42 | 43 | 44 | setting_changed.connect(reload) 45 | -------------------------------------------------------------------------------- /.github/workflows/publish-package.yml: -------------------------------------------------------------------------------- 1 | # This workflow will upload a Python Package using Twine when a release is created 2 | # For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries 3 | 4 | # This workflow uses actions that are not certified by GitHub. 5 | # They are provided by a third-party and are governed by 6 | # separate terms of service, privacy policy, and support 7 | # documentation. 8 | 9 | name: Upload Python Package 10 | 11 | on: 12 | release: 13 | types: [published] 14 | workflow_dispatch: 15 | inputs: 16 | version: 17 | description: 'Package version' 18 | required: true 19 | default: '0.0.1a' 20 | 21 | jobs: 22 | deploy: 23 | 24 | runs-on: ubuntu-latest 25 | 26 | steps: 27 | - uses: actions/checkout@v4 28 | - name: Set up Python 29 | uses: actions/setup-python@v4 30 | with: 31 | python-version: '3.x' 32 | - name: Install dependencies 33 | run: | 34 | python -m pip install --upgrade pip 35 | pip install build 36 | - name: Build package 37 | run: env VERSION=${{ github.event.inputs.version }} python -m build 38 | - name: Publish package 39 | uses: pypa/gh-action-pypi-publish@fb9fc6a4e67ca27a7a76b17bbf90be83c2d3c716 40 | with: 41 | user: __token__ 42 | password: ${{ secrets.PYPI_API_TOKEN }} 43 | -------------------------------------------------------------------------------- /tests/unittests/test_app_2/migrations/0002_split_name.py: -------------------------------------------------------------------------------- 1 | # Generated by Django 3.2 on 2021-05-24 09:19 2 | 3 | from django.db import migrations, models 4 | 5 | 6 | def split_name(apps, schema_editor): 7 | some_class = apps.get_model('test_app_2', 'SomeClass') 8 | qs = some_class.objects.all() 9 | for obj in qs: 10 | if len(obj.name) >= 1: 11 | obj.first_name = obj.name[0] 12 | if len(obj.name) == 2: 13 | obj.last_name = obj.name[1] 14 | 15 | 16 | def combine_name(apps, schema_editor): 17 | some_class = apps.get_model('test_app_2', 'SomeClass') 18 | some_class.objects.all().update( 19 | name=models.F('first_name') + models.F('last_name') 20 | ) 21 | 22 | 23 | class Migration(migrations.Migration): 24 | 25 | dependencies = [ 26 | ('test_app_2', '0001_initial'), 27 | ] 28 | 29 | operations = [ 30 | migrations.AddField( 31 | model_name='someclass', 32 | name='first_name', 33 | field=models.CharField(max_length=1, null=True), 34 | ), 35 | migrations.AddField( 36 | model_name='someclass', 37 | name='last_name', 38 | field=models.CharField(max_length=1, null=True), 39 | ), 40 | migrations.RunPython( 41 | split_name, 42 | combine_name 43 | ), 44 | migrations.RemoveField( 45 | model_name='someclass', 46 | name='name', 47 | ), 48 | ] 49 | -------------------------------------------------------------------------------- /data_migration/management/commands/squashmigrations.py: -------------------------------------------------------------------------------- 1 | from django.apps import apps 2 | from django.core.management import CommandError 3 | from django.core.management.commands.squashmigrations import Command as Migrate 4 | 5 | from data_migration.services.squasher import MigrationSquash 6 | 7 | 8 | class Command(Migrate): 9 | """Extended `squashmigrations` management command.""" 10 | 11 | def add_arguments(self, parser): # noqa D102 12 | parser.add_argument( 13 | '--extract-data-migrations', 14 | dest='extract_data_migrations', 15 | action='store_true', 16 | help='Minimize current migration tree and emplace data migrations', 17 | ) 18 | parser.add_argument( 19 | '--dry-run', 20 | dest='dry_run', 21 | action='store_true', 22 | help='Run squashing dry, does not create data_migration files.', 23 | ) 24 | super().add_arguments(parser) 25 | 26 | def handle(self, **options): # noqa D102 27 | app_label = options['app_label'] 28 | extract_data_migrations = options['extract_data_migrations'] 29 | # Validate app_label. 30 | try: 31 | apps.get_app_config(app_label) 32 | except LookupError as err: 33 | raise CommandError(str(err)) 34 | 35 | if extract_data_migrations: 36 | squasher = MigrationSquash(app_label, options.get('dry_run')) 37 | squasher.squash() 38 | return squasher.log.log 39 | else: 40 | return super().handle(**options) 41 | -------------------------------------------------------------------------------- /tests/unittests/services/test_node.py: -------------------------------------------------------------------------------- 1 | from unittest import TestCase, mock 2 | 3 | from data_migration.services.node import (Node, AlreadyAppliedError, 4 | DatabaseError) 5 | from django.db import DatabaseError as DjDatabaseError 6 | 7 | 8 | class NodeTestCase(TestCase): 9 | def tearDown(self) -> None: 10 | Node.Node.objects.all().delete() 11 | 12 | def test_unique_execution(self): 13 | node = Node(app_name='test', name='0001_initial') 14 | node.apply() 15 | 16 | with self.assertRaises(AlreadyAppliedError): 17 | node.apply() 18 | 19 | def test_apply_creates_record(self): 20 | node = Node(app_name='test', name='0001_initial') 21 | self.assertIsNone(node.pk) 22 | node.apply() 23 | self.assertIsNotNone(node.pk) 24 | 25 | def test_is_applied(self): 26 | node = Node(app_name='test', name='0001_initial') 27 | self.assertFalse(node.is_applied) 28 | node.apply() 29 | self.assertTrue(node.is_applied) 30 | 31 | @mock.patch('django.db.backends.base.base.BaseDatabaseWrapper' 32 | '.schema_editor') 33 | def test_ensure_table_side_effect(self, schema_editor_mock): 34 | schema_editor_mock.side_effect = DjDatabaseError() 35 | 36 | node = Node(app_name='test', name='0001_initial') 37 | node.has_table = lambda: False 38 | 39 | with self.assertRaises(DatabaseError) as ex: 40 | node.ensure_table() 41 | self.assertEqual(str(ex), 'Table "data_migrations" not creatable.') 42 | -------------------------------------------------------------------------------- /tests/unittests/commands/test_data_migrate.py: -------------------------------------------------------------------------------- 1 | import os 2 | from unittest import TestCase, mock 3 | 4 | from data_migration.settings import internal_settings 5 | from tests.unittests.test_app_2.helper import ResetDirectory2Context 6 | from tests.utils import FileTestCase 7 | 8 | from django.core.management import call_command 9 | 10 | this_dir = os.path.dirname(__file__) 11 | 12 | 13 | class DataMigrateCommandTestCase(FileTestCase): 14 | internal_target = os.path.join(this_dir, '../test_app_2/') 15 | needs_cleanup = False 16 | 17 | def test_explicit_app(self): 18 | with ResetDirectory2Context(): 19 | call_command('data_migrate', app_labels=['test_app_2']) 20 | prev_target = self.target 21 | self.target = os.path.join(prev_target, 'data_migrations/') 22 | self.assertTrue(self.has_file('0001_0002_split_name.py')) 23 | self.target = os.path.join(prev_target, 'migrations/') 24 | self.assertTrue(self.has_file('0001_squashed_0008.py')) 25 | 26 | def test_all_apps(self): 27 | self.assertIsNotNone(internal_settings.SQUASHABLE_APPS) 28 | with ResetDirectory2Context(): 29 | call_command('data_migrate', squash_all=True) 30 | prev_target = self.target 31 | self.target = os.path.join(prev_target, 'data_migrations/') 32 | self.assertTrue(self.has_file('0001_0002_split_name.py')) 33 | self.assertTrue(self.has_file('0002_0006_address_line_split.py')) 34 | self.target = os.path.join(prev_target, 'migrations/') 35 | self.assertTrue(self.has_file('0001_squashed_0008.py')) 36 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | from setuptools import setup 4 | 5 | 6 | def read(fname): 7 | return open(os.path.join(os.path.dirname(__file__), fname)).read() 8 | 9 | 10 | setup( 11 | name='django-data-migrations', 12 | version=os.getenv('VERSION'), 13 | packages=['data_migration'], 14 | author='Philipp Zettl', 15 | author_email='philipp.zett@godesteem.de', 16 | include_package_data=True, 17 | description='Extraction tool for data only django migrations', 18 | keywords='django,database migrations', 19 | url='https://github.com/philsupertramp/django-data-migration/', 20 | license='MIT', 21 | install_requires=[ 22 | 'django >= 2.2' 23 | ], 24 | long_description=read('README.rst'), 25 | long_description_content_type='text/x-rst', 26 | classifiers=[ 27 | # 3 - Alpha 28 | # 4 - Beta 29 | # 5 - Production/Stable 30 | "Development Status :: 4 - Beta", 31 | "Environment :: Web Environment", 32 | "Framework :: Django", 33 | "Framework :: Django :: 3.0", 34 | "Framework :: Django :: 3.1", 35 | "Framework :: Django :: 3.2", 36 | "Framework :: Django :: 2.2", 37 | "Intended Audience :: Developers", 38 | "License :: OSI Approved :: MIT License", 39 | "Operating System :: OS Independent", 40 | "Programming Language :: Python", 41 | "Programming Language :: Python :: 3", 42 | "Programming Language :: Python :: 3 :: Only", 43 | "Programming Language :: Python :: 3.6", 44 | "Programming Language :: Python :: 3.7", 45 | "Programming Language :: Python :: 3.8", 46 | "Topic :: Internet :: WWW/HTTP", 47 | "Topic :: Internet :: WWW/HTTP :: Dynamic Content" 48 | ] 49 | ) 50 | -------------------------------------------------------------------------------- /tests/integrationtests/test_node.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | 3 | from data_migration.services.node import classproperty 4 | 5 | 6 | class NodeTestCase(unittest.TestCase): 7 | def test_classproperty_class_can_be_used_without_django(self): 8 | class WrapperClass: 9 | _model_class = None 10 | 11 | @classproperty 12 | def InnerClass(cls): 13 | if cls._model_class is None: 14 | from django.db import models 15 | from django.apps.registry import Apps 16 | 17 | class DjangoModelClass(models.Model): 18 | value = models.TextField() 19 | 20 | class Meta: 21 | apps = Apps() 22 | app_label = 'model_class' 23 | db_table = 'model_class' 24 | cls._model_class = DjangoModelClass 25 | return cls._model_class 26 | 27 | def __init__(self, name): 28 | self.name = name 29 | 30 | @property 31 | def qs(self): 32 | return self._model_class.objects 33 | 34 | def __str__(self): 35 | return self.name 36 | 37 | wrapper = WrapperClass('wrapper') 38 | 39 | self.assertEqual(str(wrapper), 'wrapper') 40 | 41 | def test_classproperty_getter(self): 42 | class Foo: 43 | @classproperty 44 | def bar(cls): 45 | class Bar: 46 | pass 47 | return Bar 48 | 49 | foo = Foo() 50 | 51 | # access as member 52 | bar = foo.bar 53 | self.assertIsNotNone(bar()) 54 | 55 | # access trough class 56 | self.assertIsNotNone(Foo.bar()) 57 | -------------------------------------------------------------------------------- /tests/unittests/test_settings.py: -------------------------------------------------------------------------------- 1 | from unittest import TestCase, mock 2 | 3 | from data_migration.settings import DataMigrationSettings, reload 4 | 5 | 6 | class SettingsTestCase(TestCase): 7 | def test_uses_user_setting(self): 8 | from django.conf import settings as django_settings 9 | settings = DataMigrationSettings(django_settings) 10 | self.assertEqual(settings.SQUASHABLE_APPS, django_settings.DATA_MIGRATION.get('SQUASHABLE_APPS')) 11 | 12 | def test_update(self): 13 | from django.conf import settings as django_settings 14 | settings = DataMigrationSettings(django_settings) 15 | settings.update({'DATA_MIGRATION': {'SQUASHABLE_APPS': ['foo']}}) 16 | 17 | self.assertEqual(settings.SQUASHABLE_APPS, ['foo']) 18 | 19 | def test_reload(self): 20 | from django.conf import settings as django_settings 21 | 22 | settings = DataMigrationSettings(None, {'SQUASHABLE_APPS': ['foo']}) 23 | 24 | self.assertEqual(settings.SQUASHABLE_APPS, ['foo']) 25 | 26 | settings.reload(django_settings) 27 | 28 | self.assertNotEqual(settings.SQUASHABLE_APPS, ['foo']) 29 | 30 | def test_reload_fails_silently(self): 31 | settings = DataMigrationSettings(None, {'SQUASHABLE_APPS': ['foo']}) 32 | 33 | settings.reload(None) 34 | 35 | self.assertEqual(settings.SQUASHABLE_APPS, ['foo']) 36 | 37 | @mock.patch('data_migration.settings.DataMigrationSettings.update') 38 | def test_reload_on_signal(self, update_mock): 39 | update_payload = {'SQUASHABLE_APPS': ['foo']} 40 | reload(None, 'DATA_MIGRATION', update_payload) 41 | 42 | update_mock.assert_called_once() 43 | update_mock.assert_called_with(update_payload) 44 | update_mock.reset_mock() 45 | 46 | reload(None, 'FOO', update_payload) 47 | 48 | update_mock.assert_not_called() 49 | -------------------------------------------------------------------------------- /tests/unittests/commands/test_makemigrations.py: -------------------------------------------------------------------------------- 1 | import os 2 | from unittest import mock 3 | 4 | from django.core.management import call_command 5 | from tests.unittests.test_app.helper import ResetDirectoryContext 6 | from tests.utils import FileTestCase 7 | 8 | this_dir = os.path.dirname(__file__) 9 | 10 | 11 | def with_test_output_directory(fun): 12 | def inner(*args, **kwargs): 13 | with mock.patch('django.apps.apps.get_app_config') as dir_mock: 14 | dir_mock.return_value = mock.Mock( 15 | path=os.path.join(this_dir, 'out') 16 | ) 17 | return fun(*args, **kwargs) 18 | 19 | return inner 20 | 21 | 22 | class MakemigrationsCommandTestCase(FileTestCase): 23 | internal_target = os.path.join(this_dir, 'out/data_migrations') 24 | needs_cleanup = False 25 | 26 | def tearDown(self) -> None: 27 | if self.needs_cleanup: 28 | self.clean_directory() 29 | self.needs_cleanup = False 30 | 31 | @mock.patch('django.core.management.commands.' 32 | 'makemigrations.Command.handle') 33 | def test_extends_default(self, migrate_command): 34 | migrate_command.return_value = 'Ok.' 35 | with ResetDirectoryContext(): 36 | call_command('makemigrations', 'test_app') 37 | migrate_command.assert_called_once() 38 | 39 | @with_test_output_directory 40 | @mock.patch('django.core.management.commands.' 41 | 'makemigrations.Command.handle') 42 | def test_create_data_migration_file(self, migrate_command): 43 | with ResetDirectoryContext(): 44 | call_command('makemigrations', ['test_app'], data_migration=True) 45 | migrate_command.assert_not_called() 46 | self.assertTrue(self.has_file('0001_first.py')) 47 | self.needs_cleanup = True 48 | 49 | @with_test_output_directory 50 | @mock.patch('django.core.management.commands.' 51 | 'makemigrations.Command.handle') 52 | def test_dry_run_does_not_create_files(self, migrate_command): 53 | with ResetDirectoryContext(): 54 | call_command( 55 | 'makemigrations', 56 | ['test_app'], 57 | data_migration=True, 58 | dry_run=True 59 | ) 60 | migrate_command.assert_not_called() 61 | self.assertFalse(self.has_file('0001_first.py')) 62 | -------------------------------------------------------------------------------- /tests/unittests/commands/test_squashmigrations.py: -------------------------------------------------------------------------------- 1 | import io 2 | import os 3 | from unittest import mock, TestCase 4 | 5 | from django.core.management import call_command, CommandError 6 | from tests.unittests.test_app.helper import ResetDirectoryContext 7 | from tests.unittests.test_app_2.helper import ResetDirectory2Context 8 | from tests.utils import FileTestCase 9 | from data_migration.services.squasher import Log 10 | 11 | this_dir = os.path.dirname(__file__) 12 | 13 | 14 | def with_test_output_directory(fun): 15 | def inner(*args, **kwargs): 16 | with mock.patch('django.apps.apps.get_app_config') as dir_mock: 17 | dir_mock.return_value = mock.Mock( 18 | path=os.path.join(this_dir, 'out') 19 | ) 20 | return fun(*args, **kwargs) 21 | 22 | return inner 23 | 24 | 25 | class SquashmigrationsCommandTestCase(FileTestCase): 26 | internal_target = os.path.join(this_dir, '../test_app_2/data_migrations') 27 | needs_cleanup = False 28 | 29 | def tearDown(self) -> None: 30 | if self.needs_cleanup: 31 | self.clean_directory() 32 | self.needs_cleanup = False 33 | 34 | @mock.patch('django.core.management.commands.' 35 | 'squashmigrations.Command.handle') 36 | def test_extends_default(self, migrate_command): 37 | migrate_command.return_value = 'Ok.' 38 | with ResetDirectoryContext(): 39 | call_command('squashmigrations', 'test_app', '0001') 40 | migrate_command.assert_called_once() 41 | 42 | @mock.patch('django.core.management.commands.' 43 | 'squashmigrations.Command.handle') 44 | def test_app_not_found(self, migrate_command): 45 | migrate_command.return_value = 'Ok.' 46 | with self.assertRaises(CommandError): 47 | call_command('squashmigrations', 'foobar', '0001') 48 | 49 | def test_extended_squashing(self): 50 | with ResetDirectory2Context(): 51 | call_command( 52 | 'squashmigrations', 53 | 'test_app_2', 54 | '0001', 55 | extract_data_migrations=True 56 | ) 57 | self.assertTrue(self.has_file('0001_0002_split_name.py')) 58 | self.assertTrue(self.has_file('0002_0006_address_line_split.py')) 59 | 60 | def test_dry_run_doesnt_create_files(self): 61 | with ResetDirectory2Context(): 62 | call_command( 63 | 'squashmigrations', 64 | 'test_app_2', 65 | '0001', 66 | extract_data_migrations=True, 67 | dry_run=True 68 | ) 69 | self.assertFalse(self.has_file('0001_0002_split_name.py')) 70 | self.assertFalse(self.has_file('0002_0006_address_line_split.py')) 71 | 72 | def test_squash_without_data_migrations(self): 73 | with ResetDirectory2Context(): 74 | call_command( 75 | 'squashmigrations', 76 | 'test_app_2', 77 | '0007', 78 | extract_data_migrations=True, 79 | ) 80 | self.assertFalse(self.has_file('0007_0008_customer_is_business.py')) 81 | 82 | 83 | class LogTestCase(TestCase): 84 | def setUp(self) -> None: 85 | super().setUp() 86 | self.output = io.StringIO('') 87 | self.log = Log(self.output) 88 | 89 | def test_write(self): 90 | input_string = 'Hello' 91 | self.log.write(input_string) 92 | self.assertEqual(self.log.log, input_string + '\n') 93 | 94 | def test_flush(self): 95 | input_string = 'Hello' 96 | self.log.write(input_string) 97 | self.log.flush() 98 | self.assertEqual(self.log.stdout.getvalue(), input_string + '\n') 99 | self.assertTrue(self.output.getvalue(), input_string + '\n') 100 | -------------------------------------------------------------------------------- /data_migration/services/node.py: -------------------------------------------------------------------------------- 1 | class classproperty: 2 | """ 3 | Decorator that converts a method with a single cls argument into a property 4 | that can be accessed directly from the class. 5 | """ 6 | def __init__(self, method=None): 7 | self.fget = method 8 | 9 | def __get__(self, instance, cls=None): 10 | return self.fget(cls) 11 | 12 | 13 | class AlreadyAppliedError(Exception): 14 | def __init__(self, node: 'Node.Node'): 15 | super().__init__(f'Node {node} already applied. Do not reapply them!') 16 | 17 | 18 | class DatabaseError(Exception): 19 | def __init__(self, message): 20 | super().__init__(message) 21 | 22 | 23 | class Node: 24 | """ 25 | This is literally the same as django's MigrationRecorder 26 | """ 27 | _node_model = None 28 | 29 | @classproperty 30 | def Node(cls): 31 | """ 32 | bypass missing appconfig 33 | """ 34 | if cls._node_model is None: 35 | from django.apps.registry import Apps 36 | from django.db import models 37 | 38 | class NodeClass(models.Model): 39 | app_name = models.CharField(max_length=255) 40 | name = models.CharField(max_length=255) 41 | created_at = models.DateTimeField() 42 | 43 | class Meta: 44 | apps = Apps() 45 | app_label = 'data_migration' 46 | db_table = 'data_migrations' 47 | get_latest_by = 'created_at' 48 | constraints = [ 49 | models.UniqueConstraint( 50 | fields=['app_name', 'name'], 51 | name='unique_name_for_app' 52 | ) 53 | ] 54 | 55 | cls._node_model = NodeClass 56 | return cls._node_model 57 | 58 | def __init__(self, name, app_name, *args, **kwargs): 59 | self.pk = kwargs.get('pk') 60 | self.name = name 61 | self.app_name = app_name 62 | self.created_at = kwargs.get('created_at') 63 | 64 | @property 65 | def is_applied(self): 66 | return bool(self.created_at and self.pk) 67 | 68 | def apply(self) -> None: 69 | if self.pk: 70 | raise AlreadyAppliedError(self) 71 | 72 | self.ensure_table() 73 | from django.utils import timezone 74 | self.created_at = timezone.now() 75 | 76 | obj = self.Node( 77 | name=self.name, 78 | app_name=self.app_name, 79 | created_at=self.created_at 80 | ) 81 | obj.save() 82 | self.pk = obj.pk 83 | self.created_at = obj.created_at 84 | 85 | @property 86 | def qs(self): 87 | return self.Node.objects 88 | 89 | def exists(self): 90 | self.ensure_table() 91 | return self.qs.filter( 92 | app_name=self.app_name, 93 | name=self.name 94 | ).exists() 95 | 96 | @classmethod 97 | def flush(cls): 98 | return cls.get_qs().delete() 99 | 100 | @classmethod 101 | def get_qs(cls): 102 | node = cls('', '') 103 | node.ensure_table() 104 | return node.qs.all() 105 | 106 | def has_table(self): 107 | from django.db import connections 108 | with connections['default'].cursor() as cursor: 109 | tables = connections['default'].introspection.table_names(cursor) 110 | return self.Node._meta.db_table in tables 111 | 112 | def ensure_table(self): 113 | if self.has_table(): 114 | return 115 | from django.db import connections, DatabaseError as DjDatabaseError 116 | # Make the table 117 | try: 118 | with connections['default'].schema_editor() as editor: 119 | editor.create_model(self.Node) 120 | except DjDatabaseError as ex: 121 | raise DatabaseError( 122 | f'Table "data_migrations" not creatable ({str(ex)}' 123 | ) 124 | -------------------------------------------------------------------------------- /tests/unittests/commands/test_migrate.py: -------------------------------------------------------------------------------- 1 | import os.path 2 | import time 3 | from unittest import mock 4 | 5 | from data_migration.services.node import Node 6 | from django.core.management import call_command, CommandError 7 | from tests.unittests.test_app.helper import ResetDirectoryContext 8 | from tests.utils import TransactionalTestCase 9 | 10 | this_dir = os.path.dirname(__file__) 11 | 12 | old_value = 0 13 | new_value = 10 14 | some_other_value = old_value 15 | 16 | 17 | def set_some_value(apps, schema_editor) -> None: 18 | global some_other_value 19 | some_other_value += new_value 20 | 21 | 22 | class MigrateCommandTestCase(TransactionalTestCase): 23 | def tearDown(self) -> None: 24 | self.reset_global_state() 25 | Node.flush() 26 | 27 | @staticmethod 28 | def reset_global_state(): 29 | # reset state 30 | global some_other_value 31 | some_other_value = old_value 32 | 33 | @staticmethod 34 | def get_val(): 35 | global some_other_value 36 | return some_other_value 37 | 38 | @mock.patch('django.core.management.commands.migrate.Command.handle') 39 | def test_extends_default(self, migrate_command): 40 | migrate_command.return_value = 'Ok.' 41 | call_command('migrate') 42 | migrate_command.assert_called_once() 43 | self.assertEqual(some_other_value, old_value) 44 | 45 | @mock.patch('django.core.management.commands.migrate.Command.handle') 46 | def test_only_data(self, migrate_command): 47 | migrate_command.return_value = 'Ok.' 48 | call_command('migrate', data_migration=True) 49 | migrate_command.assert_not_called() 50 | self.assertEqual(some_other_value, old_value) 51 | 52 | @mock.patch('data_migration.services.graph.GraphNode.set_applied') 53 | @mock.patch('django.db.migrations.loader.MigrationLoader.' 54 | 'migrations_module', 55 | return_value=('django.contrib.contenttypes.' 56 | 'migrations', '__first__')) 57 | @mock.patch('django.apps.apps.get_app_config') 58 | @mock.patch('django.core.management.commands.migrate.Command.handle') 59 | def test_app_label(self, migrate_command, 60 | get_app_config_mock, migration_module_mock, 61 | set_applied_mock): 62 | global some_other_value, new_value, old_value 63 | 64 | migrate_command.return_value = 'Ok.' 65 | get_app_config_mock.return_value = mock.Mock( 66 | module=mock.Mock(__name__='tests.unittests.commands.in'), 67 | path=os.path.join(this_dir, 'in') 68 | ) 69 | self.assertEqual(some_other_value, old_value) 70 | 71 | call_command('migrate', app_label='tests.unittests.commands.in') 72 | 73 | migrate_command.assert_called_once() 74 | self.assertEqual(some_other_value, new_value) 75 | 76 | def test_validates_app_label(self): 77 | with self.assertRaises(CommandError): 78 | call_command('migrate', app_label='foobar123123') 79 | 80 | 81 | class ExtendedMigrateCommandTestCase(TransactionalTestCase): 82 | def tearDown(self) -> None: 83 | self.reset_global_state() 84 | Node.flush() 85 | 86 | @staticmethod 87 | def reset_global_state(): 88 | # reset state 89 | global some_other_value 90 | some_other_value = old_value 91 | 92 | @staticmethod 93 | def get_val(): 94 | global some_other_value 95 | return some_other_value 96 | 97 | @mock.patch('data_migration.services.graph.GraphNode.set_applied') 98 | @mock.patch('django.db.migrations.loader.MigrationLoader.' 99 | 'migrations_module', 100 | return_value=('tests.unittests.test_app.migrations', 101 | '__first__')) 102 | def test_migrate_with_leaf_migration(self, migration_module_mock, 103 | set_applied_mock): 104 | with ResetDirectoryContext(): 105 | call_command( 106 | 'migrate', 107 | app_label='test_app', 108 | migration_name='zero', 109 | data_migration=True 110 | ) 111 | 112 | self.assertEqual(self.get_val(), old_value) 113 | 114 | call_command('makemigrations', ['test_app'], data_migration=True) 115 | self.run_commit_hooks() 116 | time.sleep(1) 117 | call_command('migrate', app_label='test_app') 118 | 119 | self.assertEqual(self.get_val(), new_value) 120 | -------------------------------------------------------------------------------- /tests/unittests/services/test_file_generator.py: -------------------------------------------------------------------------------- 1 | import os 2 | from unittest import mock 3 | 4 | from django.db import connections 5 | from django.db.migrations.recorder import MigrationRecorder 6 | 7 | from data_migration.services.file_generator import DataMigrationGenerator 8 | 9 | from tests.utils import FileTestCase 10 | 11 | 12 | this_dir = os.path.dirname(__file__) 13 | 14 | 15 | def with_test_output_directory(fun): 16 | def inner(*args, **kwargs): 17 | with mock.patch('django.apps.apps.get_app_config') as dir_mock: 18 | dir_mock.return_value = mock.Mock( 19 | path=os.path.join(this_dir, 'out') 20 | ) 21 | return fun(*args, **kwargs) 22 | 23 | return inner 24 | 25 | 26 | class DataMigrationGeneratorTestCase(FileTestCase): 27 | internal_target = os.path.join(this_dir, 'out/data_migrations') 28 | 29 | def tearDown(self) -> None: 30 | self.clean_directory() 31 | 32 | @with_test_output_directory 33 | def test_generates_file(self): 34 | DataMigrationGenerator('test') 35 | self.assertTrue(self.has_file('0001_first.py')) 36 | 37 | @with_test_output_directory 38 | @mock.patch('data_migration.services.file_generator.' 39 | 'DataMigrationGenerator.render_template') 40 | def test_empty(self, render_mock): 41 | DataMigrationGenerator('test', empty=True) 42 | render_mock.assert_not_called() 43 | 44 | @with_test_output_directory 45 | @mock.patch('data_migration.services.file_generator.' 46 | 'DataMigrationGenerator.render_template') 47 | def test_without_header(self, render_mock): 48 | render_mock.return_value = '' 49 | DataMigrationGenerator('test', set_header=False) 50 | render_mock.assert_called_once() 51 | render_mock.called_with(file_name='0001_first', set_header=False) 52 | 53 | @with_test_output_directory 54 | def test_create_with_existing_directory(self): 55 | dir_path = os.path.join(this_dir, 'out/data_migrations') 56 | os.mkdir(dir_path) 57 | 58 | file = open(os.path.join(dir_path, '__init__.py'), 'x') 59 | file.close() 60 | 61 | DataMigrationGenerator('test', 'bigStart') 62 | self.assertTrue(self.has_file('0001_bigStart.py')) 63 | 64 | DataMigrationGenerator('test', 'auto') 65 | self.assertTrue(self.has_file('0002_auto.py')) 66 | 67 | DataMigrationGenerator('test') 68 | self.assertTrue(self.has_file('0003_auto.py')) 69 | 70 | @with_test_output_directory 71 | def test_sets_latest_migration_dependency(self): 72 | migration_name = '0001_init' 73 | recorder = MigrationRecorder(connections['default']) 74 | recorder.Migration(app='test', name=migration_name).save() 75 | DataMigrationGenerator('test') 76 | file_content = self.get_file('0001_first.py') 77 | 78 | self.assertIn(f'test.{migration_name}', file_content) 79 | 80 | @with_test_output_directory 81 | def test_sets_latest_data_migration_dependency(self): 82 | DataMigrationGenerator('test') 83 | DataMigrationGenerator('test') 84 | file_content = self.get_file('0002_auto.py') 85 | 86 | self.assertIn('0001_first', file_content) 87 | 88 | @with_test_output_directory 89 | def test_init_with_app_name_list(self): 90 | DataMigrationGenerator(['test']) 91 | self.assertTrue(self.has_file('0001_first.py')) 92 | 93 | @with_test_output_directory 94 | def test_with_dependencies(self): 95 | migration_dependencies = ['test.0001_init'] 96 | DataMigrationGenerator(['test'], readable_name='test', 97 | migration_dependencies=migration_dependencies) 98 | self.assertTrue(self.has_file('0001_test.py')) 99 | 100 | node = self.get_data_migration_node( 101 | 'tests.unittests.services.out.data_migrations.0001_test' 102 | ) 103 | 104 | self.assertIsNotNone(node) 105 | self.assertTrue(hasattr(node, 'migration_dependencies')) 106 | self.assertEqual( 107 | set(node.migration_dependencies), 108 | set(migration_dependencies) 109 | ) 110 | self.assertTrue(hasattr(node, 'dependencies')) 111 | 112 | @mock.patch('django.apps.apps.get_app_config') 113 | @mock.patch('importlib.import_module') 114 | def test_set_applied_fails_gracefully(self, import_mock, app_config_mock): 115 | app_config_mock.return_value = mock.Mock( 116 | path=os.path.join(this_dir, 'out'), module=mock.Mock(__name__='foo') 117 | ) 118 | import_mock.side_effect = ModuleNotFoundError() 119 | # works without issue 120 | DataMigrationGenerator(['foo']).set_applied() 121 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | django-data-migration 2 | ===================== 3 | 4 | |Test dev branch| 5 | |Pypi| 6 | 7 | | Developing and maintaining a django project over many years can start to become a constant fight against time consuming tasks including execution of a test suite, recreation of a local environment or setting up a project in a new environment. 8 | 9 | | Due to different flavors of deployment and/or different approaches within the same working environment migration files of long running django applications tent to be bloated and contain unnecessary code. Sometimes we even create migrations in the purpose of single-time usage to move, edit, duplicate or basically modify data. 10 | 11 | | Generally speaking, the idea behind it is clever. 12 | 13 | | With this approach you gained the option to trigger the execution of leaf migrations prior to starting your updated application code. 14 | 15 | .. code:: text 16 | 17 | Missing migration? 18 | / \ 19 | yes, migrate no, continue 20 | \ / 21 | restart app with new code 22 | 23 | | But on the other hand you create a new node within a already giant migration graph. 24 | | This is where ``django-data-migration`` comes in place. It is a drop-in replacement for regular migrations, without the need of a dedicated node in the migration tree. 25 | | It does that, by providing a "data-only" migration graph, that can optionally be maintained automatically in parallel with the existing migration graph, or executed independently, depending on your needs. 26 | 27 | Installation 28 | ============ 29 | 30 | Install package: 31 | 32 | | ``pip install django-data-migrations`` 33 | 34 | | Configure package in Django settings: 35 | 36 | .. code:: python 37 | 38 | INSTALLED_APPS = [ 39 | # django apps 40 | 'data_migration', 41 | # your apps 42 | ] 43 | 44 | Configuration 45 | ============= 46 | 47 | | The package is configurable using the 48 | 49 | .. code:: python 50 | 51 | DATA_MIGRATION = {} 52 | 53 | setting. 54 | 55 | Currently supported attributes: 56 | 57 | - ``SQUASHABLE_APPS``: a list of app(-label) names which allow squashing, you should only provide your own apps here 58 | 59 | 60 | Usage 61 | ===== 62 | 63 | Extended management commands: 64 | - ``makemigrations`` 65 | - ``migrate`` 66 | - ``squashmigrations`` 67 | - ``data_migrate`` 68 | 69 | ``makemigrations`` 70 | ~~~~~~~~~~~~~~~~~~ 71 | 72 | .. code:: shell 73 | 74 | # generate data migration file 75 | ./manage.py makemigrations --data-only [app_name] 76 | 77 | # generate data migration file with readable name "name_change" 78 | ./manage.py makemigrations --data-only [app_name] name_change 79 | 80 | # generate empty file 81 | ./manage.py makemigrations --data-only [app_name] --empty 82 | 83 | # generate without fileheader 84 | ./manage.py makemigrations --data-only [app_name] --no-header 85 | 86 | The ``makemigrations`` command generates a file 87 | ``[app_name]/data_migrations/[id]_[name].py`` with content like 88 | 89 | .. code:: python 90 | 91 | class Node: 92 | name = '0001_first' 93 | dependencies = () 94 | migration_dependencies = ('testapp.0001_initial', ) 95 | routines = [ 96 | ] 97 | 98 | ``migrate`` 99 | ~~~~~~~~~~~ 100 | 101 | .. code:: shell 102 | 103 | # apply data migration file 104 | ./manage.py migrate --data-only 105 | 106 | # revert complete data migration state 107 | ./manage.py migrate --data-only zero 108 | 109 | # revert partial data migration state 110 | ./manage.py migrate --data-only 0002_some_big_change 111 | 112 | 113 | 114 | ``squashmigrations`` 115 | ~~~~~~~~~~~~~~~~~~~~ 116 | 117 | | App-wise squashing of data/regular migrations. 118 | 119 | .. code:: shell 120 | 121 | # regular squashing of test_app migrations 0001-0015 122 | ./manage.py squashmigrations test_app 0001 0015 123 | 124 | # squash and replace test_app migrations 0001-0015 and extract data_migrations 125 | ./manage.py squashmigrations --extract-data-migrations test_app 0001 0015 126 | 127 | ``data_migrate`` 128 | ~~~~~~~~~~~~~~~~ 129 | 130 | | Extended squashing. Allows squashing a single app, a list of apps, or all apps at once. 131 | 132 | .. code:: shell 133 | 134 | # squash and replace all migrations at once 135 | ./manage.py data_migrate --all 136 | 137 | # squash and replace migrations app-wise 138 | ./manage.py data_migrate test_app 139 | 140 | 141 | Development 142 | =========== 143 | 144 | To develop the package further set up a local environment using the 145 | provided ``./dev-requirements.txt`` file. 146 | 147 | To run the test suite and generate a coverage report run 148 | 149 | .. code:: shell 150 | 151 | coverage run -m pytest -v tests 152 | coverage [html|report] 153 | 154 | .. |Test dev branch| image:: https://github.com/philsupertramp/django-data-migration/actions/workflows/test-dev.yml/badge.svg?branch=dev 155 | :target: https://github.com/philsupertramp/django-data-migration/actions/workflows/test-dev.yml 156 | 157 | .. |Pypi| image:: https://badge.fury.io/py/django-data-migrations.svg 158 | :target: https://badge.fury.io/py/django-data-migrations 159 | -------------------------------------------------------------------------------- /tests/unittests/services/test_graph.py: -------------------------------------------------------------------------------- 1 | import os 2 | from unittest import mock 3 | 4 | from django.db.migrations.exceptions import NodeNotFoundError 5 | 6 | from data_migration.services.node import Node 7 | from data_migration.services.graph import Graph, GraphNode 8 | from tests.utils import TransactionalTestCase 9 | 10 | old_value = 0 11 | new_value = 10 12 | some_value = old_value 13 | 14 | this_dir = os.path.dirname(__file__) 15 | 16 | 17 | def set_some_value(apps, schema_editor) -> None: 18 | global some_value, new_value 19 | some_value += new_value 20 | 21 | 22 | class GraphTestCase(TransactionalTestCase): 23 | def setUp(self): 24 | self.reset_global_state() 25 | Node.flush() 26 | 27 | @staticmethod 28 | def reset_global_state(): 29 | # reset state 30 | global some_value 31 | some_value = old_value 32 | 33 | @mock.patch('data_migration.services.graph.GraphNode.set_applied') 34 | def test_apply(self, set_applied_mock): 35 | g = Graph('test') 36 | node = GraphNode('test', '0001_init', [], [], [set_some_value]) 37 | g.push_back(node) 38 | 39 | self.assertEqual(some_value, old_value) 40 | 41 | # apply data migration 42 | g.apply() 43 | 44 | self.assertEqual(some_value, new_value) 45 | 46 | # fail save, reapplying doesn't change the outcome 47 | g.apply() 48 | 49 | self.assertEqual(some_value, new_value) 50 | 51 | @mock.patch('data_migration.services.graph.GraphNode.set_applied') 52 | def test_revert(self, set_applied_mock): 53 | g = Graph('test') 54 | g.push_back(GraphNode('test', '0001_init', [], [], [])) 55 | g.push_back(GraphNode('test', '0002_auto', ['0001_init'], [], [])) 56 | g.push_back(GraphNode('test', '0003_auto', ['0002_auto'], [], [])) 57 | g.apply() 58 | self.assertEqual(Node.get_qs().filter(app_name='test').count(), 3) 59 | g.apply('zero') 60 | self.assertEqual(Node.get_qs().filter(app_name='test').count(), 0) 61 | g.apply('0001_init') 62 | self.assertEqual(Node.get_qs().filter(app_name='test').count(), 1) 63 | g.apply('0001_init') 64 | self.assertEqual(Node.get_qs().filter(app_name='test').count(), 1) 65 | g.apply('0002_auto') 66 | self.assertEqual(Node.get_qs().filter(app_name='test').count(), 2) 67 | g.apply() 68 | self.assertEqual(Node.get_qs().filter(app_name='test').count(), 3) 69 | g.apply('zero') 70 | g.apply('0002_auto') 71 | self.assertEqual(Node.get_qs().filter(app_name='test').count(), 2) 72 | 73 | def test_wrong_base(self): 74 | g = Graph('test') 75 | with self.assertRaises(Graph.MigrationNotFoundError) as ex: 76 | g.apply('foo123') 77 | self.assertEqual(str(ex), 'Data migration "foo123" not found.') 78 | 79 | g.push_back(GraphNode('test', 'some_name', [], [], [])) 80 | 81 | with self.assertRaises(Graph.MigrationNotFoundError) as ex: 82 | g.apply('foo123') 83 | self.assertEqual(str(ex), 'Data migration "foo123" not found.') 84 | 85 | @mock.patch('data_migration.services.graph.GraphNode.set_applied') 86 | @mock.patch('django.db.migrations.loader.MigrationLoader' 87 | '.migrations_module', 88 | return_value=('django.contrib.contenttypes.migrations', 89 | '__first__')) 90 | @mock.patch('django.apps.apps.get_app_config') 91 | def test_from_dir(self, get_app_config_mock, migrations_module_mock, 92 | set_applied_mock): 93 | get_app_config_mock.return_value = mock.Mock( 94 | module=mock.Mock(__name__='tests.unittests.services'), 95 | path=this_dir 96 | ) 97 | g = Graph.from_dir('tests.unittests.services') 98 | g.apply() 99 | 100 | self.assertEqual(some_value, new_value) 101 | 102 | @mock.patch('data_migration.services.graph.GraphNode.set_applied') 103 | def test_unapplied_dependency(self, set_applied_mock): 104 | g = Graph('test') 105 | node = GraphNode('test', '0001_init', [], ['foobar.0001_init'], []) 106 | g.push_back( 107 | node 108 | ) 109 | 110 | with self.assertRaises(NodeNotFoundError): 111 | g.apply() 112 | 113 | @mock.patch('data_migration.services.graph.GraphNode.set_applied') 114 | def test_rebuilding_uses_existing_nodes(self, set_applied_mock): 115 | g = Graph('test') 116 | g.push_back(GraphNode('test', '0001_init', [], [], [])) 117 | g.apply() 118 | node_id = g.base_node.node.pk 119 | 120 | new_graph_node = GraphNode('test', '0001_init', [], [], []) 121 | 122 | self.assertEqual(new_graph_node.node.pk, node_id) 123 | 124 | def test_fail_silently(self): 125 | g = Graph('test') 126 | with self.assertRaises(Graph.EmptyGraphError): 127 | g.apply() 128 | 129 | g.apply(fail_silently=True) 130 | 131 | 132 | class GraphNodeTestCase(TransactionalTestCase): 133 | def test_revert_fails_silently(self): 134 | node = GraphNode('test', '0001_init', [], [], []) 135 | 136 | node.revert() 137 | 138 | node.apply() 139 | node.revert() 140 | -------------------------------------------------------------------------------- /tests/utils.py: -------------------------------------------------------------------------------- 1 | import os 2 | import shutil 3 | from importlib import import_module 4 | from typing import Optional, List 5 | from unittest import TestCase, mock 6 | from django.test import TransactionTestCase as DjangoTestCase 7 | from django.db import transaction 8 | from data_migration.services.graph import GraphNode 9 | import django 10 | 11 | 12 | this_dir = os.path.dirname(__file__) 13 | DB_NAME = os.path.join(this_dir, 'db.sqlite3') 14 | 15 | 16 | class FileTestCase(TestCase): 17 | internal_target = '' 18 | 19 | def setUp(self) -> None: 20 | self.target = self.internal_target 21 | 22 | def clean_directory(self): 23 | shutil.rmtree(os.path.join(self.target)) 24 | 25 | def has_file(self, name: str) -> bool: 26 | try: 27 | for f in os.listdir(self.target): 28 | if f == name: 29 | return True 30 | except FileNotFoundError: 31 | pass 32 | return False 33 | 34 | def get_file(self, name: str) -> Optional[str]: 35 | for f in os.listdir(self.target): 36 | if f == name: 37 | with open(os.path.join(self.target, f), 'r') as file: 38 | content = file.read() 39 | return content 40 | return None 41 | 42 | def get_data_migration_node(self, module_path): 43 | try: 44 | node = import_module(module_path).Node 45 | return GraphNode.from_struct(module_path.split('.')[-2], node) 46 | except AttributeError: 47 | return None 48 | 49 | 50 | class TransactionalTestCase(DjangoTestCase): 51 | def run_commit_hooks(self): 52 | """ 53 | Fake transaction commit to run delayed on_commit functions 54 | """ 55 | atomic_module = 'django.db.backends.base.base.BaseDatabaseWrapper' \ 56 | '.validate_no_atomic_block' 57 | for db_name in reversed(self._databases_names()): 58 | with mock.patch(atomic_module, lambda a: False): 59 | transaction.get_connection( 60 | using=db_name 61 | ).run_and_clear_commit_hooks() 62 | 63 | 64 | is_django_setup = False 65 | 66 | 67 | def setup_django(): 68 | global is_django_setup 69 | try: 70 | os.remove(DB_NAME) 71 | except FileNotFoundError: 72 | pass 73 | if is_django_setup: 74 | return 75 | 76 | is_django_setup = True 77 | 78 | from django.conf import settings 79 | 80 | settings.configure( 81 | SECRET_KEY='xxx', 82 | DATABASES={'default': { 83 | 'ENGINE': 'django.db.backends.sqlite3', 84 | 'NAME': DB_NAME, 85 | }}, 86 | INSTALLED_APPS=[ 87 | 'django.contrib.admin', 88 | 'django.contrib.auth', 89 | 'django.contrib.contenttypes', 90 | 'django.contrib.sessions', 91 | 'django.contrib.messages', 92 | 'django.contrib.staticfiles', 93 | 'data_migration', 94 | 'tests.unittests.test_app.apps.TestAppConfig', 95 | 'tests.unittests.test_app_2.apps.TestApp2Config', 96 | ], 97 | DATA_MIGRATION={ 98 | 'SQUASHABLE_APPS' : [ 99 | 'test_app', 100 | 'test_app_2', 101 | ], 102 | }, 103 | TEMPLATES=[ 104 | { 105 | 'BACKEND': 'django.template.backends.django.DjangoTemplates', 106 | 'DIRS': [os.path.join(this_dir, 'templates')], 107 | 'APP_DIRS': True, 108 | 'OPTIONS': { 109 | 'context_processors': [ 110 | 'django.template.context_processors.debug', 111 | 'django.template.context_processors.request', 112 | 'django.contrib.auth.context_processors.auth', 113 | 'django.contrib.messages.context_processors.messages', 114 | ], 115 | }, 116 | }, 117 | ], 118 | DEFAULT_AUTO_FIELD='django.db.models.BigAutoField', 119 | ) 120 | django.setup() 121 | migrate() 122 | 123 | 124 | def migrate(): 125 | from django.core.management import call_command 126 | from django.db import connections 127 | call_command('django_migrate') 128 | 129 | with connections['default'].cursor() as cursor: 130 | cursor.execute("PRAGMA foreign_keys = OFF;") 131 | cursor.fetchone() 132 | 133 | 134 | def teardown_django(): 135 | try: 136 | os.remove(DB_NAME) 137 | except FileNotFoundError: 138 | pass 139 | global is_django_setup 140 | is_django_setup = False 141 | 142 | 143 | class ResetDirectoryMixin: 144 | targets: List[str] = [] 145 | protected_files: List[str] = [] 146 | this_dir: str = '' 147 | 148 | def __enter__(self): 149 | return True 150 | 151 | def __exit__(self, exc_type, exc_val, exc_tb): 152 | for target in self.targets: 153 | dir_path = os.path.join(self.this_dir, target) 154 | try: 155 | files = [ 156 | os.path.join(dir_path, f) 157 | for f in os.listdir(dir_path) 158 | if f not in self.protected_files 159 | and os.path.isfile(os.path.join(dir_path, f)) 160 | ] 161 | except FileNotFoundError: 162 | continue 163 | for file in files: 164 | os.remove(file) 165 | -------------------------------------------------------------------------------- /data_migration/services/file_generator.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import os 3 | from dataclasses import dataclass 4 | from importlib import import_module 5 | from typing import Optional, List 6 | 7 | from django.apps import apps 8 | from django.db import connections 9 | from django.db.migrations.recorder import MigrationRecorder 10 | from django.template import engines 11 | from django.utils import timezone 12 | 13 | from data_migration.helper import get_package_version_string 14 | from data_migration.services.graph import GraphNode 15 | 16 | current_dir = os.path.dirname(__file__) 17 | 18 | log = logging.Logger(__file__) 19 | 20 | 21 | @dataclass 22 | class Routine: 23 | method: str 24 | module: str 25 | module_name: str 26 | file_path: str 27 | 28 | 29 | class DataMigrationGenerator: 30 | migration_template = os.path.join( 31 | current_dir, 32 | 'templates/migration.py.txt' 33 | ) 34 | 35 | def __init__(self, app_name, readable_name: Optional[str] = None, 36 | set_header: bool = True, empty: bool = False, 37 | dry_run: bool = False, routines: List[Routine] = None, 38 | migration_dependencies: List[str] = None) -> None: 39 | self.app_name = app_name 40 | if isinstance(self.app_name, list): 41 | self.app_name = self.app_name[0] 42 | log.warning( 43 | f'Initialize "{self.__class__.__name__}" using list,' 44 | f' will generate file for {self.app_name}.' 45 | ) 46 | 47 | self.clean_app_name() 48 | root_dir = apps.get_app_config(self.app_name).path 49 | self.file_dir = os.path.join(root_dir, 'data_migrations') 50 | self.set_header = set_header 51 | self.empty = empty 52 | self.dry_run = dry_run 53 | self.routines: List[Routine] = routines 54 | if migration_dependencies is None: 55 | migration_dependencies = [] 56 | self.migration_dependencies: List[str] = migration_dependencies 57 | self._gen_file(readable_name) 58 | 59 | def clean_app_name(self): 60 | self.app_name = self.app_name.replace('[\'', '').replace('\']', '') 61 | 62 | def _gen_file(self, readable_name: Optional[str] = None): 63 | empty_dir = True 64 | files = [] 65 | if os.path.isdir(self.file_dir): 66 | files = [ 67 | f for f in os.listdir(self.file_dir) 68 | if (f != '__init__.py' 69 | and os.path.isfile(os.path.join(self.file_dir, f))) 70 | ] 71 | empty_dir = len(files) == 0 72 | elif not self.dry_run: 73 | # create directory 74 | os.mkdir(self.file_dir) 75 | 76 | file = open(os.path.join(self.file_dir, '__init__.py'), 'x') 77 | file.close() 78 | 79 | latest_filename = '' 80 | if empty_dir: 81 | # create first migration file 82 | self.file_name = os.path.join( 83 | self.file_dir, 84 | f'0001_{readable_name or "first"}.py' 85 | ) 86 | else: 87 | latest_filename = sorted(files)[-1] 88 | latest_id = int(latest_filename[:4]) 89 | 90 | # 0001_first.py or xxxx_auto.py 91 | name = readable_name or ("auto" if latest_id > 0 else "first") 92 | file_name = f'{self._get_id(latest_id+1)}_{name}.py' 93 | self.file_name = os.path.join(self.file_dir, file_name) 94 | 95 | if self.dry_run: 96 | return 97 | 98 | file = open(self.file_name, 'x') 99 | file.close() 100 | 101 | if self.empty: 102 | return 103 | 104 | template_kwargs = { 105 | 'file_name': self.file_name.split('/')[-1].replace('.py', ''), 106 | 'set_header': self.set_header, 107 | 'date': timezone.now().strftime('%Y-%m-%d %H:%M'), 108 | 'package': get_package_version_string(), 109 | 'routines': self.routines, 110 | 'migration_dependencies': self.migration_dependencies 111 | } 112 | 113 | if not empty_dir and latest_filename: 114 | template_kwargs.update({ 115 | 'dependencies': [latest_filename.replace('.py', '')] 116 | }) 117 | 118 | recorder = MigrationRecorder(connections['default']) 119 | latest_migration = recorder.migration_qs.filter( 120 | app=self.app_name 121 | ).order_by('-applied').first() 122 | if latest_migration: 123 | template_kwargs['migration_dependencies'].append( 124 | f'{self.app_name}.{latest_migration.name}' 125 | ) 126 | 127 | file = open(self.file_name, 'w') 128 | with open(self.migration_template, 'r') as input_file: 129 | file.write(self.render_template( 130 | input_file.read(), **template_kwargs) 131 | ) 132 | file.close() 133 | 134 | @staticmethod 135 | def _get_id(index: int) -> str: 136 | out = str(index) 137 | while len(out) < 4: 138 | out = f'0{out}' 139 | return out 140 | 141 | @staticmethod 142 | def render_template(content, **context): 143 | django_engine = engines['django'] 144 | template = django_engine.from_string(content) 145 | 146 | return template.render(context=context) 147 | 148 | def set_applied(self): 149 | file = self.file_name.split('.')[0].split('/')[-1] 150 | app_conf = apps.get_app_config(self.app_name) 151 | module_name = f'{app_conf.module.__name__}.data_migrations.{file}' 152 | 153 | try: 154 | node = import_module(module_name).Node 155 | except ModuleNotFoundError: 156 | return 157 | node = GraphNode.from_struct(self.app_name, node) 158 | node.set_applied() 159 | -------------------------------------------------------------------------------- /data_migration/services/squasher.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | from typing import List, Dict 4 | 5 | from data_migration.services.file_generator import DataMigrationGenerator,\ 6 | Routine 7 | from django.apps import apps 8 | from django.core.management import call_command 9 | from django.db.migrations import Migration 10 | from django.db.migrations.executor import MigrationExecutor 11 | from django.db import connection 12 | 13 | 14 | MigrationPlan = List[Migration] 15 | MigrationGraph = Dict[str, List[Migration]] 16 | 17 | 18 | class Log: 19 | def __init__(self, out=sys.stdout): 20 | self.stdout = out 21 | self.log = '' 22 | 23 | def write(self, message): 24 | self.log += f'{message}\n' 25 | 26 | def flush(self): 27 | self.stdout.write(self.log) 28 | self.log = '' 29 | 30 | 31 | log = Log() 32 | sys.stdout = log 33 | 34 | 35 | class MigrationFile: 36 | def __init__(self, fn: str, replacement_string: str, pos: int): 37 | self.file_name: str = fn 38 | self.replacement_string: str = replacement_string 39 | self.position: int = pos 40 | 41 | 42 | class MigrationManager: 43 | def __init__(self, app_name: str, starting_point: str, 44 | end_point: str, dry_run: bool = False): 45 | self.app_name: str = app_name 46 | self.start: str = starting_point 47 | self.end: str = end_point 48 | self.requires_action: bool = False 49 | self.migration_files_to_touch: List[MigrationFile] = [] 50 | self.file_name = None 51 | self.new_file_name = None 52 | self.temp_file_name = None 53 | self.temp_directory = None 54 | self.dry_run = dry_run 55 | 56 | def load(self): 57 | self.file_name = self.gen_file_name() 58 | self.new_file_name = self.gen_new_file_name() 59 | self.temp_file_name = self.gen_temp_file_name() 60 | self.temp_directory = '/'.join(self.file_name.split('/')[:-1] + ['temp']) 61 | 62 | def __repr__(self): 63 | return self.file_name 64 | 65 | @property 66 | def app_path(self) -> str: 67 | return apps.get_app_config(self.app_name).path 68 | 69 | def gen_file_name(self) -> str: 70 | return f'{self.app_path}/migrations/{self.start}' \ 71 | f'_squashed_{self.end}.py' 72 | 73 | def gen_temp_file_name(self) -> str: 74 | return f'{self.app_path}/migrations/temp/{self.start}' \ 75 | f'_squashed_{self.end}.py' 76 | 77 | def gen_new_file_name(self) -> str: 78 | return f'{self.app_path}/migrations/' \ 79 | f'{self.start.split("_")[0]}' \ 80 | f'_squashed_{self.end.split("_")[0]}.py' 81 | 82 | def _parse_generated_file(self): 83 | parse_start = False 84 | with open(self.temp_file_name, 'r') as file: 85 | char_count = 0 86 | line = "–" 87 | replacement_string = '# RunPython operations to refer ' \ 88 | 'to the local versions:' 89 | while line != "": 90 | line = file.readline() 91 | if replacement_string in line: 92 | parse_start = True 93 | self.requires_action = True 94 | elif parse_start and '#' in line: 95 | partial_path = line.replace('# ', '').replace('\n', '') 96 | migration_path = f'{partial_path.replace(".", "/")}.py' 97 | self.migration_files_to_touch.append( 98 | MigrationFile( 99 | migration_path, 100 | partial_path, 101 | char_count 102 | ) 103 | ) 104 | 105 | char_count += len(line) 106 | 107 | def run(self): 108 | self.load() 109 | 110 | if self.dry_run: 111 | log.write(f'Squashing {self.app_name}: ' 112 | f'"{self.start}"–"{self.end}"') 113 | return 114 | 115 | call_command( 116 | 'django_squashmigrations', 117 | self.app_name, 118 | self.start, 119 | self.end, 120 | '--no-input', 121 | stdout=log 122 | ) 123 | 124 | def post_processing(self): 125 | """ 126 | post processing of squashed migration files, 127 | Drops part of django's squashmigration auto generated 128 | comments/commands that makes migration files unusable. 129 | """ 130 | if self.dry_run: 131 | return 132 | 133 | self.load() 134 | try: 135 | os.mkdir(self.temp_directory) 136 | except FileExistsError: 137 | pass 138 | os.rename(self.file_name, self.temp_file_name) 139 | self._parse_generated_file() 140 | if not self.requires_action: 141 | return 142 | 143 | with open(self.temp_file_name, 'r') as read_file,\ 144 | open(self.new_file_name, 'w') as write_file: 145 | stack_open = False 146 | line = "–" 147 | while line != "": 148 | line = read_file.readline() 149 | if 'RunPython(' in line: 150 | stack_open = True 151 | continue 152 | 153 | if stack_open and ')' in line: 154 | stack_open = False 155 | continue 156 | 157 | found = False 158 | for elem in self.migration_files_to_touch: 159 | if elem.replacement_string in line: 160 | found = True 161 | index = line.find(elem.replacement_string) 162 | next_space = line.find( 163 | ' ', 164 | index + len(elem.replacement_string) 165 | ) 166 | if next_space == -1: 167 | next_space = line.find( 168 | ',', 169 | index + len(elem.replacement_string) 170 | ) 171 | if next_space == -1: 172 | next_space = line.find( 173 | '\n', 174 | index + len(elem.replacement_string) 175 | ) 176 | elem.replacement_string = line[index:next_space] 177 | break 178 | 179 | if stack_open or 'code=' in line or '#' in line or found: 180 | continue 181 | 182 | write_file.write(line) 183 | 184 | os.remove(self.temp_file_name) 185 | 186 | def process_data_migrations(self): 187 | """ 188 | Generated data_migrations based on processed files. 189 | """ 190 | for elem in self.migration_files_to_touch: 191 | module_name = elem.replacement_string.split(".")[-2] 192 | module = '.'.join(elem.replacement_string.split('.')[:-1]) 193 | generator = DataMigrationGenerator( 194 | self.app_name, module_name, 195 | routines=[ 196 | Routine( 197 | method=elem.replacement_string.split('.')[-1], 198 | module=module, 199 | module_name=f'mig_{module_name}', 200 | file_path=elem.file_name, 201 | ) 202 | ], 203 | migration_dependencies=[module.replace('.migrations', '')], 204 | dry_run=self.dry_run, 205 | ) 206 | generator.set_applied() 207 | 208 | 209 | class MigrationSquash: 210 | def __init__(self, _loa: List[str], dry_run: bool = False): 211 | self.executor: MigrationExecutor = MigrationExecutor(connection) 212 | self.plan: MigrationPlan = [] 213 | self.graph: MigrationGraph = {} 214 | self.get_migration_graph() 215 | self.parse_plan() 216 | self.list_of_apps = _loa 217 | self.dry_run = dry_run 218 | 219 | @property 220 | def log(self): 221 | return log 222 | 223 | def get_migration_graph(self): 224 | django_plan = self.executor.migration_plan( 225 | self.executor.loader.graph.leaf_nodes(), 226 | clean_start=True 227 | ) 228 | self.plan = [i[0] for i in django_plan] 229 | 230 | def parse_plan(self): 231 | indices: Dict[str, int] = dict() 232 | last_key: str = '' 233 | for migration in self.plan: 234 | app = migration.app_label 235 | if app not in indices: 236 | indices[app] = 0 237 | self.graph[f'{app}_0'] = [migration] 238 | elif last_key == app: 239 | self.graph[f'{app}_{indices[app]}'].append(migration) 240 | 241 | elif len(list(self.graph.keys())) > 0 \ 242 | and self.graph[list(self.graph.keys())[-1]] != app: 243 | indices[app] += 1 244 | self.graph[f'{app}_{indices[app]}'] = [migration] 245 | else: 246 | self.graph[f'{app}_{indices[app]}'].append(migration) 247 | last_key = app 248 | 249 | def squash(self): 250 | for index, street in enumerate(self.graph.keys()): 251 | app_name: str = self.graph[street][0].app_label 252 | if len(self.graph[street]) > 1 and app_name in self.list_of_apps: 253 | from_id = self.graph[street][0].name 254 | to_id = self.graph[street][-1].name 255 | 256 | mig = MigrationManager(app_name, from_id, to_id, self.dry_run) 257 | mig.run() 258 | mig.post_processing() 259 | mig.process_data_migrations() 260 | -------------------------------------------------------------------------------- /data_migration/services/graph.py: -------------------------------------------------------------------------------- 1 | import os 2 | from importlib import import_module 3 | from types import FunctionType 4 | from typing import Optional, List 5 | 6 | from django.apps import apps 7 | from django.db import connections 8 | from django.db.migrations.exceptions import NodeNotFoundError 9 | from django.db.migrations.executor import MigrationExecutor 10 | from django.db.migrations.recorder import MigrationRecorder 11 | 12 | from data_migration.services.node import Node 13 | 14 | FunList = List[FunctionType] 15 | 16 | 17 | class GraphNode: 18 | """Migration node representation within a graph.""" 19 | 20 | def __init__(self, app_name: str, name: str, 21 | dependencies: List[str], 22 | migration_dependencies: List[str], 23 | routines: FunList) -> None: 24 | self.routines = routines 25 | self.dependencies = dependencies 26 | self.migration_dependencies = migration_dependencies 27 | self.node = self.get_or_prepare_node(app_name, name) 28 | self.next: Optional[GraphNode] = None 29 | self.previous: Optional[GraphNode] = None 30 | 31 | @staticmethod 32 | def get_or_prepare_node(app_name, name) -> Node: 33 | """ 34 | Return existing Node or object which is not created yet. 35 | 36 | :param app_name: target app 37 | :param name: name of migration 38 | :return: node with app_name=app_name, name=name 39 | """ 40 | node_obj = Node(app_name=app_name, name=name) 41 | if node_obj.exists(): 42 | node = node_obj.qs.filter( 43 | app_name=app_name, name=name).first() 44 | node_obj.pk = node.pk 45 | node_obj.created_at = node.created_at 46 | return node_obj 47 | 48 | def set_applied(self): 49 | """Queries django's migration table to determine whether node was applied before or not""" 50 | 51 | if self.node.is_applied: 52 | return 53 | 54 | recorder = MigrationRecorder(connections['default']) 55 | dependency_names = [d.split('.')[-1] for d in self.migration_dependencies] 56 | django_migrations = recorder.migration_qs.filter( 57 | app=self.node.app_name, name__in=dependency_names) 58 | if django_migrations.count() == len(dependency_names): 59 | self.node.apply() 60 | 61 | def apply(self) -> None: 62 | """ 63 | Apply node. 64 | 65 | Calling this method will execute the routines within the node. 66 | If the migration is already applied do nothing. 67 | """ 68 | if self.node.is_applied: 69 | return 70 | 71 | if self.prepare_migration_state(): 72 | connection = connections['default'] 73 | 74 | # Work out which apps have migrations and which do not 75 | executor = MigrationExecutor(connection) 76 | pre_migrate_state = executor._create_project_state( 77 | with_applied_migrations=True) 78 | current_state_apps = pre_migrate_state.apps 79 | with connection.schema_editor(atomic=True) as schema_editor: 80 | for routine in self.routines: 81 | routine( 82 | apps=current_state_apps, 83 | schema_editor=schema_editor 84 | ) 85 | self.node.apply() 86 | 87 | def revert(self) -> None: 88 | """Reverts an applied migration node.""" 89 | if not self.node.is_applied: 90 | return 91 | 92 | backup_node = Node( 93 | name=self.node.name, 94 | app_name=self.node.app_name 95 | ) 96 | self.node.qs.get(pk=self.node.pk).delete() 97 | self.node = backup_node 98 | 99 | def prepare_migration_state(self): 100 | """Migrates the django migration graph until dependencies are applied""" 101 | # Get the database we're operating from 102 | connection = connections['default'] 103 | 104 | # Work out which apps have migrations and which do not 105 | executor = MigrationExecutor(connection) 106 | pre_migrate_state = executor._create_project_state( 107 | with_applied_migrations=True) 108 | migration_graph_in_place = True 109 | 110 | if self.migration_dependencies: 111 | recorder = MigrationRecorder(connections['default']) 112 | applied_migrations = recorder.applied_migrations() 113 | unapplied_dependencies = list() 114 | for dependency in self.migration_dependencies: 115 | plan = dependency.split('.') 116 | plan = tuple([plan[-2], plan[-1]]) 117 | if plan not in applied_migrations: 118 | unapplied_dependencies.append(plan) 119 | 120 | if unapplied_dependencies: 121 | try: 122 | plan = executor.migration_plan(unapplied_dependencies) 123 | executor.migrate( 124 | unapplied_dependencies, 125 | plan=plan, 126 | state=pre_migrate_state.clone() 127 | ) 128 | migration_graph_in_place = True 129 | except NodeNotFoundError as ex: 130 | for dep in unapplied_dependencies: 131 | if not recorder.migration_qs.filter( 132 | app=dep[0], 133 | name__icontains=dep[1] 134 | ).exists(): 135 | raise ex 136 | 137 | if not migration_graph_in_place: 138 | migration_graph_in_place = ( 139 | recorder 140 | .migration_qs 141 | .filter(app=self.node.app_name) 142 | .order_by('-pk').first() in self.migration_dependencies 143 | ) 144 | 145 | return migration_graph_in_place 146 | 147 | def append(self, node) -> None: 148 | """ 149 | Append a node to the graph by attaching it to the last element. 150 | 151 | :param node: child to append 152 | """ 153 | if not self.next: 154 | self.next = node 155 | node.previous = self 156 | else: 157 | self.next.append(node) 158 | 159 | @classmethod 160 | def from_struct(cls, app_name, obj) -> 'GraphNode': 161 | """ 162 | Construct node from structured object. 163 | 164 | :param app_name: target application name 165 | :param obj: structured object to create from 166 | :return: representation as graph node 167 | """ 168 | return cls( 169 | app_name, 170 | obj.name, 171 | obj.dependencies, 172 | obj.migration_dependencies, 173 | obj.routines 174 | ) 175 | 176 | def __repr__(self) -> str: 177 | """Node representation.""" 178 | date_str = self.node.created_at.isoformat() \ 179 | if self.node.created_at else "" 180 | return f'{self.node.name}({date_str})' 181 | 182 | 183 | class Graph: 184 | """Directed graph, imitates django's migration graph.""" 185 | 186 | class MigrationNotFoundError(Exception): 187 | """Raised when non existing migration requested.""" 188 | 189 | def __init__(self, name): 190 | super().__init__(f'Data migration "{name}" not found.') 191 | 192 | class EmptyGraphError(Exception): 193 | """Raised when trying to apply empty graph.""" 194 | 195 | def __init__(self): 196 | super().__init__('Empty graph can\'t be applied.') 197 | 198 | def __init__(self, app_name: str) -> None: 199 | self.app_name = app_name 200 | self.base_node: Optional[GraphNode] = None 201 | 202 | def __repr__(self) -> str: 203 | """Representation of graph.""" 204 | node = self.base_node 205 | out = str(self.base_node) 206 | while node.next: 207 | out += f'->{node.next}' 208 | node = node.next 209 | return out 210 | 211 | def push_back(self, child: GraphNode) -> None: 212 | """ 213 | Append the graph with a new node, or set the first node as base node. 214 | 215 | :param child: node to append 216 | """ 217 | if not self.base_node: 218 | self.base_node = child 219 | else: 220 | self.base_node.append(child) 221 | 222 | def get_node(self, name: str) -> GraphNode: 223 | """ 224 | Getter for a node in the graph based on it's name. 225 | 226 | :raises Graph.MigrationNotFoundError: when base not available 227 | :param name: name of node 228 | :return: node within the graph 229 | """ 230 | if name == 'zero': 231 | return self.base_node 232 | 233 | # no name = latest applied 234 | if not name: 235 | try: 236 | latest_node = Node.Node.objects.filter( 237 | app_name=self.app_name).latest() 238 | name = latest_node.name 239 | except Node.Node.DoesNotExist: 240 | return self.base_node 241 | 242 | # search for node in tree with matching name 243 | node = self.base_node 244 | if node: 245 | if node.node.name == name: 246 | return node 247 | while node.next: 248 | if node.node.name == name: 249 | return node 250 | 251 | node = node.next 252 | 253 | raise Graph.MigrationNotFoundError(name) 254 | 255 | def apply(self, name: Optional[str] = None, 256 | fail_silently: bool = False) -> None: 257 | """ 258 | Apply the migration graph until (and including) given name. 259 | 260 | :raises Graph.EmptyGraphError(): on attempt on empty graph 261 | :param name: target migration name 262 | :param fail_silently: raise exception when applying empty graph 263 | """ 264 | node: GraphNode = self.get_node(name) 265 | if not node: 266 | if not fail_silently: 267 | raise Graph.EmptyGraphError() 268 | return 269 | 270 | # revert or apply migrations depending on current state 271 | if (node.next and node.next.node.is_applied) or name == 'zero': 272 | self.revert_graph(node, name == 'zero') 273 | else: 274 | self.forward_graph(node, not bool(name)) 275 | 276 | def forward_graph(self, node, complete: bool = True) -> None: 277 | """ 278 | Apply graph based on nodes recursive. 279 | 280 | :param node: current node to apply 281 | :param complete: indicator to apply until last node 282 | """ 283 | latest_node = node 284 | while (latest_node.previous 285 | and not latest_node.previous.node.is_applied): 286 | latest_node = latest_node.previous 287 | 288 | while latest_node: 289 | latest_node.apply() 290 | if not complete and latest_node == node: 291 | break 292 | latest_node = latest_node.next 293 | 294 | def revert_graph(self, node, complete: bool = False) -> None: 295 | """ 296 | Reverts graph based on nodes recursive. 297 | 298 | :param node: current node to revert 299 | :param complete: indicator to revert whole graph 300 | """ 301 | # walk down the tree 302 | latest_node = node 303 | while latest_node.next and latest_node.next.node.is_applied: 304 | latest_node = latest_node.next 305 | 306 | # revert until given point 307 | while latest_node: 308 | latest_node.revert() 309 | if not complete and latest_node == node: 310 | break 311 | latest_node = latest_node.previous 312 | 313 | @staticmethod 314 | def from_dir(app_name: str) -> 'Graph': 315 | """ 316 | Generate graph from given app directory. 317 | 318 | :param app_name: name of app to generate graph of 319 | :return: Fully generated graph for requested app 320 | """ 321 | app_conf = apps.get_app_config(app_name) 322 | dir_path = app_conf.path 323 | dir_path = os.path.join(dir_path, 'data_migrations') 324 | obj = Graph(app_name) 325 | files = [f for f in os.listdir(dir_path) 326 | if os.path.isfile(os.path.join(dir_path, f))] 327 | for file in files: 328 | if file == '__init__.py': 329 | continue 330 | file = file.split('.')[0] 331 | module_name = f'{app_conf.module.__name__}.data_migrations.{file}' 332 | try: 333 | node = import_module(module_name).Node 334 | except AttributeError: 335 | continue 336 | obj.push_back(GraphNode.from_struct(app_name, node)) 337 | return obj 338 | --------------------------------------------------------------------------------