├── .gitignore ├── .vscode └── launch.json ├── LICENSE ├── README.md ├── demo ├── __init__.py └── settings.py ├── requirements.txt ├── setup.py └── shifter ├── __init__.py ├── __main__.py ├── cli.py ├── config.py ├── db.py ├── map.py └── migrate.py /.gitignore: -------------------------------------------------------------------------------- 1 | env/* 2 | **/*.pyc 3 | migrations/* 4 | settingstest.py 5 | build/ 6 | *.egg-info 7 | dist/ -------------------------------------------------------------------------------- /.vscode/launch.json: -------------------------------------------------------------------------------- 1 | { 2 | "version": "0.2.0", 3 | "configurations": [ 4 | { 5 | "name": "Python", 6 | "type": "python", 7 | "request": "launch", 8 | "stopOnEntry": false, 9 | "pythonPath": "${config.python.pythonPath}", 10 | "program": "${workspaceRoot}/migrate.py", 11 | "console": "integratedTerminal", 12 | "debugOptions": [ 13 | "WaitOnAbnormalExit", 14 | "WaitOnNormalExit" 15 | ], 16 | "args": [ 17 | "migrate", 18 | "2", 19 | "" 20 | ], 21 | "env": { 22 | "CASSANDRA_SETTINGS": "settingstest" 23 | } 24 | } 25 | ] 26 | } -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rgamba/shifter/387c5dbc0c9de964afdb84296241744ccd9403a2/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # shifter 2 | 3 | Cassandra migration tool that uses pure CQL to keep the complexity away from migrations and from ORMs. 4 | 5 | shifter is based on the idea of simple pure CQL migration files than have a very specific order and should be included in your version control. 6 | 7 | ## Instalation 8 | 9 | ```bash 10 | $ pip install cql-shifter 11 | ``` 12 | 13 | ## Settings file 14 | 15 | The first thing you need to setup before using shifter is creating or pointing your `CASSANDRA_SETTINGS` file. 16 | 17 | This is done by defining the `CASSANDRA_SETTINGS` env var to the settings module of your existing application. 18 | That file should expose the following variables that shifter is going to look for: 19 | 20 | Variable name | Required | Description 21 | --- | --- | --- 22 | `CASSANDRA_SEEDS` | Yes | A list of IPs of cassandra instances. 23 | `CASSANDRA_KEYSPACE` | Yes | The name of your primary keyspace. 24 | `CASSANDRA_PORT` | No | Cassandra's port. 25 | `CASSANDRA_CQLVERSION` | No | CQL Version. Sometimes it is needed to adjust. 26 | `CASSANDRA_USER` | No | Username in case of authentication needed. 27 | `CASSANDRA_PASSWORD` | No | Password in case of authentication needed. 28 | 29 | You can take a look at `demo/settings.py` to check the defaults. In the case we would like to use that settings file we would have to `$ export CASSANDRA_SETTINGS=demo.settings` and then run any shifter command as usual. 30 | 31 | The settings file can be overriden by using the `--settings` flag in some commands (Check out the `--help` for each command). 32 | 33 | ## Clean start without keyspace schema 34 | 35 | If you need to start designing your database from scratch -this means you don't have either db schema nor migration files-, first you need to do all your schema design right in Cassandra (using cqlsh or so). 36 | 37 | When you have a *stable* version of your database running and you feel it's a good base, it's time to create out **migration genesis**, which is the base migration schema from which we are going to create any further migrations. 38 | 39 | First `cd` into your workspace root (your application root) and create and run 40 | 41 | ```bash 42 | $ mkdir migrations 43 | ``` 44 | 45 | Then initiate the migration by running 46 | 47 | ```bash 48 | $ shifter init 49 | ``` 50 | 51 | this will effectively create the migration genesis which is the `migrations/00000.cql` file. This file contains a CQL dump of your current keyspace structure. 52 | 53 | Run 54 | 55 | ```bash 56 | $ shifter migrate 57 | ``` 58 | 59 | To check everything is running smoothly just run `$ shifter status` and you should be prompted an *Already up to date* message. This means we are all set to start making further migrations. 60 | 61 | ## Clean start with a keyspace schema 62 | 63 | If you already have a nice keyspace and you want to start using shifter to control your database changes then the process is much the same as the [previous process](#clean-start-without-keyspace-schema) 64 | 65 | ## Creating a keyspace from existing migration files 66 | 67 | If you already have migrations folder and migration files inside `migrations/` then we need to create a keyspace to match the exact structure of the migration files. 68 | 69 | First `cd` into your workspace root (the directory that contains the `migrations/` folder). 70 | 71 | Then make sure to update the settings of the `CASSANDRA_SETTINGS` file to match your desired cassandra installation. 72 | 73 | Run your first migration: 74 | 75 | ```bash 76 | $ shifter migrate 77 | ``` 78 | 79 | All done. Start working! 80 | 81 | ## Creating your first database migration 82 | 83 | Once you are all set and you need to perform some change to the database there are 2 ways of doing it. 84 | 85 | Migration files are plain old CQL files. In fact you could run those files directly inside cassandra and they will work. 86 | 87 | The files are composed by a *header* section for comments, a **UP** section that performs actions to alter the schema UP and **DOWN** section that performs the inverse queries from UP. 88 | 89 | ### 1 Create a new migration file 90 | 91 | This is the preferred method as it is more reliable. 92 | 93 | #### 1.1 Create a migration 94 | 95 | ```bash 96 | $ shifter create "brief description of changes" 97 | ``` 98 | 99 | #### 1.2 Check migration file 100 | 101 | The previous command command will output the creation of a new file -say `00005_create_users_table.cql` - inside the `migrations` folder. Go ahead and open that file in your editor. 102 | 103 | #### 1.3 Modify or comment 104 | 105 | Feel free to change add/remove comments in the header section of the file `/* ... */`. The file is pretty much self-explainatory. **IMPORTANT** Each command MUST end with `;`. 106 | 107 | That's it. Now check the status of your migration: 108 | 109 | ```bash 110 | $ shifter status 111 | ``` 112 | 113 | you will see it tells you that your migrations folder is 1 movement ahead of your Cassandra database. This means you need to run a migration. 114 | 115 | To complete the migration run 116 | 117 | ```bash 118 | $ shifter migrate 119 | ``` 120 | 121 | **In case you have a syntax error or any of the queries conflict with the schema, the migration will not modify your keyspace in any way.** 122 | 123 | This is because shifter creates a keyspace replica at runtime and runs all the migrations in that replica before running it on the real keyspace. 124 | 125 | If there was any error in the migration, then the migration will be aborted and the replica will always be deleted. 126 | 127 | ### 2. Auto generate a migration 128 | 129 | If you went ahead and made some changes directly in your database, it means you have effectively outdated the migrations folder! 130 | 131 | In this case we need to update your migration folder history, you can do this by running 132 | 133 | ```bash 134 | $ shifter auto-update --name "changes description"` 135 | ``` 136 | 137 | This will output a file name that was automatically generated based on the changed you made directly on the database. 138 | 139 | That's it! Now check the status 140 | 141 | ```bash 142 | $ shifter status 143 | ``` 144 | 145 | and you should see you are up to date! 146 | 147 | _TIP:_ If you just want to output the changes that you made against the current keyspace, run 148 | 149 | ``` 150 | $ shifter auto-update --print --name sync 151 | ``` 152 | 153 | *Important:* Auto-update doesn't track column renaming or any changes in the status of a partition key or a clustering key as it would effectively destroy data. 154 | 155 | Changes of that nature will need to be tracked manually, that's why it is very importat you check manually every auto-update generated migrations. 156 | -------------------------------------------------------------------------------- /demo/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rgamba/shifter/387c5dbc0c9de964afdb84296241744ccd9403a2/demo/__init__.py -------------------------------------------------------------------------------- /demo/settings.py: -------------------------------------------------------------------------------- 1 | CASSANDRA_SEEDS = ['127.0.0.1'] 2 | CASSANDRA_PORT = '' # Use default 3 | CASSANDRA_KEYSPACE = 'my_keyspace' # CHANGE ME 4 | CASSANDRA_CQLVERSION = '3.4.2' # Sometimes cqlsh needs us to specify server cql version 5 | CASSANDRA_USER = '' # If needed 6 | CASSANDRA_PASSWORD = '' # If needed 7 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | cassandra-driver==3.7.0 2 | click==6.6 3 | futures==3.0.5 4 | invoke==0.13.0 5 | six==1.10.0 6 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup 2 | 3 | setup( 4 | name='shifter', 5 | description='A tool for migration management with Cassandra', 6 | long_description='shifter gets the pain away from managing migrations with Cassandra', 7 | url='http://github.com/rgamba/shift/', 8 | author='Ricardo Gamba', 9 | author_email='rgamba@gmail.com', 10 | license='MIT', 11 | classifiers=[ 12 | 'Development Status :: 3 - Alpha', 13 | 'Intended Audience :: Developers', 14 | 'License :: OSI Approved :: BSD License', 15 | 'Programming Language :: Python', 16 | 'Programming Language :: Python :: 2', 17 | 'Programming Language :: Python :: 2.6', 18 | 'Programming Language :: Python :: 2.7', 19 | ], 20 | version='0.1', 21 | keywords='development database migration cassandra', 22 | include_package_data=True, 23 | packages=['shifter'], 24 | install_requires=[ 25 | 'click>=6.6', 26 | 'cassandra-driver>=3.7.0', 27 | 'futures>=3.0.5', 28 | 'invoke>=0.13.0', 29 | 'six>=1.10.0' 30 | 'cqlsh' 31 | ], 32 | entry_points={ 33 | 'console_scripts': [ 34 | 'shifter = shifter.cli:cli', 35 | ], 36 | }, 37 | ) 38 | -------------------------------------------------------------------------------- /shifter/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rgamba/shifter/387c5dbc0c9de964afdb84296241744ccd9403a2/shifter/__init__.py -------------------------------------------------------------------------------- /shifter/__main__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | if __name__ == '__main__': 4 | from .cli import cli 5 | cli() 6 | -------------------------------------------------------------------------------- /shifter/cli.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | from __future__ import absolute_import, division, print_function, unicode_literals 3 | import click 4 | import warnings 5 | 6 | from .migrate import create_migration_file, create_init_migration, get_migrations_on_file 7 | from .migrate import get_last_migration, get_pending_migrations, apply_migration 8 | from .db import connect, get_current_schema, create_demo_keyspace, keyspace_exists 9 | from .db import record_migration, delete_demo_keyspace, create_migration_table, DEMO_KEYSPACE 10 | from .db import auto_migrate_keyspace, get_snapshot, update_snapshot 11 | from .config import get_config 12 | 13 | warnings.filterwarnings("ignore") 14 | 15 | # Get configuration 16 | config = get_config() 17 | 18 | @click.group() 19 | def cli(): 20 | pass 21 | 22 | 23 | @cli.command('create', short_help='Create a new migration file.') 24 | @click.argument('name', required=True) 25 | @click.option('--title', help='Migration title', default=None) 26 | @click.option('--description', help='Migration description', default=None) 27 | def create(name, title, description): 28 | """ Create a new migration file. """ 29 | file = create_migration_file(name=name, up='/* YOUR CQL GOES HERE */', 30 | down='/* CQL TO REVERSE THE QUERIES ABOVE */', 31 | title=title, description=description) 32 | click.echo('Created migration file ', nl=False) 33 | click.secho(file, bold=True, fg='green') 34 | 35 | 36 | @cli.command('init', short_help='Create the migration genesis based on the current keyspace.') 37 | def init(): 38 | """ Initiate the migration project in the current directory. """ 39 | global config 40 | # Cassandra connection. 41 | connect(config) 42 | create_init_migration(config) 43 | 44 | 45 | @cli.command('status', short_help='Show the current migration status.') 46 | @click.option('--settings', default=None, help='Settings module (not file). Must contain CASSANDRA_SEEDS and CASSANDRA_KEYSPACE defined') 47 | def status(settings): 48 | """ Get the current migration status. """ 49 | global config 50 | if settings is not None: 51 | config = get_config({'CASSANDRA_SETTINGS': settings}) 52 | # Cassandra connection. 53 | connect(config) 54 | # Check migrations on file. 55 | migrations = get_migrations_on_file() 56 | if not migrations: 57 | click.secho('No migrations found on the current directory.') 58 | return 59 | if '00000.cql' not in migrations: 60 | click.secho('There is no genesis migration found.') 61 | return 62 | if not keyspace_exists(config.get('keyspace')): 63 | click.echo('Shift hasn\'t been initialized on this keyspace.\nRun \'shift migrate\' to initiate or user the --help flag.') 64 | return 65 | last = get_last_migration(config) 66 | if last is None: 67 | click.secho('Shift hasn\'t been initialized in this keyspace.') 68 | return 69 | pending, up = get_pending_migrations(last, migrations) 70 | if len(pending) <= 0: 71 | click.echo("Already up to date.\nCurrent head is {}".format(last)) 72 | return 73 | click.echo("Cassandra is {} movements behind the current file head ({}).\nCurrent Cassandra head is {}".format(len(pending), migrations[-1], last)) 74 | 75 | 76 | @cli.command('auto-update', short_help='Auto generate the next migration targeting the current Cassandra structure.') 77 | @click.option('--print', is_flag=True, help='Just print the migrations') 78 | @click.option('--name', required=True, help='Name of the update') 79 | def auto_update(print, name): 80 | global config 81 | # Cassandra connection. 82 | connect(config) 83 | # Check migrations on file. 84 | migrations = get_migrations_on_file() 85 | if not keyspace_exists(config.get('keyspace')): 86 | click.echo('Shift hasn\'t been initialized on this keyspace.\nRun \'shift migrate\' to initiate or user the --help flag.') 87 | return 88 | last = get_last_migration(config) 89 | if last is None: 90 | click.secho('Shift hasn\'t been initialized in this keyspace.') 91 | return 92 | pending, up = get_pending_migrations(last, migrations) 93 | if len(pending) > 0: 94 | click.secho('There are pending migrations to be done, please migrate before auto-updating.') 95 | return 96 | # Create demo keyspace to compare 97 | snap = get_snapshot() 98 | if not snap: 99 | click.secho('Unable to locate the last snapshot.', fg='red') 100 | return 101 | create_demo_keyspace(snap, config.get('keyspace')) 102 | actions_up = auto_migrate_keyspace(DEMO_KEYSPACE, config.get('keyspace')) 103 | if len(actions_up) <= 0: 104 | click.secho('Cassandra is up to date with migrations on file.') 105 | delete_demo_keyspace() 106 | return 107 | actions_down = auto_migrate_keyspace(config.get('keyspace'), DEMO_KEYSPACE) 108 | delete_demo_keyspace() 109 | upquery = ';\n'.join(actions_up) + ";" 110 | downquery = ';\n'.join(actions_down) + ";" 111 | if print: 112 | click.echo('---\n' + upquery + '\n---\n') 113 | return 114 | file = create_migration_file(name, upquery, downquery) 115 | click.echo('Created migration file ', nl=False) 116 | record_migration(file, get_current_schema(config), config) 117 | click.secho(file, bold=True, fg='green') 118 | 119 | 120 | @cli.command('migrate', short_help='Migrate the current database.') 121 | @click.argument('head', required=False) 122 | @click.option('--simulate', is_flag=True, help='Just print the migrations that will be performed') 123 | @click.option('--just-demo', is_flag=True, help='Just perform the migrations in demo DB') 124 | @click.option('--settings', default=None, help='Settings module (not file). Must contain CASSANDRA_SEEDS and CASSANDRA_KEYSPACE defined') 125 | def migrate(head, simulate, just_demo, settings): 126 | """ Migrate now. """ 127 | global config 128 | if settings is not None: 129 | config = get_config({'CASSANDRA_SETTINGS': settings}) 130 | # Input validation. 131 | try: 132 | head = int(head) if head else None 133 | except Exception: 134 | click.secho('Head argument must be an integer.', fg='red') 135 | return 136 | # Cassandra connection. 137 | connect(config) 138 | # Check migrations on file. 139 | migrations = get_migrations_on_file() 140 | # Check if there is the migration genesis is present 141 | # if it's not present we cannot continue. 142 | if '00000.cql' not in migrations: 143 | click.secho('Migration genesis (00000.cql) is missing! Forgot to run init command first?', fg='red') 144 | return 145 | # Check if the keyspace exists and if we have a migrations 146 | # table configured. 147 | if not keyspace_exists(config.get('keyspace')): 148 | # Keyspace does not exist, we need to create it based on the genesis file. 149 | click.echo('Keyspace not found, creating from the genesis file.') 150 | result, err = apply_migration('00000.cql', True, None) 151 | if not result: 152 | click.secho('---\nUnable to continue due to an error genesis migration:\n\n{}\n---\n'.format(err.message), fg='red') 153 | return 154 | # Override head, it needs to go all the way from the bottom... 155 | head = None 156 | 157 | schema = get_current_schema(config) 158 | last = get_last_migration(config) 159 | if last is None: 160 | result, err = create_migration_table(config.get('keyspace')) 161 | if not result: 162 | click.secho('---\nUnable to continue due to an error:\n\n{}\n---\n'.format(err.message), fg='red') 163 | return 164 | update_snapshot(get_current_schema(config)) 165 | 166 | if len(migrations) <= 0: 167 | create_init_migration(config) 168 | pending, up = get_pending_migrations(last, migrations, head) 169 | if len(pending) <= 0: 170 | click.echo("Already up to date.") 171 | return 172 | 173 | if simulate: 174 | for p in pending: 175 | click.echo('{} will be applied {}'.format(p, 'UP' if up else 'DOWN')) 176 | return 177 | 178 | # First in demo 179 | error = False 180 | create_demo_keyspace(schema, config['keyspace']) 181 | for f in pending: 182 | res, err = apply_migration(file=f, up=up, keyspace=DEMO_KEYSPACE) 183 | if not res: 184 | error = True 185 | click.secho('---\nUnable to continue due to an error in {}:\n\n{}\n---\n'.format(f, err), fg='red') 186 | break 187 | delete_demo_keyspace() 188 | if error: 189 | return 190 | if just_demo: 191 | return 192 | # Now in real keyspace 193 | for f in pending: 194 | res, err = apply_migration(file=f, up=up, keyspace=config['keyspace']) 195 | if not res: 196 | error = True 197 | click.secho('---\nUnable to continue due to an error in {}:\n\n{}\n---\n'.format(f, err.message), fg='red') 198 | break 199 | record_migration(name=f, schema=get_current_schema(config), up=up, config=config) 200 | if error: 201 | return 202 | click.echo("Migration completed successfully.") 203 | 204 | 205 | if __name__ == "__main__": 206 | cli() 207 | -------------------------------------------------------------------------------- /shifter/config.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | from __future__ import absolute_import, division, print_function, unicode_literals 3 | import os 4 | import sys 5 | import click 6 | import importlib 7 | 8 | REQUIRED = [ 9 | 'CASSANDRA_SEEDS', 10 | 'CASSANDRA_KEYSPACE' 11 | ] 12 | OPTIONAL = [ 13 | 'CASSANDRA_PORT', 14 | 'CASSANDRA_USER', 15 | 'CASSANDRA_PASSWORD', 16 | 'CASSANDRA_CQLVERSION' 17 | ] 18 | 19 | 20 | def get_config(env_override=None): 21 | """ Get the configuration dict. """ 22 | env = os.environ 23 | if env_override is not None: 24 | for key, value in env_override.iteritems(): 25 | env[key] = value 26 | 27 | settings = None 28 | try: 29 | if 'CASSANDRA_SETTINGS' in os.environ: 30 | settings = importlib.import_module(env['CASSANDRA_SETTINGS']) 31 | except Exception: 32 | click.secho('Unable to load settings module {}!'.format(env.get('CASSANDRA_SETTINGS')), fg='red') 33 | sys.exit() 34 | 35 | # Check configuration 36 | config = {} 37 | 38 | for c in REQUIRED: 39 | if hasattr(settings, c): 40 | config[c.lower().split('_', 1).pop()] = getattr(settings, c) 41 | else: 42 | click.secho('{} is missing is settings file!'.format(c), fg='red') 43 | sys.exit() 44 | for c in OPTIONAL: 45 | if hasattr(settings, c): 46 | config[c.lower().split('_', 1).pop()] = getattr(settings, c) 47 | 48 | return config 49 | -------------------------------------------------------------------------------- /shifter/db.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | from __future__ import absolute_import, division, print_function, unicode_literals 3 | import sys 4 | import time 5 | import hashlib 6 | import copy 7 | 8 | import click 9 | from invoke import run 10 | from cassandra.cluster import Cluster 11 | from cassandra.util import max_uuid_from_time 12 | from cassandra.auth import PlainTextAuthProvider 13 | 14 | from .map import Column, Table, Keyspace 15 | 16 | 17 | DEMO_KEYSPACE = 'cm_tmp' 18 | SHIFT_TABLE = 'shift_migrations' 19 | 20 | session = None 21 | 22 | 23 | def connect(config): 24 | global session 25 | # Connect to cassandra 26 | auth_provider = None 27 | if config.get('user'): 28 | auth_provider = PlainTextAuthProvider(username=config.get('user'), password=config.get('password')) 29 | cluster = Cluster( 30 | contact_points=config.get('seeds'), 31 | port=(int(config.get('port')) if config.get('port') else 9042), 32 | auth_provider=auth_provider 33 | ) 34 | try: 35 | session = cluster.connect() 36 | except: 37 | click.secho("Unable to connect to Cassandra", fg='red') 38 | sys.exit() 39 | return session 40 | 41 | 42 | def get_session(): 43 | global session 44 | if session is None: 45 | connect() 46 | return session 47 | 48 | 49 | def get_snapshot(): 50 | try: 51 | file = open('migrations/.snapshot', 'r') 52 | content = file.read() 53 | file.close() 54 | except Exception: 55 | return None 56 | return content 57 | 58 | 59 | def update_snapshot(schema): 60 | try: 61 | file = open('migrations/.snapshot', 'w+') 62 | file.write(schema) 63 | file.close() 64 | except Exception: 65 | pass 66 | 67 | 68 | def run_cqlsh(config, command, keyspace=None): 69 | q = ['cqlsh', '-e', '"{}"'.format(command)] 70 | if config.get('user'): 71 | q.append('-u') 72 | q.append(config.get('user')) 73 | if config.get('password'): 74 | q.append('-p') 75 | q.append(config.get('password')) 76 | if keyspace: 77 | q.append('-k') 78 | q.append(keyspace) 79 | if config.get('cqlversion'): 80 | q.append('--cqlversion={}'.format(config.get('cqlversion'))) 81 | q.append(config['seeds'][0]) 82 | if config.get('port'): 83 | q.append(config['port']) 84 | return ' '.join(q) 85 | 86 | 87 | def get_current_schema(config): 88 | try: 89 | cqlsh = run_cqlsh(config, command="DESCRIBE " + config['keyspace']) 90 | out = run(cqlsh, hide='stdout') 91 | except Exception as e: 92 | click.secho("Unable to get the current DB schema: {}".format(e), fg='red') 93 | sys.exit() 94 | return out.stdout 95 | 96 | 97 | def create_demo_keyspace(schema, schema_name): 98 | schema = schema.replace("CREATE KEYSPACE {}".format(schema_name), "CREATE KEYSPACE {}".format(DEMO_KEYSPACE), 1) 99 | schema = schema.replace("{}.".format(schema_name), "{}.".format(DEMO_KEYSPACE)) 100 | try: 101 | click.echo("Creating tmp keyspace... ", nl=False) 102 | session.execute("DROP KEYSPACE IF EXISTS {}".format(DEMO_KEYSPACE)) 103 | for q in schema.replace('\n', '').split(';'): 104 | if q.strip() == "": 105 | continue 106 | session.execute(q) 107 | except Exception as e: 108 | click.secho("ERROR {}".format(e), fg='red', bold=True) 109 | sys.exit() 110 | click.secho("OK", fg='green', bold=True) 111 | 112 | 113 | def delete_demo_keyspace(): 114 | try: 115 | click.echo("Deleting tmp keyspace... ", nl=False) 116 | session.execute("DROP KEYSPACE IF EXISTS {}".format(DEMO_KEYSPACE)) 117 | click.secho("OK", fg='green', bold=True) 118 | except Exception: 119 | click.secho("ERROR", fg='red', bold=True) 120 | 121 | 122 | def record_migration(name, schema, config, up=True): 123 | session.set_keyspace(config['keyspace']) 124 | if name.endswith('.cql'): 125 | name = name[:-4] 126 | if not up: 127 | # Delete 128 | rows = session.execute( 129 | """ 130 | SELECT time FROM shift_migrations 131 | WHERE type = 'MIGRATION' 132 | AND migration = %s 133 | ALLOW FILTERING 134 | """, (name,) 135 | ) 136 | if not rows: 137 | click.secho("Unable to select last migration from DB", fg="red") 138 | return False 139 | id = rows[0].time 140 | delete = session.execute("DELETE FROM shift_migrations WHERE type = 'MIGRATION' AND time = %s", (id,)) 141 | return 142 | 143 | m = hashlib.md5() 144 | m.update(schema) 145 | session.execute( 146 | """ 147 | INSERT INTO shift_migrations(type, time, migration, hash) 148 | VALUES (%s, %s, %s, %s) 149 | """, 150 | ('MIGRATION', max_uuid_from_time(time.time()), name, m.hexdigest()) 151 | ) 152 | update_snapshot(schema) 153 | 154 | 155 | def create_migration_table(keyspace): 156 | session.set_keyspace(keyspace) 157 | click.echo("Creating shift_migrations table... ", nl=False) 158 | try: 159 | session.execute( 160 | """ 161 | CREATE TABLE IF NOT EXISTS shift_migrations( 162 | type text, 163 | time timeuuid, 164 | migration text, 165 | hash text, 166 | PRIMARY KEY (type, time) 167 | ) 168 | WITH CLUSTERING ORDER BY(time DESC) 169 | """ 170 | ) 171 | click.secho('OK', fg='green', bold=True) 172 | return (True, None) 173 | except Exception as e: 174 | click.secho('ERROR', fg='red', bold=True) 175 | return (False, e) 176 | 177 | 178 | def keyspace_exists(name): 179 | session.set_keyspace('system_schema') 180 | ks = session.execute('SELECT keyspace_name FROM keyspaces') 181 | if not ks: 182 | return False 183 | for row in ks: 184 | if row.keyspace_name == name: 185 | return True 186 | return False 187 | 188 | 189 | def auto_migrate_keyspace(source_name, target_name): 190 | """ 191 | Compare the 2 keyspaces and return a list of 192 | queries to be performed in order to sync source to target. 193 | """ 194 | source = Keyspace(name=source_name, tables=[]) 195 | stables = get_keyspace_tables(source_name) 196 | for table in stables: 197 | source.tables.append(Table( 198 | name=table, 199 | columns=get_table_columns(source_name, table) 200 | )) 201 | target = Keyspace(name=target_name, tables=[]) 202 | ttables = get_keyspace_tables(target_name) 203 | table = [] 204 | for table in ttables: 205 | target.tables.append(Table( 206 | name=table, 207 | columns=get_table_columns(target_name, table) 208 | )) 209 | return get_keyspace_diff(source, target) 210 | 211 | 212 | def get_keyspace_tables(keyspace): 213 | session.set_keyspace('system_schema') 214 | ks = session.execute('SELECT table_name FROM tables WHERE keyspace_name = %s', [keyspace]) 215 | if not ks: 216 | return [] 217 | tables = [] 218 | for row in ks: 219 | tables.append(row.table_name) 220 | return tables 221 | 222 | 223 | def get_table_columns(keyspace, table): 224 | """ Return a list of tuples (col_name, col_type). """ 225 | session.set_keyspace('system_schema') 226 | ks = session.execute('SELECT * FROM columns WHERE keyspace_name = %s AND table_name = %s', [keyspace, table]) 227 | if not ks: 228 | return [] 229 | cols = [] 230 | for row in ks: 231 | cols.append(Column( 232 | name=row.column_name, 233 | type=row.type, 234 | kind=row.kind, 235 | order=row.clustering_order, 236 | position=row.position)) 237 | return cols 238 | 239 | 240 | def get_columns_diff(source, target): 241 | if not isinstance(source, Column) or not isinstance(target, Column): 242 | raise ValueError("parameters must be Column instances") 243 | if source.kind != target.kind: 244 | return "kind" 245 | if source.position != target.position: 246 | return "position" 247 | if source.type != target.type: 248 | return "type" 249 | return None 250 | 251 | def get_tables_diff(source, target): 252 | if not isinstance(source, Table) or not isinstance(target, Table): 253 | raise ValueError("parameters must be Table instances") 254 | if source == target: 255 | return None 256 | actions = [] 257 | # Compare table name 258 | if source.name != target.name: 259 | raise ValueError("tables does not share the same name") 260 | # Columns in source table 261 | for col in source.columns: 262 | target_col = target.get_column(col.name) 263 | if not target_col: 264 | actions.append(source.drop_column_cql(col)) 265 | else: 266 | if target_col != col: 267 | dif = get_columns_diff(col, target_col) 268 | if dif == "type": 269 | actions.append(source.alter_column_type_cql(col.name, target_col.type)) 270 | # Columns in destination table 271 | for col in target.columns: 272 | source_col = source.get_column(col.name) 273 | if not source_col: 274 | actions.append(source.add_column_cql(col)) 275 | return actions 276 | 277 | 278 | def get_keyspace_diff(source, target): 279 | if not isinstance(source, Keyspace) or not isinstance(target, Keyspace): 280 | raise ValueError("parameters must be Table instances") 281 | actions = [] 282 | # Source tables 283 | for table in source.tables: 284 | target_table = target.get_table(table.name) 285 | if not target_table: 286 | actions.append(table.drop_cql()) 287 | else: 288 | a = get_tables_diff(table, target_table) 289 | if a: 290 | actions += a 291 | # Target tables 292 | for table in target.tables: 293 | source_table = source.get_table(table.name) 294 | if not source_table: 295 | actions.append(table.dump_cql()) 296 | return actions 297 | 298 | -------------------------------------------------------------------------------- /shifter/map.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | from __future__ import absolute_import, division, print_function, unicode_literals 3 | 4 | class Column(object): 5 | def __init__(self, name, type, kind='regular', order=None, position=-1): 6 | self.name = name 7 | self.type = type 8 | self.kind = kind 9 | self.order = order 10 | self.position = position 11 | 12 | def is_pk(self): 13 | return self.kind == 'partition_key' 14 | 15 | def is_clustering(self): 16 | return self.kind == 'clustering' 17 | 18 | def dump_cql(self): 19 | return '{} {}'.format(self.name, self.type.upper()) 20 | 21 | def __eq__(self, other): 22 | if not isinstance(other, Column): 23 | return False 24 | return (self.name == other.name and 25 | self.type.lower() == other.type.lower() and 26 | self.kind == other.kind and 27 | self.order == other.order and 28 | self.position == other.position) 29 | 30 | def __ne__(self, other): 31 | return not self.__eq__(other) 32 | 33 | 34 | class Table(object): 35 | def __init__(self, name, columns=[]): 36 | self.name = name 37 | self.columns = columns 38 | 39 | def primary_keys(self): 40 | pk = {} 41 | for c in self.columns: 42 | if c.is_pk(): 43 | pk[c.position] = c.name 44 | r = [] 45 | for i in sorted(pk): 46 | r.append(pk[i]) 47 | return r 48 | 49 | def clustering_columns(self): 50 | cl = {} 51 | for c in self.columns: 52 | if c.is_clustering(): 53 | cl[c.position] = c.name 54 | r = [] 55 | for i in sorted(cl): 56 | r.append(cl[i]) 57 | return r 58 | 59 | def get_column(self, name): 60 | for c in self.columns: 61 | if c.name == name: 62 | return c 63 | return None 64 | 65 | def dump_cql(self): 66 | cql = ['CREATE TABLE {} ('.format(self.name)] 67 | cql.append('\t' + ',\n\t'.join([x.dump_cql() for x in self.columns]) + ",") 68 | pk = self.primary_keys() 69 | pks = ('({})' if len(pk) > 1 else '{}').format(', '.join(pk)) 70 | cc = self.clustering_columns() 71 | clustering = ', ' + ', '.join(cc) if cc else '' 72 | cql.append('\tPRIMARY KEY ({}{})'.format(pks, clustering)) 73 | cql.append(')') 74 | return '\n'.join(cql) 75 | 76 | def __eq__(self, other): 77 | if not isinstance(other, Table): 78 | return False 79 | if len(self.columns) != len(other.columns): 80 | return False 81 | for col in self.columns: 82 | other_col = other.get_column(col.name) 83 | if not other_col: 84 | return False 85 | if col != other_col: 86 | return False 87 | return True 88 | 89 | def __ne__(self, other): 90 | return not self.__eq__(other) 91 | 92 | def drop_cql(self): 93 | return 'DROP TABLE "{}"'.format(self.name) 94 | 95 | def drop_column_cql(self, column): 96 | return 'ALTER TABLE "{}" DROP "{}"'.format(self.name, column.name) 97 | 98 | def add_column_cql(self, column): 99 | return 'ALTER TABLE "{}" ADD "{}" {}'.format(self.name, column.name, column.type) 100 | 101 | def alter_column_type_cql(self, name, type): 102 | return 'ALTER TABLE "{}" ALTER "{}" TYPE {}'.format(self.name, name, type) 103 | 104 | 105 | class Keyspace(object): 106 | def __init__(self, name, tables=[]): 107 | self.name = name 108 | self.tables = tables 109 | 110 | def get_table(self, name): 111 | for t in self.tables: 112 | if t.name == name: 113 | return t 114 | return None -------------------------------------------------------------------------------- /shifter/migrate.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | from __future__ import absolute_import, division, print_function, unicode_literals 3 | import os 4 | import sys 5 | import click 6 | import time 7 | import warnings 8 | 9 | from .db import get_current_schema, get_session 10 | from .db import update_snapshot 11 | 12 | warnings.filterwarnings("ignore") 13 | 14 | 15 | def get_last_migration(config): 16 | """ 17 | Get the last migration stored on cassandra. 18 | If there is no first migration, it will return 0 19 | If there is no table shift_migrations, it will return None 20 | """ 21 | get_session().set_keyspace(config['keyspace']) 22 | try: 23 | migrations = get_session().execute("SELECT migration FROM shift_migrations LIMIT 1") 24 | if not migrations: 25 | return 0 26 | last_migration = migrations[0].migration 27 | if last_migration.endswith('.cql'): 28 | last_migration = last_migration[:-4] 29 | except Exception as e: 30 | last_migration = None 31 | return last_migration 32 | 33 | 34 | def get_migrations_on_file(): 35 | """ Get the stored migrations on file. """ 36 | try: 37 | files = [] 38 | for f in os.listdir('migrations'): 39 | if f[-3:] == 'cql': 40 | files.append(f) 41 | except Exception: 42 | click.secho('Unable to open the migrations directory!', fg='red') 43 | sys.exit() 44 | files.sort() 45 | return files 46 | 47 | 48 | def get_head_migration_on_file(migrations): 49 | """ Get the highest migration on file. """ 50 | mig = [] 51 | for m in migrations: 52 | try: 53 | m = m.split('_')[0] 54 | m = int(m) 55 | except Exception: 56 | continue 57 | mig.append(m) 58 | if not mig: 59 | return 0 60 | return mig.pop() 61 | 62 | 63 | def get_pending_migrations(last_migration, migrations, head=None): 64 | """ 65 | Return a list of migration files that should be applied. 66 | 67 | If head is not None, it must be a positive integer pointing at the tarjet migration we 68 | are headed. If the current DB migration is ahead of the head, then the delta migrations 69 | will be applied DOWN. Otherwise the delta migrations will be applied UP. 70 | 71 | In case DOWN migrations are not found in the file, the migration wont be able to continue. 72 | Pending migrations will be returned IN ORDER in which they must be executed. 73 | """ 74 | if last_migration and '{}.cql'.format(last_migration) not in migrations: 75 | click.secho('Unable to migrate because migrations DB is ahead of migrations on file.', fg='red') 76 | sys.exit() 77 | if not last_migration: 78 | last_migration = 0 79 | else: 80 | last_migration = last_migration.split('_')[0] 81 | 82 | pointer = int(last_migration) 83 | files_head = get_head_migration_on_file(migrations) 84 | # If no head is specified, then the target head will be the 85 | # largest migration on file. 86 | target = files_head if head is None else int(head) 87 | if target > files_head: 88 | click.secho('The target migration provided does not exist in the migrations dir.', fg='red') 89 | sys.exit() 90 | # Are we migrating up or down? 91 | up = True if pointer <= target else False 92 | 93 | if not up: 94 | migrations.sort(reverse=True) 95 | 96 | pending = [] 97 | for m in migrations: 98 | try: 99 | f = m.split('_')[0] 100 | if f == '00000': 101 | continue 102 | cur_mig = int(f) 103 | except Exception: 104 | continue 105 | 106 | if up: 107 | if cur_mig <= pointer or cur_mig > target: 108 | continue 109 | else: 110 | if cur_mig <= target or cur_mig > pointer: 111 | continue 112 | 113 | pending.append(m) 114 | return (pending, up) 115 | 116 | 117 | def apply_migration(file, up, keyspace): 118 | """ 119 | Apply the migration given the raw file name. 120 | If up is True, then it will execute the up statement, else it will execute the down statement. 121 | Valid CQL format in files is as follows: 122 | 123 | --UP-- 124 | /* Your CQL statements here, separated by ; */ 125 | --DOWN-- 126 | /* Your CQL statements here. They MUST revert what the UP statements do */ 127 | 128 | """ 129 | fname = file 130 | click.echo("Applying migration {} {} ".format(file, ('UP' if up else 'DOWN')), nl=False) 131 | try: 132 | file = open('migrations/{}'.format(file), 'r') 133 | content = file.read() 134 | file.close() 135 | except Exception: 136 | click.secho('ERROR', fg='red', bold=True) 137 | return (False, 'Unable to open file {}.'.format(file)) 138 | content = content.replace('--UP--', '', 1) 139 | parts = content.split('--DOWN--') 140 | if len(parts) > 1: 141 | qryup, qrydown = parts 142 | else: 143 | qryup = parts[0] 144 | qrydown = None 145 | 146 | if not qrydown: 147 | click.secho('ERROR', fg='red', bold=True) 148 | return (False, 'File {} does not include a --DOWN-- statement.'.format(fname)) 149 | 150 | if keyspace is not None: 151 | get_session().set_keyspace(keyspace) 152 | qry = qryup if up else qrydown 153 | try: 154 | for q in qry.replace('\n', '').split(';'): 155 | q = q.strip() 156 | if q != '': 157 | get_session().execute(q.strip()) 158 | except Exception as e: 159 | click.secho('ERROR', fg='red', bold=True) 160 | return (False, e) 161 | click.secho('OK', fg='green', bold=True) 162 | return (True, None) 163 | 164 | 165 | def create_migration_file(name, up, down=None, title='', description='', 166 | genesis=False): 167 | """ 168 | Create a migration file in the migrations folder 169 | and return its filename. 170 | """ 171 | if not os.path.isdir('migrations'): 172 | os.mkdir('migrations') 173 | migrations = get_migrations_on_file() 174 | i = 1 175 | while True: 176 | count = len(migrations) if not genesis else -1 177 | file_name = [str(count + i).zfill(5)] 178 | if not genesis: 179 | file_name.append(name.strip().lower().replace(' ', '_')) 180 | file_name = '_'.join(file_name) + '.cql' 181 | if os.path.isfile('migrations/' + file_name): 182 | i += 1 183 | continue 184 | file = open('migrations/' + file_name, 'w') 185 | file.write('/*\n') 186 | if title: 187 | file.write(title) 188 | else: 189 | file.write(name) 190 | file.write('\n\n') 191 | if description: 192 | file.write(description + '\n') 193 | file.write('Created: ' + time.strftime("%d-%m-%Y") + '\n') 194 | file.write('*/\n') 195 | file.write('--UP--\n') 196 | file.write(up + '\n\n') 197 | if down: 198 | file.write('--DOWN--\n') 199 | file.write(down) 200 | return file_name 201 | 202 | 203 | def create_init_migration(config): 204 | """ 205 | Create the genesis configuration file and return it's filename. 206 | This function will also write the current keyspace schema (the one 207 | defined in the current settings) as the content of the file. 208 | """ 209 | click.echo("Creating migration genesis... ", nl=False) 210 | if os.path.isfile('migrations/00000.cql'): 211 | click.secho('ERROR (already exists)', fg='red', bold=True) 212 | return False 213 | current = get_current_schema(config) 214 | down = 'DROP KEYSPACE {};'.format(config.get('keyspace')) 215 | new_file = create_migration_file(name='', title='MIGRATION GENESIS', up=current, down=down, genesis=True) 216 | click.secho('OK', fg='green', bold=True) 217 | update_snapshot(current) 218 | return new_file 219 | --------------------------------------------------------------------------------