├── .gitignore ├── CHANGES.rst ├── LICENSE.txt ├── MANIFEST.in ├── README.rst ├── TODO.rst ├── postgresdbdiff.py ├── setup.cfg └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | env 2 | *.pyc 3 | postgres_db_diff.egg-info/* 4 | dist/* 5 | build/* 6 | -------------------------------------------------------------------------------- /CHANGES.rst: -------------------------------------------------------------------------------- 1 | Changelog 2 | ========= 3 | 4 | 0.9.1 5 | ----- 6 | 7 | * Fixing command line tool 8 | 9 | 10 | 0.9.0 11 | ----- 12 | 13 | Initial release 14 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Petras Zdanavičius 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | # Include the license file 2 | include LICENSE.txt 3 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | Postgres DB diff 2 | ================ 3 | 4 | PyPi 5 | ---- 6 | 7 | https://pypi.python.org/pypi/postgres-db-diff/ 8 | 9 | 10 | About 11 | ----- 12 | 13 | Command line tool to compare two PostgreSQL databases. It is based on parsing 14 | ``psql`` meta commands output. Such as ``\dt`` for tables and ``\dv`` for 15 | views. 16 | 17 | https://www.postgresql.org/docs/current/static/app-psql.html 18 | 19 | 20 | How to install 21 | -------------- 22 | 23 | There are two options: 24 | 25 | 1) Use any python package installing tool. Recommended ``pip``. 26 | 2) Just copy/paste ``postgresdbdiff.py`` into your dir and run it using ``python postgresdbdiff.py`` 27 | 28 | 29 | Usage 30 | ----- 31 | 32 | :: 33 | 34 | usage: postgresdbdiff.py [-h] --db1 DB1 --db2 DB2 [--diff-folder DIFF_FOLDER] 35 | 36 | optional arguments: 37 | -h, --help show this help message and exit 38 | --db1 DB1 First DB name 39 | --db2 DB2 Second DB name 40 | --diff-folder DIFF_FOLDER 41 | Directory to output diffs 42 | 43 | 44 | 45 | Example 46 | ------- 47 | 48 | Create two DBs. One using this SQL: 49 | 50 | .. code-block:: sql 51 | 52 | CREATE TABLE table_a ( 53 | id INTEGER PRIMARY KEY, 54 | test_unique VARCHAR (100) UNIQUE, 55 | test_not_null VARCHAR (100) NOT NULL, 56 | test_checks INTEGER NOT NULL 57 | ); 58 | 59 | CREATE TABLE table_b ( 60 | id INTEGER PRIMARY KEY, 61 | table_a_id integer REFERENCES table_a (id) 62 | ); 63 | 64 | CREATE TABLE table_c ( 65 | id INTEGER PRIMARY KEY 66 | ); 67 | 68 | CREATE VIEW view_a AS SELECT 69 | id, test_unique, 42 AS some_number 70 | FROM table_a; 71 | 72 | Other using this SQL: 73 | 74 | .. code-block:: sql 75 | 76 | CREATE TABLE table_a ( 77 | id INTEGER PRIMARY KEY, 78 | test_unique VARCHAR (100), 79 | test_not_null VARCHAR (100), 80 | test_checks INTEGER NOT NULL CHECK (test_checks > 0) 81 | ); 82 | 83 | CREATE TABLE table_b ( 84 | id INTEGER PRIMARY KEY, 85 | table_a_no integer REFERENCES table_a (id) 86 | ); 87 | 88 | CREATE VIEW view_a AS SELECT 89 | id, test_unique 90 | FROM table_a; 91 | 92 | Then run this command :: 93 | 94 | python postgresdbdiff.py --db1 diff_a --db2 diff_b --diff-folder diffs 95 | 96 | Output should be like this :: 97 | 98 | TABLES: additional in "diff_a" 99 | table_c 100 | 101 | TABLES: not matching 102 | table_a 103 | table_b 104 | 105 | VIEWS: not matching 106 | view_a 107 | 108 | And there should be the folder named ``diffs`` with files looking like this 109 | 110 | .. code-block:: diff 111 | 112 | # diffs/table_a.diff 113 | --- TABLES.diff_a.table_a 114 | +++ TABLES.diff_b.table_a 115 | @@ -1,12 +1,13 @@ 116 | Table "public.table_a" 117 | Column | Type | Collation | Nullable | Default 118 | ---------------+------------------------+-----------+----------+--------- 119 | id | integer | | not null | 120 | test_checks | integer | | not null | 121 | - test_not_null | character varying(100) | | not null | 122 | + test_not_null | character varying(100) | | | 123 | test_unique | character varying(100) | | | 124 | Indexes: 125 | "table_a_pkey" PRIMARY KEY, btree (id) 126 | - "table_a_test_unique_key" UNIQUE CONSTRAINT, btree (test_unique) 127 | +Check constraints: 128 | + "table_a_test_checks_check" CHECK (test_checks > 0) 129 | Referenced by: 130 | - TABLE "table_b" CONSTRAINT "table_b_table_a_id_fkey" FOREIGN KEY (table_a_id) REFERENCES table_a(id) 131 | + TABLE "table_b" CONSTRAINT "table_b_table_a_no_fkey" FOREIGN KEY (table_a_no) REFERENCES table_a(id) 132 | 133 | 134 | # diffs/table_b.diff 135 | --- TABLES.diff_a.table_b 136 | +++ TABLES.diff_b.table_b 137 | @@ -1,9 +1,9 @@ 138 | Table "public.table_b" 139 | Column | Type | Collation | Nullable | Default 140 | ------------+---------+-----------+----------+--------- 141 | id | integer | | not null | 142 | - table_a_id | integer | | | 143 | + table_a_no | integer | | | 144 | Indexes: 145 | "table_b_pkey" PRIMARY KEY, btree (id) 146 | Foreign-key constraints: 147 | - "table_b_table_a_id_fkey" FOREIGN KEY (table_a_id) REFERENCES table_a(id) 148 | + "table_b_table_a_no_fkey" FOREIGN KEY (table_a_no) REFERENCES table_a(id) 149 | 150 | 151 | # diffs/view_a.diff 152 | --- VIEWS.diff_a.view_a 153 | +++ VIEWS.diff_b.view_a 154 | @@ -1,6 +1,5 @@ 155 | View "public.view_a" 156 | Column | Type | Collation | Nullable | Default 157 | -------------+------------------------+-----------+----------+--------- 158 | id | integer | | | 159 | - some_number | integer | | | 160 | test_unique | character varying(100) | | | 161 | -------------------------------------------------------------------------------- /TODO.rst: -------------------------------------------------------------------------------- 1 | Ideas 2 | ===== 3 | 4 | Output formats: json, xml (?), html (?) 5 | 6 | 7 | Objects to compare 8 | ================== 9 | 10 | + Tables 11 | + Views 12 | 13 | - Operators (TODO: check if it is possible to have a custom operator) 14 | - Data types and domains (TODO: check WTF is that?) 15 | - Triggers 16 | - Rewrite rules (TODO: check WTF is that?) 17 | - Sequences 18 | 19 | 20 | Functions 21 | ========= 22 | 23 | \df+ 24 | 25 | 26 | Triggers 27 | ======== 28 | 29 | \dy+ 30 | 31 | CREATE EVENT TRIGGER 32 | 33 | 34 | CMD Options 35 | =========== 36 | 37 | -- Normalize constrain names 38 | 39 | 40 | Aggregates 41 | ========== 42 | 43 | CREATE AGGREGATE foo 44 | 45 | 46 | User-defined Types 47 | ================== 48 | 49 | 50 | User-defined Operators 51 | ====================== 52 | -------------------------------------------------------------------------------- /postgresdbdiff.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # If you are reading this code and thinking: why this file have not been 4 | # split into smaller and easier to read modules? The answer is quite simple: 5 | # I want users to be able just copy/paste this file and run it 6 | import argparse 7 | import difflib 8 | import os.path 9 | import subprocess 10 | import sys 11 | 12 | 13 | def check_database_name(name): 14 | try: 15 | out = db_out(name, "SELECT 42", stderr=None) 16 | except subprocess.CalledProcessError: 17 | raise argparse.ArgumentTypeError( 18 | 'Can not access DB using psql. Probably it does not exists.' 19 | ) 20 | 21 | if '42' not in out: 22 | raise argparse.ArgumentTypeError( 23 | 'Unknown problem executing SQL statements using psql. Aborting.' 24 | ) 25 | 26 | return name 27 | 28 | 29 | def check_diff_directory(name): 30 | path = os.path.join(name) 31 | if not os.path.exists(path): 32 | return name 33 | 34 | if not os.path.isdir(path): 35 | raise argparse.ArgumentTypeError('It is not a directory') 36 | 37 | if os.listdir(path): 38 | raise argparse.ArgumentTypeError('Directory must be empty') 39 | 40 | return name 41 | 42 | 43 | def parser_arguments(): 44 | parser = argparse.ArgumentParser() 45 | 46 | parser.add_argument('--db1', help='First DB name', 47 | type=check_database_name, required=True) 48 | parser.add_argument('--db2', help='Second DB name', 49 | type=check_database_name, required=True) 50 | parser.add_argument('--diff-folder', 51 | help='Directory to output diffs', 52 | type=check_diff_directory, required=False) 53 | parser.add_argument('--rowcount', 54 | help='Compare tables row count', 55 | action='store_true') 56 | 57 | return parser.parse_args() 58 | 59 | 60 | def db_out(db_name, cmd, stderr=subprocess.STDOUT): 61 | return subprocess.check_output( 62 | "psql -d '{}' -c '{}'".format(db_name, cmd), shell=True, stderr=stderr 63 | ).decode('utf-8') 64 | 65 | 66 | def get_table_rowcount(db_name, table_name, stderr=subprocess.STDOUT): 67 | cmd = 'select count(1) from "{}";'.format(table_name) 68 | output = subprocess.check_output( 69 | "psql -d '{}' -c '{}' --quiet --tuples-only".format(db_name, cmd), shell=True, stderr=stderr 70 | ).decode('utf-8') 71 | return int(output.strip()) 72 | 73 | 74 | def get_db_tables(db_name): 75 | tables = set() 76 | for line in db_out(db_name, '\\dt').splitlines(): 77 | elems = line.split() 78 | if line and elems[0] == 'public': 79 | tables.add(elems[2]) 80 | return tables 81 | 82 | 83 | def get_db_views(db_name): 84 | views = set() 85 | for line in db_out(db_name, '\\dv').splitlines(): 86 | elems = line.split() 87 | if line and elems[0] == 'public': 88 | views.add(elems[2]) 89 | return views 90 | 91 | 92 | def get_db_mat_views(db_name): 93 | views = set() 94 | for line in db_out(db_name, '\\dmv').splitlines(): 95 | elems = line.split() 96 | if line and elems[0] == 'public': 97 | views.add(elems[2]) 98 | return views 99 | 100 | 101 | def get_table_definition(db_name, table_name): 102 | lines = db_out(db_name, '\\d "{}"'.format(table_name)).splitlines() 103 | lines = [x for x in lines if x.strip()] 104 | 105 | columns_range = [None, None] 106 | indexes_range = [None, None] 107 | check_constr_range = [None, None] 108 | foreign_constr_range = [None, None] 109 | process_constr_range = [None, None] 110 | 111 | S_START = 1 112 | S_COLUMNS = 2 113 | S_INDEXES = 3 114 | S_CHECK_CONSTR = 4 115 | S_FOREIGN_CONSTR = 5 116 | S_REFERENCES = 6 117 | S_END = 7 118 | 119 | def replace_with_sorted(lines, a, b): 120 | if a is None or b is None: 121 | return lines 122 | return lines[:a] + sorted(lines[a:b]) + lines[b:] 123 | 124 | def get_after_columns_state(x): 125 | if x == 'Indexes:': 126 | return S_INDEXES 127 | elif x == 'Check constraints:': 128 | return S_CHECK_CONSTR 129 | elif x == 'Foreign-key constraints:': 130 | return S_FOREIGN_CONSTR 131 | elif x == 'Referenced by:': 132 | return S_REFERENCES 133 | return S_END 134 | 135 | def update_range(line_range, i): 136 | if line_range[0] is None: 137 | line_range[0] = i 138 | line_range[1] = i + 1 139 | else: 140 | line_range[1] = i + 1 141 | 142 | def process_start(i, x): 143 | if x[0:2] == '--': 144 | return S_COLUMNS 145 | return S_START 146 | 147 | def process_columns(i, x): 148 | if x[0] != ' ': 149 | return get_after_columns_state(x) 150 | update_range(columns_range, i) 151 | return S_COLUMNS 152 | 153 | def process_indexes(i, x): 154 | if x[0] != ' ': 155 | return get_after_columns_state(x) 156 | update_range(indexes_range, i) 157 | return S_INDEXES 158 | 159 | def process_check_constr(i, x): 160 | if x[0] != ' ': 161 | return get_after_columns_state(x) 162 | update_range(check_constr_range, i) 163 | return S_CHECK_CONSTR 164 | 165 | def process_foreign_constr(i, x): 166 | if x[0] != ' ': 167 | return get_after_columns_state(x) 168 | update_range(foreign_constr_range, i) 169 | return S_FOREIGN_CONSTR 170 | 171 | def process_references(i, x): 172 | if x[0] != ' ': 173 | return get_after_columns_state(x) 174 | update_range(process_constr_range, i) 175 | return S_REFERENCES 176 | 177 | def process_end(i, x): 178 | return S_END 179 | 180 | processes = { 181 | S_START: process_start, 182 | S_COLUMNS: process_columns, 183 | S_INDEXES: process_indexes, 184 | S_CHECK_CONSTR: process_check_constr, 185 | S_FOREIGN_CONSTR: process_foreign_constr, 186 | S_REFERENCES: process_references, 187 | S_END: process_end, 188 | } 189 | 190 | state = S_START 191 | for i, x in enumerate(lines): 192 | state = processes[state](i, x) 193 | 194 | lines = replace_with_sorted(lines, *columns_range) 195 | lines = replace_with_sorted(lines, *indexes_range) 196 | lines = replace_with_sorted(lines, *check_constr_range) 197 | lines = replace_with_sorted(lines, *foreign_constr_range) 198 | lines = replace_with_sorted(lines, *process_constr_range) 199 | return '\n'.join(lines) 200 | 201 | 202 | def compare_number_of_items(options, db1_items, db2_items, items_name): 203 | if db1_items != db2_items: 204 | additional_db1 = db1_items - db2_items 205 | additional_db2 = db2_items - db1_items 206 | 207 | if additional_db1: 208 | sys.stdout.write( 209 | '{}: additional in "{}"\n'.format(items_name, options.db1) 210 | ) 211 | for t in additional_db1: 212 | sys.stdout.write('\t{}\n'.format(t)) 213 | sys.stdout.write('\n') 214 | 215 | if additional_db2: 216 | sys.stdout.write( 217 | '{}: additional in "{}"\n'.format(items_name, options.db2) 218 | ) 219 | for t in additional_db2: 220 | sys.stdout.write('\t{}\n'.format(t)) 221 | sys.stdout.write('\n') 222 | 223 | 224 | # TODO: Using same function to compare tables and views. It is not very suited 225 | # for views. But I do not see any clear way to have cleaner interface 226 | def compare_each_table(options, db1_tables, db2_tables, items_name): 227 | not_matching_tables = [] 228 | not_matching_rowcount = [] 229 | 230 | for t in sorted(db1_tables & db2_tables): 231 | t1 = get_table_definition(options.db1, t) 232 | t2 = get_table_definition(options.db2, t) 233 | if t1 != t2: 234 | not_matching_tables.append(t) 235 | 236 | diff = difflib.unified_diff( 237 | [x + '\n' for x in t1.splitlines()], 238 | [x + '\n' for x in t2.splitlines()], 239 | '{}.{}.{}'.format(items_name, options.db1, t), 240 | '{}.{}.{}'.format(items_name, options.db2, t), 241 | n=sys.maxsize 242 | ) 243 | 244 | if options.diff_folder: 245 | if not os.path.exists(options.diff_folder): 246 | os.mkdir(options.diff_folder) 247 | filepath = os.path.join( 248 | options.diff_folder, '{}.diff'.format(t) 249 | ) 250 | with open(filepath, 'w') as f: 251 | for diff_line in diff: 252 | f.write(diff_line) 253 | 254 | elif options.rowcount: 255 | t1_rowcount = get_table_rowcount(options.db1, t) 256 | t2_rowcount = get_table_rowcount(options.db2, t) 257 | if t1_rowcount != t2_rowcount: 258 | not_matching_rowcount.append('{} ({} != {})'.format(t, t1_rowcount, t2_rowcount)) 259 | 260 | if not_matching_tables: 261 | sys.stdout.write('{}: not matching\n'.format(items_name)) 262 | for t in not_matching_tables: 263 | sys.stdout.write('\t{}\n'.format(t)) 264 | sys.stdout.write('\n') 265 | 266 | if not_matching_rowcount: 267 | sys.stdout.write('{}: not matching rowcount\n'.format(items_name)) 268 | for t in not_matching_rowcount: 269 | sys.stdout.write('\t{}\n'.format(t)) 270 | sys.stdout.write('\n') 271 | 272 | 273 | def main(): 274 | options = parser_arguments() 275 | 276 | db1_tables = get_db_tables(options.db1) 277 | db2_tables = get_db_tables(options.db2) 278 | 279 | compare_number_of_items(options, db1_tables, db2_tables, 'TABLES') 280 | compare_each_table(options, db1_tables, db2_tables, 'TABLES') 281 | 282 | db1_views = get_db_views(options.db1) 283 | db2_views = get_db_views(options.db2) 284 | compare_number_of_items(options, db1_views, db2_views, 'VIEWS') 285 | compare_each_table(options, db1_views, db2_views, 'VIEWS') 286 | 287 | db1_views = get_db_mat_views(options.db1) 288 | db2_views = get_db_mat_views(options.db2) 289 | compare_number_of_items(options, db1_views, db2_views, 'MATERIALIZED VIEWS') 290 | compare_each_table(options, db1_views, db2_views, 'MATERIALIZED VIEWS') 291 | 292 | 293 | if __name__ == "__main__": 294 | main() 295 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [bdist_wheel] 2 | universal=1 3 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import codecs 2 | import os.path 3 | from setuptools import setup 4 | 5 | 6 | here = os.path.abspath(os.path.dirname(__file__)) 7 | with codecs.open(os.path.join(here, 'README.rst'), encoding='utf-8') as f: 8 | long_description = f.read() 9 | 10 | 11 | setup( 12 | name='postgres-db-diff', 13 | version='0.9.1', # cause triggers, sequences are missing 14 | description='Command line tool to compare two PostgreSQL databases', 15 | long_description=long_description, 16 | url='https://github.com/petraszd/postgres-db-diff', 17 | author='Petras Zdanavicius', 18 | author_email='petraszd@gmail.com', 19 | 20 | 21 | keywords='postgresql database comparison command line utility', 22 | # packages=find_packages(exclude=['contrib', 'docs', 'tests']), 23 | py_modules=['postgresdbdiff'], 24 | install_requires=[], 25 | extras_require={ 26 | 'dev': [], 27 | 'test': [], 28 | }, 29 | 30 | entry_points={ 31 | 'console_scripts': ['postgres-db-diff=postgresdbdiff:main'], 32 | }, 33 | 34 | classifiers=[ 35 | 'Environment :: Console', 36 | 'License :: OSI Approved :: MIT License', 37 | 'Topic :: Utilities', 38 | 39 | 'Programming Language :: Python :: 2', 40 | 'Programming Language :: Python :: 2.7', 41 | 42 | 'Programming Language :: Python :: 3', 43 | 'Programming Language :: Python :: 3.4', 44 | 'Programming Language :: Python :: 3.5', 45 | 'Programming Language :: Python :: 3.6', 46 | ], 47 | ) 48 | --------------------------------------------------------------------------------