├── .github └── PULL_REQUEST_TEMPLATE.md ├── .gitignore ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── conf ├── .DS_Store ├── dms-task.template └── table-mappings.json ├── create_task.py └── dms-tasks.xlsx /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | *Issue #, if available:* 2 | 3 | *Description of changes:* 4 | 5 | 6 | By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. 7 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | output/* 2 | .idea/ 3 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check [existing open](https://github.com/aws-samples/dms-cloudformation-templates-generator/issues), or [recently closed](https://github.com/aws-samples/dms-cloudformation-templates-generator/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/aws-samples/dms-cloudformation-templates-generator/labels/help%20wanted) issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](https://github.com/aws-samples/dms-cloudformation-templates-generator/blob/master/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 7 | the Software, and to permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 10 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 11 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 13 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 14 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Create AWS CloudFormation templates for AWS DMS tasks using Microsoft Excel 2 | 3 | 4 | This is a small command line tool written in Python language that takes an MS Excel workbook having names of tables to be migrated, Amazon Resource Names (ARNs) of DMS Endpoints and DMS Replication Instances to be used as input and generates required DMS tasks’ AWS CloudFormation templates as output. Creation of DMS Endpoints and Replication instances is not addressed by this tool. 5 | 6 | ### Prerequisites 7 | 8 | Python 2.7 and above 9 | 10 | ``` 11 | https://www.python.org/downloads/ 12 | ``` 13 | 14 | ### Installing XLRD Python module 15 | 16 | Tool depends on XLRD Python module, if not already installed proceed to install using PIP as mentioned below. 17 | 18 | ``` 19 | pip install xlrd 20 | ``` 21 | 22 | ## Usage 23 | 24 | ``` 25 | python create_task.py --path [PATH_OF_THE_MS_EXCEL_TEMPLATE] --type [cdc | full-load | full-load-and-cdc] 26 | 27 | ``` 28 | 29 | ## Input arguments 30 | 31 | | Name | Description | Required| 32 | | :---- |:----------- |:--------| 33 | |path|Location of the MS Excel template comprising DMS tasks details whose AWS CloudFormation templates needs to be generated.| True | 34 | |type|This argument accepts three different values
**cdc** – If the DMS tasks to be created are for Change Data Capture mode only
**full-load** – If the DMS tasks to be created are for FULL-LOAD mode only
**full-load-and-cdc** – If the DMS tasks to be created are for both FULL-LOAD followed by CDC | True | 35 | 36 | ## Illustration 37 | 38 | Run the below mentioned command with the provided sample arguments to generate sample AWS CloudFormation templates of two DMS tasks i.e. DMS-CHILD and DMS-PARENT that are mentioned in the attached sample MS Excel workbook (dms-tasks.xlsx). 39 | 40 | ``` 41 | python create_task.py --path dms-tasks.xlsx --type cdc 42 | 43 | ``` 44 | 45 | The above sample command generates two AWS CloudFormation templates i.e. DMS-CHILD.template and DMS-PARENT.template in the **output** subfolder of this tool's root folder. 46 | 47 | ## License 48 | Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 49 | 50 | Permission is hereby granted, free of charge, to any person obtaining a copy of this 51 | software and associated documentation files (the "Software"), to deal in the Software 52 | without restriction, including without limitation the rights to use, copy, modify, 53 | merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 54 | permit persons to whom the Software is furnished to do so. 55 | 56 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 57 | INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 58 | PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 59 | HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 60 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 61 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 62 | -------------------------------------------------------------------------------- /conf/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/dms-cloudformation-templates-generator/00447ef5ce346e1e54e52e039303d2243c599aab/conf/.DS_Store -------------------------------------------------------------------------------- /conf/dms-task.template: -------------------------------------------------------------------------------- 1 | { 2 | "AWSTemplateFormatVersion": "2010-09-09", 3 | "Description": null, 4 | "Parameters": { 5 | "ReplicationServerARN": { 6 | "Default": "REPLICATION_INSTANCE_ARN", 7 | "NoEcho": "false", 8 | "Description": "Enter ARN of Replication Server to be used", 9 | "Type": "String" 10 | }, 11 | "SourceEndpoint": { 12 | "Default": "SOURCE_ENDPOINT_ARN", 13 | "NoEcho": "false", 14 | "Description": "Enter ARN of Source End Point", 15 | "Type": "String" 16 | }, 17 | "TargetEndpoint": { 18 | "Default": "TARGET_ENDPOINT_ARN", 19 | "NoEcho": "false", 20 | "Description": "Enter ARN of Target End Point", 21 | "Type": "String" 22 | } 23 | }, 24 | "Resources": { 25 | "TaskNameFromConfig": { 26 | "Type": "AWS::DMS::ReplicationTask", 27 | "Properties": { 28 | "SourceEndpointArn": {"Ref": "SourceEndpoint"}, 29 | "TargetEndpointArn": {"Ref": "TargetEndpoint"}, 30 | "ReplicationInstanceArn": {"Ref": "ReplicationServerARN"}, 31 | "ReplicationTaskIdentifier": "{}", 32 | "MigrationType": "{}", 33 | "ReplicationTaskSettings": "{}", 34 | "TableMappings": null 35 | } 36 | } 37 | }, 38 | "Outputs": { 39 | "StackName": { 40 | "Value": { 41 | "Ref": "AWS::StackName" 42 | } 43 | }, 44 | "Regionname": { 45 | "Value": { 46 | "Ref": "AWS::Region" 47 | } 48 | }, 49 | "SourceEndPoint": { 50 | "Value": { 51 | "Ref": "SourceEndpoint" 52 | } 53 | }, 54 | "TargetEndPoint": { 55 | "Value": { 56 | "Ref": "TargetEndpoint" 57 | } 58 | } 59 | } 60 | } 61 | -------------------------------------------------------------------------------- /conf/table-mappings.json: -------------------------------------------------------------------------------- 1 | { 2 | "selection":{ 3 | "rule-type": "selection", 4 | "rule-id": null, 5 | "rule-name": null, 6 | "object-locator": { 7 | "schema-name": null, 8 | "table-name": null 9 | }, 10 | "rule-action": "include" 11 | }, 12 | "transformation_default":{ 13 | "rule-type": "transformation", 14 | "rule-id": null, 15 | "rule-name": null, 16 | "rule-target": null, 17 | "object-locator": { 18 | }, 19 | "rule-action": null 20 | }, 21 | "transformation":[ 22 | { 23 | "rule-type": "transformation", 24 | "rule-id": null, 25 | "rule-name": null, 26 | "rule-target": "schema", 27 | "object-locator": { 28 | "schema-name": null 29 | }, 30 | "rule-action": "convert-lowercase" 31 | }, 32 | { 33 | "rule-type": "transformation", 34 | "rule-id": null, 35 | "rule-name": null, 36 | "rule-target": "table", 37 | "object-locator": { 38 | "schema-name": null, 39 | "table-name": "%" 40 | }, 41 | "rule-action": "convert-lowercase" 42 | }, 43 | { 44 | "rule-type": "transformation", 45 | "rule-id": null, 46 | "rule-name": null, 47 | "rule-target": "column", 48 | "object-locator": { 49 | "schema-name": null, 50 | "table-name": "%", 51 | "column-name": "%" 52 | }, 53 | "rule-action": "convert-lowercase" 54 | } 55 | ] 56 | } 57 | -------------------------------------------------------------------------------- /create_task.py: -------------------------------------------------------------------------------- 1 | # 2 | # Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | # 4 | # Permission is hereby granted, free of charge, to any person obtaining a copy of this 5 | # software and associated documentation files (the "Software"), to deal in the Software 6 | # without restriction, including without limitation the rights to use, copy, modify, 7 | # merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 8 | # permit persons to whom the Software is furnished to do so. 9 | # 10 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 11 | # INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 12 | # PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 13 | # HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 14 | # OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 15 | # SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 16 | # 17 | 18 | import os, argparse, json, copy, sys, re 19 | from collections import OrderedDict 20 | 21 | import xlrd 22 | 23 | # Base Path 24 | BASE_DIR = os.path.abspath(os.getcwd()) 25 | 26 | # Loading rule mappings for all tasks 27 | TABLE_MAPPINGS_TEMPLATE_FILE = os.path.join(BASE_DIR, "conf", "table-mappings.json") 28 | JSON_TABLE_MAPPINGS = json.loads(open(TABLE_MAPPINGS_TEMPLATE_FILE).read(), object_pairs_hook=OrderedDict) 29 | 30 | def create_task(name, 31 | schema_name, 32 | description, 33 | task_sheet, 34 | migration_type = "full-load", 35 | source_arn = None, 36 | target_arn = None, 37 | replica_arn = None, 38 | tags = None, 39 | cdc_start_time = None): 40 | i = 1 # counter for rule-id 41 | num_cols, task_tables= task_sheet.ncols, task_sheet.nrows 42 | task_name = re.sub(r"[^\w]",'', name) 43 | op_template = copy.deepcopy(JSON_DEFAULT_TEMPLATE_FILE) 44 | 45 | # Add endpoints and replication instance as perametes 46 | 47 | if source_arn: # Replace Source ARN in Parameters in task dms cloudformation 48 | op_template["Parameters"]["SourceEndpoint"]["Default"] = source_arn 49 | if target_arn: # Replace Target ARN in Parameters in task dms cloudformation 50 | op_template["Parameters"]["TargetEndpoint"]["Default"] = target_arn 51 | if replica_arn: # Replace ReplicaARN in Parameters in task dms cloudformation 52 | op_template["Parameters"]["ReplicationServerARN"]["Default"] = replica_arn 53 | 54 | op_template["Resources"][task_name] = op_template["Resources"].pop("TaskNameFromConfig") 55 | op_template["Resources"][task_name]["Properties"]["ReplicationTaskIdentifier"] = name 56 | op_template["Resources"][task_name]["Properties"]["MigrationType"] = migration_type 57 | op_template["Description"] = description 58 | 59 | rules = {"rules" : []} 60 | selection_json = JSON_TABLE_MAPPINGS["selection"] 61 | transformation_default_json = JSON_TABLE_MAPPINGS["transformation_default"] 62 | for row_idx in range(1, task_tables): 63 | table_name = task_sheet.cell(row_idx, 1).value 64 | exclude_columns_list = task_sheet.cell(row_idx, 2).value.split(',') 65 | exclude_columns = list() 66 | for col in exclude_columns_list: 67 | exclude_columns.append(str(col).strip()) 68 | #exclude_columns = map(str.strip, exclude_columns_list) 69 | table_json_op = copy.deepcopy(selection_json) 70 | table_json_op["rule-id"] = i 71 | table_json_op["rule-name"] = i 72 | table_json_op["object-locator"]["schema-name"] = schema_name 73 | table_json_op["object-locator"]["table-name"] = table_name 74 | rules["rules"].append(table_json_op) 75 | i += 1 76 | 77 | for exclude_column in exclude_columns: # Adding Exclude Colllumns Rule to task if exists 78 | if not exclude_column: 79 | continue 80 | exc_json_op = copy.deepcopy(transformation_default_json) 81 | exc_json_op["rule-type"] = "transformation" 82 | exc_json_op["rule-id"] = i 83 | exc_json_op["rule-name"] = i 84 | exc_json_op["rule-target"] = "column" 85 | exc_json_op["object-locator"]["schema-name"] = schema_name 86 | exc_json_op["object-locator"]["table-name"] = table_name 87 | exc_json_op["object-locator"]["column-name"] = exclude_column 88 | exc_json_op["rule-action"] = "remove-column" 89 | rules["rules"].append(exc_json_op) 90 | i += 1 91 | 92 | for trans_rule in JSON_TABLE_MAPPINGS["transformation"]: # Adding transformation rules to task 93 | trans_rule["rule-id"] = i 94 | trans_rule["rule-name"] = i 95 | trans_rule["object-locator"]["schema-name"] = schema_name 96 | rules["rules"].append(trans_rule) 97 | i += 1 98 | 99 | op_template["Resources"][task_name]["Properties"]["TableMappings"] = json.dumps(rules).replace('"', '\"') 100 | 101 | # Add Common tags if exist 102 | if tags and len(tags) > 0 : op_template["Resources"][task_name]["Properties"]["Tags"] = tags 103 | 104 | # Add CDCStartTime if applicable 105 | if cdc_start_time : op_template["Resources"][task_name]["Properties"]["CdcStartTime"] = cdc_start_time 106 | 107 | if not os.path.exists(os.path.join(BASE_DIR, "output")): # Creating output folder if not exists 108 | os.makedirs(os.path.join(BASE_DIR, "output")) 109 | 110 | with open(os.path.join(BASE_DIR, "output", name+".template"), "w") as writefile: 111 | json.dump(op_template, writefile, indent=4) 112 | print("Created Task Template for %s"%name) 113 | 114 | def form_tag(key,value): 115 | tag = dict() 116 | tag['Key'] = key 117 | tag['Value'] = value 118 | return tag 119 | 120 | def get_cell_str(cell): 121 | if cell.ctype == xlrd.XL_CELL_TEXT: return cell.value 122 | elif cell.value == int(cell.value): return int(cell.value) 123 | else: return str(ckey.value) 124 | 125 | if __name__ == '__main__': 126 | parser = argparse.ArgumentParser(description='DMS Task options') 127 | parser.add_argument('--path', required=True, help='Source Path xls') 128 | parser.add_argument('--type', required=True, choices=['cdc','full-load','full-load-and-cdc'], help='cdc/full-load/full-load-and-cdc') 129 | args = parser.parse_args() 130 | if not args.path: 131 | print("Invalid Path Provided") 132 | exit(1) 133 | 134 | xl_workbook = xlrd.open_workbook(args.path) 135 | 136 | DEFAULT_TEMPLATE_FILE = os.path.join(BASE_DIR, "conf", "dms-task.template") 137 | JSON_DEFAULT_TEMPLATE_FILE = json.loads(open(DEFAULT_TEMPLATE_FILE).read(), object_pairs_hook=OrderedDict) 138 | 139 | # Load Tasks master sheet from input file 140 | if 'DMS-Tasks' not in xl_workbook.sheet_names(): 141 | print("DMS-Tasks Sheet not found") 142 | exit(1) 143 | tasks_sheet = xl_workbook.sheet_by_name('DMS-Tasks') 144 | num_cols, num_rows = tasks_sheet.ncols, tasks_sheet.nrows 145 | tags_sheet = xl_workbook.sheet_by_name('DMS-Tags') 146 | 147 | tags = list() 148 | for r in range(1, tags_sheet.nrows): 149 | tags.append(form_tag(str(get_cell_str(tags_sheet.cell(r,1))), str(get_cell_str(tags_sheet.cell(r,2))))) 150 | 151 | for r in range(1,num_rows): 152 | task_name, task_description = tasks_sheet.cell(r, 1).value, tasks_sheet.cell(r, 2).value 153 | 154 | # Reading ARNs from config if not None will be assigned 155 | source_arn, target_arn, replica_arn = tasks_sheet.cell(r, 3).value, tasks_sheet.cell(r, 4).value, tasks_sheet.cell(r, 5).value 156 | cdc_start_time = tasks_sheet.cell(r,6) 157 | schema_name = tasks_sheet.cell(r,7).value 158 | # Task specific sheet 159 | 160 | task_sheet = xl_workbook.sheet_by_name(task_name) 161 | params = { 162 | "source_arn" : source_arn, 163 | "target_arn" : target_arn, 164 | "replica_arn" : replica_arn, 165 | "migration_type" : args.type 166 | } 167 | if len(tags) > 0: params["tags"] = tags 168 | if args.type != 'full-load' and cdc_start_time.value and cdc_start_time.ctype != xlrd.XL_CELL_TEXT: params["cdc_start_time"] = get_cell_str(cdc_start_time) 169 | 170 | create_task(task_name, 171 | schema_name, 172 | task_description, 173 | task_sheet, 174 | **params) 175 | exit(0) 176 | -------------------------------------------------------------------------------- /dms-tasks.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/dms-cloudformation-templates-generator/00447ef5ce346e1e54e52e039303d2243c599aab/dms-tasks.xlsx --------------------------------------------------------------------------------