├── .gitignore ├── CHANGELOG.md ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── images ├── cf_outputs.png ├── cf_parameters.png ├── cf_stack_name.png ├── cf_submit.png └── solution_diagram.png └── templates └── automate-rds-aurora-export.yaml /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | .env -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | - Initial commit 2 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *main* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this 4 | software and associated documentation files (the "Software"), to deal in the Software 5 | without restriction, including without limitation the rights to use, copy, modify, 6 | merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 7 | permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 10 | INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 11 | PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 12 | HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 13 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 14 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Reduce data archiving costs for compliance by automating RDS snapshot exports to Amazon S3 2 | 3 | This repository contains a CloudFormation template which deploys a serverless event-driven solution that integrates AWS Backup with the Amazon RDS export feature to automate export tasks and enables you to query the data using Amazon Athena without provisioning a new RDS instance or Aurora cluster. 4 | 5 | ## Overview 6 | The following diagram illustrates the architecture of the solution. 7 | 8 | ![Solution Diagram](images/solution_diagram.png) 9 | 10 | Let’s go through the steps shown in the diagram above: 11 | 1. You create a [backup plan](https://docs.aws.amazon.com/aws-backup/latest/devguide/about-backup-plans.html) which will put database backups to the vault created by the technical solution. 12 | 2. In this solution, we use AWS Backup as a signal source for an [EventBridge rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rules.html). 13 | 3. The EventBridge rule triggers an AWS Lambda function which starts export task for the database. This solution uses [AWS Key Management Service](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) (AWS KMS) to encrypt the database exports in Amazon S3. 14 | 4. This solution uses [Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) (Amazon S3) to store the database exports. 15 | 16 | This solution also provides an option if you don’t need to query data export using Athena. When deploying the CloudFormation template, you can choose to skip the creation of resources for step 5, 6, and 7. 17 | 18 | 5. The EventBridge rule triggers a Lambda function when the export task is completed. It uses [Amazon Simple Notification Service](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) (Amazon SNS) to send email if export task fails. 19 | 6. The Lambda function uses [AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html) to create a [database](https://docs.aws.amazon.com/glue/latest/dg/define-database.html), [crawler](https://docs.aws.amazon.com/glue/latest/dg/add-crawler.html) and [runs](https://docs.aws.amazon.com/glue/latest/dg/crawler-running.html) it. 20 | 7. After the crawler successfully runs, you can use Amazon Athena to query the data directly in Amazon S3. 21 | 22 | ## Usage 23 | To get started, create the solution resources using a CloudFormation template: 24 | 25 | 1. Download the [`templates/automate-rds-aurora-export.yaml`](templates/automate-rds-aurora-export.yaml) CloudFormation template to create a new stack. 26 | 2. For **Stack name**, enter a name. 27 | 28 | ![stack name](images/cf_stack_name.png) 29 | 30 | 3. For **KMS Key Configuration**, choose if you want a new KMS key to be created as part of this solution. If you already have an existing KMS key that you want to use, choose **No**. 31 | 4. If you choose **No** for KMS key creation, it is mandatory to enter a valid KMS key ID to be used by the solution. You need to configure key users manually after the solution deployed. Leave this field blank if you chose **Yes** for **KMS Key Configuration**. 32 | 5. Under **RDS Export Configuration**, enter a valid email address to receive notification when an S3 export task failed. 33 | 6. You can enter schema, database, or table names if you want only specific objects to be exported in comma-separated list. Otherwise, leave this field blank for all database objects to be exported. You can find more details about this parameter in the [AWS Boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/rds/client/start_export_task.html#RDS.Client.start_export_task). 34 | 7. If you choose **Yes**, the solution will make exports automatically available in Athena. 35 | 8. Click **Next**. 36 | 37 | ![parameters](images/cf_parameters.png) 38 | 39 | 9. Accept all the defaults and choose **Next**. 40 | 10. Acknowledge the creation of [AWS Identity and Access Management](https://aws.amazon.com/iam/) (IAM) [resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/resources.html) and click **Submit**. 41 | 42 | ![submit](images/cf_submit.png) 43 | 44 | The stack creation starts with the status **Create in Progress** and takes approximately 5 minutes to complete. 45 | 46 | 11. On the **Outputs** tab, take note of the following resource names: 47 | * `BackupVaultName` 48 | * `IamRoleForGlueService` 49 | * `IamRoleForLambdaBackupCompleted` 50 | * `IamRoleForLambdaExportCompleted` 51 | * `SnsTopicName` 52 | 53 | ![outputs](images/cf_outputs.png) 54 | 55 | 12. If you decided to use an existing KMS key, you need to give the IAM roles you took note of in step 11 access to your existing KMS key. You can do that by using the [AWS Management Console](http://aws.amazon.com/console) [default view](https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-modifying.html#key-policy-modifying-how-to-console-default-view) or [policy view](https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-modifying.html#key-policy-modifying-how-to-console-policy-view). 56 | 13. Check your email inbox and choose **Confirm subscription** in the email from Amazon SNS. Amazon SNS opens your web browser and displays a subscription confirmation with your subscription ID. 57 | 58 | Now you’re ready to store your all RDS or Aurora database exports on Amazon S3 automatically and make them available on Athena. This solution can work for all RDS or Aurora database backups taken using AWS Backup, which uses the backup vault created by the CloudFormation template. 59 | 60 | Before you use this solution, ensure your RDS instance supports [exporting snapshots to Amazon S3](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RDS_Fea_Regions_DB-eng.Feature.ExportSnapshotToS3.html). There might be cases when tables or rows can be excluded from the export because using incompatible data types. Review the feature limitations for [RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ExportSnapshot.html#USER_ExportSnapshot.Limits) and [Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-export-snapshot.html#aurora-export-snapshot.Limits), test the data consistency between the source database and the exported data from Athena. 61 | 62 | ## Clean up 63 | To avoid incurring future charges, delete the resources you created: 64 | 65 | 1. On the AWS Backup console, [delete the recovery points](https://docs.aws.amazon.com/aws-backup/latest/devguide/gs-cleanup-resources.html#cleanup-backups). 66 | 2. On the Amazon S3 console, [empty the S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/empty-bucket.html) created by the CloudFormation template to store the RDS database exports. 67 | 3. On the AWS CloudFormation console, [delete the stack](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-delete-stack.html) that you created for the solution. -------------------------------------------------------------------------------- /images/cf_outputs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-export-to-s3-automation/b3e6190a4007a71d506dcccc164a5e77ab184a55/images/cf_outputs.png -------------------------------------------------------------------------------- /images/cf_parameters.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-export-to-s3-automation/b3e6190a4007a71d506dcccc164a5e77ab184a55/images/cf_parameters.png -------------------------------------------------------------------------------- /images/cf_stack_name.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-export-to-s3-automation/b3e6190a4007a71d506dcccc164a5e77ab184a55/images/cf_stack_name.png -------------------------------------------------------------------------------- /images/cf_submit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-export-to-s3-automation/b3e6190a4007a71d506dcccc164a5e77ab184a55/images/cf_submit.png -------------------------------------------------------------------------------- /images/solution_diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-export-to-s3-automation/b3e6190a4007a71d506dcccc164a5e77ab184a55/images/solution_diagram.png -------------------------------------------------------------------------------- /templates/automate-rds-aurora-export.yaml: -------------------------------------------------------------------------------- 1 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # SPDX-License-Identifier: MIT-0 3 | 4 | AWSTemplateFormatVersion: 2010-09-09 5 | Description: This template deploys a serverless event-driven solution that integrates AWS Backup with the Amazon RDS export feature to automate export tasks and enables you to query the data using Amazon Athena without provisioning a new RDS instance or Aurora cluster. 6 | Metadata: 7 | AWS::CloudFormation::Interface: 8 | ParameterGroups: 9 | - 10 | Label: 11 | default: KMS Key Configuration 12 | Parameters: 13 | - CreateKmsKey 14 | - KmsKeyId 15 | - 16 | Label: 17 | default: RDS Export Configuration 18 | Parameters: 19 | - NotificationEmail 20 | - ExportOnly 21 | - RunGlueCrawler 22 | 23 | ParameterLabels: 24 | CreateKmsKey: 25 | default: 'Do you want a new KMS Key to be created that will be used to encrypy/decrypt RDS exports?' 26 | KmsKeyId: 27 | default: 'What is the KMS Key ID that you want to be used?' 28 | ExportOnly: 29 | default: 'Do you want to export only specific schemes, databases or tables?' 30 | NotificationEmail: 31 | default: 'Type a valid email address to be used for notifications' 32 | RunGlueCrawler: 33 | default: 'Do you want your database exports to be available in Athena automatically after each export?' 34 | 35 | Parameters: 36 | CreateKmsKey: 37 | AllowedValues: 38 | - 'Yes' 39 | - 'No' 40 | Default: 'Yes' 41 | Type: String 42 | Description: 'If you choose Yes, we will create a new KMS key for you to be used in RDS Exports.' 43 | KmsKeyId: 44 | Type: String 45 | Description: 'If you choose No for KMS key creation, it is mandatory to enter a valid KMS key ID. You need to define key users manually.' 46 | ExportOnly: 47 | Type: String 48 | Description: 'You can put scheme, database or table names in a comma-separated list. Otherwise leave it blank for all data to be exported. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/rds/client/start_export_task.html#RDS.Client.start_export_task' 49 | NotificationEmail: 50 | Type: String 51 | AllowedPattern: '[^@]+@[^@]+\.[^@]+' 52 | ConstraintDescription: 'Please enter a valid email address.' 53 | Description: 'You will receive notification when S3 export task failed.' 54 | RunGlueCrawler: 55 | AllowedValues: 56 | - 'Yes' 57 | - 'No' 58 | Default: 'Yes' 59 | Type: String 60 | Description: 'If you choose Yes, we will create a Glue database, table(s), crawler and run it followed by each export so you can query data using Athena.' 61 | 62 | Conditions: 63 | CreateKmsKeyCondition: !Equals [!Ref CreateKmsKey, 'Yes'] 64 | RunGlueCrawlerCondition: !Equals [!Ref RunGlueCrawler, 'Yes'] 65 | 66 | Outputs: 67 | IamRoleForLambdaBackupCompleted: 68 | Description: IAM Role Name of the Lambda which triggered after AWS Backup completed. 69 | Value: !Ref ProcessBackupCompletionNotificationRole 70 | IamRoleForLambdaExportCompleted: 71 | Description: IAM Role Name of the Lambda which triggered after RDS Export completed. 72 | Value: !Ref ProcessRdsExportCompletedNotificationRole 73 | IamRoleForGlueService: 74 | Description: IAM Role Name used by AWS Glue Service. 75 | Value: !Ref GlueRole 76 | BackupVaultName: 77 | Description: The name of the AWS Backup vault that should be used for automatically exporting the desired databases. 78 | Value: !Ref BackupVault 79 | SnsTopicName: 80 | Description: The name of the SNS Topic which you can use to receive notifications after any RDS export failed. 81 | Value: !GetAtt RdsExportFailedTopic.TopicName 82 | 83 | Resources: 84 | #BACKUP-- 85 | BackupVault: 86 | Type: AWS::Backup::BackupVault 87 | DeletionPolicy: Delete 88 | UpdateReplacePolicy: Delete 89 | Properties: 90 | BackupVaultName: !Join ['-', ['AutoExportDB-vault', !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]]] 91 | EncryptionKeyArn: !If [CreateKmsKeyCondition, !GetAtt KmsKey.Arn, !Sub 'arn:${AWS::Partition}:kms:${AWS::Region}:${AWS::AccountId}:key/${KmsKeyId}'] 92 | 93 | #KMS----- 94 | KmsKey: 95 | Type: AWS::KMS::Key 96 | Condition: CreateKmsKeyCondition 97 | Properties: 98 | Description: !Sub 'Created by Auto-RDS-Export CF Stack: ${AWS::StackId}' 99 | EnableKeyRotation: true 100 | Enabled: true 101 | KeyPolicy: 102 | Id: key-consolepolicy-3 103 | Version: '2012-10-17' 104 | Statement: 105 | - Sid: Enable IAM User Permissions 106 | Effect: Allow 107 | Principal: 108 | AWS: !Sub 'arn:${AWS::Partition}:iam::${AWS::AccountId}:root' 109 | Action: kms:* 110 | Resource: "*" 111 | - Sid: Allow use of the key 112 | Effect: Allow 113 | Principal: 114 | AWS: 115 | - !GetAtt ProcessBackupCompletionNotificationRole.Arn 116 | - !GetAtt RdsExportRole.Arn 117 | - !GetAtt GlueRole.Arn 118 | Action: 119 | - kms:Encrypt 120 | - kms:Decrypt 121 | - kms:ReEncrypt* 122 | - kms:GenerateDataKey* 123 | - kms:DescribeKey 124 | Resource: "*" 125 | - Sid: Allow attachment of persistent resources 126 | Effect: Allow 127 | Principal: 128 | AWS: 129 | - !GetAtt ProcessBackupCompletionNotificationRole.Arn 130 | - !GetAtt RdsExportRole.Arn 131 | Action: 132 | - kms:CreateGrant 133 | - kms:ListGrants 134 | - kms:RevokeGrant 135 | Resource: "*" 136 | Condition: 137 | Bool: 138 | kms:GrantIsForAWSResource: 'true' 139 | 140 | 141 | #S3----- 142 | S3Bucket: 143 | Type: AWS::S3::Bucket 144 | Properties: 145 | AccessControl: Private 146 | BucketEncryption: 147 | ServerSideEncryptionConfiguration: 148 | - ServerSideEncryptionByDefault: 149 | SSEAlgorithm: 'aws:kms' 150 | KMSMasterKeyID: !If [CreateKmsKeyCondition, !GetAtt KmsKey.Arn, !Sub 'arn:${AWS::Partition}:kms:${AWS::Region}:${AWS::AccountId}:key/${KmsKeyId}'] 151 | BucketName: !Join ['-', ['autoexportdb', !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]]] 152 | 153 | #IAM----- 154 | ProcessBackupCompletionNotificationRole: 155 | Type: AWS::IAM::Role 156 | Properties: 157 | AssumeRolePolicyDocument: 158 | Version: 2012-10-17 159 | Statement: 160 | - Effect: Allow 161 | Principal: 162 | Service: 163 | - lambda.amazonaws.com 164 | Action: 165 | - 'sts:AssumeRole' 166 | Description: 'Lambda Function Role Created by Auto RDS Export CloudFormation Template' 167 | Policies: 168 | - PolicyName: !Join ['-', ['AutoExportDB-backup-completed-policy', !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]]] 169 | PolicyDocument: 170 | Version: '2012-10-17' 171 | Statement: 172 | - Effect: Allow 173 | Action: 174 | - logs:CreateLogGroup 175 | Resource: 176 | - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:*" 177 | - Effect: Allow 178 | Action: 179 | - logs:CreateLogStream 180 | - logs:PutLogEvents 181 | Resource: 182 | - !Sub 183 | - "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/${FunctionName}:*" 184 | - FunctionName: !Join ['-', ['AutoExportDB-backup-completed', !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]]] 185 | - Effect: Allow 186 | Action: 187 | - iam:PassRole 188 | Resource: 189 | - !GetAtt RdsExportRole.Arn 190 | - Effect: Allow 191 | Action: 192 | - rds:StartExportTask 193 | - backup:DescribeBackupJob 194 | Resource: "*" #the actions above support all resources 195 | 196 | ProcessRdsExportCompletedNotificationRole: 197 | Type: AWS::IAM::Role 198 | Properties: 199 | AssumeRolePolicyDocument: 200 | Version: 2012-10-17 201 | Statement: 202 | - Effect: Allow 203 | Principal: 204 | Service: 205 | - lambda.amazonaws.com 206 | Action: 207 | - 'sts:AssumeRole' 208 | Description: 'RDS Export Role Created by Cloudformation' 209 | Policies: 210 | - PolicyName: !Join ['-', ['AutoExportDB-export-completed-policy', !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]]] 211 | PolicyDocument: 212 | Version: '2012-10-17' 213 | Statement: 214 | - Effect: Allow 215 | Action: 216 | - logs:CreateLogGroup 217 | Resource: 218 | - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:*" 219 | - Effect: Allow 220 | Action: 221 | - logs:CreateLogStream 222 | - logs:PutLogEvents 223 | Resource: 224 | - !Sub 225 | - "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/${FunctionName}:*" 226 | - FunctionName: !Join ['-', ['AutoExportDB-export-completed', !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]]] 227 | - Effect: Allow 228 | Action: 229 | - iam:PassRole 230 | - glue:CreateDatabase 231 | - glue:StartCrawler 232 | Resource: 233 | - !GetAtt GlueRole.Arn 234 | - !Sub "arn:${AWS::Partition}:glue:${AWS::Region}:${AWS::AccountId}:catalog" 235 | - !Sub "arn:${AWS::Partition}:glue:${AWS::Region}:${AWS::AccountId}:database/*" 236 | - !Sub "arn:${AWS::Partition}:glue:${AWS::Region}:${AWS::AccountId}:crawler/*" 237 | - Effect: Allow 238 | Action: 239 | - rds:DescribeExportTasks #this action supports all resources 240 | - rds:DescribeRecommendationGroups #this action supports all resources 241 | - glue:CreateCrawler #this action supports all resources 242 | - rds:DescribeRecommendations #this action supports all resources 243 | Resource: "*" 244 | - Effect: Allow 245 | Action: 246 | - rds:DescribeDBClusterSnapshots 247 | - rds:DownloadCompleteDBLogFile 248 | - rds:DownloadDBLogFilePortion 249 | - rds:DescribeDBSnapshots 250 | - rds:ListTagsForResource 251 | Resource: 252 | - !Sub "arn:${AWS::Partition}:rds:*:${AWS::AccountId}:cluster-snapshot:*" 253 | - !Sub "arn:${AWS::Partition}:rds:*:${AWS::AccountId}:db:*" 254 | - !Sub "arn:${AWS::Partition}:rds:*:${AWS::AccountId}:snapshot:*" 255 | - !Sub "arn:${AWS::Partition}:rds:*:${AWS::AccountId}:cluster:*" 256 | 257 | RdsExportRole: 258 | Type: AWS::IAM::Role 259 | Properties: 260 | AssumeRolePolicyDocument: 261 | Version: 2012-10-17 262 | Statement: 263 | - Effect: Allow 264 | Principal: 265 | Service: 266 | - export.rds.amazonaws.com 267 | Action: 268 | - 'sts:AssumeRole' 269 | Description: 'Lambda Function Role Created by Auto RDS Export CloudFormation Template' 270 | Policies: 271 | - PolicyName: !Join ['-', ['rds-export-policy', !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]]] 272 | PolicyDocument: 273 | Version: '2012-10-17' 274 | Statement: 275 | - Effect: Allow 276 | Action: 277 | - kms:Decrypt 278 | - kms:Encrypt 279 | - kms:GenerateDataKey 280 | - kms:ReEncryptTo 281 | - kms:GenerateDataKeyWithoutPlaintext 282 | - kms:DescribeKey 283 | - kms:RetireGrant 284 | - kms:CreateGrant 285 | - kms:ReEncryptFrom 286 | Resource: !Sub 'arn:${AWS::Partition}:kms:*:${AWS::AccountId}:key/*' 287 | - Effect: Allow 288 | Action: 289 | - s3:ListBucket 290 | - s3:GetBucketLocation 291 | Resource: 292 | - !Sub 293 | - 'arn:${AWS::Partition}:s3:::${BucketName}' 294 | - BucketName: !Join ['-', ['autoexportdb', !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]]] 295 | - Effect: Allow 296 | Action: 297 | - s3:PutObject* 298 | - s3:GetObject* 299 | - s3:DeleteObject* 300 | Resource: 301 | - !Sub 302 | - 'arn:${AWS::Partition}:s3:::${BucketName}/*' 303 | - BucketName: !Join ['-', ['autoexportdb', !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]]] 304 | 305 | 306 | GlueRole: 307 | Type: AWS::IAM::Role 308 | Properties: 309 | AssumeRolePolicyDocument: 310 | Version: 2012-10-17 311 | Statement: 312 | - Effect: Allow 313 | Principal: 314 | Service: 315 | - glue.amazonaws.com 316 | Action: 317 | - 'sts:AssumeRole' 318 | Description: 'Glue Role Created by Cloudformation' 319 | Policies: 320 | - PolicyName: !Join ['-', ['glue-policy', !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]]] 321 | PolicyDocument: # the policy below used per https://docs.aws.amazon.com/glue/latest/dg/create-service-policy.html 322 | Version: '2012-10-17' 323 | Statement: 324 | - Effect: Allow 325 | Action: 326 | - glue:* 327 | - s3:GetBucketLocation 328 | - s3:ListBucket 329 | - s3:ListAllMyBuckets 330 | - s3:GetBucketAcl 331 | - ec2:DescribeVpcEndpoints 332 | - ec2:DescribeRouteTables 333 | - ec2:CreateNetworkInterface 334 | - ec2:DeleteNetworkInterface 335 | - ec2:DescribeNetworkInterfaces 336 | - ec2:DescribeSecurityGroups 337 | - ec2:DescribeSubnets 338 | - ec2:DescribeVpcAttribute 339 | - iam:ListRolePolicies 340 | - iam:GetRole 341 | - iam:GetRolePolicy 342 | - cloudwatch:PutMetricData 343 | Resource: 344 | - "*" 345 | - Effect: Allow 346 | Action: 347 | - s3:CreateBucket 348 | Resource: 349 | - arn:aws:s3:::aws-glue-* 350 | - Effect: Allow 351 | Action: 352 | - s3:GetObject 353 | - s3:PutObject 354 | - s3:DeleteObject 355 | Resource: 356 | - arn:aws:s3:::aws-glue-*/* 357 | - arn:aws:s3:::*/*aws-glue-*/* 358 | - Effect: Allow 359 | Action: 360 | - s3:GetObject 361 | Resource: 362 | - arn:aws:s3:::crawler-public* 363 | - arn:aws:s3:::aws-glue-* 364 | - Effect: Allow 365 | Action: 366 | - logs:CreateLogGroup 367 | - logs:CreateLogStream 368 | - logs:PutLogEvents 369 | Resource: 370 | - arn:aws:logs:*:*:/aws-glue/* 371 | - Effect: Allow 372 | Action: 373 | - ec2:CreateTags 374 | - ec2:DeleteTags 375 | Condition: 376 | ForAllValues:StringEquals: 377 | aws:TagKeys: 378 | - aws-glue-service-resource 379 | Resource: 380 | - arn:aws:ec2:*:*:network-interface/* 381 | - arn:aws:ec2:*:*:security-group/* 382 | - arn:aws:ec2:*:*:instance/* 383 | - Effect: Allow 384 | Action: 385 | - kms:CreateAlias 386 | - kms:CreateKey 387 | - kms:DeleteAlias 388 | - kms:Describe* 389 | - kms:GenerateRandom 390 | - kms:Get* 391 | - kms:List* 392 | - kms:TagResource 393 | - kms:UntagResource 394 | - iam:ListGroups 395 | - iam:ListRoles 396 | - iam:ListUsers 397 | Resource: "*" 398 | - Effect: Allow 399 | Action: 400 | - s3:* 401 | - s3-object-lambda:* 402 | Resource: "*" 403 | 404 | ##Lambda///// 405 | ProcessRdsExportCompletedNotification: 406 | Type: AWS::Lambda::Function 407 | Condition: RunGlueCrawlerCondition 408 | Properties: 409 | Description: Processes RDS Export completion event, initiates Glue tasks. 410 | FunctionName: !Join ['-', ['AutoExportDB-export-completed', !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]]] 411 | Handler: "index.lambda_handler" 412 | MemorySize: 128 413 | Role: !GetAtt ProcessRdsExportCompletedNotificationRole.Arn 414 | Runtime: python3.9 415 | Timeout: 15 416 | Environment: 417 | Variables: 418 | GLUE_IAM_ROLE_ARN: !GetAtt GlueRole.Arn 419 | S3_BUCKET_NAME: !Join ['-', ['autoexportdb', !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]]] 420 | LOG_LEVEL: 'INFO' 421 | Code: 422 | ZipFile: | 423 | import logging 424 | import json 425 | import boto3 426 | import os 427 | from datetime import datetime 428 | from botocore.exceptions import ClientError 429 | 430 | """ 431 | The Event Bridge rule triggers this Lambda function when the export task is completed. 432 | It works for RDS and Aurora databases. 433 | It creates Glue database, crawler and runs it to make the export available in Athena. 434 | It checks s3 bucket of the export task for validation, otherwise it would create glue database for all database exports. 435 | """ 436 | 437 | #configure logger 438 | logger = logging.getLogger() 439 | logger.setLevel(os.getenv("LOG_LEVEL", logging.INFO)) 440 | 441 | #initiate boto3 442 | session = boto3.Session() 443 | rds_client = session.client('rds') 444 | glue_client = session.client('glue') 445 | 446 | def validate_env_var(var_name): 447 | value = os.environ.get(var_name) 448 | if not value: 449 | logger.error(f"ERROR: Environment variable '{var_name}' not set.") 450 | sys.exit(1) 451 | return value 452 | 453 | def lambda_handler(event, context): 454 | logger.info(event) 455 | source_arn = event['detail']['SourceArn'] 456 | source_identifier = event['detail']['SourceIdentifier'] 457 | s3_bucket = validate_env_var('S3_BUCKET_NAME') 458 | glue_iam_role_arn = validate_env_var('GLUE_IAM_ROLE_ARN') 459 | source_type = event['detail']['SourceType'] 460 | #Validate if we need to continue with glue operations for the export task 461 | response = rds_client.describe_export_tasks( 462 | SourceArn=source_arn, 463 | Filters=[ 464 | {'Name': 'status', 'Values': ['complete']}, 465 | {'Name': 's3-bucket', 'Values': [s3_bucket]} 466 | ], 467 | ) 468 | 469 | export_tasks = response['ExportTasks'] 470 | if len(export_tasks) == 0: 471 | logger.error('No Export Task found!') 472 | sys.exit(1) 473 | 474 | export_task_identifier = export_tasks[0]['ExportTaskIdentifier'] 475 | s3_prefix = export_tasks[0]['S3Prefix'] 476 | backup_job_id = source_identifier.split('awsbackup:job-')[1] 477 | 478 | logger.info('ExportTaskIdentifier: %s', export_task_identifier) 479 | logger.info('S3Prefix: %s', s3_prefix) 480 | logger.info('SourceArn: %s', source_arn) 481 | logger.info('SourceIdentifier: %s', source_identifier) 482 | logger.info('BackupJobId: %s', backup_job_id) 483 | # Check if the source database is RDS 484 | if source_type == 'SNAPSHOT': 485 | response = rds_client.describe_db_snapshots( 486 | DBSnapshotIdentifier=source_identifier, 487 | SnapshotType='awsbackup', 488 | ) 489 | db_instance_identifier = response['DBSnapshots'][0]['DBInstanceIdentifier'] 490 | db_engine = response['DBSnapshots'][0]['Engine'] 491 | db_snapshot_create_time = str(response['DBSnapshots'][0]['SnapshotCreateTime']) 492 | # Otherwise the source database can be assumed as Aurora 493 | else: 494 | response = rds_client.describe_db_cluster_snapshots( 495 | DBClusterSnapshotIdentifier=source_identifier, 496 | SnapshotType='awsbackup', 497 | ) 498 | db_instance_identifier = response['DBClusterSnapshots'][0]['DBClusterIdentifier'] 499 | db_engine = response['DBClusterSnapshots'][0]['Engine'] 500 | db_snapshot_create_time = str(response['DBClusterSnapshots'][0]['SnapshotCreateTime']) 501 | 502 | d_ss = datetime.strptime(db_snapshot_create_time, '%Y-%m-%d %H:%M:%S.%f%z') 503 | 504 | logger.info('DBInstanceIdentifier: %s', db_instance_identifier) 505 | logger.info('Engine: %s', db_engine) 506 | logger.info('SnapshotCreateTime: {}'.format(db_snapshot_create_time)) 507 | 508 | glue_db_suffix = f'{d_ss.year}{d_ss.month:02d}{d_ss.day:02d}{d_ss.hour:02d}{d_ss.minute:02d}' 509 | glue_db_name = f'rds_{db_engine}_{db_instance_identifier}_{glue_db_suffix}' 510 | 511 | response = glue_client.create_database( 512 | DatabaseInput={ 513 | 'Name': glue_db_name, 514 | 'Description': 'Database created by Auto RDS Export', 515 | } 516 | ) 517 | 518 | s3_path_for_crawler = f's3://{s3_bucket}/{s3_prefix}/{export_task_identifier}' 519 | glue_crawler_name = f'{glue_db_name}_crawler' 520 | 521 | response = glue_client.create_crawler( 522 | Name=glue_crawler_name, 523 | Role=glue_iam_role_arn, 524 | DatabaseName=glue_db_name, 525 | Description='Crawler created by Auto RDS Export', 526 | Targets={ 527 | 'S3Targets': [ 528 | { 529 | 'Path': s3_path_for_crawler 530 | }, 531 | ] 532 | }, 533 | ) 534 | 535 | response = glue_client.start_crawler(Name=glue_crawler_name) 536 | 537 | if response['ResponseMetadata']['HTTPStatusCode'] == 200: 538 | logger.info('I created Glue Database and ran Crawler.') 539 | else: 540 | logger.error('Failed to start crawler!') 541 | 542 | ProcessBackupCompletionNotification: 543 | Type: AWS::Lambda::Function 544 | Properties: 545 | Description: Processes backup completion event, initiates RDS export. 546 | FunctionName: !Join ['-', ['AutoExportDB-backup-completed', !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]]] 547 | Handler: "index.lambda_handler" 548 | MemorySize: 128 549 | Role: !GetAtt ProcessBackupCompletionNotificationRole.Arn 550 | Runtime: python3.9 551 | Timeout: 15 552 | Environment: 553 | Variables: 554 | EXPORT_ONLY: !Ref ExportOnly 555 | IAM_ROLE_ARN: !GetAtt RdsExportRole.Arn 556 | KMS_KEY_ID: !If [CreateKmsKeyCondition, !Ref KmsKey, !Ref KmsKeyId ] 557 | S3_BUCKET_NAME: !Join ['-', ['autoexportdb', !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]]] 558 | LOG_LEVEL: 'INFO' 559 | Code: 560 | ZipFile: | 561 | import sys 562 | import logging 563 | import json 564 | import boto3 565 | from botocore.exceptions import ClientError 566 | import os 567 | from datetime import datetime 568 | 569 | """ 570 | The Event Bridge rule triggers this Lambda function when the AWS Backup task is completed for the backup vault created by the solution. 571 | It starts export task for the database received in the event details. 572 | """ 573 | 574 | #configure logger 575 | logger = logging.getLogger() 576 | logger.setLevel(os.getenv("LOG_LEVEL", logging.INFO)) 577 | 578 | #initiate boto3 579 | session = boto3.Session() 580 | rds_client = session.client('rds') 581 | 582 | def validate_env_var(var_name): 583 | value = os.environ.get(var_name) 584 | if not value: 585 | logger.error(f"ERROR: Environment variable '{var_name}' not set.") 586 | sys.exit(1) 587 | return value 588 | 589 | def lambda_handler(event, context): 590 | logger.info('Received Event') 591 | logger.info(event) 592 | creation_date=event['detail']['creationDate'] 593 | export_task_id='AutoExportDB-' + str(event['detail']['backupJobId']) + '-' + datetime.now().strftime("%S%z") 594 | source_arn = event['resources'][0] 595 | s3_bucket = validate_env_var('S3_BUCKET_NAME') 596 | iam_role= validate_env_var('IAM_ROLE_ARN') 597 | kms_key= validate_env_var('KMS_KEY_ID') 598 | s3_prefix= str(creation_date.split('T')[0]).replace('-','/') 599 | export_only= os.environ.get('EXPORT_ONLY') 600 | 601 | if len(export_only) > 0: 602 | export_only = [ export_only ] 603 | else: 604 | export_only = list() 605 | 606 | logger.info('Export Only: %s', str(export_only)) 607 | 608 | try: 609 | response = rds_client.start_export_task( 610 | ExportTaskIdentifier=export_task_id, 611 | SourceArn=source_arn, 612 | S3BucketName=s3_bucket, 613 | IamRoleArn=str(iam_role), 614 | KmsKeyId=str(kms_key), 615 | S3Prefix=''.join(s3_prefix), 616 | ExportOnly=export_only) 617 | except ClientError as e: 618 | logger.error('ERROR: Could not start export task.') 619 | logger.error(e) 620 | sys.exit() 621 | 622 | logger.info('SUCCESS: Export Task started.') 623 | 624 | BackupCompletedEventPermission: 625 | Type: AWS::Lambda::Permission 626 | Properties: 627 | FunctionName: !Ref ProcessBackupCompletionNotification 628 | Action: lambda:InvokeFunction 629 | Principal: events.amazonaws.com 630 | SourceArn: !GetAtt BackupCompletedEvent.Arn 631 | 632 | ExportCompletedEventPermission: 633 | Type: AWS::Lambda::Permission 634 | Condition: RunGlueCrawlerCondition 635 | Properties: 636 | FunctionName: !Ref ProcessRdsExportCompletedNotification 637 | Action: lambda:InvokeFunction 638 | Principal: events.amazonaws.com 639 | SourceArn: !GetAtt ExportCompletedEvent.Arn 640 | 641 | ##SNS///// 642 | RdsExportFailedTopic: 643 | Type: AWS::SNS::Topic 644 | Properties: 645 | DisplayName: "Rds Export Failed" 646 | FifoTopic: false 647 | TopicName: !Join ['-', ['AutoExportDB-export-failed', !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]]] 648 | KmsMasterKeyId: !If [CreateKmsKeyCondition, !Ref KmsKey, !Ref KmsKeyId] 649 | 650 | 651 | RdsExportFailedTopicSubscription: 652 | Type: AWS::SNS::Subscription 653 | Properties: 654 | Endpoint: !Ref NotificationEmail 655 | Protocol: email 656 | TopicArn: !Ref RdsExportFailedTopic 657 | 658 | ##EventBridge///// 659 | BackupCompletedEvent: 660 | Type: AWS::Events::Rule 661 | Properties: 662 | Description: Triggered after any RDS Backup Completed 663 | EventBusName: default 664 | EventPattern: 665 | detail-type: 666 | - Backup Job State Change 667 | source: 668 | - aws.backup 669 | detail: 670 | backupVaultName: 671 | - !Ref BackupVault 672 | state: 673 | - COMPLETED 674 | resourceType: 675 | - RDS 676 | - Aurora 677 | Name: !Join ['-', ['AutoExportDB-backup-completed', !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]]] 678 | State: ENABLED 679 | Targets: 680 | - 681 | Arn: !GetAtt ProcessBackupCompletionNotification.Arn 682 | Id: !Join ['-', ['AutoExportDB-backup-completed', !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]]] 683 | 684 | ExportCompletedEvent: 685 | Type: AWS::Events::Rule 686 | Condition: RunGlueCrawlerCondition 687 | Properties: 688 | Description: Triggered after any RDS Export Completed 689 | EventPattern: 690 | source: 691 | - aws.rds 692 | detail: 693 | EventID: 694 | - RDS-EVENT-0161 695 | - RDS-EVENT-0164 696 | Name: !Join ['-', ['AutoExportDB-export-completed', !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]]] 697 | State: "ENABLED" 698 | Targets: 699 | - 700 | Arn: !GetAtt ProcessRdsExportCompletedNotification.Arn 701 | Id: !Join ['-', ['AutoExportDB-export-completed', !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]]] 702 | 703 | ExportFailedEvent: 704 | Type: AWS::Events::Rule 705 | Properties: 706 | Description: Triggered after any RDS Export Failed 707 | EventPattern: 708 | source: 709 | - aws.rds 710 | detail: 711 | EventID: 712 | - RDS-EVENT-0159 713 | - RDS-EVENT-0162 714 | State: "ENABLED" 715 | Name: !Join ['-', ['AutoExportDB-export-failed', !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]]] 716 | Targets: 717 | - 718 | Arn: 719 | !Ref RdsExportFailedTopic 720 | Id: "RdsExportFailedTopic" 721 | 722 | EventTopicPolicy: 723 | Type: 'AWS::SNS::TopicPolicy' 724 | Properties: 725 | PolicyDocument: 726 | Statement: 727 | - Effect: Allow 728 | Principal: 729 | Service: events.amazonaws.com 730 | Action: 'sns:Publish' 731 | Resource: '*' 732 | Topics: 733 | - !Ref RdsExportFailedTopic --------------------------------------------------------------------------------