├── .github └── PULL_REQUEST_TEMPLATE.md ├── .gitignore ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE.txt ├── README.md ├── build.sh ├── lambda_handler.py ├── s3snapshot.egg-info ├── PKG-INFO ├── SOURCES.txt ├── dependency_links.txt ├── entry_points.txt ├── requires.txt └── top_level.txt ├── s3snapshot ├── __init__.py ├── cli.py └── s3snapshot.py └── setup.py /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | *Issue #, if available:* 2 | 3 | *Description of changes:* 4 | 5 | 6 | By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. 7 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | .idea/ 3 | dist 4 | *.pyc 5 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check [existing open](https://github.com/awslabs/aws-s3snapshot/issues), or [recently closed](https://github.com/awslabs/aws-s3snapshot/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels ((enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/awslabs/aws-s3snapshot/labels/help%20wanted) issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](https://github.com/awslabs/aws-s3snapshot/blob/master/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 7 | the Software, and to permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 10 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 11 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 13 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 14 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # AWS EC2 Snapshot script 2 | 3 | ## Instalation Instructions 4 | 5 | ### CLI Production Mode 6 | - Install the package for running-only: 7 | 8 | ```pip install s3snapshot-X.Y.Z.tar.gz``` 9 | 10 | (Replace X,Y,Z to the current version downloaded) 11 | 12 | ### CLI Development Mode 13 | - Install the package in development mode: 14 | - 15 | ```pip install -e s3snapshot-X.Y.Z.tar.gz``` 16 | 17 | (Replace X,Y,Z to the current version downloaded) 18 | 19 | Or extract the package and run: 20 | 21 | ```python setup.py develop``` 22 | 23 | 24 | ## Lambda Function 25 | If you want to build the package from the source you need to run the `build.sh` script. 26 | This script will copy all source packages from $VIRTUAL_ENV/lib/python2.7/site-packages to the zip file. 27 | To run the script you just need to copy the s3snapshot-X.Y.Z.zip to a S3 bucket or upload directly to your lambda function 28 | 29 | The handler will be: ```lambda_handler:lambda_handler``` 30 | 31 | ## Running 32 | 33 | ### Parameters: 34 | ``` 35 | Usage: s3snapshot [OPTIONS] 36 | 37 | Options: 38 | -l, --label LABEL Label to be included in the Snapshot description 39 | -s, --stop Stop Instance before start the snapshot 40 | -sp, --stopped Check if instance is stopped before start the 41 | snapshot. (If not skip and flag error) 42 | --sns-arn SNS_ARN The SNS topic ARN to send message when finished 43 | --sns-arn-error SNS_ARN The SNS topic ARN to send message when an error 44 | occour! 45 | -f, --filter FILTER Filter list to snapshot. 46 | ex: --filter 47 | '{"instances": ["i-12345678", "i-abcdef12"], 48 | "tags": {"tag:Owner": "John", "tag:Name": "PROD"}}' 49 | --verbose Show extra information during execution 50 | -v, --version Display version number and exit. 51 | --help Show this message and exit. 52 | ``` 53 | 54 | * --filter: 55 | The filter parameter defines which instance will be included in the snapshot job. 56 | 57 | To filter only certain instance ids you can use: 58 | ``` 59 | single instance: 60 | s3snapshot --filter '{"instances": ["i-abcdef12"], "stop" : true}' 61 | 62 | multiple instances: 63 | s3snapshot --filter '{"instances": ["i-abcdef12", "i-12345678", "i-abc123ab"]}' 64 | ``` 65 | 66 | To filter only certain tags: 67 | 68 | ``` 69 | single tag: 70 | s3snapshot --filter '{"tags": {"tag:Env": "PROD"}}' 71 | 72 | multiple tags: 73 | s3snapshot --filter '{"tags": {"tag:Env": "PROD", "tag:Owner": "John"}}' 74 | ``` 75 | It's important to show that instances are single array of values while tags are array of key/pair values 76 | 77 | 78 | ### Lambda Payload 79 | 80 | * Similar to CLI parameters you must provide a valid json document with all the parameters inside. 81 | * Stop Instance can be included as a new key/value pair with { 'stop' : False } 82 | ** False = Snapshot without stop instance 83 | ** True = Stop instance before snapshot (script will start the instance after snapshot) 84 | 85 | Sample json filter: 86 | 87 | Example 1: 88 | ``` 89 | { 90 | "tags": { 91 | "tag:Env": "PROD", 92 | "tag:Owner": "John" 93 | }, 94 | "stop" : false, 95 | "stopped" : false, 96 | "verbose" : false, 97 | "sns-arn" : "arn:aws:sns:us-east-1:100000000000:Snapshot", 98 | "sns-arn-error" : "arn:aws:sns:us-east-1:100000000000:Snapshot-Err", 99 | "label" : "string to include in the description", 100 | "protected" : false 101 | } 102 | ``` 103 | 104 | Example 2: 105 | ``` 106 | { 107 | "instances": [ 108 | "i-abcdef12", 109 | "i-12345678", 110 | "i-abc123ab" 111 | ] 112 | } 113 | 114 | 115 | ``` 116 | 117 | * JSON strings must use double-quote 118 | 119 | ## Changes 120 | 121 | ### Version 0.1.5 - 2016-11-17 122 | * Bugix: Introduced a bug in the ClientError handler where I invoked create_tag inside the error handling 123 | * Included the Tagging of keys: Scripted and State:Protected 124 | 125 | ### Version 0.1.4 - 2016-11-16 126 | * Changed the time.sleep to .1 instead of 1 second 127 | 128 | ### Version 0.1.3 - 2016-11-14 129 | * Bugfix in error handling that don't have the RequestsLimitsExceeded 130 | 131 | ### Version 0.1.2 - 2016-08-05 132 | * Included the error management for RequestLimitExceeded in CreateTags API 133 | 134 | ### Version 0.1.1 - 2016-07-25 135 | * Corrected the stopped flag to start instances after the snapshot 136 | 137 | ### Version 0.1.0 - 2016-07-15 138 | * Initial version 139 | * Support both cli execution or lambda execution 140 | -------------------------------------------------------------------------------- /build.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # 4 | # Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 5 | # 6 | # SPDX-License-Identifier: MIT-0 7 | # 8 | #This script create a lambda package from the dist package generated by setuptools 9 | function usage 10 | { 11 | echo "usage: build [[-t] | [-h]]" 12 | echo "Parameters:" 13 | echo "-t | --transfer = transfer the zip package to S3 bucket (koiker-upload)" 14 | } 15 | 16 | cur_pwd=$(pwd) 17 | transfer=false 18 | bucket="your-bucket-name" 19 | 20 | # Process input parameters 21 | while [ "$1" != "" ]; do 22 | case $1 in 23 | -t | --transfer )echo "Transfering package to S3" 24 | transfer=true 25 | ;; 26 | -h | --help ) usage 27 | exit 28 | ;; 29 | esac 30 | shift 31 | done 32 | 33 | echo "AWS Lambda package builder" 34 | echo "" 35 | echo "Current directory: $cur_pwd" 36 | 37 | # Checking VIRTUAL_ENV 38 | if ! [ -n "$VIRTUAL_ENV" ]; then 39 | echo "Please create virtualenv or activate: " 40 | echo "virtualenv aws-s3snapshot" 41 | echo "source aws-s3snapshot/bin/activate" 42 | exit 1 43 | fi 44 | 45 | virtualenv=$VIRTUAL_ENV 46 | 47 | # Checking if dist and file exist 48 | if ! [ -d dist ]; then 49 | echo "There is no package to process. Please run: python setup.py sdist" 50 | exit 1 51 | fi 52 | 53 | echo "Changing to dist directory" 54 | cd dist 55 | 56 | file=$(find . -regex ".*s3snapshot-[0-9]\.[0-9]\.[0-9]\.tar\.gz" -type f | sort | tail -n1) 57 | 58 | if [ -z "$file" ]; then 59 | echo "There is no package to process. Please run: python setup.py sdist first" 60 | exit 1 61 | fi 62 | 63 | echo "Found latest file: $file" 64 | echo "" 65 | echo "Extractig files" 66 | tar -xvzf $file 67 | tmp_folder="${file%.*}" 68 | folder="${tmp_folder%.*}" 69 | 70 | echo "Entering the extracted folder: $folder" 71 | cd $folder 72 | zip_file=$folder'.zip' 73 | 74 | echo "Creating new zip file: $zip_file" 75 | zip -r9 $zip_file * 76 | 77 | echo "Adding site-packages to the zip file" 78 | tmp_folder=$(pwd) 79 | cd $VIRTUAL_ENV/lib/python2.7/site-packages/ 80 | zip -r9 $tmp_folder${zip_file#.} * -x "*boto3*" -x "*botocore*" -x "*pip*" -x "*s3transfer*" 81 | cd $tmp_folder 82 | 83 | echo "Moving zip file to dist folder" 84 | mv $zip_file .. 85 | tmp_folder=$(pwd) 86 | cd .. 87 | 88 | if [ "$transfer" = true ]; then 89 | echo "Copying file to S3 bucket" 90 | aws s3 cp $zip_file s3://$bucket 91 | fi 92 | 93 | echo "Removing temporary extracted folder: $tmp_folder" 94 | 95 | read -p "Do you wish to remove this folder? (yes/no): " yn 96 | 97 | if [ "$yn" == "yes" ]; then 98 | rm -rf $tmp_folder 99 | else 100 | echo "Keeping the temporary folder" 101 | fi 102 | 103 | cd $cur_pwd 104 | echo "Finished!" 105 | -------------------------------------------------------------------------------- /lambda_handler.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # s3backup.py 4 | # 5 | # Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 6 | # 7 | # SPDX-License-Identifier: MIT-0 8 | # 9 | """ AWS Lambda handler """ 10 | 11 | from __future__ import print_function 12 | 13 | import time 14 | 15 | import click 16 | import pkg_resources 17 | 18 | from s3snapshot.s3snapshot import s3snapshot 19 | 20 | 21 | def lambda_handler(event, context): 22 | """ 23 | This function read the event data and parse to s3snapshot function 24 | """ 25 | PACKAGE = pkg_resources.require("s3snapshot")[0].project_name 26 | VERSION = pkg_resources.require("s3snapshot")[0].version 27 | 28 | start = time.time() 29 | click.echo('[+] Start time: {0}'.format(time.strftime('%Y-%m-%d %H:%M:%S %Z', time.localtime(start)))) 30 | 31 | response = s3snapshot( 32 | event=event, 33 | context=context, 34 | start_time=start, 35 | program='{package}-{version}'.format( 36 | package=PACKAGE, version=VERSION 37 | ) 38 | ) 39 | 40 | code = response.get('result') 41 | message = '{icon} The s3snapshot was : {status}'.format( 42 | icon='[=]' if code == 'Sucessfull' else '[!]', 43 | status=code 44 | ) 45 | return message 46 | -------------------------------------------------------------------------------- /s3snapshot.egg-info/PKG-INFO: -------------------------------------------------------------------------------- 1 | Metadata-Version: 1.0 2 | Name: s3snapshot 3 | Version: 0.1.5 4 | Summary: Snapshot script 5 | Home-page: UNKNOWN 6 | Author: Rafael M. Koike 7 | Author-email: koiker@amazon.com 8 | License: Apache Software License 9 | Description: UNKNOWN 10 | Platform: UNKNOWN 11 | -------------------------------------------------------------------------------- /s3snapshot.egg-info/SOURCES.txt: -------------------------------------------------------------------------------- 1 | lambda_handler.py 2 | setup.py 3 | s3snapshot/__init__.py 4 | s3snapshot/cli.py 5 | s3snapshot/s3snapshot.py 6 | s3snapshot.egg-info/PKG-INFO 7 | s3snapshot.egg-info/SOURCES.txt 8 | s3snapshot.egg-info/dependency_links.txt 9 | s3snapshot.egg-info/entry_points.txt 10 | s3snapshot.egg-info/requires.txt 11 | s3snapshot.egg-info/top_level.txt -------------------------------------------------------------------------------- /s3snapshot.egg-info/dependency_links.txt: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /s3snapshot.egg-info/entry_points.txt: -------------------------------------------------------------------------------- 1 | 2 | [console_scripts] 3 | s3snapshot=s3snapshot.cli:cli 4 | -------------------------------------------------------------------------------- /s3snapshot.egg-info/requires.txt: -------------------------------------------------------------------------------- 1 | boto3 2 | click 3 | -------------------------------------------------------------------------------- /s3snapshot.egg-info/top_level.txt: -------------------------------------------------------------------------------- 1 | lambda_handler 2 | s3snapshot 3 | -------------------------------------------------------------------------------- /s3snapshot/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # __init__.py 4 | # 5 | # Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 6 | # 7 | # SPDX-License-Identifier: MIT-0 8 | # 9 | # 10 | 11 | __author__ = "Rafael M. Koike" 12 | __email__ = "koiker@amazon.com" 13 | __date__ = "2016-11-14" 14 | __version__ = '0.1.5' 15 | -------------------------------------------------------------------------------- /s3snapshot/cli.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # cli.py 4 | # 5 | # Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 6 | # 7 | # SPDX-License-Identifier: MIT-0 8 | # 9 | """ Backup script to copy files to S3 """ 10 | 11 | from __future__ import print_function 12 | 13 | import datetime 14 | import json 15 | import locale 16 | import sys 17 | import time 18 | 19 | import click 20 | import pkg_resources 21 | 22 | from .s3snapshot import SNS_ARN 23 | from .s3snapshot import SNS_ARN_ERROR 24 | from .s3snapshot import STOP 25 | from .s3snapshot import STOPPED 26 | from .s3snapshot import VERBOSE 27 | from .s3snapshot import s3snapshot 28 | 29 | locale.setlocale(locale.LC_ALL, '') 30 | 31 | reload(sys) 32 | sys.setdefaultencoding('utf8') 33 | 34 | 35 | @click.command() 36 | @click.option('-l', '--label', metavar='LABEL', help='Label to be included in the Snapshot description') 37 | @click.option('-s', '--stop', is_flag=True, default=STOP, help='Stop Instance before start the snapshot') 38 | @click.option('-sp', '--stopped', is_flag=True, default=STOPPED, 39 | help='Check if instance is stopped before start the snapshot. (If not skip and flag error)') 40 | @click.option('--sns-arn', metavar='SNS_ARN', help='The SNS topic ARN to send message when finished') 41 | @click.option('--sns-arn-error', metavar='SNS_ARN', help='The SNS topic ARN to send message when an error occour!') 42 | @click.option('-f', '--filter', metavar='FILTER', help=('Filter list to snapshot.\n' 43 | 'ex: --filter \'{"instances": ["i-12345678", "i-abcdef12"], "tags": {"tag:Owner": "John", "tag:Name": "PROD"}}\'''')) 44 | @click.option('--verbose', is_flag=True, default=VERBOSE, help='Show extra information during execution') 45 | @click.version_option() 46 | # Main function for CLI iteration 47 | def cli(*args, **kwargs): 48 | filter_args = kwargs.pop('filter') 49 | label = kwargs.pop('label') 50 | sns_arn = kwargs.pop('sns_arn') or SNS_ARN 51 | sns_arn_error = kwargs.pop('sns_arn_error') or SNS_ARN_ERROR 52 | # Lets build the JSON (dict) variable to pass to s3snapshot 53 | event = dict() 54 | event['stop'] = kwargs.pop('stop') 55 | event['stopped'] = kwargs.pop('stopped') 56 | event['sns-arn'] = sns_arn 57 | event['sns-arn-error'] = sns_arn_error 58 | event['label'] = label 59 | 60 | if filter_args: 61 | filter_args = json.loads(filter_args) 62 | if 'tags' in filter_args.keys(): 63 | event['tags'] = filter_args['tags'] 64 | 65 | if 'instances' in filter_args.keys(): 66 | event['instances'] = filter_args['instances'] 67 | 68 | PACKAGE = pkg_resources.require("s3snapshot")[0].project_name 69 | VERSION = pkg_resources.require("s3snapshot")[0].version 70 | 71 | click.echo('ECTP Snapshot to S3 - Version {0}'.format(VERSION)) 72 | click.echo('') 73 | 74 | if event['stop'] and event['stopped']: 75 | click.echo('[!] Unable to process. You need to choose --stop or --stopped option') 76 | return 77 | 78 | start = time.time() 79 | click.echo('[+] Start time: {0}'.format( 80 | time.strftime('%Y-%m-%d %H:%M:%S %Z', time.localtime(start))) 81 | ) 82 | 83 | response = s3snapshot( 84 | verbose=kwargs.pop('verbose'), 85 | event=event, 86 | start_time=start, 87 | program='{package}-{version}'.format( 88 | package=PACKAGE, version=VERSION 89 | ) 90 | ) 91 | 92 | elapsed = time.time() - start 93 | 94 | click.echo('[+] Elapsed time {0}'.format( 95 | datetime.timedelta(seconds=elapsed) 96 | ) 97 | ) 98 | code = response['result'] 99 | message = '{icon} The s3snapshot was : {status}'.format( 100 | icon='[=]' if code == 'Sucessfull' else '[!]', 101 | status=code 102 | ) 103 | return click.echo(message) 104 | -------------------------------------------------------------------------------- /s3snapshot/s3snapshot.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # s3backup.py 4 | # 5 | # Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 6 | # 7 | # SPDX-License-Identifier: MIT-0 8 | # 9 | """Backup script to copy files to S3""" 10 | 11 | from __future__ import print_function 12 | 13 | import datetime 14 | import json 15 | import time 16 | import traceback 17 | 18 | import boto3 19 | import botocore 20 | import click 21 | from botocore.exceptions import ClientError 22 | 23 | # Constant strings for Default Values (Change if you need to run diferently) 24 | SUCCESS = 'Successful' 25 | FAULT = 'Fault' 26 | PARTIAL = 'Partial' 27 | STOP = False 28 | STOPPED = False 29 | LABEL = '' 30 | SNS_ARN = 'arn:aws:sns:us-east-1:109881088269:SAP-Backup' 31 | SNS_ARN_ERROR = 'arn:aws:sns:us-east-1:109881088269:SAP-Backup' 32 | PROTECTED = False 33 | VERBOSE = False 34 | SLEEP_TIME = 0.1 35 | 36 | 37 | class SnapshotItem(object): 38 | def __init__(self, volume_id, instance_id, instance_name, root_device, tags, state, device_name): 39 | self.volume_id = volume_id 40 | self.instance_id = instance_id 41 | self.instance_name = instance_name 42 | self.root_device = root_device 43 | self.tags = tags 44 | self.state = state 45 | self.device_name = device_name 46 | 47 | 48 | class SnapshotName(object): 49 | def __init__(self, date, name, device, volume_id, owner_id): 50 | """ 51 | Set the variables to be used in the search of the Snapshot names 52 | """ 53 | self.date = date 54 | self.name = name 55 | self.device = device 56 | self.volume_id = volume_id 57 | self.owner_id = owner_id 58 | self.snapshot_query = 's{date}*'.format(date=date) 59 | 60 | def __call__(self, client): 61 | """ 62 | Run the search based on the variables and return the result 63 | The search structure is: 64 | sYYYYMMDD? and from the same OnwerId and VolumeId 65 | If the search result is empty the first letter to asign is 'a' otherwise is the next subsequent letter 66 | """ 67 | response = client.describe_snapshots( 68 | OwnerIds=[self.owner_id], 69 | Filters=[ 70 | {'Name': 'tag:Name', 'Values': [self.snapshot_query]}, 71 | {'Name': 'volume-id', 'Values': [self.volume_id]} 72 | ] 73 | ) 74 | 75 | if response['ResponseMetadata']['HTTPStatusCode'] == 200: 76 | tag_names = [] 77 | for snapshot in response['Snapshots']: 78 | for tag in snapshot['Tags']: 79 | if tag['Key'] == 'Name': 80 | tag_names.append(tag['Value'].split('-')[0]) 81 | 82 | # If the list of snapshot is not empty, let's sort and get the last one to increment 83 | if tag_names: 84 | tag_names.sort() 85 | return '{date}-{name}-{device}'.format( 86 | date=increment_string(tag_names[-1]), 87 | name=self.name, 88 | device=self.device 89 | ) 90 | 91 | # The list is empty. Let's build the first name 92 | else: 93 | # Return the first name with 'a' 94 | return 's{date}a-{name}-{device}'.format( 95 | date=self.date, 96 | name=self.name, 97 | device=self.device 98 | ) 99 | # Any error will result in building the first name 100 | else: 101 | # Return the first name with 'a' 102 | return 's{date}a-{name}-{device}'.format( 103 | date=self.date, 104 | name=self.name, 105 | device=self.device 106 | ) 107 | 108 | 109 | def increment_string(s): 110 | pos = ord(s[-1]) 111 | if 65 <= pos <= 90: 112 | # Upper Case Letter 113 | upper_limit = 90 114 | loop_to = 97 115 | 116 | elif 97 <= pos <= 122: 117 | # Lower Case Letter 118 | upper_limit = 122 119 | loop_to = 65 120 | 121 | else: 122 | # Character not in string.ascii_letters 123 | return '' 124 | 125 | new_pos = pos + 1 if pos + 1 <= upper_limit else loop_to 126 | return s[:-1] + chr(new_pos) 127 | 128 | 129 | def send_sns_message(sns_topic, subject, msg, msg_sms=None, msg_email=None, 130 | msg_apns=None, msg_gcm=None): 131 | """ 132 | This function send SNS message to specific topic and can format different 133 | mesages to e-mail, SMS, Apple iOS and Android 134 | """ 135 | client_sns = boto3.client('sns') 136 | sns_arn = sns_topic 137 | 138 | sns_body = dict() 139 | sns_body['default'] = msg 140 | sns_body['email'] = msg_email or msg 141 | sns_body['sms'] = msg_sms or msg 142 | sns_body['APNS'] = {'aps': {'alert': msg_apns or msg}} 143 | sns_body['GCM'] = {'data': {'message': msg_gcm or msg}} 144 | 145 | client_sns.publish( 146 | TargetArn=sns_arn, 147 | Subject=subject, 148 | MessageStructure='json', 149 | Message=json.dumps(sns_body) 150 | ) 151 | 152 | 153 | def s3snapshot(verbose=VERBOSE, start_time=time.time(), program='', event=None, context=None): 154 | """ 155 | This function read the parameters from json list and execute the snapshot 156 | 157 | list = json list with filter (instance-id or tags) 158 | list can contain stop=true/false (If the instances need to be stopped before 159 | the snapshot start) 160 | verbose: bool 161 | start_time: time 162 | event: dict 163 | program: str 164 | context: 165 | """ 166 | client = boto3.client('ec2') 167 | flag_error = False 168 | stop = STOP 169 | stopped = STOPPED 170 | label = LABEL 171 | sns_arn = SNS_ARN 172 | sns_arn_error = SNS_ARN_ERROR 173 | protected = PROTECTED 174 | error_msg = list() 175 | filter_list = [] 176 | 177 | if event: 178 | # Start parsing the arguments received from Lambda 179 | if 'tags' in event.keys(): 180 | tags = event.get('tags', {}) 181 | for key, value in tags.items(): 182 | filter_list.append({'Name': key, 'Values': [value]}) 183 | 184 | if 'instances' in event.keys(): 185 | instances = event.get('instances') 186 | filter_list.append({ 187 | 'Name': 'instance-id', 188 | 'Values': [item for item in instances] 189 | } 190 | ) 191 | 192 | if 'stop' in event.keys(): 193 | stop = event.get('stop') 194 | 195 | if 'stopped' in event.keys(): 196 | stopped = event.get('stopped') 197 | 198 | if 'verbose' in event.keys(): 199 | verbose = event.get('verbose') 200 | 201 | if 'sns-arn' in event.keys(): 202 | sns_arn = event.get('sns-arn') 203 | 204 | if 'sns-arn-error' in event.keys(): 205 | sns_arn_error = event.get('sns-arn-error') 206 | 207 | if 'label' in event.keys(): 208 | label = event.get('label') 209 | 210 | if 'protected' in event.keys(): 211 | protected = event.get('protected') 212 | 213 | click.echo('[+] The stop parameter is : {0}'.format(stop)) 214 | click.echo('[+] The stopped parameter is : {0}'.format(stopped)) 215 | 216 | try: 217 | response = client.describe_instances(Filters=filter_list) 218 | # Get the instances 219 | instances = response[response.keys()[0]] 220 | # Get the number of instances to inform in the SNS topic 221 | total_instances = len(instances) 222 | snapshot_volumes = list() 223 | 224 | for line in instances: 225 | instance = line['Instances'][0] 226 | 227 | if verbose: 228 | click.echo('[+] Instance Id to snapshot : {instance}'.format(instance=instance['InstanceId'])) 229 | click.echo('[+] Block Devices to snapshot:') 230 | 231 | for block in instance['BlockDeviceMappings']: 232 | tags = [] 233 | name = None 234 | if block.get('Ebs', None) is None: 235 | continue 236 | 237 | if instance.get('Tags'): 238 | # Pool the tags and get the Instance name to use and strip the tags that begin 'aws:' 239 | for tag in instance.get('Tags', {}): 240 | if not tag.get('Key').startswith('aws:') and not tag.get('Key') == 'Name': 241 | tags.append({'Key': tag['Key'], 'Value': tag['Value']}) 242 | 243 | elif tag.get('Key') == 'Name': 244 | # Search for the snapshots with the current device to check if there 245 | # is other snapshots from today 246 | name = SnapshotName( 247 | date=datetime.datetime.today().strftime('%Y%m%d'), 248 | name=tag.get('Value'), 249 | device=block.get('DeviceName'), 250 | volume_id=block.get('Ebs', {}).get('VolumeId'), 251 | owner_id=line.get('OwnerId') 252 | ) 253 | tags.append({'Key': tag.get('Key'), 'Value': name(client)}) 254 | 255 | if not name: 256 | # If the volume don't have Tags we create the tag Name 257 | name = SnapshotName( 258 | date=datetime.datetime.today().strftime('%Y%m%d'), 259 | name=instance.get('InstanceId'), 260 | device=block.get('DeviceName'), 261 | volume_id=block.get('Ebs', {}).get('VolumeId'), 262 | owner_id=line.get('OwnerId') 263 | ) 264 | tags.append({'Key': "Name", 'Value': name(client)}) 265 | 266 | # Add the volume to the list of items to snapshot 267 | snapshot_volumes.append( 268 | SnapshotItem( 269 | volume_id=block.get('Ebs', {}).get('VolumeId'), 270 | instance_id=instance.get('InstanceId'), 271 | instance_name=name if name else instance.get('InstanceId'), 272 | device_name=block.get('DeviceName'), 273 | root_device=True if block.get('DeviceName') == instance.get('RootDeviceName') else False, 274 | tags=tags, 275 | state=instance.get('State', {}).get('Name') 276 | ) 277 | ) 278 | 279 | if verbose: 280 | click.echo('[\] ID: [ {id} - {device} ]'.format(id=id, device=block.get('DeviceName'))) 281 | 282 | except Exception: 283 | click.echo('[!] Unable to get instances info. Check your permissions or connectivity') 284 | if verbose: 285 | click.echo('Error {error}'.format(error=traceback.format_exc())) 286 | flag_error = True 287 | error_msg.append(traceback.format_exc()) 288 | return {'result': FAULT} 289 | 290 | # Number of items 291 | total_volumes = len(snapshot_volumes) 292 | # Number of snapshots successfull (At this point none) 293 | total_success = 0 294 | # Number of snapshots failed (At this point is zero :) 295 | total_failures = 0 296 | 297 | # Start Snapshot Creation 298 | for snapshot in snapshot_volumes: 299 | click.echo( 300 | '[+] Snapshot Instance-id : {id} - Volume-id : {vol} - Block-dev : {block} - Root-dev : {root}'.format( 301 | id=snapshot.instance_id, 302 | vol=snapshot.volume_id, 303 | block=snapshot.device_name, 304 | root=snapshot.root_device 305 | ) 306 | ) 307 | 308 | snapshot_desc = 'Script {program} [Instance ID = {instance}] [Stop : {stop}] [Stopped : {stopped}] [State : {state}] {label}'.format( 309 | program=program, 310 | instance=snapshot.instance_id, 311 | stop=stop, 312 | stopped=stopped, 313 | state=snapshot.state, 314 | label=label 315 | ) 316 | 317 | # Need to stop first? 318 | instance_stopped = False 319 | if stop: 320 | click.echo('[+] Stopping instance : {id}'.format(id=snapshot.instance_id)) 321 | response = client.stop_instances(InstanceIds=[snapshot.instance_id]) 322 | 323 | try: 324 | click.echo('[+] Waiting till the instance is stopped...') 325 | client.get_waiter('instance_stopped').wait(InstanceIds=[snapshot.instance_id]) 326 | instance_stopped = True 327 | 328 | except Exception: 329 | click.echo('[!] Error waiting for instance to stop!') 330 | flag_error = True 331 | total_failures += 1 332 | error_msg.append(traceback.format_exc()) 333 | 334 | # Need to check if the instance is already stopped before start the snapshot 335 | if (stopped and snapshot.state == 'stopped') or (not stopped and stop and instance_stopped) or ( 336 | not stop and not stopped): 337 | 338 | try: 339 | # Try to avoid Throttling API requests. Wait 1 second before start. 340 | time.sleep(SLEEP_TIME) 341 | response = client.create_snapshot( 342 | DryRun=False, 343 | VolumeId=snapshot.volume_id, 344 | Description=snapshot_desc 345 | ) 346 | 347 | except Exception: 348 | click.echo( 349 | ('[!] Unable to run CreateSnapshot of:\n' 350 | 'Instance-id : {id}' 351 | ' - Volume-id : {vol}' 352 | ' - Block-dev : {block}' 353 | ' - Root-dev : {root}').format( 354 | id=snapshot.instance_id, vol=snapshot.volume_id, 355 | block=snapshot.device_name, root=snapshot.root_device 356 | ) 357 | ) 358 | flag_error = True 359 | total_failures += 1 360 | if verbose: 361 | click.echo('[!] {0}'.format(traceback.format_exc())) 362 | 363 | # Add the error message in the stack to send by e-mail later 364 | error_msg.append(traceback.format_exc()) 365 | 366 | # Did the Snapshot run correctly? 367 | if response.get('State', 'error') is not 'error': 368 | click.echo('[=] Snapshot created!') 369 | click.echo('[+] Snapshot Name : {name} - ID : {id}'.format( 370 | name=snapshot_desc, 371 | id=response['SnapshotId']) 372 | ) 373 | if verbose: 374 | # This in line function will manage datetime.datetime inside response dict 375 | date_handler = lambda obj: ( 376 | obj.isoformat() if isinstance(obj, datetime.datetime) or isinstance(obj, 377 | datetime.date) else None) 378 | click.echo('[~] response info {info}'.format( 379 | info=json.dumps(response, default=date_handler, indent=4)) 380 | ) 381 | 382 | # Add tags to Snapshot 383 | try: 384 | # loop thru each tag to ignore tags with prefix: 'aws: 385 | for tag in snapshot.tags: 386 | if not tag.get('Key', '').startswith('aws:'): 387 | client.create_tags(Resources=[response['SnapshotId']], Tags=[tag]) 388 | 389 | # Tag the State and Scripted tags 390 | client.create_tags(Resources=[response.get('SnapshotId')], Tags=[{"Key": "Scripted", "Value": "True"}]) 391 | client.create_tags(Resources=[response.get('SnapshotId')], 392 | Tags=[{"Key": "State:Protected", "Value": '{}:{}'.format(snapshot.state, protected)}]) 393 | 394 | except botocore.exceptions.ClientError: 395 | # If we have requested too fast we need to wait an run again ;-) 396 | click.echo('[!] Error writing tags... Waiting for {} secods'.format(SLEEP_TIME)) 397 | time.sleep(SLEEP_TIME) 398 | flag_error = True 399 | total_failures += 1 400 | pass 401 | 402 | except Exception: 403 | click.echo('[!] Error creating Snapshot Tags: {error}'.format(error=traceback.format_exc())) 404 | # Add the error message in the stack to send by e-mail later 405 | error_msg.append(traceback.format_exc()) 406 | flag_error = True 407 | total_failures += 1 408 | pass 409 | 410 | else: 411 | click.echo('[!] Snapshot creation Failed!') 412 | flag_error = True 413 | total_failures += 1 414 | 415 | else: 416 | click.echo('[!] Instance required to be stopped but is not stopped. Skipping') 417 | flag_error = True 418 | total_failures += 1 419 | 420 | if stop or stopped: 421 | # Bring instance back to running state 422 | # (Wait 1 second to give time to snapshot start) 423 | time.sleep(SLEEP_TIME) 424 | response = client.start_instances(InstanceIds=[snapshot.instance_id]) 425 | 426 | if not flag_error: 427 | total_success += 1 428 | # Reset the flag_error to iterate again 429 | flag_error = False 430 | click.echo('') 431 | 432 | msg_result = '' 433 | msg_result += '[=] Total Instances : {instances}\n'.format(instances=total_instances) 434 | msg_result += '[=] Total volumes to process : {total}\n'.format(total=total_volumes) 435 | msg_result += '[=] Total volumes failed : {failed}\n'.format(failed=total_failures) 436 | msg_result += '[=] Total volumes success : {success}\n'.format(success=total_success) 437 | 438 | click.echo(msg_result) 439 | 440 | if total_success == total_volumes: 441 | status = SUCCESS 442 | elif total_success > 0: 443 | status = PARTIAL 444 | else: 445 | status = FAULT 446 | 447 | # Finished Processing. Send SNS results 448 | 449 | click.echo('[+] Sending SNS topic ') 450 | elapsed_time = time.time() - start_time 451 | message_default = ( 452 | 'The snapshot of servers has been {status}\n' 453 | 'Start time: {start}\n' 454 | 'Elapsed time {elapsed}\n' 455 | '{total}').format( 456 | status=status, 457 | start=time.strftime('%Y-%m-%d %H:%M:%S %Z', time.localtime(start_time)), 458 | elapsed=datetime.timedelta(seconds=elapsed_time), 459 | total=msg_result 460 | ) 461 | if context: 462 | message_context = '[+] Json parameter passed to lambda function {json}\n'.format(json=event) 463 | message_context += ( 464 | '[~] For more information read context\n' 465 | '[~] Log stream name: {name}\n' 466 | '[~] Log group name: {group}\n' 467 | '[~] Request ID: {request}\n' 468 | '[~] Mem. limits(MB): {limits}\n').format( 469 | name=context.log_stream_name, 470 | group=context.log_group_name, 471 | request=context.aws_request_id, 472 | limits=context.memory_limit_in_mb 473 | ) 474 | message_default += message_context 475 | if verbose: 476 | click.echo('{message}'.format(message=message_context)) 477 | try: 478 | send_sns_message( 479 | sns_arn, 480 | subject='[Snapshot {status}]'.format(status=status), 481 | msg=message_default 482 | ) 483 | 484 | except Exception: 485 | click.echo('[!] Error when sending SNS message: Unable to send SNS') 486 | if verbose: 487 | click.echo('[!] Error: {0}'.format(traceback.format_exc())) 488 | error_msg.append(traceback.format_exc()) 489 | 490 | if status == FAULT or status == PARTIAL: 491 | message_default = ( 492 | 'There are errors during the snapshot processing\n' 493 | 'Bellow the information of the processing job:\n' 494 | 'Start time : {start}\n').format( 495 | start=time.strftime('%Y-%m-%d %H:%M:%S %Z', time.localtime(start_time)) 496 | ) 497 | 498 | # Include all error messages in the default msg 499 | for line in error_msg: 500 | message_default += '\n' 501 | message_default += 'error: {0}'.format(line) 502 | 503 | # Customize the error message to SMS 504 | message_sms = ('Snapshot status: {status}\n' 505 | 'There are errors during the Snapshot\n' 506 | 'Look in your e-mail for more information').format(status=status) 507 | try: 508 | send_sns_message( 509 | sns_arn_error, 510 | subject='[Snapshot {status}]'.format(status=status), 511 | msg=message_default, 512 | msg_sms=message_sms 513 | ) 514 | except: 515 | click.echo('[!] Error when sending SNS error message: Unable to send SNS') 516 | if verbose: 517 | click.echo('[!] {0}'.format(traceback.format_exc())) 518 | 519 | # Return an HTTP error code 520 | return {'result': status} 521 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # setup.py 4 | # 5 | # Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 6 | # 7 | # SPDX-License-Identifier: MIT-0 8 | # 9 | # 10 | 11 | """ 12 | S3 Snapshot tools 13 | """ 14 | 15 | import io 16 | import os 17 | import re 18 | from setuptools import setup 19 | 20 | 21 | def read(*names, **kwargs): 22 | with io.open( 23 | os.path.join(os.path.dirname(__file__), *names), 24 | encoding=kwargs.get("encoding", "utf8") 25 | ) as fp: 26 | return fp.read() 27 | 28 | 29 | def find_version(*file_paths): 30 | version_file = read(*file_paths) 31 | version_match = re.search(r"^__version__ = ['\"]([^'\"]*)['\"]", 32 | version_file, re.M) 33 | if version_match: 34 | return version_match.group(1) 35 | raise RuntimeError("Unable to find version string.") 36 | 37 | 38 | setup( 39 | name='s3snapshot', 40 | version=find_version("s3snapshot", "__init__.py"), 41 | license='Apache Software License', 42 | py_modules=['s3snapshot.s3snapshot', 's3snapshot.cli', 'lambda_handler'], 43 | author='Rafael M. Koike', 44 | author_email='koiker@amazon.com', 45 | description='Snapshot script', 46 | install_requires=[ 47 | 'boto3', 48 | 'click' 49 | ], 50 | entry_points=''' 51 | [console_scripts] 52 | s3snapshot=s3snapshot.cli:cli 53 | ''', 54 | ) 55 | --------------------------------------------------------------------------------