├── .github └── PULL_REQUEST_TEMPLATE.md ├── LICENSE ├── README.md ├── backup_codecommit.sh ├── backup_repo.sh ├── buildspec.yml ├── codecommit_backup_approach.png ├── codecommit_backup_cfn_template.yaml ├── codecommit_backup_trigger_lambda.py └── deploy.sh /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | *Issue #, if available:* 2 | 3 | *Description of changes:* 4 | 5 | 6 | By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. 7 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 7 | the Software, and to permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 10 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 11 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 13 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 14 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Introduction 2 | 3 | [AWS CodeCommit](https://aws.amazon.com/codecommit/) is a fully-managed source control service that makes it easy for companies to host secure and highly scalable private Git repositories. CodeCommit eliminates the need to operate your own source control system or worry about scaling its infrastructure. You can use CodeCommit to securely store anything from source code to binaries, and it works seamlessly with your existing Git tools. 4 | 5 | You typically don't need to worry about backing up your CodeCommit repositories as CodeCommit's architecture is highly scalable, redundant, and durable. However, there are situations where backups might be helpful. For instance, if one accidentally deletes the CloudFormation stack that created the CodeCommit repository, the entire repository and its contents 6 | are also deleted for good. Oops. 7 | 8 | As per [AWS documentation](https://aws.amazon.com/codecommit/faqs/): _"Deleting an AWS CodeCommit repository is a destructive one-way operation that cannot be undone. To restore a deleted repository, you will need to create the repository again and use either a backup or a local copy from a full clone to upload the data_". 9 | 10 | So, having a backup handy is not a bad idea - Better safe than sorry! 11 | 12 | # The Solution 13 | 14 | This project offers a serverless CodeCommit backup solution (who wants to manage servers these days?) that uses an Amazon CloudWatch event rule as a trigger (eg, trigger the backup every day at 2am UTC). (See the figure below for details). The CloudWatch event targets an AWS Lambda function that simply triggers an AWS CodeBuild container that generates a backup of all AWS CodeCommit repositories within a particular AWS account and region. The backup consists of .tar.gz files named after the repository's name and using a timestamp (eg, Repo1_2017_10_01_02_00). The backups are stored in a designated S3 bucket (eg, backup-bucket/Repo1, backup-bucket/Repo2, etc). One can use S3 lifecycle events to automatically move old backups into Amazon Glacier (cold storage) or alternatively specify an expiration policy for backup files in S3 to have them deleted after a certain period of time. Also 15 | as a security best practice, the S3 bucket storing the backups should enable [default encryption](https://aws.amazon.com/blogs/aws/new-amazon-s3-encryption-security-features/). 16 | 17 | ![approach-overview](codecommit_backup_approach.png) 18 | 19 | # Deploying the Solution 20 | 21 | * Clone this project in your local workstation (only tested on Mac OS) 22 | * Make sure you have the latest version of the [AWS Command Line Interface](http://docs.aws.amazon.com/cli/latest/userguide/installing.html) installed in your local box 23 | * Make sure the user in your AWS profile have permissions to create IAM roles, CloudWatch Event rules, CodeBuild projects and Lambda functions at a minimum (check your profiles under ~/.aws/credentials) 24 | * Open script ./deploy.sh and update these parameters as desired: AWS profile, S3 buckets, and backup schedule. The S3 buckets 25 | will not be created. They must exist already. (See parameters below.) 26 | 27 | ```bash 28 | aws_profile="default" # default AWS profile (or choose another profile) 29 | backup_schedule="cron(0 2 * * ? *)" # backups scheduled for 2am UTC, everyday 30 | scripts_s3_bucket="[S3-BUCKET-FOR-BACKUP-SCRIPTS]s" # bucket must exist in the SAME region the deployment is taking place 31 | backups_s3_bucket="[S3-BUCKET-FOR-BACKUPS" # bucket must exist and have no policy that disallows PutObject from CodeBuild 32 | stack_name="codecommit-backups" # CloudFormation stack name for the solution 33 | ``` 34 | 35 | * By default, all CodeCommit repositories within the AWS region where the solution was deployed to will be backed up everyday at 2am UTC `(cron(0 2 * * ? *))` into the S3 bucket specified 36 | * Run script ./deploy.sh to deploy the solution into your AWS account 37 | 38 | ```bash 39 | chmod +x ./deploy.sh 40 | ./deploy.sh 41 | ``` 42 | 43 | # Catch failed runs 44 | 45 | * Create an AWS SNS email topic 46 | * Create a CloudWatch Event like this one: 47 | ``` 48 | { 49 | "source": [ 50 | "aws.codebuild" 51 | ], 52 | "detail-type": [ 53 | "CodeBuild Build State Change" 54 | ], 55 | "detail": { 56 | "build-status": [ 57 | "FAILED" 58 | ], 59 | "project-name": [ 60 | "CodeCommitBackup" 61 | ] 62 | } 63 | } 64 | ``` 65 | * Assign the SNS topic as a target for this event 66 | 67 | ## License Summary 68 | 69 | This sample code is made available under the MIT-0 license. See the LICENSE file. 70 | -------------------------------------------------------------------------------- /backup_codecommit.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Copyright 2012-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 4 | # 5 | # Licensed under the Amazon Software License (the "License"). 6 | # You may not use this file except in compliance with the License. 7 | # A copy of the License is located at 8 | # 9 | # http://aws.amazon.com/asl/ 10 | # 11 | # or in the "license" file accompanying this file. This file is distributed 12 | # on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 13 | # express or implied. See the License for the specific language governing 14 | # permissions and limitations under the License. 15 | # SPDX-License-Identifier: MIT-0 16 | 17 | set -ex 18 | 19 | # variable CodeCommitBackupsS3Bucket is exported into CodeBuild environment variables 20 | backup_s3_bucket="${CodeCommitBackupsS3Bucket:-"my-s3-bucket"}" 21 | 22 | git config --global credential.helper '!aws codecommit credential-helper $@' 23 | git config --global credential.UseHttpPath true 24 | 25 | declare -a repos=(`aws codecommit list-repositories | jq -r '.repositories[].repositoryName'`) 26 | 27 | if ! parallel --joblog /tmp/log --jobs 400% --max-args=1 \ 28 | ./backup_repo.sh "${backup_s3_bucket}" {} ::: "${repos[@]}" 29 | then 30 | for i in {1..3} 31 | do 32 | sleep $(( i * 5 )) 33 | parallel --resume-failed --joblog /tmp/log --jobs 400% --max-args=1 \ 34 | ./backup_repo.sh "${backup_s3_bucket}" {} ::: "${repos[@]}" && break 35 | done 36 | fi 37 | -------------------------------------------------------------------------------- /backup_repo.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Copyright 2012-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 4 | # 5 | # Licensed under the Amazon Software License (the "License"). 6 | # You may not use this file except in compliance with the License. 7 | # A copy of the License is located at 8 | # 9 | # http://aws.amazon.com/asl/ 10 | # 11 | # or in the "license" file accompanying this file. This file is distributed 12 | # on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 13 | # express or implied. See the License for the specific language governing 14 | # permissions and limitations under the License. 15 | 16 | set -ex 17 | 18 | backup_s3_bucket="${1}" 19 | codecommitrepo="${2}" 20 | 21 | trap 'rm -rf "${codecommitrepo}"; [[ -n "${zipfile:-}" ]] && rm -f "${zipfile}"' EXIT 22 | 23 | echo "[===== Cloning repository: ${codecommitrepo} =====]" 24 | git clone "https://git-codecommit.${AWS_DEFAULT_REGION}.amazonaws.com/v1/repos/${codecommitrepo}" 25 | 26 | dt=$(date -u "+%Y_%m_%d_%H_%M") 27 | zipfile="${codecommitrepo}_backup_${dt}_UTC.tar.gz" 28 | echo "Compressing repository: ${codecommitrepo} into file: ${zipfile} and uploading to S3 bucket: ${backup_s3_bucket}/${codecommitrepo}" 29 | 30 | tar -zcvf "${zipfile}" "${codecommitrepo}/" 31 | aws s3 cp "${zipfile}" "s3://${backup_s3_bucket}/${codecommitrepo}/${zipfile}" --region $AWS_DEFAULT_REGION 32 | 33 | -------------------------------------------------------------------------------- /buildspec.yml: -------------------------------------------------------------------------------- 1 | # Copyright 2012-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # SPDX-License-Identifier: MIT-0 3 | 4 | version: 0.2 5 | 6 | phases: 7 | 8 | install: 9 | commands: 10 | - apt-get update -y 11 | - apt-get install -y jq parallel 12 | 13 | build: 14 | commands: 15 | - chmod +x backup_codecommit.sh 16 | - ./backup_codecommit.sh 17 | -------------------------------------------------------------------------------- /codecommit_backup_approach.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-codecommit-serverless-backup/7e5650e86506f69cb3d802debb69fa03a20af5ed/codecommit_backup_approach.png -------------------------------------------------------------------------------- /codecommit_backup_cfn_template.yaml: -------------------------------------------------------------------------------- 1 | # Copyright 2012-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # SPDX-License-Identifier: MIT-0 3 | 4 | AWSTemplateFormatVersion: '2010-09-09' 5 | Transform: AWS::Serverless-2016-10-31 6 | Parameters: 7 | CodeCommitBackupsScriptsS3Bucket: 8 | Type: "String" 9 | Description: "CodeCommit backup scripts" 10 | CodeCommitBackupsS3Bucket: 11 | Type: "String" 12 | Description: "S3 Bucket for CodeCommit repository backups" 13 | BackupSchedule: 14 | Type: "String" 15 | Description: "Backup schedule as a cron expression" 16 | Default: "cron(0 2 * * ? *)" 17 | BackupScriptsFile: 18 | Type: "String" 19 | Description: "Compressed S3 file containing backup scripts" 20 | Default: "codecommit_backup_scripts.zip" 21 | Resources: 22 | CodeCommitBackupLambdaRole: 23 | Type: "AWS::IAM::Role" 24 | Properties: 25 | AssumeRolePolicyDocument: 26 | Version: "2012-10-17" 27 | Statement: 28 | - 29 | Effect: "Allow" 30 | Principal: 31 | Service: 32 | - "lambda.amazonaws.com" 33 | Action: 34 | - "sts:AssumeRole" 35 | Path: "/" 36 | Policies: 37 | - 38 | PolicyName: "lambdaexecute" 39 | PolicyDocument: 40 | Version: "2012-10-17" 41 | Statement: 42 | - 43 | Effect: "Allow" 44 | Action: 45 | - "logs:*" 46 | Resource: "arn:aws:logs:*:*:*" 47 | - 48 | PolicyName: "codebuild" 49 | PolicyDocument: 50 | Version: "2012-10-17" 51 | Statement: 52 | - 53 | Effect: "Allow" 54 | Action: 55 | - "codebuild:StartBuild" 56 | Resource: !GetAtt CodeBuildProject.Arn 57 | CodeCommitBackupLambda: 58 | Type: AWS::Serverless::Function 59 | Properties: 60 | FunctionName: CodeCommitBackupLambda 61 | Handler: codecommit_backup_trigger_lambda.handler 62 | Runtime: python3.6 63 | MemorySize: 128 64 | Timeout: 30 65 | Role: !GetAtt CodeCommitBackupLambdaRole.Arn 66 | Tags: 67 | Name: CodeCommitBackup 68 | CodeCommitBackupScheduledRule: 69 | Type: "AWS::Events::Rule" 70 | Properties: 71 | Description: "Scheduled rule for CodeCommit backups" 72 | ScheduleExpression: !Ref BackupSchedule 73 | State: "ENABLED" 74 | Targets: 75 | - 76 | Arn: 77 | Fn::GetAtt: 78 | - "CodeCommitBackupLambda" 79 | - "Arn" 80 | Id: "CodeCommitBackupLambda" 81 | PermissionForEventsToInvokeLambda: 82 | Type: "AWS::Lambda::Permission" 83 | Properties: 84 | FunctionName: 85 | Ref: "CodeCommitBackupLambda" 86 | Action: "lambda:InvokeFunction" 87 | Principal: "events.amazonaws.com" 88 | CodeBuildProjectRole: 89 | Type: "AWS::IAM::Role" 90 | Properties: 91 | AssumeRolePolicyDocument: 92 | Version: "2012-10-17" 93 | Statement: 94 | - 95 | Effect: "Allow" 96 | Principal: 97 | Service: 98 | - "codebuild.amazonaws.com" 99 | Action: 100 | - "sts:AssumeRole" 101 | Path: "/" 102 | Policies: 103 | - 104 | PolicyName: "codecommit-readonly" 105 | PolicyDocument: 106 | Version: "2012-10-17" 107 | Statement: 108 | - 109 | Effect: "Allow" 110 | Action: 111 | - "codecommit:BatchGet*" 112 | - "codecommit:Get*" 113 | - "codecommit:Describe*" 114 | - "codecommit:List*" 115 | - "codecommit:GitPull" 116 | Resource: "*" 117 | - 118 | PolicyName: "logs" 119 | PolicyDocument: 120 | Version: "2012-10-17" 121 | Statement: 122 | - 123 | Effect: "Allow" 124 | Action: 125 | - "logs:CreateLogGroup" 126 | - "logs:CreateLogStream" 127 | - "logs:PutLogEvents" 128 | Resource: "*" 129 | - 130 | PolicyName: "s3-artifacts" 131 | PolicyDocument: 132 | Version: "2012-10-17" 133 | Statement: 134 | - 135 | Effect: "Allow" 136 | Action: 137 | - "s3:GetObject" 138 | - "s3:GetObjectVersion" 139 | Resource: 140 | - !Sub "arn:aws:s3:::${CodeCommitBackupsScriptsS3Bucket}/*" 141 | - 142 | PolicyName: "s3-backup" 143 | PolicyDocument: 144 | Version: "2012-10-17" 145 | Statement: 146 | - 147 | Effect: "Allow" 148 | Action: 149 | - "s3:putObject" 150 | Resource: 151 | - !Sub "arn:aws:s3:::${CodeCommitBackupsS3Bucket}/*" 152 | CodeBuildProject: 153 | Type: AWS::CodeBuild::Project 154 | Properties: 155 | Name: CodeCommitBackup 156 | Description: CodeBuild will backup all CodeCommit repo in this region 157 | ServiceRole: !GetAtt CodeBuildProjectRole.Arn 158 | Artifacts: 159 | Type: no_artifacts 160 | Environment: 161 | Type: LINUX_CONTAINER 162 | ComputeType: BUILD_GENERAL1_MEDIUM 163 | Image: aws/codebuild/python:3.5.2 164 | EnvironmentVariables: 165 | - 166 | Name: CodeCommitBackupsS3Bucket 167 | Value: !Ref CodeCommitBackupsS3Bucket 168 | Source: 169 | Location: !Sub "arn:aws:s3:::${CodeCommitBackupsScriptsS3Bucket}/${BackupScriptsFile}" 170 | Type: S3 171 | TimeoutInMinutes: 60 172 | Tags: 173 | - Key: Name 174 | Value: CodeCommitBackup 175 | -------------------------------------------------------------------------------- /codecommit_backup_trigger_lambda.py: -------------------------------------------------------------------------------- 1 | # Copyright 2012-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # SPDX-License-Identifier: MIT-0 3 | 4 | import boto3 5 | client = boto3.client('codebuild') 6 | def handler(event, context): 7 | response = client.start_build(projectName='CodeCommitBackup') 8 | output = "Triggered CodeBuild project: 'CodeCommitBackup' to back all CodeCommit repos in this account/region. Status={}".format(response["build"]["buildStatus"]) 9 | print(output) 10 | return output 11 | -------------------------------------------------------------------------------- /deploy.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Copyright 2012-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 4 | # SPDX-License-Identifier: MIT-0 5 | 6 | #----- Change these parameters to suit your environment -----# 7 | aws_profile="default" 8 | backup_schedule="cron(0 2 * * ? *)" 9 | scripts_s3_bucket="[S3-BUCKET-FOR-BACKUP-SCRIPTS]" # bucket must exist in the SAME region the deployment is taking place 10 | backups_s3_bucket="[S3-BUCKET-FOR-BACKUPS]" # bucket must exist and have no policy that disallows PutObject from CodeBuild 11 | stack_name="codecommit-backups" 12 | #----- End of user parameters -----# 13 | 14 | # You can also change these parameters but it's not required 15 | cfn_template="codecommit_backup_cfn_template.yaml" 16 | cfn_parameters="codecommit_backup_cfn_parameters.json" 17 | zipfile="codecommit_backup_scripts.zip" 18 | cfn_gen_template="/tmp/gen_codecommit_backup_cfn_template.yaml" 19 | 20 | zip -r "${zipfile}" ./ -x .git\* 21 | aws s3 --profile $aws_profile cp "${zipfile}" "s3://${scripts_s3_bucket}" 22 | rm -f "${zipfile}" 23 | 24 | aws cloudformation package \ 25 | --template-file "${cfn_template}" \ 26 | --s3-bucket "$scripts_s3_bucket" \ 27 | --output-template-file "$cfn_gen_template" 28 | 29 | aws cloudformation deploy \ 30 | --profile $aws_profile \ 31 | --stack-name "${stack_name}" \ 32 | --template-file "${cfn_gen_template}" \ 33 | --parameter-overrides \ 34 | CodeCommitBackupsScriptsS3Bucket="${scripts_s3_bucket}" \ 35 | CodeCommitBackupsS3Bucket="${backups_s3_bucket}" \ 36 | BackupSchedule="${backup_schedule}" \ 37 | BackupScriptsFile="${zipfile}" \ 38 | --capabilities CAPABILITY_IAM \ 39 | --tags "Name=${stack_name}" 40 | 41 | --------------------------------------------------------------------------------