├── .buildpacks ├── .gitignore ├── License.md ├── README.md └── bin └── backup.sh /.buildpacks: -------------------------------------------------------------------------------- 1 | https://github.com/heroku/heroku-buildpack-cli 2 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.swp 2 | tmp 3 | -------------------------------------------------------------------------------- /License.md: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2015 Karl Baum 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Simple heroku app with a bash script for capturing heroku database backups and copying to your s3 bucket. Deploy this as a separate app within heroku and schedule the script to backup your production databases which exist within another heroku project. 2 | 3 | Now using [aws cli v2](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html) - works with both `heroku-18` and `heroku-20` stacks. 4 | 5 | ## Installation 6 | 7 | 8 | First, clone this project, then change directory into the newly created directory: 9 | 10 | ``` 11 | git clone https://github.com/kbaum/heroku-database-backups.git 12 | cd heroku-database-backups 13 | ``` 14 | 15 | Create a project on heroku. 16 | 17 | ``` 18 | heroku create my-database-backups 19 | ``` 20 | Add the heroku-buildpack-cli: 21 | 22 | ``` 23 | heroku buildpacks:add https://github.com/heroku/heroku-buildpack-cli -a my-database-backups 24 | ``` 25 | 26 | Next push this project to your heroku projects git repository. 27 | 28 | ``` 29 | heroku git:remote -a my-database-backups 30 | git push heroku master 31 | ``` 32 | 33 | Now we need to set some environment variables in order to get the heroku cli working properly using the [heroku-buildpack-cli](https://github.com/heroku/heroku-buildpack-cli). 34 | 35 | ``` 36 | heroku config:add HEROKU_API_KEY=`heroku auth:token` -a my-database-backups 37 | ``` 38 | 39 | This creates a token that will quietly expire in one year. To create a long-lived authorization token instead, do this: 40 | 41 | ``` 42 | heroku config:add HEROKU_API_KEY=`heroku authorizations:create -S -d my-database-backups` -a my-database-backups 43 | ``` 44 | 45 | Next we need to add the amazon key and secret from the IAM user that you are using: 46 | 47 | ``` 48 | heroku config:add AWS_ACCESS_KEY_ID=123456 -a my-database-backups 49 | heroku config:add AWS_DEFAULT_REGION=us-east-1 -a my-database-backups 50 | heroku config:add AWS_SECRET_ACCESS_KEY=132345verybigsecret -a my-database-backups 51 | ``` 52 | 53 | And we'll need to also set the bucket and path where we would like to store our database backups: 54 | 55 | ``` 56 | heroku config:add S3_BUCKET_PATH=my-db-backup-bucket/backups -a my-database-backups 57 | ``` 58 | Be careful when setting the S3_BUCKET_PATH to leave off a trailing forward slash. Amazon console s3 browser will not be able to locate your file if your directory has "//" (S3 does not really have directories.). 59 | 60 | Finally, we need to add heroku scheduler and call [backup.sh](https://github.com/kbaum/heroku-database-backups/blob/master/bin/backup.sh) on a regular interval with the appropriate database and app. 61 | 62 | ``` 63 | heroku addons:create scheduler -a my-database-backups 64 | ``` 65 | 66 | Now open it up, in your browser with: 67 | 68 | ``` 69 | heroku addons:open scheduler -a my-database-backups 70 | ``` 71 | 72 | And add the following command to run as often as you like: 73 | 74 | ``` 75 | APP=your-app DATABASE=HEROKU_POSTGRESQL_NAVY_URL /app/bin/backup.sh 76 | ``` 77 | 78 | In the above command, APP is the name of your app within heroku that contains the database. DATABASE is the name of the database you would like to capture and backup. In our setup, DATABASE actually points to a follower database to avoid any impact to our users. Both of these environment variables can also be set within your heroku config rather than passing into the script invocation. 79 | 80 | ### Optional 81 | 82 | You can add a `HEARTBEAT_URL` to the script so a request gets sent every time a backup is made. All you have to do is add the variable value like: 83 | 84 | ``` 85 | heroku config:add HEARTBEAT_URL=https://hearbeat.url -a my-database-backups 86 | ``` 87 | 88 | If you are using [heroku's scheduled backups](https://devcenter.heroku.com/articles/heroku-postgres-backups#scheduling-backups) you might only want to archive the latest 89 | backup to S3 for long-term storage. Set the `ONLY_CAPTURE_TO_S3` variable when running the command: 90 | 91 | ``` 92 | ONLY_CAPTURE_TO_S3=true APP=your-app DATABASE=HEROKU_POSTGRESQL_NAVY_URL /app/bin/backup.sh 93 | ``` 94 | 95 | #### Tip 96 | 97 | The default timezone is `UTC`. To use your [preferred timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) in the filename timestamp, set the `TZ` variable when calling the command: 98 | 99 | ``` 100 | TZ=America/Los_Angeles APP=your-app DATABASE=HEROKU_POSTGRESQL_NAVY_URL /app/bin/backup.sh 101 | ``` -------------------------------------------------------------------------------- /bin/backup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # terminate script as soon as any command fails 4 | set -e 5 | 6 | if [[ -z "$APP" ]]; then 7 | echo "Missing APP variable which must be set to the name of your app where the db is located" 8 | exit 1 9 | fi 10 | 11 | if [[ -z "$DATABASE" ]]; then 12 | echo "Missing DATABASE variable which must be set to the name of the DATABASE you would like to backup" 13 | exit 1 14 | fi 15 | 16 | if [[ -z "$S3_BUCKET_PATH" ]]; then 17 | echo "Missing S3_BUCKET_PATH variable which must be set the directory in s3 where you would like to store your database backups" 18 | exit 1 19 | fi 20 | 21 | # install aws-cli 22 | # - this will already exist if we're running the script manually from a dyno more than once 23 | 24 | aws_command="/tmp/bin/aws" 25 | 26 | if [[ ! -f "${aws_command}" ]]; then 27 | echo "aws cli v2..." 28 | curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" 29 | unzip -q awscliv2.zip 30 | ./aws/install --bin-dir /tmp/bin --install-dir /tmp/aws 31 | fi 32 | 33 | # if the app has heroku pg:backup:schedules, we might just want to just archive the latest backup to S3 34 | # https://devcenter.heroku.com/articles/heroku-postgres-backups#scheduling-backups 35 | # 36 | # set ONLY_CAPTURE_TO_S3 when calling to skip database capture 37 | 38 | BACKUP_FILE_NAME="$(date +"%Y-%m-%d_%H-%M_%Z__")${APP}_${DATABASE}.dump" 39 | 40 | if [[ -z "$ONLY_CAPTURE_TO_S3" ]]; then 41 | heroku pg:backups capture $DATABASE --app $APP 42 | else 43 | BACKUP_FILE_NAME="archive__${BACKUP_FILE_NAME}" 44 | echo " --- Skipping database capture" 45 | fi 46 | 47 | curl -o $BACKUP_FILE_NAME `heroku pg:backups:url --app $APP` 48 | FINAL_FILE_NAME=$BACKUP_FILE_NAME 49 | 50 | if [[ -z "$NOGZIP" ]]; then 51 | gzip $BACKUP_FILE_NAME 52 | FINAL_FILE_NAME=$BACKUP_FILE_NAME.gz 53 | fi 54 | 55 | ${aws_command} s3 cp $FINAL_FILE_NAME s3://$S3_BUCKET_PATH/$APP/$DATABASE/$FINAL_FILE_NAME 56 | 57 | echo "backup $FINAL_FILE_NAME complete" 58 | 59 | if [[ -n "$HEARTBEAT_URL" ]]; then 60 | echo "Sending a request to the specified HEARTBEAT_URL that the backup was created" 61 | curl $HEARTBEAT_URL 62 | echo "heartbeat complete" 63 | fi --------------------------------------------------------------------------------