├── README.md ├── github-to-s3 ├── install.sh └── s3-to-github /README.md: -------------------------------------------------------------------------------- 1 | # github-s3 2 | 3 | At Hyper, we make lots of things. Many of those things are open source, but 4 | there are also things that are closed source. We have a limited amount of private 5 | repositories on GitHub, though, so once we haven't needed them for a while we 6 | archive them to AWS S3. 7 | 8 | These are shell scripts that make it really easy to archive and restore repositories 9 | between GitHub and AWS S3. 10 | 11 | ## Usage 12 | 13 | Clone the repository from GitHub, compress it and upload it to AWS S3: 14 | 15 | ```zsh 16 | $ github-to-s3 hyperoslo/old-repository 17 | ``` 18 | 19 | Download the repository from AWS S3, uncompress it and push it to a new private 20 | repository on GitHub: 21 | 22 | ```zsh 23 | $ s3-to-github hyperoslo/old-repository 24 | ``` 25 | 26 | ## Installation 27 | 28 | ``` 29 | $ curl -s --location http://raw.github.com/hyperoslo/github-s3/master/install.sh | sh 30 | ``` 31 | 32 | ## Configuration 33 | 34 | You'll need to set the following environment variables to use these scripts: 35 | 36 | * `GITHUB_S3_BUCKET`: The name of the bucket you want to archive to. 37 | * `AWS_DEFAULT_REGION`: The AWS region your bucket is in. 38 | * `GITHUB_S3_GITHUB_TOKEN`: A GitHub access token with access to create and 39 | delete private repositories. 40 | 41 | ## Hyper loves you 42 | 43 | [Hyper] made this. We're a bunch of folks who love building things. You should 44 | [tweet us] if you can't get it to work. In fact, you should tweet us anyway. 45 | If you're using github-s3, we probably want to [hire you]. 46 | 47 | [Hyper]: https://github.com/hyperoslo 48 | [tweet us]: http://twitter.com/hyperoslo 49 | [hire you]: http://www.hyper.no/jobs/engineers 50 | -------------------------------------------------------------------------------- /github-to-s3: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -e 4 | 5 | if [ -z $GITHUB_S3_BUCKET ]; then 6 | echo 'Error: $GITHUB_S3_BUCKET must be set to the bucket you wish to archive the repository to' 7 | return 8 | fi 9 | 10 | GITHUB_PATH=$1 11 | NAME=$(echo $GITHUB_PATH | sed 's/\//-/') 12 | ARCHIVE_PATH=/tmp/$NAME.tar.gz 13 | REPOSITORY_PATH=/tmp/$NAME 14 | 15 | rm -rf $REPOSITORY_PATH 16 | git clone --mirror git@github.com:$1 $REPOSITORY_PATH 17 | 18 | rm -rf $ARCHIVE_PATH 19 | tar -C /tmp -zcf $ARCHIVE_PATH $NAME 20 | 21 | aws s3 mv $ARCHIVE_PATH s3://$GITHUB_S3_BUCKET/$NAME.tar.gz 22 | 23 | curl -X DELETE -H "Authorization: token $GITHUB_S3_GITHUB_TOKEN" https://api.github.com/repos/$1 24 | rm -rf $REPOSITORY_PATH $ARCHIVE_PATH 25 | -------------------------------------------------------------------------------- /install.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | echo " 4 | ______ __ __ __ __ ________ 5 | / ___(_) /_/ // /_ __/ / ____/ __/_ / 6 | / (_ / / __/ _ / // / _ \/___/\ \_/_ < 7 | \___/_/\__/_//_/\_,_/_.__/ /___/____/ 8 | " 9 | 10 | echo "Downloading scripts to /usr/local/bin..." 11 | curl --progress-bar --location "https://raw.github.com/hyperoslo/github-s3/master/github-to-s3" > /usr/local/bin/github-to-s3 12 | curl --progress-bar --location "https://raw.github.com/hyperoslo/github-s3/master/s3-to-github" > /usr/local/bin/s3-to-github 13 | 14 | chmod +x /usr/local/bin/github-to-s3 15 | chmod +x /usr/local/bin/s3-to-github 16 | 17 | echo "Done." 18 | -------------------------------------------------------------------------------- /s3-to-github: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -e 4 | 5 | if [ -z $GITHUB_S3_BUCKET ]; then 6 | echo 'Error: $GITHUB_S3_BUCKET must be set to the bucket you wish to restore the repository from' 7 | return 8 | fi 9 | 10 | GITHUB_PATH=$1 11 | GITHUB_ORGANIZATION_NAME=$(echo $GITHUB_PATH | sed -E 's/^(.+)\/(.+)/\1/') 12 | GITHUB_REPOSITORY_NAME=$(echo $GITHUB_PATH | sed -E 's/^(.+)\/(.+)/\2/') 13 | NAME=$(echo $GITHUB_PATH | sed 's/\//-/') 14 | ARCHIVE_PATH=/tmp/$NAME.tar.gz 15 | REPOSITORY_PATH=/tmp/$NAME 16 | 17 | aws s3 cp s3://$GITHUB_S3_BUCKET/$NAME.tar.gz /tmp 18 | 19 | rm -rf $REPOSITORY_PATH 20 | tar -C /tmp -xf $ARCHIVE_PATH 21 | 22 | curl -s -X POST -H "Authorization: token $GITHUB_S3_GITHUB_TOKEN" \ 23 | https://api.github.com/orgs/$GITHUB_ORGANIZATION_NAME/repos \ 24 | -d "{\"name\":\"$GITHUB_REPOSITORY_NAME\", \"private\": true}" > /dev/null 25 | 26 | git -C $REPOSITORY_PATH push --mirror 27 | rm -rf $REPOSITORY_PATH $ARCHIVE_PATH 28 | --------------------------------------------------------------------------------