├── .gitignore ├── README.md ├── simplecd.sh └── webmonitor.sh /.gitignore: -------------------------------------------------------------------------------- 1 | .*.swp 2 | .idea/ 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Simple Continuous Delivery 2 | 3 | 4 | ## About 5 | 6 | SimpleCD is a Continuous Delivery system written for the Bash shell. 7 | 8 | It provides a very simple environment which allows you to continuously deliver 9 | your software to a staging and/or production environment while also running 10 | your end-to-end and/or unit tests in advance, deploying only those deliverables 11 | to your environments that don't have failing tests. 12 | 13 | SimpleCD is completely agnostic in regards to unit- and e2e-test frameworks and 14 | doesn't know how to deploy deliverables itself. These steps are defined and 15 | implemented within the projects that are to be delivered, and SimpleCD merely 16 | executes these steps. 17 | 18 | 19 | ## Installation 20 | 21 | Clone this repository. You will only need the file `simplecd.sh` in order 22 | to start deliveries. 23 | 24 | 25 | ## Preparing your application 26 | 27 | SimpleCD depends on some special files being present in your application's 28 | Git repository, the so-called SimpleCD step scripts - one for each step of a 29 | continuous delivery run. 30 | 31 | You can create up to 100 step scripts, each with a unique leading number between 32 | 00 and 99 followed by a dash (-). 33 | 34 | Here are some examples: 35 | 36 | * `00-run-unit-tests` 37 | * `10-deploy-to-staging` 38 | * `20-run-migrations-on-staging` 39 | * `30-run-e2e-tests-for-staging` 40 | 41 | and so on. Hint: If you start by numbering your initial step scripts with 42 | 00, 10, 20... instead of 00, 01, 02..., then later it's much easier to add 43 | new steps between existing steps. 44 | 45 | These files must be placed in a subfolder of your project's repository named 46 | `_simplecd`, and they must be set to executable. 47 | 48 | SimpleCD will try to execute each step by executing these scripts in the order 49 | shown above. If executing a script results in a status code > 0, then the 50 | delivery is aborted. If this is the case, and a special script 51 | `on-project-script-error` exists in the `_simplecd_` folder, then this script 52 | is executed before SimpleCD aborts delivery for good. 53 | 54 | Additionally, you can add a file `_simplecd/logreceivers.txt` with one mail 55 | address per line. If the file is present, a report of the run will be sent to 56 | the listed mail addresses. 57 | 58 | 59 | ## Usage 60 | 61 | `simplecd.sh [reset|] [--tag-on-success]` 62 | 63 | **Examples:** 64 | 65 | Monitor branch "foo" for new commits: 66 | 67 | `simplecd.sh branch foo https://github.com/johndoe/example.git` 68 | 69 | Monitor repo for new tags matching *release-** pattern: 70 | 71 | `simplecd.sh tag release-* https://github.com/johndoe/example.git` 72 | 73 | 74 | The following steps are executed: 75 | 76 | 1. Check if an instance of the given plan is already running, exit if yes 77 | 3. Pull the newest code from the remote repository 78 | 2. Check if the repo is newer than what was last delivered (new commit in branch or new matching tag) 79 | 4. Run the step scripts that are provided by the repository in subfolder 80 | `_simplecd`. 81 | 5. Mail results to the receivers listed in `_simplecd/logreceivers.txt` 82 | 83 | SimpleCD will call every script with the mode as the first, the path to the local repository clone 84 | as the second, and the name of the branch or matched tag as the third parameter, like this: 85 | 86 | `./_simplecd/00-run-unit-tests branch /var/tmp/simplecd/projects/e70081c0e267ac64454c27f5e600d214 master` 87 | 88 | `./_simplecd/00-run-unit-tests tag /var/tmp/simplecd/projects/e70081c0e267ac64454c27f5e600d214 release-1.0.3` 89 | 90 | If the keyword *reset* is provided as the fourth parameter, SimpleCD does not 91 | start a delivery, but instead removes all working data related to the given 92 | mode/repo/source combination, that is, SimpleCD resets its environment to a state 93 | as if no previous runs for this mode/repo/source had occurred. 94 | 95 | If instead an HTTP URL is provided as the fourth parameter, SimpleCD will 96 | prefix any commit id it outputs with this URL. 97 | 98 | If `--tag-on-success` is provided as the fourth or fifth parameter, then 99 | SimpleCD will annotate the rolled out commit with tag name 100 | `simplecd-rollout-`. 101 | 102 | Note that because colons are not allowed in git tag names, these are replaced 103 | with an underscore. The result looks like this: 104 | `simplecd-rollout-2019-04-13T10_50_43+00_00`. 105 | 106 | 107 | ## License 108 | 109 | The MIT License (MIT) 110 | 111 | Copyright (c) 2013 Manuel Kiessling, MeinAuto GmbH 112 | 113 | Permission is hereby granted, free of charge, to any person obtaining a copy of 114 | this software and associated documentation files (the "Software"), to deal in 115 | the Software without restriction, including without limitation the rights to 116 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 117 | the Software, and to permit persons to whom the Software is furnished to do so, 118 | subject to the following conditions: 119 | 120 | The above copyright notice and this permission notice shall be included in all 121 | copies or substantial portions of the Software. 122 | 123 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 124 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 125 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 126 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 127 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 128 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 129 | -------------------------------------------------------------------------------- /simplecd.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Main script for Simple Continuous Delivery 4 | 5 | PATH=$PATH:/bin:/usr/bin:/usr/sbin:/usr/local/bin 6 | 7 | 8 | # Functions ################################################################# 9 | 10 | shutdown () { 11 | echo $1 12 | echo "" 13 | prepend_to_maillog "Result: success." 14 | send_maillog "success" 15 | rm -f $CONTROLFILE 16 | exit 0 17 | } 18 | 19 | abort () { 20 | echo $1 21 | echo "" 22 | prepend_to_maillog "$1" 23 | prepend_to_maillog "Result: failure." 24 | send_maillog "failure" 25 | rm -f $CONTROLFILE 26 | exit 1 27 | } 28 | 29 | 30 | prepend_to_maillog () { 31 | MAILLOG="$1 32 | 33 | $MAILLOG" 34 | } 35 | 36 | append_to_maillog () { 37 | echo "$1" >> $LOGFILEFORWEBMONITOR 38 | MAILLOG="$MAILLOG 39 | 40 | $1" 41 | } 42 | 43 | send_maillog () { 44 | if [ -f $REPODIR/$SCRIPTSDIR/logreceivers.txt ]; then 45 | echo "$SUMMARY" > $WORKINGDIR/maillog.$HASH 46 | echo "" >> $WORKINGDIR/maillog.$HASH 47 | echo "" >> $WORKINGDIR/maillog.$HASH 48 | echo "Last 100 lines of log output" >> $WORKINGDIR/maillog.$HASH 49 | echo "############################" >> $WORKINGDIR/maillog.$HASH 50 | echo "" >> $WORKINGDIR/maillog.$HASH 51 | echo "$MAILLOG" | tail -100 >> $WORKINGDIR/maillog.$HASH 52 | echo "" >> $WORKINGDIR/maillog.$HASH 53 | echo "" >> $WORKINGDIR/maillog.$HASH 54 | echo "Full log output" >> $WORKINGDIR/maillog.$HASH 55 | echo "###############" >> $WORKINGDIR/maillog.$HASH 56 | echo "" >> $WORKINGDIR/maillog.$HASH 57 | echo "$MAILLOG" >> $WORKINGDIR/maillog.$HASH 58 | while read MAILRECEIVER; do 59 | echo "Mailing log of this run to $MAILRECEIVER..." 60 | mail -aFrom:`whoami`@`hostname --fqdn` -s "[simplecd][$1] $REPO - $SOURCE" $MAILRECEIVER < $WORKINGDIR/maillog.$HASH 61 | done < $REPODIR/$SCRIPTSDIR/logreceivers.txt 62 | fi 63 | } 64 | 65 | run_project_script () { 66 | STATUS=0 67 | echo "Starting project's $1 script..." 68 | append_to_maillog "" 69 | append_to_maillog "Output of project's $1 script: 70 | #######################################" 71 | echo "" 72 | $REPODIR/$SCRIPTSDIR/$1 $MODE $REPODIR $CHECKOUTSOURCE > >(tee -a $WORKINGDIR/$HASH.script.$1.log) 2> >(tee -a $WORKINGDIR/$HASH.script.$1.log >&2) > >(tee -a $LOGFILEFORWEBMONITOR) 2>(tee -a $LOGFILEFORWEBMONITOR) 73 | STATUS=$? 74 | OUTPUT=`cat $WORKINGDIR/$HASH.script.$1.log` 75 | rm $WORKINGDIR/$HASH.script.$1.log 76 | append_to_maillog "$OUTPUT" 77 | echo "" 78 | echo "Finished executing project's $1 script." 79 | 80 | if [ ! $STATUS -eq 0 ]; then 81 | if [ -x "$REPODIR/$SCRIPTSDIR/on-project-script-error" ]; then 82 | echo "Error while executing project's $1 script. Executing on-project-script-error script..." 83 | append_to_maillog "Error while executing project's $1 script. Executing on-project-script-error script..." 84 | $REPODIR/$SCRIPTSDIR/on-project-script-error > >(tee -a $WORKINGDIR/$HASH.script.on-project-script-error.log) 2> >(tee -a $WORKINGDIR/$HASH.script.on-project-script-error.log >&2) 85 | OUTPUT=`cat $WORKINGDIR/$HASH.script.on-project-script-error.log` 86 | rm $WORKINGDIR/$HASH.script.on-project-script-error.log 87 | append_to_maillog "$OUTPUT" 88 | echo "" 89 | echo "Finished executing project's on-project-script-error script." 90 | fi 91 | abort "Error while executing project's $1 script. Aborting..." 92 | fi 93 | 94 | echo "" 95 | } 96 | 97 | # Sanity checks ############################################################# 98 | 99 | # Verify that we have git 100 | 101 | git --version > /dev/null 2>&1 102 | if [ ! $? -eq 0 ]; then 103 | abort "Git not available. Aborting..." 104 | fi 105 | 106 | # Verify that we have a modern version of sort 107 | 108 | echo "foo" | sort --version-sort > /dev/null 2>&1 109 | if [ ! $? -eq 0 ]; then 110 | abort "Your version of sort does not support --version-sort. Aborting..." 111 | fi 112 | 113 | 114 | # Main routine ################################################################# 115 | 116 | MODE=$1 # "branch" or "tag" 117 | SOURCE=$2 118 | REPO=$3 119 | 120 | 121 | 122 | 123 | if [ "$4" = "reset" ]; then 124 | DORESET="yes" 125 | elif [ "$4" = "--tag-on-success" ]; then 126 | TAGONSUCCESS="yes" 127 | DORESET="no" 128 | elif [ "$4" != "" ]; then 129 | URLPREFIX=$4 130 | DORESET="no" 131 | fi 132 | 133 | if [ "$5" = "--tag-on-success" ]; then 134 | TAGONSUCCESS="yes" 135 | fi 136 | 137 | 138 | if [ "$MODE" = "" ]; then 139 | abort "Missing parameter MODE. Aborting..." 140 | fi 141 | 142 | if [ "$REPO" = "" ]; then 143 | abort "Missing parameter REPO. Aborting..." 144 | fi 145 | 146 | if [ "$SOURCE" = "" ]; then 147 | abort "Missing parameter SOURCE. Aborting..." 148 | fi 149 | 150 | 151 | if [ -x /sbin/md5 ]; then 152 | MD5BIN=/sbin/md5 153 | else 154 | MD5BIN=/usr/bin/md5sum 155 | fi 156 | 157 | WORKINGDIR=/var/tmp/simplecd 158 | SCRIPT_SRC_DIR=$(dirname "$0") 159 | HASH=`echo "$0 $MODE $REPO $SOURCE" | $MD5BIN | cut -d" " -f1` 160 | PROJECTSDIR=$WORKINGDIR/projects 161 | REPODIR=$PROJECTSDIR/$HASH 162 | CONTROLFILE=$WORKINGDIR/controlfile.$HASH 163 | LASTCOMMITIDFILE=$WORKINGDIR/last_commit_id.$HASH 164 | LASTTAGFILE=$WORKINGDIR/last_tag.$HASH 165 | LOGFILEFORWEBMONITOR=$WORKINGDIR/logforwebmonitor.$HASH.$(date +%Y-%m-%d_%H-%M) 166 | 167 | # Did the user provide the parameter "reset"? In this case 168 | # we remove everything we know about the given repo/branch combination 169 | 170 | if [ "$DORESET" = "yes" ]; then 171 | echo "Resetting SimpleCD environment for mode $MODE, repo $REPO, source $SOURCE" 172 | if [ "$MODE" = "branch" ]; then 173 | rm -f $WORKINGDIR/last_commit_id.$HASH 174 | fi 175 | if [ "$MODE" = "tag" ]; then 176 | rm -f $WORKINGDIR/last_tag.$HASH 177 | fi 178 | rm -f $WORKINGDIR/maillog.$HASH 179 | rm -rf $REPODIR 180 | rm -f $CONTROLFILE 181 | echo "done." 182 | exit 0 183 | fi 184 | 185 | # Is another process for this mode, repo and source running? 186 | 187 | if [ -f $CONTROLFILE ]; then 188 | echo "Because the control file $CONTROLFILE exists, I assume that another instance is still running. Aborting..." 189 | exit 1 190 | fi 191 | 192 | # Prepare and check the environment 193 | 194 | mkdir -p $PROJECTSDIR 195 | 196 | if [ ! -w $PROJECTSDIR ]; then 197 | abort "Cannot write to directory $PROJECTSDIR. Aborting..." 198 | fi 199 | 200 | # Create control file so no other runs are started in parallel 201 | 202 | touch $CONTROLFILE 203 | 204 | # Let's go 205 | echo "" 206 | echo "Starting delivery of source $SOURCE from repo $REPO in mode $MODE, hash of this run is $HASH" 207 | echo "" 208 | append_to_maillog "Log for delivery of source $SOURCE from repo $REPO in mode $MODE, hash of this run was $HASH" 209 | 210 | 211 | # Resolve source and check for new content 212 | 213 | if [ "$MODE" = "branch" ]; then 214 | RESOLVEDSOURCE=refs/heads/$SOURCE 215 | # Check if a new commit id is in the remote repo 216 | LASTCOMMITID=`cat $LASTCOMMITIDFILE 2> /dev/null` 217 | REMOTECOMMITID=`git ls-remote $REPO $RESOLVEDSOURCE | cut -f1` 218 | if [ "$LASTCOMMITID" = "$REMOTECOMMITID" ]; then 219 | echo "Remote commit id ($REMOTECOMMITID) has not changed since last run, won't deliver. Aborting..." 220 | rm -f $CONTROLFILE 221 | exit 0 222 | fi 223 | if [ "" = "$REMOTECOMMITID" ]; then 224 | echo "Couldn't retrieve remote commit id, won't deliver. Aborting..." 225 | rm -f $CONTROLFILE 226 | exit 0 227 | fi 228 | append_to_maillog "Local known last commit id was $LASTCOMMITID, found $REMOTECOMMITID remotely." 229 | rm -rf $REPODIR 230 | git clone $REPO $REPODIR 2>&1 | while IFS= read -r line;do echo " [GIT CLONE] $line";done 231 | cd $REPODIR 232 | git fetch 233 | CURRENTCOMMITID=$REMOTECOMMITID 234 | echo $CURRENTCOMMITID > $LASTCOMMITIDFILE 235 | CHECKOUTSOURCE=$SOURCE 236 | fi 237 | 238 | if [ "$MODE" = "tag" ]; then 239 | # Check if a new tag matching the pattern exists 240 | LASTTAG=`cat $LASTTAGFILE 2> /dev/null` 241 | LASTEXISTINGTAG=`git ls-remote --tags $REPO $SOURCE | cut -f2 | sort --version-sort | cut -d/ -f3 | tail -n1` 242 | if [ "$LASTTAG" = "$LASTEXISTINGTAG" ]; then 243 | echo "No tag newer than '$LASTTAG' found, won't deliver. Aborting..." 244 | rm -f $CONTROLFILE 245 | rm -f $LOGFILEFORWEBMONITOR 246 | exit 0 247 | fi 248 | if [ "" = "$LASTEXISTINGTAG" ]; then 249 | echo "Couldn't retrieve remote tag, won't deliver. Aborting..." 250 | rm -f $CONTROLFILE 251 | rm -f $LOGFILEFORWEBMONITOR 252 | exit 0 253 | fi 254 | append_to_maillog "Local known last tag was $LASTTAG, found $LASTEXISTINGTAG remotely." 255 | rm -rf $REPODIR 256 | git clone $REPO $REPODIR 2>&1 | while IFS= read -r line;do echo " [GIT CLONE] $line";done 257 | cd $REPODIR 258 | git fetch 259 | CURRENTCOMMITID=$LASTEXISTINGTAG 260 | echo $LASTEXISTINGTAG > $LASTTAGFILE 261 | RESOLVEDSOURCE=refs/tags/$LASTEXISTINGTAG 262 | CHECKOUTSOURCE=$LASTEXISTINGTAG 263 | fi 264 | 265 | # Make log accessible via browser 266 | $SCRIPT_SRC_DIR/webmonitor.sh $LOGFILEFORWEBMONITOR $3 & 267 | 268 | # Checkout the source 269 | 270 | git checkout $CHECKOUTSOURCE 2>&1 | while IFS= read -r line;do echo " [GIT CHECKOUT] $line";done 271 | 272 | 273 | # Create summary 274 | 275 | echo "" 276 | echo "This is what's going to be delivered:" 277 | SUMMARY=" 278 | Repository: $REPO 279 | Source: $MODE $RESOLVEDSOURCE 280 | Commit: $URLPREFIX$CURRENTCOMMITID 281 | by: `git log -n 1 $CURRENTCOMMITID --pretty=format:'%an'` 282 | at: `git log -n 1 $CURRENTCOMMITID --pretty=format:'%aD'` 283 | msg: `git log -n 1 $CURRENTCOMMITID --pretty=format:'%s'`" 284 | echo "$SUMMARY" 285 | echo "" 286 | 287 | 288 | SCRIPTSDIR=_simplecd 289 | source $REPODIR/.simplecd 290 | 291 | 292 | ## Run delivery steps from repository ########################################## 293 | 294 | for STEPFILENAME in `ls $REPODIR/$SCRIPTSDIR/[0-9][0-9]-* | sort | rev | cut -d"/" -f1 | rev`; do 295 | run_project_script $STEPFILENAME 296 | done 297 | 298 | 299 | # Tag the rolled out commit 300 | 301 | if [ "$TAGONSUCCESS" = "yes" ]; then 302 | TAGNAME="simplecd-rollout-`date --iso-8601=seconds | tr : _`" 303 | TAGMESSAGE="SimpleCD rollout on `date --rfc-2822`." 304 | git tag -a $TAGNAME -m "$TAGMESSAGE" $CURRENTCOMMITID 305 | git push origin $TAGNAME 306 | fi 307 | 308 | 309 | # Clean up the control file and finish 310 | 311 | echo "" 312 | 313 | shutdown "Delivery finished. Exiting..." 314 | -------------------------------------------------------------------------------- /webmonitor.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # this script reads a logfile from simplecd and convert it, so it can be accessed via a html page 3 | # 1. creates a html page 4 | # 2. converts logfile to html (just puts
after every line; will be extended soon) 5 | # 3. uses ajax via javascript to get the converted html-logfile 6 | 7 | [[ -w /var/www ]] || echo "Directory /var/www does not exist, aborting." && exit 1 8 | 9 | NOW=$(date '+%d.%m.%Y_%H:%M:%S') 10 | 11 | #Input variables 12 | LOGFILE=$1 13 | REPO=$(echo $2 | cut -d '/' -f 2 | cut -d '.' -f 1) 14 | 15 | # check if input variables are not null 16 | if [[ $LOGFILE == "" ]] 17 | then 18 | echo "No Logfile handed over. Aborting..." 19 | exit 1 20 | fi 21 | 22 | OUTPUT_DIRECTORY_NAME="${REPO}_${NOW}" 23 | OUTPUT_DIRECTORY_PATH=/var/www/simplecd/"${OUTPUT_DIRECTORY_NAME}" 24 | LOG_EXISTS=false 25 | PAGE_TITLE="Build $REPO on $(date '+%d.%m.%Y %H:%M:%S')" 26 | CURDIR=$(dirname "$0") 27 | 28 | echo $PAGE_TITLE 29 | 30 | # wait until Logfile exists 31 | # abort if it's not available after one minute 32 | let w=0 33 | until $LOG_EXISTS || [[ $w -gt 11 ]] 34 | do 35 | [[ -f $LOGFILE ]] && LOG_EXISTS=true || sleep 5 36 | let w=$w+1 37 | done 38 | 39 | if [[ $w > 11 ]] 40 | then 41 | echo "Logfile does not exist. Aborting..." > $LOGFILE 42 | exit 1 43 | fi 44 | 45 | # create subdirectory 46 | [[ -d $OUTPUT_DIRECTORY_PATH ]] || mkdir -p $OUTPUT_DIRECTORY_PATH 47 | 48 | 49 | 50 | # create html-file 51 | cat > $OUTPUT_DIRECTORY_PATH/index.html \ 52 | <<- _EOF_ 53 | 54 | 55 | 56 | SimpleCD Monitor 57 | 58 | 63 | 64 | 65 | 85 | 86 |
87 |

$PAGE_TITLE

88 |
89 |
90 |
91 | 92 | 93 | _EOF_ 94 | 95 | # If there is a script for notifying about the new rollout and its web output, trigger it. 96 | 97 | [[ -e $CURDIR/notification.sh ]] && $CURDIR/notification.sh $REPO $OUTPUT_DIRECTORY_NAME 98 | 99 | LASTLINE=null 100 | 101 | # read Logfile line by line 102 | # abort after one minute if no one's writing anymore 103 | let i=0 104 | let t=0 105 | while [[ t -lt 60 ]]; do 106 | let i=$i+1 107 | CURRENTLINE=$(head -n $i $LOGFILE | tail -n 1) 108 | 109 | if [[ "$LASTLINE" != $CURRENTLINE ]] 110 | then 111 | let t=0 112 | echo $CURRENTLINE "
" >> $OUTPUT_DIRECTORY_PATH/htmllog.log 113 | LASTLINE=$CURRENTLINE 114 | sleep 0.5 115 | else 116 | sleep 5 117 | let t=$t+1 118 | fi 119 | done 120 | 121 | rm -f $LOGFILE 122 | --------------------------------------------------------------------------------