├── .gitignore ├── .projmodules ├── README.md ├── ansible.cfg ├── bootstrap.sh ├── bootstrap_mac.sh ├── components ├── files │ └── ssh │ │ └── vyacheslav.pub └── nginx │ └── dhparam.pem ├── docs ├── enable_security.png └── plugins-manager.png ├── hosts.template ├── hostsbootstrap.template ├── init.sh ├── jenkins.yml ├── jenkins_bootstrap.yml ├── jenkins_vars.yml ├── molecule ├── Makefile ├── Pipfile ├── Pipfile.lock ├── Readme.md ├── default │ ├── INSTALL.rst │ ├── converge.yml │ ├── molecule.yml │ ├── playbook_vars.yml │ ├── prepare.yml │ ├── tests │ │ └── test_default.py │ └── yaml-lint.yml ├── requirements-dev.txt └── update_deps.sh ├── public └── .gitkeep ├── roles └── .gitkeep ├── setup_jenkins.sh └── setup_jenkins_only.sh /.gitignore: -------------------------------------------------------------------------------- 1 | .venv 2 | hosts 3 | hostsbootstrap 4 | public 5 | roles 6 | -------------------------------------------------------------------------------- /.projmodules: -------------------------------------------------------------------------------- 1 | [submodule "roles/sa-vnc-remote-desktop"] 2 | path = roles/sa-vnc-remote-desktop 3 | url = https://github.com/softasap/sa-vnc-remote-desktop.git 4 | [submodule "roles/sa-box-bootstrap"] 5 | path = roles/sa-box-bootstrap 6 | url = git@github.com:softasap/sa-box-bootstrap.git 7 | [submodule "roles/sa-box-jenkins"] 8 | path = roles/sa-box-jenkins 9 | url = git@github.com:softasap/sa-box-jenkins.git 10 | 11 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Dive into continious integration with Jenkins. 2 | ============================================= 3 | 4 | ## Background 5 | Nowadays continious integration is the important part of the agile software development life cycle. 6 | There is a number of tools on the market: Atlassian Bamboo, Jenkins, Jetbrains TeamCity. 7 | In my opinion Jenkins has the most optimal product community and set of really useful plugins 8 | that suits most of your software projects: you can build software, deploy software, 9 | websites, portals to various places, including AWS, DigitalOcean, bare metal servers 10 | or to run unit tests. It can be integrated with communication tools of your choice, like Slack, HipChat or 11 | email. 12 | 13 | If you haven't had a chance to try Jenkins earlier, feel free to use tutorial below to start. 14 | 15 | ## Manual installation 16 | In order to install Jenkins, we will need: 17 | - Unix system. I would recommend debian based, like ubuntu server LTS 18 | - Java runtime environment installed. I usually use Java 8 19 | - Get base Jenkins setup 20 | - Install necessary plugins 21 | - Put behind web server. 22 | 23 | ### Install Java 24 | Easist way to install Java, is using apt-get package manager 25 |
26 | sudo apt-get install python-software-properties 27 | sudo add-apt-repository ppa:webupd8team/java 28 | sudo apt-get update 29 |30 | Once you added ppa above, you can install java with the following command: 31 |
32 | sudo apt-get install oracle-java8-installer 33 |34 | 35 | ### Get base Jenkins setup 36 | You will need to execut series of the commands, namely: add jenkins signing key, 37 | register jenkins apt sources, update package lists, and install Jenkins package. 38 |
39 | wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add - 40 | sudo echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list 41 | sudo apt-get update 42 | sudo apt-get install jenkins 43 |44 | 45 | By default, it will install base Jenkins setup, which is insecure. You will 46 | need to go to the host were your Jenkins is installed, for example: http://jenkins-host:8080/. 47 | Navigate to Manage Jenkins (on the left) and choose "Configure Global Security" item 48 | on the page loaded. 49 |  50 | 51 | Now look below on the Matrix based security (select it, if it is not selected previously), 52 | and make sure Anonymous only has the Read right under the View group. 53 | Click save at the bottom of the page. After the page load, you'll see a login form, simply ignore that, 54 | go to the home page (like, for example, http://jenkins-host:8080/). You'll see this sign up form, 55 | the first signed up account will be the administrator. 56 | 57 | ### Power of plugins. 58 | Jenkins would not be so powerful without plugins. Usually I install next plugins by default: 59 | 60 | - bitbucket 61 | BitBucket plugin is designed to offer integration between BitBucket and Jenkins. BitBucket offer a Jenkins hook, but this one just trigger a build for a specific job on commit, nothing more. BitBucket plugin, like GitHub plugin already did, use the POST hook payload to check which job has to get triggered based on changed repository/branch. 62 | Plugin page: https://wiki.jenkins-ci.org/display/JENKINS/BitBucket+Plugin 63 | 64 | - bitbucket-pullrequest-builder 65 | This plugin builds pull requests from Bitbucket.org. Must have plugin, if you perfor QA deploy of each pull request submitted. 66 | Plugin page: https://wiki.jenkins-ci.org/display/JENKINS/Bitbucket+pullrequest+builder+plugin 67 | 68 | - build-pipeline-plugin 69 | This plugin provides a Build Pipeline View of upstream and downstream connected jobs that typically form a build pipeline. In addition, it offers the ability to define manual triggers for jobs that require intervention prior to execution, e.g. an approval process outside of Jenkins. Provides nice visualization of the pathes & flows. 70 | Plugin page: https://wiki.jenkins-ci.org/display/JENKINS/Build+Pipeline+Plugin 71 | 72 | - copyartifact 73 | Adds a build step to copy artifacts from another project. The plugin lets you specify which build to copy artifacts from (e.g. the last successful/stable build, by build number, or by a build parameter). You can also control the copying process by filtering the files being copied, specifying a destination directory within the target project, etc 74 | Plugin page: https://wiki.jenkins-ci.org/display/JENKINS/Copy+Artifact+Plugin 75 | - credentials 76 | Adds a build step to copy artifacts from another project. The plugin lets you specify which build to copy artifacts from (e.g. the last successful/stable build, by build number, or by a build parameter). You can also control the copying process by filtering the files being copied, specifying a destination directory within the target project, etc 77 | Plugin page: https://wiki.jenkins-ci.org/display/JENKINS/Credentials+Plugin 78 | - delivery-pipeline-plugin 79 | Visualisation of Delivery/Build Pipelines, renders pipelines based on upstream/downstream jobs. When using Jenkins as a build server it is now possible with the Delivery Pipeline Plugin to visualise one or more Delivery Pipelines in the same view even in full screen 80 | Plugin page: https://wiki.jenkins-ci.org/display/JENKINS/Delivery+Pipeline+Plugin 81 | - environment-script 82 | Environment Script Plugin allows you to have a script run after SCM checkout, before the build. If the script fails (exit code isn't zero), the build is marked as failed. 83 | Any output on standard out is parsed as environment variables that are applied to the build. It supports "override syntax" to append paths to PATH-like variables 84 | Plugin page: https://wiki.jenkins-ci.org/display/JENKINS/Environment+Script+Plugin 85 | - git 86 | Supports popular git version control system 87 | - ghprb 88 | This plugin builds pull requests in github. Must have, if your software development life cycle includes deploying pull requests 89 | to PR environment to test. 90 | Plugin page: https://wiki.jenkins-ci.org/display/JENKINS/GitHub+pull+request+builder+plugin 91 | - greenballs The most funny plugin - changes Jenkins to use green balls instead of blue for successful builds 92 | Plugin page: https://wiki.jenkins-ci.org/display/JENKINS/Green+Balls 93 | 94 | - hipchat 95 | This plugin allows your team to setup build notifications to be sent to HipChat rooms.To enable notifications add "HipChat Notifications" as a post-build step. 96 | Plugin page: https://wiki.jenkins-ci.org/display/JENKINS/HipChat+Plugin 97 | - junit 98 | Allows JUnit-format test results to be published. Note: number of tools, including Karma, PhpUNIT & other tools allow to publish test results in a JUnit format. Thus this is must have plugin for unit test flows. 99 | Plugin page: https://wiki.jenkins-ci.org/display/JENKINS/JUnit+Plugin 100 | - matrix-auth 101 | Offers matrix-based security authorization strategies (global and per-project). Good, if you have shared build server across several teams, 102 | Plugin page: https://wiki.jenkins-ci.org/display/JENKINS/Matrix+Authorization+Strategy+Plugin 103 | - parameterized-trigger 104 | This plugin lets you trigger new builds when your build has completed, with various ways of specifying parameters for the new build. 105 | You can add multiple configurations: each has a list of projects to trigger, a condition for when to trigger them (based on the result of the current build), and a parameters section. 106 | Plugin page: https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin 107 | - rebuild 108 | Plays nice the with previous one: this plug-in allows the user to rebuild a parametrized build without entering the parameters again. 109 | Plugin page: https://wiki.jenkins-ci.org/display/JENKINS/Rebuild+Plugin 110 | - ssh You can use the SSH Plugin to run shell commands on a remote machine via ssh 111 | Plugin page: https://wiki.jenkins-ci.org/display/JENKINS/SSH+plugin 112 | - s3 allows uploading artifacts to S3 with multiple options. 113 | Plugin page: https://wiki.jenkins-ci.org/display/JENKINS/S3+Plugin 114 | - throttle-concurrents 115 | This plugin allows for throttling the number of concurrent builds of a project running per node or globally. 116 | Unfortunately, this is must have plugin for Node (0.10-0.12) projects with NPM - two concurrent npm install will fail often. 117 | Plugin page: https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin 118 | 119 | Plugins are installed using Plugin manager on a Manage Jenkins Section. 120 |  121 | 122 | 123 | ### Put behind web server 124 | Usually I hide Jenkins behind nginx. Typical configuration looks like the one below 125 |
126 | server { 127 | listen 443 ssl; 128 | server_name jenkins.vagrant.dev; 129 | 130 | ssl_certificate /etc/nginx/jenkins_selfsigned.crt; 131 | ssl_certificate_key /etc/nginx/jenkins_selfsigned.key; 132 | 133 | location / { 134 | proxy_pass http://127.0.0.1:8080; 135 | proxy_set_header Host $host; 136 | proxy_set_header X-Real-IP $remote_addr; 137 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 138 | proxy_redirect off; 139 | 140 | proxy_connect_timeout 150; 141 | proxy_send_timeout 100; 142 | proxy_read_timeout 100; 143 | } 144 | ... 145 | } 146 |147 | 148 | ## Automated installation 149 | Do I install Jenkins manually each time? Of course not, I do it often for my customers. 150 | With ansible, and sa-box-jenkins role new Jenkins installation can be deployed while you drink the coffee. 151 | 152 | Let's prepare basic bootstrap project, that can be used by you in the future. 153 | It includes following files: 154 | 155 | - *bootstrap.sh* - installs ansible alongside with dependences. 156 | - *init.sh* - initializes 3rd party dependencies 157 | - *.projmodules* - fully compatible with .gitmodules git syntax, specifies list of the dependencies 158 | that will be used by the playbook. 159 | In particular, it includes ansible- by default developer_recipes (repository with set of handy deployment recipes) 160 | and ansible role called *sa-box-bootstrap* responsible for box securing steps (assuming you plan to put Jenkins on a remote hosts). 161 | 162 |
163 | [submodule "public/ansible_developer_recipes"] 164 | path = public/ansible_developer_recipes 165 | url = git@github.com:Voronenko/ansible-developer_recipes.git 166 | [submodule "roles/sa-box-bootstrap"] 167 | path = roles/sa-box-bootstrap 168 | url = git@github.com:softasap/sa-box-bootstrap.git 169 | [submodule "roles/sa-box-jenkins"] 170 | path = roles/sa-box-jenkins 171 | url = git@github.com:softasap/sa-box-jenkins.git172 | - *hosts* - list here the initial box credentials, that were provided to you for the server. Note: jenkins-bootstrap assumes, you have the fresh box with the root access only. If your box already secured, adjust credentials appropriately 173 |
174 | [jenkins-bootstrap] 175 | jenkins_bootstrap ansible_ssh_host=192.168.0.17 ansible_ssh_user=yourrootuser ansible_ssh_pass=yourpassword 176 | [jenkins] 177 | jenkins ansible_ssh_host=192.168.0.17 ansible_ssh_user=jenkins 178 |179 | - *jenkins_vars.yml* - set here specific environment overrides, like your preferred deploy user name and keys. 180 | - *jenkins_bootstrap.yml* - First step - box securing. Creates jenkins user, and secures the box using sa-box-bootstrap role. 181 | [See more details](https://github.com/softasap/sa-box-bootstrap) about the sa-box-bootstrap role 182 | In order, to override params for *sa-box-bootstrap* - pass the parameters like in example below. 183 | 184 |
185 | - hosts: all 186 | 187 | vars_files: 188 | - ./jenkins_vars.yml 189 | roles: 190 | - { 191 | role: "sa-box-bootstrap", 192 | root_dir: "{{playbook_dir}}/public/ansible_developer_recipes", 193 | deploy_user: "{{jenkins_user}}", 194 | deploy_user_keys: "{{jenkins_authorized_keys}}" 195 | }196 | - *jenkins.yml* provisioning script that configures jenkins with set of plugins and users. 197 | - *jenkins_vars.yml* configuration options for jenkins deployment. 198 | - *setup_jenkins.sh* shell script that invokes deployment in two steps: initial box bootstraping & jenkins setup 199 |
200 | #!/bin/sh 201 | 202 | ansible-playbook jenkins_bootstrap.yml --limit jenkins_bootstrap 203 | ansible-playbook jenkins.yml --limit jenkins 204 |205 | 206 | 207 | ## Configuration options for automated installation 208 | You need to override: 209 | - jenkins_authorized_keys (this is list of the keys, that allow you to login to Jenkins box under jenkins) 210 | - jenkins_domain - your agency domain 211 | - jenkins_host - name of the jenkins host (Site will be binded to jenkins_host.jenkins_domain) 212 | - java_version - your Java choice (6,7,8 supported) 213 | 214 |
215 | jenkins_user: jenkins 216 | jenkins_authorized_keys: 217 | - "{{playbook_dir}}/components/files/ssh/vyacheslav.pub" 218 | 219 | jenkins_domain: "vagrant.dev" 220 | jenkins_host: "jenkins" 221 | 222 | java_version: 8 223 |224 | 225 | -jenkins_users list of users with passwords to create. Admin and deploy are required users. 226 | Admin is used to manage instance, deploy is used to access the artifacts via deployment scripts. 227 | If you won't override passwords, default one will be used (per role), which is not the best, for public deployments. 228 |
229 | jenkins_users: 230 | - { 231 | name: "Admin", 232 | password: "AAAdmin", 233 | email: "no-reply@localhost" 234 | } 235 | - { 236 | name: "deploy", 237 | password: "DeDeDeDeploy", 238 | email: "no-reply@localhost" 239 | } 240 |241 | 242 | - jenkins_plugins Your choice of plugins to install. By default: 243 | 244 |
245 | jenkins_plugins: 246 | - bitbucket # https://wiki.jenkins-ci.org/display/JENKINS/BitBucket+Plugin 247 | - bitbucket-pullrequest-builder 248 | - build-pipeline-plugin 249 | - copyartifact # https://wiki.jenkins-ci.org/display/JENKINS/Copy+Artifact+Plugin 250 | - credentials # https://wiki.jenkins-ci.org/display/JENKINS/Credentials+Plugin 251 | - delivery-pipeline-plugin # https://wiki.jenkins-ci.org/display/JENKINS/Delivery+Pipeline+Plugin 252 | - environment-script # https://wiki.jenkins-ci.org/display/JENKINS/Environment+Script+Plugin 253 | - git 254 | - ghprb # https://wiki.jenkins-ci.org/display/JENKINS/GitHub+pull+request+builder+plugin 255 | - greenballs # https://wiki.jenkins-ci.org/display/JENKINS/Green+Balls 256 | - hipchat # https://wiki.jenkins-ci.org/display/JENKINS/HipChat+Plugin 257 | - junit # https://wiki.jenkins-ci.org/display/JENKINS/JUnit+Plugin 258 | - matrix-auth # https://wiki.jenkins-ci.org/display/JENKINS/Matrix+Authorization+Strategy+Plugin 259 | - matrix-project #https://wiki.jenkins-ci.org/display/JENKINS/Matrix+Project+Plugin 260 | - parameterized-trigger #https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin 261 | - rebuild # https://wiki.jenkins-ci.org/display/JENKINS/Rebuild+Plugin 262 | - ssh 263 | - s3 # https://wiki.jenkins-ci.org/display/JENKINS/S3+Plugin 264 | - throttle-concurrents #https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin 265 |266 | 267 | ## Code in action 268 | 269 | Code can be downloaded from repository [https://github.com/Voronenko/devops-jenkins-box-template](https://github.com/Voronenko/devops-jenkins-box-template) 270 | In order to use it - fork it, adjust parameters to your needs, and use. 271 | 272 | Running is as simple as 273 |
274 | ./setup_jenkins.sh 275 |276 | 277 | Welcome to the world of continious integration & deployment. 278 | -------------------------------------------------------------------------------- /ansible.cfg: -------------------------------------------------------------------------------- 1 | [defaults] 2 | hostfile = hosts 3 | roles_path = ./roles 4 | -------------------------------------------------------------------------------- /bootstrap.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | sudo apt-get install python-pip 4 | sudo pip install -U pip 5 | sudo -H pip install ansible 6 | -------------------------------------------------------------------------------- /bootstrap_mac.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | 4 | brew install git 5 | brew install python 6 | pip install -U pip 7 | sudo -H pip install ansible 8 | -------------------------------------------------------------------------------- /components/files/ssh/vyacheslav.pub: -------------------------------------------------------------------------------- 1 | ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzT2CJoS/GjsHxa4cWxoaVKkvGL+angx2jYlR62t4/pHZ8JNS2Q+Ptb+YL5UHiwOV74sOUn0PrKDDGoc+BSUTHX6E28Vz1YfRUrL6lLJ/JRg3ZIARXSuOdF87/FakGc83wi3YV7oFb7EtQObrDmIj01XPLATaGsfeK/0sywFgAmIDnIUWVn/asc+ijON0VCmbiXkcbb7/S+MIIOr08FtpJ6u8bJVwGCOdxn2GdcJ4Wu2TZRq20DmNWDu1iNj3JY5ADMC7rOL2F+mfuT8QjQyAX5nMJCp4ere0JdLUznZiiUZacu7vpqh9lLgxIgK1PFZwm6RiM2/s5PvPHLNJTrNLB vyacheslav@info 2 | -------------------------------------------------------------------------------- /components/nginx/dhparam.pem: -------------------------------------------------------------------------------- 1 | -----BEGIN DH PARAMETERS----- 2 | MIICCAKCAgEA/pQWKUwSCHqgCq6Shb/e45riq9Pec+vg66MKmWALy6MGpG5dSe+q 3 | 35HD3Yd2oeM4EP+CjuNmESbao1vL3dZy1EG96Xv00gaT1EqasWbBHF2NCU/zOHU+ 4 | pCZxTdqp6oqPriR8FL+B5d+iI/FKZRz6UEoKrBF6jtWbcq7OaDDq+MexrDP0YixU 5 | C02TslzISKlSGeyvEdFNj4oCTBWdG1Ieyx/56DCYbBqZQDBK7gHZbdu19vhkQqas 6 | RhyRgzzoTlBt9Pri7dxCDMeqy4GfcsqY26t4pEVwLFu6789WZX3/iFnvHeCWLrhu 7 | tFzs2E3tD2lOTMr7ptkf/82RKi2nteU18SIljiyhV/yj7u8d21fAdsNhhLE/MXdu 8 | k916MSJsIYf5XbpXD3a4inA+xfUz6b6nsxhJfiYrrwChLXS72QU4QHKj0gD3bfs5 9 | zXuL02f3RfhDL5rh2xsUt+KPShkp3mpKtTKkxx7FmFF2aT2bGVAtwdrTNlCzsKmm 10 | qKCmLN0WUGxGauooWJwiXETBbHHSLKI5E7cqbaGqWXadP2+zh/ng4DuZO+qwZLWB 11 | VxX6r6O77VaGlyMxJjqe0yykC2cohCrxpJqCbpB9PCEjnT1W2A62gPZH20hbUy6y 12 | BMExuuWZ8ifkqdgk4XzZKzvpXQrqws9mGLE3PYE9O79NoM5IZOghO7MCAQI= 13 | -----END DH PARAMETERS----- 14 | -------------------------------------------------------------------------------- /docs/enable_security.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Voronenko/devops-jenkins-box-template/c8269cca3e74ef6df29812528b907a08a779a308/docs/enable_security.png -------------------------------------------------------------------------------- /docs/plugins-manager.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Voronenko/devops-jenkins-box-template/c8269cca3e74ef6df29812528b907a08a779a308/docs/plugins-manager.png -------------------------------------------------------------------------------- /hosts.template: -------------------------------------------------------------------------------- 1 | [jenkins] 2 | 192.168.0.17 ansible_ssh_user=jenkins 3 | -------------------------------------------------------------------------------- /hostsbootstrap.template: -------------------------------------------------------------------------------- 1 | [jenkins_bootstrap] 2 | 192.168.0.17 ansible_ssh_user=your_initial_root_user ansible_ssh_pass=your_initial_root_pass_ifkeys 3 | -------------------------------------------------------------------------------- /init.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ansible-galaxy install -p ./roles -r requirements.yml $1 $2 $3 $3 $4 $5 $6 $7 $8 $9 4 | 5 | exit 0 6 | 7 | git config -f .projmodules --get-regexp '^submodule\..*\.path$' > tempfile 8 | 9 | while read -u 3 path_key path 10 | do 11 | url_key=$(echo $path_key | sed 's/\.path/.url/') 12 | url=$(git config -f .projmodules --get "$url_key") 13 | 14 | read -p "Are you sure you want to delete $path and re-initialize as a separate repository? " yn 15 | case $yn in 16 | [Yy]* ) rm -rf $path; git clone $url $path; echo "$path has been initialized";; 17 | [Nn]* ) continue;; 18 | * ) echo "Please answer yes or no.";; 19 | esac 20 | 21 | done 3