├── setup.cfg ├── MANIFEST.in ├── example ├── run.sh ├── requirements.txt ├── db.env ├── master.cfg ├── buildbot_travis-0.2.2_13_g78ad435-py2-none-any.whl ├── .gitignore ├── README.rst ├── install.sh ├── cfg.yml ├── start_buildbot.sh ├── buildbot.tac └── docker-compose.yml ├── requirements-dev.txt ├── postcss.config.js ├── install-dev.sh ├── karma.conf.js ├── .gitignore ├── Dockerfile ├── .bbtravis.yml ├── tox.ini ├── webpack.config.js ├── LICENSE ├── package.json ├── setup.py ├── CHANGES └── README.rst /setup.cfg: -------------------------------------------------------------------------------- 1 | [bdist_wheel] 2 | universal=1 3 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include buildbot_travis/VERSION 2 | recursive-include buildbot_travis/static * 3 | -------------------------------------------------------------------------------- /example/run.sh: -------------------------------------------------------------------------------- 1 | . sandbox/bin/activate 2 | buildbot upgrade-master . 3 | twistd -ny buildbot.tac 4 | -------------------------------------------------------------------------------- /example/requirements.txt: -------------------------------------------------------------------------------- 1 | supervisor ; python_version < '3.0' 2 | docker-py 3 | requests 4 | buildbot_pkg 5 | -------------------------------------------------------------------------------- /requirements-dev.txt: -------------------------------------------------------------------------------- 1 | supervisor ; python_version < '3.0' 2 | mysqlclient 3 | docker-py 4 | requests 5 | buildbot_pkg 6 | -------------------------------------------------------------------------------- /example/db.env: -------------------------------------------------------------------------------- 1 | # database parameters are shared between containers 2 | POSTGRES_PASSWORD=change_me 3 | POSTGRES_USER=buildbot 4 | POSTGRES_DB=buildbot 5 | -------------------------------------------------------------------------------- /postcss.config.js: -------------------------------------------------------------------------------- 1 | module.exports = { 2 | plugins: { 3 | autoprefixer: { 4 | browsers: ['last 2 versions'] 5 | }, 6 | }, 7 | }; 8 | -------------------------------------------------------------------------------- /example/master.cfg: -------------------------------------------------------------------------------- 1 | from buildbot_travis import TravisConfigurator 2 | c = BuildmasterConfig = {} 3 | TravisConfigurator(BuildmasterConfig, basedir).fromYaml('cfg.yml') 4 | -------------------------------------------------------------------------------- /example/buildbot_travis-0.2.2_13_g78ad435-py2-none-any.whl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/buildbot/buildbot_travis/HEAD/example/buildbot_travis-0.2.2_13_g78ad435-py2-none-any.whl -------------------------------------------------------------------------------- /example/.gitignore: -------------------------------------------------------------------------------- 1 | buildbot_travis-job/ 2 | buildbot_travis/ 3 | http.log 4 | master.cfg.sample 5 | sandbox/ 6 | slave/ 7 | state.sqlite 8 | state.sqlite-shm 9 | state.sqlite-wal 10 | -------------------------------------------------------------------------------- /install-dev.sh: -------------------------------------------------------------------------------- 1 | # this install dependancies in current directory with the help of virtualenv 2 | set -e 3 | virtualenv sandbox 4 | . sandbox/bin/activate 5 | pip install -r requirements-dev.txt 6 | pip install -e . 7 | -------------------------------------------------------------------------------- /example/README.rst: -------------------------------------------------------------------------------- 1 | install.sh 2 | ========== 3 | 4 | install dependancies in a "sandbox" virtualenv 5 | 6 | run.sh 7 | ====== 8 | 9 | run buildbot master and slave with logs in the current terminal 10 | 11 | -------------------------------------------------------------------------------- /example/install.sh: -------------------------------------------------------------------------------- 1 | # this install dependancies in current directory with the help of virtualenv 2 | set -e 3 | virtualenv sandbox 4 | . sandbox/bin/activate 5 | pip install -r requirements.txt 6 | pip install --verbose -e .. 7 | -------------------------------------------------------------------------------- /karma.conf.js: -------------------------------------------------------------------------------- 1 | const common = require('buildbot-build-common'); 2 | 3 | module.exports = function karmaConfig (config) { 4 | common.createTemplateKarmaConfig(config, { 5 | testRoot: 'src/tests.webpack.js', 6 | webpack: require('./webpack.config') 7 | }); 8 | }; 9 | -------------------------------------------------------------------------------- /example/cfg.yml: -------------------------------------------------------------------------------- 1 | env: {} 2 | not_important_files: [] 3 | projects: 4 | - branches: 5 | - master 6 | name: buildbot 7 | repository: https://github.com/tardyp/buildbot 8 | stages: [] 9 | tags: [] 10 | vcs_type: github 11 | stages: [] 12 | workers: 13 | - name: localworker 14 | type: LocalWorker -------------------------------------------------------------------------------- /example/start_buildbot.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | B=`pwd` 3 | if [ ! -f $B/buildbot.tac ] 4 | then 5 | bbtravis create-master $B 6 | cp /usr/src/buildbot_travis/example/buildbot.tac $B 7 | fi 8 | # wait for pg to start by trying to upgrade the master 9 | until buildbot upgrade-master $B 10 | do 11 | sleep 1 12 | done 13 | exec twistd -ny $B/buildbot.tac 14 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | build/ 2 | *.pyc 3 | _trial_temp/ 4 | _trial_temp.lock 5 | bin/ 6 | buildbot_travis.egg-info/ 7 | distribute-*.tar.gz 8 | include/ 9 | lib/ 10 | .tox 11 | static 12 | libs 13 | node_modules 14 | VERSION 15 | sandbox 16 | wheelhouse 17 | dist 18 | slave/ 19 | buildbot.tac 20 | run.sh 21 | *.log 22 | *.pid 23 | *.sqlite 24 | pollers/ 25 | .idea 26 | *-job/ 27 | *-try/ 28 | *-deploy/ 29 | *.sample 30 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | # please follow docker best practices 2 | # https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/ 3 | 4 | FROM buildbot/buildbot-master:master 5 | COPY example /usr/src/buildbot_travis/example 6 | 7 | RUN \ 8 | pip install buildbot_travis && \ 9 | rm -r /root/.cache 10 | 11 | EXPOSE 8010 12 | EXPOSE 9989 13 | 14 | CMD ["/usr/src/buildbot_travis/example/start_buildbot.sh"] 15 | -------------------------------------------------------------------------------- /.bbtravis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | python: 3 | - "3.5" 4 | - "3.6" 5 | - "3.7" 6 | - "3.8" 7 | 8 | install: 9 | - title: install 10 | cmd: | 11 | pip install --no-binary :all: buildbot # need no-binary to get the test framework 12 | pip install -r example/requirements.txt 13 | pip install -e . 14 | pip install mock 15 | 16 | script: 17 | - title: tests 18 | cmd: | 19 | trial buildbot_travis.tests 20 | -------------------------------------------------------------------------------- /example/buildbot.tac: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | from twisted.application import service 4 | from buildbot.master import BuildMaster 5 | 6 | # setup master 7 | basedir = os.path.abspath(os.path.dirname(__file__)) 8 | configfile = 'master.cfg' 9 | 10 | # Default umask for server 11 | umask = None 12 | 13 | # note: this line is matched against to check that this is a buildmaster 14 | # directory; do not edit it. 15 | application = service.Application('buildmaster') 16 | import sys 17 | 18 | from twisted.python.log import ILogObserver, FileLogObserver 19 | 20 | application.setComponent(ILogObserver, FileLogObserver(sys.stdout).emit) 21 | 22 | m = BuildMaster(basedir, configfile, umask) 23 | m.setServiceParent(application) 24 | 25 | -------------------------------------------------------------------------------- /example/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '2' 2 | services: 3 | postgres: 4 | env_file: db.env 5 | image: "postgres:9.4" 6 | volumes: 7 | - /var/lib/buildbot_db:/var/lib/postgresql/data 8 | expose: 9 | - 5432 10 | 11 | buildbot: 12 | # switch between local build and dockerhub image 13 | # build: ../ 14 | image: "buildbot/buildbot-travis" 15 | env_file: db.env 16 | ports: 17 | - "8010:8010" 18 | # for external non-docker workers, uncomment the following line 19 | - "9989:9989" 20 | expose: 21 | - 9989 22 | depends_on: 23 | - postgres 24 | volumes: 25 | - /var/lib/buildbot_travis/:/var/lib/buildbot/ 26 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | # Tox (http://tox.testrun.org/) is a tool for running tests 2 | # in multiple virtualenvs. This configuration file will run the 3 | # test suite on all supported python versions. To use it, "pip install tox" 4 | # and then run "tox" from this directory. 5 | 6 | [tox] 7 | envlist = py26, py27, pypy 8 | 9 | [testenv] 10 | commands = trial buildbot_travis/tests 11 | usedevelop=True 12 | pip_pre=True 13 | deps = 14 | mock 15 | buildbot_pkg 16 | 17 | [pylama] 18 | skip = */.tox/*,*/.env/* 19 | linters = pylint,pep8 20 | ignore = F0401,C0111,E731,E501,C0301,C0330 21 | 22 | [pylama:pyflakes] 23 | builtins = _ 24 | 25 | [pylama:pep8] 26 | max_line_length = 100 27 | 28 | [pylama:pylint] 29 | max_line_length = 100 30 | disable = R 31 | [flake8] 32 | max_line_length = 100 33 | 34 | -------------------------------------------------------------------------------- /webpack.config.js: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | const common = require('buildbot-build-common'); 4 | const env = require('yargs').argv.env; 5 | const pkg = require('./package.json'); 6 | 7 | var event = process.env.npm_lifecycle_event; 8 | 9 | var isTest = event === 'test' || event === 'test-watch'; 10 | var isProd = env === 'prod'; 11 | 12 | module.exports = function() { 13 | return common.createTemplateWebpackConfig({ 14 | entry: { 15 | scripts: './buildbot_travis/ui/src/app/app.module.js', 16 | styles: './buildbot_travis/ui/src/styles/styles.less', 17 | }, 18 | libraryName: pkg.name, 19 | pluginName: pkg.plugin_name, 20 | dirname: __dirname, 21 | isTest: isTest, 22 | isProd: isProd, 23 | outputPath: __dirname + '/buildbot_travis/static', 24 | extractStyles: true, 25 | provideJquery: true, 26 | }); 27 | }(); 28 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2012-2016 the buildbot_travis contributors (as per git log) 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 4 | 5 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 6 | 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 8 | -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "buildbot-travis", 3 | "plugin_name": "buildbot_travis", 4 | "private": true, 5 | "main": "src/data.module.js", 6 | "scripts": { 7 | "build": "rimraf buildbot_travis/static && webpack --bail --progress --profile --env prod", 8 | "build-dev": "rimraf buildbot_travis/static && webpack --bail --progress --profile --env dev", 9 | "dev": "webpack --bail --progress --profile --watch --env dev", 10 | "test": "karma start", 11 | "test-watch": "karma start --auto-watch --no-single-run" 12 | }, 13 | "devDependencies": { 14 | "angular-mocks": "^1.7.9", 15 | "angular-ui-bootstrap": "^2.5.6", 16 | "buildbot-build-common": "~1.0.0", 17 | "lodash": "^4.17.19", 18 | "rimraf": "^2.6.3" 19 | }, 20 | "dependencies": { 21 | "@uirouter/angularjs": "^1.0.15", 22 | "angular": "^1.8.0", 23 | "angular-animate": "^1.7.9", 24 | "angular-bootstrap-show-errors": "^2.3.0", 25 | "buildbot-data-js": "^3.0.1", 26 | "d3": "^3.5.17", 27 | "guanlecoja-ui": "^2.0.0", 28 | "jquery": "^3.5.0", 29 | "ng-tags-input": "*" 30 | } 31 | } 32 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from __future__ import absolute_import, division, print_function 2 | 3 | from setuptools import find_packages 4 | 5 | try: 6 | from buildbot_pkg import setup_www_plugin 7 | except ImportError: 8 | import sys 9 | print("Please install buildbot_pkg module in order to install that package, or use the pre-build .whl modules available on pypi", file=sys.stderr) 10 | sys.exit(1) 11 | 12 | setup_www_plugin( 13 | name='buildbot_travis', 14 | description="Travis CI implemented in Buildbot", 15 | long_description=open("README.rst").read(), 16 | keywords="buildbot travis ci", 17 | url="http://github.com/buildbot/buildbot_travis", 18 | author="Buildbot community", 19 | author_email="buildbot-devel@lists.sourceforge.net", 20 | license="MIT", 21 | packages=find_packages(exclude=['ez_setup']), 22 | include_package_data=True, 23 | zip_safe=False, 24 | entry_points={ 25 | 'buildbot.travis': [ 26 | 'git+poller = buildbot_travis.vcs.git:GitPoller', 27 | 'gerrit = buildbot_travis.vcs.gerrit:Gerrit', 28 | 'github = buildbot_travis.vcs.github:GitHub', 29 | 'gitpb = buildbot_travis.vcs.git:GitPb', 30 | # untested 'svn+poller = buildbot_travis.vcs.svn:SVNPoller', 31 | ], 32 | 'buildbot.www': [ 33 | 'buildbot_travis = buildbot_travis.ep:ep' 34 | ], 35 | 'console_scripts': [ 36 | 'bbtravis=buildbot_travis.cmdline:bbtravis', 37 | ] 38 | }, 39 | install_requires=[ 40 | 'setuptools', 41 | 'buildbot>=0.9.6', # for virtual builders features 42 | 'buildbot-www', 43 | 'buildbot-console-view', 44 | 'buildbot-waterfall-view', 45 | 'buildbot-worker', 46 | 'klein', 47 | 'urwid', 48 | 'PyYAML', 49 | 'txrequests', 50 | 'pyjade', 51 | 'txgithub', 52 | 'ldap3', 53 | 'hyper_sh', 54 | 'future' 55 | ], 56 | ) 57 | -------------------------------------------------------------------------------- /CHANGES: -------------------------------------------------------------------------------- 1 | Changelog 2 | ========= 3 | 4 | 0.6.4 5 | ----- 6 | - Another fix for python3 7 | 8 | 0.6.3 9 | ----- 10 | - Fix admin panel for python3 11 | 12 | 0.6.2 13 | ----- 14 | - use AnyControlEndpointMatcher to protect control access for everything 15 | - Allow to specify volumes to mount in docker 16 | 17 | 0.6.1 18 | ----- 19 | - publish universal wheels 20 | - Fixed possibility to use different docker image and container size according to a property for hyper. 21 | 22 | 0.6.0 23 | ----- 24 | - Fixed possibility to use different docker image according to a property 25 | - Added support for virtual builders. 26 | - Fix build with node6. 27 | - Python3 compatibility fixes. 28 | 29 | 0.5.0 30 | ----- 31 | - Added possibility to use Buildbot steps directly in the .bbtravis.yml. 32 | - Added possibility to use Interpolate from within step definition. 33 | - Global environment is no more part of the triggered build reason. 34 | 35 | 0.4.4 36 | ----- 37 | - add TreeStableTimer configurability. This avoids to create one build per commit but rather one build per push/merge. 38 | 39 | 0.4.3 40 | ----- 41 | - upgrade compatibility fixes. Fix case were tags are duplicated in the config 42 | 43 | 0.4.2 44 | ----- 45 | - README fixes for pypi page 46 | 47 | 0.4.1 48 | ----- 49 | - Minor dependencies fix to create a stable docker container 50 | 51 | 0.4.0 52 | ----- 53 | - Support for Hyper latent workers 54 | - Support for admin authentiation 55 | - Support for GitHub Pull requests 56 | 57 | 0.3.0 58 | ----- 59 | - Support for configuration of workers and auth via administration UI 60 | 61 | 0.1.1 62 | ----- 63 | Release of buildbot_travis based on buildbot nine beta1 64 | 65 | Basic support for github 66 | 67 | 0.1.0 68 | ----- 69 | First release of buildbot_travis based on buildbot nine 70 | This version was hard to use as buildbot nine was not available in pip 71 | 72 | 0.0.19 (unreleased) 73 | ------------------- 74 | 75 | - Nothing changed yet. 76 | 77 | 78 | 0.0.18 (2012-11-12) 79 | ------------------- 80 | 81 | - Fix borked release. 82 | 83 | 84 | 0.0.17 (2012-11-12) 85 | ------------------- 86 | 87 | - Update to use the latest trigger API (as it was changed in 0.8.7). 88 | 89 | - Add web UI for registering new projects. 90 | 91 | 92 | 0.0.16 (2012-11-01) 93 | ------------------- 94 | 95 | - Remove the ``name`` args from the pollers - they were removed in time for 0.8.7. 96 | 97 | 98 | 0.0.15 (2012-11-01) 99 | ------------------- 100 | 101 | - Now fully support the ``matrix`` options allowing finer control over the 102 | implied matrix generated from the list of runtimes and environment 103 | variables:: 104 | 105 | python: 106 | - 2.6 107 | - 2.7 108 | env: 109 | - FLAVOUR=apple 110 | - FLAVOUR=orange 111 | matrix: 112 | exclude: 113 | - python: 2.7 114 | env: FLAVOUR=orange 115 | include: 116 | - python: 2.7 117 | env: FLAVOUR=banana 118 | 119 | This will do an additional build of the ``banana`` build but only for python 120 | 2.7. And it will turn off the build for the ``orange`` flavour, again only 121 | for python 2.7. 122 | 123 | - Mail notification can't send a change notification if it doesn't have 124 | anything to compare to (to detect the change). So it doesn't try to. 125 | 126 | 127 | 0.0.14 (2012-06-11) 128 | ------------------- 129 | 130 | - UI hints during graceful shutdown 131 | 132 | 133 | 0.0.13 (2012-06-11) 134 | ------------------- 135 | 136 | - Only show pending builds on the right page :P 137 | 138 | 139 | 0.0.12 (2012-06-07) 140 | ------------------- 141 | 142 | - Nothing changed yet. 143 | 144 | 145 | 0.0.11 (2012-06-07) 146 | ------------------- 147 | 148 | - Fix manifest to include templates 149 | 150 | 151 | 0.0.10 (2012-06-07) 152 | ------------------- 153 | 154 | - Nothing changed yet. 155 | 156 | 157 | 0.0.9 (2012-06-07) 158 | ------------------ 159 | 160 | - Add a ``nextBuild`` handler that stops there being spawner builds without 161 | runner capacity. 162 | 163 | - Add some default ignores. 164 | 165 | - Use SingleBranchScheduler / AnyBranchScheduler 166 | 167 | - Show pending builds in web ui. 168 | 169 | 170 | 0.0.8 (2012-06-06) 171 | ------------------ 172 | 173 | - Fix unskinned 'building' view on ``/projects/wibble/`` url handler. 174 | 175 | 176 | 0.0.7 (2012-06-06) 177 | ------------------ 178 | 179 | - Brown paper bag release. 180 | 181 | 182 | 0.0.6 (2012-06-06) 183 | ------------------ 184 | 185 | - The pollers are now named - this allows them to be triggered by a change 186 | hook. 187 | 188 | 189 | 0.0.5 (2012-06-05) 190 | ------------------ 191 | 192 | - SVN poller configuration is subclassable 193 | 194 | 195 | 0.0.4 (2012-06-05) 196 | ------------------ 197 | 198 | - Custom MailNotifier that makes use of .travis.yml 199 | 200 | 201 | 0.0.3 (2012-06-05) 202 | ------------------ 203 | 204 | - Nothing changed yet. 205 | 206 | 207 | 0.0.2 (2012-05-30) 208 | ------------------ 209 | 210 | - Nothing changed yet. 211 | 212 | 213 | 0.0.1 (2012-04-22) 214 | ------------------ 215 | 216 | - Builds triggered by spawner build step get marked with a ``spawned_by`` 217 | property so we can relate them to their parent builds. 218 | 219 | - Add a HtmlResource for showing ``/console`` style views of builds. 220 | 221 | - Take advantage of features in Buildbot 0.8.6 and later. 222 | 223 | - Fix hideStepIf and flunking options 224 | 225 | - Custom steps should be interruptable 226 | 227 | 228 | 0.0.0 (2012-04-11) 229 | ------------------ 230 | 231 | - Initial release 232 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | ============================ 2 | Travis CI Compatibility Shim 3 | ============================ 4 | 5 | This is a setup of Buildbot steps, factories and configuration helpers that 6 | give you the best of buildbot and the best of Travis CI: 7 | 8 | * Builder configuration that lives with the source code 9 | * Private builds 10 | * non-github SCM support (gerrit, gitlab, github, github enterpris) 11 | * unlimitted build parallelization on your own infra 12 | 13 | 14 | Basically we provide a compatibility shim in buildbot that allows it to consume a ``.travis.yml`` file. 15 | 16 | buildbot_travis does however not support the full .travis.yml format. 17 | 18 | |travis-badge|_ |codecov-badge|_ 19 | 20 | 21 | .. |travis-badge| image:: https://travis-ci.org/buildbot/buildbot_travis.svg?branch=master 22 | .. _travis-badge: https://travis-ci.org/buildbot/buildbot_travis 23 | .. |codecov-badge| image:: http://codecov.io/github/buildbot/buildbot_travis/coverage.svg?branch=master 24 | .. _codecov-badge: http://codecov.io/github/buildbot/buildbot_travis?branch=master 25 | 26 | 27 | QuickStart 28 | ========== 29 | 30 | First you need to make sure you have the proper python 2.7 environment. On ubuntu 16.04, that would mean:: 31 | 32 | sudo apt-get install build-essential python-dev libffi-dev libssl-dev python-pip 33 | pip install virtualenv 34 | 35 | Then you create a virtualenv and install buildbot_travis via pip:: 36 | 37 | mkdir bbtravis 38 | cd bbtravis 39 | virtualenv sandbox 40 | . ./sandbox/bin/activate 41 | pip install buildbot_travis 42 | 43 | Now you can create a new master:: 44 | 45 | bbtravis create-master master 46 | 47 | Now you can start that new master:: 48 | 49 | buildbot start master 50 | 51 | And then go to the UI: http://localhost:8010 which has an administration panel where to configure the projects. 52 | 53 | 54 | QuickStart With Docker 55 | ====================== 56 | 57 | :: 58 | 59 | docker run -p 8010:8010 -p 9989:9989 buildbot/buildbot-travis 60 | 61 | 62 | QuickStart With Hyper 63 | ===================== 64 | 65 | :: 66 | 67 | IP= 68 | container=`hyper run -d -e buildbotURL=http://$IP/ -p 9989:9989 -p 80:8010 buildbot/buildbot-travis` 69 | hyper fip attach $IP $container 70 | echo go to http://$IP/#/bbtravis/config/auth to configure admin access 71 | echo go to http://$IP/#/bbtravis/config/workers to configure 72 | 73 | 74 | Buildbot Nine UI Plugin 75 | ======================= 76 | 77 | buildbot_travis is configurable via the web UI. 78 | 79 | You can edit the project list, environment variables, not_important files, deployment environments, all through the web UI. 80 | 81 | high level configuration is either stored in a yaml file or directly in the configured database. 82 | 83 | The per project config file 84 | =========================== 85 | 86 | This is a ``.travis.yml`` for a typical buildout project:: 87 | 88 | language: python 89 | 90 | before_install: python bootstrap.py 91 | install:./bin/buildout 92 | script: ./bin/test 93 | 94 | You can read more about this file format on the travis-ci website:: 95 | 96 | http://about.travis-ci.org/docs/user/build-configuration/ 97 | 98 | But features not also mentioned on this page might not currently be supported. 99 | 100 | 101 | Supported languages 102 | ------------------- 103 | 104 | The list of supported language is depending on your build worker configuration. 105 | 106 | With the help of docker, you can create as many images as you need worker configuration. 107 | 108 | 109 | Actually the language parameter of the defacto travis format does not fully leverage the full possibilities of what you can do with buildbot. 110 | 111 | You could think of selecting a different docker image according to the version of software you want to check. 112 | This can avoid the time to setup the worker environment at the beginning of your travis.yml (as you would do in travis saas) 113 | 114 | 115 | Build Steps 116 | ----------- 117 | 118 | Travis provides 6 hook points for your builds: 119 | 120 | * before_install 121 | * install 122 | * after_install 123 | * before_script 124 | * script 125 | * after_script 126 | 127 | We really don't care what you run from these hooks as long as exit code 0 means 128 | success and anything else means fail. 129 | 130 | You can provide a single command like this:: 131 | 132 | install: ./bin/buildout 133 | 134 | Or multiple commands like this:: 135 | 136 | install: 137 | - ./configure 138 | - ./bin/buildout 139 | 140 | Each element of the list in the yaml will create a single step, which is named with the first characters of your command line. 141 | 142 | If you want to create a custom name, buildbot_travis supports following syntax:: 143 | 144 | script: 145 | - | 146 | # build 147 | ./configure 148 | make 149 | - | 150 | # tests 151 | make tests 152 | 153 | 154 | Buildbot specific features 155 | -------------------------- 156 | 157 | Steps as dictionary 158 | ~~~~~~~~~~~~~~~~~~~ 159 | 160 | Original Travis just create a simple shell script to run the whole CI script. 161 | Buildbot is a little bit more powerful, and buildbot_travis can make use of it. 162 | For this you need to go out of the travis "de-facto" standard. e.g:: 163 | 164 | script: 165 | - | 166 | # build 167 | ./configure 168 | make 169 | 170 | - title: tests 171 | shell: dash 172 | condition: TESTS=='tests' 173 | cmd: make tests 174 | 175 | If yaml parser encounters a dictionary, then it will use the following keys: 176 | 177 | 178 | * ``title``: the title of the step in the UI 179 | 180 | * ``shell``: run the cmd inside the given shell. This is normally not 181 | necessary, since the buildbot worker will apply the appropriate 182 | shell (``cmd`` for Windows, ``/bin/sh`` for everything else). If the 183 | value is a list, it will be used as is. Otherwise, it is assumed 184 | that it uses the option ``-c`` to take a command string. 185 | 186 | * ``condition``: a condition to run the step. 187 | It is evaluated as a python expression, with variables beiing the environment variable generated by your matrix. 188 | The condition is evaluated at the time of the parsing of the yaml file. 189 | If the condition is not met, then the step is just not inserted in the step list. 190 | 191 | * ``cmd``: The command to run. 192 | 193 | * ``step``: The buildbot step create. 194 | See below for detailled description. 195 | if defined, ``shell``, ``title`` and ``cmd`` keys are ignored. 196 | 197 | .bbtravis.yml 198 | ~~~~~~~~~~~~~ 199 | 200 | 201 | In order to keep working with buildbot_travis and travis.org at the same time, buildbot travis will look for a .bbtravis.yml before .travis.yml. 202 | With this, you can keep your .travis.yml without any buildbot specific feature. 203 | 204 | Shallow Clone 205 | ~~~~~~~~~~~~~ 206 | 207 | * Original travis supports clone depth configuration inside the yml file (aka shallow clone). 208 | As the git clone is made before buildbot has a chance to parse the yaml, this configuration is done in the per project config in buildbot travis. 209 | Two options are available in the cfg.yml (shallow and retryFetch) e.g:: 210 | 211 | projects: 212 | - branches: 213 | - master 214 | name: buildbot 215 | repository: https://github.com/buildbot/buildbot 216 | shallow: 200 217 | mode: "full" 218 | method: "clobber" 219 | stages: [] 220 | tags: [] 221 | vcs_type: github 222 | 223 | Interpolate 224 | ~~~~~~~~~~~ 225 | 226 | Buildbot has a very useful `Interpolate `_ utility. 227 | If you prepend your scripts by ```!i`` or ``!interpolate``, then buildbot_travis will automatically create an Interpolate object:: 228 | 229 | - title: make dist 230 | cmd: !i make REVISION=%(prop:got_revision:-%(src::revision:-unknown)s)s dist 231 | 232 | Commands without shell 233 | ~~~~~~~~~~~~~~~~~~~~~~ 234 | 235 | If cmd is a list, it will run without use of shell (this can avoid to have to shell quote variables): 236 | 237 | .. code-block:: yaml 238 | 239 | script: 240 | - title: make dist 241 | cmd: [ "make", !i "REVISION=%(prop:got_revision:-%(src::revision:-unknown)s)s", "dist" ] 242 | 243 | Buildbot Steps Batteries 244 | ~~~~~~~~~~~~~~~~~~~~~~~~ 245 | 246 | Buildbot comes with battery included. It has a `tons of steps `_ in it that you could use. 247 | What if you could contruct those steps in the bbtravis.yml? 248 | Guess what? You can. 249 | 250 | .. code-block:: yaml 251 | 252 | script: 253 | - condition: TESTS=='trial' 254 | step: !Trial 255 | name: trial 256 | tests: buildbot.test 257 | 258 | Every Buildbot steps from the buildbot.plugins.steps module is available by default. 259 | If you want to use your own customs steps, you can do it with 2 methods. 260 | 261 | - Create a buildbot `plugin `_. 262 | If it is installed in your master virtual environment and recognised inside buildbot.plugins.steps, it will be available in buildbot_travis yaml parser. 263 | 264 | - If you want to define your custom step in your master.cfg directly, you will need to register your step directly in the yaml parser. 265 | 266 | .. code-block:: python 267 | 268 | from buildbot_travis.travisyml import registerStepClass 269 | 270 | class FancyStep(steps.ShellSequence): 271 | ... 272 | 273 | registerStepClass("FancyStep", FancyStep) 274 | 275 | then in your yaml: 276 | 277 | .. code-block:: yaml 278 | 279 | script: 280 | - step: !FancyStep 281 | 282 | .. note:: 283 | 284 | You can construct your steps either with arg list or keyword args, but not both e.g following are equivalent 285 | 286 | .. code-block:: yaml 287 | 288 | script: 289 | - step: !ShellCommand "true" 290 | 291 | - step: !ShellCommand 292 | - "true" 293 | 294 | - step: !ShellCommand 295 | command: "true" 296 | 297 | .. note:: 298 | 299 | Due to the way steps are initialized, ``title`` key cannot be used to override the default step name. 300 | You have to use the standard ``name`` step argument to specify it: 301 | 302 | .. code-block:: yaml 303 | 304 | script: 305 | - step: !ShellCommand 306 | command: "true" 307 | name: "always succeed" 308 | 309 | .. note:: 310 | 311 | You can also contruct your step list without passing through the dictionary structure 312 | 313 | .. code-block:: yaml 314 | 315 | script: 316 | - !ShellCommand 317 | command: "true" 318 | name: "always succeed" 319 | 320 | Status context 321 | ~~~~~~~~~~~~~~ 322 | 323 | If github_token is specified, bbtravis will create a github status for each of the builds of the matrix, with direct link to the sub build. 324 | The name of the status (aka context) is calculated using ``reporter_context`` of the project configuration. 325 | The default is ``"bb%(prop:matrix_label:+/)s%(prop:matrix_label)s"``. 326 | 327 | ``matrix_label`` is computed by the Trigger step, and is the concatenation of key and values of the matrix. 328 | because matrix can be large, and github context is limited in size, bbtravis implements a way for projects to define abbreviations for the labels. 329 | e.g .bbtravis.yml such as: 330 | 331 | .. code-block:: yaml 332 | 333 | language: python 334 | 335 | label_mapping: 336 | TWISTED: tw 337 | SQLALCHEMY: sqla 338 | SQLALCHEMY_MIGRATE: sqlam 339 | latest: l 340 | python: py 341 | 342 | Will generate context like: ``bb/py:2.6/sqla:l/sqlam:0.7.1/tw:11.1.0`` 343 | 344 | .. note:: 345 | 346 | context reporter is for now only implemented from github, but it should be easy to adapt to Gitlab, Gerrit, etc 347 | 348 | Installing dependencies 349 | ----------------------- 350 | 351 | The docker image that is used is throw away, and will start from clean state for each build. 352 | 353 | You can create a docker image with passwordless sudo, as travis does, so that you can use apt-get:: 354 | 355 | before_install: 356 | - sudo apt-get update 357 | - sudo apt-get install -y -q mydependency 358 | 359 | It is however a better practice and more optimized to just provide a prebuilt docker image which contain what you need. 360 | 361 | 362 | Environments 363 | ------------ 364 | 365 | You might want to perform multiple builds of the same piece of software. Travis 366 | delivers:: 367 | 368 | env: 369 | - FLAVOUR=blue 370 | - FLAVOUR=green 371 | - FLAVOUR=red 372 | 373 | install: 374 | - ./configure -f $FLAVOUR 375 | - ./bin/buildout 376 | 377 | Commits to this code base will cause builds for blue, green and red flavours. 378 | The environment variables can be used like ordinary environment variables 379 | inside the scripts you run from your ``.travis.yml`` and can be used in the 380 | ``.travis.yml`` itself. 381 | 382 | ``env`` is a list of environment variables. You can specify multiple variables 383 | on a single line like this:: 384 | 385 | env: 386 | - PROP1=foo PROP2=bar 387 | 388 | 389 | Build Matrix 390 | ------------ 391 | 392 | Your options for ``language`` and ``env`` create an implicit build matrix. A 393 | build matrix is a collection of all the possible combinations of the ``env`` 394 | options and language versions. You can fine tine this matrix by excluding 395 | certain combinations, or inserting additional ones. 396 | 397 | Here is an example of excluding a combination and inserting an additional 398 | build:: 399 | 400 | python: 401 | - 2.6 402 | - 2.7 403 | 404 | env: 405 | - FLAVOUR=apple 406 | - FLAVOUR=orange 407 | 408 | matrix: 409 | exclude: 410 | - python: 2.7 411 | env: FLAVOUR=orange 412 | include: 413 | - python: 2.7 414 | env: FLAVOUR=banana 415 | 416 | This will do an additional build of the ``banana`` build but only for python 417 | 2.7. And it will turn off the build for the ``orange`` flavour, again only 418 | for python 2.7. 419 | 420 | 421 | Deployment 422 | ---------- 423 | 424 | A ``Deploy`` section is available in the left side menu. In this section, a Deployment dashboard will be 425 | available once configured. 426 | 427 | This dashboard enables a streamlined, fully automated delivery process, from Commit to Production environment. 428 | Latest version of your project is just one click away from users. 429 | 430 | See the dashboard's template below 431 | 432 | ============== ========= ========= ========= ========= 433 | DELIVERABLES STAGES 434 | -------------- ------------------------------------------------ 435 | (projects) COMMIT DEV QA PROD 436 | ============== ========= ========= ========= ========= 437 | Deliverable A GIT rev 1.2.3 GIT tag GIT tag 438 | ============== ========= ========= ========= ========= 439 | 440 | For example, the version 1.2.3 (specified thanks to a GIT tag) of deliverable A is deployed in DEV stage. 441 | 442 | Here are the 5 steps to setup a Deployment dashboard in Buildbot Travis. 443 | 444 | 1) A ``Deployment`` section is available in the ``Settings`` section. 445 | In this section, the ``Deployment Environment(s)`` is the list of target environments (or Stages) 446 | where deliverables are going to be deployed. 447 | These environments should be sorted following your development process definition. 448 | Example:: 449 | 450 | COMMIT (merged dev), DEV, QA, PROD 451 | BEWARE!The first column is reserved for COMMIT stage so you do not need to define it in the Stages list. 452 | 453 | 2) Go to the ``Deploy`` section in the left side menu. You should see a Deployment dashboard like the above example. 454 | The Stages should be the same as the ones defined in 1). 455 | 456 | 3) Go to the ``Settings/Projects`` section. Add corresponding Stages to the different projects in the Stages field. 457 | Stages can be a subset of the Stages defined in 2). 458 | 459 | 4) You should see a fully configured Deployment dashboard with all the deliverables, Stages, GIT revisions and GIT 460 | tags. GIT revisions and GIT tags are available in dropdown lists. When you select a specific version, a pop_up 461 | window appears to launch the deployment procedure in the specific stage. 462 | 463 | 5) To enable push button deployments, you need to define the deployment procedures. 464 | Create deployment scripts and update the script and/or after_script sections of the ``.travis.yml`` file 465 | of each deliverable. 466 | 467 | Example:: 468 | 469 | after_script: 470 | - | 471 | # Deployment 472 | python ./deploy.py --repo "${repository}" --stage "${stage}" --version "${version}"; 473 | 474 | ${repository} is the URL of the project's (or deliverable's) repo. 475 | ${stage} is the retrieved from the Deployment dashboard. 476 | ${version} is retrieved from the Deployment dashboard. 477 | 478 | How it works 479 | ============ 480 | 481 | The basic behaviour is: 482 | 483 | * Commit is picked up (polling by default, with additional triggers via 484 | ``/change_hook/poller?poller=pollername`` web hook 485 | 486 | * Build is scheduled on a 'spawner' builder - this is a builder configured to 487 | use an ordinary slave 488 | 489 | * Checkout occurs - for the purposes of acquiring the ``.travis.yml`` rather 490 | than for actually performing a build 491 | 492 | * 'spawner' triggers a build on a 'job' builder for each environment in the 493 | build matrix defined in ``.travis.yml`` 494 | 495 | * 'job' builder does a single build in a clean latent buildslave (VM or docker) 496 | 497 | * ``setup-steps`` step dynamically appends ShellCommand steps based on 498 | contents of ``.travis.yml`` 499 | 500 | * when job is over VM orcontainer is thrown away. 501 | 502 | * The 'spawner' build acts as a way of aggregating the build results in a 503 | single pass/fail status. 504 | 505 | * MailNotifier subclass uses ``.travis.yml`` found in build history so that 506 | recipients list and whether or not to mail can be adapted accordingly. 507 | XXX: this needs to be adapted for nine 508 | 509 | 510 | CommandLine 511 | =========== 512 | ``buildbot_travis`` package comes with a ``bbtravis`` command line utility. 513 | 514 | This utility is useful to test travis.yml locally without pushing it to the CI. 515 | It allows to test either the travis.yml and the docker image used to run the workers. 516 | It allows to run only the part of the matrix that you are working on 517 | 518 | Example:: 519 | 520 | bbtravis run -d tardyp/metabbotcfg -j8 TESTS=trial TWISTED=latest 521 | 522 | This will run the resulting tests in parallel using docker image tagged tardyp/metabbotcfg and will filter only the matrix environment with TESTS=='trial' and TWISTED=='latest' 523 | 524 | UI is using urwid console UI framework, and will split the terminal into several terminal showing each matrix run. 525 | You can scroll using mouse wheel, and click to zoom and get more details. 526 | 527 | .. Note:: 528 | 529 | For now ``bbtravis`` command line utility to note support Buildbot step battery nor Interpolate contructs 530 | 531 | TODO 532 | ==== 533 | 534 | This special branch is the nine port of buildbot_travis. 535 | Compared to previous version following features are not yet available 536 | 537 | * Custom MailNotifier needs to be adapted for nine data api, in order to get the .travis.yml configuration 538 | * mergerequest should be adapted to the new collapseRequest api 539 | * SVN shall be validated (only git has been tested so far) 540 | * metrics facility is not really specific to travis, and should be available in buildbot master directly 541 | * nextBuild feature shall be reimplemented: allowed to avoid running a spawner when no '-job' slave is available 542 | 543 | Compared to original Travis format, here is a non-exaustive list of features known not to be supported 544 | 545 | * after_success, after_failure. Not implemented, but easy to add. 546 | * deploy. Deployment step would have to happen after all the matrix subbuilds are succeed 547 | 548 | 549 | And configure your hyper keys in the default hyper worker 550 | You should also configure an authentication plugin in order to protect those keys. 551 | --------------------------------------------------------------------------------