├── .gitattributes ├── .gitignore ├── .travis.yml ├── LICENSE ├── README.rst ├── docka-hub └── README.rst ├── docka-project ├── Dockerfile ├── README.rst └── app.py └── docka-salt ├── README.rst ├── master.d ├── reactor.conf └── rest_api.conf └── srv ├── pillar ├── docker.sls └── top.sls ├── reactor ├── deploy-prod.sls └── deploy-test.sls └── salt ├── docker ├── docka.sls └── init.sls ├── python └── pip.sls ├── top.sls └── yum └── epel.sls /.gitattributes: -------------------------------------------------------------------------------- 1 | .travis.yml merge=ours 2 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | docka-project/env 2 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | sudo: required 2 | 3 | language: python 4 | 5 | services: 6 | - docker 7 | 8 | before_install: 9 | - docker build -t forresta/docka-docka-docka docka-project/ 10 | - docker run -d -p 127.0.0.0:80:5000 forresta/docka-docka-docka 11 | - docker ps -a 12 | - docker run forresta/docka-docka-docka /bin/sh -c "echo 'tests call here'" 13 | 14 | script: 15 | - echo "Test code here again" 16 | 17 | after_success: 18 | - echo "Test code here again" 19 | - if [ "$TRAVIS_BRANCH" == "master" ]; then 20 | docker login -e="$DOCKER_EMAIL" -u="$DOCKER_USERNAME" -p="$DOCKER_PASSWORD"; 21 | docker push forresta/docka-docka-docka; 22 | fi 23 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2015 Forrest 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | docka-docka-docka 2 | ================= 3 | 4 | docka-docka-docka is a full environment example to help you and your developers 5 | get up to speed quickly with an environment which can be duplicated between 6 | environments. Keep in mind that this repo is simply an example that shows 7 | all components. In a real example (which you'll hopefully create) the 8 | directories populated with actual content would be different repos. This is 9 | simply an example of how to set up all the components you will need to get 10 | going. Currently it is suggested to use Ubuntu due to the known issues 11 | listed at the bottom of this Readme. It's made of three components: 12 | 13 | components 14 | ---------- 15 | 16 | docka-project 17 | ------------- 18 | 19 | docka-project uses boot2docker/docker, GitHub, and Travis CI to allow devs to 20 | write code locally, build container images, push to an environment, allow 21 | testing to occur, and merge back into a branch on GitHub which is then ready 22 | for deployment. 23 | 24 | docka-hub 25 | --------- 26 | 27 | docka-hub explains how to set up web hooks within docker hub to hook up to the 28 | salt service that is listening. 29 | 30 | docka-salt 31 | ---------- 32 | 33 | docka-salt uses SaltStack to listen for docker updates and deploy them to the 34 | server, it also handles the nginx-proxy that updates when services change. 35 | 36 | The Goal 37 | -------- 38 | 39 | docka-docka-docka was designed to use open source software that 40 | anyone would have access to. It takes advantage of these tools to show how 41 | streamlined and easy the process can be so that engineers don't have to set up 42 | an excessive number of tools to deploy their docker containers to their hosting 43 | provider. 44 | 45 | Workflow 46 | -------- 47 | 48 | The average workflow for docka-docka-docka is pretty standard. Once a docker 49 | file is written for the associated project a developer can PR against the 50 | associated repository's dev branch. The code will be tested against Travis 51 | and if the PR passes the PR can be merged into dev. Once the developer's 52 | branch is merged into the dev branch it is then accessible by other users. If 53 | everything looks good the developer can then open a PR against the test branch 54 | which is once again tested in Travis, if this PR passes and the content is 55 | merged SaltStack will then pick up the associated Docker image and push it 56 | to the test server for deployment, nginx-proxy will pick up this change for 57 | the associated service. If everything looks good here the same process can be 58 | repeated for the production branch and environment which will be automatically 59 | deployed. 60 | 61 | Note that currently docker has no way to distinguish between when 62 | ``after_success`` commands should be run based on branches. There 63 | are workarounds in the .travis.yml file that handle this. 64 | Https://github.com/travis-ci/travis-ci/issues/5065 has been 65 | opened to request support for only completing after_success based on branch in 66 | a more streamlined manner 67 | 68 | known issues 69 | ------------ 70 | 71 | The current way the states are configured only support CentOS/RHEL based 72 | distros. There should only be minor changes (primarily the repo data) required 73 | to make this support other distros. 74 | 75 | Currently on RHEL based distros it seems as though there is an issue removing 76 | containers: https://github.com/docker/docker/issues/3610#issuecomment-155532043 77 | this means that sometimes the docker deployment (unfortunately) fails. There 78 | are workarounds, however they require adding a disk which was not something 79 | that belonged in this repo. 80 | 81 | On Debian based distros there is an odd pip issue which may cause an error 82 | during the Salt run. This is caused by some pip problems as noted here: 83 | https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1306991 84 | This is currently worked around in the associated salt code. 85 | -------------------------------------------------------------------------------- /docka-hub/README.rst: -------------------------------------------------------------------------------- 1 | docka-hub 2 | ========= 3 | 4 | docka-hub is the component for hooking up docker. Signing up for docker hub is 5 | mandatory in this process, users can register here: https://hub.docker.com/ 6 | 7 | The primary purpose of this section is to detail the necessary webhooks for 8 | SaltStack. This is simply a URL to your server and is added on the repo 9 | page for your created repository via the ``Webhooks`` link. 10 | 11 | For the test environment the example call should look like: 12 | ``curl --data 'success' http://your_domain_or_server_ip:8000/hook/travis/success-test`` 13 | 14 | For the prod environment the example call should look like: 15 | 16 | ``curl --data 'success' http://your_domain_or_server_ip:8000/hook/travis/success-prod`` 17 | 18 | On hub.docker.com itself your webhooks will look like this: 19 | 20 | Test: ``http://your_domain_or_server_ip:8000/hook/travis/success-test`` 21 | 22 | Prod: ``http://your_domain_or_server_ip:8000/hook/travis/success-prod`` 23 | 24 | 25 | -------------------------------------------------------------------------------- /docka-project/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:2.7-slim 2 | RUN pip install Flask 3 | ADD . /docka-project 4 | WORKDIR /docka-project 5 | CMD python app.py 6 | -------------------------------------------------------------------------------- /docka-project/README.rst: -------------------------------------------------------------------------------- 1 | docka-project 2 | ============= 3 | 4 | This is the example project, with an example docker file. Our project consists 5 | of three key points, working locally, creating a PR which is tested, and 6 | building/storing the docker container. 7 | 8 | Using Docker Locally 9 | -------------------- 10 | 11 | If a user is running on Mac/Windows refer to the Boot2Docker installation 12 | guide: https://github.com/boot2docker/boot2docker#installation, if a user 13 | is running Linux refer to the Docker installation guide for the associated 14 | distro: http://docs.docker.com/v1.8/installation/ 15 | 16 | Creating the Container Locally 17 | ------------------------------ 18 | 19 | This repo contains an example docker file for a simple Flask app. You can use 20 | the following steps to create this image (run these within this directory): 21 | 22 | Note that these commands may need to be run as root. 23 | 24 | ``sudo service docker start`` 25 | 26 | ``sudo docker build -t docka-project .`` 27 | 28 | Once the container is created you will need to use the nginx-proxy container 29 | from jwilder (https://github.com/jwilder/nginx-proxy) to do our automated 30 | nginx proxying. To start that run this command: 31 | 32 | ``sudo docker run -d -p 80:80 -v /var/run/docker.sock:/tmp/docker.sock:ro jwilder/nginx-proxy`` 33 | 34 | Then start our Flask app container: 35 | 36 | ``sudo docker run -e VIRTUAL_HOST=test.com -t -d -p 5000:5000 docka-project`` 37 | 38 | You will need to modify your hosts file to point test.com to either your local 39 | IP address, or 0.0.0.0. 40 | 41 | The container should now be accessible from your local system so you can see 42 | the running application. Note that this process can be duplicated for as many 43 | containers as you may have, just make sure they're associated with different 44 | ports. 45 | 46 | Travis Setup for Testing and Docker 47 | ----------------------------------- 48 | 49 | Start by following the excellent getting started guide provided by Travis for 50 | your project: 51 | http://docs.travis-ci.com/user/getting-started/ 52 | 53 | Note that Travis will automatically test when you make a PR as noted here: 54 | http://docs.travis-ci.com/user/pull-requests/ 55 | 56 | It is important to ensure that you complete the instructions to use Travis and 57 | Docker (especially the environment variables so you can push to the docker 58 | hub), as users will be pushing to the Docker registry. For this project it is 59 | recommended that you use the same login as otherwise SaltStack will not know 60 | which project to pull down. 61 | 62 | At this point you should now be able to locally create your containers and run 63 | them via boot2docker or just docker and serve them via nginx-proxy just as a 64 | user would in the test/production environment. Tests should also be run when a 65 | user pushes to the repository. 66 | -------------------------------------------------------------------------------- /docka-project/app.py: -------------------------------------------------------------------------------- 1 | from flask import Flask 2 | app = Flask(__name__) 3 | 4 | @app.route('/') 5 | def hello_world(): 6 | return('Hello World! :)') 7 | 8 | if __name__ == '__main__': 9 | app.run(host="0.0.0.0") 10 | -------------------------------------------------------------------------------- /docka-salt/README.rst: -------------------------------------------------------------------------------- 1 | docka-salt 2 | ========== 3 | 4 | This is the example salt repo that you can use to set up your test and 5 | production environments. It includes all the code required to set up the 6 | listening service to handle the full deployment process. 7 | 8 | For this to work users will have ensure you've followed the steps in the 9 | ``docka-hub`` directory, otherwise the web hook will not be set up correctly 10 | to interface with the salt server. 11 | 12 | This example project contains three components. A Salt master, and two Salt 13 | minions. 14 | 15 | NOTE: This project is using CentOS 7 and Ubuntu 14, and was designed as a very 16 | basic tutorial without too much logic. This means that some code will need to 17 | be modified for other distros. 18 | 19 | Components 20 | ---------- 21 | 22 | Salt Master 23 | ----------- 24 | 25 | The Salt master contains our critical components, Salt itself, the listener 26 | for our Docker hub pushes, and the reactor code for Salt which will deploy 27 | our keys. All files in this directory should go in their respective directories 28 | on the master. 29 | 30 | Salt Minion 31 | ----------- 32 | 33 | This example uses two Salt minions, both minions connect to the master, one is 34 | treated as the production environment, and one is treated as the test 35 | environment. These minions require minimal configuration as most of the Salt 36 | code lives on the master (reactors are NOT currently supported on masterless 37 | minions, see: https://github.com/saltstack/salt/issues/15265 for updates. 38 | 39 | The production minion requires the following settings in the minion config: 40 | 41 | .. code-block:: yaml 42 | 43 | master: your_master_domain_or_ip 44 | 45 | grains: 46 | env: 47 | - prod 48 | 49 | The test minion requires the following settings in the minion config: 50 | 51 | .. code-block:: yaml 52 | 53 | master: your_master_domain_or_ip 54 | grains: 55 | env: 56 | - test 57 | 58 | When set these grains allow us to target the production or test environment 59 | as specific in our reactor states. 60 | 61 | Installation 62 | ============ 63 | 64 | Begin by installing the Salt Master (on the system you have designated 65 | as the Salt master) as specified here but only installing the 66 | Salt Master and salt-api packages: 67 | 68 | https://docs.saltstack.com/en/latest/topics/installation/ 69 | 70 | Once completed install the Salt minion using the same process on both the 71 | test and production servers. Once the master has been installed and the user 72 | has confirmed it is listening modify the minion configs with the data from the 73 | ``Salt Minion`` section above and restart both minions. On the master a 74 | privileged system user should now be able to run ``salt-key --list-all`` and 75 | see the two minions trying to connect, the user should now run 76 | ``salt-key --accept-all`` to accept both pending keys. The comamnd 77 | ``salt '*' test.ping`` should now return a response from both the production and 78 | test servers. 79 | 80 | Directories should now be copied to their relevant directory on the master 81 | system as mapped below: 82 | 83 | ``master.d -> /etc/salt/master.d/`` 84 | 85 | ``srv -> /etc/salt/srv`` 86 | -------------------------------------------------------------------------------- /docka-salt/master.d/reactor.conf: -------------------------------------------------------------------------------- 1 | reactor: 2 | - 'salt/netapi/hook/travis/success-test': 3 | - /srv/reactor/deploy-test.sls 4 | - 'salt/netapi/hook/travis/success-prod': 5 | - /srv/reactor/deploy-prod.sls 6 | -------------------------------------------------------------------------------- /docka-salt/master.d/rest_api.conf: -------------------------------------------------------------------------------- 1 | rest_cherrypy: 2 | port: 8000 3 | host: 0.0.0.0 4 | disable_ssl: True 5 | webhook_disable_auth: True 6 | -------------------------------------------------------------------------------- /docka-salt/srv/pillar/docker.sls: -------------------------------------------------------------------------------- 1 | docker-registries: 2 | https://index.docker.io/v1/: 3 | email: fakeemail@fakeaddressomg123.com 4 | password: fak3passw0rd 5 | username: fak3usernam3 6 | 7 | docker.version: auto 8 | -------------------------------------------------------------------------------- /docka-salt/srv/pillar/top.sls: -------------------------------------------------------------------------------- 1 | base: 2 | '*': 3 | - docker 4 | -------------------------------------------------------------------------------- /docka-salt/srv/reactor/deploy-prod.sls: -------------------------------------------------------------------------------- 1 | run_highstate_prod: 2 | cmd.state.highstate: 3 | - tgt: 'env:prod' 4 | - expr_form: grain 5 | -------------------------------------------------------------------------------- /docka-salt/srv/reactor/deploy-test.sls: -------------------------------------------------------------------------------- 1 | run_highstate_test: 2 | cmd.state.highstate: 3 | - tgt: 'env:test' 4 | - expr_form: grain 5 | -------------------------------------------------------------------------------- /docka-salt/srv/salt/docker/docka.sls: -------------------------------------------------------------------------------- 1 | include: 2 | - docker 3 | 4 | #nginx_container_present: 5 | # dockerng.image_present: 6 | # - name: jwilder/nginx-proxy 7 | # - insecure_registry: True 8 | 9 | nginx_container_present: 10 | cmd.run: 11 | - name: /usr/bin/docker pull jwilder/nginx-proxy 12 | 13 | install_nginx_proxy_container: 14 | dockerng.running: 15 | - name: nginx-proxy 16 | - image: jwilder/nginx-proxy 17 | - port_bindings: 18 | - 80:80 19 | - detach: True 20 | - binds: /var/run/docker.sock:/tmp/docker.sock:ro 21 | 22 | docka_container_present: 23 | cmd.run: 24 | {% if 'prod' in grains['env'] %} 25 | - name: /usr/bin/docker pull forresta/docka-docka-docka 26 | {% elif 'test' in grains['env'] %} 27 | - name: /usr/bin/docker pull forresta/docka-docka-docka-test 28 | {% endif %} 29 | 30 | #docka_container_present: 31 | # dockerng.image_present: 32 | #{% if 'prod' in grains['env'] %} 33 | # - name: forresta/docka-docka-docka 34 | #{% elif 'test' in grains['env'] %} 35 | # - name: forresta/docka-docka-docka-test 36 | #{% endif %} 37 | # - insecure_registry: True 38 | 39 | install_docka_project_container: 40 | dockerng.running: 41 | {% if 'prod' in grains['env'] %} 42 | - name: docka-docka-docka 43 | - image: forresta/docka-docka-docka 44 | {% elif 'test' in grains['env'] %} 45 | - name: docka-docka-docka-test 46 | - image: forresta/docka-docka-docka-test 47 | {% endif %} 48 | - port_bindings: 49 | - 5000:5000 50 | - environment: 51 | - VIRTUAL_HOST: test.hungryadmin.com 52 | - detach: True 53 | -------------------------------------------------------------------------------- /docka-salt/srv/salt/docker/init.sls: -------------------------------------------------------------------------------- 1 | include: 2 | - python.pip 3 | 4 | {% if grains.get('os_family') == 'RedHat' %} 5 | configure_docker_repo: 6 | pkgrepo.managed: 7 | - name: dockerrepo 8 | - hummanname: Docker Repository 9 | - baseurl: https://yum.dockerproject.org/repo/main/centos/7 10 | - gpgcheck: 0 11 | - key_url: https://yum.dockerproject.org/repo/main/centos/7 12 | - refresh_db: True 13 | {% endif %} 14 | 15 | install_docker: 16 | pkg.installed: 17 | {% if grains.get('os_family') == 'RedHat' %} 18 | - name: docker-engine 19 | {% else %} 20 | - name: docker.io 21 | {% endif %} 22 | 23 | install_docker_py: 24 | pip.installed: 25 | - name: docker-py 26 | 27 | docker_service: 28 | service.running: 29 | - name: docker 30 | - enable: True 31 | -------------------------------------------------------------------------------- /docka-salt/srv/salt/python/pip.sls: -------------------------------------------------------------------------------- 1 | {% if grains.get('os_family') == 'RedHat' %} 2 | include: 3 | - yum.epel 4 | 5 | install_pip: 6 | pkg.installed: 7 | - name: python-pip 8 | - refresh: True 9 | {% elif grains.get('os_family') == 'Debian' %} 10 | install_pip: 11 | cmd.run: 12 | - name: easy_install pip 13 | {% endif %} 14 | -------------------------------------------------------------------------------- /docka-salt/srv/salt/top.sls: -------------------------------------------------------------------------------- 1 | base: 2 | '*': 3 | - python.pip 4 | - docker.docka 5 | -------------------------------------------------------------------------------- /docka-salt/srv/salt/yum/epel.sls: -------------------------------------------------------------------------------- 1 | install_epel: 2 | pkg.installed: 3 | - name: epel-release 4 | --------------------------------------------------------------------------------