├── .gitignore ├── LICENSE.txt ├── README.md ├── docs ├── buildenvironment.md ├── howtocontribute.md ├── howtocustomize.md ├── howtoinstall.md ├── howtouse.md ├── index.md ├── qanda.md ├── usecases.md └── whatisvdist.md ├── examples ├── assemble_custom_reqspath.py ├── assemble_directory.py ├── assemble_from_subdirectory.py ├── assemble_local_pipconf.py ├── assemble_parallel.py ├── assemble_with_custom_python.py └── assemble_with_dependencies.py ├── integration-tests └── test_builder.py ├── mkdocs.yml ├── requirements.txt ├── setup.cfg ├── setup.py ├── test-requirements.txt ├── tests ├── test_build_definitions.py ├── test_builder_setup.py ├── test_buildprofile_definitions.py └── test_sources.py └── vdist ├── __init__.py ├── builder.py ├── buildmachine.py ├── defaults.py ├── profiles ├── __init__.py ├── centos.sh ├── centos6.sh ├── debian.sh └── internal_profiles.json └── source.py /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | 5 | # C extensions 6 | *.so 7 | 8 | # vim swap files 9 | *.swp 10 | 11 | # Distribution / packaging 12 | .Python 13 | env/ 14 | bin/ 15 | build/ 16 | develop-eggs/ 17 | dist/ 18 | eggs/ 19 | lib/ 20 | lib64/ 21 | parts/ 22 | sdist/ 23 | var/ 24 | include/ 25 | local/ 26 | *.egg-info/ 27 | .installed.cfg 28 | *.egg 29 | 30 | # Installer logs 31 | pip-log.txt 32 | pip-delete-this-directory.txt 33 | 34 | # Unit test / coverage reports 35 | .tox/ 36 | .coverage 37 | .cache 38 | nosetests.xml 39 | coverage.xml 40 | 41 | # Translations 42 | *.mo 43 | 44 | # Mr Developer 45 | .mr.developer.cfg 46 | .project 47 | .pydevproject 48 | 49 | # Rope 50 | .ropeproject 51 | 52 | # Django stuff: 53 | *.log 54 | *.pot 55 | 56 | # Sphinx documentation 57 | docs/_build/ 58 | 59 | # PyCharm 60 | .idea/ 61 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | The MIT License 2 | 3 | Copyright (c) 2011 Rodrigo S. Manhães 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in 13 | all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 21 | THE SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # IMPORTANT: This fork is no longer maintained. The official home of vdist can now be [found here](https://github.com/dante-signal31/vdist) 2 | 3 | --- 4 | Welcome to the home of vdist, a tool that lets you create OS packages from your Python applications in a clean and self contained manner. It uses [virtualenv](https://virtualenv.pypa.io/en/latest/), [Docker](https://www.docker.com/) and [fpm](https://github.com/jordansissel/fpm) under the hood, and it uses [Jinja2](http://jinja.pocoo.org/docs/dev/) to render its templates for each individual target OS. 5 | 6 | vdist is currently in alpha stage, but it should work just fine. Issues, feature requests, pull requests etc. are very welcome! 7 | 8 | The official documentation for vdist can be found on [ReadTheDocs](http://vdist.readthedocs.org/en/latest/) 9 | --- 10 | -------------------------------------------------------------------------------- /docs/buildenvironment.md: -------------------------------------------------------------------------------- 1 | ## Optimizing your build environment 2 | Using vdist out of the box would work fine if your context isn't all too 3 | demanding. When your demands are slightly higher (such as build speeds, 4 | continuous builds, etc.), I'd recommend getting a few things into place which 5 | can be used effectively in conjunction with vdist: 6 | - an internal Docker registry; it's easy to set up (through using Docker) 7 | 8 | - a private PyPI repository such as pypiserver or devpi; 9 | 10 | - a Continuous Integration system such as Jenkins/Bamboo/etc. 11 | 12 | - an internal OS package mirror (e.g. an APT or Yum mirror) 13 | 14 | vdist would then be used in the final stage of a CI build, where it would fire 15 | up a preprovisioned Docker image (that resides on your private Docker registry), 16 | build your project, installs your internal modules from your private PyPI 17 | repository, and leaves the resulting OS packages as deliveries for your CI 18 | system. 19 | 20 | To make a "fast" Docker image that can be used with vdist, make sure that: 21 | 22 | - it's up to date (ideally by using a Configuration Management system) 23 | 24 | - it includes an already compiled Python interpreter (not your system's 25 | interpreter, since we prefer not to be dependent on it) that installs in 26 | e.g. /opt/yourcompany 27 | 28 | - it already includes your build time dependencies (which does not mean you 29 | should leave them out of the `build_deps` parameter) 30 | 31 | - fpm is already installed (`gem install fpm` can take quite a while) 32 | 33 | Once you've created a custom Docker image, you can refer to it in your 34 | `profiles.json` like you would normally do when using Docker: 35 | ``` 36 | { 37 | "my-custom-profile": { 38 | "docker_image": "docker-internal.yourcompany.com:5000/yourcompany/yourbuildmachine:latest", 39 | "script": "debian.sh" 40 | } 41 | } 42 | ``` 43 | 44 | -------------------------------------------------------------------------------- /docs/howtocontribute.md: -------------------------------------------------------------------------------- 1 | ## How to contribute 2 | I would certainly appreciate your help! Issues, feature requests and pull 3 | requests are more than welcome. I'm guessing I would need much more effort 4 | creating more profiles, but any help is appreciated! 5 | -------------------------------------------------------------------------------- /docs/howtocustomize.md: -------------------------------------------------------------------------------- 1 | ## How to customize 2 | It could well be that in your specific case, you need different steps to be 3 | taken to get to a deployable package. vdist by default is a bit naive: it 4 | checks for a requirements.txt and installs it using pip, and it also checks 5 | for a setup.py, on which it runs an install when present. Your situation might 6 | be a bit different. To solve this, vdist offers the ability to create custom 7 | build profiles. First, create a directory called `buildprofiles` under your 8 | project directory (location can be overridden by setting the `profiles_dir` 9 | argument on the Builder instance). In this directory, you place a script called 10 | `profiles.json`. The `profiles.json` file might look like this: 11 | 12 | ``` 13 | { 14 | "centos7": { 15 | "docker_image": "yourcompany/centos7:latest", 16 | "script": "centos.sh" 17 | }, 18 | "debian": { 19 | "docker_image": "yourcompany/debian:latest", 20 | "script": "debian.sh" 21 | } 22 | } 23 | ``` 24 | 25 | This configuration file defines 2 profiles: `centos7` and `debian`. Each 26 | profile has 2 properties, called `docker_image` and `script`. The 27 | `docker_image` key indicates the name of the Docker image which will be 28 | pulled from the Docker registry by vdist. This can also be an image on 29 | your company's internal Docker registry, in which case the value for the 30 | `docker_image` property would look like 31 | "registry.company.internal:5000/user/project:version". 32 | The `script` key indicates what script to load on the build machine to 33 | actually execute the build process. These scripts are treated as 34 | [Jinja2 templates](http://jinja.pocoo.org/), and the build information 35 | you provide to vdist will be injected into these templates. 36 | 37 | By default, vdist provides the scripts `centos.sh` and `debian.sh` as 38 | generic templates for RHEL/CentOS/Fedora and Debian/Ubuntu based images, 39 | so you can use those when defining your profile. You can take a look at 40 | [the templates provided by vdist](https://github.com/objectified/vdist/tree/master/vdist/profiles) 41 | to get an idea of how they work, and how to create your own. Custom shell 42 | scripts can be put in the `buildprofiles` directory alongside your 43 | profiles.json file. All parameters that are given by you when calling 44 | `add_build()` are injected into the template, including a few more. You can 45 | refer to these scripts in your own custom profiles, while using your own 46 | Docker images. For example: your company provides a provisioned build image 47 | based on Debian (custom Python interpreter package on board, regularly 48 | maintained and all), and refers to "debian.sh" to perform the build. 49 | -------------------------------------------------------------------------------- /docs/howtoinstall.md: -------------------------------------------------------------------------------- 1 | Before installing vdist, it is important that you have the Docker daemon 2 | running. I recommend getting the latest version of Docker to use with vdist. 3 | 4 | Installing vdist is as easy as this: 5 | ``` 6 | $ pip install vdist 7 | ``` 8 | 9 | Alternatively, you can clone the source directly from Github and install its 10 | dependencies via pip. When doing that, I recommend using virtualenv. For 11 | example: 12 | 13 | ``` 14 | $ git clone https://github.com/objectified/vdist 15 | $ cd vdist 16 | $ virtualenv . 17 | $ . bin/activate 18 | $ pip install -r requirements.txt 19 | ``` 20 | -------------------------------------------------------------------------------- /docs/howtouse.md: -------------------------------------------------------------------------------- 1 | ## How to use 2 | Inside your project, there are a few basic prerequisites for vdist to work. 3 | 4 | 1. Create a requirements.txt ('pip freeze > requirements.txt' inside a 5 | virtualenv should give you a good start); you probably already have one 6 | 2. Create a small Python file that actually uses the vdist module 7 | 8 | Here is a minimal example of how to use vdist to create an OS package of 9 | "yourapp" for Ubuntu Trusty. Create a file called package.py, which would 10 | contain the following code: 11 | 12 | ```python 13 | from vdist.builder import Builder 14 | from vdist.source import git 15 | 16 | builder = Builder() 17 | 18 | builder.add_build( 19 | app='yourapp', 20 | version='1.0', 21 | source=git( 22 | uri='https://github.com/you/yourapp', 23 | branch='master' 24 | ), 25 | profile='ubuntu-trusty' 26 | ) 27 | 28 | builder.build() 29 | ``` 30 | 31 | Here is what it does: vdist will build an OS package called 'yourapp-1.0.deb' 32 | from a Git repo located at https://github.com/you/yourapp, from branch 'master' 33 | using the vdist profile 'ubuntu-trusty' (more on vdist profiles later). 34 | While doing so, it will download and compile a Python interpreter, set up a 35 | virtualenv for your application, and installs your application's dependencies 36 | into the virtualenv. The whole resulting virtualenv will be wrapped up in a 37 | package, and is the end result of the build run. Here's an example creating a 38 | build for two OS flavors at the same time: 39 | 40 | ```python 41 | from vdist.builder import Builder 42 | from vdist.source import git 43 | 44 | builder = Builder() 45 | 46 | # Add CentOS7 build 47 | builder.add_build( 48 | name='myproject :: centos7 build', 49 | app='myproject', 50 | version='1.0', 51 | source=git( 52 | uri='http://yourgithost.internal/yourcompany/yourproject', 53 | branch='master' 54 | ), 55 | profile='centos7' 56 | ) 57 | 58 | # Add Ubuntu build 59 | builder.add_build( 60 | name='myproject :: ubuntu trusty build', 61 | app='myproject', 62 | version='1.0', 63 | source=git( 64 | uri='http://yourgithost.internal/yourcompany/yourproject', 65 | branch='master' 66 | ), 67 | profile='ubuntu-trusty' 68 | ) 69 | 70 | builder.build() 71 | ``` 72 | 73 | If all goes well, running this file as a Python program will build two OS 74 | packages (an RPM for CentOS 7 and a .deb package for Ubuntu Trusty Tahr) 75 | for a project called "myproject". The two builds will be running in parallel 76 | threads, so you will see the build output of both threads at the same time, 77 | where the logging of each thread can be identified by the build name. 78 | Here's an explanation of the keyword arguments that can be given to 79 | `add_build()`: 80 | 81 | ### Required arguments: 82 | - `app` :: the name of the application to build; this should also equal the 83 | project name in Git, and is used as the prefix for the filename of the 84 | resulting package 85 | - `version` :: the version of the application; this is used when building the 86 | OS package both in the name and in its meta information 87 | - `profile` :: the name of the profile to use for this specific build; its 88 | value should be one of two things: 89 | * a vdist built-in profile (currently `centos7`, `ubuntu-trusty` and 90 | `debian-wheezy` are available) 91 | * a custom profile that you create yourself; see 92 | [How to customize](http://vdist.readthedocs.org/en/latest/howtocustomize) 93 | for instructions 94 | - `source` :: the argument that specifies how to get the source code to build 95 | from; the available source types are: 96 | * `git(uri=uri, branch=branch)`: this source type attempts to git clone by 97 | using the supplied arguments 98 | * `directory(path=path)`: this source type uses a local directory to build 99 | the project from, and uses no versioning data 100 | * `git_directory(path=path, branch=branch)`: this source type uses a git 101 | checkout in a local directory to build the project from; it checks out the 102 | supplied branch before building 103 | 104 | ### Optional arguments: 105 | - `name` :: the name of the build; this does not do anything in the build 106 | process itself, but is used in e.g. logs; when omitted, the build name is a 107 | sanitized combination of the `app`, `version` and `profile` arguments. 108 | - `build_deps` :: a list of build time dependencies; these are the names of 109 | the OS packages that need to be present on the build machine before setting 110 | up and building the project. 111 | - `runtime_deps` :: a list of run time dependencies; these names are given to 112 | the resulting OS package as dependencies, so that they act as prerequisites 113 | when installing the final OS package. 114 | - `custom_filename` :: specifies a custom filename to use when generating 115 | the OS package; within this filename, references to environment variables 116 | may be used when put in between curly braces 117 | (e.g. `foo-{ENV_VAR_ONE}-bar-{ENV_VAR_TWO}.deb`); this is useful when for 118 | example your CI system passes values such as the build number and so on. 119 | - `fpm_args` :: any extra arguments that are given to 120 | [fpm](https://github.com/jordansissel/fpm) when the actual package is being 121 | built. 122 | - `pip_args` :: any extra arguments that are given to pip when your pip 123 | requirements are being installed (a custom index url pointing to your private 124 | PyPI repository for example). 125 | - `package_install_root`:: Base directory were this package is going to be 126 | installed in target system (defaults to '*python_basedir*'). 127 | - `package_tmp_root` :: Temporal folder used in docker container to build your 128 | package (defaults to '*/tmp*'). 129 | - `working_dir` :: a subdirectory under your source tree that is to be regarded 130 | as the base directory; if set, only this directory is packaged, and the pip 131 | requirements are tried to be found here. This makes sense when you have a 132 | source repository with multiple projects under it. 133 | - `python_basedir` :: specifies one of two things: 1) where Python can be 134 | found (your company might have a prepackaged Python already installed on your 135 | custom docker container) 2) where vdist should install the compiled Python 136 | distribution on your docker container. Read vdist's various [use cases](usecases.md) 137 | to understand the nuance. Defaults to '*/opt*'. 138 | - `compile_python` :: indicates whether Python should be fetched from 139 | python.org, compiled and shipped for you; defaults to *True*. If not *True* then 140 | '*python_basedir*' should point to a python distribution already installed in 141 | docker container. 142 | - `python_version` :: specifies two things: 1) if '*compile_python*' is *True* 143 | then it means the exact python version that should be downloaded and compiled. 144 | 2) if '*compile_python*' is *False* then only mayor version number is considered 145 | (currently 2 or 3) and latest available python distribution of that mayor 146 | version is searched (in given '*python_basedir*' of your docker container) to be 147 | used. Defaults to '*2.7.9*'. 148 | - `requirements_path` :: the path to your pip requirements file, relative to 149 | your project root; this defaults to `*/requirements.txt*`. 150 | 151 | Here's another, more customized example. 152 | 153 | ```python 154 | import os 155 | 156 | from vdist.builder import Builder 157 | from vdist.source import directory 158 | 159 | # Instantiate the builder while passing it a custom location for 160 | # your profile definitions 161 | profiles_path = os.path.dirname(os.path.abspath(__file__)) 162 | 163 | builder = Builder(profiles_dir='%s/deploy/profiles' % profiles_path) 164 | 165 | # Add CentOS7 build 166 | builder.add_build( 167 | # Name of the build 168 | name='myproject :: centos6 build', 169 | 170 | # Name of the app (used for the package name) 171 | app='myproject', 172 | 173 | # The version; you might of course get this value from e.g. a file 174 | # or an environment variable set by your CI environment 175 | version='1.0', 176 | 177 | # Base the build on a directory; this would make sense when executing 178 | # vdist in the context of a CI environment 179 | source=directory(path='/home/ci/projects/myproject'), 180 | 181 | # Use the 'centos7' profile 182 | profile='centos7', 183 | 184 | # Do not compile Python during packaging, a custom Python interpreter is 185 | # already made available on the build machine 186 | compile_python=False, 187 | 188 | # The location of your custom Python interpreter as installed by an 189 | # OS package (usually from a private package repository) on your 190 | # docker container. 191 | python_basedir='/opt/yourcompany/python', 192 | # As python_version is not given, vdist is going to assume your custom 193 | # package is a Python 2 interpreter, so it will call 'python'. If your 194 | # package were a Python 3 interpreter then you should include a 195 | # python_version='3' in this configuration to make sure that vdist looks 196 | # for a 'python3' executable in 'python_basedir'. 197 | 198 | # Depend on an OS package called "yourcompany-python" which would contain 199 | # the Python interpreter; these are build dependencies, and are not 200 | # runtime dependencies. You docker container should be configured to reach 201 | # your private repository to get "yourcompany-python" package. 202 | build_deps=['yourcompany-python', 'gcc'], 203 | 204 | # Specify OS packages that should be installed when your application is 205 | # installed 206 | runtime_deps=['yourcompany-python', 'imagemagick', 'ffmpeg'], 207 | 208 | # Some extra arguments for fpm, in this case a postinstall script that 209 | # will run after your application will be installed (useful for e.g. 210 | # startup scripts, supervisor configs, etc.) 211 | fpm_args='--post-install deploy/centos7/postinstall.sh', 212 | 213 | # Extra arguments to use when your pip requirements file is being installed 214 | # by vdist; a URL to your private PyPI server, for example 215 | pip_args='--index-url https://pypi.yourcompany.com/simple/', 216 | 217 | # Find your pip requirements somewhere else instead of the project root 218 | requirements_path='deploy/requirements-prod.txt', 219 | 220 | # Specify a custom filename, including the values of environment variables 221 | # to build up the filename; these can be set by e.g. a CI system 222 | custom_filename='myapp-{GIT_TAG}-{CI_BUILD_NO}-{RELEASE_NAME}.deb' 223 | ) 224 | 225 | builder.build() 226 | ``` 227 | You can read some examples with the main vdist [use cases](usecases.md) we have 228 | identified. Additionally if you look in the 229 | [vdist examples directory](https://github.com/objectified/vdist/tree/master/examples), 230 | you will find even more examples. 231 | 232 | There are cases where you want to influence the way vdist behaves in your 233 | environment. This can be done by passing additional parameters to the vdist 234 | Builder constructor. Here's an example: 235 | 236 | ``` 237 | import os 238 | 239 | from vdist.builder import Builder 240 | from vdist.source import git 241 | 242 | profiles_dir = os.path.join(os.path.dirname(__file__), 'myprofiles') 243 | 244 | builder = Builder( 245 | profiles_dir=profiles_dir, 246 | machine_logs=False 247 | ) 248 | 249 | builder.add_build( 250 | app='myapp', 251 | source=git(uri='https://github.com/foo/myproject', branch='myrelease'), 252 | version='1.0', 253 | profile='ubuntu-trusty' 254 | ) 255 | 256 | builder.build() 257 | ``` 258 | 259 | In the above example, two things are customized for this build run: 260 | 261 | 1. vdist looks at a different directory for finding your custom profiles 262 | 263 | 2. the logging of what happens on the Docker image is turned off 264 | -------------------------------------------------------------------------------- /docs/index.md: -------------------------------------------------------------------------------- 1 | # vdist 2 | 3 | Welcome to the home of vdist, a tool that lets you create OS packages from your 4 | Python applications in a clean and self contained manner. 5 | It uses [virtualenv](https://virtualenv.pypa.io/en/latest/), 6 | [Docker](https://www.docker.com/) and [fpm](https://github.com/jordansissel/fpm) 7 | under the hood, and it uses [Jinja2](http://jinja.pocoo.org/docs/dev/) to render 8 | its templates (shell scripts) for each individual target OS. 9 | 10 | The source for vdist is available under the MIT license and can be found on 11 | [Github](https://github.com/objectified/vdist) 12 | 13 | vdist is currently in alpha stage, but it should work just fine. If you find any 14 | issues, please report issues or submit pull requests via Github. 15 | 16 | Here's a quickstart to give you an idea of how to use vdist, once you're set up. 17 | 18 | ``` 19 | from vdist.builder import Builder 20 | from vdist.source import git 21 | 22 | builder = Builder() 23 | 24 | builder.add_build( 25 | app='yourapp', 26 | version='1.0', 27 | source=git( 28 | uri='https://github.com/you/yourapp', 29 | branch='release-1.0' 30 | ), 31 | profile='ubuntu-trusty', 32 | build_deps=['libpq-dev'], 33 | runtime_deps=['libcurl3'] 34 | ) 35 | 36 | builder.build() 37 | ``` 38 | 39 | Running the above would do this: 40 | 41 | - set up a Docker container running Ubuntu Trusty Tahr 42 | 43 | - install the OS packages listed in `build_deps` (only libpq-dev in this case) 44 | 45 | - git clone the repository at https://github.com/you/yourapp 46 | 47 | - checkout the branch 'release-1.0' 48 | 49 | - set up a virtualenv for the checked out application 50 | 51 | - install your application's dependencies from requirements.txt if found in the 52 | checked out branch 53 | 54 | - wrap the virtualenv in a package called `yourapp-1.0.deb` which includes a 55 | dependency on the OS packages listed in `runtime_deps` 56 | 57 | 58 | Similarly, the same build for CentOS 6 would look like this. 59 | 60 | ``` 61 | from vdist.builder import Builder 62 | from vdist.source import git 63 | 64 | builder = Builder() 65 | 66 | builder.add_build( 67 | app='yourapp', 68 | version='1.0', 69 | source=git( 70 | uri='https://github.com/you/yourapp', 71 | branch='release-1.0' 72 | ), 73 | profile='centos7', 74 | build_deps=['postgresql-devel'], 75 | runtime_deps=['libcurl3'] 76 | ) 77 | 78 | builder.build() 79 | ``` 80 | 81 | Read more about what vdist can do 82 | [here](http://vdist.readthedocs.org/en/latest/howtouse/) 83 | -------------------------------------------------------------------------------- /docs/qanda.md: -------------------------------------------------------------------------------- 1 | ## Questions and Answers 2 | *Q: Can I use vdist without running my own OS package mirrors, Docker registry and PyPI index?* 3 | 4 | Yes you can. But your builds will be rather slow, and you won't have the advantages a private PyPI index will provide you with. It's not hard to set up, especially when using Docker. 5 | 6 | *Q: Updating the Docker image takes a long time for every build. How can I speed this up?* 7 | 8 | This will happen when you're using plain Docker images from the central Docker hub, which are probably not up to date with the latest packages. Also, vdist will try to install fpm as a ruby gem when it's not available yet. Installing ruby gems can take a long time. Next to this, vdist will by default compile a Python interpreter for you, which takes quite a while to build as well. For creating the most self contained builds with maximum speed, read the "Optimizing your build environment" section in the vdist documentation. 9 | 10 | *Q: Why do you compile a Python interpreter from scratch? Is the OS provided interpreter not good enough?* 11 | 12 | As with your module dependencies, you want to be sure that your application is as self contained as possible, using only those versions that you trust. The Python interpreter should be no exception to this. Also since we're using virtualenv, we need to make sure that the same interpreter that was used to set up the virtualenv can still be used at runtime. To have this guarantee, the interpreter is shipped. 13 | 14 | *Q: Can vdist be used to create Docker images for my application, instead of OS packages?* 15 | 16 | At the moment there is no builtin support for this, although I guess it could be done. When in high demand, I can look into it. 17 | 18 | *Q: Why didn't you just plug into setuptools/distutils, or an existing build tool like PyBuilder?* 19 | 20 | I could have done that, but it seemed that it would get a little messy. Having said that, I'm all ears when people want such a thing. 21 | 22 | *Q: How does vdist compare to [Conda](http://conda.pydata.org)?* 23 | 24 | Where vdist builds native OS packages for your target OS, Conda is a package management system. In that sense, Conda and vdist are fundamentally different things. 25 | 26 | *Q: How does vdist compare to [Python Wheels](https://wheel.readthedocs.org/en/latest/)?* 27 | 28 | If I'm not mistaken, wheels are primarily meant to create binary distributions of Python libraries. They won't necessarily help you with rolling out your applications. Wheels might be fetched from PyPI during a vdist build when available, (which is great, of course). 29 | 30 | *Q: Any future plans for vdist?* 31 | 32 | I have a few ideas, but I'm also very interested in hearing input for vdist from you. The following features jump to mind for now: 33 | 34 | - smoke testing the installation of the resulting OS packages on a fresh Docker image 35 | 36 | - the ability to commit a provisioned Docker image during the build run to a Docker registry 37 | 38 | - integrating with (for example) PyBuilder 39 | 40 | But like I said, I'm all open to ideas. 41 | 42 | -------------------------------------------------------------------------------- /docs/usecases.md: -------------------------------------------------------------------------------- 1 | ## Use cases 2 | There are many situations where you would use vdist. If your application only 3 | depended on packages from your system main repository then you could use 4 | [fpm](https://github.com/jordansissel/fpm) directly but chances are that when 5 | time passes further updates over your dependencies could break your application 6 | (this is usually known as 7 | [dependency hell](https://en.wikipedia.org/wiki/Dependency_hell)) 8 | 9 | That applies not only for external libraries your application uses but also for 10 | the specific python interpreter you need to run your application. 11 | 12 | To avoid *dependency hell* you should pack your application along with the 13 | specific version of its dependencies (packages and needed interpreter) so they 14 | are not affected by further updates. To create this bundle of your package and 15 | its dependencies is where vdist comes to play. 16 | 17 | Being aware of that, we have identified 4 main use cases for vdist: 18 | 19 | - **Scenario 1** :: Your application includes a *setup.py* to be installed over 20 | an specific compiled Python version. 21 | 22 | - **Scenario 2** :: Your application does not include a *setup.py* but still 23 | needs an specific compiled Python version. 24 | 25 | - **Scenario 3** :: Your application includes a *setup.py* but uses a prebuilt 26 | Python interpreter (maybe a custom Python package or the *vanilla* Python version 27 | available in docker container you use for building process). 28 | 29 | - **Scenario 4** :: Your application does not include a *setup.py* but uses a 30 | prebuilt Python interpreter (maybe a custom Python package or the *vanilla* 31 | Python version available in docker container you use for building process). 32 | 33 | Whether you have a *setup.py* file or not is autodetected by vdist. If you don't want 34 | to be in scenarios 1 or 3 just don't include *setup.py* file inside project given 35 | to vdist. 36 | 37 | ###Scenarios 1 and 2 38 | For scenarios 1 and 2 vdist is going to download from the official Python site 39 | the specific interpreter version you need and compile it. Finally vdist is 40 | going to install that compiled python interpreter in the folder (inside your 41 | docker container) you gave in *python_basedir* parameter. If *python_basedir* 42 | is not set, vdist installs compiled python in *app* folder inside */opt*. 43 | 44 | Afterwards, if your project includes a *setup.py* (**scenario 1**) it will be run 45 | (`python setup.py install`) using your compiled interpreter. That 46 | means that your application will end being installed in the *site_packages* 47 | folder of your compiled python distribution (if your setup.py creates entry 48 | points they will appear in the *bin* folder of your compiled python distribution). 49 | At packaging phase vdist will include only the compiled python folder inside the 50 | generated package. If you have an entry point you could prepare a post install 51 | script to link that entry point from your compiled python's *bin* folder to a 52 | system wide folder like */usr/bin*. To include that post install script in 53 | generated package just use *fpm_args* parameter. 54 | 55 | An example configurarion for scenario 1 could be: 56 | ```python 57 | builder_parameters = { 58 | "app": 'geolocate', 59 | "version": '1.3.0', 60 | "source": git( 61 | uri='https://github.com/dante-signal31/geolocate', 62 | branch='master' 63 | ), 64 | "profile": 'ubuntu-trusty', 65 | "compile_python": True, 66 | "python_version": '3.4.3', 67 | "fpm_args": '--maintainer dante.signal31@gmail.com -a native --url ' 68 | 'https://github.com/dante-signal31/geolocate --description ' 69 | '"This program accepts any text and searchs inside every IP' 70 | ' address. With each of those IP addresses, ' 71 | 'geolocate queries ' 72 | 'Maxmind GeoIP database to look for the city and ' 73 | 'country where' 74 | ' IP address or URL is located. Geolocate is designed to be' 75 | ' used in console with pipes and redirections along with ' 76 | 'applications like traceroute, nslookup, etc.' 77 | ' " --license BSD-3 --category net', 78 | "requirements_path": '/REQUIREMENTS.txt' 79 | } 80 | ``` 81 | This configuration would generate a package including only an */opt/geolocate* 82 | folder with compiled python interpreter and geolocate application installed in 83 | its *site-packages* folder. Entry points generated for geolocate will be placed 84 | in *bin* folder of the compiled interpreter. Generated entry points will use 85 | compiled intepreter and its packages so application will be entirely 86 | self-sufficient on target system. 87 | 88 | 89 | If your project doesn't include a *setup.py* (**scenario 2**) then your project 90 | files will be stored in a folder different of the compiled python one. In that 91 | case your application will end at *package_install_root* in a folder called 92 | like *app* parameter. If *package_install_root* is not set then it will be the same 93 | than *python_basedir*. That means that in this scenario vdist is going to include 94 | two folders inside generated package: one for the python interpreter and one 95 | for your application files. Your application files should be configured to use 96 | correct path to call your compiled python interpreter on target system. 97 | 98 | An example configurarion for scenario 2 could be: 99 | ```python 100 | builder_parameters = {"app": 'jtrouble', 101 | "version": '1.0.0', 102 | "source": git( 103 | uri='https://github.com/objectified/jtrouble', 104 | branch='master' 105 | ), 106 | "profile": 'ubuntu-trusty', 107 | "package_install_root": "/opt", 108 | "python_basedir": "/opt/python", 109 | "compile_python": True, 110 | "python_version": '3.4.3', } 111 | ``` 112 | This configuration would generate a package including two folders: the one with 113 | your application (*/opt/jtrouble*) and the one with your compiled python 114 | interpreter (*/opt/python*). 115 | 116 | ###Scenarios 3 and 4 117 | Scenarios 3 and 4 are respectively equivalent to scenarios 1 and 2. The main 118 | difference is that while scenarios 1 and 2 download an interpreter from the 119 | internet and compile it, scenarios 3 and 4 look for desired interpreter in the 120 | docker container file system so that interpreter should be installed before 121 | as a system package downloaded from a private repository or should be compiled 122 | and included by default in a custom docker conatiner image. If you are in a 123 | corporate enviroment you'll probably prefer this option because you'll probably 124 | have enough resources to have your own private system package repository and 125 | this way you can speed up greatly packaging process. 126 | 127 | Appart from that difference between scenarios 3 and 4 is the same than between 128 | scenarios 1 and 2. 129 | 130 | An example configuration for **scenario 3** could be: 131 | ```python 132 | builder_parameters = { 133 | "app": 'geolocate', 134 | "version": '1.3.0', 135 | "source": git( 136 | uri='https://github.com/dante-signal31/geolocate', 137 | branch='master' 138 | ), 139 | "profile": 'ubuntu-trusty', 140 | "compile_python": False, 141 | "python_version": '3.4.3', 142 | # Lets suppose custom python package is already installed and its root 143 | # folder is /usr. 144 | "python_basedir": '/usr', 145 | "fpm_args": '--maintainer dante.signal31@gmail.com -a native --url ' 146 | 'https://github.com/dante-signal31/geolocate --description ' 147 | '"This program accepts any text and searchs inside' 148 | ' every IP ' 149 | 'address. With each of those IP addresses, ' 150 | 'geolocate queries ' 151 | 'Maxmind GeoIP database to look for the city and ' 152 | 'country where' 153 | ' IP address or URL is located. Geolocate is designed to be' 154 | ' used in console with pipes and redirections along with ' 155 | 'applications like traceroute, nslookup, etc.' 156 | ' " --license BSD-3 --category net', 157 | "requirements_path": '/REQUIREMENTS.txt' 158 | } 159 | ``` 160 | Generated packaged will include just one folder: */usr/* 161 | 162 | An example configuration for **scenario 4** may be: 163 | ```python 164 | builder_parameters = { 165 | "app": 'jtrouble', 166 | "version": '1.0.0', 167 | "source": git( 168 | uri='https://github.com/objectified/jtrouble', 169 | branch='master' 170 | ), 171 | "profile": 'ubuntu-trusty', 172 | "compile_python": False, 173 | "python_version": '3.4.3', 174 | "python_basedir": '/usr', 175 | } 176 | ``` 177 | Generated package will include two folders: */usr* with python interpreter and 178 | */opt/app* folder with application inside. Note that */opt* is used for 179 | application because *package_install_root* is not set so defaul value is used. 180 | 181 | Please be aware that althoug examples for scenarios 3 and 4 include a very 182 | specific *python_version* vdist only use its major version number (currently 2 183 | or 3) to know whether call python or python3 executable at *python_basedir/bin* 184 | folder. 185 | 186 | 187 | -------------------------------------------------------------------------------- /docs/whatisvdist.md: -------------------------------------------------------------------------------- 1 | vdist (Virtualenv Distribute) is a tool that lets you build OS packages from 2 | your Python applications, while aiming to build an isolated environment for 3 | your Python project by utilizing [virtualenv](https://virtualenv.pypa.io/en/latest/). 4 | This means that your application will not depend on OS provided packages of 5 | Python modules, including their versions. The idea is largely inspired by 6 | [this article by Hynek Schlawack](https://hynek.me/articles/python-app-deployment-with-native-packages/), 7 | so vdist basically implements the ideas outlined there. 8 | 9 | In short, the following principles are the most important motivation behind 10 | vdist: 11 | 12 | - OS packages are a good idea; they make sure your application can be rolled 13 | out just like any other program, using the same tools (Salt, Ansible, Puppet, 14 | etc.) 15 | 16 | - never let your application depend on packages provided by your OS 17 | 18 | - build time dependencies are not runtime dependencies; no compilers etc. on 19 | your target system 20 | 21 | - running your internal OS package mirrors and private PyPI repositories is 22 | a good idea 23 | 24 | 25 | vdist takes an approach that's slightly different from the implementation 26 | examples found in the original article, but it's still very similar. 27 | 28 | The main objective of vdist is to create clean, self contained OS packages 29 | from Python applications. This means that OS packages created with vdist 30 | contain your application, all Python dependencies needed by your application, 31 | and a Python interpreter. Every time vdist is creating a build, it does so on 32 | a clean OS image where all *build time* OS dependencies are installed from 33 | scratch before your application is being packaged on top of it. This means 34 | that the build machine will always be reverted to its original, clean state. 35 | To facilitate this, vdist uses Docker containers. By using so called "profiles", 36 | it's fairly easy to use your own custom Docker containers to be used by vdist 37 | when your project builds. 38 | 39 | What vdist does is this: you create a Python file with some information about 40 | your project, vdist sets up a container for the specified target OS with the 41 | build time dependencies for the project, checks out your project inside the 42 | container, installs its Python dependencies in a virtualenv, optionally does 43 | some additional things and builds an OS package for you. This way, you know 44 | for sure that the Python modules needed by your application are installable 45 | on a clean installation of your target OS (including OS provided libraries, 46 | header files etc.) Also, you'll soon find out when something is missing on 47 | the target OS you want to deploy on. 48 | 49 | By default, vdist will also compile and install a fresh Python interpreter 50 | for you, to be used by your application. This interpreter is used to create 51 | the aforementioned virtualenv, and will also be used when you deploy the 52 | resulting OS package. This allows you to choose the Python interpreter you 53 | want to use, instead of being tied to the version that is shipped with the OS 54 | you're deploying on. 55 | 56 | Note that vdist is not meant to build Docker images for your project, it merely 57 | creates (nearly) self contained OS packages of your application, which you can 58 | then use to deploy on the target OS you told vdist to build for. Also, vdist is 59 | not meant to create OS packages from Python libraries (for example, the ones 60 | you find on PyPI), but for applications. For a reasonably clear distinction 61 | between the two, I suggest you read 62 | [this article by Donald Stufft](https://caremad.io/2013/07/setup-vs-requirement/). 63 | Of course, your application can definitely include a number of libraries at 64 | build time (e.g. your Django application might well include all sorts of Python 65 | modules, but you won't use your Django application as a library). 66 | 67 | For more information please refer to our [how to use](howtouse.md) and our 68 | [use cases](usecases.md) pages. 69 | 70 | 71 | -------------------------------------------------------------------------------- /examples/assemble_custom_reqspath.py: -------------------------------------------------------------------------------- 1 | """ 2 | This example shows how you can point to an alternative 3 | pip requirements file to use when building your OS package 4 | """ 5 | from vdist.builder import Builder 6 | from vdist.source import git 7 | 8 | builder = Builder() 9 | 10 | builder.add_build( 11 | name='my great project', 12 | app='myproject', 13 | version='1.0', 14 | source=git( 15 | uri='https://github.com/someuser/someproject', 16 | branch='your-release-branch' 17 | ), 18 | profile='ubuntu-trusty', 19 | # path will be used relative to your checkout directory 20 | # by default vdist will look for $checkout/requirements.txt 21 | requirements_path='/requirements/production.txt' 22 | ) 23 | 24 | builder.build() 25 | -------------------------------------------------------------------------------- /examples/assemble_directory.py: -------------------------------------------------------------------------------- 1 | from vdist.builder import Builder 2 | from vdist.source import directory, git_directory 3 | 4 | builder = Builder() 5 | 6 | # build from a local directory 7 | builder.add_build( 8 | name='my directory based build', 9 | app='myproject', 10 | version='1.0', 11 | source=directory( 12 | path='/home/user/dev/yourproject' 13 | ), 14 | profile='centos6' 15 | ) 16 | 17 | # or, build from a git repo *inside* a local directory 18 | builder.add_build( 19 | name='my directory based build', 20 | app='myproject', 21 | version='1.0', 22 | source=git_directory( 23 | path='/home/user/dev/anotherproject', 24 | branch='your-release-branch' 25 | ), 26 | profile='centos6' 27 | ) 28 | 29 | # .. and build them in parallel 30 | builder.build() 31 | -------------------------------------------------------------------------------- /examples/assemble_from_subdirectory.py: -------------------------------------------------------------------------------- 1 | """ 2 | This example shows how you can use vdist to use a subdirectory 3 | under your source tree to use as the base for your OS package 4 | You will still be able to use git branching to point to the right 5 | release, since vdist will first checkout the parent, and set apart 6 | the subdirectory after switching to the right branch 7 | """ 8 | from vdist.builder import Builder 9 | from vdist.source import git 10 | 11 | builder = Builder() 12 | 13 | builder.add_build( 14 | name='my great project', 15 | app='myproject', 16 | version='1.0', 17 | source=git( 18 | uri='https://github.com/someuser/someproject', 19 | branch='your-release-branch' 20 | ), 21 | profile='ubuntu-trusty', 22 | # specify 'subapp' as the working directory for this build; 23 | # this means that only the subapp directory will be built and 24 | # packaged 25 | # This also means that vdist will look for a pip requirements 26 | # file in this directory 27 | working_dir='subapp' 28 | ) 29 | 30 | builder.build() 31 | -------------------------------------------------------------------------------- /examples/assemble_local_pipconf.py: -------------------------------------------------------------------------------- 1 | """ 2 | This example shows how you can tell vdist to use 3 | your local pip configuration (~/.pip/pip.conf) when 4 | building. This is useful when you're using an internal 5 | PyPI index (highly recommended) 6 | """ 7 | from vdist.builder import Builder 8 | from vdist.source import git 9 | 10 | builder = Builder() 11 | 12 | builder.add_build( 13 | name='my great project', 14 | app='myproject', 15 | version='1.0', 16 | source=git( 17 | uri='https://github.com/someuser/someproject', 18 | branch='your-release-branch' 19 | ), 20 | profile='ubuntu-trusty', 21 | # this means that during the build run, vdist will 22 | # copy your pip config (~/.pip/pip.conf) to the Docker 23 | # container it uses to build the OS package, so that you 24 | # can use your internal modules as pip dependencies in 25 | # your requirements.txt / setup.py 26 | use_local_pip_conf=True 27 | ) 28 | 29 | builder.build() 30 | -------------------------------------------------------------------------------- /examples/assemble_parallel.py: -------------------------------------------------------------------------------- 1 | from vdist.builder import Builder 2 | from vdist.source import git 3 | 4 | builder = Builder() 5 | 6 | # add CentOS6 build 7 | builder.add_build( 8 | name='myproject centos6 build', 9 | app='myproject', 10 | version='1.0', 11 | source=git( 12 | uri='http://yourgithost.internal/yourcompany/yourproject', 13 | branch='master' 14 | ), 15 | profile='centos6' 16 | ) 17 | 18 | # add CentOS7 build 19 | builder.add_build( 20 | name='myproject centos7 build', 21 | app='myproject', 22 | version='1.0', 23 | source=git( 24 | uri='http://yourgithost.internal/yourcompany/yourproject', 25 | branch='master' 26 | ), 27 | profile='centos7' 28 | ) 29 | 30 | # add Ubuntu Trusty build 31 | builder.add_build( 32 | name='myproject ubuntu build', 33 | app='myproject', 34 | version='1.0', 35 | source=git( 36 | uri='http://yourgithost.internal/yourcompany/yourproject', 37 | branch='master' 38 | ), 39 | profile='ubuntu-trusty' 40 | ) 41 | 42 | # vdist will now build them all in parallel 43 | # on separate docker containers 44 | builder.build() 45 | -------------------------------------------------------------------------------- /examples/assemble_with_custom_python.py: -------------------------------------------------------------------------------- 1 | """ 2 | This example shows how you can use vdist with your 3 | own package of the Python interpreter; you might want 4 | this when you do not want vdist to compile Python 5 | every time your project builds, and you've got Python 6 | available as a custom OS package (something I'd highly 7 | recommend for cross platform stability and packaging 8 | speed) 9 | """ 10 | from vdist.builder import Builder 11 | from vdist.source import git 12 | 13 | 14 | builder = Builder() 15 | 16 | builder.add_build( 17 | name='my great project', 18 | app='myproject', 19 | version='1.0', 20 | source=git( 21 | uri='https://github.com/someuser/someproject', 22 | branch='your-release-branch' 23 | ), 24 | profile='ubuntu-trusty', 25 | # tell vdist not to compile Python 26 | compile_python=False, 27 | # point to the basedir your custom Python is installed in 28 | # vdist will look for $basedir/bin/python 29 | python_basedir='/opt/mycompany/python', 30 | # provide your custom Python OS package as a build dependency 31 | build_deps=['mycompany-python'], 32 | # provide your custom Python OS package as a runtime dependency 33 | runtime_deps=['mycompany-python'] 34 | ) 35 | 36 | builder.build() 37 | -------------------------------------------------------------------------------- /examples/assemble_with_dependencies.py: -------------------------------------------------------------------------------- 1 | from vdist.builder import Builder 2 | from vdist.source import git 3 | 4 | builder = Builder() 5 | 6 | builder.add_build( 7 | name='my project build', 8 | app='myproject', 9 | version='1.0', 10 | source=git( 11 | uri='https://github.com/someuser/someproject', 12 | branch='your-release-branch' 13 | ), 14 | profile='ubuntu-trusty', 15 | # Tell vdist that we need the OS packages "ffmpeg" and "imagemagick" 16 | # on the system this application gets deployed 17 | # These packages will be made dependencies of the resulting OS package 18 | runtime_deps=['ffmpeg', 'imagemagick'], 19 | # tell vdist that on the machine that is used for building this project, 20 | # we will need the OS packages "libimagemagick-dev" and 21 | # "libmysqlclient-dev" 22 | # before we start building 23 | build_deps=['libimagemagick-dev', 'libmysqlclient-dev'] 24 | ) 25 | 26 | builder.build() 27 | -------------------------------------------------------------------------------- /integration-tests/test_builder.py: -------------------------------------------------------------------------------- 1 | import os 2 | import re 3 | import subprocess 4 | import tempfile 5 | 6 | from vdist.builder import Builder 7 | from vdist.source import git, git_directory, directory 8 | 9 | DEB_COMPILE_FILTER = [r'[^\.]', r'\./$', r'\./usr/', r'\./opt/$'] 10 | DEB_NOCOMPILE_FILTER = [r'[^\.]', r'^\.\.', r'\./$', r'^\.$', r'\./opt/$'] 11 | 12 | 13 | def _read_deb_contents(deb_file_pathname): 14 | entries = os.popen("dpkg -c {0}".format(deb_file_pathname)).readlines() 15 | file_list = [entry.split()[-1] for entry in entries] 16 | return file_list 17 | 18 | 19 | def _read_rpm_contents(rpm_file_pathname): 20 | entries = os.popen("rpm -qlp {0}".format(rpm_file_pathname)).readlines() 21 | file_list = [entry.rstrip("\n") for entry in entries 22 | if entry.startswith("/")] 23 | return file_list 24 | 25 | 26 | def _purge_list(original_list, purgables): 27 | list_purged = [] 28 | for entry in original_list: 29 | entry_free_of_purgables = all(True if re.match(pattern, entry) is None 30 | else False 31 | for pattern in purgables) 32 | if entry_free_of_purgables: 33 | list_purged.append(entry) 34 | return list_purged 35 | 36 | 37 | def _call_builder(builder_parameters): 38 | builder = Builder() 39 | builder.add_build(**builder_parameters) 40 | builder.build() 41 | 42 | 43 | def _generate_rpm(builder_parameters, centos_version): 44 | _call_builder(builder_parameters) 45 | homedir = os.path.expanduser('~') 46 | filename_prefix = "-".join([builder_parameters["app"], 47 | builder_parameters["version"]]) 48 | rpm_filename_prefix = "-".join([builder_parameters["app"], 49 | builder_parameters["version"]]) 50 | target_file = os.path.join( 51 | homedir, 52 | '.vdist', 53 | 'dist', 54 | "".join([filename_prefix, "-{0}".format(centos_version)]), 55 | "".join([rpm_filename_prefix, '-1.x86_64.rpm']), 56 | ) 57 | assert os.path.isfile(target_file) 58 | assert os.path.getsize(target_file) > 0 59 | return target_file 60 | 61 | 62 | def _generate_deb(builder_parameters): 63 | _call_builder(builder_parameters) 64 | homedir = os.path.expanduser('~') 65 | filename_prefix = "-".join([builder_parameters["app"], 66 | builder_parameters["version"]]) 67 | deb_filename_prefix = "_".join([builder_parameters["app"], 68 | builder_parameters["version"]]) 69 | target_file = os.path.join( 70 | homedir, 71 | '.vdist', 72 | 'dist', 73 | "".join([filename_prefix, "-ubuntu-trusty"]), 74 | "".join([deb_filename_prefix, '_amd64.deb']), 75 | ) 76 | assert os.path.isfile(target_file) 77 | assert os.path.getsize(target_file) > 0 78 | return target_file 79 | 80 | 81 | def _get_purged_deb_file_list(deb_filepath, file_filter): 82 | file_list = _read_deb_contents(deb_filepath) 83 | file_list_purged = _purge_list(file_list, file_filter) 84 | return file_list_purged 85 | 86 | 87 | def test_generate_deb_from_git(): 88 | builder_parameters = {"app": 'vdist-test-generate-deb-from-git', 89 | "version": '1.0', 90 | "source": git( 91 | uri='https://github.com/objectified/vdist', 92 | branch='master' 93 | ), 94 | "profile": 'ubuntu-trusty'} 95 | _ = _generate_deb(builder_parameters) 96 | 97 | 98 | def generate_rpm_from_git(centos_version): 99 | builder_parameters = {"app": 'vdist-test-generate-rpm-from-git', 100 | "version": '1.0', 101 | "source": git( 102 | uri='https://github.com/objectified/vdist', 103 | branch='master' 104 | ), 105 | "profile": centos_version} 106 | _ = _generate_rpm(builder_parameters, centos_version) 107 | 108 | 109 | def test_generate_rpm_from_git_centos6(): 110 | generate_rpm_from_git("centos6") 111 | 112 | 113 | def test_generate_rpm_from_git_centos7(): 114 | generate_rpm_from_git("centos7") 115 | 116 | 117 | # Scenarios to test: 118 | # 1.- Project containing a setup.py and compiles Python -> only package the 119 | # whole Python basedir. 120 | # 2.- Project not containing a setup.py and compiles Python -> package both the 121 | # project dir and the Python basedir. 122 | # 3.- Project containing a setup.py and using a prebuilt Python package (e.g. 123 | # not compiling) -> package the custom Python basedir only 124 | # 4.- Project not containing a setup.py and using a prebuilt Python package -> 125 | # package both the project dir and the Python basedir 126 | # More info at: 127 | # https://github.com/objectified/vdist/pull/7#issuecomment-177818848 128 | 129 | 130 | # Scenario 1 - Project containing a setup.py and compiles Python -> only package 131 | # the whole Python basedir. 132 | def test_generate_deb_from_git_setup_compile(): 133 | builder_parameters = { 134 | "app": 'geolocate', 135 | "version": '1.3.0', 136 | "source": git( 137 | uri='https://github.com/dante-signal31/geolocate', 138 | branch='master' 139 | ), 140 | "profile": 'ubuntu-trusty', 141 | "compile_python": True, 142 | "python_version": '3.4.4', 143 | "fpm_args": '--maintainer dante.signal31@gmail.com -a native --url ' 144 | 'https://github.com/dante-signal31/geolocate --description ' 145 | '"This program accepts any text and searchs inside every IP' 146 | ' address. With each of those IP addresses, ' 147 | 'geolocate queries ' 148 | 'Maxmind GeoIP database to look for the city and ' 149 | 'country where' 150 | ' IP address or URL is located. Geolocate is designed to be' 151 | ' used in console with pipes and redirections along with ' 152 | 'applications like traceroute, nslookup, etc.' 153 | ' " --license BSD-3 --category net', 154 | "requirements_path": '/REQUIREMENTS.txt', 155 | "runtime_deps": ["libssl1.0.0", ] 156 | } 157 | target_file = _generate_deb(builder_parameters) 158 | file_list_purged = _get_purged_deb_file_list(target_file, 159 | DEB_COMPILE_FILTER) 160 | # At this point only a folder should remain if everything is correct. 161 | correct_install_path = "./opt/geolocate" 162 | assert all((True if correct_install_path in file_entry else False 163 | for file_entry in file_list_purged)) 164 | # Geolocate launcher should be in bin folder too. 165 | geolocate_launcher = "./opt/geolocate/bin/geolocate" 166 | assert geolocate_launcher in file_list_purged 167 | 168 | 169 | def generate_rpm_from_git_setup_compile(centos_version): 170 | builder_parameters = { 171 | "app": 'geolocate', 172 | "version": '1.3.0', 173 | "source": git( 174 | uri='https://github.com/dante-signal31/geolocate', 175 | branch='master' 176 | ), 177 | "profile": centos_version, 178 | "compile_python": True, 179 | "python_version": '3.4.4', 180 | "fpm_args": '--maintainer dante.signal31@gmail.com -a native --url ' 181 | 'https://github.com/dante-signal31/geolocate --description ' 182 | '"This program accepts any text and searchs inside every IP' 183 | ' address. With each of those IP addresses, ' 184 | 'geolocate queries ' 185 | 'Maxmind GeoIP database to look for the city and ' 186 | 'country where' 187 | ' IP address or URL is located. Geolocate is designed to be' 188 | ' used in console with pipes and redirections along with ' 189 | 'applications like traceroute, nslookup, etc.' 190 | ' " --license BSD-3 --category net', 191 | "requirements_path": '/REQUIREMENTS.txt', 192 | "runtime_deps": ["libssl1.0.0", ] 193 | } 194 | target_file = _generate_rpm(builder_parameters, centos_version) 195 | file_list = _read_rpm_contents(target_file) 196 | # At this point only a folder should remain if everything is correct. 197 | correct_install_path = "/opt/geolocate" 198 | assert all((True if correct_install_path in file_entry else False 199 | for file_entry in file_list)) 200 | # Geolocate launcher should be in bin folder too. 201 | geolocate_launcher = "/opt/geolocate/bin/geolocate" 202 | assert geolocate_launcher in file_list 203 | 204 | 205 | def test_generate_rpm_from_git_setup_compile_centos6(): 206 | generate_rpm_from_git_setup_compile("centos6") 207 | 208 | 209 | def test_generate_rpm_from_git_setup_compile_centos7(): 210 | generate_rpm_from_git_setup_compile("centos7") 211 | 212 | 213 | # Scenario 2.- Project not containing a setup.py and compiles Python -> package 214 | # both the project dir and the Python basedir 215 | def test_generate_deb_from_git_nosetup_compile(): 216 | builder_parameters = {"app": 'jtrouble', 217 | "version": '1.0.0', 218 | "source": git( 219 | uri='https://github.com/objectified/jtrouble', 220 | branch='master' 221 | ), 222 | "profile": 'ubuntu-trusty', 223 | "package_install_root": "/opt", 224 | "python_basedir": "/opt/python", 225 | "compile_python": True, 226 | "python_version": '3.4.4', } 227 | target_file = _generate_deb(builder_parameters) 228 | file_list_purged = _get_purged_deb_file_list(target_file, 229 | DEB_COMPILE_FILTER) 230 | # At this point only two folders should remain if everything is correct: 231 | # application folder and compiled interpreter folder. 232 | correct_folders = ["./opt/jtrouble", "./opt/python"] 233 | assert all((True if any(folder in file_entry for folder in correct_folders) 234 | else False 235 | for file_entry in file_list_purged)) 236 | assert any(correct_folders[0] in file_entry 237 | for file_entry in file_list_purged) 238 | assert any(correct_folders[1] in file_entry 239 | for file_entry in file_list_purged) 240 | 241 | 242 | def generate_rpm_from_git_nosetup_compile(centos_version): 243 | builder_parameters = {"app": 'jtrouble', 244 | "version": '1.0.0', 245 | "source": git( 246 | uri='https://github.com/objectified/jtrouble', 247 | branch='master' 248 | ), 249 | "profile": centos_version, 250 | "package_install_root": "/opt", 251 | "python_basedir": "/opt/python", 252 | "compile_python": True, 253 | "python_version": '3.4.4', } 254 | target_file = _generate_rpm(builder_parameters, centos_version) 255 | file_list = _read_rpm_contents(target_file) 256 | # At this point only two folders should remain if everything is correct: 257 | # application folder and compiled interpreter folder. 258 | correct_folders = ["/opt/jtrouble", "/opt/python"] 259 | assert all((True if any(folder in file_entry for folder in correct_folders) 260 | else False 261 | for file_entry in file_list)) 262 | assert any(correct_folders[0] in file_entry 263 | for file_entry in file_list) 264 | assert any(correct_folders[1] in file_entry 265 | for file_entry in file_list) 266 | 267 | 268 | def test_generate_rpm_from_git_nosetup_compile_centos6(): 269 | generate_rpm_from_git_nosetup_compile("centos6") 270 | 271 | 272 | def test_generate_rpm_from_git_nosetup_compile_centos7(): 273 | generate_rpm_from_git_nosetup_compile("centos7") 274 | 275 | 276 | # Scenario 3 - Project containing a setup.py and using a prebuilt Python package 277 | # (e.g. not compiling) -> package the custom Python basedir only. 278 | def test_generate_deb_from_git_setup_nocompile(): 279 | builder_parameters = { 280 | "app": 'geolocate', 281 | "version": '1.3.0', 282 | "source": git( 283 | uri='https://github.com/dante-signal31/geolocate', 284 | branch='master' 285 | ), 286 | "profile": 'ubuntu-trusty', 287 | "compile_python": False, 288 | "python_version": '3.4.4', 289 | # Lets suppose custom python package is already installed and its root 290 | # folder is /usr. Actually I'm using default installed python3 291 | # package, it's is going to be a huge package but this way don't 292 | # need a private package repository. 293 | "python_basedir": '/usr', 294 | "fpm_args": '--maintainer dante.signal31@gmail.com -a native --url ' 295 | 'https://github.com/dante-signal31/geolocate --description ' 296 | '"This program accepts any text and searchs inside' 297 | ' every IP ' 298 | 'address. With each of those IP addresses, ' 299 | 'geolocate queries ' 300 | 'Maxmind GeoIP database to look for the city and ' 301 | 'country where' 302 | ' IP address or URL is located. Geolocate is designed to be' 303 | ' used in console with pipes and redirections along with ' 304 | 'applications like traceroute, nslookup, etc.' 305 | ' " --license BSD-3 --category net', 306 | "requirements_path": '/REQUIREMENTS.txt', 307 | "runtime_deps": ["libssl1.0.0", ] 308 | } 309 | target_file = _generate_deb(builder_parameters) 310 | file_list_purged = _get_purged_deb_file_list(target_file, 311 | DEB_NOCOMPILE_FILTER) 312 | # At this point only a folder should remain if everything is correct. 313 | correct_install_path = "./usr" 314 | assert all((True if correct_install_path in file_entry else False 315 | for file_entry in file_list_purged)) 316 | # If python basedir was properly packaged then /usr/bin/python should be 317 | # there. 318 | python_interpreter = "./usr/bin/python2.7" 319 | assert python_interpreter in file_list_purged 320 | # If application was properly packaged then launcher should be in bin folder 321 | # too. 322 | geolocate_launcher = "./usr/bin/geolocate" 323 | assert geolocate_launcher in file_list_purged 324 | 325 | 326 | def generate_rpm_from_git_setup_nocompile(centos_version): 327 | builder_parameters = { 328 | "app": 'geolocate', 329 | "version": '1.3.0', 330 | "source": git( 331 | uri='https://github.com/dante-signal31/geolocate', 332 | branch='master' 333 | ), 334 | "profile": centos_version, 335 | "compile_python": False, 336 | "python_version": '3.4.4', 337 | # Lets suppose custom python package is already installed and its root 338 | # folder is /usr. Actually I'm using default installed python3 339 | # package, it's is going to be a huge package but this way don't 340 | # need a private package repository. 341 | "python_basedir": '/usr', 342 | "fpm_args": '--maintainer dante.signal31@gmail.com -a native --url ' 343 | 'https://github.com/dante-signal31/geolocate --description ' 344 | '"This program accepts any text and searchs inside' 345 | ' every IP ' 346 | 'address. With each of those IP addresses, ' 347 | 'geolocate queries ' 348 | 'Maxmind GeoIP database to look for the city and ' 349 | 'country where' 350 | ' IP address or URL is located. Geolocate is designed to be' 351 | ' used in console with pipes and redirections along with ' 352 | 'applications like traceroute, nslookup, etc.' 353 | ' " --license BSD-3 --category net', 354 | "requirements_path": '/REQUIREMENTS.txt', 355 | "runtime_deps": ["libssl1.0.0", ] 356 | } 357 | target_file = _generate_rpm(builder_parameters, centos_version) 358 | file_list = _read_rpm_contents(target_file) 359 | # At this point only a folder should remain if everything is correct. 360 | correct_install_path = "/usr" 361 | assert all((True if correct_install_path in file_entry else False 362 | for file_entry in file_list)) 363 | # If python basedir was properly packaged then /usr/bin/python should be 364 | # there. 365 | python_interpreter = "/usr/bin/python" 366 | assert python_interpreter in file_list 367 | # If application was properly packaged then launcher should be in bin folder 368 | # too. 369 | geolocate_launcher = "/usr/bin/geolocate" 370 | assert geolocate_launcher in file_list 371 | 372 | 373 | def test_generate_rpm_from_git_setup_nocompile_centos6(): 374 | generate_rpm_from_git_setup_nocompile("centos6") 375 | 376 | 377 | def test_generate_rpm_from_git_setup_nocompile_centos7(): 378 | generate_rpm_from_git_setup_nocompile("centos7") 379 | 380 | 381 | # Scenario 4.- Project not containing a setup.py and using a prebuilt Python 382 | # package -> package both the project dir and the Python basedir 383 | def test_generate_deb_from_git_nosetup_nocompile(): 384 | builder_parameters = { 385 | "app": 'jtrouble', 386 | "version": '1.0.0', 387 | "source": git( 388 | uri='https://github.com/objectified/jtrouble', 389 | branch='master' 390 | ), 391 | "profile": 'ubuntu-trusty', 392 | "compile_python": False, 393 | # Here happens the same than in 394 | # test_generate_deb_from_git_setup_nocompile() 395 | "python_version": '3.4.4', 396 | "python_basedir": '/usr', 397 | } 398 | target_file = _generate_deb(builder_parameters) 399 | file_list_purged = _get_purged_deb_file_list(target_file, 400 | DEB_NOCOMPILE_FILTER) 401 | # At this point only two folders should remain if everything is correct: 402 | # application folder and python basedir folder. 403 | correct_folders = ["./opt/jtrouble", "./usr"] 404 | assert all((True if any(folder in file_entry for folder in correct_folders) 405 | else False 406 | for file_entry in file_list_purged)) 407 | # If python basedir was properly packaged then /usr/bin/python should be 408 | # there. 409 | python_interpreter = "./usr/bin/python2.7" 410 | assert python_interpreter in file_list_purged 411 | 412 | 413 | def generate_rpm_from_git_nosetup_nocompile(centos_version): 414 | builder_parameters = { 415 | "app": 'jtrouble', 416 | "version": '1.0.0', 417 | "source": git( 418 | uri='https://github.com/objectified/jtrouble', 419 | branch='master' 420 | ), 421 | "profile": centos_version, 422 | "compile_python": False, 423 | # Here happens the same than in 424 | # test_generate_deb_from_git_setup_nocompile() 425 | "python_version": '3.4.4', 426 | "python_basedir": '/usr', 427 | } 428 | target_file = _generate_rpm(builder_parameters, centos_version) 429 | file_list = _read_rpm_contents(target_file) 430 | # At this point only two folders should remain if everything is correct: 431 | # application folder and python basedir folder. 432 | correct_folders = ["/opt/jtrouble", "/usr"] 433 | assert all((True if any(folder in file_entry for folder in correct_folders) 434 | else False 435 | for file_entry in file_list)) 436 | # If python basedir was properly packaged then /usr/bin/python should be 437 | # there. 438 | python_interpreter = "/usr/bin/python" 439 | assert python_interpreter in file_list 440 | 441 | 442 | def test_generate_rpm_from_git_nosetup_nocompile_centos6(): 443 | generate_rpm_from_git_nosetup_nocompile("centos6") 444 | 445 | 446 | def test_generate_rpm_from_git_nosetup_nocompile_centos7(): 447 | generate_rpm_from_git_nosetup_nocompile("centos7") 448 | 449 | 450 | def test_generate_deb_from_git_suffixed(): 451 | builder_parameters = {"app": 'vdist-test-generate-deb-from-git-suffixed', 452 | "version": '1.0', 453 | "source": git( 454 | uri='https://github.com/objectified/vdist.git', 455 | branch='master' 456 | ), 457 | "profile": 'ubuntu-trusty'} 458 | _ = _generate_deb(builder_parameters) 459 | 460 | 461 | def generate_rpm_from_git_suffixed(centos_version): 462 | builder_parameters = {"app": 'vdist-test-generate-deb-from-git-suffixed', 463 | "version": '1.0', 464 | "source": git( 465 | uri='https://github.com/objectified/vdist.git', 466 | branch='master' 467 | ), 468 | "profile": centos_version} 469 | _ = _generate_rpm(builder_parameters, centos_version) 470 | 471 | 472 | def test_generate_rpm_from_git_suffixed_centos6(): 473 | generate_rpm_from_git_suffixed("centos6") 474 | 475 | 476 | def test_generate_rpm_from_git_suffixed_centos7(): 477 | generate_rpm_from_git_suffixed("centos7") 478 | 479 | 480 | def test_generate_deb_from_git_directory(): 481 | tempdir = tempfile.gettempdir() 482 | checkout_dir = os.path.join(tempdir, 'vdist') 483 | 484 | git_p = subprocess.Popen( 485 | ['git', 'clone', 486 | 'https://github.com/objectified/vdist', 487 | checkout_dir]) 488 | git_p.communicate() 489 | 490 | builder_parameters = {"app": 'vdist-test-generate-deb-from-git-dir', 491 | "version": '1.0', 492 | "source": git_directory(path=checkout_dir, 493 | branch='master'), 494 | "profile": 'ubuntu-trusty'} 495 | _ = _generate_deb(builder_parameters) 496 | 497 | 498 | def generate_rpm_from_git_directory(centos_version): 499 | tempdir = tempfile.gettempdir() 500 | checkout_dir = os.path.join(tempdir, 'vdist') 501 | 502 | git_p = subprocess.Popen( 503 | ['git', 'clone', 504 | 'https://github.com/objectified/vdist', 505 | checkout_dir]) 506 | git_p.communicate() 507 | 508 | builder_parameters = {"app": 'vdist-test-generate-deb-from-git-dir', 509 | "version": '1.0', 510 | "source": git_directory(path=checkout_dir, 511 | branch='master'), 512 | "profile": centos_version} 513 | _ = _generate_rpm(builder_parameters, centos_version) 514 | 515 | 516 | def test_generate_rpm_from_git_directory_centos6(): 517 | generate_rpm_from_git_directory("centos6") 518 | 519 | 520 | def test_generate_rpm_from_git_directory_centos7(): 521 | generate_rpm_from_git_directory("centos7") 522 | 523 | 524 | def test_generate_deb_from_directory(): 525 | tempdir = tempfile.gettempdir() 526 | checkout_dir = os.path.join(tempdir, 'vdist') 527 | 528 | git_p = subprocess.Popen( 529 | ['git', 'clone', 530 | 'https://github.com/objectified/vdist', 531 | checkout_dir]) 532 | git_p.communicate() 533 | 534 | builder_parameters = {"app": 'vdist-test-generate-deb-from-dir', 535 | "version": '1.0', 536 | "source": directory(path=checkout_dir, ), 537 | "profile": 'ubuntu-trusty'} 538 | _ = _generate_deb(builder_parameters) 539 | 540 | 541 | def generate_rpm_from_directory(centos_version): 542 | tempdir = tempfile.gettempdir() 543 | checkout_dir = os.path.join(tempdir, 'vdist') 544 | 545 | git_p = subprocess.Popen( 546 | ['git', 'clone', 547 | 'https://github.com/objectified/vdist', 548 | checkout_dir]) 549 | git_p.communicate() 550 | 551 | builder_parameters = {"app": 'vdist-test-generate-deb-from-dir', 552 | "version": '1.0', 553 | "source": directory(path=checkout_dir, ), 554 | "profile": centos_version} 555 | _ = _generate_rpm(builder_parameters, centos_version) 556 | 557 | 558 | def test_generate_rpm_from_directory_centos6(): 559 | generate_rpm_from_directory("centos6") 560 | 561 | 562 | def test_generate_rpm_from_directory_centos7(): 563 | generate_rpm_from_directory("centos7") 564 | 565 | -------------------------------------------------------------------------------- /mkdocs.yml: -------------------------------------------------------------------------------- 1 | site_name: vdist 2 | theme: readthedocs 3 | pages: 4 | - [index.md, Home] 5 | - [whatisvdist.md, What is vdist] 6 | - [usecases.md, Use Cases] 7 | - [howtoinstall.md, How to install] 8 | - [howtouse.md, How to use] 9 | - [howtocustomize.md, How to customize] 10 | - [buildenvironment.md, Optimizing your build environment] 11 | - [qanda.md, Questions and Answers] 12 | - [howtocontribute.md, How to contribute] 13 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | --index-url https://pypi.python.org/simple/ 2 | 3 | -e . 4 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [metadata] 2 | description-file = README.md 3 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup, find_packages 2 | 3 | setup( 4 | name='vdist', 5 | version='0.6', 6 | description='Create OS packages from Python ' 7 | 'projects using Docker containers', 8 | long_description='Create OS packages from Python ' 9 | 'projects using Docker containers', 10 | author='objectified, dante-signal31', 11 | author_email='objectified@gmail.com, dante.signal31@gmail.com', 12 | license='MIT', 13 | url='https://github.com/dante-signal31/vdist', 14 | packages=find_packages(), 15 | install_requires=['jinja2==2.7.3'], 16 | package_data={'': ['internal_profiles.json', '*.sh']}, 17 | tests_require=['pytest'], 18 | classifiers=[ 19 | 'Development Status :: 3 - Alpha', 20 | 'Environment :: Console', 21 | 'Intended Audience :: Developers', 22 | 'Operating System :: MacOS :: MacOS X', 23 | 'Operating System :: Unix', 24 | 'Operating System :: POSIX', 25 | 'License :: OSI Approved :: MIT License', 26 | 'Programming Language :: Python :: 2.6', 27 | 'Programming Language :: Python :: 2.7', 28 | ], 29 | keywords='python docker deployment packaging', 30 | ) 31 | -------------------------------------------------------------------------------- /test-requirements.txt: -------------------------------------------------------------------------------- 1 | pytest==2.8.0 2 | -------------------------------------------------------------------------------- /tests/test_build_definitions.py: -------------------------------------------------------------------------------- 1 | from vdist.builder import Build 2 | from vdist.source import git, directory, git_directory 3 | 4 | 5 | def test_build_projectroot_from_uri(): 6 | build = Build( 7 | name='my build', 8 | app='myapp', 9 | version='1.0', 10 | source=git( 11 | uri='https://github.com/objectified/vdist', 12 | branch='release-1.0' 13 | ), 14 | profile='ubuntu-trusty' 15 | ) 16 | assert build.get_project_root_from_source() == 'vdist' 17 | 18 | 19 | def test_build_projectroot_from_uri_git_suffix(): 20 | build = Build( 21 | name='my build', 22 | app='myapp', 23 | version='1.0', 24 | source=git( 25 | uri='https://github.com/objectified/vdist.git', 26 | branch='release-1.0' 27 | ), 28 | profile='ubuntu-trusty' 29 | ) 30 | assert build.get_project_root_from_source() == 'vdist' 31 | 32 | 33 | def test_build_projectroot_from_directory(): 34 | build = Build( 35 | name='my build', 36 | app='myapp', 37 | version='1.0', 38 | source=directory(path='/var/tmp/vdist'), 39 | profile='ubuntu-trusty' 40 | ) 41 | assert build.get_project_root_from_source() == 'vdist' 42 | 43 | 44 | def test_build_projectroot_from_git_directory(): 45 | build = Build( 46 | name='my build', 47 | app='myapp', 48 | version='1.0', 49 | source=git_directory( 50 | path='/var/tmp/vdist', 51 | branch='release-1.0' 52 | ), 53 | profile='ubuntu-trusty' 54 | ) 55 | assert build.get_project_root_from_source() == 'vdist' 56 | 57 | 58 | def test_build_no_name(): 59 | build = Build( 60 | app='myapp', 61 | version='1.0', 62 | source=git_directory( 63 | path='/var/tmp/vdist', 64 | branch='release-1.0' 65 | ), 66 | profile='ubuntu-trusty' 67 | ) 68 | assert build.name == 'myapp-1.0-ubuntu-trusty' 69 | 70 | 71 | def test_build_get_safe_dirname(): 72 | build = Build( 73 | name='my build', 74 | app='myapp-foo @#^&_', 75 | version='1.0', 76 | source=git_directory( 77 | path='/var/tmp/vdist', 78 | branch='release-1.0' 79 | ), 80 | profile='ubuntu-trusty' 81 | ) 82 | assert build.get_safe_dirname() == 'myapp-foo______-1.0-ubuntu-trusty' 83 | -------------------------------------------------------------------------------- /tests/test_builder_setup.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | from vdist.builder import Builder, NoBuildsFoundException 4 | 5 | 6 | def test_builder_nobuilds(): 7 | b = Builder() 8 | 9 | with pytest.raises(NoBuildsFoundException): 10 | b.build() 11 | 12 | 13 | def test_internal_profile_loads(): 14 | b = Builder() 15 | 16 | profiles = b.get_available_profiles() 17 | 18 | internal_profile_ids = ['ubuntu-trusty', 'centos6', 'debian-wheezy'] 19 | 20 | for profile_id in internal_profile_ids: 21 | assert profile_id in profiles 22 | -------------------------------------------------------------------------------- /tests/test_buildprofile_definitions.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | from vdist.builder import BuildProfile 4 | 5 | 6 | def test_buildprofile_invalid_required_args(): 7 | with pytest.raises(AttributeError): 8 | BuildProfile() 9 | 10 | 11 | def test_buildprofile_valid_required_args(): 12 | m = BuildProfile( 13 | profile_id='some_profile_id', 14 | docker_image='some_docker_image', 15 | script='foo.sh' 16 | ) 17 | assert m.validate() 18 | 19 | 20 | def test_buildprofile_insecure_registry_arg(): 21 | m = BuildProfile( 22 | profile_id='some_profile_id', 23 | docker_image='some_docker_image', 24 | script='foo.sh', 25 | insecure_registry="true" 26 | ) 27 | assert m.insecure_registry is True 28 | 29 | m = BuildProfile( 30 | profile_id='some_profile_id', 31 | docker_image='some_docker_image', 32 | script='foo.sh' 33 | ) 34 | assert m.insecure_registry is False 35 | 36 | 37 | def test_buildprofile_invalid_arg(): 38 | with pytest.raises(AttributeError): 39 | BuildProfile( 40 | profile_id='some_profile_id', 41 | docker_image='some_docker_image', 42 | script='foo.sh', 43 | some_garbage='blah' 44 | ) 45 | -------------------------------------------------------------------------------- /tests/test_sources.py: -------------------------------------------------------------------------------- 1 | from vdist.source import git, directory, git_directory 2 | 3 | 4 | def test_source_type_git(): 5 | s = git(uri='https://github.com/objectified/vdist') 6 | assert s['uri'] == 'https://github.com/objectified/vdist' 7 | 8 | s = git(uri='https://github.com/objectified/vdist.git') 9 | assert s['uri'] == 'https://github.com/objectified/vdist' 10 | 11 | 12 | def test_source_type_directory(): 13 | s = directory(path='/foo/bar/baz/') 14 | assert s['path'] == '/foo/bar/baz' 15 | 16 | 17 | def test_source_type_git_directory(): 18 | s = git_directory(path='/foo/bar/', branch='foo') 19 | assert s['branch'] == 'foo' 20 | assert s['path'] == '/foo/bar' 21 | -------------------------------------------------------------------------------- /vdist/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/objectified/vdist/f2a82294144b1a624cbb32aac3a5ae6923ddde2a/vdist/__init__.py -------------------------------------------------------------------------------- /vdist/builder.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import os 3 | import shutil 4 | import re 5 | import json 6 | import threading 7 | 8 | from jinja2 import Environment, FileSystemLoader 9 | 10 | from vdist import defaults 11 | from vdist.buildmachine import BuildMachine 12 | 13 | 14 | class BuildProfile(object): 15 | 16 | def __init__(self, **kwargs): 17 | self.required_attrs = ['profile_id', 'docker_image', 'script'] 18 | self.optional_attrs = ['insecure_registry'] 19 | 20 | for arg in kwargs: 21 | if arg not in self.required_attrs and \ 22 | arg not in self.optional_attrs: 23 | raise AttributeError('attribute not allowed: %s' % arg) 24 | 25 | self.__dict__.update(kwargs) 26 | 27 | self.validate() 28 | 29 | if hasattr(self, 'insecure_registry') and \ 30 | self.insecure_registry == 'true': 31 | self.insecure_registry = True 32 | else: 33 | self.insecure_registry = False 34 | 35 | def validate(self): 36 | for attr in self.required_attrs: 37 | if not hasattr(self, attr): 38 | raise AttributeError( 39 | 'build profile misses attribute: %s' % attr) 40 | return True 41 | 42 | def __str__(self): 43 | return str(self.__dict__) 44 | 45 | 46 | class Build(object): 47 | 48 | def __init__(self, app, version, source, profile, 49 | name=None, use_local_pip_conf=False, build_deps=None, 50 | runtime_deps=None, custom_filename=None, 51 | fpm_args='', pip_args='', 52 | package_install_root=None, 53 | package_tmp_root=None, 54 | working_dir='', python_basedir=None, 55 | compile_python=True, 56 | python_version=defaults.PYTHON_VERSION, 57 | requirements_path='/requirements.txt'): 58 | self.app = app 59 | self.version = version.format(**os.environ) 60 | self.source = source 61 | self.use_local_pip_conf = use_local_pip_conf 62 | if package_install_root is None: 63 | self.package_install_root = defaults.PACKAGE_INSTALL_ROOT.format(**os.environ) 64 | else: 65 | self.package_install_root = package_install_root.format(**os.environ) 66 | if package_tmp_root is None: 67 | self.package_tmp_root = defaults.PACKAGE_TMP_ROOT.format(**os.environ) 68 | else: 69 | self.package_tmp_root = package_tmp_root.format(**os.environ) 70 | self.working_dir = working_dir.format(**os.environ) 71 | self.requirements_path = requirements_path.format(**os.environ) 72 | if python_basedir is None: 73 | self.python_basedir = "/".join([defaults.PYTHON_BASEDIR, app]).format(**os.environ) 74 | else: 75 | self.python_basedir = python_basedir.format(**os.environ) 76 | self.compile_python = compile_python 77 | self.python_version = python_version.format(**os.environ) 78 | if custom_filename: 79 | self.custom_filename = custom_filename.format(**os.environ) 80 | else: 81 | self.custom_filename = None 82 | 83 | self.build_deps = [] 84 | if build_deps: 85 | self.build_deps = build_deps 86 | 87 | self.runtime_deps = [] 88 | if runtime_deps: 89 | self.runtime_deps = runtime_deps 90 | 91 | self.profile = profile 92 | self.fpm_args = fpm_args.format(**os.environ) 93 | self.pip_args = pip_args.format(**os.environ) 94 | 95 | if not name: 96 | self.name = self.get_safe_dirname() 97 | else: 98 | self.name = name 99 | 100 | def __str__(self): 101 | return str(self.__dict__) 102 | 103 | def get_project_root_from_source(self): 104 | if self.source['type'] == 'git': 105 | return os.path.basename(self.source['uri'].rstrip('/')) 106 | if self.source['type'] in ['directory', 'git_directory']: 107 | return os.path.basename(self.source['path'].rstrip('/')) 108 | return '' 109 | 110 | def get_safe_dirname(self): 111 | return re.sub( 112 | '[^A-Za-z0-9\.\-]', 113 | '_', 114 | '-'.join( 115 | [self.app, self.version, self.profile] 116 | ) 117 | ) 118 | 119 | 120 | class Builder(object): 121 | 122 | def __init__( 123 | self, 124 | profiles_dir=defaults.LOCAL_PROFILES_DIR, 125 | machine_logs=True): 126 | logging.basicConfig(format='%(asctime)s %(levelname)s ' 127 | '[%(threadName)s] %(name)s %(message)s', 128 | level=logging.INFO) 129 | self.logger = logging.getLogger('Builder') 130 | 131 | self.build_basedir = defaults.BUILD_BASEDIR 132 | self.profiles = {} 133 | self.builds = [] 134 | 135 | self.machine_logs = machine_logs 136 | self.local_profiles_dir = profiles_dir 137 | 138 | def add_build(self, **kwargs): 139 | self.builds.append(Build(**kwargs)) 140 | 141 | def _create_vdist_dir(self): 142 | vdist_path = os.path.join(os.path.expanduser('~'), '.vdist') 143 | if not os.path.exists(vdist_path): 144 | self.logger.info('Creating: %s' % vdist_path) 145 | os.mkdir(vdist_path) 146 | 147 | def _add_profiles_from_file(self, config_file): 148 | with open(config_file) as f: 149 | profiles = json.loads(f.read()) 150 | 151 | for profile_id in profiles: 152 | profile = BuildProfile( 153 | profile_id=profile_id, 154 | docker_image=profiles[profile_id]['docker_image'], 155 | script=profiles[profile_id]['script'], 156 | insecure_registry=profiles[profile_id].get( 157 | 'insecure_registry', 'false') 158 | ) 159 | 160 | self.profiles[profile_id] = profile 161 | 162 | def _load_profiles(self): 163 | internal_profiles = os.path.join( 164 | os.path.dirname(__file__), 165 | 'profiles', 'internal_profiles.json') 166 | self._add_profiles_from_file(internal_profiles) 167 | 168 | local_profiles = os.path.join( 169 | self.local_profiles_dir, defaults.LOCAL_PROFILES_FILE) 170 | if os.path.isfile(local_profiles): 171 | self._add_profiles_from_file(local_profiles) 172 | 173 | def _render_template(self, build): 174 | internal_template_dir = os.path.join( 175 | os.path.dirname(__file__), 'profiles') 176 | 177 | local_template_dir = os.path.abspath(self.local_profiles_dir) 178 | 179 | env = Environment(loader=FileSystemLoader( 180 | [internal_template_dir, local_template_dir])) 181 | 182 | if build.profile not in self.profiles: 183 | raise BuildProfileNotFoundException( 184 | 'profile not found: %s' % build.profile) 185 | 186 | profile = self.profiles[build.profile] 187 | template_name = profile.script 188 | template = env.get_template(template_name) 189 | 190 | scratch_dir = os.path.join( 191 | defaults.SHARED_DIR, 192 | defaults.SCRATCH_DIR 193 | ) 194 | 195 | # local uid and gid are needed to correctly set permissions 196 | # on the created artifacts after the build completes 197 | return template.render( 198 | local_uid=os.getuid(), 199 | local_gid=os.getgid(), 200 | project_root=build.get_project_root_from_source(), 201 | shared_dir=defaults.SHARED_DIR, 202 | scratch_dir=scratch_dir, 203 | **build.__dict__ 204 | ) 205 | 206 | def _clean_build_basedir(self): 207 | if os.path.exists(self.build_basedir): 208 | shutil.rmtree(self.build_basedir) 209 | 210 | def _create_build_basedir(self): 211 | os.mkdir(self.build_basedir) 212 | 213 | @staticmethod 214 | def _write_build_script(path, script): 215 | with open(path, 'w+') as f: 216 | f.write(script) 217 | os.chmod(path, 0o777) 218 | 219 | def _populate_scratch_dir(self, scratch_dir, build): 220 | # write rendered build script to scratch dir 221 | self._write_build_script( 222 | os.path.join(scratch_dir, defaults.SCRATCH_BUILDSCRIPT_NAME), 223 | self._render_template(build) 224 | ) 225 | 226 | # copy local ~/.pip if necessary 227 | if build.use_local_pip_conf: 228 | shutil.copytree( 229 | os.path.join(os.path.expanduser('~'), '.pip'), 230 | os.path.join(scratch_dir, '.pip') 231 | ) 232 | 233 | # local source type, copy local dir to scratch dir 234 | if build.source['type'] in ['directory', 'git_directory']: 235 | if not os.path.exists(build.source['path']): 236 | raise ValueError( 237 | 'path does not exist: %s' % build.source['path']) 238 | else: 239 | subdir = os.path.basename(build.source['path']) 240 | shutil.copytree( 241 | build.source['path'].rstrip('/'), 242 | os.path.join(scratch_dir, subdir) 243 | ) 244 | 245 | def _create_build_dir(self, build): 246 | subdir_name = build.get_safe_dirname() 247 | 248 | build_dir = os.path.join(self.build_basedir, subdir_name) 249 | 250 | if os.path.exists(build_dir): 251 | shutil.rmtree(build_dir) 252 | 253 | os.mkdir(build_dir) 254 | 255 | # create "scratch" subdirectory for stuff needed at build time 256 | scratch_dir = os.path.join(build_dir, defaults.SCRATCH_DIR) 257 | os.mkdir(scratch_dir) 258 | 259 | # write necessary stuff to scratch_dir 260 | self._populate_scratch_dir(scratch_dir, build) 261 | 262 | return build_dir 263 | 264 | def run_build(self, build): 265 | profile = self.profiles[build.profile] 266 | 267 | build_dir = self._create_build_dir(build) 268 | 269 | self.logger.info('launching docker image: %s' % profile.docker_image) 270 | 271 | build_machine = BuildMachine( 272 | machine_logs=self.machine_logs, 273 | image=profile.docker_image, 274 | insecure_registry=profile.insecure_registry 275 | ) 276 | 277 | self.logger.info('Running build machine for: %s' % build.name) 278 | build_machine.launch(build_dir=build_dir) 279 | 280 | self.logger.info('Shutting down build machine: %s' % build.name) 281 | build_machine.shutdown() 282 | 283 | self.logger.info('*** Resulting OS packages are in: %s ***' % build_dir) 284 | 285 | def get_available_profiles(self): 286 | self._load_profiles() 287 | return self.profiles 288 | 289 | def build(self): 290 | self._create_vdist_dir() 291 | self._load_profiles() 292 | self._clean_build_basedir() 293 | self._create_build_basedir() 294 | 295 | if len(self.builds) < 1: 296 | raise NoBuildsFoundException() 297 | 298 | threads = [] 299 | 300 | for build in self.builds: 301 | t = threading.Thread( 302 | name=build.name, 303 | target=self.run_build, 304 | args=(build,) 305 | ) 306 | threads.append(t) 307 | t.start() 308 | 309 | for t in threads: 310 | t.join() 311 | 312 | 313 | class BuildProfileNotFoundException(Exception): 314 | pass 315 | 316 | 317 | class TemplateNotFoundException(Exception): 318 | pass 319 | 320 | 321 | class NoBuildsFoundException(Exception): 322 | pass 323 | -------------------------------------------------------------------------------- /vdist/buildmachine.py: -------------------------------------------------------------------------------- 1 | import itertools 2 | import logging 3 | import os 4 | import subprocess 5 | 6 | from vdist import defaults 7 | 8 | 9 | class BuildMachine(object): 10 | 11 | def __init__(self, machine_logs=True, image=None, insecure_registry=False, 12 | docker_cli='docker'): 13 | self.logger = logging.getLogger('BuildMachine') 14 | 15 | self.machine_logs = machine_logs 16 | self.image = image 17 | 18 | self.container_id = None 19 | 20 | self.docker_cli = docker_cli 21 | 22 | self.insecure_registry = insecure_registry 23 | 24 | def _run_cli(self, cmd): 25 | self.logger.info('Running command: "%s"' % cmd) 26 | p = subprocess.Popen( 27 | cmd, 28 | shell=True, 29 | stdin=subprocess.PIPE, 30 | stdout=subprocess.PIPE, 31 | stderr=subprocess.PIPE 32 | ) 33 | 34 | media = [p.stdout, p.stderr] 35 | first_line = self._read_from_media(media) 36 | 37 | p.stdout.close() 38 | p.stderr.close() 39 | p.wait() 40 | 41 | return first_line 42 | 43 | def _read_from_media(self, media): 44 | first_line = None 45 | for input_to_read in media: 46 | for line in iter(input_to_read.readline, b''): 47 | line = str(line.decode("UTF-8")).strip() 48 | if not first_line: 49 | first_line = line 50 | self.logger.info(line) 51 | return first_line 52 | 53 | @staticmethod 54 | def _binds_to_shell_volumes(binds): 55 | if defaults.PYTHON3_INTERPRETER: 56 | vol_list = ['-v %s:%s' % (k, v) for k, v in binds.items()] 57 | else: 58 | vol_list = ['-v %s:%s' % (k, v) for k, v in binds.iteritems()] 59 | return ' '.join(vol_list) 60 | 61 | def launch(self, build_dir, extra_binds=None): 62 | binds = {build_dir: defaults.SHARED_DIR} 63 | if extra_binds: 64 | binds = list(itertools.chain(binds.items(), extra_binds.items())) 65 | path_to_command = os.path.join( 66 | defaults.SHARED_DIR, 67 | defaults.SCRATCH_DIR, 68 | defaults.SCRATCH_BUILDSCRIPT_NAME 69 | ) 70 | self.logger.info('Starting container: %s' % self.image) 71 | self.container_id = self._run_cli( 72 | '%s run -d -ti %s %s bash' % 73 | (self.docker_cli, 74 | self._binds_to_shell_volumes(binds), 75 | self.image)) 76 | 77 | self._run_cli( 78 | '%s exec %s %s' % 79 | (self.docker_cli, self.container_id, path_to_command)) 80 | 81 | def shutdown(self): 82 | self.logger.info('Stopping container: %s' % self.container_id) 83 | self._run_cli('%s stop %s' % (self.docker_cli, self.container_id)) 84 | 85 | self.logger.info('Removing container: %s' % self.container_id) 86 | self._run_cli('%s rm -f %s' % (self.docker_cli, self.container_id)) 87 | -------------------------------------------------------------------------------- /vdist/defaults.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | 4 | PYTHON_BASEDIR = '/opt' 5 | PYTHON_VERSION = '2.7.9' 6 | LOCAL_PROFILES_DIR = 'buildprofiles' 7 | LOCAL_PROFILES_FILE = 'profiles.json' 8 | VDIST_USERDIR = os.path.join(os.path.expanduser('~'), '.vdist') 9 | BUILD_BASEDIR = os.path.join(VDIST_USERDIR, 'dist') 10 | SCRATCH_BUILDSCRIPT_NAME = 'buildscript.sh' 11 | SCRATCH_DIR = 'scratch' 12 | SHARED_DIR = '/work' 13 | PACKAGE_INSTALL_ROOT = PYTHON_BASEDIR 14 | PACKAGE_TMP_ROOT = '/tmp' 15 | 16 | PYTHON3_INTERPRETER = True if sys.version_info[0] == 3 else False 17 | -------------------------------------------------------------------------------- /vdist/profiles/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/objectified/vdist/f2a82294144b1a624cbb32aac3a5ae6923ddde2a/vdist/profiles/__init__.py -------------------------------------------------------------------------------- /vdist/profiles/centos.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -x 2 | PYTHON_VERSION="{{python_version}}" 3 | PYTHON_BASEDIR="{{python_basedir}}" 4 | 5 | # fail on error 6 | set -e 7 | 8 | # install general prerequisites 9 | yum -y update 10 | yum install -y ruby-devel curl libyaml-devel which tar rpm-build rubygems git python-setuptools zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel epel-release 11 | yum -y install python34 12 | curl -O https://bootstrap.pypa.io/get-pip.py 13 | /usr/bin/python3 get-pip.py 14 | yum groupinstall -y "Development Tools" 15 | 16 | # install build dependencies needed for this specific build 17 | {% if build_deps %} 18 | yum install -y {{build_deps|join(' ')}} 19 | {% endif %} 20 | 21 | # only install when needed, to save time with 22 | # pre-provisioned containers 23 | if [ ! -f /usr/bin/fpm ]; then 24 | gem install fpm 25 | fi 26 | 27 | # install prerequisites 28 | easy_install virtualenv 29 | 30 | {% if compile_python %} 31 | # compile and install python 32 | cd /var/tmp 33 | curl -O https://www.python.org/ftp/python/$PYTHON_VERSION/Python-$PYTHON_VERSION.tgz 34 | tar xzvf Python-$PYTHON_VERSION.tgz 35 | cd Python-$PYTHON_VERSION 36 | ./configure --prefix=$PYTHON_BASEDIR 37 | make && make install 38 | {% endif %} 39 | 40 | if [ ! -d {{package_tmp_root}} ]; then 41 | mkdir -p {{package_tmp_root}} 42 | fi 43 | 44 | cd {{package_tmp_root}} 45 | 46 | {% if source.type == 'git' %} 47 | 48 | git clone {{source.uri}} 49 | cd {{project_root}} 50 | git checkout {{source.branch}} 51 | 52 | {% elif source.type in ['directory', 'git_directory'] %} 53 | 54 | cp -r {{scratch_dir}}/{{project_root}} . 55 | cd {{package_tmp_root}}/{{project_root}} 56 | 57 | {% if source.type == 'git_directory' %} 58 | git checkout {{source.branch}} 59 | {% endif %} 60 | 61 | {% else %} 62 | 63 | echo "invalid source type, exiting." 64 | exit 1 65 | 66 | {% endif %} 67 | 68 | {% if use_local_pip_conf %} 69 | cp -r {{scratch_dir}}/.pip ~ 70 | {% endif %} 71 | 72 | # when working_dir is set, assume that is the base and remove the rest 73 | {% if working_dir %} 74 | mv {{working_dir}} {{package_tmp_root}} && rm -rf {{package_tmp_root}}/{{project_root}} 75 | cd {{package_tmp_root}}/{{working_dir}} 76 | 77 | # reset project_root 78 | {% set project_root = working_dir %} 79 | {% endif %} 80 | 81 | # brutally remove virtualenv stuff from the current directory 82 | rm -rf bin include lib local 83 | 84 | if [[ ${PYTHON_VERSION:0:1} == "2" ]]; then 85 | PYTHON_BIN="$PYTHON_BASEDIR/bin/python" 86 | PIP_BIN="$PYTHON_BASEDIR/bin/pip" 87 | $PYTHON_BIN -m ensurepip 88 | else 89 | PYTHON_BIN="$PYTHON_BASEDIR/bin/python3" 90 | PIP_BIN="$PYTHON_BASEDIR/bin/pip3" 91 | fi 92 | 93 | if [ -f "$PWD{{requirements_path}}" ]; then 94 | $PIP_BIN install -U pip setuptools 95 | virtualenv -p $PYTHON_BIN . 96 | source bin/activate 97 | $PIP_BIN install {{pip_args}} -r $PWD{{requirements_path}} 98 | fi 99 | 100 | if [ -f "setup.py" ]; then 101 | $PYTHON_BIN setup.py install 102 | built=true 103 | else 104 | built=false 105 | fi 106 | 107 | cd / 108 | 109 | # get rid of VCS info 110 | find {{package_tmp_root}} -type d -name '.git' -print0 | xargs -0 rm -rf 111 | find {{package_tmp_root}} -type d -name '.svn' -print0 | xargs -0 rm -rf 112 | 113 | if $built; then 114 | {% if custom_filename %} 115 | fpm -s dir -t rpm -n {{app}} -p {{package_tmp_root}}/{{custom_filename}} -v {{version}} {% for dep in runtime_deps %} --depends {{dep}} {% endfor %} {{fpm_args}} $PYTHON_BASEDIR 116 | {% else %} 117 | fpm -s dir -t rpm -n {{app}} -p {{package_tmp_root}} -v {{version}} {% for dep in runtime_deps %} --depends {{dep}} {% endfor %} {{fpm_args}} $PYTHON_BASEDIR 118 | {% endif %} 119 | cp {{package_tmp_root}}/*rpm {{shared_dir}} 120 | else 121 | mkdir -p {{package_install_root}}/{{app}} 122 | cp -r {{package_tmp_root}}/{{app}}/* {{package_install_root}}/{{app}}/. 123 | {% if custom_filename %} 124 | fpm -s dir -t rpm -n {{app}} -p {{package_tmp_root}}/{{custom_filename}} -v {{version}} {% for dep in runtime_deps %} --depends {{dep}} {% endfor %} {{fpm_args}} {{package_install_root}}/{{project_root}} $PYTHON_BASEDIR 125 | {% else %} 126 | fpm -s dir -t rpm -n {{app}} -p {{package_tmp_root}} -v {{version}} {% for dep in runtime_deps %} --depends {{dep}} {% endfor %} {{fpm_args}} {{package_install_root}}/{{project_root}} $PYTHON_BASEDIR 127 | {% endif %} 128 | cp {{package_tmp_root}}/*rpm {{shared_dir}} 129 | fi 130 | 131 | chown -R {{local_uid}}:{{local_gid}} {{shared_dir}} 132 | -------------------------------------------------------------------------------- /vdist/profiles/centos6.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -x 2 | PYTHON_VERSION="{{python_version}}" 3 | PYTHON_BASEDIR="{{python_basedir}}" 4 | CONTAINER_PYTHON3_VERSION="5" 5 | 6 | # fail on error 7 | set -e 8 | 9 | # install general prerequisites 10 | yum -y update 11 | yum groupinstall -y "Development Tools" 12 | yum install -y ruby-devel curl libyaml-devel which tar rpm-build rubygems git python-setuptools zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel gcc gcc-c++ 13 | yum install -y yum-utils 14 | 15 | # Python 3 RPM installation to get basic support in that Python version. 16 | # Idea taken from: http://stackoverflow.com/questions/8087184/problems-installing-python3-on-rhel 17 | yum install -y https://centos6.iuscommunity.org/ius-release.rpm 18 | yum install -y python3${CONTAINER_PYTHON3_VERSION}u python3${CONTAINER_PYTHON3_VERSION}u-pip 19 | ln -s /usr/bin/python3.$CONTAINER_PYTHON3_VERSION /usr/bin/python3 20 | ln -s /usr/bin/pip3.$CONTAINER_PYTHON3_VERSION /usr/bin/pip3 21 | 22 | # install build dependencies needed for this specific build 23 | {% if build_deps %} 24 | yum install -y {{build_deps|join(' ')}} 25 | {% endif %} 26 | 27 | # only install when needed, to save time with 28 | # pre-provisioned containers 29 | if [ ! -f /usr/bin/fpm ]; then 30 | # Latest ruby 1.5.0 fails to install in centos 6. 31 | # For more info read: https://github.com/jordansissel/fpm/issues/1090 32 | # So force to 1.4.0 needed. 33 | gem install fpm --version 1.4.0 34 | fi 35 | 36 | # install prerequisites 37 | easy_install virtualenv 38 | 39 | {% if compile_python %} 40 | # compile and install python 41 | cd /var/tmp 42 | curl -O https://www.python.org/ftp/python/$PYTHON_VERSION/Python-$PYTHON_VERSION.tgz 43 | tar xzvf Python-$PYTHON_VERSION.tgz 44 | cd Python-$PYTHON_VERSION 45 | # Configure fails if folder in rpath doesn't exists before. 46 | # Creating it, even empty, before configure seems to solve issue. 47 | # More info in: 48 | # http://koansys.com/tech/building-python-with-enable-shared-in-non-standard-location 49 | mkdir -p ${PYTHON_BASEDIR}/lib 50 | ./configure --prefix=$PYTHON_BASEDIR --enable-shared LDFLAGS="-Wl,-rpath ${PYTHON_BASEDIR}/lib" 51 | make 52 | make altinstall 53 | PYTHON_MAIN_VERSION=${PYTHON_VERSION:0:3} 54 | if [[ ${PYTHON_VERSION:0:1} == "2" ]]; then 55 | ln -s $PYTHON_BASEDIR/bin/python$PYTHON_MAIN_VERSION $PYTHON_BASEDIR/bin/python 56 | # At this point pip does not exists yet so we're creating a dead link 57 | # but later we are going to install pip through ensurepip module 58 | # so this is going to be fixed. 59 | ln -s $PYTHON_BASEDIR/bin/pip$PYTHON_MAIN_VERSION $PYTHON_BASEDIR/bin/pip 60 | else 61 | ln -s $PYTHON_BASEDIR/bin/python$PYTHON_MAIN_VERSION $PYTHON_BASEDIR/bin/python3 62 | ln -s $PYTHON_BASEDIR/bin/pip$PYTHON_MAIN_VERSION $PYTHON_BASEDIR/bin/pip3 63 | fi 64 | {% endif %} 65 | 66 | if [ ! -d {{package_tmp_root}} ]; then 67 | mkdir -p {{package_tmp_root}} 68 | fi 69 | 70 | cd {{package_tmp_root}} 71 | 72 | {% if source.type == 'git' %} 73 | 74 | git clone {{source.uri}} 75 | cd {{project_root}} 76 | git checkout {{source.branch}} 77 | 78 | {% elif source.type in ['directory', 'git_directory'] %} 79 | 80 | cp -r {{scratch_dir}}/{{project_root}} . 81 | cd {{package_tmp_root}}/{{project_root}} 82 | 83 | {% if source.type == 'git_directory' %} 84 | git checkout {{source.branch}} 85 | {% endif %} 86 | 87 | {% else %} 88 | 89 | echo "invalid source type, exiting." 90 | exit 1 91 | 92 | {% endif %} 93 | 94 | {% if use_local_pip_conf %} 95 | cp -r {{scratch_dir}}/.pip ~ 96 | {% endif %} 97 | 98 | # when working_dir is set, assume that is the base and remove the rest 99 | {% if working_dir %} 100 | mv {{working_dir}} {{package_tmp_root}} && rm -rf {{package_tmp_root}}/{{project_root}} 101 | cd {{package_tmp_root}}/{{working_dir}} 102 | 103 | # reset project_root 104 | {% set project_root = working_dir %} 105 | {% endif %} 106 | 107 | # brutally remove virtualenv stuff from the current directory 108 | rm -rf bin include lib local 109 | 110 | if [[ ${PYTHON_VERSION:0:1} == "2" ]]; then 111 | PYTHON_BIN="$PYTHON_BASEDIR/bin/python" 112 | PIP_BIN="$PYTHON_BASEDIR/bin/pip" 113 | $PYTHON_BIN -m ensurepip 114 | else 115 | PYTHON_BIN="$PYTHON_BASEDIR/bin/python3" 116 | PIP_BIN="$PYTHON_BASEDIR/bin/pip3" 117 | fi 118 | 119 | if [ -f "$PWD{{requirements_path}}" ]; then 120 | $PIP_BIN install -U pip setuptools 121 | virtualenv -p $PYTHON_BIN . 122 | source bin/activate 123 | $PIP_BIN install {{pip_args}} -r $PWD{{requirements_path}} 124 | fi 125 | 126 | if [ -f "setup.py" ]; then 127 | $PYTHON_BIN setup.py install 128 | built=true 129 | else 130 | built=false 131 | fi 132 | 133 | cd / 134 | 135 | # get rid of VCS info 136 | find {{package_tmp_root}} -type d -name '.git' -print0 | xargs -0 rm -rf 137 | find {{package_tmp_root}} -type d -name '.svn' -print0 | xargs -0 rm -rf 138 | 139 | if $built; then 140 | {% if custom_filename %} 141 | fpm -s dir -t rpm -n {{app}} -p {{package_tmp_root}}/{{custom_filename}} -v {{version}} {% for dep in runtime_deps %} --depends {{dep}} {% endfor %} {{fpm_args}} $PYTHON_BASEDIR 142 | {% else %} 143 | fpm -s dir -t rpm -n {{app}} -p {{package_tmp_root}} -v {{version}} {% for dep in runtime_deps %} --depends {{dep}} {% endfor %} {{fpm_args}} $PYTHON_BASEDIR 144 | {% endif %} 145 | cp {{package_tmp_root}}/*rpm {{shared_dir}} 146 | else 147 | mkdir -p {{package_install_root}}/{{app}} 148 | cp -r {{package_tmp_root}}/{{app}}/* {{package_install_root}}/{{app}}/. 149 | {% if custom_filename %} 150 | fpm -s dir -t rpm -n {{app}} -p {{package_tmp_root}}/{{custom_filename}} -v {{version}} {% for dep in runtime_deps %} --depends {{dep}} {% endfor %} {{fpm_args}} {{package_install_root}}/{{project_root}} $PYTHON_BASEDIR 151 | {% else %} 152 | fpm -s dir -t rpm -n {{app}} -p {{package_tmp_root}} -v {{version}} {% for dep in runtime_deps %} --depends {{dep}} {% endfor %} {{fpm_args}} {{package_install_root}}/{{project_root}} $PYTHON_BASEDIR 153 | {% endif %} 154 | cp {{package_tmp_root}}/*rpm {{shared_dir}} 155 | fi 156 | 157 | chown -R {{local_uid}}:{{local_gid}} {{shared_dir}} 158 | -------------------------------------------------------------------------------- /vdist/profiles/debian.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -x 2 | PYTHON_VERSION="{{python_version}}" 3 | PYTHON_BASEDIR="{{python_basedir}}" 4 | 5 | # fail on error 6 | set -e 7 | 8 | # install fpm 9 | apt-get update 10 | apt-get install ruby-dev build-essential git python-virtualenv curl libssl-dev libsqlite3-dev libgdbm-dev libreadline-dev libbz2-dev libncurses5-dev tk-dev python3 python3-pip -y 11 | 12 | # only install when needed, to save time with 13 | # pre-provisioned containers 14 | if [ ! -f /usr/bin/fpm ]; then 15 | gem install fpm 16 | fi 17 | 18 | # install build dependencies 19 | {% if build_deps %} 20 | apt-get install -y {{build_deps|join(' ')}} 21 | {% endif %} 22 | 23 | {% if compile_python %} 24 | apt-get build-dep python -y 25 | apt-get install libssl-dev -y 26 | 27 | # compile and install python 28 | cd /var/tmp 29 | curl -O https://www.python.org/ftp/python/$PYTHON_VERSION/Python-$PYTHON_VERSION.tgz 30 | tar xzvf Python-$PYTHON_VERSION.tgz 31 | cd Python-$PYTHON_VERSION 32 | ./configure --prefix=$PYTHON_BASEDIR --with-ensurepip=install 33 | make && make install 34 | {% endif %} 35 | 36 | if [ ! -d {{package_tmp_root}} ]; then 37 | mkdir -p {{package_tmp_root}} 38 | fi 39 | 40 | cd {{package_tmp_root}} 41 | 42 | {% if source.type == 'git' %} 43 | 44 | git clone {{source.uri}} 45 | cd {{project_root}} 46 | git checkout {{source.branch}} 47 | 48 | {% elif source.type in ['directory', 'git_directory'] %} 49 | 50 | cp -r {{scratch_dir}}/{{project_root}} . 51 | cd {{package_tmp_root}}/{{project_root}} 52 | 53 | {% if source.type == 'git_directory' %} 54 | git checkout {{source.branch}} 55 | {% endif %} 56 | 57 | {% else %} 58 | 59 | echo "invalid source type, exiting." 60 | exit 1 61 | 62 | 63 | {% endif %} 64 | 65 | {% if use_local_pip_conf %} 66 | cp -r {{scratch_dir}}/.pip ~ 67 | {% endif %} 68 | 69 | # when working_dir is set, assume that is the base and remove the rest 70 | {% if working_dir %} 71 | mv {{working_dir}} {{package_tmp_root}} && rm -rf {{package_tmp_root}}/{{project_root}} 72 | cd {{package_tmp_root}}/{{working_dir}} 73 | 74 | # reset project_root 75 | {% set project_root = working_dir %} 76 | {% endif %} 77 | 78 | # brutally remove virtualenv stuff from the current directory 79 | rm -rf bin include lib local 80 | 81 | if [[ ${PYTHON_VERSION:0:1} == "2" ]]; then 82 | PYTHON_BIN="$PYTHON_BASEDIR/bin/python" 83 | PIP_BIN="$PYTHON_BASEDIR/bin/pip" 84 | else 85 | PYTHON_BIN="$PYTHON_BASEDIR/bin/python3" 86 | PIP_BIN="$PYTHON_BASEDIR/bin/pip3" 87 | fi 88 | 89 | if [ -f "$PWD{{requirements_path}}" ]; then 90 | $PIP_BIN install -U pip setuptools 91 | virtualenv -p $PYTHON_BIN . 92 | source bin/activate 93 | $PIP_BIN install {{pip_args}} -r $PWD{{requirements_path}} 94 | fi 95 | 96 | if [ -f "setup.py" ]; then 97 | $PYTHON_BIN setup.py install 98 | built=true 99 | else 100 | built=false 101 | fi 102 | 103 | cd / 104 | 105 | # get rid of VCS info 106 | find {{package_tmp_root}} -type d -name '.git' -print0 | xargs -0 rm -rf 107 | find {{package_tmp_root}} -type d -name '.svn' -print0 | xargs -0 rm -rf 108 | 109 | if $built; then 110 | {% if custom_filename %} 111 | fpm -s dir -t deb -n {{app}} -p {{package_tmp_root}}/{{custom_filename}} -v {{version}} {% for dep in runtime_deps %} --depends {{dep}} {% endfor %} {{fpm_args}} $PYTHON_BASEDIR 112 | {% else %} 113 | fpm -s dir -t deb -n {{app}} -p {{package_tmp_root}} -v {{version}} {% for dep in runtime_deps %} --depends {{dep}} {% endfor %} {{fpm_args}} $PYTHON_BASEDIR 114 | {% endif %} 115 | cp {{package_tmp_root}}/*deb {{shared_dir}} 116 | else 117 | mkdir -p {{package_install_root}}/{{app}} 118 | cp -r {{package_tmp_root}}/{{app}}/* {{package_install_root}}/{{app}}/. 119 | {% if custom_filename %} 120 | fpm -s dir -t deb -n {{app}} -p {{package_tmp_root}}/{{custom_filename}} -v {{version}} {% for dep in runtime_deps %} --depends {{dep}} {% endfor %} {{fpm_args}} {{package_install_root}}/{{project_root}} $PYTHON_BASEDIR 121 | {% else %} 122 | fpm -s dir -t deb -n {{app}} -p {{package_tmp_root}} -v {{version}} {% for dep in runtime_deps %} --depends {{dep}} {% endfor %} {{fpm_args}} {{package_install_root}}/{{project_root}} $PYTHON_BASEDIR 123 | {% endif %} 124 | cp {{package_tmp_root}}/*deb {{shared_dir}} 125 | fi 126 | 127 | chown -R {{local_uid}}:{{local_gid}} {{shared_dir}} 128 | -------------------------------------------------------------------------------- /vdist/profiles/internal_profiles.json: -------------------------------------------------------------------------------- 1 | { 2 | "centos7": { 3 | "docker_image": "centos:centos7", 4 | "script": "centos.sh" 5 | }, 6 | "centos6": { 7 | "docker_image": "centos:centos6", 8 | "script": "centos6.sh" 9 | }, 10 | "ubuntu-trusty": { 11 | "docker_image": "ubuntu:trusty", 12 | "script": "debian.sh" 13 | }, 14 | "debian-wheezy": { 15 | "docker_image": "debian:wheezy", 16 | "script": "debian.sh" 17 | } 18 | } 19 | -------------------------------------------------------------------------------- /vdist/source.py: -------------------------------------------------------------------------------- 1 | def git(uri=None, branch='master'): 2 | if uri.endswith('.git'): 3 | uri = uri[:-4] 4 | return dict(type='git', uri=uri, branch=branch) 5 | 6 | 7 | def directory(path=None): 8 | path = path.rstrip('/') 9 | return dict(type='directory', path=path) 10 | 11 | 12 | def git_directory(path=None, branch='master'): 13 | path = path.rstrip('/') 14 | return dict(type='git_directory', path=path, branch=branch) 15 | --------------------------------------------------------------------------------