├── demo ├── .gitignore ├── src │ ├── demo-a.sh │ └── demo-b.sh ├── builder-support │ ├── dockerfiles │ │ ├── Dockerfile.target.centos-10 │ │ ├── Dockerfile.target.centos-7 │ │ ├── Dockerfile.rpmtest │ │ ├── Dockerfile.target.rocky-9 │ │ ├── Dockerfile.target.rocky-8 │ │ ├── Dockerfile.target.sdist │ │ └── Dockerfile.rpmbuild │ ├── post-build │ ├── install-tests │ │ ├── test-exec.sh │ │ └── check-installed-files.sh │ ├── usage.include.txt │ ├── specs │ │ ├── a.spec │ │ └── b.spec │ ├── post-build-test │ └── vendor-specs │ │ └── lua-argparse.spec ├── prepare.sh └── README.md ├── .gitignore ├── templating ├── testdata │ ├── test-template-include2.txt │ ├── test-template-include1.txt │ ├── test-expected.txt │ └── test-template.txt ├── test-templating.sh └── templating.sh ├── helpers ├── set-configure-ac-version.sh ├── buildrequires-from-specs ├── list-rock-contents.lua ├── generate-yum-provenance.py ├── generate-deb-provenance.py ├── generate-dnf-provenance.py ├── build-debs.sh ├── build-rocks.sh ├── build-specs.sh └── functions.sh ├── Dockerfile-kaniko ├── tests ├── test-rocky-8-reproducible.sh ├── test-rocky-9-reproducible.sh └── test_versioning.sh ├── LICENSE ├── .github └── workflows │ └── tests.yml ├── docker-cleanup.sh ├── gen-version └── README.md /demo/.gitignore: -------------------------------------------------------------------------------- 1 | /builder 2 | -------------------------------------------------------------------------------- /demo/src/demo-a.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | echo "This is demo A" 3 | -------------------------------------------------------------------------------- /demo/src/demo-b.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | echo "This is demo B" 3 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | /tmp 2 | /cache 3 | *.pyc 4 | *.tmp 5 | .DS_Store 6 | .*.sw? 7 | -------------------------------------------------------------------------------- /demo/builder-support/dockerfiles/Dockerfile.target.centos-10: -------------------------------------------------------------------------------- 1 | # bogus target for sort testing 2 | -------------------------------------------------------------------------------- /demo/builder-support/post-build: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | echo "*** This is builder-support/post-build" 4 | -------------------------------------------------------------------------------- /templating/testdata/test-template-include2.txt: -------------------------------------------------------------------------------- 1 | [Second include start] 2 | @EXEC echo "Hello world!" 3 | [Second include end] 4 | -------------------------------------------------------------------------------- /templating/testdata/test-template-include1.txt: -------------------------------------------------------------------------------- 1 | [Include file 1 start] 2 | @EVAL value of FOO in include is also $FOO 3 | [Include file 1 end] 4 | -------------------------------------------------------------------------------- /helpers/set-configure-ac-version.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | sed -E -i -e "s/AC_INIT[[( ]+([^]]+).*/AC_INIT([\1], [${BUILDER_VERSION}])/" configure.ac 4 | -------------------------------------------------------------------------------- /demo/builder-support/install-tests/test-exec.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | set -ex 4 | 5 | /usr/bin/demo-a | grep 'demo A' 6 | /usr/bin/demo-b | grep 'demo B' 7 | 8 | -------------------------------------------------------------------------------- /demo/builder-support/usage.include.txt: -------------------------------------------------------------------------------- 1 | Modules (by default all are built): 2 | a - demo module A 3 | b - demo module B 4 | vendor - vendor packages 5 | 6 | -------------------------------------------------------------------------------- /templating/test-templating.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -e 4 | 5 | cd testdata 6 | 7 | if ! diff -u test-expected.txt <(../templating.sh test-template.txt) ; then 8 | echo 9 | echo "FAILED" 10 | exit 1 11 | fi 12 | 13 | echo "PASSED" 14 | 15 | -------------------------------------------------------------------------------- /demo/prepare.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Prepare demo 3 | # This copies the builder from the parent dir to a builder/ subdir. 4 | # This is needed, because Docker will not include symlinked parent directories into the 5 | # build context. 6 | 7 | set -x 8 | mkdir builder/ 9 | git tag 0.1.42 10 | rsync -rv --exclude .git --exclude demo --exclude tmp --exclude cache --exclude tests ../ builder/ 11 | 12 | set +x 13 | echo 14 | echo "DONE. Now run ./builder/build.sh" 15 | -------------------------------------------------------------------------------- /demo/README.md: -------------------------------------------------------------------------------- 1 | # Builder demo 2 | 3 | This simple demo serves both as a demonstration of how to use the builder 4 | and as a test case for builder development. 5 | 6 | To run the build, run this from the current folder: 7 | 8 | ./prepare.sh 9 | ./builder/build.sh -B MYCOOLARG=iLikeTests 10 | 11 | The prepare script copies the builder files into builder/. Generally 12 | you would use git submodules instead, but this does not work for here 13 | as we keep the demo inside the same repository. 14 | -------------------------------------------------------------------------------- /Dockerfile-kaniko: -------------------------------------------------------------------------------- 1 | FROM alpine:3.11 2 | 3 | ENV DOCKER_CONFIG='/kaniko/.docker' 4 | ENV GOOGLE_APPLICATION_CREDENTIALS='/kaniko/.docker/config.json' 5 | ENV PATH=/kaniko:$PATH 6 | ENV SSL_CERT_DIR=/kaniko/ssl/certs 7 | 8 | RUN apk --no-cache add \ 9 | bash \ 10 | git \ 11 | grep \ 12 | openssh-client \ 13 | perl \ 14 | rsync \ 15 | sed \ 16 | tree 17 | 18 | COPY --from=gcr.io/kaniko-project/executor /kaniko /kaniko 19 | 20 | ENTRYPOINT ["/kaniko/executor"] 21 | -------------------------------------------------------------------------------- /demo/builder-support/install-tests/check-installed-files.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | failed=0 4 | 5 | check_file() { 6 | [ ! -f "$1" ] && echo "FAILED: missing file: $1" && failed=1 7 | } 8 | check_exec() { 9 | if [ ! -f "$1" ]; then 10 | echo "FAILED: missing executable file: $1" 11 | failed=1 12 | elif [ ! -x "$1" ]; then 13 | echo "FAILED: file not executable: $1" 14 | failed=1 15 | fi 16 | } 17 | 18 | check_exec /usr/bin/demo-a 19 | check_exec /usr/bin/demo-b 20 | 21 | [ "$failed" = "1" ] && exit 1 22 | exit 0 23 | -------------------------------------------------------------------------------- /helpers/buildrequires-from-specs: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """Parses a spec file and extracts all build dependencies""" 3 | 4 | import sys 5 | 6 | builddeps = set() 7 | 8 | for specfile in sys.argv: 9 | with open(specfile, 'r', encoding='utf-8') as f: 10 | for line in f: 11 | if line.startswith('BuildRequires:'): 12 | _, values = line.split(':', 1) 13 | for value in values.split(','): 14 | value = value.strip() 15 | name = value.split(' ')[0] # ignore version 16 | builddeps.add(name) 17 | 18 | for name in sorted(builddeps): 19 | print(name) 20 | -------------------------------------------------------------------------------- /helpers/list-rock-contents.lua: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env lua5.1 2 | -- Build helper that extracts a list of required files from a rockspec. 3 | -- The builder uses this list to create the source tarball. 4 | 5 | local rockspecpath = arg[1] 6 | local rockspec = assert(loadfile(rockspecpath)) 7 | -- This will set globals 8 | rockspec() 9 | 10 | if build then 11 | if build.modules then 12 | for name,path in pairs(build.modules) do 13 | print(path) 14 | end 15 | end 16 | if build.install then 17 | for type,spec in pairs(build.install) do 18 | for name,path in pairs(spec) do 19 | print(path) 20 | end 21 | end 22 | end 23 | end 24 | -------------------------------------------------------------------------------- /tests/test-rocky-8-reproducible.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Test if rocky-8 RPM builds are reproducible 3 | # Must be run from demo dir 4 | 5 | set -ex 6 | 7 | # First build 8 | ./builder/build.sh -B MYCOOLARG=iLikeTests rocky-8 9 | 10 | # Record hashes 11 | sha256sum \ 12 | builder/tmp/latest/rocky-8/dist/noarch/*.rpm \ 13 | builder/tmp/latest/sdist/*.tar.gz \ 14 | > /tmp/sha256sum.txt 15 | 16 | # Second build after cleaning and adding a file to invalidate the build context 17 | rm -rf ./builder/tmp/latest/rocky-8 18 | rm -rf ./builder/tmp/latest/sdist 19 | ./builder/build.sh -B MYCOOLARG=iLikeTests -b build-again rocky-8 20 | 21 | # Check hashes, should be identical 22 | sha256sum -c /tmp/sha256sum.txt 23 | 24 | -------------------------------------------------------------------------------- /tests/test-rocky-9-reproducible.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Test if rocky-9 RPM builds are reproducible 3 | # Must be run from demo dir 4 | 5 | set -ex 6 | 7 | # First build 8 | ./builder/build.sh -B MYCOOLARG=iLikeTests rocky-9 9 | 10 | # Record hashes 11 | sha256sum \ 12 | builder/tmp/latest/rocky-9/dist/noarch/*.rpm \ 13 | builder/tmp/latest/sdist/*.tar.gz \ 14 | > /tmp/sha256sum.txt 15 | 16 | # Second build after cleaning and adding a file to invalidate the build context 17 | rm -rf ./builder/tmp/latest/rocky-9 18 | rm -rf ./builder/tmp/latest/sdist 19 | ./builder/build.sh -B MYCOOLARG=iLikeTests -b build-again rocky-9 20 | 21 | # Check hashes, should be identical 22 | sha256sum -c /tmp/sha256sum.txt 23 | 24 | -------------------------------------------------------------------------------- /demo/builder-support/dockerfiles/Dockerfile.target.centos-7: -------------------------------------------------------------------------------- 1 | # First do the source builds 2 | @INCLUDE Dockerfile.target.sdist 3 | 4 | # This defines the distribution base layer 5 | # Put only the bare minimum of common commands here, without dev tools 6 | FROM centos:7 as dist-base 7 | ARG BUILDER_CACHE_BUSTER= 8 | RUN yum install -y epel-release 9 | # Python 3.4+ is needed for the builder helpers 10 | RUN yum install -y /usr/bin/python3 11 | 12 | # Do the actual rpm build 13 | @INCLUDE Dockerfile.rpmbuild 14 | 15 | # Generate provenance 16 | RUN /build/builder/helpers/generate-yum-provenance.py /dist/rpm-provenance.json 17 | 18 | # Do a test install and verify 19 | # Can be skipped with skiptests=1 in the environment 20 | @EXEC [ "$skiptests" = "" ] && include Dockerfile.rpmtest 21 | 22 | -------------------------------------------------------------------------------- /demo/builder-support/specs/a.spec: -------------------------------------------------------------------------------- 1 | %if %{?rhel} < 6 2 | exit 1 3 | %endif 4 | 5 | %define src_name demo-a 6 | %define src_version %{getenv:BUILDER_VERSION} 7 | 8 | Name: %{src_name} 9 | Version: %{getenv:BUILDER_RPM_VERSION} 10 | Release: %{getenv:BUILDER_RPM_RELEASE}%{dist} 11 | Summary: PowerDNS builder demo package A 12 | BuildArch: noarch 13 | 14 | Group: System 15 | License: MIT 16 | URL: https://github.com/PowerDNS/pdns-builder 17 | Source0: %{src_name}-%{src_version}.tar.gz 18 | 19 | %description 20 | A demo package for the PowerDNS builder. 21 | 22 | %prep 23 | %autosetup -n %{src_name}-%{src_version} 24 | 25 | %install 26 | ls 27 | %{__mkdir} -p %{buildroot}%{_bindir} 28 | %{__cp} demo-a.sh %{buildroot}%{_bindir}/demo-a 29 | 30 | %files 31 | %{_bindir}/* 32 | -------------------------------------------------------------------------------- /demo/builder-support/specs/b.spec: -------------------------------------------------------------------------------- 1 | %if %{?rhel} < 6 2 | exit 1 3 | %endif 4 | 5 | %define src_name demo-b 6 | %define src_version %{getenv:BUILDER_VERSION} 7 | 8 | Name: %{src_name} 9 | Version: %{getenv:BUILDER_RPM_VERSION} 10 | Release: %{getenv:BUILDER_RPM_RELEASE}%{dist} 11 | Summary: PowerDNS builder demo package A 12 | BuildArch: noarch 13 | 14 | Group: System 15 | License: MIT 16 | URL: https://github.com/PowerDNS/pdns-builder 17 | Source0: %{src_name}-%{src_version}.tar.gz 18 | 19 | %description 20 | A demo package for the PowerDNS builder. 21 | 22 | %prep 23 | %autosetup -n %{src_name}-%{src_version} 24 | 25 | %install 26 | ls 27 | %{__mkdir} -p %{buildroot}%{_bindir} 28 | %{__cp} demo-b.sh %{buildroot}%{_bindir}/demo-b 29 | 30 | %files 31 | %{_bindir}/* 32 | -------------------------------------------------------------------------------- /demo/builder-support/dockerfiles/Dockerfile.rpmtest: -------------------------------------------------------------------------------- 1 | # Install the built rpms and test them 2 | FROM dist-base as dist 3 | # If you want to install extra packages or do generic configuration, 4 | # do it before the COPY. Either here, or in the dist-base layer. 5 | 6 | # Test script requirements 7 | RUN yum install -y redis 8 | 9 | COPY --from=sdist /sdist /sdist 10 | COPY --from=package-builder /dist /dist 11 | 12 | # Install built packages with dependencies 13 | RUN yum localinstall -y /dist/*/*.rpm 14 | 15 | # Installation tests 16 | COPY builder-support/install-tests /build/builder-support/install-tests 17 | WORKDIR /build 18 | RUN builder-support/install-tests/check-installed-files.sh 19 | RUN builder-support/install-tests/test-exec.sh 20 | 21 | # Copy cache from package builder image, so that the builder can copy them out 22 | @IF [ ! -z "$BUILDER_CACHE" ] 23 | COPY --from=package-builder /cache/new /cache/new 24 | @ENDIF 25 | -------------------------------------------------------------------------------- /demo/builder-support/dockerfiles/Dockerfile.target.rocky-9: -------------------------------------------------------------------------------- 1 | # First do the source builds 2 | @INCLUDE Dockerfile.target.sdist 3 | 4 | # This defines the distribution base layer 5 | # Put only the bare minimum of common commands here, without dev tools 6 | FROM rockylinux:9 as dist-base 7 | ARG BUILDER_CACHE_BUSTER= 8 | #RUN dnf install -y epel-release 9 | # Python 3.4+ is needed for the builder helpers 10 | RUN dnf install -y /usr/bin/python3 11 | RUN dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm 12 | RUN dnf install -y dnf-plugins-core 13 | RUN dnf config-manager --set-enabled crb 14 | 15 | # Do the actual rpm build 16 | @INCLUDE Dockerfile.rpmbuild 17 | 18 | # Generate provenance 19 | RUN /build/builder/helpers/generate-dnf-provenance.py /dist/rpm-provenance.json 20 | 21 | # Do a test install and verify 22 | # Can be skipped with skiptests=1 in the environment 23 | @EXEC [ "$skiptests" = "" ] && include Dockerfile.rpmtest 24 | 25 | -------------------------------------------------------------------------------- /demo/builder-support/dockerfiles/Dockerfile.target.rocky-8: -------------------------------------------------------------------------------- 1 | # First do the source builds 2 | @INCLUDE Dockerfile.target.sdist 3 | 4 | # This defines the distribution base layer 5 | # Put only the bare minimum of common commands here, without dev tools 6 | FROM rockylinux:8 as dist-base 7 | ARG BUILDER_CACHE_BUSTER= 8 | #RUN dnf install -y epel-release 9 | # Python 3.4+ is needed for the builder helpers 10 | RUN dnf install -y /usr/bin/python3 11 | RUN dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm 12 | RUN dnf install -y dnf-plugins-core 13 | RUN dnf config-manager --set-enabled powertools 14 | 15 | # Do the actual rpm build 16 | @INCLUDE Dockerfile.rpmbuild 17 | 18 | # Generate provenance 19 | RUN /build/builder/helpers/generate-dnf-provenance.py /dist/rpm-provenance.json 20 | 21 | # Do a test install and verify 22 | # Can be skipped with skiptests=1 in the environment 23 | @EXEC [ "$skiptests" = "" ] && include Dockerfile.rpmtest 24 | 25 | -------------------------------------------------------------------------------- /demo/builder-support/post-build-test: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | echo "*** This is builder-support/post-build-test" 4 | 5 | if [ "$BUILDER_TARGET" = "sdist" ]; then 6 | echo "Skipping post-build-test for sdist target" 7 | exit 0 8 | fi 9 | 10 | # Pretend to do some tests that require a redis server 11 | 12 | # This id is the container id, which is also used as the --link hostname. 13 | redis_id=$(docker create redis:4-alpine) 14 | 15 | # Remove container on exit 16 | function cleanup_container { 17 | docker stop "$redis_id" > /dev/null 18 | docker rm "$redis_id" > /dev/null 19 | echo "Redis container removed" 20 | } 21 | trap cleanup_container EXIT 22 | 23 | docker start "$redis_id" 24 | 25 | # Run redis-cli the built image (the last stage with install tests), connect 26 | # to our temporary Redis 4.x container and verify the Redis version is 4.x. 27 | docker run -i --link "$redis_id" "$BUILDER_IMAGE" redis-cli -h "$redis_id" INFO server \ 28 | | grep redis_version:4 29 | 30 | -------------------------------------------------------------------------------- /templating/testdata/test-expected.txt: -------------------------------------------------------------------------------- 1 | Lines can start with @INCLUDE, @EVAL or @EXEC for special processing, they 2 | have no effect when not at the start of a line. 3 | 4 | 5 | Direct @INCLUDE: 6 | [Include file 1 start] 7 | value of FOO in include is also 123 8 | [Include file 1 end] 9 | 10 | 11 | The value of FOO is 123 12 | 13 | Empty @EXEC: 14 | @EXEC 15 | 16 | Conditional include in @EXEC: 17 | [Second include start] 18 | Hello world! 19 | [Second include end] 20 | 21 | 22 | This line is only printed if $FOO = "123", which is the case. 23 | Nested IF that is true. 24 | In between IFs. 25 | Last line of first IF. 26 | 27 | Triple nested IF that is true. 28 | Also true. 29 | Second level. 30 | First level. 31 | 32 | 33 | true1 34 | true1 35 | 36 | true1 37 | true2 38 | true2 39 | true1 40 | 41 | # Test @IF with extra indenting after the @ 42 | 43 | true1 44 | true2 45 | true2 46 | Other directives also get indenting 47 | true1 48 | 49 | 50 | Other lines are printed unchanged. 51 | 52 | -------------------------------------------------------------------------------- /demo/builder-support/vendor-specs/lua-argparse.spec: -------------------------------------------------------------------------------- 1 | %define luaver 5.1 2 | %define luapkgdir %{_datadir}/lua/%{luaver} 3 | 4 | Name: lua-argparse 5 | Version: 0.6.0 6 | Release: 1pdns%{?dist} 7 | Summary: Feature-rich command line parser for Lua 8 | 9 | Group: Development/Libraries 10 | License: MIT 11 | URL: https://github.com/mpeterv/argparse 12 | Source0: https://github.com/mpeterv/argparse/archive/%{version}.tar.gz 13 | 14 | BuildArch: noarch 15 | 16 | BuildRequires: lua >= %{luaver}, lua-devel >= %{luaver} 17 | Requires: lua >= %{luaver} 18 | 19 | %description 20 | Argparse is a feature-rich command line parser for Lua inspired by argparse for Python. 21 | 22 | %prep 23 | %setup -q -n argparse-%{version} 24 | 25 | %install 26 | %{__rm} -rf %{buildroot} 27 | %{__mkdir_p} %{buildroot}%{luapkgdir} 28 | %{__mkdir_p} %{buildroot}%{luapkgdir} 29 | %{__install} -m 0664 src/argparse.lua %{buildroot}%{luapkgdir} 30 | 31 | %clean 32 | %{__rm} -rf %{buildroot} 33 | 34 | %files 35 | %defattr(-,root,root,-) 36 | %doc README.md LICENSE 37 | %{luapkgdir}/* 38 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License 2 | 3 | Copyright (c) 2017 PowerDNS.COM BV. https://www.powerdns.com/ 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in 13 | all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 21 | THE SOFTWARE. 22 | -------------------------------------------------------------------------------- /helpers/generate-yum-provenance.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """ 4 | This script uses yum and rpm to generate in-toto material provenance and 5 | writes the resulting JSON to stdout or to argv[0] if provided. 6 | """ 7 | 8 | from __future__ import print_function 9 | import yum 10 | import json 11 | import sys 12 | import urllib 13 | 14 | yb = yum.YumBase() 15 | yb.setCacheDir() 16 | 17 | in_toto_data = list() 18 | in_toto_fmt = "pkg:rpm/{origin}/{name}@{epoch}{version}-{release}?arch={arch}" 19 | 20 | sack = yb.rpmdb 21 | sack.preloadPackageChecksums() 22 | 23 | for pkg in sack.returnPackages(): 24 | in_toto_data.append( 25 | { 26 | "uri": in_toto_fmt.format( 27 | origin=urllib.quote(pkg.vendor), 28 | name=pkg.name, 29 | epoch=pkg.epoch + ':' if pkg.epoch != '0' else '', 30 | version=pkg.version, 31 | release=pkg.release, 32 | arch=pkg.arch 33 | ), 34 | "digest": { 35 | 'sha256': pkg.yumdb_info.checksum_data 36 | } 37 | } 38 | ) 39 | 40 | if len(sys.argv) > 1: 41 | with open(sys.argv[1], 'w') as f: 42 | json.dump(in_toto_data, f) 43 | else: 44 | print(json.dumps(in_toto_data)) 45 | -------------------------------------------------------------------------------- /helpers/generate-deb-provenance.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """ 4 | This script reads the apt cache and writes in-toto material provenance 5 | to stdout in JSON format or argv[1] when provided. 6 | """ 7 | 8 | from __future__ import print_function 9 | import apt.cache 10 | import json 11 | import sys 12 | 13 | cache = apt.cache.Cache() 14 | in_toto_data = list() 15 | 16 | for pkg_name in cache.keys(): 17 | pkg = cache[pkg_name] 18 | if not pkg.is_installed: 19 | continue 20 | # Generated by mk-build-deps in build-debs.sh 21 | # Has no digest, so let's ignore it 22 | if pkg_name.endswith('-build-deps'): 23 | continue 24 | 25 | in_toto_data.append( 26 | { 27 | "uri": "pkg:deb/{origin}/{name}@{version}?arch={arch}".format( 28 | origin=pkg.installed.origins[0].origin, 29 | name=pkg.shortname, 30 | version=pkg.installed.version, 31 | arch=pkg.installed.architecture 32 | ), 33 | "digest": { 34 | "sha256": pkg.installed.sha256 35 | } 36 | } 37 | ) 38 | 39 | if len(sys.argv) > 1: 40 | with open(sys.argv[1], 'w') as f: 41 | json.dump(in_toto_data, f) 42 | else: 43 | print(json.dumps(in_toto_data)) 44 | -------------------------------------------------------------------------------- /demo/builder-support/dockerfiles/Dockerfile.target.sdist: -------------------------------------------------------------------------------- 1 | # This builds the source distributions for all components 2 | 3 | # ========================================================================= 4 | # Deliberately using alpine with incompatible libc to ensure we are 5 | # not sneaking in any binaries, and because it's light and up to date. 6 | FROM alpine:3.13 as sdist 7 | ARG BUILDER_CACHE_BUSTER= 8 | RUN apk add --no-cache tar 9 | ARG BUILDER_VERSION 10 | ARG SOURCE_DATE_EPOCH 11 | 12 | RUN echo "::: SOURCE_DATE_EPOCH=$SOURCE_DATE_EPOCH"; if [ "$SOURCE_DATE_EPOCH" -lt "1000" ]; then echo "::: INVALID SOURCE_DATE_EPOCH" ; exit 99 ; fi 13 | 14 | # Copying minimal set of files to avoid cache invalidation 15 | RUN mkdir /build 16 | WORKDIR /build 17 | COPY src/ src/ 18 | RUN mkdir /sdist 19 | 20 | # Test if the -B option works 21 | ARG MYCOOLARG 22 | RUN test "${MYCOOLARG}" = 'iLikeTests' 23 | 24 | # Build module A 25 | @IF [ ! -z "$M_all$M_a" ] 26 | RUN cd src/ && tar --clamp-mtime --mtime="@$SOURCE_DATE_EPOCH" -cvzf /sdist/demo-a-$BUILDER_VERSION.tar.gz --transform "s,^,demo-a-$BUILDER_VERSION/," demo-a.sh 27 | @ENDIF 28 | 29 | # Build module B 30 | @IF [ ! -z "$M_all$M_a" ] 31 | RUN cd src/ && tar --clamp-mtime --mtime="@$SOURCE_DATE_EPOCH" -cvzf /sdist/demo-b-$BUILDER_VERSION.tar.gz --transform "s,^,demo-b-$BUILDER_VERSION/," demo-b.sh 32 | @ENDIF 33 | 34 | # Show contents for build debugging 35 | RUN ls -l /sdist/* 36 | 37 | -------------------------------------------------------------------------------- /.github/workflows/tests.yml: -------------------------------------------------------------------------------- 1 | --- 2 | name: 'Run tests' 3 | 4 | on: 5 | push: 6 | pull_request: 7 | # Disabled, because Github appears to disable the whole workflow if you have a schedule 8 | # and no updates for 60 days... 9 | # schedule: 10 | #- cron: '0 7 * * *' 11 | 12 | jobs: 13 | tests: 14 | name: Run tests 15 | # on a ubuntu-20.04 VM 16 | runs-on: ubuntu-20.04 17 | defaults: 18 | run: 19 | working-directory: ./demo/ 20 | steps: 21 | - uses: actions/checkout@v2.3.4 22 | with: 23 | fetch-depth: 5 24 | submodules: recursive 25 | - run: ./test-templating.sh 26 | working-directory: ./templating 27 | - run: ./tests/test_versioning.sh 28 | working-directory: . 29 | - run: ./prepare.sh 30 | - run: ./builder/build.sh -B MYCOOLARG=iLikeTests sdist 31 | - run: ./builder/build.sh -B MYCOOLARG=iLikeTests rocky-9 32 | # Again, now very fast due to the layer cache 33 | - run: ./builder/build.sh -B MYCOOLARG=iLikeTests rocky-9 34 | # Three cache builds: 35 | # - First one will write the vendor cache 36 | - run: ./builder/build.sh -c -B MYCOOLARG=iLikeTests rocky-9 37 | # - Second one will use the vendor cache, but the Docker layer cache gets invalidated by the new cache file 38 | - run: ./builder/build.sh -c -B MYCOOLARG=iLikeTests rocky-9 39 | # - Third one is very fast due to the Docker layer cache 40 | - run: ./builder/build.sh -c -B MYCOOLARG=iLikeTests rocky-9 41 | # Do a reproducible rocky-8 build (does not work for centos-7) 42 | - run: ../tests/test-rocky-8-reproducible.sh 43 | - run: ../tests/test-rocky-9-reproducible.sh 44 | -------------------------------------------------------------------------------- /templating/testdata/test-template.txt: -------------------------------------------------------------------------------- 1 | Lines can start with @INCLUDE, @EVAL or @EXEC for special processing, they 2 | have no effect when not at the start of a line. 3 | 4 | @EXEC FOO=123 5 | 6 | Direct @INCLUDE: 7 | @INCLUDE test-template-include1.txt 8 | 9 | @EVAL The value of FOO is $FOO 10 | 11 | Empty @EXEC: 12 | @EXEC 13 | 14 | Conditional include in @EXEC: 15 | @EXEC [ "$FOO" = "123" ] && include test-template-include2.txt 16 | 17 | @IF [ "$FOO" = "123" ] 18 | This line is only printed if $FOO = "123", which is the case. 19 | @IF true 20 | Nested IF that is true. 21 | @ENDIF 22 | In between IFs. 23 | @IF [ "$FOO" = "wrong" ] 24 | THIS WILL NEVER BE PRINTED. 25 | @ENDIF 26 | Last line of first IF. 27 | @ENDIF 28 | 29 | @IF true 30 | @IF true 31 | @IF true 32 | Triple nested IF that is true. 33 | @ENDIF 34 | @IF true 35 | Also true. 36 | @ENDIF 37 | Second level. 38 | @ENDIF 39 | First level. 40 | @ENDIF 41 | 42 | @IF false 43 | @IF true 44 | @IF true 45 | Triple nested IF that is FALSE. 46 | @ENDIF 47 | @ENDIF 48 | @ENDIF 49 | 50 | @IF true 51 | true1 52 | @IF false 53 | false2 54 | @IF true 55 | Triple nested IF that is FALSE. 56 | @ENDIF 57 | false2 MUST NOT APPEAR, BUG! 58 | @ENDIF 59 | true1 60 | @ENDIF 61 | 62 | @IF true 63 | true1 64 | @IF true 65 | true2 66 | @IF false 67 | Triple nested IF that is FALSE. 68 | @ENDIF 69 | true2 70 | @ENDIF 71 | true1 72 | @ENDIF 73 | 74 | # Test @IF with extra indenting after the @ 75 | 76 | @IF true 77 | true1 78 | @ IF true 79 | true2 80 | @ IF false 81 | Triple nested IF that is FALSE. 82 | @ ENDIF 83 | true2 84 | @ EXEC echo "Other directives also get indenting" 85 | @ ENDIF 86 | true1 87 | @ENDIF 88 | 89 | 90 | Other lines are printed unchanged. 91 | -------------------------------------------------------------------------------- /helpers/generate-dnf-provenance.py: -------------------------------------------------------------------------------- 1 | #!/usr/libexec/platform-python 2 | """ 3 | This script uses yum and rpm to generate in-toto material provenance and 4 | writes the resulting JSON to stdout or to argv[0] if provided. 5 | """ 6 | 7 | import dnf 8 | import json 9 | import sys 10 | import urllib.parse 11 | 12 | in_toto_data = list() 13 | in_toto_fmt = "pkg:rpm/{origin}/{name}@{epoch}{version}-{release}?arch={arch}" 14 | 15 | with dnf.Base() as db: 16 | db.fill_sack() 17 | q = db.sack.query() 18 | 19 | for pkg in q.installed(): 20 | in_toto_data.append( 21 | { 22 | "uri": in_toto_fmt.format( 23 | origin=urllib.parse.quote(pkg.vendor), 24 | name=pkg.name, 25 | epoch=str(pkg.epoch) + ':' if pkg.epoch != 0 else '', 26 | version=pkg.version, 27 | release=pkg.release, 28 | arch=pkg.arch 29 | ), 30 | "digest": { 31 | # The DNF documentation says: 32 | # The checksum is returned only for packages from 33 | # repository. The checksum is not returned for 34 | # installed package or packages from commandline 35 | # repository. 36 | # Which is super lame, so we use the header checksum to 37 | # have _something_. 38 | 'sha1': pkg.hdr_chksum[1].hex() 39 | } 40 | } 41 | ) 42 | 43 | if len(sys.argv) > 1: 44 | with open(sys.argv[1], 'w') as f: 45 | json.dump(in_toto_data, f) 46 | else: 47 | print(json.dumps(in_toto_data)) 48 | -------------------------------------------------------------------------------- /docker-cleanup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Cleanup docker files: untagged containers and images. 3 | # 4 | # Use `docker-cleanup -n` for a dry run to see what would be deleted. 5 | 6 | untagged_containers() { 7 | # Print containers using untagged images: $1 is used with awk's print: 0=line, 1=column 1. 8 | docker ps -a -f status=exited | awk '$2 ~ "[0-9a-f]{12}" {print $'$1'}' 9 | } 10 | 11 | compose_run_containers() { 12 | docker ps -a -f status=exited | egrep '_run_[0-9]{1,2}' | sed 's/ .*//' 13 | } 14 | 15 | non_compose_containers() { 16 | # random name foo_bar, like pedantic_euclid 17 | # extra filter to make sure we don't remove compose containers like nginx_nginx_1 18 | docker ps -a -f status=exited | egrep ' [a-z]+_[a-z]+$' | egrep -v '_[0-9]{1,2}' | sed 's/ .*//' 19 | } 20 | 21 | untagged_images() { 22 | # Print untagged images: $1 is used with awk's print: 0=line, 3=column 3. 23 | # NOTE: intermediate images (via -a) seem to only cause 24 | # "Error: Conflict, foobarid wasn't deleted" messages. 25 | # Might be useful sometimes when Docker messed things up?! 26 | # docker images -a | awk '$1 == "" {print $'$1'}' 27 | docker images | tail -n +2 | awk '$1 == "" {print $'$1'}' 28 | } 29 | 30 | # Dry-run. 31 | if [ "$1" = "-n" ]; then 32 | echo "=== Containers with uncommitted images: ===" 33 | untagged_containers 0 34 | echo 35 | 36 | echo "=== Uncommitted images: ===" 37 | untagged_images 0 38 | 39 | exit 40 | fi 41 | 42 | # Remove containers with untagged images. 43 | echo "Removing untagged containers:" >&2 44 | untagged_containers 1 | xargs docker rm --volumes=true 45 | echo "Removing compose run containers:" >&2 46 | compose_run_containers 1 | xargs docker rm --volumes=true 47 | echo "Removing non-compose containers:" >&2 48 | non_compose_containers 1 | xargs docker rm --volumes=true 49 | 50 | # Remove untagged images 51 | echo "Removing images:" >&2 52 | untagged_images 3 | xargs docker rmi 53 | -------------------------------------------------------------------------------- /gen-version: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | VERSION="unknown" 3 | 4 | DIRTY="" 5 | git status | grep -q clean || DIRTY='.dirty' 6 | 7 | # Special environment variable to signal that we are building a release, as this 8 | # has consequences for the version number. 9 | if [ "${IS_RELEASE}" = "YES" ]; then 10 | TAG="$(git describe --tags --exact-match 2> /dev/null)" 11 | if [ -n "${TAG}" ]; then 12 | # We're on a tag 13 | echo "${TAG}${DIRTY}" > .version 14 | printf "${TAG}${DIRTY}" 15 | exit 0 16 | fi 17 | echo 'This is not a tag, either tag this commit or do not set $IS_RELEASE' >&2 18 | exit 1 19 | fi 20 | 21 | # 22 | # Generate the version number based on the branch 23 | # 24 | if [ ! -z "$(git rev-parse --abbrev-ref HEAD 2> /dev/null)" ]; then 25 | # Possible `git describe --tags` outputs: 26 | # 2.4.1 (when on a tag) 27 | # 2.4.1-alpha1 (when on a tag) 28 | # 2.4.2-1-gdad05a9 (not on a tag) 29 | # 2.4.2-alpha1-1-gdad05a9 (not on a tag) 30 | 31 | OIFS=$IFS 32 | IFS='-' GIT_VERSION=( $(git describe --tags 2> /dev/null) ) 33 | IFS=$OIFS 34 | LAST_TAG="${GIT_VERSION[0]}" 35 | COMMITS_SINCE_TAG='' 36 | GIT_HASH='' 37 | 38 | if [ ${#GIT_VERSION[@]} -eq 1 ]; then 39 | # We're on a tag, but IS_RELEASE was unset 40 | COMMITS_SINCE_TAG=0 41 | fi 42 | 43 | if [ ${#GIT_VERSION[@]} -eq 2 ]; then 44 | # On a tag for a pre-release, e.g. 1.2.3-beta2 45 | LAST_TAG="${LAST_TAG}-${GIT_VERSION[1]}" 46 | COMMITS_SINCE_TAG=0 47 | fi 48 | 49 | if [ ${#GIT_VERSION[@]} -eq 3 ]; then 50 | # Not on a tag 51 | # 1.2.3-100-g123456 52 | COMMITS_SINCE_TAG="${GIT_VERSION[1]}" 53 | GIT_HASH="${GIT_VERSION[2]}" 54 | fi 55 | 56 | if [ ${#GIT_VERSION[@]} -eq 4 ]; then 57 | # Not on a tag, but a pre-release was made before 58 | # 1.2.3-rc1-100-g123456 59 | LAST_TAG="${LAST_TAG}-${GIT_VERSION[1]}" 60 | COMMITS_SINCE_TAG="${GIT_VERSION[2]}" 61 | GIT_HASH="${GIT_VERSION[3]}" 62 | fi 63 | 64 | if [ -z "${GIT_HASH}" ]; then 65 | GIT_HASH="g$(git show --no-patch --format=format:%h HEAD 2>/dev/null)" 66 | if [ -z "$GIT_HASH" ]; then 67 | GIT_HASH="g$(git show --format=format:%h HEAD | head -n1)" 68 | fi 69 | fi 70 | 71 | if [ -z "${LAST_TAG}" ]; then 72 | LAST_TAG=0.0.0 73 | COMMITS_SINCE_TAG="$(git log --oneline | wc -l)" 74 | fi 75 | 76 | BRANCH="$(git rev-parse --abbrev-ref HEAD)" 77 | 78 | # if CI_COMMIT_REF_SLUG is set, we're running inside GitLab CI and this name is safe to use 79 | if [ -n "${CI_COMMIT_REF_SLUG}" ]; then 80 | BRANCH="${CI_COMMIT_REF_SLUG}" 81 | fi 82 | 83 | BRANCH=".$(echo ${BRANCH} | perl -p -e 's/[^[:alnum:]]//g;')" 84 | 85 | # If the branch is master, remove it 86 | [ "${BRANCH}" = ".master" ] && BRANCH='' 87 | 88 | VERSION="${LAST_TAG}.${COMMITS_SINCE_TAG}${BRANCH}.${GIT_HASH}${DIRTY}" 89 | fi 90 | 91 | printf $VERSION | perl -p -e 's/^v//;' 92 | -------------------------------------------------------------------------------- /demo/builder-support/dockerfiles/Dockerfile.rpmbuild: -------------------------------------------------------------------------------- 1 | # Generic RPM building, which should work for centos and oraclelinux 2 | 3 | ############################################################################## 4 | # Shared package building base image 5 | 6 | FROM dist-base as package-builder-base 7 | RUN yum install -y rpm-build rpmdevtools && rpmdev-setuptree 8 | 9 | RUN mkdir -p /dist /build 10 | WORKDIR /build 11 | 12 | ############################################################################## 13 | # Our package-builder target image 14 | 15 | FROM package-builder-base as package-builder 16 | 17 | # Only ADD/COPY the files you really need for efficient docker caching. 18 | ADD builder/helpers/ /build/builder/helpers/ 19 | 20 | # Used for -p option to only build specific spec files 21 | ARG BUILDER_PACKAGE_MATCH 22 | 23 | # Enables the builder rpm file cache (requires -c flag). 24 | # This must only be used for vendor deps, because it is based on the hash of the spec. 25 | # It might slow down your second build if you have docker layer caching, but 26 | # it's useful for Travis and if you build different modules and branches all 27 | # the time. 28 | @IF [ ! -z "$BUILDER_CACHE" ] 29 | @EVAL ADD builder/cache/${BUILDER_TARGET}/ /cache/old/ 30 | ENV BUILDER_CACHE 1 31 | RUN mkdir /cache/new 32 | @ENDIF 33 | 34 | # Vendor specs 35 | # These do not depend on code in this repository and have independent versioning. 36 | # Only vendor specs can safely use the builder cache for these reasons. 37 | @IF [ ! -z "$M_all$M_vendor" ] 38 | COPY builder-support/vendor-specs/ builder-support/vendor-specs/ 39 | RUN BUILDER_CACHE_THIS=1 BUILDER_SOURCE_DATE_FROM_SPEC_MTIME=1 builder/helpers/build-specs.sh builder-support/vendor-specs/*.spec 40 | @ENDIF 41 | 42 | # You can override these build args for faster Python builds 43 | # by adding these to your local .env 44 | # For pip: http://doc.devpi.net/latest/quickstart-pypimirror.html 45 | #ARG PIP_INDEX_URL 46 | #ARG PIP_TRUSTED_HOST 47 | 48 | # Set after vendor builds to not invalidate their cached layers every time 49 | ARG BUILDER_VERSION 50 | ARG BUILDER_RELEASE 51 | ARG SOURCE_DATE_EPOCH 52 | COPY --from=sdist /sdist /sdist 53 | RUN for file in /sdist/* ; do ln -s $file /root/rpmbuild/SOURCES/ ; done && ls /root/rpmbuild/SOURCES/ 54 | 55 | COPY builder-support/specs/ builder-support/specs/ 56 | 57 | @IF [ ! -z "$M_all$M_a" ] 58 | RUN builder/helpers/build-specs.sh builder-support/specs/a.spec 59 | @ENDIF 60 | 61 | @IF [ ! -z "$M_all$M_b" ] 62 | RUN builder/helpers/build-specs.sh builder-support/specs/b.spec 63 | @ENDIF 64 | 65 | # mv across layers with overlay2 is buggy in some kernel versions (results in empty dirs) 66 | # See: https://github.com/moby/moby/issues/33733 67 | #RUN mv /root/rpmbuild/RPMS/* /dist/ 68 | RUN cp -R /root/rpmbuild/RPMS/* /dist/ 69 | 70 | # Write text files with lists of all files in the rpms 71 | RUN rpm -qlp --queryformat '\n# %{RPMTAG_NAME}\n' /dist/*/*.rpm > /dist/rpm-contents-all.txt 72 | -------------------------------------------------------------------------------- /templating/templating.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # Very simple template language 3 | 4 | if [ "$1" = "" ]; then 5 | echo "USAGE: $0 " 6 | echo 7 | echo "Template syntax:" 8 | echo 9 | echo 'Lines can start with @INCLUDE, @EVAL or @EXEC for special processing:' 10 | echo '@INCLUDE foo.txt' 11 | echo '@EVAL My home dir is $HOME' 12 | echo '@EXEC uname -a' 13 | echo '@EXEC [ "$foo" = "bar" ] && include bar.txt' 14 | echo '@IF [ "$foo" = "bar" ]' 15 | echo 'This line is only printed if $foo = "bar"' 16 | echo '@ENDIF' 17 | echo 'Other lines are printed unchanged.' 18 | echo 19 | echo "Environment variables:" 20 | echo " tmpl_debug: If set, markers are printed around included files" 21 | echo " tmpl_comment: Characters to use to start the marker comments (default: #)" 22 | echo " tmpl_prefix: Regexp to match processing directive prefixes (default: @)" 23 | exit 1 24 | fi 25 | 26 | tmpl_comment=${tmpl_comment:-#} 27 | tmpl_prefix=${tmpl_prefix:-@} 28 | 29 | include() { 30 | [ "$tmpl_debug" != "" ] && echo "$tmpl_comment $1" 31 | # Current level of @IF we are in 32 | local iflevel=0 33 | # Set to the @IF level that disabled the current block, if any 34 | local ifdisablelevel=0 35 | local line 36 | local condition 37 | ( cat "$1" && echo ) | while IFS= read -r line; do 38 | 39 | if [[ $line =~ ^${tmpl_prefix}\ *IF\ (.*) ]]; then 40 | [ "$tmpl_debug" != "" ] && echo "$tmpl_comment $line" 41 | iflevel=$((iflevel+1)) 42 | if ! [ $ifdisablelevel -gt 0 ]; then 43 | # Only if not already in a disabled IF statement 44 | condition="${BASH_REMATCH[1]}" 45 | if ! eval "$condition" ; then 46 | # Disabled at the current IF level 47 | ifdisablelevel=$iflevel 48 | fi 49 | fi 50 | 51 | elif [[ $line =~ ^${tmpl_prefix}\ *ENDIF ]]; then 52 | [ "$tmpl_debug" != "" ] && echo "$tmpl_comment $line" 53 | if [ $iflevel = 0 ] ; then 54 | echo "ERROR: @ENDIF without matching @IF in file $1" > /dev/stderr 55 | exit 30 56 | fi 57 | iflevel=$((iflevel-1)) 58 | if [ $ifdisablelevel -gt $iflevel ]; then 59 | # We left the IF block level that was disabled 60 | ifdisablelevel=0 61 | fi 62 | 63 | elif [ $ifdisablelevel -gt 0 ]; then 64 | # nothing, in IF that evaluated to false 65 | [ "$tmpl_debug" != "" ] && echo "$tmpl_comment $line" 66 | 67 | 68 | elif [[ $line =~ ^${tmpl_prefix}\ *INCLUDE\ +([^ ]*) ]]; then 69 | include="${BASH_REMATCH[1]}" 70 | include $include 71 | 72 | elif [[ $line =~ ^${tmpl_prefix}\ *EVAL\ (.*) ]]; then 73 | line="${BASH_REMATCH[1]}" 74 | eval echo "\"$line\"" 75 | 76 | elif [[ $line =~ ^${tmpl_prefix}\ *EXEC\ (.*) ]]; then 77 | line="${BASH_REMATCH[1]}" 78 | eval "$line" 79 | 80 | else 81 | echo "$line" 82 | fi 83 | 84 | done 85 | [ "$tmpl_debug" != "" ] && echo "$tmpl_comment /$1" 86 | } 87 | 88 | include "$1" 89 | 90 | exit 0 91 | -------------------------------------------------------------------------------- /helpers/build-debs.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Build debian packages, after installing dependencies 3 | # This assumes the source is unpacked and a debian/ directory exists 4 | 5 | helpers=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd ) 6 | 7 | source "$helpers/functions.sh" 8 | 9 | dirs=() 10 | for dir in "$@"; do 11 | # If BUILDER_PACKAGE_MATCH is set, only build the packages that match, otherwise build all 12 | if [ -z "$BUILDER_PACKAGE_MATCH" ] || [[ $dir = *$BUILDER_PACKAGE_MATCH* ]]; then 13 | if [ ! -d ${dir}/debian ]; then 14 | echo "${dir}/debian does not exist, can not build!" 15 | continue 16 | fi 17 | dirs+=($dir) 18 | fi 19 | done 20 | 21 | if [ "${#dirs[@]}" = "0" ]; then 22 | echo "No debian package directories matched, nothing to do" 23 | exit 0 24 | fi 25 | 26 | for dir in "${dirs[@]}"; do 27 | # Install all build-deps 28 | pushd "${dir}" 29 | mk-build-deps -i -t 'apt-get -y -o Debug::pkgProblemResolver=yes --no-install-recommends' || exit 1 30 | popd 31 | done 32 | 33 | for dir in "${dirs[@]}"; do 34 | echo "===================================================================" 35 | echo "-> ${dir}" 36 | pushd "${dir}" 37 | # If there's a changelog, this is probably a vendor dependency or versioned 38 | # outside of pdns-builder 39 | if [ ! -f debian/changelog ]; then 40 | # Parse the Source name 41 | sourcename=`grep '^Source: ' debian/control | sed 's,^Source: ,,'` 42 | if [ -z "${sourcename}" ]; then 43 | echo "Unable to parse name of the source from ${dir}" 44 | exit 1 45 | fi 46 | # Let's try really hard to find the release name of the distribution 47 | # Prefer something that sorts well over time 48 | distro_release="$(source /etc/os-release; [ ! -z ${ID} ] && [ ! -z ${VERSION_ID} ] && echo -n ${ID}${VERSION_ID})" # this will look like 'debian12' or 'ubuntu22.04' 49 | if [ -z "${distro_release}" ]; then 50 | # we should only end up here on Debian Testing 51 | distro="$(source /etc/os-release; echo -n ${ID})" 52 | if [ ! -z "${distro}" ]; then 53 | releasename="$(perl -n -e '/PRETTY_NAME="Debian GNU\/Linux (.*)\/sid"/ && print $1' /etc/os-release)" 54 | if [ ! -z "${releasename}" ]; then 55 | apt-get -y --no-install-recommends install distro-info-data 56 | releasenum="$(grep ${releasename} /usr/share/distro-info/debian.csv | cut -f1 -d,)" 57 | if [ ! -z "${releasenum}" ]; then 58 | distro_release="${distro}${releasenum}" 59 | fi 60 | fi 61 | fi 62 | fi 63 | if [ -z "${distro_release}" ]; then 64 | echo 'Unable to determine distribution codename!' 65 | exit 1 66 | fi 67 | if [ -z "$BUILDER_EPOCH" ]; then 68 | epoch_string="" 69 | else 70 | epoch_string="${BUILDER_EPOCH}:" 71 | fi 72 | echo "EPOCH_STRING=${epoch_string}" 73 | set_debian_versions 74 | cat > debian/changelog << EOF 75 | $sourcename (${epoch_string}${BUILDER_DEB_VERSION}-${BUILDER_DEB_RELEASE}.${distro_release}) unstable; urgency=medium 76 | 77 | * Automatic build 78 | 79 | -- PowerDNS.COM AutoBuilder $(date -R) 80 | EOF 81 | fi 82 | 83 | # allow build to use all available processors 84 | export DEB_BUILD_OPTIONS='parallel='`nproc` 85 | 86 | dpkg-buildpackage -b || exit 1 87 | popd 88 | done 89 | -------------------------------------------------------------------------------- /tests/test_versioning.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | exitcode=0 4 | assert_equal() { 5 | if [ "$2" != "$3" ]; then 6 | echo "${1}: ${2} != ${3}" 7 | exitcode=1 8 | fi 9 | } 10 | 11 | set -e 12 | 13 | source "helpers/functions.sh" 14 | 15 | builder_release='1pdns' 16 | 17 | src_versions=(1.0.0 18 | 1.0.0.dirty 19 | 1.0.0-beta1 20 | 1.0.0-beta1.dirty 21 | 1.1.0-rc2.0.g123456 22 | 1.1.0-rc2.0.g123456.dirty 23 | 1.1.0.15.g123456 24 | 1.1.0.15.g123456.dirty 25 | 1.1.0.15.branchname.g123456 26 | 1.1.0.15.branchname.g123456.dirty 27 | 1.2.0-alpha1.10.branch.g123456 28 | 1.2.0-alpha1.10.branch.g123456.dirty 29 | 1.2.3.130.HEAD.gbac839b2) 30 | deb_versions=(1.0.0 31 | 1.0.0+dirty 32 | 1.0.0~beta1 33 | 1.0.0~beta1+dirty 34 | 1.1.0~rc2+0.g123456 35 | 1.1.0~rc2+0.g123456.dirty 36 | 1.1.0+15.g123456 37 | 1.1.0+15.g123456.dirty 38 | 1.1.0+branchname.15.g123456 39 | 1.1.0+branchname.15.g123456.dirty 40 | 1.2.0~alpha1+branch.10.g123456 41 | 1.2.0~alpha1+branch.10.g123456.dirty 42 | 1.2.3+HEAD.130.gbac839b2) 43 | rpm_versions=(1.0.0 44 | 1.0.0 45 | 1.0.0 46 | 1.0.0 47 | 1.1.0 48 | 1.1.0 49 | 1.1.0 50 | 1.1.0 51 | 1.1.0 52 | 1.1.0 53 | 1.2.0 54 | 1.2.0 55 | 1.2.3) 56 | rpm_releases=($builder_release 57 | dirty.$builder_release 58 | 0.beta1.$builder_release 59 | 0.beta1.dirty.$builder_release 60 | 0.rc2.0.g123456.$builder_release 61 | 0.rc2.0.g123456.dirty.$builder_release 62 | 15.g123456.$builder_release 63 | 15.g123456.dirty.$builder_release 64 | branchname.15.g123456.$builder_release 65 | branchname.15.g123456.dirty.$builder_release 66 | 0.alpha1.branch.10.g123456.$builder_release 67 | 0.alpha1.branch.10.g123456.dirty.$builder_release 68 | HEAD.130.gbac839b2.$builder_release) 69 | 70 | # These comply to PEP 440 71 | py_versions=(1.0.0 72 | 1.0.0+dirty 73 | 1.0.0b1 74 | 1.0.0b1+dirty 75 | 1.1.0rc2+0.g123456 76 | 1.1.0rc2+0.g123456.dirty 77 | 1.1.0+15.g123456 78 | 1.1.0+15.g123456.dirty 79 | 1.1.0+15.branchname.g123456 80 | 1.1.0+15.branchname.g123456.dirty 81 | 1.2.0a1+10.branch.g123456 82 | 1.2.0a1+10.branch.g123456.dirty 83 | 1.2.3+130.head.gbac839b2) 84 | 85 | for ctr in ${!src_versions[@]}; do 86 | BUILDER_VERSION=${src_versions[$ctr]} 87 | BUILDER_RELEASE=$builder_release 88 | set_debian_versions 89 | assert_equal BUILDER_DEB_VERSION $BUILDER_DEB_VERSION ${deb_versions[$ctr]} 90 | assert_equal BUILDER_DEB_RELEASE $BUILDER_DEB_RELEASE $builder_release 91 | 92 | set_rpm_versions 93 | assert_equal BUILDER_RPM_VERSION $BUILDER_RPM_VERSION ${rpm_versions[$ctr]} 94 | assert_equal BUILDER_RPM_RELEASE $BUILDER_RPM_RELEASE ${rpm_releases[$ctr]} 95 | 96 | set_python_src_versions 97 | assert_equal BUILDER_PYTHON_SRC_VERSION $BUILDER_PYTHON_SRC_VERSION ${py_versions[$ctr]} 98 | 99 | assert_equal BUILDER_VERSION $BUILDER_VERSION ${src_versions[$ctr]} 100 | assert_equal BUILDER_RELEASE $BUILDER_RELEASE $builder_release 101 | done 102 | 103 | exit "$exitcode" 104 | -------------------------------------------------------------------------------- /helpers/build-rocks.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # This must be run from the root of the repo 3 | # This script creates source distributions for all rockspecs in builder/luarocks: 4 | # - It fills in the version and copies it to the right filename luarocks expects 5 | # - It creates a tarball with all the files specified in the rockspec 6 | # 7 | # NOTE: This is not used for external dependencies. Externally hosted rocks 8 | # should be built directly. 9 | # 10 | # The source builds can be triggered manually by running this **from the repo root**: 11 | # 12 | # BUILDER_VERSION=`./build-scripts/gen-version` ./builder/helpers/build-rocks.sh foo.luarock 13 | # 14 | # Related tools and locations: 15 | # 16 | # - `helpers/build-rocks.sh`: creates source archives, rockspecs and rocks for all input 17 | # rockspecs under `builder/rockspecs/`. 18 | # It expects the version to be available in the `BUILDER_VERSION` env var. 19 | # - `helpers/list-rock-contents.lua`: lists all files that need to be included in the 20 | # source archive from a rockspec. 21 | # 22 | # Rules for writing the rockspecs: 23 | # 24 | # - `__VERSION__` will be replaced by a version string. From the input version 25 | # (typically generated by `gen-version`), `+` will be replaced by `.` and 26 | # `-1` will be appended as a rockspec revision, as required by LuaRocks. 27 | # - Only `build.type = 'builtin'` is supported (because of `list-rock-contents.lua`). 28 | # - Only files under `build.modules` and `build.install` will be included in the 29 | # source tarball. No support for compiled modules. 30 | # 31 | # To test a built rock, you can do: 32 | # 33 | # luarocks install --deps-mode=none --tree /tmp/foo ./builder/tmp/some-package-*.src.rock 34 | # 35 | # You can check the tree with: 36 | # 37 | # tree /tmp/foo 38 | # 39 | # You can omit `--deps-mode=none` to also install all dependencies. This switch is 40 | # useful when building distribution packages, as you then want to manage dependencies 41 | # in the spec files. 42 | # 43 | 44 | # We need the GNU version of tar for --transform 45 | [ -z "$tar" ] && tar=`which gtar tar | grep '^/' | head -1` 46 | if ! $tar --version | grep -q GNU; then 47 | echo "ERROR: could not find GNU tar (as gtar or tar)" 48 | echo "On macOS: brew install gnu-tar" 49 | exit 1 50 | fi 51 | 52 | [ -z "$tar_prefix" ] && tar_prefix=dist 53 | 54 | set -e 55 | 56 | version=`echo "${BUILDER_VERSION:-0.0.0}" | sed 's/[-+]/./'` 57 | 58 | build_dir=builder/tmp 59 | [ ! -d "$build_dir" ] && mkdir "$build_dir" 60 | 61 | color_reset='\x1B[0m' 62 | color_black='\x1B[1;30m'; 63 | color_red='\x1B[1;31m'; 64 | color_green='\x1B[1;32m'; 65 | color_yellow='\x1B[1;33m'; 66 | color_blue='\x1B[1;34m'; 67 | color_purple='\x1B[1;35m'; 68 | color_cyan='\x1B[1;36m'; 69 | color_white='\x1B[1;37m'; 70 | 71 | luarocks=luarocks-5.1 72 | 73 | for rockspec in "$@"; do 74 | echo 75 | echo "${color_yellow}* Building rock for $rockspec${color_reset}" 76 | name=`basename "$rockspec" .rockspec` 77 | 78 | # Generate a rockspec file with version 79 | outrockspecname="$name-$version-1.rockspec" 80 | cat "$rockspec" | sed "s/__VERSION__/$version/" > "$build_dir/$outrockspecname" 81 | echo "Created $build_dir/$outrockspecname" 82 | #cat "$build_dir/$outrockspecname" 83 | "$luarocks" lint "$build_dir/$outrockspecname" 84 | 85 | # Archive with all required files 86 | archive="$build_dir/$name-$version.tar.gz" 87 | files=`./builder/helpers/list-rock-contents.lua "$rockspec"` 88 | echo "$files" | sed 's/^/ /' 89 | "$tar" --transform "s|^|$tar_prefix/|" --dereference -czf "$archive" $files 90 | 91 | # Create rock 92 | cd "$build_dir" 93 | "$luarocks" pack "$outrockspecname" 94 | cd - 95 | 96 | if [ ! -s "$build_dir/$outrockspecname" ]; then 97 | echo "${color_red}ERROR: It looks like your outdated version of luarocks truncates rockspecs!${color_reset}" 98 | exit 1 99 | fi 100 | echo "${color_green}SUCCESS${color_reset}" 101 | done 102 | 103 | 104 | -------------------------------------------------------------------------------- /helpers/build-specs.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Build multiple rpm specs, after installing the build dependencies 3 | 4 | set -e # exit on helper error 5 | 6 | helpers=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd ) 7 | 8 | source "$helpers/functions.sh" 9 | 10 | specs=() 11 | for spec in "$@"; do 12 | # If BUILDER_PACKAGE_MATCH is set, only build the specs that match, otherwise build all 13 | if [ -z "$BUILDER_PACKAGE_MATCH" ] || [[ $spec = *$BUILDER_PACKAGE_MATCH* ]]; then 14 | specs+=($spec) 15 | fi 16 | done 17 | 18 | if [ "${#specs[@]}" = "0" ]; then 19 | echo "No specs matched, nothing to do" 20 | exit 0 21 | fi 22 | 23 | # Used for caching rpms between builds 24 | rpm_file_root=/root/rpmbuild/RPMS/ 25 | function rpm_file_list { 26 | find "$rpm_file_root" -type f | sed "s|$rpm_file_root||" | sort 27 | } 28 | function file_hash { 29 | local spec="$1" 30 | local n=$(basename "$spec" .spec) 31 | local h=$(sha1sum "$1" | cut -d' ' -f1) 32 | echo "$n.$h" 33 | } 34 | function check_cache { 35 | local spec="$1" 36 | local h=$(file_hash "$spec") 37 | if [ -f "/cache/old/$h.tar" ]; then 38 | echo "* FOUND IN CACHE: $spec" 39 | tar -C "$rpm_file_root" -xvf "/cache/old/$h.tar" 40 | return 0 41 | fi 42 | return 1 43 | } 44 | cache= 45 | if [ ! -z "$BUILDER_CACHE" ] && [ ! -z "$BUILDER_CACHE_THIS" ]; then 46 | cache=1 47 | mkdir -p /cache/new 48 | fi 49 | 50 | set_rpm_versions 51 | set_python_src_versions 52 | # Parse the specfiles to evaluate conditionals for builddeps, and store them in tempfiles 53 | # Also check for specs we need to skip (BUILDER_SKIP) 54 | tmpdir=$(mktemp -d /tmp/build-specs.parsed.XXXXXX) 55 | trap "rm -rf -- '$tmpdir'" INT TERM HUP EXIT 56 | declare -A skip_specs # associative array (dict) 57 | if [ -x /usr/bin/rpmspec ]; then 58 | # RHEL >= 7 has this tool 59 | for spec in "${specs[@]}"; do 60 | # First check if we have the rpms cached 61 | if [ "$cache" = "1" ] && check_cache "$spec"; then 62 | skip_specs["$spec"]=1 63 | echo "::: $spec (cached)" 64 | continue 65 | fi 66 | 67 | name=$(basename "$spec") 68 | tmpfile="$tmpdir/$name" 69 | rpmspec -P "$spec" > "$tmpfile" 70 | if grep --silent 'BUILDER_SKIP' "$tmpfile"; then 71 | echo "BUILDER_SKIP: $spec will be skipped" 72 | skip_specs["$spec"]=1 73 | rm -f "$tmpfile" 74 | fi 75 | done 76 | touch "$tmpdir/__empty.spec" # To prevent an error because of an empty dir 77 | reqs=`$helpers/buildrequires-from-specs $tmpdir/*.spec` 78 | else 79 | # For RHEL 6 let's just try to install all we find 80 | # You can add 'BUILDER_EL6_SKIP' somewhere in the spec to skip it (comment is ok for this one) 81 | reqs=`$helpers/buildrequires-from-specs "${specs[@]}"` 82 | for spec in "${specs[@]}"; do 83 | # First check if we have the rpms cached 84 | if [ "$cache" = "1" ] && check_cache "$spec"; then 85 | skip_specs["$spec"]=1 86 | continue 87 | fi 88 | 89 | if grep --silent 'BUILDER_EL6_SKIP' "$spec"; then 90 | echo "BUILDER_EL6_SKIP: $spec will be skipped" 91 | skip_specs["$spec"]=1 92 | fi 93 | done 94 | fi 95 | 96 | set -ex 97 | 98 | if [ ! -z "$reqs" ]; then 99 | yum install -y $reqs 100 | fi 101 | 102 | function new_rpms { 103 | diff -u /tmp/rpms-before /tmp/rpms-after | tee /tmp/rpms-diff | grep -v '^[+][+]' | grep '^[+]' | sed 's/^[+]//' 104 | } 105 | 106 | rpmbuild_options="" 107 | if [ -n "${BUILDER_SKIP_CHECKS}" ]; then 108 | rpmbuild_options="--nocheck" 109 | fi 110 | 111 | for spec in "${specs[@]}"; do 112 | echo "===================================================================" 113 | echo "-> $spec" 114 | if [ -z "${skip_specs[$spec]}" ]; then 115 | echo "::: $spec" 116 | 117 | if [ -n "${BUILDER_EPOCH}" ] && grep -q BUILDER_RPM_VERSION "$spec"; then 118 | sed -i "/Name:/a Epoch: ${BUILDER_EPOCH}" "$spec" 119 | fi 120 | 121 | # Use the modification time of the spec file as the SOURCE_DATE_EPOCH if 122 | # BUILDER_SOURCE_DATE_FROM_SPEC_MTIME is set. This is useful for vendor packages 123 | # that have independent versioning. 124 | if [ -n "${BUILDER_SOURCE_DATE_FROM_SPEC_MTIME}" ]; then 125 | SOURCE_DATE_EPOCH=$(stat -c '%Y' "$spec") 126 | export SOURCE_DATE_EPOCH 127 | fi 128 | 129 | # Download sources 130 | spectool -g -R "$spec" 131 | 132 | # Build the rpm and record which files are new 133 | rpm_file_list > /tmp/rpms-before 134 | # NOTE: source_date_epoch_from_changelog is always overridden by SOURCE_DATE_EPOCH if that is set. 135 | # See https://fossies.org/linux/rpm/build/build.c#l_298 136 | rpmbuild \ 137 | ${rpmbuild_options} \ 138 | --define "_sdistdir /sdist" \ 139 | --define "_buildhost reproducible" \ 140 | --define "source_date_epoch_from_changelog Y" \ 141 | --define "clamp_mtime_to_source_date_epoch Y" \ 142 | --define "use_source_date_epoch_as_buildtime Y" \ 143 | -ba "$spec" 144 | rpm_file_list > /tmp/rpms-after 145 | 146 | new_rpms | sed 's/^/NEW: /' 147 | cat /tmp/rpms-diff | sed 's/^/DIFF: /' 148 | if [ "$cache" = "1" ]; then 149 | h=$(file_hash "$spec") 150 | tar -C "$rpm_file_root" -cvf "/cache/new/$h.tar" $(new_rpms) 151 | fi 152 | else 153 | echo "Skipping spec (BUILDER_SKIP or in cache)" 154 | fi 155 | done 156 | -------------------------------------------------------------------------------- /helpers/functions.sh: -------------------------------------------------------------------------------- 1 | # https://stackoverflow.com/questions/1527049/join-elements-of-an-array#17841619 2 | function join_by { local IFS="$1"; shift; echo "$*"; } 3 | 4 | set_python_src_versions() { 5 | # setuptools is very strict about PEP 440 6 | # See https://peps.python.org/pep-0440/ 7 | 8 | # BUILDER_VERSION BUILDER_PYTHON_SRC_VERSION 9 | # 1.2.3 => 1.2.3 10 | # 1.2.3.dirty => 1.2.3+dirty 11 | # 1.2.3.0.g123456 => 1.2.3+0.g123456 12 | # 1.2.3-alpha1 => 1.2.3a1 13 | # 1.2.3-alpha1.0.g123456 => 1.2.3a1+0.g123456 14 | # 1.2.3-alpha1.15.g123456 => 1.2.3a1+15.g123456 15 | # 1.2.3-rc2.12.branch.g123456 => 1.2.3rc2+branch.12.g123456 16 | # 1.2.3.15.mybranch.g123456 => 1.2.3+15.mybranch.g123456 17 | # 1.2.3.15.g123456 => 1.2.3+15.g123456 18 | # 1.2.3.15.g123456.dirty => 1.2.3+15.g123456.dirty 19 | # 1.2.3.130.HEAD.gbac839b2 => 1.2.3+130.head.gbac839b2 20 | export BUILDER_PYTHON_SRC_VERSION="$(echo ${BUILDER_VERSION} | perl -pe 's,-alpha([0-9]+),a\1+,' | perl -pe 's,-beta([0-9]+),b\1+,' | perl -pe 's,-rc([0-9]+),rc\1+,' | perl -pe 's,\+$,,' | perl -pe 's,\+\.,+,' | perl -pe 's,^([0-9]+\.[0-9]+\.[0-9]+)\.(.*)$,\1+\2,' | tr A-Z a-z )" 21 | } 22 | 23 | set_debian_versions() { 24 | # Examples (BUILDER_RELEASE before is assumed to be 1pdns 25 | # BUILDER_VERSION BUILDER_DEB_VERSION after BUILDER_DEB_RELEASE after 26 | # 1.2.3 => 1.2.3 1pdns 27 | # 1.2.3.0.g123456 => 1.2.3+0.g123456 1pdns 28 | # 1.2.3-alpha1 => 1.2.3~alpha1 1pdns 29 | # 1.2.3-alpha1.0.g123456 => 1.2.3~alpha1+0.g123456 1pdns 30 | # 1.2.3-alpha1.15.g123456 => 1.2.3~alpha1+15.g123456 1pdns 31 | # 1.2.3-rc2.12.branch.g123456 => 1.2.3~rc2+branch.12.g123456 1pdns 32 | # 1.2.3.15.mybranch.g123456 => 1.2.3+mybranch.15.g123456 1pdns 33 | # 1.2.3.15.g123456 => 1.2.3+15.g123456 1pdns 34 | OIFS=$IFS 35 | IFS='-' version_elems=($BUILDER_VERSION) 36 | IFS=$OIFS 37 | version='' 38 | if [ ${#version_elems[@]} -gt 1 ]; then 39 | version=${version_elems[0]} 40 | OIFS=$IFS 41 | IFS='.' version_elems=(${version_elems[1]}) 42 | IFS=$OIFS 43 | 44 | # version_elems now contains e.g. 45 | # alpha1 46 | # alpha1 dirty 47 | # alpha1 15 g123456 48 | # alpha1 15 g123456 dirty 49 | # alpha1 15 mybranch g123456 50 | # alpha1 15 mybranch g123456 dirty 51 | version="${version}~${version_elems[0]}" 52 | if [ ${#version_elems[@]} -eq 2 ]; then 53 | version="${version}+$(join_by . ${version_elems[@]:1})" 54 | fi 55 | if [ ${#version_elems[@]} -ge 3 ]; then 56 | if [[ "${version_elems[2]}" =~ ^g[0-9a-e]+ ]]; then 57 | version="${version}+$(join_by . ${version_elems[@]:1})" 58 | else 59 | version="${version}+${version_elems[2]}.${version_elems[1]}.$(join_by . ${version_elems[@]:3})" 60 | fi 61 | fi 62 | else 63 | OIFS=$IFS 64 | IFS='.' version_elems=(${BUILDER_VERSION}) 65 | IFS=$OIFS 66 | # version_elems now contains e.g. 67 | # 1 2 3 68 | # 1 2 3 15 g123456 69 | # 1 2 3 15 g123456 dirty 70 | # 1 2 3 15 mybranch g123456 71 | # 1 2 3 15 mybranch g123456 dirty 72 | version=$(join_by . ${version_elems[@]:0:3}) 73 | if [ ${#version_elems[@]} -eq 4 ]; then 74 | version="${version}+$(join_by . ${version_elems[@]:3})" 75 | fi 76 | if [ ${#version_elems[@]} -ge 5 ]; then 77 | if [[ "${version_elems[4]}" =~ ^g[0-9a-e]+ ]]; then 78 | version="${version}+$(join_by . ${version_elems[@]:3})" 79 | else 80 | version="${version}+${version_elems[4]}.${version_elems[3]}.$(join_by . ${version_elems[@]:5})" 81 | fi 82 | fi 83 | fi 84 | export BUILDER_DEB_VERSION=$version 85 | export BUILDER_DEB_RELEASE=${BUILDER_RELEASE} 86 | } 87 | 88 | set_rpm_versions() { 89 | # Examples (BUILDER_RELEASE before is assumed to be 1pdns 90 | # BUILDER_VERSION BUILDER_RPM_VERSION after BUILDER_RPM_RELEASE after 91 | # 1.2.3 => 1.2.3 1pdns 92 | # 1.2.3.dirty => 1.2.3 dirty.1pdns 93 | # 1.2.3.0.g123456 => 1.2.3 0.g123456.1pdns 94 | # 1.2.3.0.g123456.dirty => 1.2.3 0.g123456.dirty.1pdns 95 | # 1.2.3-alpha1 => 1.2.3 0.alpha1.1pdns 96 | # 1.2.3-alpha1.dirty => 1.2.3 0.alpha1.dirty.1pdns 97 | # 1.2.3-alpha1.0.g123456 => 1.2.3 0.alpha1.0.g12456.1pdns 98 | # 1.2.3-alpha1.0.g123456.dirty=> 1.2.3 0.alpha1.0.g12456.dirty.1pdns 99 | # 1.2.3-alpha1.15.g123456 => 1.2.3 0.alpha1.15.g12456.1pdns 100 | # 1.2.3-rc2.12.branch.g123456 => 1.2.3 0.rc2.branch.12.g123456.1pdns 101 | # 1.2.3.15.mybranch.g123456 => 1.2.3 mybranch.15.g123456.1pdns 102 | # 1.2.3.15.g123456 => 1.2.3 15.g123456.1pdns 103 | OIFS=$IFS 104 | IFS='-' version_elems=($BUILDER_VERSION) 105 | IFS=$OIFS 106 | prerel='' 107 | if [ ${#version_elems[@]} -gt 1 ]; then 108 | # There's a dash in the version number, indicating a pre-release 109 | # Take the version number 110 | BUILDER_RPM_VERSION=${version_elems[0]} 111 | OIFS=$IFS 112 | IFS='.' version_elems=(${version_elems[1]}) 113 | IFS=$OIFS 114 | prerel="0.${version_elems[0]}." 115 | version_elems=(${version_elems[@]:1}) 116 | else 117 | OIFS=$IFS 118 | IFS='.' version_elems=(${version_elems}) 119 | IFS=$OIFS 120 | BUILDER_RPM_VERSION=$(join_by . ${version_elems[@]:0:3}) 121 | version_elems=(${version_elems[@]:3}) 122 | fi 123 | 124 | # version_elems now contains everything _after_ the version, sans pre-release info 125 | # e.g. 126 | # (empty) 127 | # dirty 128 | # 0 g123456 129 | # 12 g123456 dirty 130 | # 12 branch g123456 131 | # 12 branch g123456 dirty 132 | release='' 133 | if [ ${#version_elems[@]} -eq 0 ]; then 134 | # This is a release 135 | export BUILDER_RPM_RELEASE="${prerel}${BUILDER_RELEASE}" 136 | elif [ ${#version_elems[@]} -le 2 ]; then 137 | export BUILDER_RPM_RELEASE="${prerel}$(join_by . ${version_elems[@]}).${BUILDER_RELEASE}" 138 | else 139 | if [[ "${version_elems[1]}" =~ ^g[0-9a-e]+ ]]; then 140 | export BUILDER_RPM_RELEASE="${prerel}$(join_by . ${version_elems[@]}).${BUILDER_RELEASE}" 141 | else 142 | export BUILDER_RPM_RELEASE="${prerel}${version_elems[1]}.${version_elems[0]}.$(join_by . ${version_elems[@]:2}).${BUILDER_RELEASE}" 143 | fi 144 | fi 145 | export BUILDER_RPM_VERSION 146 | } 147 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # PowerDNS Builder 2 | 3 | [![Build Status](https://travis-ci.org/PowerDNS/pdns-builder.svg?branch=master)](https://travis-ci.org/PowerDNS/pdns-builder) 4 | 5 | A reusable Docker based distribution package builder. 6 | 7 | ## Quickstart 8 | 9 | To only build generic source packages: 10 | 11 | ./builder/build.sh sdist 12 | 13 | To build for CentOS 7: 14 | 15 | ./builder/build.sh centos-7 16 | 17 | Packages will end up in `builder/tmp//`. 18 | 19 | The build script supports various commandline options. See: 20 | 21 | ./builder/build.sh -h 22 | 23 | 24 | ## Build requirements 25 | 26 | * Docker >= 17.05 27 | * git 28 | * bash 29 | * tree (optional) 30 | 31 | 32 | ## How does this work? 33 | 34 | The build process for distribution packages consists of three steps: 35 | 36 | 1. Create generic source distributions for all components. This step also 37 | performs any more complicated generic steps, like building plain dist assets 38 | using webpack. 39 | 40 | 2. Create rpms or debs from these source packages (and build specs) only, with 41 | no other access to the source, in a container with all build dependencies 42 | installed. 43 | 44 | 3. Install the distribution packages and test them in a clean container without 45 | the build dependencies. 46 | 47 | The `sdist` target only performs the first step. The install test is skippable 48 | with `-s`, but using this option is not recommended. 49 | 50 | The builder expects to be put in `builder/` in the repository to build and to 51 | find a `builder-support/` directory next to it with all repository specific 52 | build configurations. 53 | 54 | The implementation uses [Docker 17.05+ multi-stage builds][multistage] to 55 | implement these steps. `builder-support/dockerfiles/` is expecte to contain 56 | [simple templates][templ] for Dockerfiles that will be combined into a 57 | temporary Dockerfile for the build. 58 | This allows us to split different build steps and substeps into separate 59 | stages, while only having to trigger a single Docker build. 60 | 61 | The build script does not know or care how the actual build is performed, it just 62 | expects the build artifacts to end up in `/sdist` and `/dist` inside 63 | the final image after the build. 64 | 65 | [multistage]: https://docs.docker.com/engine/userguide/eng-image/multistage-build/ 66 | [templ]: ./templating/templating.sh 67 | 68 | 69 | ## Autobuild deployment notes 70 | 71 | If you want to run these builds on a frequent basis, such as in a buildbot that 72 | automatically builds on new commits, keep the following in mind: 73 | 74 | ### Disk usage 75 | 76 | Every build will create a fair amount of new Docker layers that will take up 77 | disk space. Make sure you have something like 20 GB or more disk space 78 | available for builds, and run the included [docker-cleanup.sh](docker-cleanup.sh) 79 | once a day in a cron job to remove containers and images that are no longer 80 | referenced by a tag. 81 | 82 | Do keep in mind that after a cleanup, a new build will have to start from 83 | scratch, so you do not want to run it too often. Once a day is probably a 84 | fair compromise between build time and disk usage, assuming you will not be 85 | building hundreds of times per day. 86 | 87 | ### Base image freshness 88 | 89 | Docker will never pull newer versions of the base images by itself. After a 90 | while the base images might become outdated. You could add an 91 | `apt update && apt upgrade` or equivalent to the start of each base image, but 92 | this will take longer and longer to execute. 93 | 94 | Instead, you could `docker pull ` in a cron job every night for every 95 | base image used in the builds. It's best to do this before the docker cleanup 96 | script, so that it can cleanup all the old layers from the previous image. 97 | 98 | Script to find all official images and pull them: 99 | 100 | images=`docker images --format '{{.Repository}}:{{.Tag}}' --filter dangling=false | grep -v '[_/-]'` 101 | for image in $images; do docker pull "$image"; done 102 | 103 | You probably do not want to add `set -e` here. 104 | 105 | NOTE: This will also try to pull any images you tagged locally without any 106 | `-`, `_` or `/` in the name, and skip any non-official Docker images. Please 107 | adapt to your use case. 108 | 109 | ### Concurrent builds 110 | 111 | With the current build script it is unsafe run several builds for the same 112 | target at the same time, because the temporary Dockerfile name and image tag 113 | will clash. We considered adding the version number to the tag, but this would 114 | make cleanup harder. Maybe we need to add an option to the build script to 115 | override the tag, if we need concurrent builds. 116 | 117 | Concurrent builds for different targets (like oraclelinux-6.8 and centos-7 118 | in parallel) should be safe, though. 119 | 120 | 121 | ## Implementation details 122 | 123 | ### Dockerfiles 124 | 125 | The Dockerfile templates are expected in `builder-support/dockerfiles/`. 126 | Note that these Dockerfiles are repository specific and not included with 127 | the builder distribution. 128 | 129 | The files that start with `Dockerfile.target.` are used as build targets. 130 | For example, `Dockerfile.target.centos-7` would be used for the `centos-7` target. 131 | `Dockerfile.target.sdist` is used for the `sdist` target, but also included by 132 | all the other targets to performs the source builds. 133 | 134 | To allow for reusability of include files, the following stage naming conventions 135 | should be observed: 136 | 137 | * `sdist` is the final source dist stage that contains all source dists in `/sdist`, 138 | which will be copied by the binary package builder. 139 | * `dist-base` is the stage used as base image for both the package builder and 140 | the installation test. 141 | * `package-builder` is the final binary package build stage that contains binary 142 | packages in `/dist`, which will be installed in the installation test. 143 | 144 | The last stage to appear in the Dockerfile will be the resulting image of the 145 | docker build. This one must have source dists in `/sdist` and binaries in 146 | `/dist`, as this is where the build scripts copies the result artifacts from. 147 | Please keep in mind that the test stage could be skipped, so these also have to 148 | exist at the end of the package builder stage. 149 | 150 | #### Docker caching 151 | 152 | If editing Dockerfiles, try to maximize the efficiency of docker layer caches. 153 | For example: 154 | 155 | * Only COPY/ADD files that are really needed at that point in the build process. 156 | For example, the installation tests live in a different folder than the 157 | build helpers, so that updating installation tests does not invalidate the 158 | layers that build the RPMs. 159 | * Vendor specs should be built before you COPY your source artifacts, so that 160 | they are only rebuilt if their spec files change and not every time your code 161 | changes. 162 | * For the same reason, build ARGs should be set as late as possible. 163 | * If you have a slow build step, like building an Angular project using Webpack, 164 | you should consider doing this in a separate stage and only apply any 165 | versioning after the actual build, so that the expensive steps can be cached. 166 | 167 | If you have a build step that relies on external, changing state (such as 168 | `apt-get update`), you may want to avoid caching this step forever. To do so, 169 | put `ARG BUILDER_CACHE_BUSTER=` before the step, and pass `-b daily` or `-b 170 | weekly` to build.sh. 171 | 172 | #### Templating 173 | 174 | Templating is done using a simple template engine written in bash. 175 | 176 | Example text template: 177 | 178 | Lines can start with @INCLUDE, @EVAL or @EXEC for special processing: 179 | @INCLUDE foo.txt 180 | @EVAL My home dir is $HOME 181 | @EXEC uname -a 182 | @EXEC [ "$foo" = "bar" ] && include bar.txt 183 | @IF [ "$foo" = "bar" ] 184 | This line is only printed if $foo = "bar" (cannot be nested) 185 | @INCLUDE bar.txt 186 | @ENDIF 187 | Other lines are printed unchanged. 188 | 189 | The commands behind `@EXEC` and `@IF` can be any bash commands. `include` is 190 | an internal bash function used to implement `@INCLUDE`. Note that `@IF` 191 | currently cannot be nested. 192 | 193 | The templating implementation can be found in `templating/templating.sh`. 194 | 195 | #### Post Build steps 196 | 197 | When certain steps or commands are needed after building, add an executable 198 | file called `post-build` to `builder-support`. After a build, this file will 199 | be run. 200 | 201 | 202 | ### Reproducible builds 203 | 204 | The builder has a few features to help with creating reproducible builds. 205 | 206 | The builder sets a `SOURCE_DATE_EPOCH` build argument with the timestamp of the last 207 | commit as the value. This is not automatically propagated to the build environment. 208 | If you want to use this, add this to your Dockerfile at the place where you want to 209 | start using it: 210 | 211 | ``` 212 | ARG SOURCE_DATE_EPOCH 213 | ``` 214 | 215 | This will probably be the same place that you inject the `BUILDER_VERSION`. 216 | 217 | For vendor dependency builds, you probably do not want to use it, as it could make their 218 | artifacts change with every version change. Instead, you may want to set the 219 | `BUILDER_SOURCE_DATE_FROM_SPEC_MTIME` env var when building RPMs. If this is set, the 220 | build script will use the modification time of the spec file as the `SOURCE_DATE_EPOCH`. 221 | Example usage: 222 | 223 | ``` 224 | RUN BUILDER_SOURCE_DATE_FROM_SPEC_MTIME=1 builder/helpers/build-specs.sh builder-support/vendor-specs/*.spec 225 | ``` 226 | 227 | The RPM build script always defines the following variables for reproducible RPM builds: 228 | 229 | ``` 230 | --define "_buildhost reproducible" 231 | --define "source_date_epoch_from_changelog Y" 232 | --define "clamp_mtime_to_source_date_epoch Y" 233 | --define "use_source_date_epoch_as_buildtime Y" 234 | ``` 235 | 236 | The `source_date_epoch_from_changelog` variable only has effect when no `SOURCE_DATE_EPOCH` is set. 237 | These variables are only supported in RHEL 8+ and derived distributions. RHEL 7 does not appear 238 | to support reproducible RPM builds. 239 | 240 | Keep in mind that the builder an only do so much, as any part of your build pipeline 241 | that creates non-reproducible artifacts will result in non-reproducible build output. 242 | For example, if the base image you use upgrades the compiler, the compiled output 243 | will likely change. 244 | 245 | 246 | --------------------------------------------------------------------------------