├── License.txt ├── README.md ├── builder ├── freebsd.shlib ├── freebsd.startx ├── netbsd.shlib └── netbsd.startx ├── controller ├── index.js └── package.json ├── doc ├── deployment.png ├── deployment.puml ├── github.png ├── sequence.png └── sequence.puml ├── lib ├── config.shlib ├── deploy.shlib ├── event.shlib └── shared.shlib └── pmci /License.txt: -------------------------------------------------------------------------------- 1 | BSD 2-Clause License 2 | 3 | Copyright (c) 2018, Bill Zissimopoulos 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | * Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | * Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 17 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 18 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 20 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 21 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 22 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 23 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 24 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 25 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Poor Man's CI 2 | 3 | Poor Man's CI (PMCI - Poor Man's Continuous Integration) is a collection of scripts that taken together work as a simple CI solution that runs on Google Cloud. While there are many advanced hosted CI systems today, and many of them are free for open source projects, none of them seem to offer a solution for the BSD operating systems (FreeBSD, NetBSD, OpenBSD, etc.) 4 | 5 | The architecture of Poor Man's CI is system agnostic. However in the implementation provided in this repository the only supported systems are FreeBSD and NetBSD. Support for additional systems is possible. 6 | 7 | Poor Man's CI runs on the Google Cloud. It is possible to set it up so that the service fits within the Google Cloud ["Always Free"](https://cloud.google.com/free/docs/always-free-usage-limits) limits. In doing so the provided CI is not only hosted, but is also free! (Disclaimer: I am not affiliated with Google and do not otherwise endorse their products.) 8 | 9 | ## ARCHITECTURE 10 | 11 | A CI solution listens for "commit" (or more usually "push") events, builds the associated repository at the appropriate place in its history and reports the results. Poor Man's CI implements this very basic CI scenario using a simple architecture, which we present in this section. 12 | 13 | Poor Man's CI consists of the following components and their interactions: 14 | 15 | - **`Controller`**: Controls the overall process of accepting GitHub `push` events and starting builds. The `Controller` runs in the Cloud Functions environment and is implemented by the files in the `controller` source directory. It consists of the following components: 16 | - **`Listener`**: Listens for GitHub `push` events and posts them as `work` messages to the `workq` PubSub. 17 | - **`Dispatcher`**: Receives `work` messages from the `workq` PubSub and a free instance `name` from the `Builder Pool`. It instantiates a `builder` instance named `name` in the Compute Engine environment and passes it the link of a repository to build. 18 | - **`Collector`**: Receives `done` messages from the `doneq` PubSub and posts the freed instance `name` back to the `Builder Pool`. 19 | - **`PubSub Topics`**: 20 | - **`workq`**: Transports `work` messages that contain the link of the repository to build. 21 | - **`poolq`**: Implements the `Builder Pool`, which contains the `name`'s of available `builder` instances. To acquire a `builder` name, pull a message from the `poolq`. To release a `builder` name, post it back into the `poolq`. 22 | - **`doneq`**: Transports `done` messages (`builder` instance terminate and delete events). These message contain the `name` of freed `builder` instances. 23 | - **`builder`**: A `builder` is a Compute Engine instance that performs a build of a repository and shuts down when the build is complete. A `builder` is instantiated from a VM `image` and a `startx` (startup-exit) script. 24 | - **`Build Logs`**: A Storage bucket that contains the logs of builds performed by `builder` instances. 25 | - **`Logging Sink`**: A `Logging Sink` captures `builder` instance terminate and delete events and posts them into the `doneq`. 26 | 27 | A structural view of the system is presented below: 28 | 29 | ![Deployment Diagram](doc/deployment.png) 30 | 31 | A behavioral view of the system follows: 32 | 33 | ![Sequence Diagram](doc/sequence.png) 34 | 35 | ## DEPLOYMENT 36 | 37 | Prerequisites: 38 | 39 | - Empty project in Google Cloud. 40 | 41 | - Google Cloud SDK installed. 42 | 43 | - `gcloud init` command has been run. 44 | 45 | Instructions: 46 | - Obtain a `SECRET` that will guard access to your PMCI deployment. 47 | ``` 48 | $ openssl rand -hex 16 49 | SECRET 50 | ``` 51 | 52 | - Deploy PMCI to your project: 53 | ``` 54 | $ ./pmci deploy SECRET 55 | ``` 56 | 57 | - Obtain your personal access `TOKEN` by visiting github.com > Account > Settings > Developer settings > Personal access tokens. 58 | 59 | - On every project you want to use PMCI go to github.com > Project > Settings > Webhooks > Add Webhook. 60 | - URL: `https://REGION-PROJECT.cloudfunctions.net/listener?secret=SECRET&image=IMAGE&token=TOKEN` 61 | - Set `REGION` and `PROJECT` accordingly. 62 | - Set `IMAGE` to `freebsd` or `netbsd`. 63 | - Content-type: `application/json` 64 | - "Just the `push` event." 65 | 66 | - Add a shell script named `.pmci/IMAGE.sh` (where `IMAGE` is `freebsd` or `netbsd`) to your project. This script will be run by PMCI on every push. For example, here is my [cgofuse](https://github.com/billziss-gh/cgofuse) script for FreeBSD: 67 | ```shell 68 | set -ex 69 | 70 | # FUSE 71 | kldload fuse 72 | pkg install -y fusefs-libs 73 | 74 | # cgofuse: build and test 75 | export GOPATH=/tmp/go 76 | mkdir -p /tmp/go/src/github.com/billziss-gh 77 | cp -R /tmp/repo/cgofuse /tmp/go/src/github.com/billziss-gh 78 | cd /tmp/go/src/github.com/billziss-gh/cgofuse 79 | go build ./examples/memfs 80 | go build ./examples/passthrough 81 | go test -v ./fuse 82 | ``` 83 | 84 | - You should now have working FreeBSD and/or NetBSD builds! Try pushing something into your GitHub project. 85 | 86 | - To undeploy PMCI: 87 | ``` 88 | $ ./pmci undeploy 89 | ``` 90 | 91 | **NOTE**: The default deployment uses a single builder instance of `f1-micro` with a 30GB HDD. This fits within the "Always Free" tier and is therefore free. However it is also extremely slow and can even run out of memory when compiling bigger projects (e.g. it runs out of memory 1 out of 5 times when compiling Go in June 2018). Here are some ways to improve the performance: 92 | 93 | - Use a machine type that is faster and has more memory, such as `n1-standard-1`. 94 | 95 | - Use a larger HDD or an SSD. 96 | 97 | - FreeBSD only: Use a custom image that has already performed `firstboot`. The default FreeBSD image `freebsd-11-1-release-amd64` performs a system update and other expensive work when booted for the first time (i.e. the `/firstboot` file exists). An image that has already done this work boots much faster. 98 | ``` 99 | $ ./pmci freebsd_builder_create builder0 100 | # wait until builder has fully booted; it will do so twice; 101 | # when the login prompt is presented in the serial console proceed 102 | $ ./pmci builder_stop builder0 103 | $ ./pmci builder_image_create builder0 freebsd-builder 104 | $ ./pmci builder_delete builder0 105 | # now modify your controller/index.js file to point to your custom freebsd-builder image 106 | $ ./pmci deploy SECRET 107 | ``` 108 | 109 | ### BADGES 110 | 111 | PMCI supports status badges that show the last status of your build. Use them as follows in Markdown: 112 | 113 | Badge: 114 | ```markdown 115 | ![PMCI](http://storage.googleapis.com/PROJECT-logs/github.com/USER/REPO/IMAGE/badge.svg) 116 | ``` 117 | 118 | Badge that links to the build log: 119 | ```markdown 120 | [![PMCI](http://storage.googleapis.com/PROJECT-logs/github.com/USER/REPO/IMAGE/badge.svg)](http://storage.googleapis.com/PROJECT-logs/github.com/USER/REPO/IMAGE/build.html) 121 | ``` 122 | 123 | ## BUGS 124 | 125 | - The `Builder Pool` is currently implemented as a PubSub; messages in the PubSub contain the names of available `builder` instances. Unfortunately a PubSub retains its messages for a maximum of 7 days. It is therefore possible that messages will be discarded and that your PMCI deployment will suddenly find itself out of builder instances. If this happens you can reseed the `Builder Pool` by running the commands below. However this is a serious BUG that should be fixed. For a related discussion see https://tinyurl.com/ybkycuub. 126 | ``` 127 | $ ./pmci queue_post poolq builder0 128 | # ./pmci queue_post poolq builder1 129 | # ... repeat for as many builders as you want 130 | ``` 131 | 132 | - The `Dispatcher` is implemented as a Retry Background Cloud Function. It accepts `work` messages from the `workq` and attempts to pull a free `name` from the `poolq`. If that fails it returns an error, which instructs the infrastructure to retry. Because the infrastructure does not provide any retry controls, this currently happens immediately and the `Dispatcher` spins unproductively. This is currently mitigated by a "sleep" (`setTimeout`), but the Cloud Functions system still counts the Function as running and charges it accordingly. While this fits within the "Always Free" limits, it is something that should eventually be fixed (perhaps by the PubSub team). For a related discussion see https://tinyurl.com/yb2vbwfd. 133 | -------------------------------------------------------------------------------- /builder/freebsd.shlib: -------------------------------------------------------------------------------- 1 | # freebsd.shlib 2 | # 3 | # Copyright 2018 Bill Zissimopoulos 4 | 5 | # This file is part of "Poor Man's CI". 6 | # 7 | # It is licensed under the BSD license. The full license text can be found 8 | # in the License.txt file at the root of this project. 9 | 10 | Help="$Help 11 | freebsd_builder_create INSTANCE_NAMES BUILDER_ARG_NAME=VALUE" 12 | freebsd_builder_create() 13 | { 14 | local name="$1"; shift 15 | rm -f /tmp/startx.$$ 16 | for a in "$@"; do 17 | echo "$a" >>/tmp/startx.$$ 18 | done 19 | cat \ 20 | "$ProgDir/lib/config.shlib" \ 21 | "$ProgDir/lib/shared.shlib" \ 22 | "$ProgDir/builder/freebsd.startx" \ 23 | >>/tmp/startx.$$ 24 | builder_create \ 25 | --image-project=freebsd-org-cloud-dev \ 26 | --image=freebsd-11-1-release-amd64 \ 27 | --boot-disk-size="${BUILDER_BOOT_DISK_SIZE}GB" \ 28 | --metadata-from-file=startup-script=/tmp/startx.$$ \ 29 | "$name" 30 | local ec=$? 31 | rm -f /tmp/startx.$$ 32 | return $ec 33 | } 34 | -------------------------------------------------------------------------------- /builder/freebsd.startx: -------------------------------------------------------------------------------- 1 | # freebsd.startx 2 | # 3 | # Copyright 2018 Bill Zissimopoulos 4 | 5 | # This file is part of "Poor Man's CI". 6 | # 7 | # It is licensed under the BSD license. The full license text can be found 8 | # in the License.txt file at the root of this project. 9 | 10 | exec >>/var/log/startx.log 11 | exec 2>&1 12 | 13 | echo ">>>>STARTX $(date +%Y-%m-%dT%H:%M:%S%z)" 14 | trap 'set +x; echo "<<<>/etc/rc.conf 19 | 20 | # upgrade: cloud sdk; see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225255 21 | pkg delete -y google-cloud-sdk 22 | rm -rf /usr/local/google-cloud-sdk/ 23 | pkg install -y google-cloud-sdk 24 | # gcloud components update 25 | 26 | # install: base packages 27 | pkg install -y git 28 | pkg install -y go 29 | 30 | # install: custom Go 31 | # (fixes signal handling bug; remove when Go 1.11 is available) 32 | gsutil cp gs://$STORAGE_BUCKET/go-custom.freebsd-amd64.tar.xz /tmp/go-custom.tar.xz 33 | tar -xJf /tmp/go-custom.tar.xz -C /usr/local 34 | rm /tmp/go-custom.tar.xz 35 | else 36 | # use: custom Go 37 | export GOROOT=/usr/local/go-custom 38 | export PATH=/usr/local/go-custom/bin:$PATH 39 | 40 | # clone repo and build 41 | # also stops builder if started with $BUILDER_ARG_CLONE_URL 42 | mkdir -p /tmp/repo 43 | cd /tmp/repo 44 | builder_work 45 | fi 46 | -------------------------------------------------------------------------------- /builder/netbsd.shlib: -------------------------------------------------------------------------------- 1 | # netbsd.shlib 2 | # 3 | # Copyright 2018 Bill Zissimopoulos 4 | 5 | # This file is part of "Poor Man's CI". 6 | # 7 | # It is licensed under the BSD license. The full license text can be found 8 | # in the License.txt file at the root of this project. 9 | 10 | Help="$Help 11 | netbsd_builder_create INSTANCE_NAMES BUILDER_ARG_NAME=VALUE" 12 | netbsd_builder_create() 13 | { 14 | local name="$1"; shift 15 | rm -f /tmp/startx.$$ 16 | for a in "$@"; do 17 | echo "$a" >>/tmp/startx.$$ 18 | done 19 | cat \ 20 | "$ProgDir/lib/config.shlib" \ 21 | "$ProgDir/lib/shared.shlib" \ 22 | "$ProgDir/builder/netbsd.startx" \ 23 | >>/tmp/startx.$$ 24 | builder_create \ 25 | --image-project=poor-mans-ci \ 26 | --image=netbsd-builder \ 27 | --boot-disk-size="10GB" \ 28 | --metadata-from-file=startup-script=/tmp/startx.$$ \ 29 | "$name" 30 | local ec=$? 31 | rm -f /tmp/startx.$$ 32 | return $ec 33 | } 34 | -------------------------------------------------------------------------------- /builder/netbsd.startx: -------------------------------------------------------------------------------- 1 | # netbsd.startx 2 | # 3 | # Copyright 2018 Bill Zissimopoulos 4 | 5 | # This file is part of "Poor Man's CI". 6 | # 7 | # It is licensed under the BSD license. The full license text can be found 8 | # in the License.txt file at the root of this project. 9 | 10 | exec >>/var/log/startx.log 11 | exec 2>&1 12 | 13 | echo ">>>>STARTX $(date +%Y-%m-%dT%H:%M:%S%z)" 14 | trap 'set +x; echo "<<< 146 | { 147 | // check if the request has our secret 148 | if (req.query.secret != package.config.SECRET) 149 | { 150 | rsp.status(400).send("You don't know the password!") 151 | return 152 | } 153 | 154 | if (req.query.image === undefined || 155 | req.query.token === undefined) 156 | { 157 | rsp.status(400).send("query: missing image or token") 158 | return 159 | } 160 | 161 | if (!(req.query.image in vmconf)) 162 | { 163 | rsp.status(400).send("query: unknown image " + req.query.image) 164 | return 165 | } 166 | 167 | if (req.body.repository === undefined || 168 | req.body.repository.clone_url === undefined) 169 | { 170 | rsp.status(400).send("body: missing clone_url") 171 | return 172 | } 173 | 174 | if (req.body.after == "0000000000000000000000000000000000000000") 175 | { 176 | rsp.status(200).end() 177 | return 178 | } 179 | 180 | // post event to workq 181 | attributes = 182 | { 183 | image: req.query.image, 184 | token: req.query.token, 185 | clone_url: req.body.repository.clone_url, 186 | } 187 | if (req.body.after !== undefined) 188 | attributes.commit = req.body.after 189 | queue_post("workq", attributes). 190 | then(_ => 191 | { 192 | console.log("workq: posted work" + 193 | " on " + req.query.image + 194 | " for " + req.body.repository.clone_url + 195 | " commit " + req.body.after) 196 | rsp.status(200).end() 197 | }). 198 | catch(err => 199 | { 200 | console.error(err) 201 | rsp.status(500).end() 202 | }) 203 | } 204 | 205 | exports.dispatcher = (evt) => 206 | { 207 | // Retry Background Function: 208 | // - Promise.resolve means that message is accepted. 209 | // - Promise.reject means that message will be retried. 210 | 211 | // received message from workq 212 | message = evt.data 213 | 214 | if (message.attributes === undefined || 215 | message.attributes.image === undefined || 216 | message.attributes.token === undefined || 217 | message.attributes.clone_url === undefined) 218 | // invalid workq message; do not retry 219 | return Promise.resolve("workq: skip invalid message: " + String(message)) 220 | 221 | // have work; pull instance from poolq 222 | return queue_recv("poolq"). 223 | then(responses => 224 | { 225 | response = responses[0] 226 | received = response.received_messages 227 | if (response.received_messages === undefined) 228 | received = response.receivedMessages 229 | if (received === undefined || 230 | received.length == 0) 231 | { 232 | // HACK: if we reject this message it will be retried immediately, 233 | // and we will spin unproductively. Ideally the infrastructure would 234 | // give us some control over when to be retried, but this is not the 235 | // case. So let's try the next best thing: sleep for a while instead. 236 | // 237 | // See https://tinyurl.com/yb2vbwfd 238 | // 239 | // Remove this if PubSub implements retry controls. 240 | return new Promise(resolve => setTimeout(resolve, 60000)). 241 | then(_ => 242 | { 243 | return Promise.reject("poolq: no available instance") 244 | }) 245 | 246 | // no pool instance; retry 247 | //return Promise.reject("poolq: no available instance") 248 | } 249 | 250 | ackId = received[0].ack_id 251 | if (ackId === undefined) 252 | ackId = received[0].ackId 253 | poolmsg = received[0].message 254 | if (poolmsg.attributes === undefined || 255 | poolmsg.attributes.instance === undefined) 256 | { 257 | // invalid poolq message; acknowledge it and retry the workq message 258 | queue_ack("poolq", ackId) 259 | return Promise.reject("poolq: skip invalid message: " + String(poolmsg)) 260 | } 261 | 262 | console.log("poolq: received instance " + poolmsg.attributes.instance) 263 | 264 | // have builder instance name; create new builder 265 | return builder_create( 266 | message.attributes.image, 267 | poolmsg.attributes.instance, 268 | message.attributes.token, 269 | message.attributes.clone_url, 270 | message.attributes.commit). 271 | then(_ => 272 | { 273 | // acknowledge poolq message 274 | return queue_ack("poolq", ackId) 275 | }). 276 | then(_ => 277 | { 278 | console.log("builder: created " + 279 | message.attributes.image + " instance " + poolmsg.attributes.instance + 280 | " for " + message.attributes.clone_url + 281 | " commit " + message.attributes.commit) 282 | }) 283 | }) 284 | } 285 | 286 | exports.collector = (evt) => 287 | { 288 | // NO-Retry Background Function: 289 | // - Promise.resolve means that message is accepted. 290 | // - Promise.reject means error, but message is NOT retried. 291 | 292 | // received message from doneq 293 | message = evt.data 294 | 295 | if (message.data == null) 296 | return Promise.reject("doneq: skip invalid message: " + String(message)) 297 | 298 | message = JSON.parse(Buffer(message.data, "base64").toString()).jsonPayload 299 | instance = message.resource.name 300 | 301 | if (!instance.startsWith("builder")) 302 | return Promise.resolve("doneq: ignoring non-builder instance " + instance) 303 | 304 | switch (message.event_subtype) 305 | { 306 | case "compute.instances.stop": 307 | case "compute.instances.guestTerminate": 308 | return zone.vm(instance).delete(). 309 | then(_ => 310 | { 311 | console.log("builder: deleted " + instance) 312 | }) 313 | case "compute.instances.delete": 314 | attributes = 315 | { 316 | instance: instance, 317 | } 318 | return queue_post("poolq", attributes). 319 | then(_ => 320 | { 321 | console.log("poolq: posted instance " + instance) 322 | }) 323 | default: 324 | return Promise.reject("doneq: invalid message: unknown event_subtype: " + String(message)) 325 | } 326 | } 327 | 328 | function queue_post(topic, attributes) 329 | { 330 | message = 331 | { 332 | data: "", 333 | attributes: attributes, 334 | } 335 | request = 336 | { 337 | topic: pubcli.topicPath(project, topic), 338 | messages: [message], 339 | } 340 | return pubcli.publish(request) 341 | } 342 | 343 | function queue_recv(topic) 344 | { 345 | request = 346 | { 347 | subscription: subcli.subscriptionPath(project, topic), 348 | maxMessages: 1, 349 | returnImmediately: true, 350 | } 351 | return subcli.pull(request) 352 | } 353 | 354 | function queue_ack(topic, ackId) 355 | { 356 | request = 357 | { 358 | subscription: subcli.subscriptionPath(project, topic), 359 | ackIds: [ackId], 360 | } 361 | return subcli.acknowledge(request) 362 | } 363 | 364 | function builder_create(image, instance, token, clone_url, commit) 365 | { 366 | // prepare builder args 367 | args = "" 368 | args += `BUILDER_ARG_SRCHOST_TOKEN=${token}\n` 369 | args += `BUILDER_ARG_CLONE_URL=${clone_url}\n` 370 | if (commit !== undefined) 371 | args += `BUILDER_ARG_COMMIT=${commit}\n` 372 | 373 | // create builder instance 374 | conf = JSON.parse(JSON.stringify(vmconf[image])) 375 | conf.metadata.items[0].value = args + conf.metadata.items[0].value 376 | return zone.vm(instance).create(conf) 377 | } 378 | -------------------------------------------------------------------------------- /controller/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "dependencies": { 3 | "@google-cloud/compute": "^0.10.0", 4 | "@google-cloud/pubsub": "^0.18.0" 5 | } 6 | } -------------------------------------------------------------------------------- /doc/deployment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/billziss-gh/pmci/47c0ed4bc18295abf7dc581413418c77af4ec504/doc/deployment.png -------------------------------------------------------------------------------- /doc/deployment.puml: -------------------------------------------------------------------------------- 1 | skinparam componentStyle uml2 2 | 3 | artifact controller 4 | together { 5 | artifact startx 6 | artifact image 7 | } 8 | 9 | together { 10 | cloud "" as GitHub 11 | 12 | node "Cloud Functions" { 13 | component Controller { 14 | component Listener 15 | component Dispatcher 16 | component Collector 17 | } 18 | } 19 | 20 | node "Compute Engine" { 21 | node builder 22 | } 23 | } 24 | 25 | together { 26 | queue workq 27 | component "Builder\nPool" { 28 | queue poolq 29 | } 30 | queue doneq 31 | entity "Logging Sink" as done 32 | } 33 | 34 | database Storage { 35 | database "Build\nLogs" as Logs 36 | } 37 | 38 | controller .down.> Listener: <> 39 | controller .down.> Dispatcher: <> 40 | controller .down.> Collector: <> 41 | 42 | image .down.> builder: <> 43 | startx .down.> builder: <> 44 | 45 | GitHub -right- Listener: HTTP 46 | Listener -down-> workq: publish 47 | Dispatcher <-down- workq: push 48 | Dispatcher <-down- poolq: pull 49 | Collector -down-> poolq: publish 50 | Collector <-down- doneq: push 51 | 52 | done -left-> doneq: publish 53 | 54 | builder -right- Logs -------------------------------------------------------------------------------- /doc/github.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/billziss-gh/pmci/47c0ed4bc18295abf7dc581413418c77af4ec504/doc/github.png -------------------------------------------------------------------------------- /doc/sequence.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/billziss-gh/pmci/47c0ed4bc18295abf7dc581413418c77af4ec504/doc/sequence.png -------------------------------------------------------------------------------- /doc/sequence.puml: -------------------------------------------------------------------------------- 1 | hide footbox 2 | skinparam BoxPadding 40 3 | 4 | participant "" as GitHub 5 | box "Controller" 6 | participant Listener 7 | participant Dispatcher 8 | participant Collector 9 | end box 10 | participant "Builder\nPool" as Pool 11 | 12 | GitHub ->> Listener: push(repo) 13 | activate Listener 14 | deactivate Listener 15 | loop retry delivery while\nno builder available 16 | Listener ->> Dispatcher: work(repo) 17 | activate Dispatcher 18 | end 19 | Pool ->> Dispatcher: name 20 | create builder 21 | Dispatcher -> builder: new(name, repo) 22 | activate builder 23 | deactivate Dispatcher 24 | builder -> builder: build(repo) 25 | activate builder #Salmon 26 | deactivate builder 27 | builder -> "Build\nLogs": store(build_log) 28 | builder ->> GitHub: status 29 | builder ->> Collector: done(name) 30 | destroy builder 31 | activate Collector 32 | Collector ->> Pool: name 33 | deactivate Collector 34 | note over builder #Salmon 35 | Salmon color: 36 | CI processing. 37 | end note 38 | -------------------------------------------------------------------------------- /lib/config.shlib: -------------------------------------------------------------------------------- 1 | # config.shlib 2 | # 3 | # Copyright 2018 Bill Zissimopoulos 4 | 5 | # This file is part of "Poor Man's CI". 6 | # 7 | # It is licensed under the BSD license. The full license text can be found 8 | # in the License.txt file at the root of this project. 9 | 10 | PROJECT="poor-mans-ci" 11 | REGION="us-central1" 12 | BUILDER_ZONE="us-central1-c" 13 | BUILDER_MACHINE_TYPE="f1-micro" 14 | BUILDER_MIN_CPU_PLATFORM="Intel Skylake" 15 | BUILDER_BOOT_DISK_SIZE="30" 16 | BUILDER_SERVICE_ACCOUNT="default" 17 | STORAGE_BUCKET="pmci" 18 | STORAGE_BUCKET_LOGS="pmci-logs" 19 | 20 | Help="$Help 21 | config_json VARIABLES... # read stdin as JSON and update 'config' entry" 22 | config_json() 23 | { 24 | ( 25 | if [ "$BUILDER_SERVICE_ACCOUNT" == "default" ]; then 26 | BUILDER_SERVICE_ACCOUNT="$(gcloud iam service-accounts list "--format=value(email)" | \ 27 | grep compute@developer.gserviceaccount.com)" 28 | fi 29 | set -- \ 30 | PROJECT \ 31 | REGION \ 32 | BUILDER_ZONE \ 33 | BUILDER_MACHINE_TYPE \ 34 | BUILDER_MIN_CPU_PLATFORM \ 35 | BUILDER_BOOT_DISK_SIZE \ 36 | BUILDER_SERVICE_ACCOUNT \ 37 | STORAGE_BUCKET_LOGS \ 38 | "$@" 39 | for k in "$@"; do 40 | eval v="\$$k" 41 | export $k="$v" 42 | done 43 | python -c ' 44 | import json, os, sys 45 | obj = json.load(sys.stdin) 46 | cfg = obj.setdefault("config", {}) 47 | for k in sys.argv[1:]: 48 | cfg[k] = os.environ.get(k, "") 49 | json.dump(obj, sys.stdout, sort_keys=True, indent=4, separators=(",", ": ")) 50 | ' "$@" 51 | ) 52 | } 53 | -------------------------------------------------------------------------------- /lib/deploy.shlib: -------------------------------------------------------------------------------- 1 | # deploy.shlib 2 | # 3 | # Copyright 2018 Bill Zissimopoulos 4 | 5 | # This file is part of "Poor Man's CI". 6 | # 7 | # It is licensed under the BSD license. The full license text can be found 8 | # in the License.txt file at the root of this project. 9 | 10 | Help="$Help 11 | deploy SECRET # deploy to Google Cloud" 12 | deploy() 13 | { 14 | queue_topic_create workq 300 15 | queue_topic_create doneq 300 16 | queue_create poolq 300 17 | builder_done_sink_create done doneq 18 | 19 | deploy_controller "$1" 20 | 21 | # seed the pool 22 | queue_post poolq instance=builder0 >/dev/null 23 | queue_post poolq instance=builder1 >/dev/null 24 | } 25 | 26 | Help="$Help 27 | deploy_controller SECRET # deploy controller to Google Cloud" 28 | deploy_controller() 29 | { 30 | rm -rf /tmp/generate.$$ 31 | mkdir /tmp/generate.$$ 32 | _prepare_controller_package "$1" 33 | function_deploy_topic doneq collector --source=/tmp/generate.$$ 34 | function_deploy_topic workq dispatcher --timeout=300s --retry --source=/tmp/generate.$$ 35 | function_deploy_http listener --source=/tmp/generate.$$ 36 | rm -rf /tmp/generate.$$ 37 | } 38 | 39 | _prepare_controller_package() 40 | { 41 | cp -R "$ProgDir/controller"/* /tmp/generate.$$ 42 | local SECRET="${1:-$(keyring get PMCI secret)}" 43 | local FREEBSD_BUILDER_STARTX=$(cat \ 44 | "$ProgDir/lib/config.shlib" \ 45 | "$ProgDir/lib/shared.shlib" \ 46 | "$ProgDir/builder/freebsd.startx" \ 47 | ) 48 | local NETBSD_BUILDER_STARTX=$(cat \ 49 | "$ProgDir/lib/config.shlib" \ 50 | "$ProgDir/lib/shared.shlib" \ 51 | "$ProgDir/builder/netbsd.startx" \ 52 | ) 53 | config_json SECRET \ 54 | FREEBSD_BUILDER_STARTX NETBSD_BUILDER_STARTX \ 55 | /tmp/generate.$$/package.json.new 57 | mv /tmp/generate.$$/package.json.new /tmp/generate.$$/package.json 58 | } 59 | 60 | Help="$Help 61 | undeploy # undeploy from Google Cloud" 62 | undeploy() 63 | { 64 | function_delete listener 65 | function_delete dispatcher 66 | function_delete collector 67 | 68 | builder_done_sink_delete done 69 | queue_delete poolq 70 | queue_topic_delete doneq 71 | queue_topic_delete workq 72 | } 73 | -------------------------------------------------------------------------------- /lib/event.shlib: -------------------------------------------------------------------------------- 1 | # event.shlib 2 | # 3 | # Copyright 2018 Bill Zissimopoulos 4 | 5 | # This file is part of "Poor Man's CI". 6 | # 7 | # It is licensed under the BSD license. The full license text can be found 8 | # in the License.txt file at the root of this project. 9 | 10 | Help="$Help 11 | github_push IMAGE USER/REPO [COMMIT] # requires GitHub secret and token in keyring" 12 | github_push() 13 | { 14 | local secret="$(keyring get PMCI secret)" 15 | local token="$(keyring get PMCI github)" 16 | local image="$1" 17 | local clone_url="https://github.com/$2.git" 18 | 19 | local data 20 | if [ -n "$3" ]; then 21 | data="{ \ 22 | \"after\": \"$3\", \ 23 | \"repository\": { \ 24 | \"clone_url\": \"$clone_url\" \ 25 | } \ 26 | }" 27 | else 28 | data="{ \ 29 | \"repository\": { \ 30 | \"clone_url\": \"$clone_url\" \ 31 | } \ 32 | }" 33 | fi 34 | 35 | curl -X POST \ 36 | "https://$REGION-$PROJECT.cloudfunctions.net/listener?secret=$secret&image=$image&token=$token" \ 37 | -H "Content-type: application/json" \ 38 | -d "$data" 39 | echo 40 | } 41 | -------------------------------------------------------------------------------- /lib/shared.shlib: -------------------------------------------------------------------------------- 1 | # shared.shlib 2 | # 3 | # Copyright 2018 Bill Zissimopoulos 4 | 5 | # This file is part of "Poor Man's CI". 6 | # 7 | # It is licensed under the BSD license. The full license text can be found 8 | # in the License.txt file at the root of this project. 9 | 10 | Help="$Help 11 | builder_create INSTANCE_NAMES [FLAGS]..." 12 | builder_create() 13 | { 14 | gcloud beta compute instances create \ 15 | --machine-type="$BUILDER_MACHINE_TYPE" \ 16 | --min-cpu-platform="$BUILDER_MIN_CPU_PLATFORM" \ 17 | --maintenance-policy=MIGRATE \ 18 | --scopes=default,compute-rw,storage-rw \ 19 | "$@" 20 | } 21 | 22 | Help="$Help 23 | builder_delete INSTANCE_NAMES [FLAGS]..." 24 | builder_delete() 25 | { 26 | gcloud compute instances delete \ 27 | "$@" 28 | } 29 | 30 | Help="$Help 31 | builder_done_sink_create SINK_NAME QUEUE_NAME" 32 | builder_done_sink_create() 33 | { 34 | local sink=$1 35 | local queue=$2 36 | gcloud logging sinks create --no-user-output-enabled \ 37 | $sink pubsub.googleapis.com/projects/$PROJECT/topics/$queue \ 38 | --log-filter 'resource.type="gce_instance" 39 | logName="projects/'$PROJECT'/logs/compute.googleapis.com%2Factivity_log" 40 | jsonPayload.event_type:"GCE_OPERATION_DONE" 41 | (jsonPayload.event_subtype:"compute.instances.stop" OR jsonPayload.event_subtype:"compute.instances.guestTerminate" OR jsonPayload.event_subtype:"compute.instances.delete")' 42 | set -- $(gcloud logging sinks describe $sink | grep ^writerIdentity:) 43 | gcloud projects add-iam-policy-binding $PROJECT \ 44 | --member=$2 \ 45 | --role=roles/pubsub.publisher >/dev/null 46 | } 47 | 48 | Help="$Help 49 | builder_done_sink_delete SINK_NAME" 50 | builder_done_sink_delete() 51 | { 52 | local sink=$1 53 | set -- $(gcloud logging sinks describe $sink | grep ^writerIdentity:) 54 | gcloud projects remove-iam-policy-binding $PROJECT \ 55 | --member=$2 \ 56 | --role=roles/pubsub.publisher >/dev/null 57 | gcloud logging sinks delete --quiet $sink 58 | } 59 | 60 | Help="$Help 61 | builder_image_create SOURCE_DISK IMAGE_NAME" 62 | builder_image_create() 63 | { 64 | local disk="$1" 65 | local image="$2" 66 | gcloud compute images create "$image" \ 67 | --source-disk="$disk" \ 68 | --source-disk-zone="$BUILDER_ZONE" 69 | } 70 | 71 | Help="$Help 72 | builder_image_delete IMAGE_NAME" 73 | builder_image_delete() 74 | { 75 | local image="$1" 76 | gcloud compute images delete "$image" 77 | } 78 | 79 | Help="$Help 80 | builder_name # works only within GCE" 81 | builder_name() 82 | { 83 | if [ -x /usr/local/share/google/get_metadata_value ]; then 84 | /usr/local/share/google/get_metadata_value name 85 | else 86 | curl http://metadata.google.internal/computeMetadata/v1/instance/name \ 87 | -H Metadata-Flavor:Google 88 | fi 89 | } 90 | 91 | Help="$Help 92 | builder_stop INSTANCE_NAMES" 93 | builder_stop() 94 | { 95 | gcloud compute instances stop \ 96 | "$@" 97 | } 98 | 99 | Help="$Help 100 | builder_work # works only within GCE" 101 | builder_work() 102 | { 103 | local clone_url="$BUILDER_ARG_CLONE_URL" 104 | local commit=$BUILDER_ARG_COMMIT 105 | 106 | if [ -n "$clone_url" ]; then 107 | if git clone "$clone_url"; then 108 | local srchostpath=${clone_url#*//} 109 | local srchost=$(dirname $(dirname "$srchostpath")) 110 | local user=$(basename $(dirname "$srchostpath")) 111 | local repo=$(basename "$srchostpath") 112 | repo="${repo%.*}" 113 | 114 | cd "$repo" 115 | 116 | [ -n "$commit" ] && git checkout "$commit" 117 | commit=$(git rev-parse HEAD) 118 | 119 | local os=$(uname -s | tr A-Z a-z) 120 | if [ -f .pmci/$os.sh ]; then 121 | sh .pmci/$os.sh >/var/log/build.log 2>&1 122 | 123 | local ec=$? 124 | local badge 125 | if [ $ec -eq 0 ]; then 126 | echo >>/var/log/build.log 127 | echo BUILD SUCCESS >>/var/log/build.log 128 | badge=build-success 129 | else 130 | echo >>/var/log/build.log 131 | echo BUILD FAILURE >>/var/log/build.log 132 | badge=build-failure 133 | fi 134 | 135 | echo "" >/tmp/build.html 136 | 137 | gsutil \ 138 | -h "Content-type:text/plain" \ 139 | -h "Content-Disposition" \ 140 | -h "Cache-Control:public,max-age=3600" \ 141 | cp \ 142 | -a public-read \ 143 | /var/log/build.log \ 144 | gs://$STORAGE_BUCKET_LOGS/$srchost/$user/$repo/$os/$commit.log 145 | 146 | gsutil \ 147 | -h "Content-type:text/html" \ 148 | -h "Content-Disposition" \ 149 | -h "Cache-Control:private" \ 150 | cp \ 151 | -a public-read \ 152 | /tmp/build.html \ 153 | gs://$STORAGE_BUCKET_LOGS/$srchost/$user/$repo/$os/build.html 154 | 155 | gsutil \ 156 | -h "Content-type:image/svg+xml" \ 157 | -h "Content-Disposition" \ 158 | -h "Cache-Control:private" \ 159 | cp \ 160 | -a public-read \ 161 | gs://$STORAGE_BUCKET/$badge.svg \ 162 | gs://$STORAGE_BUCKET_LOGS/$srchost/$user/$repo/$os/badge.svg 163 | 164 | if [ $ec -eq 0 ]; then 165 | srchost_post_status $srchost $user $repo $os $commit success 166 | else 167 | srchost_post_status $srchost $user $repo $os $commit failure 168 | fi 169 | fi 170 | fi 171 | 172 | # shutdown -p now 173 | builder_stop --zone $BUILDER_ZONE $(builder_name) 174 | fi 175 | } 176 | 177 | Help="$Help 178 | srchost_post_status SRCHOST USER REPO OS COMMIT STATUS # works only within GCE" 179 | srchost_post_status() 180 | { 181 | local srchost=$1 182 | local user=$2 183 | local repo=$3 184 | local os=$4 185 | local commit=$5 186 | local status=$6 187 | curl -s -S -X POST https://api.github.com/repos/$user/$repo/statuses/$commit \ 188 | -H "Authorization: token $BUILDER_ARG_SRCHOST_TOKEN" \ 189 | -d "{ \ 190 | \"context\": \"continuous-integration/pmci/$os\", \ 191 | \"state\": \"$status\", \ 192 | \"target_url\": \ 193 | \"http://storage.googleapis.com/$STORAGE_BUCKET_LOGS/$srchost/$user/$repo/$os/$commit.log\" \ 194 | }" \ 195 | --retry 5 \ 196 | --retry-max-time 30 197 | } 198 | 199 | Help="$Help 200 | function_deploy_http NAME ..." 201 | function_deploy_http() 202 | { 203 | gcloud functions deploy --memory=128MB --trigger-http "$@" >/dev/null 204 | } 205 | 206 | Help="$Help 207 | function_deploy_topic QUEUE_NAME NAME ..." 208 | function_deploy_topic() 209 | { 210 | local queue_name=$1; shift 211 | gcloud functions deploy --memory=128MB --trigger-topic=$queue_name "$@" >/dev/null 212 | } 213 | 214 | Help="$Help 215 | function_delete NAME ..." 216 | function_delete() 217 | { 218 | gcloud functions delete --quiet "$@" 219 | } 220 | 221 | Help="$Help 222 | queue_create QUEUE_NAME ACK_DEADLINE" 223 | queue_create() 224 | { 225 | gcloud pubsub topics create $1 226 | gcloud pubsub subscriptions create $1 --topic=$1 --ack-deadline=$2 227 | } 228 | 229 | Help="$Help 230 | queue_topic_create QUEUE_NAME" 231 | queue_topic_create() 232 | { 233 | gcloud pubsub topics create $1 234 | } 235 | 236 | Help="$Help 237 | queue_delete QUEUE_NAME" 238 | queue_delete() 239 | { 240 | gcloud pubsub topics delete $1 241 | gcloud pubsub subscriptions delete $1 242 | } 243 | 244 | Help="$Help 245 | queue_topic_delete QUEUE_NAME" 246 | queue_topic_delete() 247 | { 248 | gcloud pubsub topics delete $1 249 | } 250 | 251 | Help="$Help 252 | queue_post QUEUE_NAME NAME=VALUE[,...]" 253 | queue_post() 254 | { 255 | gcloud pubsub topics publish $1 "--attribute=$2" 256 | } 257 | 258 | Help="$Help 259 | queue_recv QUEUE_NAME" 260 | queue_recv() 261 | { 262 | gcloud pubsub subscriptions pull $1 --auto-ack "--format=value(message.attributes)" 263 | } 264 | 265 | Help="$Help 266 | queue_peek QUEUE_NAME" 267 | queue_peek() 268 | { 269 | gcloud pubsub subscriptions pull $1 "--format=value(message.attributes)" 270 | } 271 | 272 | version() 273 | { 274 | echo "0.9" 275 | } 276 | -------------------------------------------------------------------------------- /pmci: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | # pmci 4 | # 5 | # Copyright 2018 Bill Zissimopoulos 6 | 7 | # This file is part of "Poor Man's CI". 8 | # 9 | # It is licensed under the BSD license. The full license text can be found 10 | # in the License.txt file at the root of this project. 11 | 12 | ProgDir=$(cd $(dirname "$0") && pwd) 13 | ProgName=$(basename "$0") 14 | 15 | Help="" 16 | for f in event deploy config shared; do 17 | . $ProgDir/lib/$f.shlib 18 | done 19 | for f in $(find $ProgDir/builder -type f -name '*.shlib' | sort); do 20 | . $f 21 | done 22 | 23 | if [ $# -eq 0 ]; then 24 | echo "usage: $ProgName COMMAND ARGS...\n" 1>&2 25 | echo "$Help" | while read name desc; do 26 | [ -n "$name" ] && echo " ${name}\n \t${desc}" 1>&2 27 | done 28 | exit 2 29 | fi 30 | 31 | # set -ex 32 | "$@" 33 | --------------------------------------------------------------------------------