├── specs ├── models │ └── test.sh ├── features │ └── test.sh └── controllers │ └── test.sh ├── .buildkite ├── screenshot.png ├── template.yml └── pipeline.sh ├── LICENSE.md └── README.md /specs/models/test.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -eu 4 | 5 | echo "--- :thumbsup: Model tests" -------------------------------------------------------------------------------- /specs/features/test.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -eu 4 | 5 | echo "--- :thumbsup: Feature tests" -------------------------------------------------------------------------------- /specs/controllers/test.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -eu 4 | 5 | echo "--- :thumbsup: Controller tests" -------------------------------------------------------------------------------- /.buildkite/screenshot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/buildkite/dynamic-pipeline-example/HEAD/.buildkite/screenshot.png -------------------------------------------------------------------------------- /.buildkite/template.yml: -------------------------------------------------------------------------------- 1 | name: "Dynamic Pipeline Example" 2 | description: "An example pipeline that dynamically generates steps using a shell script." 3 | emoji: ":twisted_rightwards_arrows:" 4 | languages: 5 | - "Shell" 6 | steps: 7 | - label: ":pipeline:" 8 | command: ".buildkite/pipeline.sh | buildkite-agent pipeline upload" -------------------------------------------------------------------------------- /.buildkite/pipeline.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -eu 4 | 5 | echo "steps:" 6 | 7 | # Wait for the pipeline to be full generated before running subsequent steps 8 | # Can be omitted to run as uploaded! 9 | echo " - wait" 10 | 11 | # A step for each dir in specs/ 12 | 13 | find specs/* -type d | while read -r D; do 14 | echo " - command: \"$D/test.sh\"" 15 | echo " label: \"$(basename "$D")\"" 16 | done 17 | 18 | # A deploy step only if it's the main branch 19 | 20 | if [[ "$BUILDKITE_BRANCH" == "main" ]]; then 21 | echo " - wait" 22 | echo " - command: \"echo Deploy!\"" 23 | echo " label: \":rocket:\"" 24 | fi 25 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2025 Buildkite Pty Ltd 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Buildkite Dynamic Pipeline Steps Example 2 | 3 | [![Build status](https://badge.buildkite.com/c2e1cdc85d67dec83d2031e511bc208e911d178e54b59fec8b.svg?branch=main)](https://buildkite.com/buildkite/dynamic-pipeline-example/builds/latest?branch=main) 4 | [![Add to Buildkite](https://img.shields.io/badge/Add%20to%20Buildkite-14CC80)](https://buildkite.com/new) 5 | 6 | This repository is an example [Buildkite](https://buildkite.com/) pipeline that shows how to programmatically generate dynamic steps using a shell script. 7 | 8 | 👉 **See this example in action:** [buildkite/dynamic-pipeline-example](https://buildkite.com/buildkite/dynamic-pipeline-example/builds/latest?branch=main) 9 | 10 | See the full [Getting Started Guide](https://buildkite.com/docs/guides/getting-started) for step-by-step instructions on how to get this running, or try it yourself: 11 | 12 | [![Add to Buildkite](https://buildkite.com/button.svg)](https://buildkite.com/new) 13 | 14 | 15 | Screenshot of Buildkite dynamic pipeline example 16 | 17 | 18 | 19 | 20 | ## How does it work? 21 | 22 | This pipeline starts with a single job that runs: 23 | ```bash 24 | .buildkite/pipeline.sh | buildkite-agent pipeline upload 25 | ``` 26 | 27 | The script [.buildkite/pipeline.sh](.buildkite/pipeline.sh) does the following: 28 | * Creates a test step for each subdirectory in [specs](specs/) 29 | * Adds a deploy step if the build is on the `main` branch 30 | 31 | For non-main branches build it generates: 32 | 33 | ```yml 34 | steps: 35 | - command: "specs/controllers/test.sh" 36 | label: "controllers" 37 | - command: "specs/features/test.sh" 38 | label: "features" 39 | - command: "specs/models/test.sh" 40 | label: "models" 41 | ``` 42 | 43 | For a `main` branch build it generates: 44 | 45 | ```yml 46 | steps: 47 | - command: "specs/controllers/test.sh" 48 | label: "controllers" 49 | - command: "specs/features/test.sh" 50 | label: "features" 51 | - command: "specs/models/test.sh" 52 | label: "models" 53 | - wait 54 | - command: "echo Deploy!" 55 | label: ":rocket:" 56 | ``` 57 | 58 | ## More ideas 59 | 60 | What else could you do? The possibilities are endless. You can use this technique for custom deploy workflows, QA gates, conditional rollbacks, etc. 61 | 62 | Tools like [Jobsworth](https://github.com/saymedia/jobsworth) use dynamic steps to manage complex deployment logic. 63 | 64 | 65 | 66 | ## License 67 | 68 | See [LICENSE.md](LICENSE.md) (MIT) 69 | --------------------------------------------------------------------------------