├── .gitignore ├── ALPHA-GETTING-STARTED.md ├── MPTGraphic.png ├── PIPELINE_TEMPLATES.md ├── README.md ├── examples ├── README.md └── pipeline-templates-v1 │ ├── bake-deploy │ ├── README.md │ ├── bake-deploy-config.yml │ ├── findImage-deploy-config.yml │ └── template.yml │ ├── complex-wait │ ├── README.md │ ├── child-2-template.yml │ ├── child-template.yml │ ├── combined-template.yml │ ├── complex-wait-ui.png │ ├── mypipeline-config.yml │ ├── mypipelineCombined-config.yml │ ├── only-config.yml │ └── root-template.yml │ └── configuration-inheritance │ ├── README.md │ ├── configuration-inheritance-config.yml │ └── template.yml └── schemas └── pipeline-templates-v1 ├── configuration-schema.yml └── template-schema.yml /.gitignore: -------------------------------------------------------------------------------- 1 | gh-md-toc 2 | scratch 3 | -------------------------------------------------------------------------------- /ALPHA-GETTING-STARTED.md: -------------------------------------------------------------------------------- 1 | # alpha getting started 2 | 3 | This document is intended to help people get started with declarative pipelines. 4 | Things are still in a highly volatile state, but if you're looking to evaluate 5 | or help contribute bringing pipeline templates to a more stable place, this doc 6 | will help you get going. 7 | 8 | ## enabling pipeline templates 9 | 10 | ### With halyard 11 | 12 | With recent versions of halyard you can 13 | 14 | ``` 15 | hal config features edit --pipeline-templates true 16 | ``` 17 | 18 | then 19 | 20 | ``` 21 | hal deploy apply 22 | ``` 23 | 24 | ### Alternatively 25 | 26 | In `orca-local.yml`: 27 | 28 | ``` 29 | pipelineTemplates: 30 | enabled: true 31 | jinja: 32 | enabled: true 33 | ``` 34 | 35 | A handlebars renderer is currently in the codebase as well, but is being 36 | replaced by Jinja, so just enable that now to avoid hassles in the future. 37 | 38 | #### enabling UI support in Deck 39 | 40 | In `settings.js` add the following to the `features` map: 41 | ``` 42 | pipelineTemplates: true 43 | ``` 44 | 45 | ## Create a template 46 | 47 | Templates are currently only resolvable via HTTP(S). For evaluation and early 48 | development efforts, I found creating gists and linking to their raw content 49 | was the easiest way to iterate. 50 | 51 | Here's a [super barebones template](https://gist.githubusercontent.com/robzienert/04f326f3077df176b1788b30e06ed981/raw/b9eed8643e9028d27f21c3dee7ca3b0b1f8c9fee/barebones.yml) 52 | (single wait stage) to get you running. 53 | 54 | ## Create a template using the CLI ([roer](https://github.com/spinnaker/roer)) 55 | 56 | ```roer pipeline-template publish template.yml``` 57 | 58 | ## Create a pipeline in the UI 59 | 60 | The UI supports creating pipelines given it's enabled as described above. Just create a new pipeline and choose Create from: "Template". Choose your template and configure it using the UI. 61 | 62 | ![Create a pipeline in the UI](https://user-images.githubusercontent.com/1511533/28893520-68d1b110-77da-11e7-935d-b509464026d9.png) 63 | 64 | ## Create a pipeline using the CLI (roer) 65 | 66 | You can use a thin spinnaker CLI to publish both a template or a configuration to create a pipeline using [roer](https://github.com/spinnaker/roer). 67 | 68 | ```roer pipeline save pipeline-config.yml``` 69 | 70 | 71 | 72 | 73 | # questions 74 | 75 | If you have any questions on getting started, ask in Slack. 76 | -------------------------------------------------------------------------------- /MPTGraphic.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spinnaker/dcd-spec/b8f8953095308b977ef192c4eeea28c5216612cc/MPTGraphic.png -------------------------------------------------------------------------------- /PIPELINE_TEMPLATES.md: -------------------------------------------------------------------------------- 1 | # Pipeline Templates Spec 2 | 3 | **IMPORTANT: We are working on a simpler templating strategy, which we expect to release this year.** 4 | 5 | Pipeline Templates are a means for standardizing and distributing reusable 6 | Pipelines within a single application, different applications or even different 7 | Spinnaker deployments. They are the base abstraction of Pipelines and map 8 | very closely to the Pipeline JSON configuration format that the UI generates. 9 | 10 | For usage information, please see [ALPHA-GETTING-STARTED.md](ALPHA-GETTING-STARTED.md). 11 | 12 | * [pipeline templates examples](https://github.com/spinnaker/pipeline-templates) 13 | * [orca-pipelinetemplate implementation](https://github.com/spinnaker/orca/tree/master/orca-pipelinetemplate) 14 | * [dcd-converter]() 15 | 16 | # TOC 17 | 18 | * [Taxonomy](#taxonomy) 19 | * [Key Concepts](#key-concepts) 20 | * [Jinja Templating](#jinja-templating) 21 | * [Custom Tags & Filters](#custom-tags--filters) 22 | * [Template & Configuration Schemas](#template--configuration-schemas) 23 | * [Templates](#templates) 24 | * [Configurations](#configurations) 25 | * [Template Loaders](#template-loaders) 26 | * [Variables](#variables) 27 | * [Stages](#stages) 28 | * [Dependencies](#dependencies) 29 | * [Conditional Stages](#conditional-stages) 30 | * [Modules](#modules) 31 | * [Partials](#partials) 32 | * [Injection](#injection) 33 | * [Inheritance Control](#inheritance-control) 34 | * [FAQ](#faq) 35 | 36 | # Taxonomy 37 | 38 | * *Pipeline Template (template)*: An abstract, composable definition of a Spinnaker Pipeline 39 | * *Pipeline Configuration (configuration)*: A variable-binding and user-facing configuration 40 | format that combines with a Pipeline Template to make it runnable. 41 | * *Spinnaker Pipeline (pipeline)*: (what you see in the UI) a pipeline, represented internally by json, that can be run by orca. 42 | 43 | # Key Concepts 44 | 45 | * Pipeline Templates have two main components: Templates and Configurations. 46 | Templates can inherit and decorate parent templates. Configurations are 47 | concrete implementations of a Template. 48 | * Composability of Templates are done via Modules. Configurations can configure 49 | a module or entirely replace modules if they don't fit. 50 | * Configurations can inject new stages into the final pipeline graph and modify 51 | individual objects with JSONPath. 52 | * Templates can use Jinja expression syntax for better flow control. 53 | 54 | # Explained in Pictures 55 | ![mpt-graphic](MPTGraphic.png "Logo Title Text 1") 56 | 57 | # Template & Configuration Schemas 58 | 59 | Templates and Configuration schemas feel pretty similar, but do have some 60 | important differences. Templates, on their own, cannot be executed as pipelines. 61 | Configurations bring the additional variable bindings and customization that 62 | make Templates executable. 63 | 64 | You may notice that both Template and Configuration have a stanza named 65 | `configuration`: This represents the Pipeline Configuration view as seen in the 66 | UI. **NOTE:** This stanza, while defined in the spec, is not currently consumed 67 | by Spinnaker itself. 68 | 69 | This section primarily gives explanations of Templates & Configurations. For 70 | actual reference schemas, please take a look at [schemas/pipeline-templates-v1](schemas/pipeline-templates-v1). For examples, please take a look at [examples/pipeline-templates-v1](examples/pipeline-templates-v1). 71 | 72 | ## Templates 73 | 74 | ```yaml 75 | schema: "1" 76 | id: myTemplate 77 | source: https://example.com/myParentTemplate.yml 78 | protect: true 79 | metadata: 80 | name: Default Bake & Tag 81 | description: A generic application bake & tag pipeline. 82 | owner: example@example.com 83 | scopes: [global] 84 | variables: [] 85 | configuration: 86 | concurrentExecutions: {} 87 | triggers: [] 88 | parameters: [] 89 | notifications: [] 90 | description: "" 91 | stages: [] 92 | modules: [] 93 | partials: [] 94 | ``` 95 | 96 | * `schema`: A string value defining the version of the Template schema, which 97 | will increment in major versions only. 98 | * `id`: A globally unique identifier of the Template. This is used both by 99 | Templates and Configurations to reference a Template. 100 | * `source`: An optional field for defining the parent template to inherit from. 101 | If no value is assigned, the template is considered a root Template. 102 | * `protect`: A flag (defaults false) to control whether or not configurations 103 | can change or mutate the template's defined stage graph at plan/exec time. 104 | Side effects of template variables are not included in this protection. 105 | * `metadata`: A map of additional metadata used for rendering Templates in the 106 | UI. `name`, `description` and `scopes` are required. 107 | * `scopes`: A list of free-form strings used to group templates. For templates 108 | stored within Spinnaker, scopes have special meaning: 109 | * Users can query Spinnaker's API for templates that have a given scope or 110 | have a scope matching a given regex. 111 | * When configuring a pipeline template, Spinnaker's UI offers 112 | templates that have either `global` scope or `{applicationName}` scope. 113 | * `variables`: An explicit list of variables used by the template. Variables are 114 | flattened together when multiple templates are inherited. 115 | * `configuration`: A map of pipeline configurations. This is a 1-to-1 mapping of 116 | the pipeline configuration you'd see in the UI. 117 | * `stages`: A list of stages in the pipeline. 118 | * `modules`: A list of modules available to the pipeline. 119 | * `partials`: A list of reusable groups of stages that can be inserted into the 120 | pipeline. 121 | 122 | ## Configurations 123 | 124 | ```yaml 125 | schema: "1" 126 | pipeline: 127 | application: myApp 128 | name: My App Pipeline 129 | template: 130 | source: file://myTemplate.yml 131 | variables: {} 132 | configuration: 133 | inherit: [] 134 | triggers: [] 135 | parameters: [] 136 | notifications: [] 137 | description: "" 138 | stages: [] 139 | modules: [] 140 | partials: [] 141 | ``` 142 | 143 | * `schema`: A string value defining the version of the Configuration schema. 144 | This will likely be in lock-step with the Template schema version. 145 | * `pipeline`: Pipeline configuration, as well as template sourcing information. 146 | The variables field is a flat key/value map of concrete variable values that 147 | parent templates have defined. 148 | * `template`: A reference to the Template that the Configuration is applied 149 | against. See the Template Loaders section below for more information. 150 | * `configuration`: Pipeline configuration with a 1-1 mapping as you'd see in the 151 | Spinnaker UI. The `inherit` field is an explicit list of keys (e.g. `triggers`, 152 | `parameters`) that the configuration should inherit from parent templates. By 153 | default, configurations do not inherit any configurations. 154 | * `stages`: Any additional stages added to the pipeline graph. 155 | * `modules`: A list of modules available to the pipeline. 156 | * `partials`: A list of reusable groups of stages that can be inserted into the 157 | pipeline. 158 | 159 | # Template Loaders 160 | 161 | Templates can be loaded from different sources, or URI schemes: 162 | 163 | * `file:///path/to/my/template.yml` 164 | * `https://example.com/template.yml` 165 | * `spinnaker://template` 166 | 167 | The *file scheme* is offered mostly for running internal test harnesses, but it 168 | can be used in production as well. The path will resolve to whatever filesystem 169 | is associated to the server running `orca`. 170 | 171 | The *http and https schemes* is useful if you want to store your templates in 172 | a separate service, or potentially link to a Gist. It's handy for getting started, 173 | development and easy sharing. 174 | 175 | The *spinnaker scheme* references templates that have been saved into Spinnaker 176 | itself (depending on how you configure Front50: Redis, S3, GCS, etc). Generally 177 | speaking, this is the most resilient option. Templates are published into 178 | Spinnaker using the API. 179 | 180 | When using the *spinnaker scheme*, it's recommended to namespace your template 181 | IDs, as they are globally unique, regardless of the scope you provide them while 182 | publishing. 183 | 184 | # Jinja Templating 185 | 186 | Templates can use jinjava (Java implementation of Jinja) to offer greater 187 | productivity while creating and using templates. Jinja templating is only 188 | allowed in string values so that the JSON transport can always be valid. The 189 | results of Jinja templates can and often will result in non-string values (even 190 | object graphs). 191 | 192 | For more information about what's possible with Jinja, and specifically jinjava: 193 | 194 | * https://github.com/HubSpot/jinjava 195 | * http://jinja.pocoo.org/docs/2.9/ (Python implementation; good reference) 196 | 197 | Jinja is supported in the following areas: 198 | 199 | * Stage & module `when` stanzas 200 | * Module `definition` stanza (or any nested value inside) 201 | * Stage `config` stanza (or any nested value inside) 202 | * Stage `name` stanza 203 | * Template `metadata` values 204 | 205 | ## Custom Tags & Filters 206 | 207 | We're continually extending the Jinja implementation with new Tags and Filters. 208 | 209 | * `frigga` filter: Used to parse string values and return Frigga naming 210 | convention substrings. `{{ "orca-main"|frigga('stack') }} == "main" }}` 211 | * `json` filter: Output a value as a JSON object. `{{ myVar|json }}` 212 | * `base64` filter: Output a value as a Base64 encoded string. `{{ myVar|base64 }}` 213 | 214 | **IMPORTANT**: If your Jinja template is intended to return a list, map or 215 | object graph, you must ensure the output is *valid YAML or JSON*. 216 | 217 | 218 | 219 | # Variables 220 | 221 | Variables are used during the Jinja template rendering phase to build up stages. 222 | They have optional hinted types and can be used within the Template they are 223 | defined in, or in child Templates and Configurations. They required a `name`, 224 | `description` and optionally `type`, `defaultValue`, `group` and `example` 225 | fields. 226 | 227 | Dashes are not allowed in variable names. 228 | 229 | The `type` field accepts: 230 | 231 | * `string` 232 | * `int` 233 | * `float` 234 | * `boolean` 235 | * `list` 236 | * `object` (default) 237 | 238 | ```yaml 239 | variables: 240 | - name: regions 241 | description: A list of AWS regions to deploy into. Markdown supported. 242 | type: list 243 | defaultValue: 244 | - us-east-1 245 | - us-west-2 246 | group: Deployment 247 | example: | 248 | Free-form text. Typical usage would be for object variable types where the 249 | format could be in-obvious. 250 | ``` 251 | 252 | # Stages 253 | 254 | A Stage is directly analogous to a Pipeline Stage in the UI. It is defined by a 255 | minimum of `id`, `type` and `config`. 256 | 257 | Optional fields include `dependsOn`, `inject` and `when`, which are used 258 | for graph mutation (which we'll address later) 259 | 260 | ```yaml 261 | stages: 262 | - id: myBakeStage 263 | dependsOn: 264 | - myParentStage 265 | inject: SEE_INJECTED_DOCS_BELOW 266 | name: My fancy bake stage name 267 | type: bake 268 | config: 269 | package: foo 270 | executionOptions: 271 | onStageFailure: haltEntirePipeline 272 | # ... 273 | notifications: [] 274 | comments: "" 275 | when: 276 | - "{{ appSupportsBake == 'myAppName' }} 277 | ``` 278 | 279 | The `config` map is a 1-for-1 mapping of the stage type configuration that you 280 | would see looking at a stage in a Pipeline's JSON configuration. 281 | 282 | ## Dependencies 283 | 284 | To create a Pipeline consisting of a series of Stages, the stage definition has 285 | the concept of `dependsOn`, which takes a list of stage IDs: These are direct 286 | parents of the stage. For more advanced control, the `inject` stanza is offered, 287 | which will be covered later. In most cases, `dependsOn` is all you'll need to 288 | perform standard branch fork and join operations. 289 | 290 | ## Conditional Stages 291 | 292 | Configuring a Pipeline via the UI, you can conditionally execute stages based on 293 | runtime information. With Pipeline Templates, you can conditionally include or 294 | exclude entire branches of stages before the Pipeline is executed. 295 | 296 | The `when` stanza takes a list of Jinja expressions that are evaluated together 297 | as `AND` statements. These expressions must evaluate to a boolean value. 298 | 299 | If a Stage is conditionally excluded, the stage graph will automatically be 300 | recalculated. 301 | 302 | ```yaml 303 | stages: 304 | - id: one 305 | type: wait 306 | config: {} 307 | when: 308 | - "{{ true }}" 309 | - id: two 310 | dependsOn: 311 | - one 312 | type: wait 313 | config: {} 314 | when: 315 | - "{{ false }}" 316 | - id: three 317 | dependsOn: 318 | - two 319 | type: wait 320 | config: {} 321 | 322 | ## rendered to... 323 | 324 | stages: 325 | - id: one 326 | type: wait 327 | config: {} 328 | when: 329 | - "{{ true }}" 330 | - id: three 331 | dependsOn: 332 | - one 333 | type: wait 334 | config: {} 335 | ``` 336 | 337 | Conditional stages are supported within Partials as well. 338 | 339 | # Modules 340 | 341 | Modules are used to represent common functionality within stages, like stage configuration. 342 | 343 | Modules can be referenced by each other, the Template they are defined in, 344 | child Templates and Configurations. Furthermore, they can be replaced by child 345 | Templates and Configurations as well. 346 | 347 | Modules, combined with Jinja templating, can be powerful for looping over 348 | common template blocks, as well as swapping out cloud provider functionality or 349 | conditionally including certain configuration options defined from a common 350 | parent Template. 351 | 352 | At minimum, a module must have an `id`, `usage` and `definition`. Optionally, 353 | `variables` can be defined. Note, while modules can directly reference variables 354 | defined by the Template, any variables defined by the module will be strictly 355 | scoped to the Module only. 356 | 357 | ```yaml 358 | # Template snippet 359 | id: myExampleTemplate 360 | metadata: 361 | name: myExampleTemplate 362 | description: Modules example template 363 | owner: example@example.com 364 | scopes: [global] 365 | variables: 366 | - name: regions 367 | description: A list of AWS regions to deploy into 368 | type: list 369 | stages: 370 | - id: deploy 371 | type: deploy 372 | config: 373 | clusters: | 374 | {% for region in regions %} 375 | - {% module deployClusterAws region=region %} 376 | {% endfor %} 377 | 378 | modules: 379 | - id: deployClusterAws 380 | usage: Defines a deploy stage cluster using the AWS cloud provider 381 | variables: 382 | - name: region 383 | description: The AWS region to deploy into 384 | definition: 385 | provider: aws 386 | account: myAccount 387 | region: "{{ region }}" 388 | ``` 389 | 390 | Modules may be used anywhere Jinja expressions are supported, and can output as 391 | little or as much data as necessary. Combined with configuration-level module 392 | overriding, this offers a considerable amount of options for extensibility. 393 | 394 | # Partials 395 | 396 | **DRAFT: This feature is either not yet implemented, or currently under test.** 397 | 398 | For cases where you need to reuse entire parts of a Template's stage graph, such 399 | as a group of stages, Partials exist. This behaves similarly to Modules, but 400 | rather than templating a the configuration of a single stage, you can template 401 | groups of stages together. 402 | 403 | The goal of Partials is primarily to replace using Child Pipelines, which teams 404 | often use to reuse common logic. The issue in using Child Pipelines is restart 405 | functionality can be at the incorrect granularity and is often slow. More 406 | importantly, however, is correlation of a Child Pipeline's execution is difficult 407 | to visualize within the Pipelines UI. 408 | 409 | Partials are a root-level element in the Template, and like modules, can be 410 | accessed or replaced by children Templates. 411 | 412 | A Partial has the required fields `id`, `usage` and `stages`, with an optional 413 | field of `variables`. Just like Modules, a Partial will inherit all variables 414 | of the Template, but variables defined within a Partial will be locally-scoped. 415 | 416 | Once defined, a Partial is referenced within a Template's Stage list similar 417 | to how a normal Stage would be, except with a special type `partial`. The type 418 | value takes a format of `partial.PARTIAL_ID`, so for example, if a partial exists 419 | with the ID `myPartial`, the `type` value would be `partial.myPartial`. 420 | 421 | The `config` value would be setting the variable bindings of the Partial, and 422 | like any Stage config stanza, supports full Jinja expressions. 423 | 424 | An example, where a Template needs to support building and publishing an artifact 425 | targeted at different web browsers using Jenkins jobs. 426 | 427 | ```yaml 428 | schema: '1' 429 | id: partialsExampleTemplate 430 | stages: 431 | - id: firstWait 432 | type: wait 433 | config: 434 | waitTime: 5 435 | - id: buildChrome 436 | type: partial.buildBrowser 437 | dependsOn: 438 | - firstWait 439 | config: 440 | target: chrome 441 | - id: finalWait 442 | type: wait 443 | dependsOn: 444 | - buildChrome 445 | config: 446 | waitTime: 5 447 | 448 | partials: 449 | - id: buildBrowser 450 | usage: Builds the pipeline artifact targeting the a specified browser. 451 | variables: 452 | - name: target 453 | description: The target browser to build for 454 | stages: 455 | - id: buildTarget 456 | type: jenkins 457 | name: Build {{ target }} 458 | config: 459 | # etc... 460 | - id: publishTarget 461 | type: jenkins 462 | name: Publish {{ target }} 463 | dependsOn: 464 | - buildTarget 465 | config: 466 | # etc ... 467 | ``` 468 | 469 | The resultant Pipeline would look sort of like so: 470 | 471 | ```yaml 472 | schema: '1' 473 | id: partialsExampleTemplate 474 | stages: 475 | - id: firstWait 476 | type: wait 477 | config: 478 | waitTime: 5 479 | - id: buildChrome.buildTarget 480 | type: jenkins 481 | dependsOn: 482 | - wait 483 | config: 484 | # blah 485 | - id: buildChrome.publishTarget 486 | type: jenkins 487 | dependsOn: 488 | - buildChrome.buildTarget 489 | config: 490 | # blah 491 | - id: finalWait 492 | type: wait 493 | dependsOn: 494 | - buildChrome.publishTarget 495 | config: 496 | waitTime: 5 497 | ``` 498 | 499 | A couple key things to note here: 500 | 501 | 1. The resultant stage graph namespaces stage IDs generated by the Partial 502 | by `{{ partialId }}.{{ internalPartialStageId }}`. This ensures that the 503 | stage names can be uniquely referenced by injections and so-on. 504 | 2. The resultant stage graph will correctly resolve child stage `dependsOn` 505 | dependencies. Note that the stage `finalWait` stage only depends on 506 | `buildChrome.publishTarget`, rather than every stage defined by the Partial. 507 | 508 | # Injection 509 | 510 | A child Template or Configuration can make mutations to the Pipeline stage 511 | graph defined by parent Templates. Injecting a Stage will cause the stage graph 512 | to be automatically recalculated. 513 | 514 | You should consider injection an advanced option where standard `dependsOn` is 515 | not sufficient. 516 | 517 | ``` 518 | # "inject after target" behavior 519 | Target --> 1..* Children 520 | Target --> Injected --> 1..* Children 521 | ``` 522 | 523 | ```yaml 524 | # Config: Single-stage injection 525 | pipeline: 526 | template: 527 | # This template defines a pipeline "bake" -> "deploy" (these are ids, 528 | # as well as the stage type). 529 | source: spinnaker://myPipelineTemplate 530 | stages: 531 | # We want to add a manualJudgement stage to propagate authentication 532 | - id: manualJudgment 533 | type: manualJudgment 534 | inject: 535 | before: 536 | - deploy 537 | config: 538 | propagateAuthentication: true 539 | notifications: 540 | - type: slack 541 | address: spinnaker 542 | message: 543 | manualJudgmentContinue: 544 | text: "judgment delivered" 545 | when: 546 | - manualJudgment 547 | ``` 548 | 549 | In the above example, `manualJudgment` will be injected into the graph before 550 | the `deploy` stage. 551 | 552 | The available hooks for injection are: 553 | 554 | * `before`: List of stage IDs. 555 | * `after`: List of stage IDs. 556 | * `first`: Boolean. 557 | * `last`: Boolean. 558 | 559 | # Inheritance Control 560 | 561 | **This feature is not fully supported and is not recommended.** 562 | 563 | In some rare cases, you want to inherit a Stage, but need to make limited, 564 | un-templated changes to it. Stages support the inclusion of an 565 | `inheritanceControl` stanza which allows for more powerful expressions in 566 | modifying nested list elements or maps. Inheritance control has three different 567 | control methods, all of which require a `path` selector. The path selector 568 | uses JSONPath. 569 | 570 | * `merge`: Merge maps together or append to lists. 571 | * `replace`: Replace an object with a new object at a path. 572 | * `remove`: Removes an object from the path. 573 | 574 | In the following example, the Template defines a deploy stage that assumes a 575 | collection of "paved road" ports on a load balancer. The application you're 576 | building pipelines for fits this template perfectly, but you just need to modify 577 | the listeners. 578 | 579 | This is a very advanced feature. If you find yourself having to use this 580 | pattern often, you should strongly consider if you can approach the templating 581 | problems you have differently. 582 | 583 | ```yaml 584 | # Template 585 | id: myTemplate 586 | stages: 587 | - id: deploy 588 | type: deploy 589 | config: 590 | clusters: 591 | - provider: aws 592 | loadBalancers: 593 | - instancePort: 80 594 | instanceProtocol: "http" 595 | lbPort: 80 596 | lbProtocol: "http" 597 | - instancePort: 8443 598 | instanceProtocol: "https" 599 | lbPort: 8443 600 | lbProtocol: "https" 601 | ``` 602 | 603 | ```yaml 604 | # Configuration 605 | stages: 606 | - id: deploy 607 | type: deploy 608 | inheritanceControl: 609 | merge: 610 | - path: $.clusters[?(@.provider==aws)].loadBalancers 611 | value: 612 | instancePort: 9000 613 | instanceProtocol: http 614 | lbPort: 9000 615 | lbProtocol: http 616 | replace: 617 | - path: $.clusters[?(@.provider==aws)].loadBalancers[?(@.instancePort==80)] 618 | value: 619 | instancePort: 8080 620 | instanceProtocol: http 621 | lbPort: 80 622 | lbProtocol: http 623 | remove: 624 | - path: $.clusters[?(@.provider==aws)].loadBalancers[?(@.instancePort==8443)] 625 | ``` 626 | 627 | The result would become: 628 | 629 | ```yaml 630 | # Template 631 | id: myTemplate 632 | stages: 633 | - id: deploy 634 | type: deploy 635 | config: 636 | clusters: 637 | - provider: aws 638 | loadBalancers: 639 | - instancePort: 8080 640 | instanceProtocol: http 641 | lbPort: 80 642 | lbProtocol: http 643 | - instancePort: 9000 644 | instanceProtocol: http 645 | lbPort: 9000 646 | lbProtocol: http 647 | ``` 648 | 649 | # FAQ 650 | 651 | ## Q. Why YAML? 652 | 653 | We feel that YAML is much easier to write and grok for humans, while being easy 654 | to convert to JSON, Spinnaker's internal pipeline storage format. Since these 655 | files are intended to be version-controlled alongside your code, we felt it 656 | necessary to choose a format that was more human-first leaning. 657 | 658 | ## Q. What are the differences between Template Variables and Pipline Parameters? 659 | 660 | One potential confusing part about Pipeline Templates are the differences 661 | between Variables and Parameters. Variables are a concept for Pipeline Templates 662 | only and are not available at pipeline execution runtime, as they're only used 663 | during Jinja templating. 664 | 665 | Variables are can be used to help build stages and modify the stage graph prior 666 | to execution. 667 | 668 | Parameters are variables that are made available for execution runtime. These 669 | are the values that are available via the Configuration UI, and are presented 670 | when manually executing a Pipeline. 671 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # dcd-spec 2 | 3 | A living document of the Spinnaker Pipeline Templates and greater Declarative 4 | Continuous Delivery config language, a way to store your app, pipelines and 5 | infrastructure as code within Spinnaker. 6 | 7 | This repository is organized by major DCD features. 8 | 9 | * [Pipeline Templates Spec](PIPELINE_TEMPLATES.md) 10 | 11 | Other resources: 12 | 13 | * [pipeline-templates](https://github.com/spinnaker/pipeline-templates) - 14 | Public repo of Pipeline Templates for examples and sharing. 15 | * [spin-dcd-converter](https://github.com/robzienert/spin-dcd-converter) - 16 | Tool to bootstrap converting existing Pipelines to Pipeline Template YAML. 17 | * [pipeline2dcd](https://github.com/sergi/pipeline2dcd) - 18 | Web browser extension to convert JSON Spinnaker pipelines to DCD pipelines. 19 | -------------------------------------------------------------------------------- /examples/README.md: -------------------------------------------------------------------------------- 1 | # dcd-examples 2 | 3 | Examples of how to use MPT, organized by schema version. 4 | 5 | Templates are linked using the `file://myfile.yml` format. 6 | To use templates in a hosted manner, change all the `source:` fields to represent the URL 7 | of the raw file or the `spinnaker://` format. 8 | See [my gists](https://gist.github.com/emjburns/b769578860ecbb84d64ea72b544e205c) for an example. 9 | -------------------------------------------------------------------------------- /examples/pipeline-templates-v1/bake-deploy/README.md: -------------------------------------------------------------------------------- 1 | # This example shows a bake and deploy pipeline with the option to override to a find image and deploy pipeline 2 | 3 | ## Option 1: Bake 4 | `template.yml` shows a sample multi region bake and deploy. `bake-deploy-config.yml` shows the config for that pipeline, providing a stage name for the first stage. 5 | 6 | 7 | ## Option 2: Find Image 8 | `template.yml` remains the same. `findImage-deploy-config.yml` shows replacing the bake stage with a findImage stage. 9 | -------------------------------------------------------------------------------- /examples/pipeline-templates-v1/bake-deploy/bake-deploy-config.yml: -------------------------------------------------------------------------------- 1 | --- 2 | schema: "1" 3 | pipeline: 4 | application: orca 5 | name: Bake and Deploy 6 | template: 7 | source: file://template.yml 8 | variables: 9 | myCustomFirstStageName: "Bake" 10 | stages: [] -------------------------------------------------------------------------------- /examples/pipeline-templates-v1/bake-deploy/findImage-deploy-config.yml: -------------------------------------------------------------------------------- 1 | --- 2 | schema: "1" 3 | pipeline: 4 | application: orca 5 | name: Find Image and Deploy 6 | template: 7 | source: file://template.yml 8 | variables: 9 | myCustomFirstStageName: "Find Image" 10 | stages: 11 | - id: bake1 12 | type: findImageFromTags 13 | name: "{{ myCustomFirstStageName }}" 14 | config: 15 | cloudProvider: aws 16 | cloudProviderType: aws 17 | packageName: "{{ application }}" 18 | regions: 19 | - us-east-1 20 | - us-west-1 21 | - us-west-2 22 | - eu-west-1 23 | tags: 24 | stack: test 25 | -------------------------------------------------------------------------------- /examples/pipeline-templates-v1/bake-deploy/template.yml: -------------------------------------------------------------------------------- 1 | --- 2 | schema: "1" 3 | id: bakeDeploy 4 | metadata: 5 | name: Bake and Deploy 6 | description: Example bake and deploy template 7 | variables: 8 | - name: myCustomFirstStageName 9 | stages: 10 | - id: bake1 11 | type: bake 12 | name: "{{ myCustomFirstStageName }}" 13 | config: 14 | baseLabel: release 15 | baseOs: trusty 16 | cloudProviderType: aws 17 | enhancedNetworking: false 18 | extendedAttributes: {} 19 | overrideTimeout: true 20 | package: orca 21 | regions: 22 | - us-east-1 23 | - us-west-1 24 | - us-west-2 25 | - eu-west-1 26 | sendNotifications: true 27 | showAdvancedOptions: true 28 | stageTimeoutMs: 900000 29 | storeType: ebs 30 | user: example@example.com 31 | vmType: hvm 32 | - id: deploy2 33 | type: deploy 34 | dependsOn: 35 | - bake1 36 | name: Deploy 37 | config: 38 | clusters: 39 | - account: test 40 | application: orca 41 | availabilityZones: 42 | us-west-1: 43 | - us-west-1a 44 | - us-west-1c 45 | capacity: 46 | desired: 1 47 | max: 1 48 | min: 1 49 | cloudProvider: aws 50 | cooldown: 10 51 | copySourceCustomBlockDeviceMappings: true 52 | ebsOptimized: false 53 | enabledMetrics: [] 54 | freeFormDetails: demo 55 | healthCheckGracePeriod: 600 56 | healthCheckType: EC2 57 | iamRole: myIAMRole 58 | instanceMonitoring: false 59 | instanceType: m3.large 60 | interestingHealthProviderNames: 61 | - Amazon 62 | keyPair: keypair 63 | loadBalancers: 64 | - orca-demo-frontend 65 | maxRemainingAsgs: 2 66 | preferSourceCapacity: true 67 | provider: aws 68 | scaleDown: true 69 | securityGroups: [] 70 | stack: test 71 | strategy: redblack 72 | subnetType: mySubnet 73 | suspendedProcesses: [] 74 | tags: {} 75 | targetGroups: [] 76 | targetHealthyDeployPercentage: 100 77 | terminationPolicies: 78 | - Default 79 | useAmiBlockDeviceMappings: false 80 | useSourceCapacity: true 81 | - account: test 82 | application: orca 83 | availabilityZones: 84 | us-east-1: 85 | - us-east-1c 86 | - us-east-1d 87 | - us-east-1e 88 | capacity: 89 | desired: 0 90 | max: 0 91 | min: 0 92 | cloudProvider: aws 93 | cooldown: 10 94 | ebsOptimized: false 95 | freeFormDetails: demo 96 | healthCheckGracePeriod: 600 97 | healthCheckType: EC2 98 | iamRole: myIAMRole 99 | instanceMonitoring: false 100 | instanceType: m3.large 101 | interestingHealthProviderNames: 102 | - Amazon 103 | keyPair: keypair 104 | provider: aws 105 | securityGroups: [] 106 | stack: test 107 | strategy: highlander 108 | subnetType: mySubnet 109 | suspendedProcesses: [] 110 | tags: {} 111 | targetHealthyDeployPercentage: 100 112 | terminationPolicies: 113 | - Default 114 | useSourceCapacity: false 115 | -------------------------------------------------------------------------------- /examples/pipeline-templates-v1/complex-wait/README.md: -------------------------------------------------------------------------------- 1 | # This example shows a complex wait pipeline. 2 | The UI view for this pipeline is shown in [complex-wait-ui.png](complex-wait-ui.png). Same UI for all three options. 3 | 4 | ## Option 1: Inheritance 5 | `root-template.yml`, `child-template.yml`, and `child-2-template.yml` show an inheritence structure and the use variable. `mypipeline-config.yml` adds another stage at the configuration level. 6 | 7 | 8 | ## Option 2: One template 9 | `combined-template.yml` shows the same stages represented in one template. `mypipelineCombined-config.yml` shows the config for that template that produces the same pipeline. 10 | 11 | ## Option 3: All Config 12 | `only-config.yml` shows how you can create the same pipeline with one config file. You might want to do this if you are only using this logic for one pipeline, or you are working to transition to managed pipeline templates and want to first convert each pipeline to code. -------------------------------------------------------------------------------- /examples/pipeline-templates-v1/complex-wait/child-2-template.yml: -------------------------------------------------------------------------------- 1 | schema: "1" 2 | id: child-2-template 3 | source: file://child-template.yml 4 | variables: 5 | - name: childWaitTime 6 | description: pause time for another wait 7 | metadata: 8 | name: A Second Child template 9 | description: A second child template 10 | stages: 11 | - id: waitChild2 12 | type: wait 13 | dependsOn: 14 | - wait1 15 | config: 16 | waitTime: "{{ childWaitTime }}" -------------------------------------------------------------------------------- /examples/pipeline-templates-v1/complex-wait/child-template.yml: -------------------------------------------------------------------------------- 1 | schema: "1" 2 | id: child-template 3 | source: file://root-template.yml # Indicates that this template inherits from the root-template 4 | metadata: 5 | name: Child template 6 | description: A child template 7 | stages: 8 | - id: waitChild1 9 | type: wait 10 | dependsOn: 11 | - wait1 # Depending on a stage from the root-template 12 | config: 13 | waitTime: "{{ waitTime }}" # Using a variable from the root-template -------------------------------------------------------------------------------- /examples/pipeline-templates-v1/complex-wait/combined-template.yml: -------------------------------------------------------------------------------- 1 | # "Combined" template, exact same as child-2-template.yml + 2 | # the stage defined in the config 3 | 4 | schema: "1" 5 | id: combined-template 6 | metadata: 7 | name: Less simple wait template 8 | description: A template with many waits 9 | variables: 10 | - name: waitTime 11 | description: The time a wait stage should pause 12 | type: int 13 | - name: childWaitTime 14 | description: pause time for another wait 15 | stages: 16 | - id: wait1 17 | type: wait 18 | config: 19 | waitTime: "{{ waitTime }}" 20 | - id: waitChild1 21 | type: wait 22 | dependsOn: 23 | - wait1 24 | config: 25 | waitTime: "{{ waitTime }}" 26 | - id: waitChild2 27 | type: wait 28 | dependsOn: 29 | - wait1 30 | config: 31 | waitTime: "{{ childWaitTime }}" 32 | - id: finalWait 33 | type: wait 34 | dependsOn: 35 | - waitChild1 36 | - waitChild2 37 | config: 38 | waitTime: 10 39 | -------------------------------------------------------------------------------- /examples/pipeline-templates-v1/complex-wait/complex-wait-ui.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spinnaker/dcd-spec/b8f8953095308b977ef192c4eeea28c5216612cc/examples/pipeline-templates-v1/complex-wait/complex-wait-ui.png -------------------------------------------------------------------------------- /examples/pipeline-templates-v1/complex-wait/mypipeline-config.yml: -------------------------------------------------------------------------------- 1 | schema: "1" 2 | pipeline: # Combines with the template to make a runnable spinnaker pipeline 3 | application: myApp 4 | name: My super awesome pipeline 5 | template: 6 | source: file://child-2-template.yml # What template this config applies to 7 | variables: # The actual values for all vars defined in all templates in the chain 8 | waitTime: 20 9 | childWaitTime: 15 10 | configuration: # Configuration of the pipeline 11 | notifications: 12 | - address: example@example.com 13 | level: pipeline 14 | name: email0 15 | type: email 16 | when: 17 | - pipeline.failed 18 | triggers: [] 19 | stages: 20 | - id: finalwait 21 | type: wait 22 | dependsOn: 23 | - waitChild1 24 | - waitChild2 25 | config: 26 | waitTime: 10 -------------------------------------------------------------------------------- /examples/pipeline-templates-v1/complex-wait/mypipelineCombined-config.yml: -------------------------------------------------------------------------------- 1 | # Exactly the same configuration as mypipeline-config.yml 2 | # EXCEPT source points to combined template, and all stages 3 | # are defined in the template 4 | 5 | schema: "1" 6 | pipeline: 7 | application: myApp 8 | name: My super awesome pipeline 9 | template: 10 | source: file://combined-template.yml 11 | variables: 12 | waitTime: 20 13 | childWaitTime: 15 14 | configuration: 15 | notifications: 16 | - address: example@example.com 17 | level: pipeline 18 | name: email0 19 | type: email 20 | when: 21 | - pipeline.failed 22 | triggers: [] 23 | 24 | -------------------------------------------------------------------------------- /examples/pipeline-templates-v1/complex-wait/only-config.yml: -------------------------------------------------------------------------------- 1 | schema: "1" 2 | pipeline: 3 | application: myApp 4 | name: My super awesome pipeline 5 | variables: 6 | waitTime: 20 7 | childWaitTime: 15 8 | configuration: 9 | notifications: 10 | - address: example@example.com 11 | level: pipeline 12 | name: email0 13 | type: email 14 | when: 15 | - pipeline.failed 16 | triggers: [] 17 | stages: 18 | - id: wait1 19 | type: wait 20 | config: 21 | waitTime: "{{ waitTime }}" 22 | - id: waitChild1 23 | type: wait 24 | dependsOn: 25 | - wait1 26 | config: 27 | waitTime: "{{ waitTime }}" 28 | - id: waitChild2 29 | type: wait 30 | dependsOn: 31 | - wait1 32 | config: 33 | waitTime: "{{ childWaitTime }}" 34 | - id: finalWait 35 | type: wait 36 | dependsOn: 37 | - waitChild1 38 | - waitChild2 39 | config: 40 | waitTime: 10 -------------------------------------------------------------------------------- /examples/pipeline-templates-v1/complex-wait/root-template.yml: -------------------------------------------------------------------------------- 1 | schema: "1" # Schema must match for all 2 | id: root-template 3 | metadata: 4 | name: Simple wait template 5 | description: Extendable root template 6 | variables: # Variables available to all that inherit 7 | - name: waitTime 8 | description: The time a wait stage should pause 9 | type: int 10 | stages: # Stages available to all that inherit 11 | - id: wait1 12 | type: wait 13 | config: 14 | waitTime: "{{ waitTime }}" # Variables can be used anywhere -------------------------------------------------------------------------------- /examples/pipeline-templates-v1/configuration-inheritance/README.md: -------------------------------------------------------------------------------- 1 | # Inheriting Configuration from a Template 2 | 3 | This example shows how to include configuration from your Template in your Pipeline 4 | 5 | This enables you to define parameters, and other `configuration` items in your template and allow configs to use these blocks without redefining them 6 | 7 | - `template.yml` shows a configuration block defined with a trigger, some parameters and `concurrentExecutions` flags defined 8 | - `configuration-inheritance-config.yml` shows how you can include your Templates `configuration` using the `inherit` option 9 | -------------------------------------------------------------------------------- /examples/pipeline-templates-v1/configuration-inheritance/configuration-inheritance-config.yml: -------------------------------------------------------------------------------- 1 | --- 2 | schema: "1" 3 | pipeline: 4 | application: orca 5 | name: Simple Configuraiton Inheritance Example 6 | template: 7 | source: file://template.yml 8 | variables: 9 | jenkinsTriggerJob: 'JobName' 10 | jenkinsMaster: 'JenkinsMasterName' 11 | jenkinsPropertyFile: 'properties.json' 12 | configuration: 13 | inherit: ['concurrentExecutions', 'triggers', 'parameters'] 14 | -------------------------------------------------------------------------------- /examples/pipeline-templates-v1/configuration-inheritance/template.yml: -------------------------------------------------------------------------------- 1 | --- 2 | schema: "1" 3 | id: configurationInheritance 4 | metadata: 5 | name: Configuration Inheritance Example 6 | description: A simple inheritance example of how to inherit parameters from a template 7 | scopes: 8 | - global 9 | 10 | configuration: 11 | triggers: 12 | - type: jenkins 13 | job: "{{ jenkinsTriggerJob }}" 14 | master: "{{ jenkinsMaster }}" 15 | propertyFile: "{{ jenkinsPropertyFile }}" 16 | enabled: true 17 | parameters: 18 | - name: baseOs 19 | description: The baseOs to bake the image with 20 | concurrentExecutions: 21 | limitConcurrent: true 22 | 23 | stages: 24 | - id: bake 25 | type: bake 26 | name: Bake 27 | config: 28 | baseOs: ${trigger.parameters["baseOs"]} 29 | baseLabel: release 30 | cloudProviderType: aws 31 | enhancedNetworking: false 32 | extendedAttributes: {} 33 | overrideTimeout: true 34 | package: orca 35 | regions: 36 | - us-east-1 37 | sendNotifications: true 38 | showAdvancedOptions: true 39 | stageTimeoutMs: 900000 40 | storeType: ebs 41 | user: example@example.com 42 | vmType: hvm 43 | -------------------------------------------------------------------------------- /schemas/pipeline-templates-v1/configuration-schema.yml: -------------------------------------------------------------------------------- 1 | # TODO 2 | -------------------------------------------------------------------------------- /schemas/pipeline-templates-v1/template-schema.yml: -------------------------------------------------------------------------------- 1 | # TODO 2 | --------------------------------------------------------------------------------