├── docs ├── _config.yml ├── setup-workflows.png ├── conditional-workflows.md ├── transitioning.md ├── parameter-scopes.md ├── pipeline-parameters.md ├── README.md ├── path-filtering.md ├── scheduled-pipelines.md ├── setup-workflows.md ├── breaking.md └── api-changes.md ├── README.md ├── .circleci ├── config.yml └── runthrough.sh └── designs-for-feedback └── trigger-workflows-in-pipeline.md /docs/_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-minimal -------------------------------------------------------------------------------- /docs/setup-workflows.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CircleCI-Archived/api-preview-docs/HEAD/docs/setup-workflows.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # CircleCI API Preview 2 | 3 | This repository has information about the preview release of CircleCI API v2. 4 | 5 | ## DEVELOPING WITH CIRCLECI API v2 6 | 7 | * See the [documentation](docs/) in this repository for reference information. 8 | * Breaking changes to the API will be announced in this document [breaking](docs/breaking.md). 9 | * We keep [an up-to-date OpenAPI document](https://circleci.com/api/v2/openapi.json) which describes the v2 API 10 | 11 | ## FEEDBACK 12 | We welcome your feedback on the v2 API. A few ways to reach us include: 13 | * Feel free to use Issues on this repository for feature requests or bug reports. 14 | 15 | * Talk to your account team at [CircleCI](https://circleci.com/), or ask them to set up time to speak with us about your needs. 16 | * Tweet [@circleci](https://twitter.com/CircleCI) and mention API v2 - we tend to see those within a day or two. 17 | 18 | ## NEXT STEPS 19 | * Read the [docs](docs/) in this repository for details. 20 | -------------------------------------------------------------------------------- /.circleci/config.yml: -------------------------------------------------------------------------------- 1 | parameters: 2 | spec-root: 3 | type: string 4 | default: "./specs/" 5 | 6 | 7 | version: 2.1 8 | 9 | orbs: 10 | tools: circleci/node@volatile 11 | spectacle: 12 | executors: 13 | default: 14 | docker: 15 | - image: sourcey/spectacle 16 | commands: 17 | build-docs: 18 | parameters: 19 | file: 20 | type: string 21 | description: "local path to the openAPI json" 22 | save-artifact: 23 | type: boolean 24 | default: true 25 | publish-dir: 26 | type: string 27 | default: "~/spectacle/" 28 | steps: 29 | - run: spectacle -t <> << parameters.file >> 30 | - when: 31 | condition: << parameters.save-artifact >> 32 | steps: 33 | store_artifacts: 34 | path: << parameters.publish-dir >> 35 | jobs: 36 | generate-docs: 37 | executor: spectacle/default 38 | steps: 39 | - checkout 40 | - run: curl -o circleci-openapi.json https://circleci.com/api/v2/openapi.json 41 | - spectacle/build-docs: 42 | file: circleci-openapi.json 43 | workflows: 44 | main: 45 | jobs: 46 | - generate-docs 47 | 48 | -------------------------------------------------------------------------------- /docs/conditional-workflows.md: -------------------------------------------------------------------------------- 1 | # Conditional Workflows 2 | New as of June 2019, you can use a `when` clause (we also support the inverse clause `unless`) under a workflow declaration with a boolean value to decide whether or not to run that workflow. 3 | 4 | The most common use of this construct is to use a [pipeline parameter](pipeline-parameters.md) as the value, allowing an API trigger to pass that parameter to determine which workflows to run. 5 | 6 | Below is an example configuration using two different pipeline parameters, one used to drive whether a particular workflow will run and another to determine if a particular step will run. 7 | 8 | ```yaml 9 | version: 2.1 10 | 11 | parameters: 12 | run_integration_tests: 13 | type: boolean 14 | default: false 15 | deploy: 16 | type: boolean 17 | default: false 18 | 19 | workflows: 20 | version: 2 21 | integration_tests: 22 | when: << pipeline.parameters.run_integration_tests >> 23 | jobs: 24 | - mytestjob 25 | 26 | jobs: 27 | mytestjob: 28 | steps: 29 | - checkout 30 | - when: 31 | condition: << pipeline.parameters.deploy >> 32 | steps: 33 | - run: echo "deploying" 34 | ``` 35 | 36 | The above would prevent the workflow `integration_tests` from being triggered 37 | unless it was invoked explicitly when the pipeline is triggered with the following in the POST body: 38 | 39 | ```json 40 | { 41 | "parameters": { 42 | "run_integration_tests": true 43 | } 44 | } 45 | ``` 46 | 47 | The `when` key actually accepts any boolean, not just pipeline parameters, 48 | though pipeline parameters will be the primary use of this feature until we implement others. `when` also has an inverse clause called `unless`, which inverts truthiness of the condition. 49 | -------------------------------------------------------------------------------- /docs/transitioning.md: -------------------------------------------------------------------------------- 1 | # Transitioning to Pipelines 2 | 3 | This document outlines transitioning to pipelines. 4 | 5 | ## Pipelines with 2.0 Configuration 6 | 7 | When using 2.0 configuration in combination with pipelines, CircleCI 8 | will inject the `CIRCLE_COMPARE_URL` environment variable into all 9 | jobs for backwards compatibility. This environment variable is 10 | generated in a different way than the one that is available in legacy 11 | jobs, and is not always available. 12 | 13 | It is not injected when there is no meaningful previous revision, 14 | which includes the first push of commits to an empty repository, and 15 | when a new branch is created/pushed without any additional commits. 16 | 17 | ## Opting Into Pipelines on a Branch 18 | 19 | There are two main ways of trying out pipelines on a branch without 20 | committing by changing the project-wide setting. One of them is by 21 | using version 2.1 configuration, the other is by including the 22 | `experimental` stanza. 23 | 24 | Note that these methods currently apply to webhooks as well as the 25 | version 2 "pipeline trigger" API, but not the version 1.1 "job 26 | trigger" API yet. We will announce support for using the version 1.1 27 | API with pipelines soon. 28 | 29 | ### Using Version 2.1 Configuration 30 | 31 | Configuration version 2.1 has always depended on pipelines. We are now 32 | enabling pipelines for build requests with version 2.1 configuration 33 | automatically, so trying out pipelines on a branch this way only 34 | requires upgrading the configuration version. 35 | 36 | This also allows use of the 2.1-exclusive features, like pipeline 37 | values. 38 | 39 | ### Using the `experimental` Stanza 40 | 41 | Alternatively there is a configuration stanza which enables pipelines 42 | using a version 2.0 configuration: 43 | 44 | ```yaml 45 | experimental: 46 | pipelines: true 47 | ``` 48 | -------------------------------------------------------------------------------- /docs/parameter-scopes.md: -------------------------------------------------------------------------------- 1 | # Config Processing Stages and Parameter Scopes 2 | 3 | ## Processing Stages 4 | 5 | Configuration processing happens in the following stages: 6 | 7 | - Pipeline parameters are resolved and type-checked 8 | - Pipeline parameters are replaced in the orb statement 9 | - Orbs are imported 10 | - The remaining configuration is processed, element parameters are 11 | resolved, type-checked, and substituted 12 | 13 | ## Element Parameter Scope 14 | 15 | Element parameters use lexical scoping, so parameters are in scope 16 | within the element they are defined in, e.g. a job, a command, or an 17 | executor. If a element with parameters calls another element with 18 | parameters, like in the example below, the inner element does not 19 | inherit the scope of the calling element. 20 | 21 | ```yaml 22 | version: 2.1 23 | 24 | commands: 25 | print: 26 | parameters: 27 | message: 28 | type: string 29 | steps: 30 | - run: echo << parameters.message >> 31 | 32 | jobs: 33 | cat-file: 34 | parameters: 35 | file: 36 | type: string 37 | steps: 38 | - print: 39 | message: Printing << parameters.file >> 40 | - run: cat << parameters.file >> 41 | 42 | workflows: 43 | my-workflow: 44 | jobs: 45 | - cat-file: 46 | file: test.txt 47 | ``` 48 | 49 | Even though the `print` command is called from the `cat-file` job, the 50 | `file` parameter would not be in scope inside the `print`. This 51 | ensures that all parameters are always bound to a valid value, and the 52 | set of available parameters is always known. 53 | 54 | ## Pipeline Value Scope 55 | 56 | Pipeline values, the pipeline-wide values that are provided by 57 | CircleCI (e.g. `<< pipeline.number >>`) are always in scope. 58 | 59 | ## Pipeline Parameter Scope 60 | 61 | Pipeline parameters which are defined in configuration are always in 62 | scope, with two exceptions: 63 | 64 | - Pipeline parameters are not in scope for the definition of other 65 | pipeline parameters, so they cannot depend on one another 66 | - Pipeline parameters are not in scope in the body of orbs, even 67 | inline orbs, to prevent data leaks 68 | -------------------------------------------------------------------------------- /docs/pipeline-parameters.md: -------------------------------------------------------------------------------- 1 | # Pipeline Parameters 2 | 3 | Pipeline parameters are new with API v2. To use pipeline parameters you must use configuration version 2.1 or higher. 4 | 5 | ## Declaring and using pipeline parameters in configuration 6 | 7 | Pipeline parameters are declared using a `parameters` stanza in the top level keys of your `.circleci/config.yml` file. You can then reference the value of the parameter as a config variable in the scope `pipeline.parameters`. 8 | 9 | The example belows shows a config with two pipeline parameters, `image-tag` and `workingdir` both used on the subsequent config stanzas: 10 | 11 | ``` 12 | version: 2.1 13 | parameters: 14 | image-tag: 15 | type: string 16 | default: "latest" 17 | workingdir: 18 | type: string 19 | default: "~/main" 20 | 21 | jobs: 22 | build: 23 | docker: 24 | - image: circleci/node:<< pipeline.parameters.image-tag >> 25 | environment: 26 | IMAGETAG: << pipeline.parameters.image-tag >> 27 | working_directory: << pipeline.parameters.workingdir >> 28 | steps: 29 | - run: echo "Image tag used was ${IMAGETAG}" 30 | - run: echo "$(pwd) == << pipeline.parameters.workingdir >>" 31 | ``` 32 | 33 | 34 | ## Passing parameters when triggering pipelines via the API 35 | Use the API v2 endpoint to trigger a pipeline, passing the `parameters` key in the JSON packet in your POST body. 36 | 37 | The example below triggers a pipeline with the parameters in the above config example (_NOTE: To pass a parameter when triggering a pipeline via the API the parameter must be declared in the configuration file._). 38 | 39 | ``` 40 | curl -u ${CIRCLECI_TOKEN}: -X POST --header "Content-Type: application/json" -d '{ 41 | "parameters": { 42 | "workingdir": "./myspecialdir", 43 | "image-tag": "4.8.2" 44 | } 45 | }' https://circleci.com/api/v2/project/:project_slug/pipeline 46 | ``` 47 | 48 | ## The scope of pipeline parameters 49 | Pipeline parameters can only be resolved in the `.circleci/config.yml` file in which they are declared. Pipeline parameters are not available in orbs, including orbs declared locally in your config.yml file. We made this design decision because access to the pipeline scope in orbs would break encapsulation and create a hard dependency between the orb and the calling config, jeopardizing determinism and creating surface area of vulnerability. 50 | 51 | 52 | -------------------------------------------------------------------------------- /docs/README.md: -------------------------------------------------------------------------------- 1 | # CircleCI API v2 Preview Documentation 2 | 3 | ## BEFORE YOU START: IMPORTANT DISCLAIMER 4 | The CircleCI v2 API is currently in Preview release. You may use it at will, but this evaluation product is not yet fully supported or considered generally available. We have no plans to change any live endpoints in v2, and we are treating it as production software, but the risk of disruption or breaking changes is higher than our generally available features. All breaking changes will be announced ahead of time in our [breaking changes log](breaking.md) 5 | 6 | Use of the v2 API as long as this notice is in the master branch (current as of June 2019) is done at your own risk and is governed by CircleCI’s Terms of Service. 7 | 8 | PREVIEWS ARE PROVIDED "AS-IS," "WITH ALL FAULTS," AND "AS AVAILABLE," AND ARE EXCLUDED FROM THE SERVICE LEVEL AGREEMENTS AND LIMITED WARRANTY. Previews may not be covered by customer support. Previews may be subject to reduced or different security, compliance and privacy commitments, as further explained in the Terms of Service, Privacy Policy and any additional notices provided with the Preview. We may change or discontinue Previews at any time without notice. We also may choose not to release a Preview into "General Availability." 9 | 10 | 11 | # RESOURCES IN THE PREVIEW DOCS 12 | * [New endpoints in API v2](api-changes.md) 13 | * [New: Pipeline parameters](pipeline-parameters.md) 14 | * [New: Conditional workflows](conditional-workflows.md) 15 | 16 | 17 | 18 | # tldr 19 | 20 | The CircleCI v2 API behaves very similarly to our v1.1 API. For instance, to get back basic information about the user associated with an API token in v2 you could run (assuming you have set CIRCLECI_TOKEN in your environment): 21 | 22 | `curl -u ${CIRCLECI_TOKEN}: https://circleci.com/api/v2/me` 23 | 24 | 25 | ## GETTING STARTED WITH CIRCLECI API v2 26 | The v2 API is very similar to our currently documented v1.1 API with some notable exceptions. For details on see the file [docs/api-changes.md](api-changes.md) in this repository, originally at . 27 | 28 | Here is a simple example using `curl` to trigger a pipeline with parameters (NOTE: see [docs/pipeline-parameters.md](pipeline-parameters.md) in this repository for more on pipeline parameters, as they behave differently from parameters in the 1.1 job triggering endpoint): 29 | 30 | ``` 31 | curl -u ${CIRCLECI_TOKEN}: -X POST --header "Content-Type: application/json" -d '{ 32 | "parameters": { 33 | "myparam": "./myspecialdir", 34 | "myspecialversion": "4.8.2" 35 | } 36 | }' https://circleci.com/api/v2/project/${project_slug}/pipeline 37 | ``` 38 | 39 | In the above example the `project_slug` would take the form `:vcs/:org/:project`. For instance the project slug `gh/CircleCI-Public/circleci-cli` tells CircleCI to use the project found in the GitHub organization `CircleCI-Public` in the repository named `circleci-cli`. 40 | 41 | # Where next? 42 | 43 | We publish the the current OpenAPI spec as 44 | [JSON](https://circleci.com/api/v2/openapi.json) and 45 | [YAML](https://circleci.com/api/v2/openapi.yml) as part of the API. 46 | 47 | -------------------------------------------------------------------------------- /designs-for-feedback/trigger-workflows-in-pipeline.md: -------------------------------------------------------------------------------- 1 | 2 | # NOTE: THIS DESIGN DOCUMENT IS NOW OBSOLETE. 3 | PLEASE SEE [THE DOC ON CONDITIONAL WORKFLOWS](../docs/conditional-workflows.md) FOR INFO ON THE ACCEPTED IMPLEMENTATION. 4 | 5 | 6 | 7 | 8 | 9 | 10 | ## Problem statement 11 | Users want to be able to trigger a pipeline, selecting a specific workflow in 12 | the configuration to run. They also need a way for such manually triggered 13 | workflows to not always run when the pipeline is triggered without specifying a 14 | particular workflow. 15 | 16 | ## Proposed solution 17 | We will add new semantics to workflows configuration which will use pipeline 18 | parameters (and potentially other types of values in the future) to determine 19 | whether a particular workflow should run within a pipeline. Standard pipeline 20 | parameters can then be passed in to select which workflows should be run. 21 | 22 | ## Design options for configuration of workflows 23 | If a pipeline configuration contains multiple workflows the behavior today is to 24 | run them both immediately upon triggering. With the ability to trigger a 25 | particular workflow via the API, users will likely want to be able to configure 26 | workflows intended to be run only when explicitly triggered and not run when a 27 | normal webhook from GitHub or other VCS system is received upon a push to the 28 | repo. Thus, we need to introduce some mechanism to declare under what 29 | circumstances certain workflows in a configuration should be run. 30 | 31 | The currently planned solution uses the existing pipeline parameters feature 32 | coupled with some new semantics within workflows to decide whether a workflow 33 | will be run. This feature requires pipelines to be enabled on the project, and a 34 | configuration of version 2.1 (or newer). 35 | 36 | ```yaml 37 | version: 2.1 38 | 39 | parameters: 40 | run_integration_tests: 41 | type: boolean 42 | default: false 43 | deploy: 44 | type: boolean 45 | default: false 46 | 47 | workflows: 48 | version: 2 49 | integration_tests: 50 | when: << pipeline.parameters.run_integration_tests >> 51 | jobs: 52 | - tests 53 | - when: 54 | condition: << pipeline.parameters.deploy >> 55 | steps: 56 | - deploy 57 | 58 | jobs: 59 | ... 60 | ``` 61 | 62 | The above would prevent the workflow `integration_tests` from being triggered 63 | unless it was invoked explicitly when the pipeline is triggered with: 64 | 65 | ```json 66 | { 67 | "parameters": { 68 | "run_integration_tests": true 69 | } 70 | } 71 | ``` 72 | 73 | The `when` key actually accepts any boolean, not just pipeline parameters, 74 | though pipeline parameters will be the only meaningful use of this feature until 75 | we implement others. 76 | 77 | `when` would also come with an alternative of `unless`, which inverts truthiness 78 | of the condition. 79 | 80 | The use of pipeline parameters has the following benefits: 81 | 82 | - Pipeline parameters are an existing feature which is planned to be used more 83 | widely, and thus limit the need for additional documentation and generally 84 | limit the semantics of configuration through reuse. 85 | - They allow custom grouping and defaults, giving users more control over the 86 | structure of their workflows without incurring additional complexity on our 87 | side. Users could for example reuse the same parameter for several workflows 88 | which should always run together. 89 | -------------------------------------------------------------------------------- /docs/path-filtering.md: -------------------------------------------------------------------------------- 1 | # Setup Workflows, Path Filtering, and Monorepos 2 | 3 | CircleCI has so far not had any first-class monorepo support. One common problem this entails for monorepo users is that every push runs the entire CI suite. 4 | 5 | The work to be run is defined in CircleCI configuration, which is fetched and compiled before any workflow is created or run. As such it cannot be dynamically changed based on the changes in the repository, for example just build the components of a monorepo which have actually changed. 6 | 7 | 8 | ## Setup Workflows 9 | 10 | Setup workflows are a new opt-in CircleCI feature which allows the running of a workflow to dynamically generate a CircleCI configuration and pipeline parameters, and then run the resulting work within the same pipeline. This way it sidesteps the limitation outlined above and enables dynamic work discovery. 11 | 12 | To this end, the pipeline enters a new `setup` phase (as denoted by the `setup` stanza in the configuration), during which a single workflow can be run. The jobs in this workflow have access to a special `continuation-key`, which can be used to call the public continuation API at [/api/v2/pipeline/continue](https://circleci.com/docs/api/v2/#operation/continuePipeline). This call includes a new configuration file, as well as pipeline parameters. The pipeline is then advanced to the regular `created` phase. A pipeline can only be continued once, and only within 6 hours. 13 | 14 | If a pipeline enters the `setup` phase, the configuration used initially is called the `setup config`, and is stored alongside the regular configuration. 15 | 16 | The `continuation-key` is injected into the jobs as a secret environment variable, `$CIRCLE_CONTINUATION_KEY`, and excluded from API responses and step output. We strongly recommend against extracting it from the job, as it allows submission of arbitrary configurations, and thus could potentially be used to extract secrets from restricted contexts. 17 | 18 | ![img](./setup-workflows.png "Flow") 19 | 20 | 21 | ## Using the Path Filtering Orb 22 | 23 | To reduce the boilerplate required for a monorepo setup, we have created a set of orbs to implement path filtering and [automatic continuation](https://circleci.com/developer/orbs/orb/sandbox/continuation). 24 | 25 | First, you need to enable setup workflows for your project (Project Settings -> Advanced -> At the bottom). After signing up to the preview, navigate to the project settings page on CircleCI and enable the "Run Setup Workflows" setting. 26 | 27 | The following steps can happen on a feature branch. 28 | 29 | Move your existing config to `.circleci/continue_config.yml`. 30 | 31 | Separate out the tests for different components into separate workflows, and gate them on pipeline parameters, like so: 32 | 33 | ```yaml 34 | version: 2.1 35 | 36 | parameters: 37 | build-server: 38 | type: boolean 39 | default: false 40 | build-client: 41 | type: boolean 42 | default: false 43 | 44 | jobs: 45 | - [...] 46 | 47 | workflows: 48 | build-server: 49 | when: << pipeline.parameters.build-server >> 50 | jobs: 51 | - [...] 52 | build-client: 53 | when: << pipeline.parameters.build-client >> 54 | jobs: 55 | - [...] 56 | ``` 57 | 58 | The `equal` statements ensure that the workflows always run on the default branch, i.e. after merging. 59 | 60 | Place this config at `.circleci/config.yml`: 61 | 62 | ```yaml 63 | version: 2.1 64 | 65 | setup: true 66 | 67 | orbs: 68 | path-filtering: circleci/path-filtering@0.0.1 69 | 70 | workflows: 71 | setup-workflow: 72 | jobs: 73 | - path-filtering/filter: 74 | mapping: | 75 | src/server/.* build-server true 76 | src/client/.* build-client true 77 | # Optional, defaults to main: 78 | base-revision: origin/develop 79 | ``` 80 | 81 | Match the paths and parameters in the mapping to your project. 82 | 83 | The mappings consist of three whitespace-separated elements, a regular expression matching a path, a pipeline parameter, and a value for that parameter. 84 | 85 | The mappings are evaluated in order. If the regular expression matches any file changed between the base revision and the current `HEAD`, the pipeline parameter will be set to the value specified. Later matches override earlier ones. **Note**: If `base-revision` is a non-default branch, it needs to be prefixed with `origin/`, as it will not be fetched by default. 86 | 87 | The regular expressions support the full [Python re syntax](https://docs.python.org/3.8/library/re.html#regular-expression-syntax), and are automatically enclosed by `^` and `$` to prevent partial matches. 88 | 89 | 90 | ## Further Reading 91 | 92 | You can find more information about the elements used here: 93 | 94 | - [Setup Workflows](./setup-workflows.md) 95 | - [Pipeline Variables](https://circleci.com/docs/2.0/pipeline-variables/) 96 | - [Logic Statements](https://circleci.com/docs/2.0/configuration-reference/#logic-statements) 97 | - [Conditional Steps](https://circleci.com/docs/2.0/reusing-config/#defining-conditional-steps) 98 | - [Introduction to Orbs](https://circleci.com/docs/2.0/orb-intro/) 99 | -------------------------------------------------------------------------------- /docs/scheduled-pipelines.md: -------------------------------------------------------------------------------- 1 | Schedule Pipelines allow users to trigger pipelines periodically based on a schedule. 2 | 3 | Since the scheduled run is based on pipelines, scheduled pipelines have all the features that come with using pipelines: 4 | 5 | - Can control the actor that's associated with the pipeline, which can enable the use of restricted contexts 6 | - Can use dynamic config via setup workflows 7 | - Can modify the schedule without having to change the config.yml 8 | - Can interact with auto-cancelling of pipelines 9 | - Can specify pipeline parameters associated with a schedule 10 | 11 | CircleCI has APIs that allows users to create, view, edit, and delete Scheduled Pipelines. 12 | 13 | **Note**: *At this time, the UI for Scheduled Pipelines is not available. It will become available soon. 14 | 15 | ## Getting started with scheduled pipelines in CircleCI 16 | 17 | ### Start from scratch 18 | If your project has no scheduled workflows and would like to try out scheduled pipelines: 19 | 20 | - Have your CCI token ready or create a new one following [these steps](https://circleci.com/docs/2.0/managing-api-tokens/) 21 | - Create a new schedule using the new Pipelines Schedule API, for example: 22 | 23 | ``` 24 | curl --location --request POST 'https://circleci.com/api/v2/project//schedule' \ 25 | --header 'circle-token: ' \ 26 | --header 'Content-Type: application/json' \ 27 | --data-raw '{ 28 | "name": "my schedule name", 29 | "description": "some description", 30 | "attribution-actor": "system", 31 | "parameters": { 32 | "branch": "main" 33 | 34 | }, 35 | "timetable": { 36 | "per-hour": 3, 37 | "hours-of-day": [1,15], 38 | "days-of-week": ["MON", "WED"] 39 | } 40 | }' 41 | ``` 42 | 43 | - Check out the `schedule` section under the [open-api docs](https://circleci.com/docs/api/v2/) for additional information 44 | 45 | ### Migrating from scheduled workflows to scheduled pipelines 46 | 47 | Currently, using Scheduled Workflows has numerous shortcomings. Some of them are listed below: 48 | 49 | - Cannot control the actor, so scheduled workflows can't use restricted contexts 50 | - Cannot control the interaction with auto-cancelling of pipelines 51 | - Cannot use scheduled workflow together with dynamic config without hakcy workarounds 52 | - Cannot change or cancel scheduled workflows on a branch without triggering a pipeline 53 | - Cannot kick off test runs for scheduled workflows without changing the schedule 54 | - Cannot restrict scheduled workflows from PR branches if you want the workflow to run on webhooks 55 | 56 | To migrate from scheduled workflows to scheduled pipelines, one can follow the steps below: 57 | 58 | - Find the scheduled trigger in your project's .circleci/config.yml 59 | - For example, it might look like this one: 60 | 61 | ``` 62 | daily-run-workflow: 63 | triggers: 64 | - schedule: 65 | # Every day, 0421Z. 66 | cron: "21 4 * * *" 67 | filters: 68 | branches: 69 | only: 70 | - main 71 | jobs: 72 | - test 73 | - build 74 | ``` 75 | 76 | - Interpret the frequency your trigger needs to run from the cron expression 77 | - Use the same step from `Starting from scratch` section above to create the schedule via the API 78 | - In the circleci config file, remove the `triggers` section so that it looks like a normal workflow 79 | 80 | ``` 81 | daily-run-workflow: 82 | jobs: 83 | - test 84 | - build 85 | ``` 86 | 87 | - Add workflows filtering. As scheduled pipeline is essentially a triggered pipeline, it will run every workflow in the config. 88 | - One way to implement workflows filtering is by using the pipeline values, for example: 89 | 90 | ``` 91 | daily-run-workflow: 92 | when: 93 | and: 94 | - equal: [ scheduled_pipeline, << pipeline.trigger_source >> ] 95 | - equal: [ "my schedule name", << pipeline.schedule.name >> ] 96 | jobs: 97 | - test 98 | - build 99 | ``` 100 | 101 | - Please note that in the above example, the second `equal` under `when` is not strictly necessary. The `pipeline.schedule.name` is an available pipeline value when the pipeline is triggered by a schedule. 102 | 103 | 104 | - Add workflows filtering for workflows that should NOT run when a schedule triggers: 105 | 106 | ``` 107 | daily-run-workflow: 108 | when: 109 | and: 110 | - equal: [ scheduled_pipeline, << pipeline.trigger_source >> ] 111 | - equal: [ "my schedule name", << pipeline.schedule.name >> ] 112 | jobs: 113 | - test 114 | - build 115 | 116 | other-workflow: 117 | when: 118 | not: 119 | equal: [ scheduled_pipeline, << pipeline.trigger_source >> ] 120 | jobs: 121 | - build 122 | - deploy 123 | ``` 124 | 125 | ## Scheduled pipeline FAQs 126 | 127 | **Q:** How do I find the schedules that I've created? 128 | 129 | **A:** As scheduled pipelines are stored directly in CircleCI, we now have a UUID associated with each schedule. Or you could list all the schedules under a single project 130 | 131 | ``` 132 | curl --location --request GET 'https://circleci.com/api/v2/project//schedule' \ 133 | --header 'circle-token: ' 134 | ``` 135 | 136 | **Q:** Why is my Scheduled Pipeline not running? 137 | 138 | **A:** There can be many reasons for this. For example: 139 | - Is the actor who is set for the scheduled pipelines still part of the organization? 140 | - Is the branch set for the schedule deleted? 141 | - Is your github organization using SAML protection? SAML tokens expire often, which can cause requests to github to fail. 142 | -------------------------------------------------------------------------------- /docs/setup-workflows.md: -------------------------------------------------------------------------------- 1 | # Setup Workflows 2 | 3 | Setup workflows are a new CircleCI feature which allow users to 4 | dynamically generate configuration within a job, and by extension 5 | dynamically decide which work to execute. The resulting work will be 6 | executed within the same pipeline. 7 | 8 | Prior to this feature, configuration of a pipeline was relatively 9 | static: while we had [conditional steps](https://circleci.com/docs/2.0/reusing-config/#defining-conditional-steps) and [pipeline parameters](https://circleci.com/docs/2.0/pipeline-variables/#pipeline-parameters-in-configuration), it was 10 | not possible to manipulate a pipeline's configuration in an arbitrary 11 | way. 12 | 13 | Setup workflows allow us to dynamically generate or manipulate our 14 | configuration. This is a powerful means of extension and abstraction, 15 | and enables features that were difficult to accomplish with static 16 | configuration - for example [path filtering](./path-filtering.md) for monorepo support, or 17 | [merging several configuration files](https://github.com/circle-makotom/circle-advanced-setup-workflow). 18 | 19 | ## Concepts 20 | 21 | Setup workflows introduce new, optional pipeline lifecycle phases, 22 | which allow work to be executed. Where previously a pipeline has 23 | consisted of two phases: 24 | 25 | 1. Configuration Processing 26 | 1. Work Execution 27 | 28 | it can now have two additional phases: 29 | 30 | 1. *Setup* Configuration Processing 31 | 1. *Setup* Work Execution 32 | 1. Configuration Processing 33 | 1. Work Execution 34 | 35 | ![Schematic](./setup-workflows.png) 36 | 37 | During the first phase, the _setup configuration_ is processed, which 38 | describes the work to be executed during the _setup work execution_ 39 | phase. The workflow executed as part of the _setup work execution_ phase 40 | is called a _setup workflow_. _Setup workflows_ are tagged as such in the 41 | UI and API. The _setup workflow_ then continues the pipeline to enter 42 | the next phase. 43 | 44 | Behind the scenes this continuation is implemented as a call to a 45 | [public pipeline continuation API](https://circleci.com/docs/api/v2/#operation/continuePipeline). This API accepts a _continuation key_, 46 | which is a secret, unique-per-pipeline key, that is automatically 47 | injected into the environment of jobs executed as part of a setup 48 | workflow. We advise against extracting this key. It also accepts a 49 | configuration string, as well as a set of pipeline parameters. 50 | 51 | We provide several orbs to cover common use cases for setup workflows, 52 | such as [path filtering](https://circleci.com/developer/orbs/orb/circleci/path-filtering). 53 | 54 | This means each pipeline that has entered the setup phase has two 55 | configurations, the _setup configuration_, and the regular 56 | configuration. 57 | 58 | ## Limitations 59 | 60 | Some limitations apply to setup workflows: 61 | 62 | - the setup phase requires configuration version 2.1 or higher 63 | - a pipeline can only be continued once 64 | - a pipeline can only be continued within six hours of its creation 65 | - a pipeline cannot be continued with another setup configuration 66 | - there can only be one workflow in the setup configuration 67 | - pipeline parameters submitted at continuation time cannot overlap 68 | with pipeline parameters submitted at trigger time 69 | - pipeline parameters declared in the setup configuration must also be 70 | declared in the continuation configuration, and can be used at 71 | continuation time 72 | 73 | ## Enabling Setup Workflows 74 | 75 | To enable setup workflows, simply enable "Setup Workflows" in the 76 | project settings (under "Advanced"). Now you can push a setup 77 | configuration to a branch. To designate a configuration as a setup 78 | configuration, and thus trigger the setup phase, use the top-level 79 | `setup: true` stanza (see below for a full example). Regardless of the 80 | project setting, only setup configurations will trigger the setup 81 | phase. 82 | 83 | ## Full Example 84 | 85 | In this example we presume that a `generate-config` script or executable 86 | exists, which outputs a YAML string, based on some work it performs. 87 | It could potentially inspect git history, or pipeline values that get 88 | passed to it, or do anything else you can do inside a job. 89 | 90 | ```yaml 91 | version: 2.1 92 | 93 | setup: true 94 | 95 | orbs: 96 | continuation: circleci/continuation@0.1.2 97 | 98 | jobs: 99 | setup: 100 | executor: continuation/default 101 | steps: 102 | - checkout 103 | - run: 104 | name: Generate config 105 | command: | 106 | ./generate-config > generated_config.yml 107 | - continuation/continue: 108 | configuration_path: generated_config.yml 109 | 110 | workflows: 111 | setup: 112 | jobs: 113 | - setup 114 | ``` 115 | 116 | ## Advanced Topics 117 | 118 | ### Using Custom Executors 119 | 120 | Alternative executors can be used, but require certain dependencies to 121 | be installed for the continuation step to work (currently: `curl`, 122 | `jq`). 123 | 124 | ### Choosing not to Continue a Pipeline 125 | 126 | If the setup workflow decides that no further work shall be executed, 127 | it is good practice to finish the pipeline, avoiding accidental 128 | continuation. The continuation orb has a command for this: 129 | 130 | ```yaml 131 | steps: 132 | - continuation/finish 133 | ``` 134 | 135 | ### Not Using the Continuation Orb 136 | 137 | If you have special requirements not covered by the continuation orb, 138 | you can implement the same functionality in different ways. Refer to 139 | the [orb source code](https://app.circleci.com/pipelines/github/CircleCI-Public/continuation-orb) for reference. 140 | -------------------------------------------------------------------------------- /docs/breaking.md: -------------------------------------------------------------------------------- 1 | # Breaking Changes 2 | 3 | ## Dec 9, 2019 4 | The `workflow` key in the [Get a pipeline](https://circleci.com/docs/api/v2/#get-a-pipeline) `GET /pipeline/{pipeline-id}` response is now deleted 5 | 6 | ## Nov 21, 2019 7 | 8 | The endpoint to [Get a pipeline's workflows](https://circleci.com/docs/api/v2/#get-a-pipeline-39-s-workflows) `GET /pipeline/{pipeline-id}/workflow` has been released. 9 | 10 | The `workflow` key in the [Get a pipeline](https://circleci.com/docs/api/v2/#get-a-pipeline) `GET /pipeline/{pipeline-id}` response is now deprecated and is scheduled to be deleted on Dec 9, 2019 11 | 12 | Instances where the `workflow` key is being referenced from the [Get a pipeline](https://circleci.com/docs/api/v2/#get-a-pipeline) `GET /pipeline/{pipeline-id}` response should be modified to make a call to [Get a pipeline's workflows](https://circleci.com/docs/api/v2/#get-a-pipeline-39-s-workflows) `GET /pipeline/{pipeline-id}/workflow` instead. 13 | 14 | 15 | ## Nov 11, 2019 16 | 17 | The `/workflow/:workflow-id/jobs` endpoint has been moved to `/workflow/:workflow-id/job` for consistency. The old route was removed on November 20th, 2019. 18 | 19 | ## Oct 17, 2019 20 | 21 | - `start_time` and `stop_time` parameters have been renamed to `started_at` and `stopped_at` to be consistent with other time-based information fields like `created_at` and `queued_at`. This change affects the [Get a Workflow's Job](https://circleci.com/docs/api/v2/#get-a-workflow-39-s-jobs) endpoint 22 | - Cancelling a job is no longer a `POST` to `/project/:project-slug/job/:jobnum`. The job cancellation action is now a `POST` to `/project/:project-slug/job/:jobnum/cancel` 23 | 24 | # Proposed Breaking Changes 25 | 26 | ## Log 27 | 28 | ### Nov 13, 2019 29 | After some feedback, we looked into rolling `scheduled workflows` into `GET /pipeline/{pipeline-id}/workflow` response. We can accomplish this by creating a pipeline for each trigger of a `scheduled workflow`. Proposed breaking changes have been revised to reflect this. 30 | 31 | ### Nov 5, 2019 32 | 33 | - Route changes will not be pluralized, instead they will be all singular 34 | - `workflows` will be removed from the [Get a pipeline](https://circleci.com/docs/api/v2/#get-a-pipeline) response. This information will now be retrieved in two new endpoints: 35 | - Get workflows by pipeline `GET /pipeline/{pipeline-id}/workflow` 36 | - ~~Get scheduled workflows by project `GET /project/{project-slug}/scheduled-workflow`~~ 37 | 38 | ### Oct 25, 2019 39 | 40 | Fixed some typos and clarified some of the changes, specifically around pluralization and error responses. 41 | 42 | ### Oct 23, 2019 43 | 44 | Initial list of proposed changes to v2 preview 45 | 46 | ## Proposed Changes 47 | 48 | The API v2 preview was made available earlier this year with the intent of exposing [new workflow and pipeline control endpoints](https://github.com/CircleCI-Public/api-preview-docs/blob/master/docs/api-changes.md#new-endpoints-available-for-preview-use) to our API users. With the v2 Preview endpoints in place, the team at CircleCI also began to migrate API v1 and v1.1 endpoints into v2. 49 | 50 | As we began to migrate endpoints into v2, the team took some time to re-evaluate the design and UX of the overall API itself. It was during this process that we identified some gaps and inconsistencies with how the v2 endpoints behaved. These changes would require us to introduce breaking changes, but contributes to a more consistent and scalable API v2 experience. 51 | 52 | The changes we are making fall into one of five general categories: 53 | 54 | - [Removed endpoints](#removed-endpoints) 55 | - [Consistent HTTP responses](#consistent-http-responses) 56 | - [Changes to routes](#changes-to-routes) 57 | - [Paginated responses](#paginated-responses) 58 | - [Changes to Reponses](#changes-to-responses) 59 | 60 | ### Removed endpoints 61 | 62 | The list of removed endpoints have already been announced in the original API v2 preview announcement. You can find the list of endpoints that didn't make the transition from v1.1 to v2 [here](https://github.com/CircleCI-Public/api-preview-docs/blob/master/docs/api-changes.md#endpoints-likely-being-removed-in-api-v2-still-available-in-v11-for-now). 63 | 64 | In addition to that list, the following endpoint has been removed: 65 | 66 | #### Add User to Build 67 | 68 | `POST project/:vcs-type/:username/:project/:build_num/ssh-users?circle-token=:token` has been removed because this operation is no longer supported on the CircleCI 2.0 platform. 69 | 70 | ### Consistent HTTP Responses 71 | 72 | We've caught a few situations where some API endpoints don't respond with the expected HTTP Status code. We will be standardizing our HTTP responses to make sure that all our `v2` endpoints provide a consistent HTTP response status. 73 | 74 | #### Successful Status Codes 75 | 76 | 200 OK - responds with a custom payload 77 | 201 Created - responds with the created entity 78 | 202 Accepted - responds with `{"message": "Accepted."}` 79 | 80 | #### Error Status Codes 81 | 82 | ##### User Errors 83 | 84 | For user fault errors, the error message will be human readable and have enough information for a developer to understand and address the problem. 85 | 86 | ###### 404 Not Found 87 | 88 | Generally responds with: 89 | 90 | ``` 91 | {"message": "some-entity-name not found."} 92 | ``` 93 | 94 | For example, you will see this error code when a resource is not found, or if the URI doesn't exist or is malformed. 95 | 96 | ###### 400 Bad Request 97 | 98 | Generally responds with: 99 | 100 | ``` 101 | {"message": "some-entity-name not found."} 102 | ``` 103 | 104 | For example, you will typically see this error code when query parameters are missing or invalid, or if the request body is invalid. 105 | 106 | #### Server Fault Codes 107 | 108 | The relevant HTTP 5xx status codes will be used in the situation where a server error occurs. The return body will contain a message. For example: 109 | 110 | - 500 Internal Server Error - responds with `{"message": "Internal server error."}` 111 | - 503 Service Unavailable - responds with `{"message": "Service unavailable."}` 112 | 113 | ### Changes to Routes 114 | 115 | - [Get Test Metadata](https://circleci.com/docs/api/v2/#get-test-metadata) endpoint will move from `/project/{project-slug}/{job-number}/tests` to `/project/{project-slug}/job/{job-number}/tests`. Note the addition of `job/` to the route 116 | - [Get a workflow's jobs](https://circleci.com/docs/api/v2/#get-a-workflow-39-s-jobs) `/workflow/{id}/jobs` will be singular and now live under `/workflow/{id}/job` 117 | 118 | 119 | ### Paginated Responses 120 | 121 | - [Get an environment variable](https://circleci.com/docs/api/v2/#get-an-environment-variable) response will be paginated 122 | - [Get all checkout keys](https://circleci.com/docs/api/v2/#get-all-checkout-keys) response will be paginated 123 | 124 | ### Changes to Responses 125 | 126 | #### Get a Pipeline 127 | A pipeline could potentially have an unbounded set of workflows associated with it. For that reason we are removing `workflow` from the [get a pipeline](https://circleci.com/docs/api/v2/#get-a-pipeline) response and introducing two new endpoints: 128 | 129 | 1. Get a pipeline's workflows `GET /pipeline/{pipeline-id}/workflow` 130 | -------------------------------------------------------------------------------- /docs/api-changes.md: -------------------------------------------------------------------------------- 1 | 2 | # What's new in API v2? 3 | 4 | ## BEFORE YOU START: IMPORTANT DISCLAIMER 5 | The CircleCI v2 API is currently in Preview release. You may use it at will, but this evaluation product is not yet fully supported or considered generally available. We have no plans to change any live endpoints in v2, and we are treating it as production software, but the risk of disruption or breaking changes is higher than our generally available features. 6 | 7 | Use of the v2 API as long as this notice is in the master branch (current as of June 2019) is done at your own risk and is governed by CircleCI’s Terms of Service. 8 | 9 | PREVIEWS ARE PROVIDED "AS-IS," "WITH ALL FAULTS," AND "AS AVAILABLE," AND ARE EXCLUDED FROM THE SERVICE LEVEL AGREEMENTS AND LIMITED WARRANTY. Previews may not be covered by customer support. Previews may be subject to reduced or different security, compliance and privacy commitments, as further explained in the Terms of Service, Privacy Policy and any additional notices provided with the Preview. We may change or discontinue Previews at any time without notice. We also may choose not to release a Preview into "General Availability." 10 | 11 | ## NOTE ON AUTHENTICATION 12 | Use of the below endpoints is similar to use of the version 1.1 endpoints in terms of authentication. A simple way to authenticate is to send your API token as the username of the HTTP request. For instance, let's say you have set `CIRCLECI_TOKEN` in your shell's environment. You could then use `curl` with that token like this: 13 | 14 | `curl -u ${CIRCLECI_TOKEN}: https://circleci.com/api/v2/me` 15 | 16 | Please refer to the main CirlceCI API documentation for more details on authenticating with a token. 17 | 18 | ## NEW: THE `project_slug` as a string 19 | The CircleCI v2 API is backwards compatible with previous API versions in the way it identifies your projects using repository name. For instance, if you want to pull information from CircleCI about the GitHub repository you can refer to that in the CircleCI API as `gh/CircleCI-Public/circleci-cli`, which is a "triplet" of the project type, the name of your "organization", and the name of the repository. For the project type you can use `github` or `bitbucket` as well as the shorter forms `gh` or `bb`, which are now supported in API v2). The organization is your username or organization name in your version control system. 20 | 21 | With API v2 we are introducing a string representation of the triplet called the `project_slug`, takes the form: `//`. The `project_slug` is included in the payload when pulling information about a project as well as when looking up a pipeline or workflow by ID. The `project_slug` can then be used to get information about the project. It's possible in the future we could change the shape of a `project_slug`, but in all cases it would be usable as a human-readable identifier for a given project. 22 | 23 | ## NEW: Insights endpoints 24 | 25 | With CircleCI v2 Insights endpoints, you can view the recent runs of your named workflows. The insights endpoints contains status, duration and credits consumed information. 26 | 27 | ## NEW ENDPOINTS AVAILABLE FOR PREVIEW USE. 28 | Note, all endpoints below are relative to: 29 | 30 | `https://circleci.com/api/v2` 31 | 32 | ### GET /workflow/:id 33 | Retrieve an individual workflow by its unique ID. 34 | 35 | ### GET /workflow/:id/job 36 | Retrieve the jobs of an individual workflow by its unique ID. Note that for now we're returning all jobs, but we reserve the right to paginate them in the future. The shape of the pagination will not change, but the default number of jobs may be reduced. 37 | 38 | ### GET /project/:project_slug 39 | Retrieve an individual project by its unique slug. 40 | 41 | ### POST /project/:project_slug/pipeline 42 | Trigger a new pipeline run on a project. 43 | 44 | To trigger with parameters you pass a `parameters` map inside a JSON object as part of the POST body. For details on passing pipeline parameters when triggering pipelines with the API see the [pipeline parameters documentation](pipeline-parameters.md). Note that pipeline parameters can also be used to populate a `when` or `unless` clause on a workflow to conditionally run one or more workflows. See the [conditional workflows](conditional-workflows.md) doc for more information. 45 | 46 | To trigger on a specific branch pass a parameter `branch` in the post body. For instance, it might look like: 47 | 48 | ``` 49 | curl -u ${CIRCLECI_TOKEN}: -X POST --header "Content-Type: application/json" -d '{ 50 | "branch": "dev" 51 | }' https://circleci.com/api/v2/project/${project_slug}/pipeline 52 | ``` 53 | 54 | 55 | ### GET /project/:project_slug/pipeline/ 56 | Retrieve recent set of pipelines for a project. 57 | 58 | ### GET /pipeline/:id 59 | Retrieve a particular pipeline by its unique ID. This will return basic information about the pipeline, including triggering information and the IDs of running workflows associated with the pipeline. Please note that workflows are created asyncronously to the creation of a pipeline, so in automated scripts if you trigger a pipeline it may not immediately have knowledge of all workflows that will be run as a result of your trigger. You may need to make subsequent, delayed requests to get all workflow IDs. In most cases this should be a matter of a few seconds, but in some situations the queues between our pipeline ingestion machinery and our workflows conductor can take longer to proceed. 60 | 61 | ### GET /pipeline/:id/config 62 | Retrieve the configuration (both the source and compiled versions) for a given pipeline. 63 | 64 | ### Run only workflows conditioned on API parameters 65 | Use the new `when` clause under a workflow, you can use the value of a boolean pipeline parameter to conditionally start specific workflows. See the documentation on [conditional workflows](conditional-workflows.md) for more. 66 | 67 | ### GET openapi.json or GET openapi.yml 68 | Gives you the current production OpenAPI spec for the v2 API (eg: ) 69 | 70 | ### GET /insights/:project-slug/workflows?branch=":branch-name" 71 | Retrieve aggregate data about project workflows for a specified branch. The aggregation window is the shorter of the last 90 days, or the last 250 executions. If no branch is selected, CircleCI will provided data for the default branch. The payload contains the following fields: 72 | 73 | - Workflow name 74 | - Aggregation window start (UTC) 75 | - Aggregation window end (UTC) 76 | - Successful runs 77 | - Failed runs 78 | - Total runs 79 | - Success rate 80 | - Throughput (average number of runs / day) 81 | - Total credits used 82 | - Duration statistics: max, min, mean, median, p95, standard deviation (all in seconds) 83 | 84 | ### GET /insights/:project-slug/workflows/:workflow-name?branch=":branch-name" 85 | Retrieve the recent 250 runs (within the last 90 days) of a named project workflow for a specified branch. If no branch is selected, CircleCI will provide data for the default branch. The payload contains the following fields: 86 | 87 | - Workflow ID 88 | - Status 89 | - Credits used 90 | - Duration (seconds) 91 | - Created at (UTC) 92 | - Stopped at (UTC) 93 | 94 | ## Endpoints likely being removed in API v2 (still available in v1.1 for now) 95 | ### POST /project/:vcs-type/:username/:project 96 | In v2 and forward you can only trigger a pipeline using the new endpoint. This endpoint will be removed in v2 and deprecated in v1.1 (note that this endpoint is not compatible with pipelines, so if you turn on pipelines it won't work properly. Please migrate to the new endpoint) 97 | 98 | ### POST /project/:vcs-type/:username/:project/:build_num/retry 99 | 100 | ### POST /project/:vcs-type/:username/:project/build 101 | Replaced by POST /project/:vcs-type/:username/:project/pipeline 102 | 103 | ### DELETE /project/:vcs-type/:username/:project/build-cache 104 | Note that 'build cache' is a concept from legacy CircleCI. Caches in our new platform are immutable. 105 | 106 | ### GET /recent-builds 107 | This will be replaced by the ability to retrieve recent pipelines. 108 | 109 | ## Other changes in v2 110 | * If you do not provide an ACCEPT header, the v2 API will return JSON. This is a change from 1.1, which would default to EDN. 111 | -------------------------------------------------------------------------------- /.circleci/runthrough.sh: -------------------------------------------------------------------------------- 1 | #! /bin/bash 2 | set -e 3 | set -o pipefail 4 | set -o functrace 5 | 6 | # TODO: Not tolerant of initial trigger failing to create any workflows in time - should have some retry logic. 7 | # TODO: Does not check for pipeline errors 8 | # TODO: Does not know all possible status values for workflows 9 | # TODO: Only gets jobs for workflows still running when initially retrieved 10 | 11 | 12 | #******************** 13 | # VARIABLES 14 | # ******************* 15 | 16 | # **** !!!! 17 | # **** CHANGE THESE TO REFLECT YOUR PROJECT 18 | # **** AND THE PARAMETERS YOU WANT TO SEND 19 | # **** !!!! 20 | project_slug="gh/ndintenfass/scratch" 21 | parameter_map='{"workingdir": "~/myspecialdir", "image-tag": "4.8.2"}' 22 | 23 | 24 | # YOU LIKELY DO NOT NEED TO EDIT THESE OR ANYTHING BELOW HERE 25 | path_to_cli_config='~/.circleci/cli.yml' 26 | circleci_root='https://circleci.com/' 27 | api_root="${circleci_root}api/v2/" 28 | cli_config_path="${HOME}/.circleci/cli.yml" 29 | 30 | # branch="tryapi" 31 | 32 | #******************** 33 | # FUNCTIONS 34 | # ******************* 35 | 36 | current_section=0 37 | current_step=0 38 | section () { 39 | ((current_section++)) 40 | # reset the step numbering at each section 41 | current_step=0 42 | echo "************* $(roman current_section). ${1} *************" 43 | } 44 | 45 | 46 | step () { 47 | ((current_step++)) 48 | echo "${current_step}. ${@: -1}" 49 | echo "" 50 | } 51 | 52 | error () { 53 | section 'GAME OVER' 54 | echo $1 55 | exit 1 56 | } 57 | 58 | require_command () { 59 | cli_exists=$(which $1) 60 | if [[ -z $cli_exists ]]; then 61 | error "${1} is required, but you don't seem to have it. Aborting because this script won't work without ${1}." 62 | fi 63 | echo "${1} is required and present. Proceeding..." 64 | } 65 | 66 | gather () { 67 | read -p read -p "${1}`echo $'\n> '`" var 68 | echo $var 69 | return 0 70 | } 71 | 72 | 73 | choice () { 74 | while true; do 75 | read -p "${1}. Answer y or n. " yn 76 | case $yn in 77 | [Yy]* ) echo; break;; 78 | [Nn]* ) section "IN THAT CASE, HAVE A GOOD DAY. SEE YOU NEXT TIME."; echo 'exiting...'; exit;; 79 | * ) echo "Please answer y or n.";; 80 | esac 81 | done 82 | } 83 | 84 | set_token () { 85 | step "You are using circleci CLI version $(circleci --skip-update-check version). This script will now look in your circleci CLI config to get a token value to use. (NOTE: For now this script assumes you use the default location for that config, but a nice improvement later would be to update this script to suss out the location using the diagnostic command of the circleci CLI.)" 86 | if [ -f "$cli_config_path" ]; then 87 | circle_token=$(yq r "${cli_config_path}" token) 88 | else 89 | circle_token=$(gather \ 90 | "You do not appear to have a config for your CLI at ${cli_config_path}, so to use this script you will need to enter your token manually.") 91 | fi 92 | step "A CircleCI token is set." 93 | } 94 | 95 | ensure_project_slug () { 96 | if [[ ! $project_slug ]]; then 97 | project_slug=$(gather \ 98 | "Enter your project slug in the form ':type/:org/:project',\ 99 | for instance valid project slug would look like\ 100 | 'gh/CircleCI-Public/circleci-cli'.") 101 | fi 102 | step "Your project_slug is set to ${project_slug}. You can change this in the file $( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" 103 | vcs_slug=$(awk -F/ '{print $1}' <<< $project_slug) 104 | org_name=$(awk -F/ '{print $2}' <<< $project_slug) 105 | project_name=$(awk -F/ '{print $3}' <<< $project_slug) 106 | } 107 | 108 | 109 | 110 | post () { 111 | local url="${api_root}${1}" 112 | printf "HTTP POST ${url}\n\n" > /dev/tty 113 | 114 | curl -su ${circle_token}: -X POST \ 115 | --header "Content-Type: application/json" \ 116 | -d "$2"\ 117 | "${url}" 118 | } 119 | 120 | get () { 121 | local url="${api_root}${1}" 122 | printf "HTTP GET ${url}\n\n" > /dev/tty 123 | curl -su ${circle_token}: \ 124 | --header "Content-Type: application/json" \ 125 | "${url}" 126 | } 127 | 128 | pretty_json () { 129 | # jq . <<< "${result}" | sed 's/^/ /' 130 | echo "\`\`\`" 131 | jq . <<< "${result}" 132 | echo "\`\`\`" 133 | } 134 | 135 | roman() { 136 | local values=( 1000 900 500 400 100 90 50 40 10 5 4 1 ) 137 | local roman=( 138 | [1000]=M [900]=CM [500]=D [400]=CD 139 | [100]=C [90]=XC [50]=L [40]=XL 140 | [10]=X [9]=IX [5]=V [4]=IV 141 | [1]=I 142 | ) 143 | local nvmber="" 144 | local num=$1 145 | for value in ${values[@]}; do 146 | while (( num >= value )); do 147 | nvmber+=${roman[value]} 148 | ((num -= value)) 149 | done 150 | done 151 | echo $nvmber 152 | } 153 | 154 | #******************** 155 | # PREAMBLE AND SETUP 156 | # ******************* 157 | section 'INTRO: A QUICK RUN THROUGH OF THE v2 API' 158 | step 'Hello, this script runs through some really simple cases enabled by the CircleCI v2 API's 159 | 160 | section 'CHECK PREREQUISITES' 161 | require_command git 162 | require_command circleci 163 | require_command yq 164 | require_command jq 165 | require_command awk 166 | section 'SETUP SOME VARIABLES' 167 | ensure_project_slug 168 | set_token 169 | 170 | #******************** 171 | # TRIGGER AND RETRIEVE PIPELINE 172 | # ******************* 173 | section 'TRY TRIGGERING AND RETRIEVING A PIPELINE' 174 | post_path="project/${project_slug}/pipeline" 175 | step "Attemping to trigger a pipeline with a POST to ${post_path}" 176 | params="{\"parameters\": ${parameter_map} }" 177 | result=$(post $post_path "${params}") 178 | pretty_json $result 179 | pipeline_id=$(echo $result | jq -r .id) 180 | step "Successfully created pipeline with ID $pipeline_id" 181 | get_path="pipeline/${pipeline_id}" 182 | step "GET pipeline by ID: /${get_path} - the raw payload is below" 183 | result=$(get $get_path) 184 | workflow_count=$(echo $result | jq -r '.workflows | length' ) 185 | 186 | pretty_json $result 187 | 188 | #******************** 189 | # GET WORKFLOWS 190 | # ******************* 191 | section 'GET THE WORKFLOWS FOR THE ABOVE PIPELINE' 192 | 193 | # jq -r .workflows.ids[] <<<$result 194 | 195 | 196 | step "You should now be able to see the ${workflow_count} workflow(s) for this pipeline here:" 197 | workflow_url_for_project="${circleci_root}${vcs_slug}/${org_name}/workflows/${project_name}" 198 | echo $workflow_url_for_project 199 | echo "" 200 | workflow_ids=($(echo $result | jq -r '.workflows[].id | @sh')) 201 | #DUMB HACK TO STRIP SINGLE QUOTES THAT CAN LIKELY BE SOLVED MORE ELEGANTLY 202 | workflow_ids=(${workflow_ids[@]//\'/}) 203 | step "Now let's loop over the ${workflow_count} workflow(s) and get info about each one" 204 | declare -a running_workflows 205 | for id in ${workflow_ids[@]}; do 206 | get_path="workflow/${id}" 207 | step "GET workflow by ID: /${get_path} - the raw payload is below" 208 | result=$(get $get_path) 209 | pretty_json $result 210 | status=$(echo $result | jq -r '.status') 211 | if [[ $status == "running" ]]; then 212 | echo "Still running, so add it to the list of workflows to poll down below..." 213 | running_workflows+=($id) 214 | else 215 | echo "Finished running with status: ${status}" 216 | fi 217 | done 218 | step "There's ${#running_workflows[@]} workflow(s) running. Below we loop through them, polling each one for a while to see if it will finish" 219 | 220 | for i in "${!running_workflows[@]}"; do 221 | id=${running_workflows[i]} 222 | still_running=true 223 | maxloops=25 224 | loops=0 225 | wait="1s" 226 | 227 | get_path="workflow/${id}" 228 | step "Poll every ${wait} to GET workflow /${get_path}" 229 | echo "NOTE: If you prefer to see it in the UI visit:" 230 | echo "${circleci_root}workflow-run/${id}" 231 | printf "polling, please wait..." 232 | while [ $still_running ]; do 233 | ((loops+=1)) 234 | result=$(get $get_path) 235 | status=$(echo $result | jq -r '.status') 236 | if [[ $status != "running" ]]; then 237 | still_running=false 238 | printf "Finished with status of ${status}!\n" 239 | echo "Now retrieve info on the jobs of this workflow" 240 | get_path="workflow/${id}/jobs" 241 | echo "GET workflow jobs: /${get_path}" 242 | result=$(get $get_path) 243 | pretty_json $result 244 | break 245 | else 246 | sleep $wait 247 | printf "." 248 | continue 249 | fi 250 | if [[ $loops == $maxloops ]]; then 251 | echo "Max loops of ${maxloops} has been reached, so stopping querying for this workflow." 252 | break 253 | fi 254 | done 255 | done 256 | 257 | #******************** 258 | # GET RECENT PIPELINES FOR THE PROJECT 259 | # ******************* 260 | section 'GET RECENT PIPELINES' 261 | get_path="project/${project_slug}/pipeline" 262 | step "GET recent pipelines for project ${project_slug} - The latest two from '.items' are below" 263 | result=$(get $get_path) 264 | echo $result | jq .items[0:2] 265 | 266 | 267 | --------------------------------------------------------------------------------