├── .gitignore
├── .github
└── CODEOWNERS
├── misc
└── reality_check.png
├── renovate.json
├── LICENSE
├── test-results
└── test-results.xml
├── README.md
└── .circleci
└── config.yml
/.gitignore:
--------------------------------------------------------------------------------
1 | .DS_Store
--------------------------------------------------------------------------------
/.github/CODEOWNERS:
--------------------------------------------------------------------------------
1 | * @circleci/on-prem
2 |
--------------------------------------------------------------------------------
/misc/reality_check.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/circleci/realitycheck/HEAD/misc/reality_check.png
--------------------------------------------------------------------------------
/renovate.json:
--------------------------------------------------------------------------------
1 | {
2 | "$schema": "https://docs.renovatebot.com/renovate-schema.json",
3 | "extends": [
4 | "config:recommended"
5 | ]
6 | }
7 |
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 | MIT License
2 |
3 | Copyright (c) 2017 CircleCI
4 |
5 | Permission is hereby granted, free of charge, to any person obtaining a copy
6 | of this software and associated documentation files (the "Software"), to deal
7 | in the Software without restriction, including without limitation the rights
8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 |
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 |
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 |
--------------------------------------------------------------------------------
/test-results/test-results.xml:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
11 |
12 |
13 |
14 |
15 |
16 |
17 |
18 |
19 |
20 |
21 |
22 |
23 |
24 |
25 |
26 |
27 |
28 |
29 |
30 |
31 |
32 |
33 |
34 |
35 |
36 |
37 |
38 |
39 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # CircleCI Reality Check
2 | This repository is used to check for a proper installation of CircleCI Server. It implements a sample app that validates some basic CircleCI features in parallel workflows.
3 |
4 | 
5 |
6 | ---
7 |
8 | ## Installation
9 | To install and run reality check on your CircleCI Server installation, follow these steps.
10 |
11 | 1. Fork this repository to the GitHub environment tied to your server installation. [See here for more information](https://support.circleci.com/hc/en-us/articles/360011235534).
12 |
13 | 2. Set up your repository project in your CircleCI Server installation.
14 |
15 | 3. Generate a **personal api token - not a project token**, and add it as an environment variable in your project with the name `CIRCLE_TOKEN`.
16 |
17 | 4. Determine the base URL of your install, including protocol, and remove the trailing slash and "app" subdomain if present. Add it as an environment variable in your project with the name `CIRCLE_HOSTNAME`.
18 |
19 | > *Note*: As an example, if your dashboard is accessible at `https://app.server.example.com/dashboard/` your `CIRCLE_HOSTNAME` is `https://server.example.com`
20 |
21 |
22 | 5. Set the environment variable in your project with the name `CIRCLE_CLOUD_PROVIDER` as either `gcp`, `aws`, or `other`. depending on your installation.
23 |
24 | 6. You can __optionally__ set `CIRCLE_WINDOWS_EXECUTOR` to `true` in order to run a set of verification jobs for the windows execution environment. This requires a supported CircleCI Windows image, see instructions in [CircleCI-Public/circleci-server-windows-image-builder](https://github.com/CircleCI-Public/circleci-server-windows-image-builder).
25 |
26 | 7. Configure the following contexts and keys (their values can be anything). Docs on how to set up contexts [can be found here](https://circleci.com/docs/2.0/contexts/).
27 |
28 | Context Name | Environment Variable Key Name | Value
29 | -------------------|------------------------------- |-----------------------------
30 | `org-global` | `CONTEXT_END_TO_END_TEST_VAR` | `1`
31 | `individual-local` | `MULTI_CONTEXT_END_TO_END_VAR` | `1`
32 |
33 | ### Example AWS Server Installation
34 | ```bash
35 | # Project Environment Variables
36 | CIRCLE_TOKEN=123456789-personal-access-token
37 | CIRCLE_HOSTNAME=https://aws-server-install.example.com
38 | CIRCLE_CLOUD_PROVIDER=aws
39 | CIRCLE_WINDOWS_EXECUTOR=true # Optional, only if using a Windowws AMI
40 |
41 | # org-global context environment variables
42 | CONTEXT_END_TO_END_TEST_VAR=1
43 |
44 | # individual-local context environment variables
45 | MULTI_CONTEXT_END_TO_END_VAR=1
46 | ```
47 |
48 | ### Example GCP Server Installation
49 | ```bash
50 | # Project Environment Variables
51 | CIRCLE_TOKEN=123456789-personal-access-token
52 | CIRCLE_HOSTNAME=https://gcp-server-install.example.com
53 | CIRCLE_CLOUD_PROVIDER=gcp
54 | CIRCLE_WINDOWS_EXECUTOR=true # Optional, only if using a Windows AMI
55 |
56 | # org-global context environment variables
57 | CONTEXT_END_TO_END_TEST_VAR=1
58 |
59 | # individual-local context environment variables
60 | MULTI_CONTEXT_END_TO_END_VAR=1
61 | ```
62 |
63 | ---
64 |
65 | ## Running Reality Check
66 | Once installed, to run reality check, make any commit to the default branch of your forked repository. If installed correctly, your CircleCI Server Installation will pick up the commit and build it in parallel workflows.
67 |
68 | ```bash
69 | git commit --allow-empty -m "Trigger Build, Empty commit" && git push
70 | ```
71 |
72 | ---
73 |
74 | ## Reality Check Workflows
75 | Descriptions of the three workflows follow.
76 |
77 | ## `resource_class` workflow
78 |
79 | Tests all known `resource_class` options—queries the CircleCI API to verify that jobs ran with the requested resources.
80 |
81 | ### Prerequisites
82 | - Your Nomad client instances must be large enough to accommodate these resource classes — see our [Configuration Reference](https://circleci.com/docs/2.0/configuration-reference/#resource_class) for details on the available resource classes and [our server docs](https://circleci.com/docs/2.0/nomad/#auto-scaling-policy-best-practices) about sizing your Nomad client instances.
83 | - The base URL of your CircleCI installation (e.g. https://circleci.com) must be specified via a `CIRCLE_HOSTNAME` project environment variable
84 | - A personal API token (see `CIRCLE_HOSTNAME/account/api` URL endpoint) must be stored as a `CIRCLE_TOKEN` project environment variable
85 |
86 | ## VM service workflow
87 | Tests the functionality of the [`machine` executor](https://circleci.com/docs/2.0/executor-types/#using-machine), [Remote Docker Environment](https://circleci.com/docs/2.0/building-docker-images), and [Docker Layer Caching](https://circleci.com/docs/2.0/docker-layer-caching).
88 |
89 | ## Features workflow
90 | - Tests ability to save and restore [caches](circleci.com/docs/2.0/caching)
91 | - Tests writing to and reading from [workspaces](https://circleci.com/docs/2.0/workflows/#using-workspaces-to-share-data-among-jobs)
92 | - Tests the default `org-global` [context](https://circleci.com/docs/2.0/contexts) (*NOTE:* needs a key called `CONTEXT_END_TO_END_TEST_VAR` to exist in a context called `org-global`)
93 | - Tests multiple contexts (*NOTE:* needs a key called `MULTI_CONTEXT_END_TO_END_VAR` to exist in a context called `individual-local`)
94 | - Tests upload/storage of [artifacts](https://circleci.com/docs/2.0/artifacts) and [test results](https://circleci.com/docs/2.0/collect-test-data)
95 |
96 |
97 | ## GCP Jobs Workflow
98 | Tests android machine images on Google Cloud Platform. First checks if the environment is configured for GCP with the CIRCLE_CLOUD_PROVIDER environment variable.
99 |
100 | ### Prerequisites
101 | You will need to have your CIRCLE_TOKEN environment variable set to a personal access token, and will need your CIRCLE_CLOUD_PROVIDER set to `gcp` in lower-case. You must also have nomad clients configured, and your vm-service configured.
102 |
103 | ## AWS Jobs Workflow
104 | Tests ARM based images on AWS. First checks if the environment is configured for AWS with the CIRCLE_CLOUD_PROVIDER environment variable.
105 |
106 | ### Prerequisites
107 | You will need to have your CIRCLE_TOKEN environment variable set to a personal access token, and will need your CIRCLE_CLOUD_PROVIDER set to `aws` in lower-case. You must also have nomad clients configured, and your vm-service configured.
108 |
109 | ---
110 |
111 | ## Contributing
112 | If you have more ideas for things that should tested, please submit a PR against the open-source repository on GitHub where this project is maintained: .
113 | See the current CI status of the main repo at .
114 | View the LICENSE file in this repository for licensing information.
115 |
--------------------------------------------------------------------------------
/.circleci/config.yml:
--------------------------------------------------------------------------------
1 | version: 2.1
2 |
3 | executors:
4 | docker: # Docker using the Base Convenience Image
5 | parameters:
6 | resource_class:
7 | type: string
8 | default: "small"
9 | docker:
10 | - image: 'cimg/base:2025.05-20.04'
11 | resource_class: <>
12 | windows: # Windows using the default windows image
13 | parameters:
14 | resource_class:
15 | type: string
16 | default: "medium"
17 | machine:
18 | image: windows-default
19 | shell: 'powershell.exe -ExecutionPolicy Bypass'
20 | resource_class: <>
21 |
22 | commands:
23 | # resource_class verification commands to execute depending on executor_type
24 | verify_resource_class:
25 | parameters:
26 | resource_class:
27 | type: string
28 | default: "small"
29 | executor_type:
30 | type: string
31 | default: "docker"
32 | steps:
33 | - when:
34 | condition:
35 | equal: [ "docker", << parameters.executor_type >> ]
36 | steps:
37 | - run:
38 | name: verify required Environment Variables
39 | command: |
40 | if [ -z "${CIRCLE_HOSTNAME}" -o -z "${CIRCLE_TOKEN}" ];then
41 | echo "You must provide 2 Environment Variables in project settings for this job to run."
42 | echo "CIRCLE_HOSTNAME: Should be the scheme://domain of your install. \"https://ci.example.com\""
43 | echo "CIRCLE_TOKEN: Should be the API Key of an admin or project level with Scope:All"
44 | exit 1
45 | fi
46 | - run:
47 | name: Verify that job ran with the requested resource_class option
48 | command: |
49 | curl -k \
50 | "${CIRCLE_HOSTNAME%/}/api/v1.1/project/github/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME/$CIRCLE_BUILD_NUM?\
51 | circle-token=$CIRCLE_TOKEN" | \
52 | jq '.picard.resource_class.class' | grep <>
53 | - when:
54 | condition:
55 | equal: [ "windows", << parameters.executor_type >> ]
56 | steps:
57 | - run:
58 | name: verify required Environment Variables
59 | command: |
60 | if (!((Test-Path env:CIRCLE_HOSTNAME) -and (Test-Path env:CIRCLE_TOKEN))) {
61 | Write-Host "You must provide 2 Environment Variables in project settings for this job to run."
62 | Write-Host "CIRCLE_HOSTNAME: Should be the scheme://domain of your install. "https://ci.example.com""
63 | Write-Host "CIRCLE_TOKEN: Should be the API Key of an admin or project level with Scope:All"
64 | Exit 1
65 | }
66 | - run:
67 | name: Verify that job ran with the requested resource_class option
68 | command: |
69 | $job = Invoke-RestMethod -URI "$env:CIRCLE_HOSTNAME/api/v2/project/gh/$env:CIRCLE_PROJECT_USERNAME/$env:CIRCLE_PROJECT_REPONAME/job/$env:CIRCLE_BUILD_NUM" `
70 | -Headers @{ "Circle-Token" = "$env:CIRCLE_TOKEN" }
71 | $job.executor.resource_class | Select-String -Pattern "<>"
72 |
73 | remote_docker_defaults: &remote_docker_defaults
74 | docker: [{image: 'docker:25.0-git'}]
75 |
76 | basic_docker_build: &basic_docker_build
77 | name: "Build a really basic docker image"
78 | command: |
79 | dockerfile=Dockerfile
80 | echo "FROM alpine:latest" > $dockerfile
81 | echo "RUN echo hello" >> $dockerfile
82 | docker build -f $dockerfile --tag throwaway:$CIRCLE_BUILD_NUM .
83 | docker run --rm throwaway:$CIRCLE_BUILD_NUM
84 |
85 | contexts_defaults: &contexts_defaults
86 | docker: [{image: 'alpine:latest'}]
87 | working_directory: /a/contexts/test
88 |
89 | workspaces_defaults: &workspaces_defaults
90 | docker: [{image: 'cimg/node:current'}]
91 | working_directory: ~/foo/bar
92 |
93 | jobs:
94 | env-subst-example:
95 | executor: docker
96 | environment:
97 | foo: bar
98 | alpha: beta
99 | steps:
100 | - checkout
101 | - run:
102 | command: echo 'the value of foo is "$foo" and next to alpha is "$alpha"' > template.tmpl
103 | - run:
104 | command: cat template.tmpl
105 | - run:
106 | command: circleci env subst < template.tmpl > output.txt
107 | - run:
108 | command: grep "bar" output.txt | grep "beta"
109 |
110 | # job definition for verifying the resource_class per each executor_type
111 | executor_resource_class:
112 | parameters:
113 | resource_class:
114 | type: string
115 | default: "small"
116 | executor_type:
117 | type: string
118 | default: "docker"
119 | executor:
120 | name: <>
121 | resource_class: <>
122 | steps:
123 | - verify_resource_class:
124 | resource_class: <>
125 | executor_type: <>
126 |
127 | check_if_environment_is_aws:
128 | executor: docker
129 | steps:
130 | - run:
131 | name: Verify AWS Environment
132 | command: |
133 | if [ -z "${CIRCLE_CLOUD_PROVIDER}" ];then
134 | echo "You must provide the CIRCLE_CLOUD_PROVIDER environment variable in project settings for this job to run.";
135 | exit 1;
136 | fi
137 | if [[ "${CIRCLE_CLOUD_PROVIDER}" != 'aws' ]]; then
138 | echo "This installation is not in an AWS environment. Cancelling downstream workflow.";
139 | curl -X POST "${CIRCLE_HOSTNAME%/}/api/v2/workflow/$CIRCLE_WORKFLOW_ID/cancel?circle-token=$CIRCLE_TOKEN"
140 | fi
141 |
142 | check_if_environment_is_gcp:
143 | executor: docker
144 | steps:
145 | - run:
146 | name: Verify GCP Environment
147 | command: |
148 | if [ -z "${CIRCLE_CLOUD_PROVIDER}" ];then
149 | echo "You must provide the CIRCLE_CLOUD_PROVIDER environment variable in project settings for this job to run.";
150 | exit 1;
151 | fi
152 | if [[ "${CIRCLE_CLOUD_PROVIDER}" != 'gcp' ]]; then
153 | echo "This installation is not in an GCP environment. Cancelling downstream workflow.";
154 | curl -X POST "${CIRCLE_HOSTNAME%/}/api/v2/workflow/$CIRCLE_WORKFLOW_ID/cancel?circle-token=$CIRCLE_TOKEN"
155 | fi
156 |
157 | check_if_windows_is_enabled:
158 | executor: docker
159 | steps:
160 | - run:
161 | name: Verify if Windows enabled env flag is set to true
162 | command: |
163 | if [[ "${CIRCLE_WINDOWS_EXECUTOR}" != "true" ]]; then
164 | echo "Windows executor environment flag is not set to \"true\". Cancelling downstream workflow."
165 | curl -X POST "${CIRCLE_HOSTNAME%/}/api/v2/workflow/$CIRCLE_WORKFLOW_ID/cancel?circle-token=$CIRCLE_TOKEN"
166 | fi
167 |
168 | # vm jobs
169 | machine:
170 | machine: true
171 | steps:
172 | - run:
173 | <<: *basic_docker_build
174 | - run: |
175 | echo $SLEEP
176 | date
177 | sleep $SLEEP
178 | date
179 | echo 'Done sleeping.'
180 | environment:
181 | SLEEP: 1
182 |
183 | android_machine:
184 | machine:
185 | image: android-default
186 | steps:
187 | - run:
188 | <<: *basic_docker_build
189 | - run: |
190 | echo $SLEEP
191 | date
192 | sleep $SLEEP
193 | date
194 | echo 'Done sleeping.'
195 | environment:
196 | SLEEP: 1
197 |
198 | aws_arm:
199 | machine:
200 | image: arm-default
201 | resource_class: arm.medium
202 | steps:
203 | - run:
204 | <<: *basic_docker_build
205 | - run: |
206 | echo $SLEEP
207 | date
208 | sleep $SLEEP
209 | date
210 | echo 'Done sleeping.'
211 | environment:
212 | SLEEP: 1
213 |
214 | remote_docker:
215 | <<: *remote_docker_defaults
216 | steps:
217 | - run: which docker
218 | - run: docker -v
219 | - setup_remote_docker
220 | - run:
221 | <<: *basic_docker_build
222 | - run: docker version
223 |
224 | docker_layer_caching:
225 | <<: *remote_docker_defaults
226 | steps:
227 | - run: which docker
228 | - run: docker -v
229 | - setup_remote_docker:
230 | docker_layer_caching: true
231 | - run:
232 | <<: *basic_docker_build
233 | - run: docker version
234 |
235 | machine_dlc:
236 | machine:
237 | docker_layer_caching: true
238 | steps:
239 | - run: which docker
240 | - run: docker -v
241 | - run:
242 | <<: *basic_docker_build
243 | - run: docker version
244 |
245 | # feature jobs
246 | contexts:
247 | <<: *contexts_defaults
248 | steps:
249 | - run: env | grep CONTEXT_END_TO_END_TEST_VAR
250 |
251 | multi-contexts:
252 | <<: *contexts_defaults
253 | steps:
254 | - run: env | grep MULTI_CONTEXT_END_TO_END_VAR
255 |
256 | write_workspace:
257 | <<: *workspaces_defaults
258 | steps:
259 | - run: mkdir stuff
260 | - run: echo 5 >./stuff/thing
261 | - persist_to_workspace:
262 | root: .
263 | paths:
264 | - stuff
265 |
266 | read_workspace:
267 | <<: *workspaces_defaults
268 | steps:
269 | - attach_workspace:
270 | at: ./attached
271 | - run: |
272 | if [[ $(< ./attached/stuff/thing) != '5' ]]; then
273 | echo 'Sadness, the value is not what we expected, so the workspace write/read did not work'
274 | exit 1
275 | else
276 | echo 'Yay, value matches'
277 | exit 0
278 | fi
279 |
280 | save_and_restore_cache:
281 | docker:
282 | - image: cimg/python:3.13
283 | working_directory: ~/realitycheck
284 | steps:
285 | - checkout
286 |
287 | - run: mkdir -p stuff
288 | - run: echo 5 >./stuff/thing
289 |
290 | - save_cache:
291 | key: v3-file-cache-{{ .BuildNum }}
292 | paths:
293 | - ./stuff/thing
294 |
295 | - run: rm -rf ./stuff/thing
296 | - run: sleep 5
297 |
298 | - restore_cache:
299 | keys:
300 | - v3-file-cache-{{ .BuildNum }}
301 |
302 | - run: |
303 | if [[ $(< stuff/thing) != '5' ]]; then
304 | exit 1
305 | else
306 | echo "The cache was populated"
307 | fi
308 |
309 | artifacts_test_results:
310 | docker:
311 | - image: python:3.13.7
312 | working_directory: ~/realitycheck
313 | steps:
314 | - checkout
315 |
316 | - run:
317 | name: Creating Dummy Artifacts
318 | command: |
319 | echo "my artifact file" > /tmp/artifact-1.txt;
320 | mkdir /tmp/artifacts;
321 | echo "my artifact files in a dir" > /tmp/artifacts/artifact-2.txt;
322 |
323 | - store_artifacts:
324 | path: /tmp/artifact-1.txt
325 | destination: artifact-file.txt
326 |
327 | - store_artifacts:
328 | path: /tmp/artifacts
329 |
330 | - store_test_results:
331 | path: test-results
332 |
333 | workflows:
334 | docker_resource_class_jobs:
335 | jobs:
336 | - executor_resource_class:
337 | matrix:
338 | parameters:
339 | resource_class: [small, medium, medium+, large, xlarge]
340 | executor_type: [docker] # default, but re-defining for clarity
341 |
342 | windows_resource_class_jobs:
343 | jobs:
344 | - check_if_windows_is_enabled
345 | - executor_resource_class:
346 | matrix:
347 | parameters:
348 | resource_class: [windows.medium, windows.large, windows.xlarge]
349 | executor_type: [windows]
350 | requires:
351 | - check_if_windows_is_enabled
352 |
353 | vm_jobs:
354 | jobs:
355 | - machine
356 | - remote_docker
357 | - docker_layer_caching
358 | - machine_dlc
359 |
360 | aws_jobs:
361 | jobs:
362 | - check_if_environment_is_aws
363 | - aws_arm:
364 | requires:
365 | - check_if_environment_is_aws
366 |
367 | gcp_jobs:
368 | jobs:
369 | - check_if_environment_is_gcp
370 | - android_machine:
371 | requires:
372 | - check_if_environment_is_gcp
373 |
374 | feature_jobs:
375 | jobs:
376 | - save_and_restore_cache
377 | - contexts:
378 | context: org-global
379 | - multi-contexts:
380 | context: individual-local
381 | - write_workspace
382 | - read_workspace:
383 | requires:
384 | - write_workspace
385 | - artifacts_test_results
386 |
387 | other_jobs:
388 | jobs:
389 | - env-subst-example
390 |
--------------------------------------------------------------------------------