├── .github ├── contributing.md ├── issue_template.md └── pull_request_template.md ├── .gitignore ├── LICENSE ├── NOTICE ├── README.md ├── docs ├── deploy-without-helm.md ├── eirini-diego-feature-parity.md ├── scaling-and-ha.md ├── security-guidelines.md ├── security-overview.md └── security-overview.png ├── helm ├── .helmignore ├── Chart.yaml ├── templates │ ├── core │ │ ├── api-configmap.yml │ │ ├── api-deployment.yml │ │ ├── api-rbac.yml │ │ ├── api-service.yml │ │ ├── instance-index-env-injector-configmap.yml │ │ ├── instance-index-env-injector-deployment.yml │ │ ├── instance-index-env-injector-rbac.yml │ │ ├── instance-index-env-injector-service.yml │ │ ├── instance-index-env-injector-webhook.yml │ │ ├── migration-job.yml │ │ ├── migration-rbac.yml │ │ ├── task-reporter-configmap.yml │ │ ├── task-reporter-deployment.yml │ │ └── task-reporter-rbac.yml │ ├── events │ │ ├── event-reporter-configmap.yml │ │ ├── event-reporter-deployment.yml │ │ └── event-reporter-rbac.yml │ └── workloads │ │ ├── app-rbac.yml │ │ ├── core │ │ ├── api-rbac.yml │ │ └── task-reporter-rbac.yml │ │ ├── events │ │ └── event-reporter-rbac.yml │ │ └── namespace.yml └── values.yaml └── scripts ├── assets ├── cf-for-k8s-value-overrides.yml ├── value-overrides.yml └── wiremock.yml ├── cleanup.sh ├── deploy.sh ├── generate-secrets.sh ├── helpers └── print.sh └── render-templates.sh /.github/contributing.md: -------------------------------------------------------------------------------- 1 | # Contributing to Eirini 2 | 3 | Thank you for taking the time to contribute to Eirini. You can find more information about the project at: 4 | 1. [#eirini-dev](https://cloudfoundry.slack.com/messages/eirini-dev) slack channel. 5 | 1. [PivotalTracker](https://www.pivotaltracker.com/n/projects/2172361) show our current backlog and what we have planned for the near future. 6 | 1. [CI pipeline](https://ci.flintstone.cf.cloud.ibm.com/teams/eirini/pipelines/ci) 7 | 1. [Project page](https://www.cloudfoundry.org/project-eirini/) 8 | 9 | # How to contribute 10 | 11 | ## Submitting a bug report 12 | 13 | We use PivotalTracker to track bugs and a story is automatically created when you submit an issue on GitHub. To create the issue please use the following [template](ISSUE_TEMPLATE.md). 14 | 15 | ## Submitting a pull request 16 | 17 | Pull requests are the best way to propose changes to the codebase. When a pull request is submitted, a story is automatically created in the Icebox in our PivotalTracker. 18 | 19 | 1. Sign and submit the appropriate [individual](https://www.cloudfoundry.org/wp-content/uploads/2017/01/CFF_Individual_CLA.pdf) or [corporate](https://www.cloudfoundry.org/wp-content/uploads/2017/01/CFF_Corporate_CLA.pdf) CLA 20 | 1. Fork this project into your GitHub organisation or username 21 | 1. Clone your fork on your local machine 22 | 1. Make sure you are up-to-date with the upstream `develop` branch 23 | 1. Create your feature branch 24 | 1. Commit your changes on your feature branch 25 | 1. [Run unit tests and static checks](#unit-tests--static-checks) 26 | 1. Push your feature branch to your fork 27 | 1. Issue a Pull request against **develop** 28 | 29 | Thanks, 30 | The Eirini team :heart: 31 | 32 | -------------------------------------------------------------------------------- /.github/issue_template.md: -------------------------------------------------------------------------------- 1 | ### Description 2 | 3 | [Fill in] 4 | 5 | ### Steps to reproduce 6 | 7 | [Fill in] 8 | 9 | ### What was expected to happen 10 | 11 | [Fill in] 12 | 13 | ### What actually happened 14 | 15 | [Fill in] 16 | 17 | ### Suggested fix (optional) 18 | 19 | [Fill in] 20 | 21 | ### Additional information (optional) 22 | 23 | [Fill in] 24 | 25 | -------------------------------------------------------------------------------- /.github/pull_request_template.md: -------------------------------------------------------------------------------- 1 | Thanks for contributing to Eirini! In order for your pull request to be accepted, we would like to ask you the following: 2 | 3 | 1. Please base your PR off the `develop` branch. 4 | 1. Describe the change on a conceptual level. How does it work, and why should it exist? Ideally, you contribute a few words to the README, too. 5 | 1. Please provide automated tests (preferred) or instructions for manually testing your change. 6 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cloudfoundry/eirini-release/11a34b7ea3aec29285612a47e96395dc0b0f9d00/.gitignore -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | -------------------------------------------------------------------------------- /NOTICE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2018-Present CloudFoundry.org Foundation, Inc. All Rights Reserved. 2 | 3 | This project is licensed to you under the Apache License, Version 2.0 (the "License"). 4 | You may not use this project except in compliance with the License. 5 | 6 | This project may include a number of subcomponents with separate copyright notices 7 | and license terms. Your use of these subcomponents is subject to the terms and 8 | conditions of the subcomponent's license, as noted in the LICENSE file. 9 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Eirini Release 2 | 3 | This is a `helm` release for Project [Eirini](https://code.cloudfoundry.org/eirini). In a nutshell _Eirini_ is a Kubernetes backend for 4 | Cloud Foundry, made in the effort to decouple Cloud Foundry from Diego, the only current option of a scheduler. It deploys CF apps 5 | to a kube backend, using OCI images and Kube deployments. 6 | 7 | ## Installation 8 | 9 | The following CFAR (Cloud Foundry Application Runtime) distributions deploy CF on top of Kubernetes and bundle Eirini with it: 10 | 11 | - [cf-for-k8s](https://github.com/cloudfoundry/cf-for-k8s) 12 | - [KubeCF](https://github.com/cloudfoundry-incubator/kubecf) 13 | 14 | ## Building the yaml release 15 | 16 | To build the pure yaml files included in our release please run: 17 | 18 | ```shell 19 | ./scripts/render-templates.sh 20 | ``` 21 | 22 | This will produce the yamls for all [eirini components](https://github.com/cloudfoundry-incubator/eirini/tree/65789e8ccb3f80986a34d9679733c53156a8e394#components) in separate directories. The components needed for cf-for-k8s are `core`, `events` and `workloads`. 23 | 24 | ## Security 25 | 26 | ### Security Overview 27 | 28 | Of an overview of how secure Eirini is compared to other popular container runtimes please look at [this table](./docs/security-overview.md) 29 | 30 | ### Securing the Eirini Deployment 31 | 32 | To learn about how you can use Kubernetes security primitives to make your deployment more secure, please take a look at our [Security Guidelines](docs/security-guidelines.md). 33 | 34 | ## Scalability 35 | 36 | As of v1.5.0 a single instance of the eirini deployment can take a sustained load of 90 parallel desire LRP operations. A desire operation takes about 300ms on average when under load. 37 | 38 | In order to better understand this result we have to state some condtitions that we assumed when performing the tests: 39 | 40 | - Performing well under load means that the eirini server will respond in less than 30s. 41 | - The tests were performed directly against the Eirini API (bypassing the cloud controller) and agains a sufficiently large cluster in order to make sure the eirini is the only bottleneck. So these results apply to eirini in isolation. The whole cf system will be as scalable as it's weakest subsystem. 42 | - There results describe the throughput of eirini itself. Our measurements apply from the moment a desire request is placed to the moment a stateful set is created on Kubernetes. These are not scalability results for Kubernetes. 43 | 44 | For details about high availability see [this doc](docs/scaling-and-ha.md). 45 | 46 | ## Differences with Diego 47 | 48 | We are working hard towards feature parity with Diego, but still there are some differences in behaviour 49 | 50 | ### Environment variables 51 | 52 | It is not possible to set environment variables containing `:` to your apps containers because of Kubernetes restrictions. 53 | 54 | ### Docker images running with the root user 55 | 56 | By default Eirini does not allow docker images running with the root user. Diego allows this because the application runs in a separate user namespace, which is not supported in Kubernetes as of now. However, you can configure Eirini to allow such docker images - see [Security Guidelines](docs/security-guidelines.md#application-podsecuritypolicy) for more information. 57 | 58 | ### Task retries and parallelism 59 | 60 | Tasks in Diego are run [at most once](https://github.com/cloudfoundry/diego-notes/blob/926024b/notes/lrp-task-states-and-transitions.md#task-states) and once completed you can determine whether they failed or not. In Eirini we run tasks as Jobs in Kubernetes with both `completions` and `parallelism` set to 1. However, [as per the Kubernetes documentation](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#handling-pod-and-container-failures), there is no guarantee that the task won't be ran more than once. 61 | 62 | ## Troubleshooting 63 | 64 | ### Disk full on blobstore 65 | 66 | If all the CF apps are running, it is safe to delete all files in `/var/vcap/store/shared/cc-droplets/sh/a2/` directory on the `blobstore-0` pod. 67 | To do so, you can run this command: 68 | 69 | ```bash 70 | kubectl exec -n blobstore-0 -c blobstore -- \ 71 | /bin/sh -c 'rm -rf /var/vcap/store/shared/cc-droplets/sh/a2/sha256:*' 72 | ``` 73 | 74 | ## Resources 75 | 76 | - [Eirini Continuous Integration Pipeline](https://jetson.eirini.cf-app.com/) 77 | -------------------------------------------------------------------------------- /docs/deploy-without-helm.md: -------------------------------------------------------------------------------- 1 | # Deploy without Helm 2 | 3 | The deployment YAML files are found in the `eirini.tgz` asset of the latest [eirini release](https://github.com/cloudfoundry-incubator/eirini-release/releases). 4 | 5 | If you require set of deployment YAML files from the latest passing eirini build, you can `git checkout` the master branch of [eirini-release](https://github.com/cloudfoundry-incubator/eirini-release) and generate the files using a script: 6 | 7 | ```sh 8 | cd 9 | ./scripts/render-templates.sh 10 | ``` 11 | 12 | ## Components 13 | 14 | Throughout these deployment YAML files, the eirini components are configured to run in the `cf-system` namespace and to deploy LRPs and Tasks to the `cf-workloads` namespace. 15 | These namespaces should be created prior to applying the deployment YAML. 16 | 17 | ### Core Components 18 | 19 | The core eirini components consist of: 20 | 21 | - eirini-api (the REST interface that CloudController uses to communicate with Eirini) 22 | - instance-env-injector (a mutating webhook that injects the `CF_INSTANCE_INDEX` env variable into LRP pods) 23 | - task-reporter (calls back to CloudController on the status of completed tasks) 24 | - event-reporter (notifies CloudController of LRP crashes). 25 | This is found in the events directory. 26 | 27 | ## Workloads 28 | 29 | The `workloads` directory contains minimal RBAC configuration required in the workloads namespace. 30 | It provides a Service Account and associated Pod Security Policy for running LRPs. 31 | It also gives appropriate role permissions to eirini components that need to interact with resources in the workloads namespace. 32 | 33 | ## Configuration 34 | 35 | ### Config Maps 36 | 37 | Components are configured using the various `*-configmap.yml` files provided: 38 | 39 | | Component | Configmap YAML | 40 | | -------------- | ------------------------------------- | 41 | | Eirini API | `core/api-configmap.yml` | 42 | | Task Reporter | `core/task-reporter-configmap.yml` | 43 | | Event Reporter | `events/event-reporter-configmap.yml` | 44 | 45 | Each configmap is documented describing the options. 46 | Where a certain configuration requires changes to other file, this is noted there. 47 | 48 | ### Secrets 49 | 50 | Eirini depends on the following secrets, which must be named and constructed as follows: 51 | 52 | - `capi-tls` (optional when `cc_tls_disabled` is set to true in the component's configmap) 53 | 54 | - `tls.crt`: client certificate used for mTLS 55 | - `tls.key`: key for client certificate 56 | - `tls.ca`: CA used to validate CAPI's server certificate 57 | 58 | - `eirini-certs` (optional when `serve_plaintext` is set to true in the API configmap) 59 | 60 | - `tls.crt`: server certificate 61 | - `tls.key`: key for server certificate 62 | - `tls.ca`: CA used to validate client certificates 63 | 64 | - `eirini-instance-index-env-injector-certs` (mandatory - required by the mutating webhook configuration) 65 | 66 | - `tls.crt`: server certificate 67 | - `tls.key`: key for server certificate 68 | - `tls.ca`: CA used to validate injector webhook's server certificate 69 | 70 | ## Deployment 71 | 72 | You can create the Eirini objects using `kubectl`. 73 | First extract the `eirini.tgz` tarball (or run the `render-templates.sh` script). 74 | Then run: 75 | 76 | ```bash 77 | kubectl apply --recursive=true -f 78 | ``` 79 | 80 | Wait for all pods in the `cf-system` namespace to be in the RUNNING state. 81 | That's it! 82 | -------------------------------------------------------------------------------- /docs/eirini-diego-feature-parity.md: -------------------------------------------------------------------------------- 1 | # Eirini - Diego feature parity 2 | 3 | | Feature | Description | Link | CATs | Diego | Eirini | Notes | 4 | | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | :--: | :---: | :----: | --------------------------------------------------------------------------------------------------------- | 5 | | tcp routing | Allows TCP connections to apps | | ✓ | ✓ | ✗ | | 6 | | CPU throttling | Throttle cpu of badly behaving apps | https://www.cloudfoundry.org/blog/better-way-split-cake-cpu-entitlements/ | ✓ | ✓ | ✗ | | 7 | | cf push (with docker image) / USER as string | Garden automatically maps the user to a user ID | | ✗ | ✓ | ✗ | | 8 | | isolation segments | Isolation segments provide dedicated pools of resources where you can deploy apps and isolate workloads. | https://docs.cloudfoundry.org/adminguide/isolation-segments.html | ✓ | ✓ | ✗ | | 9 | | docker private registry push | Deploy docker images from private registries | https://docs.cloudfoundry.org/devguide/deploy-apps/push-docker.html#private-repo | ✓ | ✓ | ✓ | This is excluded in kubecf CATs, but should work once tasks are updated to latest eirini functionality | 10 | | route services | | https://docs.cloudfoundry.org/services/route-services.html | ✓ | ✓ | ✗ | Does not work in kubecf but maybe that's a setup issue in kubecf (e.g. route_services_secret missing ?) | 11 | | ssh | cf ssh into running app instances | https://docs.cloudfoundry.org/devguide/deploy-apps/ssh-apps.html | ✓ | ✓ | ✗ | kubecf implements this with [eirinix hook](https://github.com/cloudfoundry-incubator/eirini-ssh) | 12 | | routing isolation segments | Operators can configure and manage routing for isolation segments in Cloud Foundry (CF) | https://docs.cloudfoundry.org/adminguide/routing-is.html | ✓ | ✓ | ✗ | | 13 | | security groups | collection of egress rules that specify the protocols, ports, and IP address ranges where app or task instances send traffic | https://docs.cloudfoundry.org/concepts/asg.html | ✓ | ✓ | ✗ | Does not work in kubecf, but we think it's not a responsibility of Eirini | 14 | | tasks | run a one-off task for a running app | https://docs.cloudfoundry.org/devguide/using-tasks.html | ✓ | ✓ | ? | running tasks fail in kubecf v2.6.1, but we think that is because they are using an older Eirini (v1.8.0) | 15 | | windows support | run containers on windows nodes | | ✓ | ✓ | ✗ | This is not supported in the whole cf-4-k8s deployment, it's not just an Eirini feature | 16 | | sidecars | run additional app processes in an adjacent container | https://docs.cloudfoundry.org/devguide/sidecars.html | ✗ | ✓ | ✗ | | 17 | | multi-process apps | use a single command to push apps that run multiple processes, such as a web app that has a UI process and a worker process | https://docs.cloudfoundry.org/devguide/multiple-processes.html | ✗ | ✓ | ✗ | | 18 | | v3 | CF v3 API | https://docs.cloudfoundry.org/devguide/v3-commands.html | ✓ | ✓ | ? | We think this works but have no way of testing currently | 19 | | non-numeric "USER" in Docker image | `cf push myapp --docker-image myrepo/myapp:mytag` where the Docker image has USER specified as a string | https://docs.cloudfoundry.org/devguide/deploy-apps/push-docker.html | ✗ | ✓ | ✗ | Kubernetes doesn't support User namespaces (while Docker and Garden do). This means, whatever UID you have in your container is gonna match the user of the same UID on the host. For that reason, Eirini enforces RunAsNonRoot which means, Kubernetes will reject any image that either doesn't specify a USER (since root is the default), or specifies a user as a string (since Kubernetes can't check whether that user is root or not). The workaround is to specify the USER using the numeric UID in the Dockerfile. | 20 | -------------------------------------------------------------------------------- /docs/scaling-and-ha.md: -------------------------------------------------------------------------------- 1 | ## Scaling and High Availablity Configuration 2 | 3 | There is a mixture of high availability and scaling available in eirini components. 4 | Components responsible for reacting to configuration and events cannot be scaled, as such reactions should not be duplicated. 5 | These use leadership elections to ensure only a single instance is active at a given time. 6 | Others, which provide services, can be scaled horizontally providing both increased bandwidth and availability. 7 | 8 | Scaling is configured using the replicas property of the deployment spec in both cases. 9 | 10 | | Eirini Component | Scaling type | Notes | 11 | | --------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | 12 | | REST API | Horizontal | Request to the API are distributed round robin to the instances by default (this is based on the Kubernetes Service) | 13 | | Event Reporter | HA only | Leadership election in the controller runtime ensures only a single instance handles a given crash event | 14 | | Task Reporter | HA only | Leadership election in the controller runtime ensures only a single instance handles a given task completion event | 15 | | Instance Index Env Injector | Horizontal | Request to the hook service are distributed round robin to the instances by default (this is based on the Kubernetes Service). The registration job should not be scaled | 16 | | Metrics Collector | None | Leadership election is required but not yet implemented, so do not scale this component | 17 | -------------------------------------------------------------------------------- /docs/security-guidelines.md: -------------------------------------------------------------------------------- 1 | ## Service Account 2 | 3 | When an app is pushed with Eirini, the pods are assigned the default Service Account in `app_namespace`. By default, when the cluster is deployed with `RBAC` authentication method, that Service Account should not have any read/write permissions to the Kubernetes API. Since `RBAC` is preferred to `ABAC`, we recommend using the former. 4 | 5 | ## Network policies 6 | 7 | Apps pushed by Eirini currently cannot be accessed directly from another app container. This is accomplished by creating a [NetworkPolicy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) resource in the namespace in which Eirini deploys apps. 8 | 9 | In order to use network policies in your cluster, you must use a compatible container network plug-in, otherwise creating a `NetworkPolicy` resource will have no effect. 10 | 11 | Both [IKS](https://cloud.ibm.com/docs/containers?topic=containers-network_policies) (is automatically setup) and [GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/network-policy#enabling_network_policy_enforcement) (has to be enabled) support a network plug-in called [Calico](https://www.projectcalico.org/), which supports defining network policies. 12 | 13 | For other implementations of the Kubernetes networking model, take a look [here](https://kubernetes.io/docs/concepts/cluster-administration/networking/#how-to-implement-the-kubernetes-networking-model). Keep in mind that not all implementations support defining network polcies (e.g. Flannel). For a more detailed comparison between different plugins, take a look [here](https://docs.google.com/spreadsheets/d/1qCOlor16Wp5mHd6MQxB5gUEQILnijyDLIExEpqmee2k/edit#gid=0) (not maintained by us). 14 | 15 | ## Application PodSecurityPolicy 16 | 17 | _Note: For this section, ensure that PodSecurityPolicy support is enabled on your cluster. This is platform specific (e.g. in GKE this is not enabled by default)._ 18 | 19 | By default, Eirini attaches a specific Service Account to all application pods. This service account permissions can be found [here](../helm/eirini/templates/app-pod-security-policy.yaml) and they don't allow pods to be run with the root user. You can relax this limitation by doing the following steps: 20 | 21 | 1. Set the `allow_run_image_as_root` property in the Eirini ConfigMap to `true` by executing 22 | 23 | ``` 24 | kubectl edit configmap eirini -n 25 | ``` 26 | 27 | 2. Restart the Eirini pod so the new change can be applied. 28 | 29 | ``` 30 | kubectl delete pod -n 31 | ``` 32 | 33 | 3. Apply a more relaxed PodSecurityPolicy in the namespace in which eirini schedules applications. 34 | Example of a relaxed PSP 35 | 36 | ``` 37 | apiVersion: policy/v1beta1 38 | kind: PodSecurityPolicy 39 | metadata: 40 | annotations: 41 | seccomp.security.alpha.kubernetes.io/allowedProfileNames: runtime/default 42 | seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default 43 | name: eirini-app-privileged-psp 44 | namespace: eirini 45 | spec: 46 | allowPrivilegeEscalation: false 47 | fsGroup: 48 | rule: RunAsAny 49 | runAsUser: 50 | rule: RunAsAny 51 | seLinux: 52 | rule: RunAsAny 53 | supplementalGroups: 54 | rule: RunAsAny 55 | ``` 56 | 57 | 4. Add the new privileged PSP to the default Service Account role by executing: 58 | 59 | ``` 60 | kubectl patch -n eirini role eirini-app-role --type='json' -p '[{"op":"add","path":"/rules/0/resourceNames/-","value":"eirini-app-privileged-psp"}]' 61 | ``` 62 | 63 | ### Securing Kubernetes API Endpoint 64 | 65 | The Kubernetes API is available in all pods by default at `https://kubernetes.default`. Eirini does not mount 66 | service account credentials to the pod and uses default service account in the namespace. This prevents Eirini pods from using Kubernetes API. 67 | To completely disallow access to this from application instances, you'd need to apply this network policy: 68 | 69 | ```yaml 70 | apiVersion: extensions/v1beta1 71 | kind: NetworkPolicy 72 | metadata: 73 | name: eirini-egress-policy 74 | namespace: eirini 75 | spec: 76 | egress: 77 | - to: 78 | - ipBlock: 79 | cidr: 0.0.0.0/0 80 | except: 81 | - /32 82 | podSelector: {} 83 | policyTypes: 84 | - Egress 85 | ``` 86 | 87 | You can get IP address of the master by running `kubectl get endpoints` command. If there are multiple Kubernetes API nodes, IP address 88 | of each of them would need to be specified in the `except` array. 89 | -------------------------------------------------------------------------------- /docs/security-overview.md: -------------------------------------------------------------------------------- 1 | # Security Overview 2 | 3 | The following table provides an overview of container security mechanisms across various container systems. 4 | Table last updated 12/08/19. [Link to spreadsheet](https://docs.google.com/spreadsheets/d/1Rwg-C5B4yhyqUrKe_9ozhpiPKpo0Ck6RYBIKC0uzzvQ/edit?usp=sharing) 5 | 6 | ![security overview](security-overview.png) 7 | 8 | * \* Possible with [mutating webhooks](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#webhook-configuration) 9 | * \*\* https://github.com/kubernetes/kubernetes/pull/64005 10 | * \*\*\* Application is restarted after reaching the limit. The limit is configured globally for every application. 11 | * \*\*\*\* Fewer masked paths than garden/docker (e.g. /proc/scsi) 12 | 13 | * User Namespaces - False, not possible in Kubernetes yet 14 | * Rootless - False, not possible in Kubernetes yet 15 | * Seccomp - True, runtime default is applied 16 | * AppArmor - True, runtime default is applied 17 | * Root Capability Dropping - True, runtime default is applied 18 | * No New Privileges - True, [`allowPrivilegeEscalation` is set to `false`](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) 19 | * Cgroups - True if container processes' access to physical resources restricted by Cgroups 20 | * Disk Quotas - True, using [ephemeral storage limits](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#requests-and-limits-setting-for-local-ephemeral-storage). 21 | * Procfs/Sysfs limits - True, runtime default is applied 22 | * Bridge networking - Depends, see table for further info 23 | * Hypervisor Isolation - True if Kubernetes is deployed [with this runtime](https://github.com/kubernetes/frakti) 24 | * SELinux - False, can be [configured in the pod definition](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#assign-selinux-labels-to-a-container) 25 | 26 | * Table inspired by: https://blog.jessfraz.com/post/containers-security-and-echo-chambers 27 | 28 | ## Environments 29 | 30 | * Cloud Foundry Application Runtime v7.4.0 - Standard deployment on Xenial trusty stemcell 31 | * Kubernetes v1.13.3 - Standard deployment on GCP 32 | -------------------------------------------------------------------------------- /docs/security-overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cloudfoundry/eirini-release/11a34b7ea3aec29285612a47e96395dc0b0f9d00/docs/security-overview.png -------------------------------------------------------------------------------- /helm/.helmignore: -------------------------------------------------------------------------------- 1 | # Patterns to ignore when building packages. 2 | # This supports shell glob matching, relative path matching, and 3 | # negation (prefixed with !). Only one pattern per line. 4 | .DS_Store 5 | # Common VCS dirs 6 | .git/ 7 | .gitignore 8 | .bzr/ 9 | .bzrignore 10 | .hg/ 11 | .hgignore 12 | .svn/ 13 | # Common backup files 14 | *.swp 15 | *.bak 16 | *.tmp 17 | *~ 18 | # Various IDEs 19 | .project 20 | .idea/ 21 | *.tmproj 22 | -------------------------------------------------------------------------------- /helm/Chart.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | appVersion: "0.3.0" 3 | description: A Helm chart for Cloud Foundry Eirini 4 | name: eirini 5 | version: 0.0.0-dev 6 | -------------------------------------------------------------------------------- /helm/templates/core/api-configmap.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: ConfigMap 4 | metadata: 5 | name: eirini-api 6 | namespace: {{ .Release.Namespace }} 7 | data: 8 | api.yml: | 9 | # app_namespace is the namespace used by Eirini to deploy LRPs that do 10 | # not specify their own namespace in the request. 11 | app_namespace: {{ .Values.workloads.default_namespace }} 12 | 13 | # serve_plaintext specifies whether the Eirini REST API should be served 14 | # over plain HTTP or not. 15 | # 16 | # When serve_plaintext is false, which is the default, a secret must be 17 | # provided for the eirini-certs volume in core/api-deployment.yml. This 18 | # must have entries tls.ca, tls.crt and tls.key, for the TLS certificate 19 | # authority, certificate and key respectively, used to serve TLS. 20 | serve_plaintext: {{ .Values.api.serve_plaintext }} 21 | 22 | # plaintext_port is the port used by Eirini to serve its REST API over 23 | # plain HTTP. Required when serve_plaintext is set to true. 24 | plaintext_port: {{ .Values.api.plaintext_port }} 25 | 26 | # tls_port is the port used by Eirini to serve its REST API over HTTPS 27 | # Required when serve_plaintext is set to false or omitted. 28 | tls_port: {{ .Values.api.tls_port }} 29 | 30 | # cc_tls_disabled specifies wether Eirini should communicate to the Cloud 31 | # Controller via HTTPS or not. This should be set to false if TLS is 32 | # handled transparently, e.g. by a service mesh. 33 | # 34 | # When cc_tls_disabled is false, which is the default, a secret must be 35 | # provided for the cc-certs volume in core/api-deployment.yml. This must 36 | # have entries tls.ca, tls.crt and tls.key, for the TLS certificate 37 | # authority, client certificate and key respectively, used for mTLS with 38 | # the Cloud Controller. 39 | cc_tls_disabled: {{ .Values.cc_api.tls_disabled }} 40 | 41 | # application_service_account is name of the service account used by 42 | # running LRPs and tasks. It must match the service account name in 43 | # workloads/app-rbac.yml 44 | application_service_account: eirini 45 | 46 | # registry_secret_name is the name of the secret containing the docker 47 | # credentials to pull LRP images 48 | registry_secret_name: {{ .Values.api.registry_secret_name }} 49 | 50 | # allow_run_image_as_root will allow containers to run as root when set 51 | # to true. As kubernetes does not use user namespaces, this will be the 52 | # same root user as on the kubernetes node, and so is a security concern. 53 | # It should be generally left as false. 54 | allow_run_image_as_root: false 55 | 56 | # unsafe_allow_automount_service_account_token when set to true causes 57 | # Kubernetes to mount the service account token in the LRP and task 58 | # containers. This gives the code running there access to the Kubernetes 59 | # API with the privileges of the application service account. 60 | # 61 | # This should generally be left as the default false unless there is a 62 | # good reason and the implications are understood. It is required when 63 | # running cf-for-k8s in a kind cluster, for example. 64 | unsafe_allow_automount_service_account_token: {{ .Values.api.unsafe_allow_automount_service_account_token }} 65 | -------------------------------------------------------------------------------- /helm/templates/core/api-deployment.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: apps/v1 3 | kind: Deployment 4 | metadata: 5 | name: eirini-api 6 | namespace: {{ .Release.Namespace }} 7 | spec: 8 | selector: 9 | matchLabels: 10 | name: eirini-api 11 | template: 12 | metadata: 13 | labels: 14 | name: eirini-api 15 | spec: 16 | dnsPolicy: ClusterFirst 17 | serviceAccountName: eirini-api 18 | volumes: 19 | - name: config-map 20 | configMap: 21 | name: eirini-api 22 | - name: cc-certs 23 | # The secret below is required when api.cc_tls_disabled is false in 24 | # the core/api-configmap.yml. It must have keys tls.ca, tls.crt and 25 | # tls.key for the certificate authority, client certificate and 26 | # private key respectively, used for mTLS with the Cloud Controller. 27 | secret: 28 | secretName: {{ .Values.cc_api.tls_secret_name }} 29 | optional: true 30 | - name: eirini-certs 31 | # The secret below is required when api.serve_plaintext is false in 32 | # the core/api-configmap.yml. It must have keys tls.ca, tls.crt and 33 | # tls.key for the certificate authority, client certificate and 34 | # private key respectively, used to serve the REST API on TLS. 35 | secret: 36 | secretName: {{ .Values.api.tls_secret_name }} 37 | optional: true 38 | securityContext: 39 | runAsNonRoot: true 40 | containers: 41 | - name: api 42 | image: {{ .Values.images.api }} 43 | args: [ "--config", "/etc/eirini/config/api.yml" ] 44 | imagePullPolicy: IfNotPresent 45 | volumeMounts: 46 | - name: config-map 47 | mountPath: /etc/eirini/config 48 | - name: eirini-certs 49 | mountPath: /etc/eirini/certs 50 | - name: cc-certs 51 | mountPath: /etc/cf-api/certs 52 | ports: 53 | {{- if .Values.api.serve_plaintext }} 54 | - containerPort: {{ .Values.api.plaintext_port }} 55 | name: http 56 | {{- else }} 57 | - containerPort: {{ .Values.api.tls_port }} 58 | name: https 59 | {{- end }} 60 | resources: 61 | requests: 62 | cpu: 20m 63 | memory: 20Mi 64 | limits: 65 | cpu: 100m 66 | memory: 100Mi 67 | -------------------------------------------------------------------------------- /helm/templates/core/api-rbac.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: ServiceAccount 4 | metadata: 5 | name: eirini-api 6 | namespace: {{ .Release.Namespace }} 7 | 8 | --- 9 | apiVersion: rbac.authorization.k8s.io/v1 10 | kind: Role 11 | metadata: 12 | name: eirini-api-role 13 | namespace: {{ .Release.Namespace }} 14 | rules: 15 | - apiGroups: 16 | - policy 17 | resources: 18 | - podsecuritypolicies 19 | verbs: 20 | - use 21 | resourceNames: 22 | - eirini-api 23 | 24 | --- 25 | apiVersion: rbac.authorization.k8s.io/v1 26 | kind: RoleBinding 27 | metadata: 28 | name: eirini-api-rolebinding 29 | namespace: {{ .Release.Namespace }} 30 | roleRef: 31 | kind: Role 32 | name: eirini-api-role 33 | apiGroup: rbac.authorization.k8s.io 34 | subjects: 35 | - kind: ServiceAccount 36 | name: eirini-api 37 | namespace: {{ .Release.Namespace }} 38 | 39 | --- 40 | apiVersion: policy/v1beta1 41 | kind: PodSecurityPolicy 42 | metadata: 43 | name: eirini-api 44 | annotations: 45 | seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'runtime/default' 46 | seccomp.security.alpha.kubernetes.io/defaultProfileName: 'runtime/default' 47 | spec: 48 | privileged: false 49 | allowPrivilegeEscalation: false 50 | requiredDropCapabilities: 51 | - ALL 52 | volumes: 53 | - 'configMap' 54 | - 'emptyDir' 55 | - 'projected' 56 | - 'secret' 57 | - 'downwardAPI' 58 | hostNetwork: false 59 | hostIPC: false 60 | hostPID: false 61 | runAsUser: 62 | rule: 'MustRunAsNonRoot' 63 | seLinux: 64 | # This policy assumes the nodes are using AppArmor rather than SELinux. 65 | rule: 'RunAsAny' 66 | supplementalGroups: 67 | rule: 'MustRunAs' 68 | ranges: 69 | # Forbid adding the root group. 70 | - min: 1 71 | max: 65535 72 | fsGroup: 73 | rule: 'MustRunAs' 74 | ranges: 75 | # Forbid adding the root group. 76 | - min: 1 77 | max: 65535 78 | readOnlyRootFilesystem: false 79 | 80 | --- 81 | apiVersion: rbac.authorization.k8s.io/v1 82 | kind: ClusterRole 83 | metadata: 84 | name: eirini-cluster-role 85 | rules: 86 | - apiGroups: 87 | - batch 88 | resources: 89 | - jobs 90 | verbs: 91 | - list 92 | - apiGroups: 93 | - apps 94 | resources: 95 | - statefulsets 96 | verbs: 97 | - list 98 | - apiGroups: 99 | - "" 100 | resources: 101 | - events 102 | verbs: 103 | - list 104 | - apiGroups: 105 | - "" 106 | resources: 107 | - pods 108 | verbs: 109 | - list 110 | --- 111 | # Bind to the default service account 112 | apiVersion: rbac.authorization.k8s.io/v1 113 | kind: ClusterRoleBinding 114 | metadata: 115 | name: eirini-cluster-rolebinding 116 | roleRef: 117 | kind: ClusterRole 118 | name: eirini-cluster-role 119 | apiGroup: rbac.authorization.k8s.io 120 | subjects: 121 | - kind: ServiceAccount 122 | name: eirini-api 123 | namespace: {{ .Release.Namespace }} 124 | 125 | -------------------------------------------------------------------------------- /helm/templates/core/api-service.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: Service 4 | metadata: 5 | name: eirini-api 6 | namespace: {{ .Release.Namespace }} 7 | spec: 8 | ports: 9 | {{- if .Values.api.serve_plaintext }} 10 | - port: {{ .Values.api.plaintext_port }} 11 | protocol: TCP 12 | name: http 13 | {{- else }} 14 | - port: {{ .Values.api.tls_port }} 15 | protocol: TCP 16 | name: https 17 | {{- end }} 18 | selector: 19 | name: eirini-api 20 | -------------------------------------------------------------------------------- /helm/templates/core/instance-index-env-injector-configmap.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: ConfigMap 4 | metadata: 5 | name: eirini-instance-index-env-injector 6 | namespace: {{ .Release.Namespace }} 7 | data: 8 | instance-index-env-injector.yml: | 9 | # service_port is the port exposed by the instance index env injector 10 | # webhook Service. 11 | service_port: 8443 12 | -------------------------------------------------------------------------------- /helm/templates/core/instance-index-env-injector-deployment.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: apps/v1 3 | kind: Deployment 4 | metadata: 5 | name: eirini-instance-index-env-injector 6 | namespace: {{ .Release.Namespace }} 7 | spec: 8 | selector: 9 | matchLabels: 10 | name: eirini-instance-index-env-injector 11 | template: 12 | metadata: 13 | labels: 14 | name: eirini-instance-index-env-injector 15 | annotations: 16 | sidecar.istio.io/inject: "false" 17 | spec: 18 | serviceAccountName: eirini-instance-index-env-injector 19 | volumes: 20 | - name: config-map-volume 21 | configMap: 22 | name: eirini-instance-index-env-injector 23 | - name: certs 24 | secret: 25 | secretName: {{ .Values.instance_index_env_injector.certs_secret_name }} 26 | securityContext: 27 | runAsNonRoot: true 28 | containers: 29 | - name: instance-index-env-injector 30 | image: {{ .Values.images.instance_index_env_injector }} 31 | args: [ "--config", "/etc/eirini/config/instance-index-env-injector.yml" ] 32 | imagePullPolicy: IfNotPresent 33 | volumeMounts: 34 | - name: config-map-volume 35 | mountPath: /etc/eirini/config 36 | - name: certs 37 | mountPath: /etc/eirini/certs 38 | ports: 39 | - containerPort: 8443 40 | name: https 41 | resources: 42 | requests: 43 | cpu: 20m 44 | memory: 20Mi 45 | limits: 46 | cpu: 100m 47 | memory: 100Mi 48 | 49 | -------------------------------------------------------------------------------- /helm/templates/core/instance-index-env-injector-rbac.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: ServiceAccount 4 | metadata: 5 | name: eirini-instance-index-env-injector 6 | namespace: {{ .Release.Namespace }} 7 | 8 | --- 9 | apiVersion: rbac.authorization.k8s.io/v1 10 | kind: Role 11 | metadata: 12 | name: eirini-instance-index-env-injector-psp 13 | namespace: {{ .Release.Namespace }} 14 | rules: 15 | - apiGroups: 16 | - policy 17 | resources: 18 | - podsecuritypolicies 19 | verbs: 20 | - use 21 | resourceNames: 22 | - eirini-instance-index-env-injector 23 | 24 | --- 25 | apiVersion: rbac.authorization.k8s.io/v1 26 | kind: RoleBinding 27 | metadata: 28 | name: eirini-instance-index-env-injector-psp 29 | namespace: {{ .Release.Namespace }} 30 | roleRef: 31 | kind: Role 32 | name: eirini-instance-index-env-injector-psp 33 | apiGroup: rbac.authorization.k8s.io 34 | subjects: 35 | - kind: ServiceAccount 36 | name: eirini-instance-index-env-injector 37 | namespace: {{ .Release.Namespace }} 38 | 39 | --- 40 | apiVersion: policy/v1beta1 41 | kind: PodSecurityPolicy 42 | metadata: 43 | name: eirini-instance-index-env-injector 44 | annotations: 45 | seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'runtime/default' 46 | seccomp.security.alpha.kubernetes.io/defaultProfileName: 'runtime/default' 47 | spec: 48 | privileged: false 49 | allowPrivilegeEscalation: false 50 | requiredDropCapabilities: 51 | - ALL 52 | volumes: 53 | - 'configMap' 54 | hostNetwork: false 55 | hostIPC: false 56 | hostPID: false 57 | runAsUser: 58 | rule: 'MustRunAsNonRoot' 59 | seLinux: 60 | # This policy assumes the nodes are using AppArmor rather than SELinux. 61 | rule: 'RunAsAny' 62 | supplementalGroups: 63 | rule: 'MustRunAs' 64 | ranges: 65 | # Forbid adding the root group. 66 | - min: 1 67 | max: 65535 68 | fsGroup: 69 | rule: 'MustRunAs' 70 | ranges: 71 | # Forbid adding the root group. 72 | - min: 1 73 | max: 65535 74 | readOnlyRootFilesystem: false 75 | -------------------------------------------------------------------------------- /helm/templates/core/instance-index-env-injector-service.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: Service 4 | metadata: 5 | name: eirini-instance-index-env-injector 6 | namespace: {{ .Release.Namespace }} 7 | spec: 8 | ports: 9 | - port: 443 10 | targetPort: 8443 11 | protocol: TCP 12 | name: https 13 | selector: 14 | name: eirini-instance-index-env-injector 15 | 16 | -------------------------------------------------------------------------------- /helm/templates/core/instance-index-env-injector-webhook.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: admissionregistration.k8s.io/v1 3 | kind: MutatingWebhookConfiguration 4 | metadata: 5 | name: eirini-x-mutating-hook 6 | webhooks: 7 | - name: instance-index-env-injector.eirini.cloudfoundry.org 8 | objectSelector: 9 | matchLabels: 10 | cloudfoundry.org/source_type: APP 11 | rules: 12 | - apiGroups: [""] 13 | apiVersions: ["v1"] 14 | operations: ["CREATE"] 15 | resources: ["pods"] 16 | scope: "Namespaced" 17 | clientConfig: 18 | service: 19 | namespace: {{ .Release.Namespace }} 20 | name: eirini-instance-index-env-injector 21 | caBundle: {{ .Values.webhook_ca_bundle }} 22 | admissionReviewVersions: ["v1beta1"] 23 | sideEffects: None 24 | timeoutSeconds: 10 25 | 26 | -------------------------------------------------------------------------------- /helm/templates/core/migration-job.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: batch/v1 3 | kind: Job 4 | metadata: 5 | name: eirini-app-migration 6 | namespace: {{ .Release.Namespace }} 7 | annotations: 8 | kapp.k14s.io/update-strategy: fallback-on-replace 9 | spec: 10 | backoffLimit: 4 11 | template: 12 | metadata: 13 | labels: 14 | name: eirini-app-migration 15 | annotations: 16 | sidecar.istio.io/inject: "false" 17 | spec: 18 | serviceAccountName: eirini-migration 19 | securityContext: 20 | runAsNonRoot: true 21 | containers: 22 | - name: migration 23 | image: {{ .Values.images.migration }} 24 | imagePullPolicy: IfNotPresent 25 | resources: 26 | requests: 27 | cpu: 20m 28 | memory: 20Mi 29 | limits: 30 | cpu: 100m 31 | memory: 100Mi 32 | restartPolicy: OnFailure 33 | -------------------------------------------------------------------------------- /helm/templates/core/migration-rbac.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: ServiceAccount 4 | metadata: 5 | name: eirini-migration 6 | namespace: {{ .Release.Namespace }} 7 | 8 | --- 9 | apiVersion: rbac.authorization.k8s.io/v1 10 | kind: Role 11 | metadata: 12 | name: eirini-migration-role 13 | namespace: {{ .Release.Namespace }} 14 | rules: 15 | - apiGroups: 16 | - policy 17 | resources: 18 | - podsecuritypolicies 19 | verbs: 20 | - use 21 | resourceNames: 22 | - eirini-migration 23 | 24 | --- 25 | apiVersion: rbac.authorization.k8s.io/v1 26 | kind: RoleBinding 27 | metadata: 28 | name: eirini-migration-rolebinding 29 | namespace: {{ .Release.Namespace }} 30 | roleRef: 31 | kind: Role 32 | name: eirini-migration-role 33 | apiGroup: rbac.authorization.k8s.io 34 | subjects: 35 | - kind: ServiceAccount 36 | name: eirini-migration 37 | namespace: {{ .Release.Namespace }} 38 | 39 | --- 40 | apiVersion: policy/v1beta1 41 | kind: PodSecurityPolicy 42 | metadata: 43 | name: eirini-migration 44 | annotations: 45 | seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'runtime/default' 46 | seccomp.security.alpha.kubernetes.io/defaultProfileName: 'runtime/default' 47 | spec: 48 | privileged: false 49 | allowPrivilegeEscalation: false 50 | requiredDropCapabilities: 51 | - ALL 52 | volumes: 53 | - 'configMap' 54 | - 'emptyDir' 55 | - 'projected' 56 | - 'secret' 57 | - 'downwardAPI' 58 | hostNetwork: false 59 | hostIPC: false 60 | hostPID: false 61 | runAsUser: 62 | rule: 'MustRunAsNonRoot' 63 | seLinux: 64 | # This policy assumes the nodes are using AppArmor rather than SELinux. 65 | rule: 'RunAsAny' 66 | supplementalGroups: 67 | rule: 'MustRunAs' 68 | ranges: 69 | # Forbid adding the root group. 70 | - min: 1 71 | max: 65535 72 | fsGroup: 73 | rule: 'MustRunAs' 74 | ranges: 75 | # Forbid adding the root group. 76 | - min: 1 77 | max: 65535 78 | readOnlyRootFilesystem: false 79 | 80 | --- 81 | apiVersion: rbac.authorization.k8s.io/v1 82 | kind: ClusterRole 83 | metadata: 84 | name: eirini-migration-cluster-role 85 | rules: 86 | - apiGroups: 87 | - apps 88 | resources: 89 | - statefulsets 90 | verbs: 91 | - list 92 | - patch 93 | - apiGroups: 94 | - batch 95 | resources: 96 | - jobs 97 | verbs: 98 | - list 99 | - patch 100 | - apiGroups: 101 | - policy 102 | resources: 103 | - poddisruptionbudgets 104 | verbs: 105 | - get 106 | - patch 107 | - apiGroups: 108 | - "" 109 | resources: 110 | - secrets 111 | verbs: 112 | - get 113 | - patch 114 | --- 115 | # Bind to the default service account 116 | apiVersion: rbac.authorization.k8s.io/v1 117 | kind: ClusterRoleBinding 118 | metadata: 119 | name: eirini-migration-cluster-rolebinding 120 | roleRef: 121 | kind: ClusterRole 122 | name: eirini-migration-cluster-role 123 | apiGroup: rbac.authorization.k8s.io 124 | subjects: 125 | - kind: ServiceAccount 126 | name: eirini-migration 127 | namespace: {{ .Release.Namespace }} 128 | 129 | -------------------------------------------------------------------------------- /helm/templates/core/task-reporter-configmap.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: ConfigMap 4 | metadata: 5 | name: eirini-task-reporter 6 | namespace: {{ .Release.Namespace }} 7 | data: 8 | task-reporter.yml: | 9 | # cc_tls_disabled specifies wether Eirini should communicate to the Cloud 10 | # Controller via HTTPS or not. This should be set to false if TLS is 11 | # handled transparently, e.g. by a service mesh. 12 | # 13 | # When cc_tls_disabled is false, which is the default, a secret must be 14 | # provided for the cc-certs volume in core/api-deployment.yml. This must 15 | # have entries tls.ca, tls.crt and tls.key, for the TLS certificate 16 | # authority, client certificate and key respectively, used for mTLS with 17 | # the Cloud Controller. 18 | cc_tls_disabled: {{ .Values.cc_api.tls_disabled }} 19 | 20 | # completion_callback_retry_limit is the number of times Eirini will retry 21 | # to call the Cloud Controller completion callback in case the Cloud 22 | # Controller is unreachable. 23 | completion_callback_retry_limit: {{ .Values.tasks.completion_callback_retry_limit }} 24 | 25 | # ttl_seconds is the number of seconds Eirini will wait before deleting the 26 | # Job associated to a completed Task. 27 | ttl_seconds: {{ .Values.tasks.ttl_seconds }} 28 | -------------------------------------------------------------------------------- /helm/templates/core/task-reporter-deployment.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: apps/v1 3 | kind: Deployment 4 | metadata: 5 | name: eirini-task-reporter 6 | namespace: {{ .Release.Namespace }} 7 | spec: 8 | selector: 9 | matchLabels: 10 | name: eirini-task-reporter 11 | replicas: 2 12 | template: 13 | metadata: 14 | labels: 15 | name: eirini-task-reporter 16 | spec: 17 | dnsPolicy: ClusterFirst 18 | serviceAccountName: eirini-task-reporter 19 | securityContext: 20 | runAsNonRoot: true 21 | containers: 22 | - name: task-reporter 23 | image: {{ .Values.images.task_reporter }} 24 | args: [ "--config", "/etc/eirini/config/task-reporter.yml" ] 25 | imagePullPolicy: IfNotPresent 26 | resources: 27 | requests: 28 | cpu: 15m 29 | memory: 15Mi 30 | limits: 31 | cpu: 100m 32 | memory: 100Mi 33 | volumeMounts: 34 | - name: config-map-volume 35 | mountPath: /etc/eirini/config 36 | - name: cc-certs 37 | mountPath: /etc/cf-api/certs 38 | volumes: 39 | - name: config-map-volume 40 | configMap: 41 | name: eirini-task-reporter 42 | items: 43 | - key: task-reporter.yml 44 | path: task-reporter.yml 45 | - name: cc-certs 46 | secret: 47 | secretName: {{ .Values.cc_api.tls_secret_name }} 48 | optional: true 49 | -------------------------------------------------------------------------------- /helm/templates/core/task-reporter-rbac.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: ServiceAccount 4 | metadata: 5 | name: eirini-task-reporter 6 | namespace: {{ .Release.Namespace }} 7 | 8 | --- 9 | apiVersion: rbac.authorization.k8s.io/v1 10 | kind: Role 11 | metadata: 12 | name: eirini-task-reporter-psp 13 | namespace: {{ .Release.Namespace }} 14 | rules: 15 | - apiGroups: 16 | - policy 17 | resources: 18 | - podsecuritypolicies 19 | verbs: 20 | - use 21 | resourceNames: 22 | - eirini-task-reporter 23 | - apiGroups: 24 | - "" 25 | resources: 26 | - configmaps 27 | verbs: 28 | - get 29 | - create 30 | - update 31 | - apiGroups: 32 | - "coordination.k8s.io" 33 | resources: 34 | - leases 35 | verbs: 36 | - get 37 | - create 38 | - update 39 | - apiGroups: 40 | - "" 41 | resources: 42 | - events 43 | verbs: 44 | - create 45 | 46 | --- 47 | apiVersion: rbac.authorization.k8s.io/v1 48 | kind: RoleBinding 49 | metadata: 50 | name: eirini-task-reporter-psp 51 | namespace: {{ .Release.Namespace }} 52 | roleRef: 53 | kind: Role 54 | name: eirini-task-reporter-psp 55 | apiGroup: rbac.authorization.k8s.io 56 | subjects: 57 | - kind: ServiceAccount 58 | name: eirini-task-reporter 59 | namespace: {{ .Release.Namespace }} 60 | 61 | --- 62 | apiVersion: policy/v1beta1 63 | kind: PodSecurityPolicy 64 | metadata: 65 | name: eirini-task-reporter 66 | annotations: 67 | seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'runtime/default' 68 | seccomp.security.alpha.kubernetes.io/defaultProfileName: 'runtime/default' 69 | spec: 70 | privileged: false 71 | allowPrivilegeEscalation: false 72 | requiredDropCapabilities: 73 | - ALL 74 | volumes: 75 | - 'configMap' 76 | - 'secret' 77 | - 'projected' 78 | hostNetwork: false 79 | hostIPC: false 80 | hostPID: false 81 | runAsUser: 82 | rule: 'MustRunAsNonRoot' 83 | seLinux: 84 | # This policy assumes the nodes are using AppArmor rather than SELinux. 85 | rule: 'RunAsAny' 86 | supplementalGroups: 87 | rule: 'MustRunAs' 88 | ranges: 89 | # Forbid adding the root group. 90 | - min: 1 91 | max: 65535 92 | fsGroup: 93 | rule: 'MustRunAs' 94 | ranges: 95 | # Forbid adding the root group. 96 | - min: 1 97 | max: 65535 98 | readOnlyRootFilesystem: false 99 | 100 | --- 101 | apiVersion: rbac.authorization.k8s.io/v1 102 | kind: ClusterRole 103 | metadata: 104 | name: eirini-task-reporter 105 | rules: 106 | - apiGroups: 107 | - "" 108 | resources: 109 | - pods 110 | verbs: 111 | - get 112 | - watch 113 | - list 114 | - apiGroups: 115 | - batch 116 | resources: 117 | - jobs 118 | verbs: 119 | - list 120 | 121 | --- 122 | apiVersion: rbac.authorization.k8s.io/v1 123 | kind: ClusterRoleBinding 124 | metadata: 125 | name: eirini-task-reporter 126 | roleRef: 127 | kind: ClusterRole 128 | name: eirini-task-reporter 129 | apiGroup: rbac.authorization.k8s.io 130 | subjects: 131 | - kind: ServiceAccount 132 | name: eirini-task-reporter 133 | namespace: {{ .Release.Namespace }} 134 | -------------------------------------------------------------------------------- /helm/templates/events/event-reporter-configmap.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: ConfigMap 4 | metadata: 5 | name: eirini-event-reporter 6 | namespace: {{ .Release.Namespace }} 7 | data: 8 | events.yml: | 9 | # cc_internal_api is the URL used by Eirini to call the Cloud Controller. 10 | cc_internal_api: "{{ .Values.cc_api.scheme }}://{{ .Values.cc_api.host }}:{{ .Values.cc_api.port }}" 11 | 12 | # cc_tls_disabled specifies wether Eirini should communicate to the Cloud 13 | # Controller via HTTPS or not. This should be set to false if TLS is 14 | # handled transparently, e.g. by a service mesh. 15 | # 16 | # When cc_tls_disabled is false, which is the default, a secret must be 17 | # provided for the cc-certs volume in core/api-deployment.yml. This must 18 | # have entries tls.ca, tls.crt and tls.key, for the TLS certificate 19 | # authority, client certificate and key respectively, used for mTLS with 20 | # the Cloud Controller. 21 | cc_tls_disabled: {{ .Values.cc_api.tls_disabled }} 22 | -------------------------------------------------------------------------------- /helm/templates/events/event-reporter-deployment.yml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: Deployment 3 | metadata: 4 | name: eirini-event-reporter 5 | namespace: {{ .Release.Namespace }} 6 | spec: 7 | selector: 8 | matchLabels: 9 | name: eirini-event-reporter 10 | replicas: 2 11 | template: 12 | metadata: 13 | labels: 14 | name: eirini-event-reporter 15 | spec: 16 | dnsPolicy: ClusterFirst 17 | serviceAccountName: eirini-event-reporter 18 | volumes: 19 | - name: config-map-volume 20 | configMap: 21 | name: eirini-event-reporter 22 | - name: cc-certs 23 | secret: 24 | secretName: {{ .Values.cc_api.tls_secret_name }} 25 | optional: true 26 | securityContext: 27 | runAsNonRoot: true 28 | containers: 29 | - name: event-reporter 30 | image: {{ .Values.images.event_reporter }} 31 | args: [ "--config", "/etc/eirini-event-reporter/config/events.yml" ] 32 | imagePullPolicy: IfNotPresent 33 | volumeMounts: 34 | - name: config-map-volume 35 | mountPath: /etc/eirini-event-reporter/config 36 | - name: cc-certs 37 | mountPath: /etc/cf-api/certs 38 | resources: 39 | requests: 40 | cpu: 15m 41 | memory: 15Mi 42 | limits: 43 | cpu: 100m 44 | memory: 100Mi 45 | -------------------------------------------------------------------------------- /helm/templates/events/event-reporter-rbac.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: ServiceAccount 4 | metadata: 5 | name: eirini-event-reporter 6 | namespace: {{ .Release.Namespace }} 7 | 8 | --- 9 | apiVersion: rbac.authorization.k8s.io/v1 10 | kind: Role 11 | metadata: 12 | name: eirini-event-reporter-psp 13 | namespace: {{ .Release.Namespace }} 14 | rules: 15 | - apiGroups: 16 | - policy 17 | resources: 18 | - podsecuritypolicies 19 | verbs: 20 | - use 21 | resourceNames: 22 | - eirini-event-reporter 23 | - apiGroups: 24 | - "" 25 | resources: 26 | - configmaps 27 | verbs: 28 | - get 29 | - create 30 | - update 31 | - apiGroups: 32 | - "coordination.k8s.io" 33 | resources: 34 | - leases 35 | verbs: 36 | - get 37 | - create 38 | - update 39 | - apiGroups: 40 | - "" 41 | resources: 42 | - events 43 | verbs: 44 | - create 45 | 46 | --- 47 | apiVersion: rbac.authorization.k8s.io/v1 48 | kind: RoleBinding 49 | metadata: 50 | name: eirini-event-reporter-psp 51 | namespace: {{ .Release.Namespace }} 52 | roleRef: 53 | kind: Role 54 | name: eirini-event-reporter-psp 55 | apiGroup: rbac.authorization.k8s.io 56 | subjects: 57 | - kind: ServiceAccount 58 | name: eirini-event-reporter 59 | namespace: {{ .Release.Namespace }} 60 | 61 | --- 62 | apiVersion: policy/v1beta1 63 | kind: PodSecurityPolicy 64 | metadata: 65 | name: eirini-event-reporter 66 | annotations: 67 | seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'runtime/default' 68 | seccomp.security.alpha.kubernetes.io/defaultProfileName: 'runtime/default' 69 | spec: 70 | privileged: false 71 | allowPrivilegeEscalation: false 72 | requiredDropCapabilities: 73 | - ALL 74 | volumes: 75 | - 'configMap' 76 | - 'secret' 77 | hostNetwork: false 78 | hostIPC: false 79 | hostPID: false 80 | runAsUser: 81 | # we use non-numeric user ID, so K8s can't verify if it is root or not 82 | rule: 'RunAsAny' 83 | seLinux: 84 | # This policy assumes the nodes are using AppArmor rather than SELinux. 85 | rule: 'RunAsAny' 86 | supplementalGroups: 87 | rule: 'MustRunAs' 88 | ranges: 89 | # Forbid adding the root group. 90 | - min: 1 91 | max: 65535 92 | fsGroup: 93 | rule: 'MustRunAs' 94 | ranges: 95 | # Forbid adding the root group. 96 | - min: 1 97 | max: 65535 98 | readOnlyRootFilesystem: false 99 | 100 | --- 101 | apiVersion: rbac.authorization.k8s.io/v1 102 | kind: ClusterRole 103 | metadata: 104 | name: eirini-event-reporter-nodes-policy 105 | rules: 106 | - apiGroups: 107 | - "" 108 | resources: 109 | - "nodes" 110 | - "nodes/proxy" 111 | verbs: 112 | - "get" 113 | - "list" 114 | --- 115 | apiVersion: rbac.authorization.k8s.io/v1 116 | kind: ClusterRoleBinding 117 | metadata: 118 | name: eirini-event-reporter-nodes-policy 119 | subjects: 120 | - kind: ServiceAccount 121 | name: eirini-event-reporter 122 | namespace: {{ .Release.Namespace }} 123 | roleRef: 124 | kind: ClusterRole 125 | name: eirini-event-reporter-nodes-policy 126 | apiGroup: rbac.authorization.k8s.io 127 | 128 | --- 129 | apiVersion: rbac.authorization.k8s.io/v1 130 | kind: ClusterRole 131 | metadata: 132 | name: eirini-event-reporter 133 | rules: 134 | - apiGroups: 135 | - "" 136 | resources: 137 | - pods 138 | - events 139 | verbs: 140 | - get 141 | - list 142 | - watch 143 | 144 | --- 145 | apiVersion: rbac.authorization.k8s.io/v1 146 | kind: ClusterRoleBinding 147 | metadata: 148 | name: eirini-event-reporter 149 | roleRef: 150 | kind: ClusterRole 151 | name: eirini-event-reporter 152 | apiGroup: rbac.authorization.k8s.io 153 | subjects: 154 | - kind: ServiceAccount 155 | name: eirini-event-reporter 156 | namespace: {{ .Release.Namespace }} 157 | -------------------------------------------------------------------------------- /helm/templates/workloads/app-rbac.yml: -------------------------------------------------------------------------------- 1 | {{- range prepend (.Values.workloads.namespaces | default list) .Values.workloads.default_namespace }} 2 | --- 3 | apiVersion: v1 4 | kind: ServiceAccount 5 | metadata: 6 | name: eirini 7 | namespace: {{ . }} 8 | automountServiceAccountToken: false 9 | 10 | --- 11 | apiVersion: rbac.authorization.k8s.io/v1 12 | kind: Role 13 | metadata: 14 | name: eirini-workloads-app-role 15 | namespace: {{ . }} 16 | rules: 17 | - apiGroups: ['policy'] 18 | resources: ['podsecuritypolicies'] 19 | verbs: ['use'] 20 | resourceNames: 21 | - eirini-workloads-app-psp 22 | 23 | --- 24 | # Bind to the default service account 25 | apiVersion: rbac.authorization.k8s.io/v1 26 | kind: RoleBinding 27 | metadata: 28 | name: eirini-workloads-app-rolebinding 29 | namespace: {{ . }} 30 | roleRef: 31 | kind: Role 32 | name: eirini-workloads-app-role 33 | apiGroup: rbac.authorization.k8s.io 34 | subjects: 35 | - kind: ServiceAccount 36 | name: eirini 37 | namespace: {{ . }} 38 | {{- end }} 39 | 40 | --- 41 | apiVersion: policy/v1beta1 42 | kind: PodSecurityPolicy 43 | metadata: 44 | name: eirini-workloads-app-psp 45 | annotations: 46 | seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default,runtime/default' 47 | seccomp.security.alpha.kubernetes.io/defaultProfileName: 'runtime/default' 48 | spec: 49 | privileged: false 50 | allowPrivilegeEscalation: false 51 | requiredDropCapabilities: 52 | - ALL 53 | hostNetwork: false 54 | hostIPC: false 55 | hostPID: false 56 | runAsUser: 57 | rule: 'MustRunAsNonRoot' 58 | seLinux: 59 | # This policy assumes the nodes are using AppArmor rather than SELinux. 60 | rule: 'RunAsAny' 61 | supplementalGroups: 62 | rule: 'MustRunAs' 63 | ranges: 64 | # Forbid adding the root group. 65 | - min: 1 66 | max: 65535 67 | fsGroup: 68 | rule: 'MustRunAs' 69 | ranges: 70 | # Forbid adding the root group. 71 | - min: 1 72 | max: 65535 73 | readOnlyRootFilesystem: false 74 | -------------------------------------------------------------------------------- /helm/templates/workloads/core/api-rbac.yml: -------------------------------------------------------------------------------- 1 | {{- $releaseNamespace := .Release.Namespace }} 2 | {{- range prepend (.Values.workloads.namespaces | default list) .Values.workloads.default_namespace }} 3 | --- 4 | apiVersion: rbac.authorization.k8s.io/v1 5 | kind: Role 6 | metadata: 7 | name: eirini-namespaced-role 8 | namespace: {{ . }} 9 | rules: 10 | - apiGroups: 11 | - batch 12 | resources: 13 | - jobs 14 | verbs: 15 | - create 16 | - delete 17 | - apiGroups: 18 | - apps 19 | resources: 20 | - statefulsets 21 | verbs: 22 | - create 23 | - update 24 | - delete 25 | - apiGroups: 26 | - "" 27 | resources: 28 | - pods 29 | verbs: 30 | - delete 31 | - apiGroups: 32 | - policy 33 | resources: 34 | - poddisruptionbudgets 35 | verbs: 36 | - create 37 | - delete 38 | - apiGroups: 39 | - "" 40 | resources: 41 | - secrets 42 | verbs: 43 | - create 44 | - delete 45 | - patch 46 | 47 | --- 48 | # Bind to the default service account 49 | apiVersion: rbac.authorization.k8s.io/v1 50 | kind: RoleBinding 51 | metadata: 52 | name: eirini-namespaced-rolebinding 53 | namespace: {{ . }} 54 | roleRef: 55 | kind: Role 56 | apiGroup: rbac.authorization.k8s.io 57 | name: eirini-namespaced-role 58 | subjects: 59 | - kind: ServiceAccount 60 | name: eirini-api 61 | namespace: {{ $releaseNamespace }} 62 | {{- end }} 63 | -------------------------------------------------------------------------------- /helm/templates/workloads/core/task-reporter-rbac.yml: -------------------------------------------------------------------------------- 1 | {{- $releaseNamespace := .Release.Namespace }} 2 | {{- range prepend (.Values.workloads.namespaces | default list) .Values.workloads.default_namespace }} 3 | --- 4 | apiVersion: rbac.authorization.k8s.io/v1 5 | kind: Role 6 | metadata: 7 | name: eirini-task-reporter-namespaced 8 | namespace: {{ . }} 9 | rules: 10 | - apiGroups: 11 | - batch 12 | resources: 13 | - jobs 14 | verbs: 15 | - delete 16 | - patch 17 | - apiGroups: 18 | - "" 19 | resources: 20 | - pods 21 | verbs: 22 | - patch 23 | - apiGroups: 24 | - "" 25 | resources: 26 | - secrets 27 | verbs: 28 | - list 29 | - delete 30 | --- 31 | 32 | apiVersion: rbac.authorization.k8s.io/v1 33 | kind: RoleBinding 34 | metadata: 35 | name: eirini-task-reporter-namespaced 36 | namespace: {{ . }} 37 | roleRef: 38 | kind: Role 39 | name: eirini-task-reporter-namespaced 40 | apiGroup: rbac.authorization.k8s.io 41 | subjects: 42 | - kind: ServiceAccount 43 | name: eirini-task-reporter 44 | namespace: {{ $releaseNamespace }} 45 | {{- end }} 46 | -------------------------------------------------------------------------------- /helm/templates/workloads/events/event-reporter-rbac.yml: -------------------------------------------------------------------------------- 1 | {{- $releaseNamespace := .Release.Namespace }} 2 | {{- range prepend (.Values.workloads.namespaces | default list) .Values.workloads.default_namespace }} 3 | --- 4 | apiVersion: rbac.authorization.k8s.io/v1 5 | kind: Role 6 | metadata: 7 | name: eirini-event-reporter-namespaced 8 | namespace: {{ . }} 9 | rules: 10 | - apiGroups: 11 | - "" 12 | resources: 13 | - pods 14 | verbs: 15 | - patch 16 | --- 17 | 18 | apiVersion: rbac.authorization.k8s.io/v1 19 | kind: RoleBinding 20 | metadata: 21 | name: eirini-event-reporter-namespaced 22 | namespace: {{ . }} 23 | roleRef: 24 | kind: Role 25 | name: eirini-event-reporter-namespaced 26 | apiGroup: rbac.authorization.k8s.io 27 | subjects: 28 | - kind: ServiceAccount 29 | name: eirini-event-reporter 30 | namespace: {{ $releaseNamespace }} 31 | {{- end }} 32 | -------------------------------------------------------------------------------- /helm/templates/workloads/namespace.yml: -------------------------------------------------------------------------------- 1 | {{- if .Values.workloads.create_namespaces }} 2 | {{- range prepend (.Values.workloads.namespaces | default list) .Values.workloads.default_namespace }} 3 | --- 4 | apiVersion: v1 5 | kind: Namespace 6 | metadata: 7 | name: {{ . }} 8 | {{- end }} 9 | {{- end }} 10 | -------------------------------------------------------------------------------- /helm/values.yaml: -------------------------------------------------------------------------------- 1 | api: 2 | # registry_secret_name is the name of the secret containing the docker 3 | # credentials to pull LRP images 4 | registry_secret_name: app-registry-credentials 5 | 6 | # unsafe_allow_automount_service_account_token when set to true causes 7 | # Kubernetes to mount the service account token in the LRP and task 8 | # containers. This gives the code running there access to the Kubernetes 9 | # API with the privileges of the application service account. 10 | # 11 | # This should generally be left as the default false unless there is a 12 | # good reason and the implications are understood. It is required when 13 | # running cf-for-k8s in a kind cluster, for example. 14 | unsafe_allow_automount_service_account_token: false 15 | 16 | # tls_port is the port used by Eirini to serve its REST API over HTTPS 17 | # Required when serve_plaintext is set to false or omitted. 18 | tls_port: 8085 19 | 20 | # plaintext_port is the port used by Eirini to serve its REST API over 21 | # plain HTTP. Required when serve_plaintext is set to true. 22 | plaintext_port: 8080 23 | 24 | # serve_plaintext specifies whether the Eirini REST API should be served 25 | # over plain HTTP or not. 26 | # 27 | # When serve_plaintext is false, which is the default, a secret must be 28 | # provided for the eirini-certs volume in core/api-deployment.yml. This 29 | # must have entries tls.ca, tls.crt and tls.key, for the TLS certificate 30 | # authority, certificate and key respectively, used to serve TLS. 31 | serve_plaintext: true 32 | 33 | # tls_secret_name is required when api.cc_tls_disabled is false in the 34 | # core/api-configmap.yml. It must have keys tls.ca, tls.crt and tls.key for 35 | # the certificate authority, client certificate and private key respectively, 36 | # used for mTLS with the Cloud Controller. 37 | tls_secret_name: eirini-internal-tls-certs 38 | 39 | # prometheus_port is the port used to expose Prometheus metrics. When set 40 | # to 0, the metrics endpoint is disabled. 41 | prometheus_port: 8080 42 | 43 | tasks: 44 | # completion_callback_retry_limit is the number of times Eirini will retry 45 | # to call the Cloud Controller completion callback in case the Cloud 46 | # Controller is unreachable. 47 | completion_callback_retry_limit: 10 48 | 49 | # ttl_seconds is the number of seconds Eirini will wait before deleting the 50 | # Job associated to a completed Task. 51 | ttl_seconds: 5 52 | 53 | workloads: 54 | # default_namespace is the namespace used by Eirini to deploy LRPs that do 55 | # not specify their own namespace in the request. 56 | default_namespace: cf-workloads 57 | 58 | namespaces: [] 59 | create_namespaces: false 60 | 61 | cc_api: 62 | # host is the host used by Eirini to call the Cloud Controller. 63 | host: "capi.cf-system.svc.cluster.local" 64 | 65 | # port is the port used by Eirini to call the Cloud Controller. 66 | port: 9023 67 | 68 | # scheme is the URL scheme used by Eirini to call the Cloud Controller. 69 | scheme: http 70 | 71 | # tls_disabled specifies wether Eirini should communicate to the Cloud 72 | # Controller via HTTPS or not. This should be set to true if TLS is handled 73 | # transparently, e.g. by a service mesh. 74 | # 75 | # When tls_disabled is false, a secret must be provided for the cc-certs 76 | # volume via tls_secret_name. This must have entries tls.ca, tls.crt and 77 | # tls.key, for the TLS certificate authority, client certificate and key 78 | # respectively, used for mTLS with the Cloud Controller. 79 | tls_disabled: true 80 | 81 | # tls_secret_name is required when api.cc_tls_disabled is false in the 82 | # core/api-configmap.yml. It must have keys tls.ca, tls.crt and tls.key for 83 | # the certificate authority, client certificate and private key respectively, 84 | # used for mTLS with the Cloud Controller. 85 | tls_secret_name: eirini-internal-tls-certs 86 | 87 | instance_index_env_injector: 88 | certs_secret_name: eirini-instance-index-env-injector-certs 89 | 90 | images: 91 | api: eirini/api@sha256:e17fdf0d2d2473aedf74b710b0e31ccfcd6954eebd97652707aa41879216f263 92 | instance_index_env_injector: eirini/instance-index-env-injector@sha256:4d827b47f913d3baa333f1abd1fc9a0c13ec35eb1194fb1180eb0b0122d9841e 93 | task_reporter: eirini/task-reporter@sha256:6ac28d1dabc7c1147ae3f057deacfb3e3f4febb739fd5a8e1561a72813f4f9fa 94 | event_reporter: eirini/event-reporter@sha256:9931ee78ea5075d50e3573a9c19daaaf97bcf37dd84a3f9d48a0df5040131f0b 95 | migration: eirini/migration@sha256:94b2419427efbc350f3edf29b7fff754d928aa330b09a91e1855ce4eff117edd 96 | -------------------------------------------------------------------------------- /scripts/assets/cf-for-k8s-value-overrides.yml: -------------------------------------------------------------------------------- 1 | cc_api: 2 | tls_disabled: true 3 | -------------------------------------------------------------------------------- /scripts/assets/value-overrides.yml: -------------------------------------------------------------------------------- 1 | api: 2 | serve_plaintext: false 3 | tls_secret_name: eirini-certs 4 | 5 | workloads: 6 | default_namespace: eirini-workloads 7 | create_namespaces: true 8 | 9 | cc_api: 10 | host: "cc-wiremock.eirini-core.svc.cluster.local" 11 | scheme: https 12 | port: 443 13 | tls_secret_name: capi-tls 14 | tls_disabled: false 15 | 16 | routing: 17 | nats: 18 | host: "nats-client.eirini-core.svc.cluster.local" 19 | -------------------------------------------------------------------------------- /scripts/assets/wiremock.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: apps/v1 3 | kind: Deployment 4 | metadata: 5 | name: cc-wiremock 6 | namespace: eirini-core 7 | spec: 8 | selector: 9 | matchLabels: 10 | name: cc-wiremock 11 | template: 12 | metadata: 13 | labels: 14 | name: cc-wiremock 15 | spec: 16 | volumes: 17 | - name: wiremock-keystore 18 | secret: 19 | secretName: wiremock-keystore 20 | optional: true 21 | containers: 22 | - name: wiremock 23 | env: 24 | - name: KEYSTORE_PASSWORD 25 | valueFrom: 26 | secretKeyRef: 27 | name: wiremock-keystore 28 | key: ks.pass 29 | image: wiremock/wiremock 30 | args: ["--verbose", "--https-port", "8081", "--https-keystore", "/etc/wiremock/keystore/keystore.pkcs12", "--keystore-type", "PKCS12", "--keystore-password", "$(KEYSTORE_PASSWORD)", "--key-manager-password", "$(KEYSTORE_PASSWORD)"] 31 | volumeMounts: 32 | - name: wiremock-keystore 33 | mountPath: /etc/wiremock/keystore 34 | ports: 35 | - containerPort: 8080 36 | name: http 37 | - containerPort: 8081 38 | name: https 39 | 40 | --- 41 | apiVersion: v1 42 | kind: Service 43 | metadata: 44 | name: cc-wiremock 45 | namespace: eirini-core 46 | spec: 47 | type: ClusterIP 48 | ports: 49 | - port: 80 50 | targetPort: 8080 51 | protocol: TCP 52 | name: http 53 | - port: 443 54 | targetPort: 8081 55 | protocol: TCP 56 | name: https 57 | selector: 58 | name: cc-wiremock 59 | -------------------------------------------------------------------------------- /scripts/cleanup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -euo pipefail 4 | 5 | SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" 6 | readonly SYSTEM_NAMESPACE=eirini-core 7 | 8 | source "$SCRIPT_DIR/helpers/print.sh" 9 | 10 | delete_eirini() { 11 | helm --namespace "$SYSTEM_NAMESPACE" delete eirini || true 12 | helm --namespace "$SYSTEM_NAMESPACE" delete nats || true 13 | helm --namespace "$SYSTEM_NAMESPACE" delete prometheus || true 14 | kubectl delete -f "$SCRIPT_DIR/assets/wiremock.yml" || true 15 | kubectl delete namespace "$SYSTEM_NAMESPACE" --wait || true 16 | } 17 | 18 | main() { 19 | print_disclaimer 20 | delete_eirini 21 | } 22 | 23 | main 24 | -------------------------------------------------------------------------------- /scripts/deploy.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -euo pipefail 4 | 5 | ROOT_DIR="$(cd "$(dirname "$0")/.." && pwd)" 6 | SCRIPT_DIR="$ROOT_DIR/scripts" 7 | 8 | export KUBECONFIG 9 | KUBECONFIG=${KUBECONFIG:-$HOME/.kube/config} 10 | KUBECONFIG=$(readlink -f "$KUBECONFIG") 11 | 12 | export GOOGLE_APPLICATION_CREDENTIALS 13 | GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS:-""} 14 | if [[ -n $GOOGLE_APPLICATION_CREDENTIALS ]]; then 15 | GOOGLE_APPLICATION_CREDENTIALS=$(readlink -f "$GOOGLE_APPLICATION_CREDENTIALS") 16 | fi 17 | export WIREMOCK_KEYSTORE_PASSWORD 18 | WIREMOCK_KEYSTORE_PASSWORD=${WIREMOCK_KEYSTORE_PASSWORD:-""} 19 | 20 | readonly SYSTEM_NAMESPACE=eirini-core 21 | 22 | source "$SCRIPT_DIR/helpers/print.sh" 23 | 24 | main() { 25 | print_disclaimer 26 | generate_secrets 27 | install_prometheus 28 | install_eirini "$@" 29 | } 30 | 31 | generate_secrets() { 32 | "$SCRIPT_DIR/generate-secrets.sh" "*.${SYSTEM_NAMESPACE}.svc" "$WIREMOCK_KEYSTORE_PASSWORD" 33 | } 34 | 35 | install_prometheus() { 36 | helm repo add prometheus-community https://prometheus-community.github.io/helm-charts 37 | helm repo update 38 | helm upgrade prometheus \ 39 | --install prometheus-community/prometheus \ 40 | --namespace "$SYSTEM_NAMESPACE" \ 41 | --wait 42 | } 43 | 44 | install_eirini() { 45 | local env_injector_ca_bundle 46 | env_injector_ca_bundle="$(kubectl get secret -n $SYSTEM_NAMESPACE eirini-instance-index-env-injector-certs -o jsonpath="{.data['tls\.ca']}")" 47 | helm upgrade eirini \ 48 | --install "$ROOT_DIR/helm" \ 49 | --namespace "$SYSTEM_NAMESPACE" \ 50 | --values "$SCRIPT_DIR/assets/value-overrides.yml" \ 51 | --set "webhook_ca_bundle=$env_injector_ca_bundle" \ 52 | --wait \ 53 | "$@" 54 | 55 | # Install wiremock to mock the cloud controller 56 | kubectl apply -f "$SCRIPT_DIR/assets/wiremock.yml" 57 | } 58 | 59 | main "$@" 60 | -------------------------------------------------------------------------------- /scripts/generate-secrets.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -eu 4 | 5 | echo "Will now generate tls.ca tls.crt and tls.key files" 6 | 7 | mkdir -p keys 8 | trap 'rm -rf keys' EXIT 9 | 10 | otherDNS=$1 11 | keystore_password=$2 12 | 13 | pushd keys 14 | { 15 | kubectl create namespace eirini-core || true 16 | 17 | openssl req -x509 -newkey rsa:4096 -keyout tls.key -out tls.crt -nodes -subj '/CN=localhost' -addext "subjectAltName = DNS:$otherDNS, DNS:$otherDNS.cluster.local" -days 365 18 | 19 | for secret_name in eirini-certs loggregator-certs capi-tls eirini-instance-index-env-injector-certs; do 20 | if kubectl -n eirini-core get secret "$secret_name" >/dev/null 2>&1; then 21 | kubectl delete secret -n eirini-core "$secret_name" 22 | fi 23 | echo "Creating the $secret_name secret in your kubernetes cluster" 24 | kubectl create secret -n eirini-core generic "$secret_name" --from-file=tls.crt=./tls.crt --from-file=tls.ca=./tls.crt --from-file=tls.key=./tls.key 25 | done 26 | 27 | if kubectl -n eirini-core get secret wiremock-keystore >/dev/null 2>&1; then 28 | kubectl delete secret -n eirini-core wiremock-keystore 29 | fi 30 | pem_file=$(mktemp) 31 | keystore_file=$(mktemp) 32 | cat ./tls.key >"$pem_file" 33 | cat ./tls.crt >>"$pem_file" 34 | openssl pkcs12 -export -in "$pem_file" -out "$keystore_file" -password "pass:$keystore_password" 35 | 36 | kubectl create secret -n eirini-core generic wiremock-keystore --from-file=keystore.pkcs12="$keystore_file" --from-literal=ks.pass="$keystore_password" 37 | rm "$pem_file" 38 | rm "$keystore_file" 39 | 40 | echo "Done!" 41 | } 42 | popd 43 | -------------------------------------------------------------------------------- /scripts/helpers/print.sh: -------------------------------------------------------------------------------- 1 | RED=1 2 | GREEN=2 3 | BLUE=4 4 | 5 | print_message() { 6 | message=$1 7 | colour=$2 8 | printf "\\r\\033[00;3%sm%s\\033[0m\\n" "$colour" "$message" 9 | } 10 | 11 | warning=$( 12 | cat <