├── .vscode └── spellright.dict ├── DETAILED.md └── README.md /.vscode/spellright.dict: -------------------------------------------------------------------------------- 1 | Kubernetes 2 | Istio 3 | sha256 4 | containerd 5 | KubeCon 6 | cloudfoundry 7 | configmaps 8 | Dockerhub 9 | seccomd 10 | apparmor 11 | resourcesprovides 12 | rootfs 13 | -------------------------------------------------------------------------------- /DETAILED.md: -------------------------------------------------------------------------------- 1 | # Kubernetes-idiomatic Cloud Foundry components guidelines 2 | 3 | Also check [shorter version](./README.md) 4 | 5 | * [Structure](#structure) 6 | * [Wording](#wording) 7 | * [Production readiness criteria](#production-readiness-criteria) 8 | * [Resilience](#resilience) 9 | * [Availability](#availability) 10 | * [Failure Recovery](#failure-recovery) 11 | * [Isolation](#isolation) 12 | * [Operability](#operability) 13 | * [Resource Planning](#resource-planning) 14 | * [Health Monitoring](#health-monitoring) 15 | * [Logging](#logging) 16 | * [Diagnostics Tooling](#diagnostics-tooling) 17 | * [Customisation](#customisation) 18 | * [Upgrades](#upgrades) 19 | * [Security](#security) 20 | * [Open Source](#open-source) 21 | * [Guidelines](#guidelines) 22 | * [Code](#code) 23 | * [Health endpoint](#health-endpoint) 24 | * [Log location](#log-location) 25 | * [Passing configuration to the application](#passing-configuration-to-the-application) 26 | * [Ability to rollback](#ability-to-rollback) 27 | * [Work with signals](#work-with-signals) 28 | * [Resilience to cluster upgrades](#resilience-to-cluster-upgrades) 29 | * [Labels for objects managed by component](#labels-for-objects-managed-by-component) 30 | * [Packaging](#packaging) 31 | * [Non-root user](#non-root-user) 32 | * [Image metadata](#image-metadata) 33 | * [Base image](#base-image) 34 | * [Dependencies](#dependencies) 35 | * [Keeping base image up to date](#keeping-base-image-up-to-date) 36 | * [Image location](#image-location) 37 | * [Packaging instructions](#packaging-instructions) 38 | * [Pod specification](#pod-specification) 39 | * [Image referenced by sha256](#image-referenced-by-sha256) 40 | * [Pod labels](#pod-labels) 41 | * [Readiness probe](#readiness-probe) 42 | * [Liveness probe](#liveness-probe) 43 | * [Number of containers](#number-of-containers) 44 | * [Number of init containers](#number-of-init-containers) 45 | * [Pod requests](#pod-requests) 46 | * [Pod limits](#pod-limits) 47 | * [Pod service account](#pod-service-account) 48 | * [Service account token](#service-account-token) 49 | * [Pod security configuration](#pod-security-configuration) 50 | * [Using certificates](#using-certificates) 51 | * [Pod port names](#pod-port-names) 52 | * [Affinity](#affinity) 53 | * [Service specification](#service-specification) 54 | * [Using services](#using-services) 55 | * [Service pod names](#service-pod-names) 56 | * [Service labels](#service-labels) 57 | * [Other Kubernetes objects](#other-kubernetes-objects) 58 | * [Component Object Labels](#component-object-labels) 59 | * [Pod Disruption Budget](#pod-disruption-budget) 60 | * [Pod Security Policy](#pod-security-policy) 61 | * [Networking policy](#networking-policy) 62 | * [Istio RBAC](#istio-rbac) 63 | * [Access outside the cluster](#access-outside-the-cluster) 64 | * [Using secrets](#using-secrets) 65 | * [Service accounts](#service-accounts) 66 | * [Replicas count](#replicas-count) 67 | * [Work with other components](#work-with-other-components) 68 | * [Optional parts](#optional-parts) 69 | * [Using custom DNS addresses](#using-custom-dns-addresses) 70 | * [Documentation](#documentation) 71 | * [Kubernetes versions support](#kubernetes-versions-support) 72 | * [Container runtime support](#container-runtime-support) 73 | 74 | ## Structure 75 | 76 | The document consists of two big parts: [criteria grouped in themes](#production-readiness-criteria) and [guidelines that solving some of the problems the criteria defined](#guidelines). The document intends to help developers to create a component that the platform engineer would be able to operate using existing Kubernetes patterns. 77 | 78 | ## Wording 79 | 80 | Component - a single application. 81 | Platform engineer - a person responsible for deploying and operating Cloud Foundry. 82 | 83 | ## Production readiness criteria 84 | 85 | ### Resilience 86 | 87 | Ability to provide an acceptable level of service in the face of faults. 88 | 89 | #### Availability 90 | 91 | Guidelines: 92 | 93 | * [Image referenced by sha256](#image-referenced-by-sha256) 94 | * [Liveness probe](#liveness-probe) 95 | * [Number of containers](#number-of-containers) 96 | * [Pod requests](#pod-requests) 97 | * [Affinity](#affinity) 98 | * [Replica count](#replicas-count) 99 | 100 | #### Failure Recovery 101 | 102 | Guidelines: 103 | 104 | * [Storing dependencies on the image](#dependencies) 105 | * [Number of containers](#number-of-containers) 106 | * [Number of init containers](#number-of-init-containers) 107 | 108 | #### Isolation 109 | 110 | Guidelines: 111 | 112 | * [Number of containers](#number-of-containers) 113 | * [Affinity](#affinity) 114 | 115 | ### Operability 116 | 117 | #### Resource Planning 118 | 119 | Guidelines: 120 | 121 | * [Storing dependencies on the image](#dependencies) 122 | * [Number of containers](#number-of-containers) 123 | * [Pod requests](#pod-requests) 124 | * [Pod limits](#pod-limits) 125 | * [Affinity](#affinity) 126 | 127 | #### Health Monitoring 128 | 129 | Guidelines: 130 | 131 | * [Health checks](#health-endpoint) 132 | 133 | #### Logging 134 | 135 | Guidelines: 136 | 137 | * [Log location](#log-location) 138 | * [Pod labels](#pod-labels) 139 | * [Service labels](#service-labels) 140 | * [Labels for objects managed by component](#labels-for-objects-managed-by-component) 141 | * [Component objects labels](#component-object-labels) 142 | 143 | #### Diagnostics Tooling 144 | 145 | Guidelines: 146 | 147 | * [Image metadata](#image-metadata) 148 | * [Pod labels](#pod-labels) 149 | * [Service labels](#service-labels) 150 | * [Labels for objects managed by component](#labels-for-objects-managed-by-component) 151 | * [Component objects labels](#component-object-labels) 152 | 153 | #### Customisation 154 | 155 | Guidelines: 156 | 157 | * [Passing configuration to the application](#passing-configuration-to-the-application) 158 | * [Pod service account](#pod-service-account) 159 | * [Using certificates](#using-certificates) 160 | * [Pod port names](#pod-port-names) 161 | * [Networking policy](#networking-policy) 162 | * [Istio RBAC](#istio-rbac) 163 | * [Access outside the cluster](#access-outside-the-cluster) 164 | * [Using secrets](#using-secrets) 165 | * [Service accounts](#service-accounts) 166 | 167 | #### Upgrades 168 | 169 | As a platform operator, I can upgrade the Kubernetes cluster with the minimum application downtime. 170 | 171 | As a platform operator, I can upgrade Cloud Foundry with minimum control plane downtime. 172 | 173 | Guidelines: 174 | 175 | * [Health checks](#health-endpoint) 176 | * [Rollback](#ability-to-rollback) 177 | * [Work with signals](#work-with-signals) 178 | * [Resilience to cluster upgrades](#resilience-to-cluster-upgrades) 179 | * [Storing dependencies on the image](#dependencies) 180 | * [Image referenced by sha256](#image-referenced-by-sha256) 181 | * [Number of containers](#number-of-containers) 182 | * [Number of init containers](#number-of-init-containers) 183 | * [Pod disruption budget](#pod-disruption-budget) 184 | 185 | ### Security 186 | 187 | Guidelines: 188 | 189 | * [Non-root user](#non-root-user) 190 | * [Storing dependencies on the image](#dependencies) 191 | * [Keeping base image up to date](#keeping-base-image-up-to-date) 192 | * [Pod service account](#pod-service-account) 193 | * [Service account token](#service-account-token) 194 | * [Pod security configuration](#pod-security-configuration) 195 | * [Pod security policy](#pod-security-policy) 196 | * [Networking policy](#networking-policy) 197 | * [Istio RBAC](#istio-rbac) 198 | * [Service accounts](#service-accounts) 199 | 200 | ### Open Source 201 | 202 | As an open-source contributor, I can submit a patch for the component(that passing tests). 203 | 204 | As an open-source platform architect, I can consume the component. 205 | 206 | Guidelines: 207 | 208 | * [Image metadata](#image-metadata) 209 | * [Base image](#base-image) 210 | * [Image location](#image-location) 211 | * [Packaging instructions](#packaging-instructions) 212 | * [Networking policy](#networking-policy) 213 | * [Istio RBAC](#istio-rbac) 214 | * [Access outside the cluster](#access-outside-the-cluster) 215 | * [Using secrets](#using-secrets) 216 | 217 | ## Guidelines 218 | 219 | ### Code 220 | 221 | #### Health endpoint 222 | 223 | Every component must have an endpoint with health information or crash if it is unhealthy. 224 | 225 | Improves: 226 | 227 | * [operability](#health-monitoring). 228 | * [upgrades](#upgrades) by allowing using blue-green deployments. 229 | 230 | Read more: 231 | 232 | #### Log location 233 | 234 | All logs must go in stdout/stderr 235 | 236 | Improves: 237 | 238 | * [logging](#logging) 239 | 240 | Reasons: 241 | 242 | * Kubernetes handles the stdout and stderr output of the pod. It saves the logs to the known disk location and the operator can later check the logs from the previous version of the pod for some time after the restart. 243 | * One of the security recommendations for Kubernetes applications is to use [ReadOnlyRootFilesystem](https://kubernetes.io/docs/concepts/policy/pod-security-policy/#volumes-and-file-systems) and as a result do not write anything on the disk. 244 | 245 | #### Passing configuration to the application 246 | 247 | The component must be able to use up-to-date configuration. 248 | 249 | Possible solutions: 250 | 251 | * Read the config from the config files. Ideally, a `conf.d` style config dir so that you can compose the config from multiple configmaps and secrets. The application should check for the file changes and update its config when it changes. 252 | * The configs are immutable and the component only consumes new configuration through deploying between ConfigMaps and Secrets. 253 | * The configs are stored in config files and when the config is updated, the liveness probe starts to fail that requires Kubernetes to restart the pod. 254 | 255 | * Problem: It is harder to achieve high availability here. 256 | 257 | * The configs are stored in config files. When the config is updated, the pod gets restarted. 258 | 259 | * Problem: The pod requires access to Kubernetes API. 260 | 261 | * The hash of the config will be stored in the pod specification later. 262 | 263 | * Problem: The platform engineer needs to know how to do the manual config update. 264 | 265 | * The configuration is passed via command-line flags in the pod specification 266 | 267 | * Problem: It is not possible to use the Kubernetes secrets. 268 | 269 | Improves: 270 | 271 | * [customisation](#customisation) 272 | 273 | #### Ability to rollback 274 | 275 | The application with version n-1 should be able to start with the database that has been migrated to version n 276 | 277 | Improves: 278 | 279 | * [application upgrade](#upgrades) 280 | 281 | Reason: 282 | 283 | It is very easy to implement blue-green deployment with the Kubernetes object and do the rollback. Ability to work with a previous version gives the platform engineer the ability to rollback using the `kubectl rollout undo` command. 284 | 285 | See also: 286 | 287 | [Kubernetes documentation about deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) 288 | 289 | #### Work with signals 290 | 291 | The application must respect SIGTERM signal and start sending NotReady probe 292 | 293 | Improves: 294 | 295 | * [Upgrades](#upgrades) 296 | 297 | See also: 298 | 299 | * [Termination of Pods](https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods) 300 | * [Closed issue in Kubernetes](https://github.com/kubernetes/kubernetes/issues/64510) and [the blog entry with a better explanation](https://freecontent.manning.com/handling-client-requests-properly-with-kubernetes/) 301 | 302 | #### Resilience to cluster upgrades 303 | 304 | The component is either expected to work during K8s control plane downtime or has a clear notice README that to achieve high availability, the control plane of Kubernetes must have multiple replicas. 305 | 306 | Improves: 307 | 308 | * [Kubernetes cluster upgrades](#upgrades) 309 | 310 | #### Labels for objects managed by component 311 | 312 | Kubernetes resources that created by a component should have a label `app.kubernetes.io/managed-by` with value `cloudfoundry` 313 | 314 | Improves: 315 | 316 | * [logging](#logging) 317 | * [diagnostic tooling](#diagnostics-tooling) 318 | 319 | ### Packaging 320 | 321 | #### Non-root user 322 | 323 | All components should run with a non-root user unless it is completely impossible, including “FROM scratch” images. The user UID should not be 1337 due to [Istio limitations](https://istio.io/docs/ops/deployment/requirements/). 324 | 325 | Improves: 326 | 327 | * [security](#security) 328 | 329 | See also: 330 | 331 | [Pod security policy](#pod-security-policy) 332 | 333 | #### Image metadata 334 | 335 | All components images should have labels in the metadata with repo URL and SHA of the commit it the metadata as [recommended by OCI](https://github.com/opencontainers/image-spec/blob/master/annotations.md#pre-defined-annotation-keys) 336 | 337 | Improves: 338 | 339 | * [open source](#image-metadata) 340 | * [diagnostics tooling](#diagnostics-tooling) 341 | 342 | Reasons: 343 | 344 | * This allows the operator to check if the component has a CVE. 345 | * This also helps scanners that work off of artefact metadata to determine source code/provenance (e.g. OSL, security) 346 | 347 | See also: 348 | 349 | [Eirini docker image](https://github.com/cloudfoundry-incubator/eirini/blob/16a093f7e43e56779c43b74143aee855173f1748/docker/opi/Dockerfile#L15-L17) as an example 350 | 351 | #### Base image 352 | 353 | The default base image should come from `cloudfoundry/stacks`. All components should have the possibility to change the base image 354 | 355 | Improves: 356 | 357 | * [open source](#open-source) 358 | 359 | Reasons: 360 | 361 | * The base layer for the images should be the same. 362 | * Other Cloud Foundry Foundation members want to build their own base images. 363 | 364 | #### Dependencies 365 | 366 | The run image should not have packages required for compilation, only for running. i.e. don’t have Go package or JDK in the final image, For java, only JRE should be shipped. It should be possible to get a list of current dependencies installed inside. 367 | 368 | Improves: 369 | 370 | * [upgrades](#upgrades) 371 | * [resource planning](#resource-planning) 372 | * [recovery](#failure-recovery) 373 | * [security](#security) 374 | 375 | Reasons: 376 | 377 | * It takes more time to pull a new version of the image in case of scaling, upgrade or recovery. 378 | * Images stored on the worker node disk and might require bigger disks. 379 | * The unneeded packages require upgrading it more often. 380 | * The operator needs to know if the packages in the image up-to-date or not. 381 | 382 | See also: 383 | 384 | [Kubernetes best practices](https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-how-and-why-to-build-small-container-images) by Google Cloud 385 | 386 | #### Keeping base image up to date 387 | 388 | Images are continuously updated with the new version of the base layer. 389 | 390 | Improves: 391 | 392 | * [security](#security) 393 | 394 | #### Image location 395 | 396 | Images are stored in the CFF organisation under Dockerhub. 397 | 398 | Improves: 399 | 400 | * [open source](#open-source) 401 | 402 | #### Packaging instructions 403 | 404 | The component has clear instructions on how to build its container image 405 | 406 | Improves: 407 | 408 | * [open source](#open-source) 409 | 410 | ### Pod specification 411 | 412 | #### Image referenced by sha256 413 | 414 | The image references must always include sha256. 415 | 416 | Reasons: 417 | 418 | The tags in Docker registries are mutable, this can cause two different versions of the application to run on the cluster due to node restart. Sha256 provides immutable version. Both tags and versions can be used at the same time. 419 | 420 | Improves: 421 | 422 | * [Kubernetes cluster upgrades](#upgrades) 423 | * [Availability](#availability) by starting the same version of the component during the recovery event 424 | 425 | See also: 426 | 427 | [Kubernetes Configuration Best Practices](https://kubernetes.io/docs/concepts/configuration/overview/#container-images) 428 | 429 | #### Pod labels 430 | 431 | The component should have [the labels that suggested by Kubernetes](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/#labels), for example, `app.kubernetes.io/name`, `app.kubernetes.io/version`. The `app.kubernetes.io/part-of` is set to `cloudfoundry` and `app.kubernetes.io/component` set to component name. 432 | 433 | Improves: 434 | 435 | * [logging](#logging) 436 | * [diagnostic tooling](#diagnostics-tooling) 437 | 438 | #### Readiness probe 439 | 440 | The readiness probe for the main container must be always present. 441 | 442 | Improves: 443 | 444 | * [availability](#availability) 445 | * [upgrades](#upgrades) 446 | 447 | Reasons: 448 | 449 | * the pod won't serve traffic until it is ready. 450 | * the rolling upgrade will wait for the pod to come up before deleting the existing pod. 451 | * if [pod disruption budget](#pod-disruption-budget) is set, the node draining will proceed only if pods are ready. 452 | 453 | See also: 454 | 455 | [Kubernetes documentation about probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-readiness-probes) 456 | 457 | #### Liveness probe 458 | 459 | The liveness probe should only fail if the application is in an unrecoverable state and has to be restarted. Ideally, the liveness probe should not be set and the application should crash. If it is present, it should point to a different endpoint than the readiness probe. 460 | 461 | Improves: 462 | 463 | * [availability](#availability) 464 | 465 | Reasons: 466 | 467 | * Failing liveness probe causes the restart of the pod - this might take more time if the pod is scheduled on the different node. 468 | * Crashing of the pod might increase pressure on the running pods. 469 | 470 | See also: 471 | 472 | * [Outage report](https://keepingitclassless.net/2018/12/december-4-nre-labs-outage-post-mortem/) 473 | * [Liveness probes are dangerous](https://srcco.de/posts/kubernetes-liveness-probes-are-dangerous.html) 474 | 475 | #### Number of containers 476 | 477 | The pod must have as little containers as possible. Ideally, a single pod should have a single container. Most of native Kubernetes deployments have up to three containers. 478 | 479 | Improves: 480 | 481 | * [upgrades](#upgrades) 482 | * [availability](#availability) 483 | * [failure recovery](#failure-recovery) 484 | * [isolation](#isolation) 485 | * [resource planning](#resource-planning) 486 | 487 | Reasons: 488 | 489 | * All the containers are scheduled on a single node. They require more resources and slow down the start of the pod. 490 | * When one container is down, the pod won't be processing requests. 491 | * When the configuration is changed for a single container, the whole pod has to restart. 492 | 493 | #### Number of init containers 494 | 495 | The non-short-living pod must have up as little init containers as possible. Most of native Kubernetes deployments to 2 init containers in its spec. 496 | 497 | Improves: 498 | 499 | * [upgrades](#upgrades) 500 | * [failure recovery](#failure-recovery) 501 | 502 | Reasons: 503 | 504 | * The pod executes init containers sequentially and this increases the time for the pod to start. Consider using Kubernetes jobs instead. 505 | * Crashes in init containers do not show up in pod crash count. 506 | * The logs from init containers is impossible to get from the pod after the pod is started. 507 | 508 | #### Pod requests 509 | 510 | Each container in a pod must always have configurable CPU & memory requests with sane defaults(required to run 50 applications /start 5 at the same time) 511 | 512 | Improves: 513 | 514 | * [Resource Planning](#resource-planning) 515 | * [Availability](#availability) 516 | 517 | Reasons: 518 | 519 | * Kubernetes blocks the resources on the node and does not allow to schedule more applications that are possible. Overcommitting slows down the deployment. 520 | 521 | See all: 522 | 523 | [Managing Compute Resources](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/) 524 | 525 | #### Pod limits 526 | 527 | Memory limits are optional, but if they are present they must be at least 50% bigger than requests. CPU limits must be never set. 528 | 529 | Improves: 530 | 531 | * [Resource planning](#resource-planning) 532 | 533 | Reasons: 534 | 535 | * [CPU limits are broken now](https://github.com/kubernetes/kubernetes/issues/67577) 536 | * Pod will restart if it uses more memory than allowed 537 | 538 | See also: 539 | 540 | * [KubeCon presentation](https://www.youtube.com/watch?v=UE7QX98-kO0) 541 | 542 | #### Pod service account 543 | 544 | Each component must have its own service account. It must never use the `default` service account. 545 | 546 | Improves: 547 | 548 | * [security](#security) 549 | * [customisation](#customisation) 550 | 551 | Reason: 552 | 553 | This allows attaching [pod security policy](#pod-security-policy) to the pod. 554 | 555 | #### Service account token 556 | 557 | If the pod does not need access to the Kubernetes API, the service account token is not mounted to it 558 | 559 | Improves: 560 | 561 | * [security](#security) 562 | 563 | #### Pod security configuration 564 | 565 | The pod spec should satisfy [the restricted pod security policy provided by Kubernetes](https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/policy/restricted-psp.yaml) 566 | 567 | * Pod should drop all capabilities. 568 | * Pod should have proper `seccomd` or `apparmor` annotation. 569 | * Pod should have property readOnlyRootFilesystem=readOnlyRootFilesystem. 570 | * Pod should set `securityContext.runAsNonRoot` property. 571 | 572 | Improves: 573 | 574 | * [security](#security) 575 | 576 | See also: 577 | 578 | * [Pod security policies](#pod-security-policy) 579 | * [Non-root user for container](#non-root-user) 580 | 581 | #### Using certificates 582 | 583 | If a pod requires TLS/SSL cert/keys for public consumption it must support utilising cert-manager. 584 | 585 | Improves: 586 | 587 | * [customisation](#customisation) 588 | 589 | Reason: 590 | 591 | Kubernetes secrets have a special format for certificates and the operators expect the components to use it. 592 | 593 | #### Pod port names 594 | 595 | Ports that are exposed by pod must have a name which should be the same as in the corresponding service 596 | 597 | Improves: 598 | 599 | * [customisation](#customisation) 600 | 601 | Reason: 602 | 603 | Istio requires proper [port names](https://istio.io/docs/ops/deployment/requirements/). 604 | 605 | #### Affinity 606 | 607 | The specification allows setting affinity and anti-affinity rules. 608 | 609 | Improves: 610 | 611 | * [availability](#availability) 612 | * [isolation](#isolation) 613 | * [resource planning](#resource-planning) 614 | 615 | See also: 616 | 617 | [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/) 618 | 619 | ### Service specification 620 | 621 | #### Using services 622 | 623 | All pods should be part of services. 624 | 625 | Reason: 626 | 627 | * [Istio requirements](https://istio.io/docs/ops/deployment/requirements/). 628 | 629 | #### Service pod names 630 | 631 | The component creates a service if it has to be accessed by other components. Service ports should have the name of format `[-]`, e.g. `grpc-api` or `tcp-database`. See more in Istio documentation. 632 | 633 | Reason: 634 | 635 | * [Istio requirements](https://istio.io/docs/ops/deployment/requirements/). 636 | 637 | #### Service labels 638 | 639 | The service must have the same labels as a [pod](#pod-labels) 640 | 641 | Improves: 642 | 643 | * [logging](#logging) 644 | * [diagnostics](#diagnostics-tooling) 645 | 646 | ### Other Kubernetes objects 647 | 648 | #### Component Object Labels 649 | 650 | Each Kubernetes object that is deployed should have the same base labels [as the pod](#pod-labels) this object will be attached to. 651 | 652 | Improves: 653 | 654 | * [logging](#logging) 655 | * [diagnostics](#diagnostics-tooling) 656 | 657 | #### Pod Disruption Budget 658 | 659 | If the process is expected to have no downtime, it has PodDisruptionBudget 660 | 661 | Improves: 662 | 663 | * [Kubernetes cluster upgrades](#upgrades) 664 | 665 | Reason: 666 | 667 | Metadata from a higher level is ignored when the node is drained. The pod disruption budget prevents too many pods from going down. 668 | 669 | See also: 670 | 671 | [Kubernetes documentation about disruptions](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/) 672 | 673 | #### Pod Security Policy 674 | 675 | Minimal pod security policy is provided. Ideally, it should be the same (or stricter) as [Kubernetes provided policy](https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/policy/restricted-psp.yaml) 676 | 677 | Improves: 678 | 679 | * [Security](#security) 680 | 681 | #### Networking policy 682 | 683 | Sample networking policy is provided 684 | 685 | Improves: 686 | 687 | * [Security](#security) 688 | * [Customisation](#customisation) 689 | * [Open Source](#open-source) 690 | 691 | #### Istio RBAC 692 | 693 | Sample Istio RBAC is provided 694 | 695 | Improves: 696 | 697 | * [Security](#security) 698 | * [Customisation](#customisation) 699 | * [Open Source](#open-source) 700 | 701 | #### Access outside the cluster 702 | 703 | If the component has to be accessed externally, it writes a K8s Ingress resource or a set of Istio VirtualService + Gateway resourcesproviders ingress with free form annotations and the ability to provide a load balancer 704 | 705 | Improves: 706 | 707 | * [Customisation](#customisation) 708 | * [Open Source](#open-source) 709 | 710 | #### Using secrets 711 | 712 | If the component needs some secrets, it has an option to use an existing Kubernetes object with the predefined format. The Kubernetes object name can be provided by the platform engineer. 713 | The secret for certificates uses known K8s format 714 | 715 | Improves: 716 | 717 | * [Customisation](#customisation) 718 | * [Open Source](#open-source) 719 | 720 | #### Service accounts 721 | 722 | Each component creates and attached its own service account. 723 | 724 | Improves: 725 | 726 | * [Security](#security) 727 | * [Customisation](#customisation) 728 | 729 | #### Deployment 730 | 731 | Each stateless component is deployed as a deployment 732 | 733 | #### Replicas count 734 | 735 | The number of replicas is not specified in the template unless it can only be deployed as a single copy. 736 | 737 | Improves: 738 | 739 | * [Availability](#availability) 740 | 741 | ### Work with other components 742 | 743 | #### Optional parts 744 | 745 | If the component has a soft dependency(can work without it) on another component, the depending part can be skipped. i.e. Eirini deploys with rootfs patcher, but rootfs patcher can be skipped in the deployment. 746 | 747 | #### Using custom DNS addresses 748 | 749 | The address for the dependent component can always be specified by the platform engineer and has a sane default 750 | 751 | Improves: 752 | 753 | * [Customisation](#customisation) 754 | 755 | Reason: 756 | 757 | Integration with KubeCF and CF-4-K8s requires using different DNS addresses. 758 | 759 | ### Documentation 760 | 761 | Each non-alpha property that platform engineer can specify is documented in README. 762 | 763 | Improves: 764 | 765 | * [Customisation](#customisation) 766 | 767 | ### Kubernetes versions support 768 | 769 | Each component is expected to support all supported by CNCF versions of Kubernetes by using correct API specification. 770 | 771 | Improves: 772 | 773 | * [upgrades](#upgrades) 774 | 775 | Reason: 776 | 777 | APIs get deprecated and has to be fixed in advance. 778 | 779 | #### Container runtime support 780 | 781 | The component must support both Docker and containerd as container runtimes. 782 | 783 | Improves: 784 | 785 | * [Kubernetes cluster customisation](#customisation) 786 | 787 | Reason: 788 | 789 | Docker is widely used, containerd implements CRI(container runtime interface) and is used in several public cloud offerings. 790 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Kubernetes-idiomatic Cloud Foundry components guidelines 2 | 3 | See the description for each guideline in [DETAILED document](DETAILED.md) 4 | 5 | ## Code 6 | 7 | * Every component must have an endpoint with health information or crash if it is unhealthy. 8 | * All logs must go in stdout/stderr 9 | * The component must be able to use up-to-date configuration 10 | * The application with version n-1 should be able to start with the database that has been migrated to version n 11 | * The application must respect SIGTERM signal and start sending NotReady probe 12 | * The component is either expected to work during K8s control plane downtime or has a clear notice README that to achieve high availability, the control plane of Kubernetes must have multiple replicas. 13 | * Kubernetes resources that created by a component should have a label `app.kubernetes.io/managed-by` with value `cloudfoundry` 14 | 15 | ## Packaging 16 | 17 | * All components should run with a non-root user(but not the user with UID 1337) unless it is completely impossible. Even “FROM scratch” images 18 | * All components images should have labels in the metadata with repo URL and SHA of the commit it the metadata 19 | * All components should have the possibility to change the base image 20 | * The default base image should come from `cloudfoundry/stacks` 21 | * The run image should not have packages required for compilation, only for running. i.e. don’t have Go package or JDK in the final image, For java, only JRE should be shipped 22 | * Images are continuously updated with the new version of the base layer. 23 | * Images are stored in the CFF organisation under Dockerhub 24 | * The component has clear instructions on how to build its container image 25 | 26 | ## Pod specification 27 | 28 | * The image references must always include sha256 29 | * The component should have the labels that suggested by Kubernetes such as `app.kubernetes.io/name`, `app.kubernetes.io/version` and `app.kubernetes.io/component`. The `app.kubernetes.io/part-of` is set to Cloud Foundry 30 | * The readiness probe for the main container must be always present 31 | * Liveness probe if present should point to a different endpoint than the readiness probe. Ideally, something that does not require any processing 32 | * In general, a single pod should have a single container. There are might be edge-cases, but the pod shouldn’t have more than 5 containers in its spec 33 | * The long-living pod can have up to 2 init containers in its spec 34 | * Each container in a pod must always have configurable CPU & memory requests with sane defaults(required to run 50 applications /start 5 at the same time) 35 | * Memory limits are optional, but if they are present they must be at least 50% bigger than requests 36 | * CPU limits must be never set 37 | * Each component must have its service account. It must never use the `default` service account 38 | * If the pod does not need access to the Kubernetes API, the service account token is not mounted to it 39 | * The pod spec should satisfy [the restricted pod security policy provided by Kubernetes](https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/policy/restricted-psp.yaml) 40 | * Pod should drop all capabilities 41 | * Pod should have proper `seccomd` or `apparmor` annotation 42 | * Pod should have property readOnlyRootFilesystem=readOnlyRootFilesystem 43 | * If a pod requires TLS/SSL cert/keys for public consumption it must support utilising cert-manager. 44 | * Ports that are exposed by pod must have a name which should be the same as in the corresponding service 45 | * The specification allows setting affinity and anti-affinity rules 46 | 47 | ## Service specification 48 | 49 | * The component creates a service for its pods. 50 | * Service ports should have the name of format `[-]`, e.g. `grpc-api` or `tcp-database`. See more in Istio documentation 51 | * The service must have the same labels as a pod 52 | 53 | ## Other Kubernetes objects 54 | 55 | * If the process is expected to have no downtime it has PodDisruptionBudget 56 | * Minimal pod security policy is provided. Ideally, it should be the same (or stricter) as [Kubernetes provided policy](https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/policy/restricted-psp.yaml) 57 | * Sample networking policy is provided 58 | * Sample Istio RBAC is provided 59 | * If the component has to be accessed externally, it writes a K8s Ingress resource or a set of Istio VirtualService + Gateway resources provides ingress with free form annotations and the ability to provide a load balancer 60 | * If the component needs some secrets, it has an option to use an existing Kubernetes object with the predefined format. The Kubernetes object name can be provided by the platform engineer. 61 | * The secret for certificates uses known K8s format 62 | * Each stateless component is deployed as a deployment 63 | * The number of replicas is not specified in the template unless it can only be deployed as a single copy. 64 | * Each component creates and attached its own service account. 65 | 66 | ## Work with other components 67 | 68 | * If the component has a soft dependency(can work without it) on another component, the depending part can be skipped. i.e. Eirini deploys with rootfs patcher, but rootfs patcher can be skipped in the deployment. 69 | * The address for the dependent component can always be specified by the platform engineer and has a sane default 70 | 71 | ## Documentation 72 | 73 | * Each non-alpha property that platform engineer can specify is documented in README 74 | 75 | ## Kubernetes versions support 76 | 77 | * Each component is expected to support all supported by CNCF versions of Kubernetes by using correct API specification. 78 | * If API specification differs in version x and x+2, the component has by default the version that is supported in CFCR. Optionally, it can provide a flag to use a different API version 79 | * The component must support both Docker and containerd as container runtimes. 80 | --------------------------------------------------------------------------------