├── .gitignore ├── CODEOWNERS ├── LICENSE ├── README.md ├── alignment.md ├── architecture.md ├── architecture_diagrams.pptx ├── demos ├── operator-demo │ └── README.md └── ssh-demo │ ├── Dockerfile │ ├── README.md │ ├── aa-offline_fs_kbc-keys.json │ ├── ccv0-ssh │ ├── cri-container-config.yaml │ ├── cri-sandbox-config.yaml │ └── k8s-cc-ssh.yaml ├── guides ├── eaa_verdictd.md ├── enclave-cc.md ├── ephemeral_storage.md ├── nontee_demo.md ├── sev.md └── troubleshooting.md ├── images ├── ApplicationTEEProtection.png ├── CC_SGX_container.jpg ├── CC_SGX_container.png ├── CC_TEE_VM.jpg ├── CC_TEE_container.jpg ├── CC_TEE_container.png ├── CC_all_blocks.jpg ├── COCO_ccv1_TEE.png ├── COCO_ccv1_enclave.png ├── COCO_ccv1_peerpods_TEE.png ├── CloudNativeTEEProtection.png ├── RoadmapNov2021.jpg ├── RoadmapOct2021.jpg ├── RoadmapSept2021.jpg ├── coco_logo.png └── persona_model.svg ├── overview.md ├── quickstart.md ├── releases ├── v0.1.0.md ├── v0.2.0.md ├── v0.3.0.md ├── v0.4.0.md ├── v0.5.0.md └── v0.6.0.md ├── roadmap.md ├── threats_overview.md ├── trust_model.md └── trust_model_personas.md /.gitignore: -------------------------------------------------------------------------------- 1 | /.idea 2 | /.DS_Store 3 | -------------------------------------------------------------------------------- /CODEOWNERS: -------------------------------------------------------------------------------- 1 | # This is a comment. 2 | # Each line is a file pattern followed by one or more owners. 3 | 4 | # These owners will be the default owners for everything in 5 | # the repo. Unless a later match takes precedence, 6 | # @global-owner1 and @global-owner2 will be requested for 7 | # review when someone opens a pull request. 8 | * @confidential-containers/documentation-maintainers 9 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # THIS REPOSITORY IS ARCHIVED. 4 | 5 | # Please switch to [confidential-containers repository](https://github.com/confidential-containers/confidential-containers) 6 | 7 | ---- 8 | 9 | 10 | ## Confidential Containers Documentation 11 | 12 | Confidential Containers is an open source community working to leverage 13 | [Trusted Execution Environments](https://en.wikipedia.org/wiki/Trusted_execution_environment) 14 | to protect containers and data and to deliver cloud native 15 | confidential computing. 16 | 17 | **We have a new release every 6 weeks!** 18 | See [Release Notes](./releases/) or [Quickstart Guide](./quickstart.md) 19 | 20 | Our key considerations are: 21 | - Allow cloud native application owners to enforce application security requirements 22 | - Transparent deployment of unmodified containers 23 | - Support for multiple TEE and hardware platforms 24 | - A trust model which separates Cloud Service Providers (CSPs) from guest applications 25 | - Least privilege principles for the Kubernetes cluster administration capabilities which impact 26 | delivering Confidential Computing for guest applications or data inside the TEE 27 | 28 | ## Further Detail 29 | 30 | [![asciicast](https://asciinema.org/a/eGHhZdQY3uYnDalFAfuB7VYqF.svg)](https://asciinema.org/a/eGHhZdQY3uYnDalFAfuB7VYqF) 31 | [![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fconfidential-containers%2Fdocumentation.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2Fconfidential-containers%2Fdocumentation?ref=badge_shield) 32 | 33 | - [Project Overview](./overview.md) 34 | - [Project Architecture](./architecture.md) 35 | - [Our Roadmap](./roadmap.md) 36 | - [Alignment with other Projects](alignment.md) 37 | 38 | 39 | ### Associated Repositories 40 | - [Kubernetes Operator for Confidential Computing](https://github.com/confidential-containers/confidential-containers-operator) 41 | : An operator to deploy confidential containers runtime (and required configs) on a Kubernetes cluster 42 | 43 | 44 | ## License 45 | [![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fconfidential-containers%2Fdocumentation.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fconfidential-containers%2Fdocumentation?ref=badge_large) 46 | -------------------------------------------------------------------------------- /alignment.md: -------------------------------------------------------------------------------- 1 | # Confidential Containers and Other Projects 2 | 3 | Confidential Containers are connected to a wide array of projects. Some projects are directly implicated in the design of Confidential Containers. Others might be a good pairing for Confidential Containers, and others might be transparently supported or consumed by or via Confidential Containers. If work needs to be done to align a project with Confidential Containers, it will generally involve reconciling the strict trust model of Confidential Containers and the tradional cloud native trust model. 4 | 5 | ## Directly Related 6 | 7 | Given that Confidential Containers are deployed via containerd/cri-o and Kubernetes, these projects are directly implicated. With Confidential Containers, container images are pulled inside the enclave and are essentially not present on the host. Some changes will be needed higher up the stack to accommodate this. Work is already ongoing in at least one of these communities (containerd) to support Confidential Containers. 8 | 9 | ## Potentially Related 10 | 11 | These projects might pair nicely with Confidential Containers, but are not required. 12 | 13 | ### SPIFFE/SPIRE 14 | 15 | [SPIFFE and SPIRE](https://spiffe.io/) can be used to assign a provable identity to every workload. With some adjustments this might be a good way to deliver secrets to confidential containers or even the containers themselves. The attestation process would most likely have to be linked to platform-specific hardware attestation, perhaps in conjunction with the Attestation Agent. The SPIRE agent would need to be moved inside the enclave and the SPIRE server could not be provided by the Cloud Service Provider (CSP). Further investigation is encouraged. 16 | 17 | ### Service Mesh 18 | 19 | A service mesh ([Istio](https://istio.io/), [Envoy](https://www.envoyproxy.io/)) could be a good way to extend the security guarantees of Confidential Containers to the network, essentially by ensuring that traffic is encrypted prior to leaving the enclave. The service mesh would need to be configured such that the CSP could not manipulate sensitive guest network policies. In the case of Istio, the Istio Daemon has a broad authority to reconfigure networking rules. This would break the trust model of Confidential Containers. 20 | 21 | ### Observability 22 | 23 | With Confidential Containers, logs must be carefully configured to avoid exposing Confidential information to the CSP. Locking down the workload logging, monitoring, and tracing difficult. The CNCF has many projects that standardize or centralize logs. These include [Fluentd](https://www.fluentd.org/), [Jaeger](https://www.jaegertracing.io/), [Open Telemetry](https://opentelemetry.io/), [Thanos](https://thanos.io/), and [Cortex](https://cortexmetrics.io/). Perhaps one of these could be used to securely deliver confidential logs to the guest owner. More investigation is needed 24 | 25 | ## Consumable 26 | 27 | Since Confidential Containers are a drop-in replacement for traditional containers, Confidential Containers should be able to consume or support a wide range of container-related projects including container registries like [Harbor](https://goharbor.io/) or [Dragonfly](https://d7y.io/). Tools for defining applications or creating build pipelines should work with minimal modification. Of course any services that can be deployed as containers should also work as Confidential Containers with new security properties. 28 | -------------------------------------------------------------------------------- /architecture.md: -------------------------------------------------------------------------------- 1 | # Confidential Computing Overview 2 | 3 | > Confidential Computing protects data in use by performing computation in a 4 | > hardware-based Trusted Execution Environment. 5 | > These secure and isolated environments prevent unauthorized access or 6 | > modification of applications and data while they are in use, thereby increasing 7 | > the security level of organizations that manage sensitive and regulated data. 8 | > 9 | > A Trusted Execution Environment (TEE) is commonly defined as an environment that 10 | > provides a level of assurance of data integrity, data confidentiality, and code 11 | > integrity. A hardware-based TEE uses hardware-backed techniques to provide 12 | > increased security guarantees for the execution of code and protection of data 13 | > within that environment. 14 | 15 | Key Characteristics of Confidential Computing, as defined by the 16 | [Confidential Computing Consortium](https://confidentialcomputing.io/wp-content/uploads/sites/85/2021/03/confidentialcomputing_outreach_whitepaper-8-5x11-1.pdf): 17 | 18 | - **Data confidentiality:** Unauthorized entities cannot view data while it is 19 | in use within the TEE 20 | - **Data integrity:** Unauthorized entities cannot add, remove, or alter data 21 | while it is in use within the TEE 22 | - **Code integrity:** Unauthorized entities cannot add, remove, or alter code 23 | executing in the TEE 24 | 25 | # Confidential Computing Building Blocks 26 | 27 | ## Key Components 28 | 29 | The following diagram shows how the different building blocks for Confidential 30 | Computing (CC) come together: 31 | 32 | ![CC_all_blocks](./images/CC_all_blocks.jpg) 33 | 34 | Let’s start by describing the lower level blocks of the CC solution 35 | (**colored blue**): 36 | 37 | - **Infra layer** - be it on-premises (bare metal, VMware etc.) or public clouds 38 | (AWS, Azure, GCP, etc.) 39 | - **Hardware with CC support** - this includes 2 distinct models: 40 | - **VM-based TEEs** - In this model, memory is encrypted along a 41 | traditional VM boundary running on top of a VMM. AMD SEV-SNP, Intel TDX, 42 | IBM Secure Execution and Protected Execution Functionality (PEF) are 43 | examples of VM-based TEEs. 44 | - **Process-based TEEs** - In this model, a process that needs to run 45 | securely is divided into two components: trusted and untrusted. 46 | The trusted component resides in encrypted memory and handles confidential 47 | computing, while the untrusted component interfaces with the operating 48 | system and propagates I/O from encrypted memory to the rest of the system. 49 | Intel SGX is an example of a process-based TEE. 50 | - **Hypervisor** - this includes hypervisors such as QEMU/KVM, cloud hypervisor, 51 | public cloud provider hypervisors etc... 52 | 53 | The **Confidential Computing services block** contains a number of services 54 | (**colored pink**) which are required for creating a holistic CC platform which 55 | the customer can then use to build their solution. This currently includes the 56 | following services: 57 | 58 | - **Attestation service** - The primary purpose of the attestation service is 59 | to validate the evidence provided by the hardware TEE. This is the *Verifier*, 60 | as defined in the [RATS architecture](https://www.rfc-editor.org/rfc/rfc9334). 61 | - **Key Broker Service (KBS)** - The KBS is the *Relying Party*, as defined by 62 | the [RATS architecture](https://www.rfc-editor.org/rfc/rfc9334). 63 | Following are its primary functions: 64 | - Receive evidence from the *Attester* (confidential VM or container) via a 65 | challenge-response protocol. 66 | - Relay the evidence to the Attestation Service for verification. 67 | - Apply appraisal policy for the returned Attestation Results to assess the 68 | trustworthiness of the *Attester*. 69 | - Interact with the Key Management Service to retrieve the keys and then 70 | send them back to the *Attester*. 71 | - **Key management service** - A service for securely storing, managing and 72 | backing up of cryptographic keys used by applications and users. 73 | - **Image build service** - Services used to build confidential containers or VM 74 | images for end users. 75 | - **Image registry** - A service that is used to store encrypted and/or signed 76 | container and VM images required for CC workloads. Examples of such registries 77 | include [Quay.io](https://quay.io/), [Docker Hub](https://hub.docker.com/), 78 | CSPs provided registries, etc. 79 | 80 | Both the **Confidential VM** and the **Confidential Containers** block contain 81 | the **Enclave Software Stack**. This stack is made of the trusted CC components 82 | responsible for collectively running the whole attestation flows, and for 83 | example: 84 | - Unblocking a confidential VM boot process after injecting a guest owned secret 85 | released by the relying party. 86 | - Decrypting and unpacking a protected container image, and running its 87 | associated workload. 88 | 89 | ## Confidential Container Stack 90 | 91 | The CC components can be arranged together and configured to provide 92 | containerized workloads with the Confidential Computing security guarantees. 93 | 94 | The two following sections describe a generic software architecture for 95 | Kubernetes pods to run in either VM-based or process-based TEEs. 96 | 97 | Although internal implementations for the two approaches differ, they share the 98 | same goals and attributes: 99 | 100 | - Remove cloud and infrastructure providers from the guest application Trusted 101 | Computing Base (TCB). 102 | - Integrate natively with the Kubernetes control plane. 103 | - Provide an unmodified Kubernetes user and developer experience. 104 | - Deploy unmodified workloads. 105 | 106 | ### VM-based TEE 107 | 108 | VM-based TEEs (e.g. AMD SEV, IBM SE or Intel TDX) can be used to build a 109 | confidential containers software architecture: 110 | 111 | ![CC_TEE_container](./images/CC_TEE_container.png) 112 | 113 | 114 | Following is the workflow when deploying a Kubernetes pod with VM-based TEEs: 115 | 116 | - CC workload preparation 117 | - User builds the container image(s) (e.g. with tools like `podman`). 118 | - User signs/encrypts the container image(s). 119 | - User pushes the container image(s) to the image registry. 120 | - Deploying the CC workload in k8s 121 | - User deploys the workload (`kubectl apply -f cc_workload.yaml`). 122 | - Kubernetes schedules the workload to target host having the required 123 | capability to run confidential containers. 124 | - CC workload execution flow (**red connector** in the diagram) 125 | - Confidential containers runtime on the host starts the VM TEE (The enclave). 126 | - Enclave (agent) performs remote attestation: **steps 1-2 in the diagram**. 127 | - Enclave (agent) gets the keys required to verify/decrypt the containers 128 | image(s): **steps 3-4 in the diagram**. 129 | - Enclave (image management) downloads the container image(s) : **step 5 in 130 | the diagram**. 131 | - Enclave verifies/decrypts the container image(s) : **step 6 in the diagram**. 132 | - Enclave starts the container workload. 133 | 134 | ### Process-based TEE 135 | 136 | The confidential containers software architecture can also be built on top of 137 | process-based TEEs like e.g. Intel SGX: 138 | 139 | ![CC_SGX_container](./images/CC_SGX_container.png) 140 | 141 | Following is the workflow when deploying a Kubernetes pod with a process-based 142 | TEEs. The main differences from the VM-based TEE approach are the last 3 steps 143 | involving interaction between 2 enclave processes: 144 | 145 | - CC workload preparation 146 | - User builds the container image(s) (e.g. with tools like `podman`). 147 | - User signs/encrypts the container image(s). 148 | - User pushes the container image(s) to the image registry. 149 | - Deploying the CC workload in k8s 150 | - User deploys the workload (`kubectl apply -f cc_workload.yaml`). 151 | - Kubernetes schedules the workload to target host having the required 152 | capability to run confidential containers. 153 | - CC workload execution flow (**red connector** in the diagram) 154 | - Confidential containers runtime on the host starts the enclave agent. 155 | - Enclave (agent) performs remote attestation: **steps 1-2 in the diagram**. 156 | - Enclave (agent) gets the keys required to verify/decrypt the containers 157 | image(s): **steps 3-4 in the diagram**. 158 | - Enclave (image management) downloads the container image(s) : **step 5 in 159 | the diagram**. 160 | - Enclave verifies, decrypts and writes the container image(s) to a local 161 | encrypted filesystem: **step 6 in the diagram**. 162 | - The runtime starts the app enclave which reads the container bundle from the 163 | encrypted filesystem. 164 | - Secure use of the encrypted filesystem is facilitated by a key exchange 165 | between the agent and app enclaves using either sealing or local attestation. 166 | 167 | As can be seen, the flows for process and VM-based TEEs are almost identical. It 168 | should be noted that process-based TEE requires a few additional software 169 | components like `libOS` without which the application requires re-architecting. 170 | This is not the case for VM-based TEEs. Conversely, Process based TEEs do not 171 | require a separate VM and CC-aware hypervisor. 172 | 173 | The up-to-date design for the Confidential Containers process-based architecture 174 | is maintained in the [enclave-cc documentation](https://github.com/confidential-containers/enclave-cc/blob/main/docs/design.md). 175 | 176 | # CNCF Confidential Containers 177 | 178 | The CNCF Confidential Containers project is an implementation of the 179 | confidential containers architecture described in the previous section. 180 | 181 | It relies on several major cloud native components like `containerd`, both the 182 | `Kata Containers` and the `enclave-cc` runtimes, or the `ocicrypt` container 183 | image encryption APIs. 184 | It also depends on the standard Linux virtualization stack, including the `KVM` 185 | hypervisor and open source VMMs like `QEMU` or `cloud-hypervisor`. 186 | 187 | The project supports both VM-based and process-based TEEs, with the key 188 | objective to reuse the generic components around enclave agent and attestation 189 | services across both TEE models. 190 | 191 | It is a [Cloud Native Computing Foundation Sandbox](https://www.cncf.io/projects/confidential-containers/) 192 | project. 193 | 194 | The following diagram shows the upcoming v1 architecture to run Confidential 195 | Containers using VM-based TEEs and the Kata Containers runtime: 196 | 197 | ![COCO_ccv1_TEE](./images/COCO_ccv1_TEE.png) 198 | 199 | The following diagram shows the upcoming v1 architecture to run Confidential 200 | Containers using VM-based TEEs by leveraging the peer-pods approach. This relies on Kata Containers remote hypervisor support and the [cloud-api-adaptor](https://github.com/confidential-containers/cloud-api-adaptor/) project: 201 | 202 | ![COCO_ccv1_TEE](./images/COCO_ccv1_peerpods_TEE.png) 203 | 204 | The following diagram shows the upcoming v1 architecture to run Confidential 205 | Containers using the Intel SGX process-based TEE. It relies on the 206 | [enclave-cc](https://github.com/confidential-containers/enclave-cc) project: 207 | 208 | ![COCO_ccv1_enclave](./images/COCO_ccv1_enclave.png) 209 | -------------------------------------------------------------------------------- /architecture_diagrams.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/confidential-containers/documentation/8e55631d899d83e34b99792150669968a037d17f/architecture_diagrams.pptx -------------------------------------------------------------------------------- /demos/operator-demo/README.md: -------------------------------------------------------------------------------- 1 | 2 | - [CCv0 Operator Demo](#ccv0-operator-demo) 3 | * [Demo Video](#demo-video) 4 | * [Demo Environment setup](#demo-environment-setup) 5 | + [Kubernetes cluster](#kubernetes-cluster) 6 | + [Replace containerd ](#replace-containerd) 7 | + [Verify if the cluster nodes are all up](#verify-if-the-cluster-nodes-are-all-up) 8 | * [Operator Setup](#operator-setup) 9 | * [Confidential Containers Runtime setup](#confidential-containers-runtime-setup) 10 | 11 | 12 | # CCv0 Operator Demo 13 | 14 | ## Demo Video 15 | 16 | The demo shows CCv0 Kata runtime installation and configuration using the [operator](https://github.com/confidential-containers/operator). 17 | 18 | [Watch the demo in youtube](https://www.youtube.com/watch?v=4cM3IhfnJLQ) 19 | 20 | 21 | ## Demo Environment setup 22 | 23 | ### Kubernetes cluster 24 | 25 | Setup a two nodes Kubernetes cluster using Ubuntu 20.04. 26 | You can use your preferred Kubernetes setup tool. Here is an example 27 | using [kcli](https://kcli.readthedocs.io/en/latest/) 28 | 29 | Download ubuntu 20.04 image if not present 30 | ``` 31 | kcli download image ubuntu2004 32 | ``` 33 | 34 | Install the cluster 35 | ``` 36 | kcli create kube generic -P image=ubuntu2004 -P workers=1 testk8s 37 | ``` 38 | 39 | ### Replace containerd 40 | 41 | Replace containerd on the worker node by building a new containerd from the following 42 | branch - https://github.com/confidential-containers/containerd/tree/ali-CCv0 43 | 44 | Modify systemd configuration to use the new binary and restart containerd and kubelet 45 | 46 | ### Verify if the cluster nodes are all up 47 | 48 | ``` 49 | kubectl get nodes 50 | ``` 51 | Sample output from my demo environment 52 | ``` 53 | NAME STATUS ROLES AGE VERSION 54 | cck8s-demo-master-0 Ready control-plane,master 25d v1.22.3 55 | cck8s-demo-worker-0 Ready worker 25d v1.22.3 56 | ``` 57 | 58 | ## Operator Setup 59 | 60 | ``` 61 | kubectl apply -f https://raw.githubusercontent.com/confidential-containers/operator/ccv0-demo/deploy/deploy.yaml 62 | ``` 63 | 64 | The operator installs everything under the `confidential-containers-system` namespace 65 | 66 | Verify if the operator is running 67 | 68 | ``` 69 | kubectl get pods -n confidential-containers-system 70 | ``` 71 | 72 | Sample output from my demo environment 73 | ``` 74 | NAME READY STATUS RESTARTS AGE 75 | cc-operator-controller-manager-7f8d6dd988-t9zdm 2/2 Running 0 13s 76 | ``` 77 | 78 | ## Confidential Containers Runtime setup 79 | 80 | Creating a `CCruntime` object sets up the container runtime 81 | The default payload image sets up the CCv0 demo image of the kata-containers runtime. 82 | 83 | ```yaml 84 | cat << EOF | kubectl create -f - 85 | apiVersion: confidentialcontainers.org/v1beta1 86 | kind: CcRuntime 87 | metadata: 88 | name: ccruntime-sample 89 | namespace: confidential-containers-system 90 | spec: 91 | # Add fields here 92 | runtimeName: kata 93 | config: 94 | installType: bundle 95 | payloadImage: quay.io/confidential-containers/runtime-payload:ccv0-ssh-demo 96 | EOF 97 | ``` 98 | This will create an install daemonset targeting the worker nodes for installation. 99 | You can verify the status under the `confidential-containers-system` namespace 100 | 101 | ``` 102 | kubectl get pods -n confidential-containers-system 103 | NAME READY STATUS RESTARTS AGE 104 | cc-operator-controller-manager-7f8d6dd988-t9zdm 2/2 Running 0 82s 105 | cc-operator-daemon-install-p9ntc 1/1 Running 0 45s 106 | ``` 107 | 108 | On successful installation, you'll see the following `runtimeClasses` being setup 109 | 110 | ``` 111 | kubectl get runtimeclasses.node.k8s.io 112 | NAME HANDLER AGE 113 | kata kata 92s 114 | kata-cc kata-cc 92s 115 | kata-qemu kata-qemu 92s 116 | ``` 117 | `kata-cc` runtimeclass uses CCv0 specific configurations. 118 | 119 | Now you can deploy the PODs targeting the specific runtimeclasses. 120 | The [SSH demo](/demos/ssh-demo) can be used as a compatible workload. 121 | -------------------------------------------------------------------------------- /demos/ssh-demo/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM alpine:3.14 2 | RUN apk update && apk upgrade && apk add openssh-server 3 | RUN ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -P "" 4 | # A password needs to be set for login to work. An empty password is 5 | # unproblematic as password-based login to root is not allowed. 6 | RUN passwd -d root 7 | # Generate with `ssh-keygen -t ed25519 -f ccv0-ssh -P "" -C ""` 8 | COPY ccv0-ssh.pub /root/.ssh/authorized_keys 9 | ENTRYPOINT /usr/sbin/sshd -D 10 | -------------------------------------------------------------------------------- /demos/ssh-demo/README.md: -------------------------------------------------------------------------------- 1 | # SSH demo 2 | 3 | To demonstrate confidential containers capabilities, we run a pod with SSH public key authentication. 4 | 5 | Compared to the execution of and login to a shell on a pod, an SSH connection is cryptographically secured and requires a private key. 6 | It cannot be established by unauthorized parties, such as someone who controls the node. 7 | The container image contains the SSH host key that can be used for impersonating the host we will connect to. 8 | Because this container image is encrypted, and the key to decrypting this image is only provided in measurable ways (e.g. attestation or encrypted initrd), and because the pod/guest memory is protected, even someone who controls the node cannot steal this key. 9 | 10 | ## Using a pre-provided container image 11 | 12 | If you would rather build the image with your own keys, skip to [Building the container image](#building-the-container-image). 13 | The [operator](/demos/operator-demo) can be used to set up a compatible runtime. 14 | 15 | A demo image is provided at [docker.io/katadocker/ccv0-ssh](https://hub.docker.com/r/katadocker/ccv0-ssh). 16 | It is encrypted with [Attestation Agent](https://github.com/confidential-containers/attestation-agent)'s [offline file system key broker](https://github.com/confidential-containers/attestation-agent/tree/64c12fbecfe90ba974d5fe4896bf997308df298d/src/kbc_modules/offline_fs_kbc) and [`aa-offline_fs_kbc-keys.json`](./aa-offline_fs_kbc-keys.json) as its key file. 17 | The private key for establishing an SSH connection to this container is given in [`ccv0-ssh`](./ccv0-ssh). 18 | To use it with SSH, its permissions should be adjusted: `chmod 600 ccv0-ssh`. 19 | The host key fingerprint is `SHA256:wK7uOpqpYQczcgV00fGCh+X97sJL3f6G1Ku4rvlwtR0`. 20 | 21 | All keys shown here are for demonstration purposes. 22 | To achieve actually confidential containers, use a hardware trusted execution environment and **do not** reuse these keys. 23 | 24 | Continue at [Connecting to the guest](#connecting-to-the-guest). 25 | 26 | ## Building the container image 27 | 28 | The image built should be encrypted. 29 | To receive a decryption key at run time, the Confidential Containers project utilizes the [Attestation Agent](https://github.com/confidential-containers/attestation-agent). 30 | 31 | ### Generating SSH keys 32 | 33 | ```sh 34 | $ ssh-keygen -t ed25519 -f ccv0-ssh -P "" -C "" 35 | ``` 36 | 37 | generates an SSH key `ccv0-ssh` and the correspondent public key `ccv0-ssh.pub`. 38 | 39 | ### Building the image 40 | 41 | The provided [`Dockerfile`](./Dockerfile) expects `ccv0-sh.pub` to exist. 42 | Using Docker, you can build with 43 | 44 | ```sh 45 | $ docker build -t ccv0-ssh . 46 | ``` 47 | 48 | Alternatively, Buildah can be used (`buildah build` or formerly `buildah bud`). 49 | The SSH host key fingerprint is displayed during the build. 50 | 51 | ## Connecting to the guest 52 | 53 | A [Kubernetes YAML file](./k8s-cc-ssh.yaml) specifying the [`kata`](https://github.com/kata-containers/kata-containers) runtime is included. 54 | If you use a [self-built image](#building-the-container-image), you should replace the image specification with the image you built. 55 | The default tag points to an `amd64` image, an `s390x` tag is also available. 56 | With common CNI setups, on the same host, with the service running, you can connect via SSH with 57 | 58 | ```sh 59 | $ ssh -i ccv0-ssh root@$(kubectl get service ccv0-ssh -o jsonpath="{.spec.clusterIP}") 60 | ``` 61 | 62 | You will be prompted about whether the host key fingerprint is correct. 63 | This fingerprint should match the one specified above/displayed in the Docker build. 64 | 65 | `crictl`-compatible [sandbox](./cri-sandbox-config.yaml) and [container](./cri-container-config.yaml) configurations are also included, which forward the pod SSH port (22) to 2222 on the host (use the `-p` flag in SSH). 66 | -------------------------------------------------------------------------------- /demos/ssh-demo/aa-offline_fs_kbc-keys.json: -------------------------------------------------------------------------------- 1 | { 2 | "default/key/ssh-demo": "HUlOu8NWz8si11OZUzUJMnjiq/iZyHBJZMSD3BaqgMc=" 3 | } 4 | -------------------------------------------------------------------------------- /demos/ssh-demo/ccv0-ssh: -------------------------------------------------------------------------------- 1 | -----BEGIN OPENSSH PRIVATE KEY----- 2 | b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW 3 | QyNTUxOQAAACAfiGV2X4o+6AgjVBaY/ZR2UvZp84dVYF5bpNZGMLylQwAAAIhawtHJWsLR 4 | yQAAAAtzc2gtZWQyNTUxOQAAACAfiGV2X4o+6AgjVBaY/ZR2UvZp84dVYF5bpNZGMLylQw 5 | AAAEAwWYIBvBxQZgk0irFku3Lj1Xbfb8dHtVM/kkz/Uz/l2h+IZXZfij7oCCNUFpj9lHZS 6 | 9mnzh1VgXluk1kYwvKVDAAAAAAECAwQF 7 | -----END OPENSSH PRIVATE KEY----- 8 | -------------------------------------------------------------------------------- /demos/ssh-demo/cri-container-config.yaml: -------------------------------------------------------------------------------- 1 | metadata: 2 | name: ccv0-ssh 3 | image: 4 | image: docker.io/katadocker/ccv0-ssh 5 | -------------------------------------------------------------------------------- /demos/ssh-demo/cri-sandbox-config.yaml: -------------------------------------------------------------------------------- 1 | metadata: 2 | name: ccv0-ssh-pod 3 | hostname: ccv0 4 | port_mappings: 5 | - container_port: 22 6 | host_port: 2222 7 | -------------------------------------------------------------------------------- /demos/ssh-demo/k8s-cc-ssh.yaml: -------------------------------------------------------------------------------- 1 | kind: Service 2 | apiVersion: v1 3 | metadata: 4 | name: ccv0-ssh 5 | spec: 6 | selector: 7 | app: ccv0-ssh 8 | ports: 9 | - port: 22 10 | --- 11 | kind: Deployment 12 | apiVersion: apps/v1 13 | metadata: 14 | name: ccv0-ssh 15 | spec: 16 | selector: 17 | matchLabels: 18 | app: ccv0-ssh 19 | template: 20 | metadata: 21 | labels: 22 | app: ccv0-ssh 23 | spec: 24 | runtimeClassName: kata 25 | containers: 26 | - name: ccv0-ssh 27 | image: ghcr.io/confidential-containers/test-container:multi-arch-encrypted 28 | imagePullPolicy: Always 29 | -------------------------------------------------------------------------------- /guides/eaa_verdictd.md: -------------------------------------------------------------------------------- 1 | # EAA Verdictd Guide 2 | 3 | EAA is used to perform attestation at runtime and provide guest with confidential resources such as keys. 4 | It is based on [rats-tls](https://github.com/inclavare-containers/rats-tls). 5 | 6 | [Verdictd](https://github.com/inclavare-containers/verdictd) is the Key Broker Service and Attestation Service of EAA. 7 | The EAA KBC is an optional module in the attestation-agent at compile time, 8 | which can be used to communicate with Verdictd. 9 | The communication is established on the encrypted channel provided by rats-tls. 10 | 11 | EAA can now be used on Intel TDX and Intel SGX platforms. 12 | 13 | ## Create encrypted image 14 | 15 | Before build encrypted image, you need to make sure Skopeo and Verdictd(EAA KBS) have been installed: 16 | - [Skopeo](https://github.com/containers/skopeo): the command line utility to perform encryption operations. 17 | - [Verdictd](https://github.com/inclavare-containers/verdictd): EAA Key Broker Service and Attestation Service. 18 | 19 | 1. Pull unencrypted image. 20 | 21 | Here use `alpine:latest` for example: 22 | 23 | ```sh 24 | ${SKOPEO_HOME}/bin/skopeo copy --insecure-policy　docker://docker.io/library/alpine:latest oci:busybox 25 | ``` 26 | 27 | 2. Follow the [Verdictd README #Generate encrypted container image](https://github.com/inclavare-containers/verdictd#generate-encrypted-container-image) to encrypt the image. 28 | 29 | 3. Publish the encrypted image to your registry. 30 | 31 | ## Deploy encrypted image 32 | 33 | 1. Build rootfs with EAA component: 34 | 35 | Specify `AA_KBC=eaa_kbc` parameters when using kata-containers `rootfs.sh` scripts to create rootfs. 36 | 37 | 2. Launch Verdictd 38 | 39 | Verdictd performs remote attestation at runtime and provides the key needed to decrypt the image. 40 | It is actually both Key Broker Service and Attestation Service of EAA. 41 | So when deploy the encrypted image, Verdictd is needed to be launched: 42 | 43 | ```sh 44 | verdictd --listen <$ip>:<$port> --mutual 45 | ``` 46 | 47 | > **Note** The communication between Verdictd and EAA KBC is based on rats-tls, 48 | so you need to confirm that [rats-tls](https://github.com/inclavare-containers/rats-tls) has been correctly installed in your running environment. 49 | 50 | 3. Agent Configuration 51 | 52 | Add configuration `aa_kbc_params= 'eaa_kbc::<$IP>:<$PORT>'` to agent config file, the IP and PORT should be consistent with verdictd. 53 | -------------------------------------------------------------------------------- /guides/enclave-cc.md: -------------------------------------------------------------------------------- 1 | # Enclave-CC 2 | 3 | `enclave-cc` provides process-based isolation using SGX. 4 | This guide assumes that you already have a Kubernetes cluster 5 | and have deployed the operator as described in the **Installation** 6 | section of the [quickstart guide](../quickstart.md). 7 | 8 | ## Configuring enclave-cc custom resource to use a different KBC 9 | 10 | **Note** The KBC configuration changes to the enclave-cc custom resource yaml 11 | must be made **before** deploying it. 12 | 13 | Enclave CC supports Verdictd and in order to use it, users will have to 14 | properly configure a decrypt_config.conf, in order to set the `KBC` (`sample_kbc` 15 | or `eaa_kbc`) `IP`,`PORT`, and the `SECURITY_VALIDATE` (`false` or `true`) 16 | ```json 17 | { 18 | "key_provider": "provider:attestation-agent:KBC::IP:PORT", 19 | "security_validate": SECURITY_VALIDATE 20 | } 21 | ``` 22 | 23 | Once that's set according to the users needs, the user will then have to run: 24 | `cat decrypt_config.conf | base64 -w 0` in order to get the data encoded and 25 | set it accordingly [here](https://github.com/confidential-containers/operator/blob/6f241fbc056f0a5d9e1bd2c10b2cedc0782b99ff/config/samples/enclave-cc/base/ccruntime-enclave-cc.yaml#L124). 26 | 27 | ## Creating a sample CoCo workload using enclave-cc 28 | 29 | As an example, we setup a sample *hello world* 30 | workload with an encrypted and cosign signed container image using the `enclave-cc` runtime class for process based TEEs. 31 | The deployment below assumes the hardware SGX mode build is installed by the operator. To try on a non-TEE system, please 32 | use simulate SGX mode build. 33 | 34 | The example uses a trivial hello world C application: 35 | ```yaml 36 | apiVersion: v1 37 | kind: Pod 38 | metadata: 39 | name: enclave-cc-pod 40 | spec: 41 | containers: 42 | - image: ghcr.io/confidential-containers/test-container-enclave-cc:encrypted 43 | name: hello-world 44 | workingDir: "/run/rune/boot_instance/" 45 | resources: 46 | limits: 47 | sgx.intel.com/epc: 600Mi 48 | env: 49 | - name: OCCLUM_RELEASE_ENCLAVE 50 | value: "1" 51 | command: 52 | - /run/rune/boot_instance/build/bin/occlum-run 53 | - /bin/hello_world 54 | runtimeClassName: enclave-cc 55 | 56 | ``` 57 | 58 | **Note** When the hardware SGX mode payload is used in an SGX enabled cluster, `sgx.intel.com/epc: 600Mi` 59 | resource request must be added to the pod spec. 60 | 61 | Again, create a pod YAML file as previously described (this time we named it `enclave-cc-pod.yaml`) . 62 | 63 | Create the workload: 64 | ```sh 65 | kubectl apply -f enclave-cc-pod.yaml 66 | ``` 67 | Output: 68 | ``` 69 | pod/enclave-cc-pod created 70 | ``` 71 | 72 | Ensure the pod was created successfully (in running state): 73 | ```sh 74 | kubectl get pods 75 | ``` 76 | Output: 77 | ``` 78 | NAME READY STATUS RESTARTS AGE 79 | enclave-cc-pod 1/1 Running 0 22s 80 | ``` 81 | 82 | Check the pod is running as expected: 83 | ```sh 84 | kubectl logs enclave-cc-pod | head -5 85 | ``` 86 | Output: 87 | ``` 88 | ["init"] 89 | Hello world! 90 | 91 | Hello world! 92 | 93 | ``` 94 | 95 | We can also verify the host does not have the image for others to use: 96 | ```sh 97 | crictl -r unix:///run/containerd/containerd.sock image ls | grep helloworld_enc 98 | ``` 99 | -------------------------------------------------------------------------------- /guides/ephemeral_storage.md: -------------------------------------------------------------------------------- 1 | # Trusted Ephemeral Storage for container images 2 | 3 | With CoCo, container images are pulled inside the guest VM. 4 | By default container images are saved in guest memory which is protected by CC hardware. 5 | Since memory is an expensive resource, CoCo implemented [trusted ephemeral storage](https://github.com/confidential-containers/documentation/issues/39) for container image and RW layer. 6 | 7 | This solution is verified with Kubernetes CSI driver [open-local](https://github.com/alibaba/open-local). Please follow this [user guide](https://github.com/alibaba/open-local/blob/main/docs/user-guide/user-guide.md) to install open-local. 8 | 9 | We can use following example `trusted_store_cc.yaml` to have a try: 10 | ```yaml 11 | apiVersion: v1 12 | kind: Pod 13 | metadata: 14 | name: trusted-lvm-block 15 | spec: 16 | runtimeClassName: kata-qemu-tdx 17 | containers: 18 | - name: sidecar-trusted-store 19 | image: pause 20 | volumeDevices: 21 | - devicePath: "/dev/trusted_store" 22 | name: trusted-store 23 | - name: application 24 | image: busybox 25 | command: 26 | - sh 27 | - "-c" 28 | - | 29 | sleep 10000 30 | volumes: 31 | - name: trusted-store 32 | persistentVolumeClaim: 33 | claimName: trusted-store-block-pvc 34 | --- 35 | apiVersion: v1 36 | kind: PersistentVolumeClaim 37 | metadata: 38 | name: trusted-store-block-pvc 39 | spec: 40 | volumeMode: Block 41 | accessModes: 42 | - ReadWriteOnce 43 | resources: 44 | requests: 45 | storage: 1Gi 46 | storageClassName: open-local-lvm 47 | ``` 48 | Before deploy the workload, we can follow this [documentation](https://github.com/kata-containers/kata-containers/blob/CCv0/docs/how-to/how-to-build-and-test-ccv0.md) and use [ccv0.sh](https://github.com/kata-containers/kata-containers/blob/CCv0/docs/how-to/ccv0.sh) to enable CoCo console debug(optional, check whether working as expected). 49 | 50 | Create the workload: 51 | ```sh 52 | kubectl apply -f trusted_store_cc.yaml 53 | ``` 54 | 55 | Ensure the pod was created successfully (in running state): 56 | ```sh 57 | kubectl get pods 58 | ``` 59 | 60 | Output: 61 | ``` 62 | NAME READY STATUS RESTARTS AGE 63 | trusted-lvm-block 2/2 Running 0 31s 64 | ``` 65 | 66 | After we enable the debug option, we can login into the VM with `ccv0.sh` script: 67 | ```sh 68 | ./ccv0.sh -d open_kata_shell 69 | ``` 70 | 71 | Check container image is saved in encrypted storage with following commands: 72 | ```sh 73 | root@localhost:/# lsblk --fs 74 | NAME FSTYPE LABEL UUID FSAVAIL FSUSE% MOUNTPOINT 75 | sda 76 | └─ephemeral_image_encrypted_disk 906M 0% /run/image 77 | 78 | root@localhost:/# cryptsetup status ephemeral_image_encrypted_disk 79 | /dev/mapper/ephemeral_image_encrypted_disk is active and is in use. 80 | type: LUKS2 81 | cipher: aes-xts-plain64 82 | keysize: 512 bits 83 | key location: dm-crypt 84 | device: /dev/sda 85 | sector size: 4096 86 | offset: 32768 sectors 87 | size: 2064384 sectors 88 | mode: read/write 89 | 90 | root@localhost:/# mount|grep image 91 | /dev/mapper/ephemeral_image_encrypted_disk on /run/image type ext4 (rw,relatime) 92 | 93 | root@localhost:/# ls /run/image/ 94 | layers lost+found overlay 95 | ``` 96 | -------------------------------------------------------------------------------- /guides/nontee_demo.md: -------------------------------------------------------------------------------- 1 | # Encrypted Container Images without Hardware Support 2 | 3 | Without Confidential Computing hardware, there is no way to securely provision 4 | the keys for an encrypted image. Nonetheless, in this demo we describe how to 5 | test encrypted images support with the non-tee `kata`/`kata-qemu` runtimeclass. 6 | 7 | ## Creating a CoCo workload using a pre-existing encrypted image 8 | 9 | We will now proceed to download and run a sample encrypted container image using the CoCo building blocks. 10 | 11 | A demo container image is provided at [docker.io/katadocker/ccv0-ssh](https://hub.docker.com/r/katadocker/ccv0-ssh). 12 | It is encrypted with [Attestation Agent](https://github.com/confidential-containers/attestation-agent)'s [offline file system key broker](https://github.com/confidential-containers/attestation-agent/tree/64c12fbecfe90ba974d5fe4896bf997308df298d/src/kbc_modules/offline_fs_kbc) and [`aa-offline_fs_kbc-keys.json`](https://github.com/confidential-containers/documentation/blob/main/demos/ssh-demo/aa-offline_fs_kbc-keys.json) as its key file. 13 | 14 | We have prepared a sample CoCo operator custom resource that is based on the standard `ccruntime.yaml`, but in addition has the the decryption keys and configuration required to decrypt this sample container image. 15 | > **Note** All pods started with this sample resource will be able to decrypt the sample container and all keys shown are for demo purposes only and should not be used in production. 16 | 17 | To test out creating a workload from the sample encrypted container image, we can take the following steps: 18 | 19 | ### Swap out the standard custom resource for our sample 20 | 21 | Support for multiple custom resources in not available in the current release. Consequently, if a custom resource already exists, then you'll need to remove it first before deploying a new one. We can remove the standard custom resource with: 22 | ```sh 23 | kubectl delete -k github.com/confidential-containers/operator/config/samples/ccruntime/?ref= 24 | ``` 25 | and in it's place install the modified version with the sample container's decryption key: 26 | ```sh 27 | kubectl apply -k github.com/confidential-containers/operator/config/samples/ccruntime/ssh-demo?ref= 28 | ``` 29 | Wait until each pod has the STATUS of Running. 30 | ```sh 31 | kubectl get pods -n confidential-containers-system --watch 32 | ``` 33 | ### Test creating a workload from the sample encrypted image 34 | 35 | Create a new Kubernetes deployment that uses the `docker.io/katadocker/ccv0-ssh` container image with: 36 | ```sh 37 | cat << EOF > ccv0-ssh-demo.yaml 38 | kind: Service 39 | apiVersion: v1 40 | metadata: 41 | name: ccv0-ssh 42 | spec: 43 | selector: 44 | app: ccv0-ssh 45 | ports: 46 | - port: 22 47 | --- 48 | kind: Deployment 49 | apiVersion: apps/v1 50 | metadata: 51 | name: ccv0-ssh 52 | spec: 53 | selector: 54 | matchLabels: 55 | app: ccv0-ssh 56 | template: 57 | metadata: 58 | labels: 59 | app: ccv0-ssh 60 | spec: 61 | runtimeClassName: kata 62 | containers: 63 | - name: ccv0-ssh 64 | image: docker.io/katadocker/ccv0-ssh 65 | imagePullPolicy: Always 66 | EOF 67 | ``` 68 | 69 | Apply this with: 70 | ```sh 71 | kubectl apply -f ccv0-ssh-demo.yaml 72 | ``` 73 | and wait for the pod to start. This process should show that we are able to pull the encrypted image, and using the decryption key configured in the CoCo sample guest image, decrypt the container image and create a workload using it. 74 | 75 | The demo image has an SSH host key embedded in it, which is protected by it's encryption, but we can download the sample private key and use this to ssh into the container to validate it hasn't been tampered with. 76 | 77 | Download the SSH key with: 78 | ```sh 79 | curl -Lo ccv0-ssh https://raw.githubusercontent.com/confidential-containers/documentation/main/demos/ssh-demo/ccv0-ssh 80 | ``` 81 | Ensure that the permissions are set correctly with: 82 | ```sh 83 | chmod 600 ccv0-ssh 84 | ``` 85 | 86 | We can then use the key to ssh into the container: 87 | ```sh 88 | $ ssh -i ccv0-ssh root@$(kubectl get service ccv0-ssh -o jsonpath="{.spec.clusterIP}") 89 | ``` 90 | You will be prompted about whether the host key fingerprint is correct. This fingerprint should match the one specified in the container image: `wK7uOpqpYQczcgV00fGCh+X97sJL3f6G1Ku4rvlwtR0.` 91 | -------------------------------------------------------------------------------- /guides/sev.md: -------------------------------------------------------------------------------- 1 | # SEV-ES Guide 2 | 3 | Confidential Containers supports SEV(-ES) with pre-attestation using 4 | [simple-kbs](https://github.com/confidential-containers/simple-kbs). 5 | 6 | This guide covers platform-specific setup for SEV(-ES) and walks through 7 | complete flows for attestation and encrypted images. 8 | 9 | ## Creating a CoCo workload using a pre-existing encrypted image on SEV 10 | 11 | ### Platform Setup 12 | 13 | To enable SEV on the host platform, first ensure that it is supported. Then follow these instructions to enable SEV: 14 | 15 | [AMD SEV - Prepare Host OS](https://github.com/AMDESE/AMDSEV#prepare-host-os) 16 | 17 | ### Install sevctl and Export SEV Certificate Chain 18 | 19 | [sevctl](https://github.com/virtee/sevctl) is the SEV command line utility and is needed to export the SEV certificate chain. 20 | 21 | Follow these steps to install `sevctl`: 22 | 23 | * Debian / Ubuntu: 24 | 25 | ``` 26 | # Rust must be installed to build 27 | curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh 28 | source $HOME/.cargo/env 29 | sudo apt install -y musl-dev musl-tools 30 | 31 | # Additional packages are required to build 32 | sudo apt install -y pkg-config libssl-dev asciidoctor 33 | 34 | # Clone the repository 35 | git clone https://github.com/virtee/sevctl.git 36 | 37 | # Build 38 | (cd sevctl && cargo build) 39 | ``` 40 | 41 | * CentOS / Fedora / RHEL: 42 | 43 | ``` 44 | sudo dnf install sevctl 45 | ``` 46 | 47 | > **Note** Due to [this bug](https://bugzilla.redhat.com/show_bug.cgi?id=2037963) on sevctl for RHEL and Fedora you might need to build the tool from sources to pick the fix up. 48 | 49 | If using the SEV kata configuration template file, the SEV certificate chain must be placed in `/opt/sev`. Export the SEV certificate chain using the following commands: 50 | 51 | ``` 52 | sudo mkdir -p /opt/sev 53 | sudo ./sevctl/target/debug/sevctl export --full /opt/sev/cert_chain.cert 54 | ``` 55 | 56 | ### Setup and Run the simple-kbs 57 | 58 | 59 | By default, the `kata-qemu-sev` runtime class uses pre-attestation with the 60 | `online-sev-kbc` and [simple-kbs](https://github.com/confidential-containers/simple-kbs) to attest the guest and provision secrets. 61 | `simple-kbs` is a basic prototype key broker which can validate a guest measurement according to a specified policy and conditionally release secrets. 62 | To use encrypted images, signed images, or authenticated registries with SEV, you should setup `simple-kbs`. 63 | If you simply want to run an unencrypted container image, you can disable pre-attestation by adding the following annotation 64 | `io.katacontainers.config.pre_attestation.enabled: "false"` to your pod. 65 | 66 | If you are using pre-attestation, you will need to add an annotation to your pod configuration which contains the URI of a `simple-kbs` instance. 67 | This annotation should be of the form `io.katacontainers.config.pre_attestation.uri: ":44444"`. 68 | Port 44444 is the default port per the directions below, but it may be configured to use another port. 69 | The KBS IP must be accessible from inside the guest. 70 | Usually it should be the public IP of the node where `simple-kbs` runs. 71 | 72 | The SEV policy can also be set by adding `io.katacontainers.config.sev.policy: ""` to your pod configuration. 73 | Setting the second bit of the policy enables SEV-ES. 74 | For more information see chapter 3 of the Secure Encrypted Virtualization API. 75 | The SEV policy is not the same as the policies that drive `simple-kbs`. 76 | 77 | The CoCo project has created a sample encrypted container image ([encrypted-image-tests](ghcr.io/fitzthum/encrypted-image-tests:encrypted)). This image is encrypted using a key that comes already provisioned inside the `simple-kbs` for ease of testing. No `simple-kbs` policy is required to get things running. 78 | 79 | The image encryption key and key for SSH access have been attached to the CoCo sample encrypted container image as docker labels. This image is meant for TEST purposes only as these keys are published publicly. In a production use case, these keys would be generated by the workload administrator and kept secret. For further details, see the section how to [Create an Encrypted Image](#create-an-encrypted-image). 80 | 81 | To learn more about creating custom policies, see the section on [Creating a simple-kbs Policy to Verify the SEV Firmware Measurement](#creating-a-simple-kbs-policy-to-verify-the-sev-firmware-measurement). 82 | 83 | `docker-compose` is required to run the `simple-kbs` and its database in docker containers: 84 | 85 | * Debian / Ubuntu: 86 | 87 | ``` 88 | sudo apt install docker-compose 89 | ``` 90 | 91 | * CentOS / Fedora / RHEL: 92 | 93 | ``` 94 | sudo dnf install docker-compose-plugin 95 | ``` 96 | 97 | Clone the repository for specified tag: 98 | ``` 99 | simple_kbs_tag="0.1.1" 100 | git clone https://github.com/confidential-containers/simple-kbs.git 101 | (cd simple-kbs && git checkout -b "branch_${simple_kbs_tag}" "${simple_kbs_tag}") 102 | ``` 103 | 104 | Run the service with `docker-compose`: 105 | 106 | * Debian / Ubuntu: 107 | 108 | ``` 109 | (cd simple-kbs && sudo docker-compose up -d) 110 | ``` 111 | 112 | * CentOS / Fedora / RHEL: 113 | 114 | ``` 115 | (cd simple-kbs && sudo docker compose up -d) 116 | ``` 117 | 118 | ### Launch the Pod and Verify SEV Encryption 119 | 120 | Here is a sample kubernetes service yaml for an encrypted image: 121 | 122 | ``` 123 | kind: Service 124 | apiVersion: v1 125 | metadata: 126 | name: encrypted-image-tests 127 | spec: 128 | selector: 129 | app: encrypted-image-tests 130 | ports: 131 | - port: 22 132 | --- 133 | kind: Deployment 134 | apiVersion: apps/v1 135 | metadata: 136 | name: encrypted-image-tests 137 | spec: 138 | selector: 139 | matchLabels: 140 | app: encrypted-image-tests 141 | template: 142 | metadata: 143 | labels: 144 | app: encrypted-image-tests 145 | spec: 146 | runtimeClassName: kata-qemu-sev 147 | containers: 148 | - name: encrypted-image-tests 149 | image: ghcr.io/fitzthum/encrypted-image-tests:encrypted 150 | imagePullPolicy: Always 151 | ``` 152 | 153 | Save this service yaml to a file named `encrypted-image-tests.yaml`. Notice the image URL specified points to the previously described CoCo sample encrypted container image. `kata-qemu-sev` must also be specified as the `runtimeClassName`. 154 | 155 | Start the service: 156 | 157 | ``` 158 | kubectl apply -f encrypted-image-tests.yaml 159 | ``` 160 | 161 | Check for pod errors: 162 | 163 | ``` 164 | pod_name=$(kubectl get pod -o wide | grep encrypted-image-tests | awk '{print $1;}') 165 | kubectl describe pod ${pod_name} 166 | ``` 167 | 168 | If there are no errors, a CoCo encrypted container with SEV has been successfully launched! 169 | 170 | ### Verify SEV Memory Encryption 171 | 172 | The container `dmesg` report can be parsed to verify SEV memory encryption. 173 | 174 | Get pod IP: 175 | 176 | ``` 177 | pod_ip=$(kubectl get pod -o wide | grep encrypted-image-tests | awk '{print $6;}') 178 | ``` 179 | 180 | Get the CoCo sample encrypted container image SSH access key from docker image label and save it to a file. 181 | Currently the docker client cannot pull encrypted images. We can inspect the unencrypted image instead, 182 | which has the same labels. You could also use `skopeo inspect` to get the labels from the encrypted image. 183 | 184 | ``` 185 | docker pull ghcr.io/fitzthum/encrypted-image-tests:unencrypted 186 | docker inspect ghcr.io/fitzthum/encrypted-image-tests:unencrypted | \ 187 | jq -r '.[0].Config.Labels.ssh_key' \ 188 | | sed "s|\(-----BEGIN OPENSSH PRIVATE KEY-----\)|\1\n|g" \ 189 | | sed "s|\(-----END OPENSSH PRIVATE KEY-----\)|\n\1|g" \ 190 | > encrypted-image-tests 191 | ``` 192 | 193 | Set permissions on the SSH private key file: 194 | 195 | ``` 196 | chmod 600 encrypted-image-tests 197 | ``` 198 | 199 | Run a SSH command to parse the container `dmesg` output for SEV enabled messages: 200 | 201 | ``` 202 | ssh -i encrypted-image-tests \ 203 | -o "StrictHostKeyChecking no" \ 204 | -t root@${pod_ip} \ 205 | 'dmesg | grep SEV' 206 | ``` 207 | 208 | The output should look something like this: 209 | ``` 210 | [ 0.150045] Memory Encryption Features active: AMD SEV 211 | ``` 212 | 213 | ## Create an Encrypted Image 214 | 215 | If SSH access to the container is desired, create a keypair: 216 | 217 | ``` 218 | ssh-keygen -t ed25519 -f encrypted-image-tests -P "" -C "" <<< y 219 | ``` 220 | 221 | The above command will save the keypair in a file named `encrypted-image-tests`. 222 | 223 | Here is a sample Dockerfile to create a docker image: 224 | 225 | ``` 226 | FROM alpine:3.16 227 | 228 | # Update and install openssh-server 229 | RUN apk update && apk upgrade && apk add openssh-server 230 | 231 | # Generate container ssh key 232 | RUN ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -P "" 233 | 234 | # A password needs to be set for login to work. An empty password is 235 | # unproblematic as password-based login to root is not allowed. 236 | RUN passwd -d root 237 | 238 | # Copy the remote generated public key to the container authorized_keys 239 | # Generate with 'ssh-keygen -t ed25519 -f encrypted-image-tests -P "" -C ""' 240 | COPY encrypted-image-tests.pub /root/.ssh/authorized_keys 241 | 242 | # Entry point - run sshd 243 | ENTRYPOINT /usr/sbin/sshd -D 244 | ``` 245 | 246 | Store this `Dockerfile` in the same directory as the `encrypted-image-tests` ssh keypair. 247 | 248 | Build image: 249 | 250 | ``` 251 | docker build -t encrypted-image-tests . 252 | ``` 253 | 254 | Tag and upload this unencrypted docker image to a registry: 255 | 256 | ``` 257 | docker tag encrypted-image-tests:latest [REGISTRY_URL]:unencrypted 258 | docker push [REGISTRY_URL]:unencrypted 259 | ``` 260 | 261 | Be sure to replace `[REGISTRY_URL]` with the desired registry URL. 262 | 263 | [skopeo](https://github.com/containers/skopeo) is required to encrypt the container image. Follow the instructions here to install `skopeo`: 264 | 265 | [skopeo Installation](https://github.com/containers/skopeo/blob/main/install.md) 266 | 267 | The Attestation Agent hosts a grpc service to support encrypting the image. Clone the repository: 268 | 269 | ``` 270 | attestation_agent_tag="v0.1.0" 271 | git clone https://github.com/confidential-containers/attestation-agent.git 272 | (cd attestation-agent && git checkout -b "branch_${attestation_agent_tag}" "${attestation_agent_tag}") 273 | ``` 274 | 275 | Run the offline_fs_kbs: 276 | 277 | ``` 278 | (cd attestation-agent/sample_keyprovider/src/enc_mods/offline_fs_kbs \ 279 | && cargo run --release --features offline_fs_kbs -- --keyprovider_sock 127.0.0.1:50001 &) 280 | ``` 281 | 282 | Create the Attestation Agent keyprovider: 283 | 284 | ``` 285 | cat > attestation-agent/sample_keyprovider/src/enc_mods/offline_fs_kbs/ocicrypt.conf < keys.json <` 21 | Sometimes this will give you a useful message pointing to a failed attestation 22 | or some sort of missing environment setup. Most of the time you will see a 23 | generic message such as the following: 24 | 25 | ``` 26 | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim: Failed to Check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 637456061:1024: unknown 27 | ``` 28 | 29 | Unfortunately, because this is a generic message, you'll need to go deeper to figure out 30 | what is going on. 31 | 32 | ## CoCo Debugging 33 | 34 | A good next step is to figure out if things are breaking before or after the VM boots. 35 | You can see if there is a hypervisor process running with something like this. 36 | ```bash 37 | ps -ef | grep qemu 38 | ``` 39 | 40 | If you are using a different hypervisor, adjust your command accordingly. 41 | If there are no hypervisor processes running on the worker node, the VM has 42 | either failed to start or was shutdown. If there is a hypervisor process, 43 | the problem is probably inside the guest. 44 | 45 | Now is a good time to enable debug output for Kata and containerd. 46 | To do this, first look at the containerd config file located at 47 | `/etc/containerd/config.toml`. At the bottom of the file there should 48 | be a section for each runtime class. For example: 49 | 50 | ```toml 51 | [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu-sev] 52 | cri_handler = "cc" 53 | runtime_type = "io.containerd.kata-qemu-sev.v2" 54 | privileged_without_host_devices = true 55 | pod_annotations = ["io.katacontainers.*"] 56 | [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu-sev.options] 57 | ConfigPath = "/opt/confidential-containers/share/defaults/kata-containers/configuration-qemu-sev.toml" 58 | ``` 59 | 60 | The `ConfigPath` entry on the final line shows the path to the Kata configuration file that will be used 61 | for that runtime class. 62 | 63 | While you are looking at the containerd config, find the `[debug]` section near the top and set `level` 64 | to `debug`. Make sure to restart containerd after editing the containerd config file. 65 | You can do this with `sudo systemctl restart containerd`. 66 | 67 | Now go to the Kata config file that matches your runtime class and enable every debug option available. 68 | You do not need to restart any daemons when changing the Kata config file; just run another pod 69 | or hope that Kubernetes restarts your existing pod. Note that enabling debug options in the Kata 70 | config file can change the attestation evidence of a confidential guest. 71 | 72 | Now you should be able to view logs from containerd with the following: 73 | ``` 74 | sudo journalctl -xeu containerd 75 | ``` 76 | 77 | Kata writes many messages to this log. It's good to know what you're looking for. There are many 78 | generic messages that are insignificant, often arising from a VM not shutting down cleanly 79 | after an unrelated issue. 80 | 81 | ### VM Doesn't Start 82 | If the VM has failed to start, you might have a problem with confidential 83 | computing support on your worker node. Make sure that you can start 84 | confidential VMs without confidential containers. 85 | 86 | Check the containerd log for any obvious errors regarding VM boot. 87 | Try searching the log for the string `error` or for the name 88 | of your hypervisor i.e. `qemu` or `qemu-system-x86_64`. 89 | 90 | If there are no obvious errors, try finding the hypervisor 91 | commandline. This should be in the containerd log if you have enabled 92 | debug messages correctly. 93 | 94 | It might be tempting to try running the hypervisor command directly 95 | from the command line, but this usually isn't productive. Instead, 96 | try starting a standalone VM using the same kernel, initrd/disk, 97 | command line, firmware, and hypervisor that Kata uses. 98 | This might uncover some kind of system misconfiguration. 99 | You can also find these values in the Kata config file, but looking 100 | in the log is more direct. 101 | 102 | Another way to print the hypervisor command is to create a bash script 103 | that prints any arguments it is called with to a file. Then modify the 104 | Kata config file so that the hypervisor path points to this scipt 105 | rather than to the hypervisor. This method can also be used to add 106 | additional parameters to the command line. Just have the bash script 107 | call the hypervisor with whatever arguments it received plus any that 108 | you want to add. This could be useful for enabling debugging or tracing 109 | flags in your hypervisor. For instance, if you are using QEMU and SEV 110 | you might want to add the argument `--trace 'kvm_sev_*'`. Make sure 111 | that QEMU was built with an appropriate tracing backend. 112 | 113 | ### VM Does Start 114 | 115 | If the VM does start, search the containerd log for the string `vmconsole`. 116 | This will show you any guest serial output. You might see some errors 117 | coming from the kernel as the guest tries to boot. You might also see the 118 | Kata agent starting. If the Kata agent has started, you can match 119 | the output to the source to get some clues about what is happening. 120 | You might also see something more obvious, like a panic coming from 121 | the Kata agent. 122 | 123 | #### failed to create shim task: failed to mount "/run/kata-containers/shared/containers/CONTAINER_NAME/rootfs" 124 | 125 | If your CoCo Pod gets an error like the one shown below, then it is likely the image pull policy is set to **IfNotPresent**, and the image has been found in the kubelet cache. It fails because the container runtime will not delegate to the Kata agent to pull the image inside the VM and the agent in turn will try to mount the bundle rootfs that only exist in the host filesystem. 126 | 127 | Therefore, you must ensure that the image pull policy is set to **Always** for any CoCo pod. This way the images are always handled entirely by the agent inside the VM. It is worth mentioning we recognize that this behavior is sub-optimal, so the community provides solutions to avoid constant image downloads for each workload. 128 | 129 | ``` 130 | Events: 131 | Type Reason Age From Message 132 | ---- ------ ---- ---- ------- 133 | Normal Scheduled 20s default-scheduler Successfully assigned default/coco-fedora-69d9f84cd7-j597j to virtlab1012 134 | Normal Pulled 5s (x3 over 19s) kubelet Container image "docker.io/wainersm/coco-fedora_sshd@sha256:a7108f9f0080c429beb66e2cf0abff143c9eb9c7cf4dcde3241bc56c938d33b9" already present on machine 135 | Normal Created 5s (x3 over 19s) kubelet Created container coco-fedora 136 | Warning Failed 5s (x3 over 19s) kubelet Error: failed to create containerd task: failed to create shim task: failed to mount "/run/kata-containers/shared/containers/coco-fedora/rootfs" to "/run/kata-containers/coco-fedora/rootfs", with error: ENOENT: No such file or directory: unknown 137 | Warning BackOff 4s (x3 over 18s) kubelet Back-off restarting failed container 138 | ``` 139 | 140 | #### Debug Console 141 | 142 | One very useful debugging tool is the Kata guest debug console. You can 143 | enable this by editing the Kata agent configuration file and adding the lines 144 | ``` toml 145 | debug_console = true 146 | debug_console_vport = 1026 147 | ``` 148 | 149 | Enabling the debug console via the Kata Configuration file will overwrite 150 | any settings in the agent configuration file in the guest initrd. 151 | Enabling the debug console will also change the launch measurement. 152 | 153 | Once you've started a pod with the new configuration, get the id of the pod 154 | you want to access. Do this via `ps -ef | grep qemu` or equivalent. 155 | The id is the long id that shows up in many different arguments. 156 | It should look like `1a9ab65be63b8b03dfd0c75036d27f0ed09eab38abb45337fea83acd3cd7bacd`. 157 | Once you have the id, you can use it to access the debug console. 158 | ``` 159 | sudo /opt/confidential-containers/bin/kata-runtime exec 160 | ``` 161 | You might need to symlink the appropriate Kata configuration file for your runtime 162 | class if the `kata-runtime` tries to look at the wrong one. 163 | 164 | The debug console gives you access to the guest VM. This is a great way to 165 | investigate missing dependencies or incorrect configurations. 166 | 167 | #### Guest Firmware Logs 168 | 169 | If the VM is running but there is no guest output in the log, 170 | the guest may have stalled in the firmware. Firmware output will 171 | depend on your firmware and hypervisor. If you are using QEMU and OVMF, 172 | you can see the OVMF output by adding `-global isa-debugcon.iobase=0x402` 173 | and `-debugcon file:/tmp/ovmf.log` to the QEMU command line using the 174 | redirect script described above. 175 | -------------------------------------------------------------------------------- /images/ApplicationTEEProtection.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/confidential-containers/documentation/8e55631d899d83e34b99792150669968a037d17f/images/ApplicationTEEProtection.png -------------------------------------------------------------------------------- /images/CC_SGX_container.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/confidential-containers/documentation/8e55631d899d83e34b99792150669968a037d17f/images/CC_SGX_container.jpg -------------------------------------------------------------------------------- /images/CC_SGX_container.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/confidential-containers/documentation/8e55631d899d83e34b99792150669968a037d17f/images/CC_SGX_container.png -------------------------------------------------------------------------------- /images/CC_TEE_VM.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/confidential-containers/documentation/8e55631d899d83e34b99792150669968a037d17f/images/CC_TEE_VM.jpg -------------------------------------------------------------------------------- /images/CC_TEE_container.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/confidential-containers/documentation/8e55631d899d83e34b99792150669968a037d17f/images/CC_TEE_container.jpg -------------------------------------------------------------------------------- /images/CC_TEE_container.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/confidential-containers/documentation/8e55631d899d83e34b99792150669968a037d17f/images/CC_TEE_container.png -------------------------------------------------------------------------------- /images/CC_all_blocks.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/confidential-containers/documentation/8e55631d899d83e34b99792150669968a037d17f/images/CC_all_blocks.jpg -------------------------------------------------------------------------------- /images/COCO_ccv1_TEE.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/confidential-containers/documentation/8e55631d899d83e34b99792150669968a037d17f/images/COCO_ccv1_TEE.png -------------------------------------------------------------------------------- /images/COCO_ccv1_enclave.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/confidential-containers/documentation/8e55631d899d83e34b99792150669968a037d17f/images/COCO_ccv1_enclave.png -------------------------------------------------------------------------------- /images/COCO_ccv1_peerpods_TEE.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/confidential-containers/documentation/8e55631d899d83e34b99792150669968a037d17f/images/COCO_ccv1_peerpods_TEE.png -------------------------------------------------------------------------------- /images/CloudNativeTEEProtection.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/confidential-containers/documentation/8e55631d899d83e34b99792150669968a037d17f/images/CloudNativeTEEProtection.png -------------------------------------------------------------------------------- /images/RoadmapNov2021.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/confidential-containers/documentation/8e55631d899d83e34b99792150669968a037d17f/images/RoadmapNov2021.jpg -------------------------------------------------------------------------------- /images/RoadmapOct2021.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/confidential-containers/documentation/8e55631d899d83e34b99792150669968a037d17f/images/RoadmapOct2021.jpg -------------------------------------------------------------------------------- /images/RoadmapSept2021.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/confidential-containers/documentation/8e55631d899d83e34b99792150669968a037d17f/images/RoadmapSept2021.jpg -------------------------------------------------------------------------------- /images/coco_logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/confidential-containers/documentation/8e55631d899d83e34b99792150669968a037d17f/images/coco_logo.png -------------------------------------------------------------------------------- /images/persona_model.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 |
Container Image Registry 
Container Image Registry 
Infrastructure as a Service (IaaS)
Infrastructure as a Service (IaaS)
Infrastructure
Operator
Inf...
Orchestration/Cluster
Orchestration/Cluster
Orchestration Operator
Orc...
Application / Pods/ Containers
Application / Pods/ Containers
Workload Provider
Wor...
Data
Data
Data Owner
Dat...
Confidential Computing Boundary
Confidential Comp...
Isolation can protect the outer layers (Cloud and Cluster) from the workload code and containers
Isolation can protect the outer layers (Cloud and Clu...
Confidentiality is the reverse and protects the code and data from the outer layers and personas.
Confidentiality is the reverse and protects the code...
Container Image Provider
Con...Text is not SVG - cannot display -------------------------------------------------------------------------------- /overview.md: -------------------------------------------------------------------------------- 1 | 2 | # Confidential Containers 3 | 4 | We are interested in integrating existing [Trusted Execution Environments](https://en.wikipedia.org/wiki/Trusted_execution_environment) 5 | (TEE) infrastructure support and technologies with the cloud native world. 6 | 7 | Key considerations are: 8 | - Allow cloud native application owners to enforce application security requirements 9 | - Transparent deployment of unmodified containers 10 | - Support multiple TEE and hardware platforms 11 | - Introduce a trust model which separates Cloud Service Providers (CSPs) from guest applications 12 | - Apply least privilege principles to the Kubernetes cluster administration capabilities which 13 | impact delivering Confidential Computing for guest applications or data inside the TEE 14 | 15 | TEE's can be used to encapsulate different levels of the architecture stack with three key levels 16 | being node v pod v container. Container isolation was initially 17 | [provided in kata-containers](https://github.com/kata-containers/kata-containers/blob/main/docs/use-cases/using-Intel-SGX-and-kata.md). 18 | As an outgrowth of that project we are now expanding to address pod level support for confidential 19 | computing. Node level introduces significant challenges around least privilege for kubernetes 20 | cluster administration. We will explore the combination of pod and container level isolation, and we 21 | expect that challenges explored will have relevance to future understanding of the use of TEE's at 22 | the node level. 23 | 24 | 25 | ## Why? 26 | Security has long been a significant concern with data encryption at rest and in flight 27 | assumed to be a key part of any offering. Trusted Execution Environments look to address the *data 28 | in use* security concern. 29 | 30 | Cloud Computing adoption continues to accelerate whether it be Public, Private or increasingly 31 | common a Hybrid approach and with it the trust boundaries change. Consideration of insider threats 32 | needs to now consider the cloud provider, infrastructure provider, and managed service provider. 33 | 34 | Certain industries are heavily focused on compliance to standards. Governments, too, are concerned 35 | both [collectively](https://www.un.org/counterterrorism/cybersecurity) and 36 | [individually](https://www.whitehouse.gov/briefing-room/presidential-actions/2021/05/12/executive-order-on-improving-the-nations-cybersecurity/). 37 | The standards expected to protect software solutions continue to evolve towards a concept of 38 | Confidential Computing. 39 | 40 | Confidential Containers seek to address the growing concerns and needs of these three areas 41 | combining and growing in the future. 42 | 43 | ## How? 44 | ### Trusted Execution Environments 45 | 46 | We are actively working to support multiple TEE Technologies: 47 | - AMD Secure Encrypted Virtualization (SEV, SEV-ES) 48 | - Intel Software Guard Extensions (SGX) 49 | - Intel Trusted Domain Extensions (TDX) 50 | - IBM 51 | - Protected Execution Facility (PEF) 52 | - Secure Execution for Linux (SE) 53 | 54 | 55 | ![TEE Protects an application](./images/ApplicationTEEProtection.png) 56 | 57 | The TEE seeks to protect the application and data from outside threats, with the application owner 58 | having complete control of all communication across the TEE boundary. The application is considered 59 | a single complete entity and once supplied with the resources it requires, the TEE protects those 60 | resources (memory and CPU) from the infrastructure and all communication across the TEE boundary is 61 | under the control of the application owner. 62 | 63 | ### Cloud Native Execution Environments 64 | ![TEE protects an orchestrated pod](./images/CloudNativeTEEProtection.png) 65 | 66 | However, moving to a more cloud native approach, the application is now considered a group of one or 67 | more containers, with shared storage and network resources (pod). This pod is also subject to an 68 | orchestration layer which needs to dynamically interact with the pods and containers with respect to 69 | provisioning, deployment, networking, scaling, availability, and lifecycle management. 70 | 71 | How does the application owner trust the orchestration actions required to deliver on the cloud 72 | native promise, take advantage of the TEE capabilities and deliver on their compliance/security 73 | requirements? 74 | 75 | Our goal is to establish patterns using TEE's to lock out the Cloud Service Provider (CSP), 76 | establish deprivileged orchestration (Kubernetes admin) and still be able to build and deploy cloud 77 | native workloads. 78 | 79 | ### Integration of Technologies 80 | With many TEE technologies requiring a KVM boundary between the host 81 | and guest, [Kata Containers](https://katacontainers.io/) are the basis for our initial work. The 82 | Kata Containers project already supports a KVM boundary between a Kubernetes Node and Kubernetes Pod 83 | and focuses on reducing the concern of a guest attacking the host, in this case a breakout from 84 | containers within the pod attacking the Kubernetes Node. 85 | 86 | Requests from TEE vendors to support their technology within Kata Containers project led to 87 | a consideration of 88 | [Confidential Computing Enablement](https://github.com/kata-containers/kata-containers/issues/1332). 89 | How to protect the guest from the host (in this case to protect the containers and workloads within 90 | a Pod from the Kubernetes Node)? 91 | 92 | Initial exploration led to the realisation that this was not a problem possible to solve within the 93 | Kata Containers project alone. It raises considerations that need to be discussed and resolved in 94 | other areas of the cloud native stack, from container runtime to CSI to orchestration (kubelet) and 95 | brings in new projects or concerns such as attestation, reconsidering trust domains and least 96 | privilege capabilities for Kubernetes admins. 97 | -------------------------------------------------------------------------------- /quickstart.md: -------------------------------------------------------------------------------- 1 | # Getting Started 2 | 3 | Confidential Containers (CoCo) provides a uniform workflow, trust model, and feature set 4 | across a wide array of platforms and deployment models. 5 | 6 | In general, using CoCo involves the following steps: 7 | 8 | - Install CoCo using the operator 9 | - Add a runtimeClassName to your pod yaml 10 | - Deploy signed/encrypted container images (optional) 11 | - Setup attestation (optional) 12 | 13 | This guide walks through these steps and touches on some platform-specific configurations. 14 | For more advanced features, specific hardware setup, and troubleshooting information, 15 | see the [guides](./guides) directory. 16 | 17 | Confidential Containers is still maturing. See [release notes](./releases) for currrent 18 | hardware support and limitations. 19 | 20 | # Installation 21 | 22 | You can enable Confidential Containers in an existing Kubernetes cluster using the Confidential Containers Operator. 23 | When installation is finished, your cluster will have new runtime classes for different hardware platforms, 24 | including a generic runtime for testing CoCo without confidential hardware support, a runtime using a remote hypervisor 25 | that allows for cloud integration, a runtime for process-based isolation using SGX, as well as runtimes for TDX and SEV. 26 | 27 | ## Prerequisites 28 | 29 | To run the operator you must have an existing Kubernetes cluster that meets the followng requirements. 30 | 31 | - Ensure a minimum of 8GB RAM and 4 vCPU for the Kubernetes cluster node 32 | - Only containerd runtime based Kubernetes clusters are supported with the current CoCo release 33 | - The minimum Kubernetes version should be 1.24 34 | - Ensure at least one Kubernetes node in the cluster is having the label `node-role.kubernetes.io/worker=` 35 | - Ensure SELinux is disabled or not enforced (https://github.com/confidential-containers/operator/issues/115) 36 | 37 | For more details on the operator, including the custom resources managed by the operator, refer to the operator [docs](https://github.com/confidential-containers/operator). 38 | 39 | > **Note** If you need to quickly deploy a single-node test cluster, you can 40 | use the [run-local.sh 41 | script](https://github.com/confidential-containers/operator/blob/main/tests/e2e/run-local.sh) 42 | from the operator test suite, which will setup a single-node cluster on your 43 | machine for testing purpose. 44 | This script requires `ansible-playbook`, which you can install on CentOS/RHEL using 45 | `dnf install ansible-core`, and the Ansible `docker_container` module, which you can 46 | get using `ansible-galaxy collection install community.docker`. 47 | 48 | > **Note** You can also use a Kind or Minikube cluster with containerd runtime to try out the CoCo stack 49 | for development purposes. Make sure to use the `kata-clh` runtime class for your workloads when using Kind or 50 | Minikube, [as QEMU is known to **not** be working with Kind or Minikube](https://github.com/confidential-containers/operator/issues/124). 51 | Also, with the `enclave-cc` runtime class, the cluster must be prepared so that `/opt/confidential-containers` 52 | on the worker nodes is **not** on an overlayfs mount but the path is a `hostPath` mount (see 53 | [a sample configuration](https://github.com/confidential-containers/operator/blob/cf6a4f38114f7c5b71daec6cb666b1b40bcea140/tests/e2e/enclave-cc-kind-config.yaml#L6-L8)) 54 | 55 | ## Operator Installation 56 | 57 | ### Deploy the operator 58 | 59 | Deploy the operator by running the following command where `` needs to be substituted 60 | with the desired [release tag](https://github.com/confidential-containers/operator/tags). 61 | 62 | ``` 63 | kubectl apply -k github.com/confidential-containers/operator/config/release?ref= 64 | ``` 65 | 66 | For example, to deploy the `v0.3.0` release run: 67 | ``` 68 | kubectl apply -k github.com/confidential-containers/operator/config/release?ref=v0.3.0 69 | ``` 70 | 71 | Wait until each pod has the STATUS of Running. 72 | 73 | ``` 74 | kubectl get pods -n confidential-containers-system --watch 75 | ``` 76 | 77 | ### Create the custom resource 78 | 79 | Creating a custom resource installs the required CC runtime pieces into the cluster node and creates 80 | the `RuntimeClasses` 81 | 82 | ``` 83 | kubectl apply -k github.com/confidential-containers/operator/config/samples/ccruntime/?ref= 84 | ``` 85 | 86 | The current present overlays are: `default` and `s390x` 87 | 88 | For example, to deploy the `v0.3.0` release for `x86_64`, run: 89 | ``` 90 | kubectl apply -k github.com/confidential-containers/operator/config/samples/ccruntime/default?ref=v0.3.0 91 | ``` 92 | 93 | And to deploy `v0.3.0` release for `s390x`, run: 94 | ``` 95 | kubectl apply -k github.com/confidential-containers/operator/config/samples/ccruntime/s390x?ref=v0.3.0 96 | ``` 97 | 98 | Wait until each pod has the STATUS of Running. 99 | 100 | ``` 101 | kubectl get pods -n confidential-containers-system --watch 102 | ``` 103 | 104 | #### Create custom resource for enclave-cc 105 | 106 | **Note** For `enclave-cc` certain configuration changes, such as setting the 107 | URI of the KBS, must be made **before** applying the custom resource. 108 | Please see the [enclave-cc guide](./guides/enclave-cc.md) for more information. 109 | 110 | `enclave-cc` is a form of Confidential Containers that uses process-based isolation. 111 | `enclave-cc` can be installed with the following custom resources. 112 | ``` 113 | kubectl apply -k github.com/confidential-containers/operator/config/samples/enclave-cc/sim?ref= 114 | ``` 115 | or 116 | ``` 117 | kubectl apply -k github.com/confidential-containers/operator/config/samples/enclave-cc/hw?ref= 118 | ``` 119 | for the **simulated** SGX mode build or **hardware** SGX mode build, respectively. 120 | 121 | ### Verify Installation 122 | 123 | Check the `RuntimeClasses` that got created. 124 | 125 | ``` 126 | kubectl get runtimeclass 127 | ``` 128 | Output: 129 | ``` 130 | NAME HANDLER AGE 131 | kata kata 9m55s 132 | kata-clh kata-clh 9m55s 133 | kata-clh-tdx kata-clh-tdx 9m55s 134 | kata-qemu kata-qemu 9m55s 135 | kata-qemu-tdx kata-qemu-tdx 9m55s 136 | kata-qemu-sev kata-qemu-sev 9m55s 137 | ``` 138 | 139 | Details on each of the runtime classes: 140 | 141 | - *kata* - standard kata runtime using the QEMU hypervisor including all CoCo building blocks for a non CC HW 142 | - *kata-clh* - standard kata runtime using the cloud hypervisor including all CoCo building blocks for a non CC HW 143 | - *kata-clh-tdx* - using the Cloud Hypervisor, with TD-Shim, and support for Intel TDX CC HW 144 | - *kata-qemu* - same as kata 145 | - *kata-qemu-tdx* - using QEMU, with TDVF, and support for Intel TDX CC HW, prepared for using Verdictd and EAA KBC. 146 | - *kata-qemu-sev* - using QEMU, and support for AMD SEV HW 147 | 148 | If you are using `enclave-cc` you should see the following runtime classes. 149 | 150 | ``` 151 | kubectl get runtimeclass 152 | ``` 153 | Output: 154 | ``` 155 | NAME HANDLER AGE 156 | enclave-cc enclave-cc 9m55s 157 | ``` 158 | 159 | ### Platform Setup 160 | 161 | While the operator deploys all the required binaries and artifacts and sets up runtime classes that use them, 162 | certain platforms may require additional configuration to enable confidential computing. For example, the host 163 | kernel and firmware might need to be configured. 164 | See the [guides](./guides) for more information. 165 | 166 | # Running a workload 167 | 168 | ## Creating a sample CoCo workload 169 | 170 | Once you've used the operator to install Confidential Containers, you can run a pod with CoCo by simply adding a runtime class. 171 | First, we will use the `kata` runtime class which uses CoCo without hardware support. 172 | Initially we will try this with an unencrypted container image. 173 | 174 | In our example we will be using the bitnami/nginx image as described in the following yaml: 175 | ``` 176 | apiVersion: v1 177 | kind: Pod 178 | metadata: 179 | labels: 180 | run: nginx 181 | name: nginx 182 | spec: 183 | containers: 184 | - image: bitnami/nginx:1.22.0 185 | name: nginx 186 | dnsPolicy: ClusterFirst 187 | runtimeClassName: kata 188 | ``` 189 | 190 | Setting the runtimeClassName is usually the only change needed to the pod yaml, but some platforms 191 | support additional annotations for configuring the enclave. See the [guides](./guides) for 192 | more details. 193 | 194 | With Confidential Containers, the workload container images are never downloaded on the host. 195 | For verifying that the container image doesn’t exist on the host you should log into the k8s node and ensure the following command returns an empty result: 196 | ``` 197 | root@cluster01-master-0:/home/ubuntu# crictl -r unix:///run/containerd/containerd.sock image ls | grep bitnami/nginx 198 | ``` 199 | You will run this command again after the container has started. 200 | 201 | Create a pod YAML file as previously described (we named it `nginx.yaml`) . 202 | 203 | Create the workload: 204 | ``` 205 | kubectl apply -f nginx.yaml 206 | ``` 207 | Output: 208 | ``` 209 | pod/nginx created 210 | ``` 211 | 212 | Ensure the pod was created successfully (in running state): 213 | ``` 214 | kubectl get pods 215 | ``` 216 | Output: 217 | ``` 218 | NAME READY STATUS RESTARTS AGE 219 | nginx 1/1 Running 0 3m50s 220 | ``` 221 | 222 | Now go back to the k8s node and ensure that you still don’t have any bitnami/nginx images on it: 223 | ``` 224 | root@cluster01-master-0:/home/ubuntu# crictl -r unix:///run/containerd/containerd.sock image ls | grep bitnami/nginx 225 | ``` 226 | 227 | ## Encrypted and/or signed images with attestation 228 | 229 | The previous example does not involve any attestation because the workload container isn't signed or encrypted 230 | and the workload itself does not require any secrets. 231 | 232 | This is not the case for most real workloads. It is recommended to use CoCo with signed and/or encrypted images. 233 | The workload itself can also request secrets from the attestation agent in the guest. 234 | 235 | Secrets are provisioned to the guest in conjunction with an attestation, which is based on hardware evidence. 236 | The rest of this guide focuses on setting up more substantial encrypted/signed workloads using attestation 237 | and confidential hardware. 238 | 239 | See [this guide](./guides/nontee_demo.md) if you would like to deploy an example encrypted image without 240 | confidential hardware. 241 | 242 | CoCo has a modular attestation interface and there are a few options for attestation. 243 | CoCo provides a generic Key Broker Service (KBS) that the rest of this guide will be focused on. 244 | The SEV runtime class uses `simple-kbs`, which is described in the [SEV guide](./guides/sev.md). 245 | There is also `eaa_kbc`/`verdictd` which is described [here](./guides/eaa_verdictd.md). 246 | 247 | ### Select Runtime Class 248 | 249 | To use CoCo with confidential hardware, first switch to the appropriate runtime class. 250 | TDX has two runtime classes, `kata-qemu-tdx` and `kata-clh-tdx`. One uses QEMU as VMM and TDVF as firmware. The other uses Cloud Hypervisor as VMM and TD-Shim as firmware. 251 | 252 | For SEV(-ES) use the `kata-qemu-sev` runtime class and follow the [SEV guide](./guides/sev.md). 253 | 254 | For `enclave-cc` follow the [enclave-cc guide](./guides/enclave-cc.md). 255 | 256 | ### Deploy and Configure tenant-side CoCo Key Broker System cluster 257 | 258 | The following describes how to run and provision the generic KBS. 259 | The KBS should be run in a trusted environment. The KBS is not just one service, 260 | but a combination of several. 261 | 262 | A tenant-side CoCo Key Broker System cluster includes: 263 | - Key Broker Service (KBS): Brokering service for confidential resources. 264 | - Attestation Service (AS): Verifier for remote attestation. 265 | - Reference Value Provicer Service (RVPS): Provides reference values for AS. 266 | - CoCo Keyprovider: Component to encrypt the images following ocicrypt spec. 267 | 268 | To quick start the KBS cluster, a `docker-compose` yaml is provided to launch. 269 | 270 | ```shell 271 | # Clone KBS git repository 272 | git clone https://github.com/confidential-containers/kbs.git 273 | cd kbs 274 | export KBS_DIR_PATH=$(pwd) 275 | 276 | # Generate a user auth key pair 277 | openssl genpkey -algorithm ed25519 > config/private.key 278 | openssl pkey -in config/private.key -pubout -out config/public.pub 279 | 280 | # Start KBS cluster 281 | docker-compose up -d 282 | ``` 283 | 284 | If configuration of KBS cluster is required, edit the following config files and restart the KBS cluster with `docker-compose`: 285 | 286 | - `$KBS_DIR_PATH/config/kbs-config.json`: configuration for Key Broker Service. 287 | - `$KBS_DIR_PATH/config/as-config.json`: configuration for Attestation Service. 288 | - `$KBS_DIR_PATH/config/sgx_default_qcnl.conf`: configuration for Intel TDX/SGX verification. 289 | 290 | When KBS cluster is running, you can modify the policy file used by AS policy engine ([OPA](https://www.openpolicyagent.org/)) at any time: 291 | 292 | - `$KBS_DIR_PATH/data/attestation-service/opa/policy.rego`: Policy file for evidence verification of AS, refer to [AS Policy Engine](https://github.com/confidential-containers/attestation-service#policy-engine) for more infomation. 293 | 294 | ### Encrypting an Image 295 | 296 | [skopeo](https://github.com/containers/skopeo) is required to encrypt the container image. 297 | Follow the [instructions](https://github.com/containers/skopeo/blob/main/install.md) to install `skopeo`. 298 | 299 | Use `skopeo` to encrypt an image on the same node of the KBS cluster (use busybox:latest for example): 300 | 301 | ```shell 302 | # edit ocicrypt.conf 303 | tee > ocicrypt.conf <.toml` to add `agent.aa_kbc_params=cc_kbc::` to kernal parameters. Here `RUNTIME_CLASS_SUFFIX` is something like `qemu-tdx`, `KBS_URI` is the address of Key Broker Service in KBS cluster like `http://123.123.123.123:8080`. 402 | 403 | Deploy encrypted image as a workload: 404 | 405 | ```shell 406 | kubectl apply -f encrypted-image-test-busybox.yaml 407 | ``` 408 | -------------------------------------------------------------------------------- /releases/v0.1.0.md: -------------------------------------------------------------------------------- 1 | # Release Notes for v0.1.0 2 | This is the first full release of Confidential Containers. 3 | The goal of this release is to provide a stable, simple, and well-documented base for the Confidential Containers project. 4 | The Confidential Containers operator is the focal point of the release. 5 | The operator allows users to install Confidential Containers on an existing Kubernetes cluster. 6 | This release also provides core Confidential Containers features, such as being able to run encrypted containers on Intel-TDX and AMD-SEV. 7 | 8 | Please see the [quickstart guide](../quickstart.md) for details on how to try out Confidential Containers" 9 | 10 | ## Hardware Support 11 | Confidential Containers is tested with attestation on the following platforms: 12 | - Intel TDX 13 | - AMD SEV 14 | 15 | The following platforms are untested or partially supported: 16 | - AMD SEV-ES 17 | - IBM Z SE 18 | 19 | The following platforms are in development: 20 | - Intel SGX 21 | - AMD SEV-SNP 22 | 23 | ## Limitations 24 | 25 | The following are known limitations of this release: 26 | 27 | - Platform support is currently limited, and rapidly changing 28 | * S390x is not supported by the CoCo operator 29 | * AMD SEV-ES has not been tested. 30 | * AMD SEV does not support container image signature validation. 31 | - Attestation and key brokering support is still under development 32 | * The disk-based key broker client (KBC) is used when there is no HW support, but is not suitable for production (except with encrypted VM images). 33 | * Currently, there are two KBS that can be used: 34 | - simple-kbs: simple key broker service (KBS) for SEV(-ES). 35 | - [Verdictd](https://github.com/inclavare-containers/verdictd): An external project with which Attestation Agent can conduct remote attestation communication and key acquisition via EAA KBC 36 | * The full-featured generic KBS and the corresponding KBC are still in the development stage. 37 | * For developers, other KBCs can be experimented with. 38 | * AMD SEV must use a KBS even for unencrypted images. 39 | - The format of encrypted container images is still subject to change 40 | * The oci-crypt container image format itself may still change 41 | * The tools to generate images are not in their final form 42 | * The image format itself is subject to change in upcoming releases 43 | * Image repository support for encrypted images is unequal 44 | - CoCo currently requires a custom build of `containerd` 45 | * The CoCo operator will deploy the correct version of `containerd` for you 46 | * Changes are required to delegate `PullImage` to the agent in the virtual machine 47 | * The required changes are not part of the vanilla `containerd` 48 | * The final form of the required changes in `containerd` is expected to be different 49 | * `crio` is not supported 50 | - CoCo is not fully integrated with the orchestration ecosystem (Kubernetes, OpenShift) 51 | * OpenShift is a non-started at the moment due to their dependency on CRIO 52 | * Existing APIs do not fully support the CoCo security and threat model 53 | * Some commands accessing confidential data, such as `kubectl exec`, may either fail to work, or incorrectly expose information to the host 54 | * Container image sharing is not possible in this release 55 | * Container images are downloaded by the guest (with encryption), not by the host 56 | * As a result, the same image will be downloaded separately by every pod using it, not shared between pods on the same host. 57 | - The CoCo community aspires to adopting open source security best practices, but not all practices are adopted yet. 58 | * We track our status with the OpenSSF Best Practices Badge, which was at 43% at the time of this release. 59 | * The main gaps are in test coverage, both general and security tests. 60 | * Vulnerability reporting mechanisms also need to be created. Public github issues are still appropriate for this release until private reporting is established. 61 | 62 | 63 | ## CVE Fixes 64 | 65 | None - This is our first release. 66 | -------------------------------------------------------------------------------- /releases/v0.2.0.md: -------------------------------------------------------------------------------- 1 | # Release Notes for v0.2.0 2 | Confidential Containers has adopted a six-week release cadence. 3 | This is our first release on this schedule. 4 | This release mainly features incremental improvements to our build system and tests 5 | as well as minor features, adjustments, and cleanup. 6 | 7 | Please see the [quickstart guide](../quickstart.md) for details on how to try out Confidential Containers 8 | 9 | ## What's new 10 | 11 | - Kata CI uses existing Kata tooling to build components. 12 | - Kata CI caches build environments for components. 13 | - Pod VM can be launched with measured boot. See [more info](https://github.com/confidential-containers/documentation/issues/40) 14 | - Incremental advances in signature support including verification of cosign-signed images. 15 | - Enclave-cc added to operator, providing initial SGX support. 16 | - KBS no longer required to use unencrypted images with SEV. 17 | - More rigorous versioning of sub-projects 18 | 19 | ## Hardware Support 20 | Confidential Containers is tested with attestation on the following platforms: 21 | - Intel TDX 22 | - AMD SEV 23 | 24 | The following platforms are untested or partially supported: 25 | - Intel SGX 26 | - AMD SEV-ES 27 | - IBM Z SE 28 | 29 | The following platforms are in development: 30 | - AMD SEV-SNP 31 | 32 | ## Limitations 33 | 34 | The following are known limitations of this release: 35 | 36 | - Platform support is currently limited, and rapidly changing 37 | * s390x is not supported by the CoCo operator 38 | * AMD SEV-ES has not been tested. 39 | * AMD SEV does not support container image signature validation. 40 | * s390x does not support cosign signature validation 41 | - SELinux is not supported on the host and must be set to permissive if in use. 42 | - Attestation and key brokering support is still under development 43 | * The disk-based key broker client (KBC) is used for non-tee testing, but is not suitable for production, except with encrypted VM images. 44 | * Currently, there are two KBS that can be used: 45 | - simple-kbs: simple key broker service (KBS) for SEV(-ES). 46 | - [Verdictd](https://github.com/inclavare-containers/verdictd): An external project with which Attestation Agent can conduct remote attestation communication and key acquisition via EAA KBC 47 | * The full-featured generic KBS and the corresponding KBC are still in the development stage. 48 | * For developers, other KBCs can be experimented with. 49 | * AMD SEV must use a KBS even for unencrypted images. 50 | - The format of encrypted container images is still subject to change 51 | * The oci-crypt container image format itself may still change 52 | * The tools to generate images are not in their final form 53 | * The image format itself is subject to change in upcoming releases 54 | * Image repository support for encrypted images is unequal 55 | - CoCo currently requires a custom build of `containerd` 56 | * The CoCo operator will deploy the correct version of `containerd` for you 57 | * Changes are required to delegate `PullImage` to the agent in the virtual machine 58 | * The required changes are not part of the vanilla `containerd` 59 | * The final form of the required changes in `containerd` is expected to be different 60 | * `crio` is not supported 61 | - CoCo is not fully integrated with the orchestration ecosystem (Kubernetes, OpenShift) 62 | * OpenShift is a non-starter at the moment due to its dependency on [CRI-O](https://github.com/cri-o/cri-o) 63 | * Existing APIs do not fully support the CoCo security and threat model. [More info](https://github.com/confidential-containers/community/issues/53) 64 | * Some commands accessing confidential data, such as `kubectl exec`, may either fail to work, or incorrectly expose information to the host 65 | * Container image sharing is not possible in this release 66 | * Container images are downloaded by the guest (with encryption), not by the host 67 | * As a result, the same image will be downloaded separately by every pod using it, not shared between pods on the same host. [More info](https://github.com/confidential-containers/community/issues/66) 68 | - The CoCo community aspires to adopting open source security best practices, but not all practices are adopted yet. 69 | * We track our status with the OpenSSF Best Practices Badge, which increased to 46% at the time of this release. 70 | * The main gaps are in test coverage, both general and security tests. 71 | * Vulnerability reporting mechanisms also need to be created. Public github issues are still appropriate for this release until private reporting is established. 72 | 73 | 74 | ## CVE Fixes 75 | 76 | None 77 | -------------------------------------------------------------------------------- /releases/v0.3.0.md: -------------------------------------------------------------------------------- 1 | # Release Notes for v0.3.0 2 | Release Date: January 20th, 2023 3 | 4 | Code Freeze: January 13th, 2023 5 | 6 | Please see the [quickstart guide](../quickstart.md) for details on how to try out Confidential Containers 7 | 8 | ## What's new 9 | - Support for pulling images from authenticated container registries. See [design info](https://github.com/confidential-containers/image-rs/blob/main/docs/image_auth.md). 10 | - Significantly reduced resource requirements for image pulling 11 | - Attestation support for AMD SEV-ES 12 | - `kata-qemu-tdx` supports and has been tested with Verdictd 13 | - Support for `get_resource` endpoint with SEV(-ES) 14 | - Enabled cosign signature support in enclave-cc / SGX 15 | - SEV attestation bug fixes 16 | - Measured rootfs now works with `kata-clh`, `kata-qemu`, `kata-clh-tdx`, and `kata-qemu-tdx` runtime classes. 17 | - IBM zSystems / LinuxONE (s390x) enablement and CI verification on non-TEE environments 18 | - Enhanced docs, config, CI pipeline and test coverage for enclave-cc / SGX 19 | 20 | ## Hardware Support 21 | Confidential Containers is tested with attestation on the following platforms: 22 | - Intel TDX 23 | - AMD SEV 24 | 25 | The following platforms are untested or partially supported: 26 | - Intel SGX 27 | - AMD SEV-ES 28 | - IBM Secure Execution (SE) on IBM zSystems & LinuxONE 29 | 30 | The following platforms are in development: 31 | - AMD SEV-SNP 32 | 33 | ## Limitations 34 | 35 | The following are known limitations of this release: 36 | 37 | - Platform support is currently limited, and rapidly changing 38 | * AMD SEV-ES is not tested in the CI. 39 | * Image signature validation has not been tested with AMD SEV. 40 | * s390x does not support cosign signature validation 41 | - SELinux is not supported on the host and must be set to permissive if in use. 42 | - Attestation and key brokering support is still under development 43 | * The disk-based key broker client (KBC) is used for non-tee testing, but is not suitable for production, except with encrypted VM images. 44 | * Currently, there are two KBS that can be used: 45 | - simple-kbs: simple key broker service (KBS) for SEV(-ES). 46 | - [Verdictd](https://github.com/inclavare-containers/verdictd): An external project with which Attestation Agent can conduct remote attestation communication and key acquisition via EAA KBC 47 | * The full-featured generic KBS and the corresponding KBC are still in the development stage. 48 | * For developers, other KBCs can be experimented with. 49 | * AMD SEV must use a KBS even for unencrypted images. 50 | - The format of encrypted container images is still subject to change 51 | * The oci-crypt container image format itself may still change 52 | * The tools to generate images are not in their final form 53 | * The image format itself is subject to change in upcoming releases 54 | * Image repository support for encrypted images is unequal 55 | - CoCo currently requires a custom build of `containerd` 56 | * The CoCo operator will deploy the correct version of `containerd` for you 57 | * Changes are required to delegate `PullImage` to the agent in the virtual machine 58 | * The required changes are not part of the vanilla `containerd` 59 | * The final form of the required changes in `containerd` is expected to be different 60 | * `crio` is not supported 61 | - CoCo is not fully integrated with the orchestration ecosystem (Kubernetes, OpenShift) 62 | * OpenShift is a non-starter at the moment due to its dependency on [CRI-O](https://github.com/cri-o/cri-o) 63 | * Existing APIs do not fully support the CoCo security and threat model. [More info](https://github.com/confidential-containers/community/issues/53) 64 | * Some commands accessing confidential data, such as `kubectl exec`, may either fail to work, or incorrectly expose information to the host 65 | * Container image sharing is not possible in this release 66 | * Container images are downloaded by the guest (with encryption), not by the host 67 | * As a result, the same image will be downloaded separately by every pod using it, not shared between pods on the same host. [More info](https://github.com/confidential-containers/community/issues/66) 68 | - The CoCo community aspires to adopting open source security best practices, but not all practices are adopted yet. 69 | * We track our status with the OpenSSF Best Practices Badge, which increased to 49% at the time of this release. 70 | * The main gaps are in test coverage, both general and security tests. 71 | * Vulnerability reporting mechanisms also need to be created. Public github issues are still appropriate for this release until private reporting is established. 72 | 73 | 74 | ## CVE Fixes 75 | 76 | None 77 | -------------------------------------------------------------------------------- /releases/v0.4.0.md: -------------------------------------------------------------------------------- 1 | # Release Notes for v0.4.0 2 | Release Date: 2023-03-03 3 | 4 | Please see the [quickstart guide](../quickstart.md) for details on how to try out Confidential 5 | Containers. 6 | 7 | Please refer to our [Acronyms](https://github.com/confidential-containers/documentation/wiki/Acronyms) 8 | and [Glossary](https://github.com/confidential-containers/documentation/wiki/Glossary) pages for a 9 | definition of the acronyms used in this document. 10 | 11 | ## What's new 12 | - This release focused on reducing technical debt. You will not observe as many new features in 13 | this release but you will be running on top of more robust code. 14 | - Skopeo and umoci dependencies are removed with our image-rs component fully integrated 15 | - Improved CI for SEV 16 | - Improved container support for enclave-cc / SGX 17 | 18 | ## Hardware Support 19 | Confidential Containers is tested with attestation on the following platforms: 20 | - Intel TDX 21 | - AMD SEV 22 | 23 | The following platforms are untested or partially supported: 24 | - Intel SGX 25 | - AMD SEV-ES 26 | - IBM Secure Execution (SE) on IBM zSystems (s390x) running LinuxONE 27 | 28 | The following platforms are in development: 29 | - AMD SEV-SNP 30 | 31 | ## Limitations 32 | 33 | The following are known limitations of this release: 34 | 35 | - Platform support is currently limited, and rapidly changing 36 | * AMD SEV-ES is not tested in the CI. 37 | * Image signature validation has not been tested with AMD SEV. 38 | * s390x does not support cosign signature validation 39 | - SELinux is not supported on the host and must be set to permissive if in use. 40 | - Attestation and key brokering support is still under development 41 | * The disk-based key broker client (KBC) is used for non-tee testing, but is not suitable for 42 | production, except with encrypted VM images. 43 | * Currently, there are two key broker services (KBS) that can be used: 44 | - simple-kbs: simple key broker service for SEV(-ES). 45 | - [Verdictd](https://github.com/inclavare-containers/verdictd): An external project with which 46 | Attestation Agent can conduct remote attestation communication and key acquisition via EAA KBC 47 | * The full-featured generic KBS and the corresponding KBC are still in the development stage. 48 | - The format of encrypted container images is still subject to change 49 | * The [oci-crypt](https://github.com/containers/ocicrypt) container image format itself may still change 50 | * The tools to generate images are not in their final form 51 | * The image format itself is subject to change in upcoming releases 52 | * Image repository support for encrypted images is unequal 53 | - CoCo currently requires a custom build of `containerd` 54 | * The CoCo operator will deploy the correct version of `containerd` for you 55 | * Changes are required to delegate `PullImage` to the agent in the virtual machine 56 | * The required changes are not part of the vanilla `containerd` 57 | * The final form of the required changes in `containerd` is expected to be different 58 | * `crio` is not supported 59 | - CoCo is not fully integrated with the orchestration ecosystem (Kubernetes, OpenShift) 60 | * OpenShift is a non-starter at the moment due to its dependency on [CRI-O](https://github.com/cri-o/cri-o) 61 | * Existing APIs do not fully support the CoCo security and threat model. [More info](https://github.com/confidential-containers/community/issues/53) 62 | * Some commands accessing confidential data, such as `kubectl exec`, may either fail to work, or incorrectly expose information to the host 63 | * Container image sharing is not possible in this release 64 | * Container images are downloaded by the guest (with encryption), not by the host 65 | * As a result, the same image will be downloaded separately by every pod using it, not shared between pods on the same host. [More info](https://github.com/confidential-containers/community/issues/66) 66 | - The CoCo community aspires to adopting open source security best practices, but not all practices are adopted yet. 67 | * We track our status with the OpenSSF Best Practices Badge, which increased to 49% at the time of this release. 68 | * The main gaps are in test coverage, both general and security tests. 69 | * Vulnerability reporting mechanisms also need to be created. Public github issues are still appropriate for this release until private reporting is established. 70 | 71 | 72 | ## CVE Fixes 73 | 74 | None 75 | -------------------------------------------------------------------------------- /releases/v0.5.0.md: -------------------------------------------------------------------------------- 1 | # Release Notes for v0.5.0 2 | Release Date: 2023-04-14 3 | 4 | > **Warning** 5 | > This release includes breaking changes to the format of encrypted images. See below for more details. 6 | > Images that were encrypted using tooling from previous releases will fail with this release. 7 | > The process for validating signed images is also slightly different. 8 | 9 | Please see the [quickstart guide](../quickstart.md) for details on how to try out Confidential 10 | Containers. 11 | 12 | Please refer to our [Acronyms](https://github.com/confidential-containers/documentation/wiki/Acronyms) 13 | and [Glossary](https://github.com/confidential-containers/documentation/wiki/Glossary) pages for a 14 | definition of the acronyms used in this document. 15 | 16 | ## What's new 17 | - Process-based isolation is now fully supported with SGX hardware added to enclave-cc CI 18 | - Remote hypervisor support added to the CoCo operator, which helps to enable creating containers 19 | as ‘peer pods’, either locally, or on Cloud Service Provider Infrastructure. 20 | See [README](https://github.com/confidential-containers/cloud-api-adaptor/blob/v0.5.0/README.md) for more information and installation instructions. 21 | 22 | - [KBS Resource URI Scheme](https://github.com/confidential-containers/attestation-agent/blob/main/docs/KBS_URI.md) 23 | is published to identify all confidential resources. 24 | - Different KBCs now share image encryption format allowing for interchangeable use. 25 | - Generic Key Broker System (KBS) is now supported. This includes the [KBS](https://github.com/confidential-containers/kbs) 26 | itself, which relies on the [Attestation Service (AS)](https://github.com/confidential-containers/attestation-service) 27 | for attestation evidence verification. Reference Values are provided to the `AS` by 28 | the [Reference Value Provider Service (RVPS)](https://github.com/confidential-containers/attestation-service/blob/main/docs/rvps.md). 29 | Currently only TDX and a sample mode are supported with generic KBS. Other platforms are in development. 30 | - SEV configuration can be set with annotations. 31 | - SEV-ES is now tested in the CI. 32 | - Some developmental SEV-SNP components can be manually enabled to test SNP containers without attestation. 33 | 34 | ## Hardware Support 35 | Confidential Containers is tested with attestation on the following platforms: 36 | - Intel TDX 37 | - AMD SEV(-ES) 38 | - Intel SGX 39 | 40 | The following platforms are untested or partially supported: 41 | - IBM Secure Execution (SE) on IBM zSystems (s390x) running LinuxONE 42 | 43 | The following platforms are in development: 44 | - AMD SEV-SNP 45 | 46 | ## Limitations 47 | The following are known limitations of this release: 48 | 49 | - Platform support is currently limited, and rapidly changing 50 | * Image signature validation with AMD SEV-ES is not covered by CI. 51 | * s390x does not support cosign signature validation 52 | - SELinux is not supported on the host and must be set to permissive if in use. 53 | - Attestation and key brokering support varies by platform. 54 | * The generic KBS is only supported on TDX. Other platforms have different solutions. 55 | - The format of encrypted container images is still subject to change 56 | * The [oci-crypt](https://github.com/containers/ocicrypt) container image format itself may still change 57 | * The tools to generate images are not in their final form 58 | * The image format itself is subject to change in upcoming releases 59 | * Image repository support for encrypted images is unequal 60 | - CoCo currently requires a custom build of `containerd` 61 | * The CoCo operator will deploy the correct version of `containerd` for you 62 | * Changes are required to delegate `PullImage` to the agent in the virtual machine 63 | * The required changes are not part of the vanilla `containerd` 64 | * The final form of the required changes in `containerd` is expected to be different 65 | * `crio` is not supported 66 | - CoCo is not fully integrated with the orchestration ecosystem (Kubernetes, OpenShift) 67 | * OpenShift support is not yet complete. 68 | * Existing APIs do not fully support the CoCo security and threat model. [More info](https://github.com/confidential-containers/community/issues/53) 69 | * Some commands accessing confidential data, such as `kubectl exec`, may either fail to work, or incorrectly expose information to the host 70 | * Container image sharing is not possible in this release 71 | * Container images are downloaded by the guest (with encryption), not by the host 72 | * As a result, the same image will be downloaded separately by every pod using it, not shared between pods on the same host. [More info](https://github.com/confidential-containers/community/issues/66) 73 | - The CoCo community aspires to adopting open source security best practices, but not all practices are adopted yet. 74 | * We track our status with the OpenSSF Best Practices Badge, which increased from 49% to 64% at the time of this release. 75 | * All CoCo repos now have automated tests, including linting, incorporated into CI. 76 | * Vulnerability reporting mechanisms still need to be created. Public github issues are still appropriate for this release until private reporting is established. 77 | 78 | 79 | ## CVE Fixes 80 | 81 | None 82 | -------------------------------------------------------------------------------- /releases/v0.6.0.md: -------------------------------------------------------------------------------- 1 | # Release Notes for v0.6.0 2 | Release Date: June 7th, 2023 3 | 4 | Please see the [quickstart guide](../quickstart.md) for details on how to try out Confidential 5 | Containers. 6 | 7 | Please refer to our [Acronyms](https://github.com/confidential-containers/documentation/wiki/Acronyms) 8 | and [Glossary](https://github.com/confidential-containers/documentation/wiki/Glossary) pages for a 9 | definition of the acronyms used in this document. 10 | 11 | ## What's new 12 | - Support for attesting pod VMs with Azure vTPMs on SEV-SNP 13 | - Support for using Project Amber as an attestation service 14 | - Support for Cosign signature validation with s390x 15 | - Pulling guest images with many layers can no longer cause guest CPU starvation. 16 | - Attestation Service upgraded to avoid several security issues in Go packages. 17 | - CC-KBC & KBS support with SGX attester/verifier for Occlum and CI for enclave-cc 18 | 19 | ## Hardware Support 20 | Confidential Containers is tested with attestation on the following platforms: 21 | - Intel TDX 22 | - AMD SEV(-ES) 23 | - Intel SGX 24 | 25 | The following platforms are untested or partially supported: 26 | - IBM Secure Execution (SE) on IBM zSystems (s390x) running LinuxONE 27 | - AMD SEV-SNP 28 | 29 | The following platforms are in development: 30 | - ARM CCA 31 | 32 | ## Limitations 33 | The following are known limitations of this release: 34 | 35 | - Platform support is rapidly changing 36 | * Image signature validation with AMD SEV-ES is not covered by CI. 37 | - SELinux is not supported on the host and must be set to permissive if in use. 38 | - The generic KBS does not yet supported all platforms. 39 | - The format of encrypted container images is still subject to change 40 | * The [oci-crypt](https://github.com/containers/ocicrypt) container image format itself may still change 41 | * The tools to generate images are not in their final form 42 | * The image format itself is subject to change in upcoming releases 43 | * Not all image repositories support encrypted container images. 44 | - CoCo currently requires a custom build of `containerd`, which is installed by the operator. 45 | * Codepath for pulling images will change significantly in future releases. 46 | * `crio` is only supported with `cloud-api-adaptor`. 47 | - Complete integration with Kubernetes is still in progress. 48 | * OpenShift support is not yet complete. 49 | * Existing APIs do not fully support the CoCo security and threat model. [More info](https://github.com/confidential-containers/community/issues/53) 50 | * Some commands accessing confidential data, such as `kubectl exec`, may either fail to work, or incorrectly expose information to the host 51 | * Container images must be downloaded separately (inside guest) for each pod. [More info](https://github.com/confidential-containers/community/issues/66) 52 | - The CoCo community aspires to adopting open source security best practices, but not all practices are adopted yet. 53 | * We track our status with the OpenSSF Best Practices Badge, which remained at 64% at the time of this release. 54 | * Vulnerability reporting mechanisms still need to be created. Public github issues are still appropriate for this release until private reporting is established. 55 | 56 | 57 | ## CVE Fixes 58 | 59 | None 60 | -------------------------------------------------------------------------------- /roadmap.md: -------------------------------------------------------------------------------- 1 | # Confidential Containers Roadmap 2 | When looking at the project's roadmap we distinguish between the short-term roadmap (2-4 months) vs. 3 | the mid/long-term roadmap (4-12 months): 4 | - The **short-term roadmap** is focused on achieving an end-to-end, easy to deploy confidential 5 | containers solution using at least one HW encryption solution and integrated to k8s (with forked 6 | versions if needed) 7 | - The **mid/long-term solutions** focuses on maturing the components of the short-term solution 8 | and adding a number of enhancements both to the solution and the project (such as CI, 9 | interoperability with other projects etc.) 10 | 11 | # Short-Term Roadmap 12 | The short-term roadmap aims to achieve the following: 13 | - MVP stack for running confidential containers 14 | - Based on and compatible with Kata Containers 2 15 | - Based on at least one confidential computing implementation (SEV, TDX, SE, etc) 16 | - Integration with Kubernetes: kubectl apply -f confidential-pod.yaml 17 | 18 | The work is targeted to be completed by end of November 2021 and includes 3 milestones: 19 | ![September 2021](./images/RoadmapSept2021.jpg) 20 | - **September 2021** 21 | - Unencrypted image pulled inside the guest, kept in tmpfs 22 | - Pod/Container runs from pulled image 23 | - Agent API is restricted 24 | - crictl only 25 | 26 | ![October 2021](./images/RoadmapOct2021.jpg) 27 | - **October 2021** 28 | - Encrypted image pulled inside the guest, kept in tmpfs 29 | - Image is decrypted with a pre-provisioned key (No attestation) 30 | 31 | ![November 2021](./images/RoadmapNov2021.jpg) 32 | - **November 2021** 33 | - Image is optionally stored on an encrypted, ephemeral block device 34 | - Image is decrypted with a key obtained from a key brokering service (KBS) 35 | - Integration with kubelet 36 | 37 | For additional details on each milestone see [Confidential Containers v0](https://docs.google.com/presentation/d/1SIqLogbauLf6lG53cIBPMOFadRT23aXuTGC8q-Ernfw/edit#slide=id.p). 38 | 39 | Tasks are tracked on a weekly basis through a dedicated spreadsheet. 40 | For more information see [Confidential Containers V0 Plan](https://docs.google.com/spreadsheets/d/1M_MijAutym4hMg8KtIye1jIDAUMUWsFCri9nq4dqGvA/edit#gid=0&fvid=1397558749). 41 | 42 | 43 | # Mid-Term Roadmap 44 | 45 | Continue our journey using knowledge and support of Subject Matter Experts (SME's) in other 46 | projects to form stronger opinions on what is needed from components which can be integrated to 47 | deliver the confidential containers objectives. 48 | 49 | - Harden the code used for the demos 50 | - Improve CI/CD pipeline 51 | - Clarify the release process 52 | - Establish processes and tools to support planning, prioritisation, and work in progress 53 | - Simple process to get up and running regardless of underlying Trusted Execution Environment 54 | technology 55 | - Develop a small, simple, secure, lightweight and high performance OCI container image 56 | management library [image-rs](https://github.com/confidential-containers/image-rs) for 57 | confidential containers. 58 | - Develop small, simple shim firmware ([td-shim](https://github.com/confidential-containers/td-shim)) 59 | in support of trusted execution environment for use with cloud native confidential containers. 60 | - Document threat model and trust model, what are we protecting, how are we achieving it. 61 | - Identify technical convergence points with other confidential computing projects both inside 62 | and outside CNCF. 63 | 64 | # Longer-Term Roadmap 65 | 66 | Focused meetings will be set up to discuss architecture and the priority of longer-term objectives 67 | in the process of being set up. 68 | 69 | Each meeting will have an agreed focus with people sharing material/thoughts ahead of time. 70 | 71 | Topics under consideration: 72 | - CI/CD + repositories 73 | - Community structure and expectations 74 | - 2 on Mid-Term Architecture 75 | - Attestation 76 | - Images 77 | - Runtimes 78 | 79 | Proposed Topics to influence long-term direction/architecture: 80 | - Baremetal / Peer Pod 81 | - Composability of alternative technologies to deliver confidential containers 82 | - Performance 83 | - Identity / Service Mesh 84 | - Reproducible builds/demos 85 | - Edge Computing 86 | - Reduce footprint of image pull 87 | 88 | -------------------------------------------------------------------------------- /threats_overview.md: -------------------------------------------------------------------------------- 1 | # Threat Vectors/Profiles 2 | 3 | Links to further documentation detailing specific threats and how Confidential Containers uses 4 | the trust concepts described in the context of the [Trust Model](./trust_model.md) will be added here. 5 | 6 | Current TODO List for Threats to be covered is tracked under Issues [#2](https://github.com/confidential-containers/documentation/issues/29) 7 | -------------------------------------------------------------------------------- /trust_model.md: -------------------------------------------------------------------------------- 1 | # Trust Model for Confidential Containers 2 | A clear definition of trust for the confidential containers project is needed to ensure the 3 | components and architecture deliver the security principles expected for cloud native 4 | confidential computing. It provides the solid foundations and unifying security principles 5 | against which we can assess architecture and implementation ideas and discussions. 6 | 7 | ## Trust Model Definition 8 | The [Trust Modeling for Security Architecture Development article](https://www.informit.com/articles/article.aspx?p=31546) 9 | defines Trust Modeling as : 10 | 11 | > A trust model identifies the specific mechanisms that are necessary to respond to a specific 12 | > threat profile. 13 | 14 | > A trust model must include implicit or explicit validation of an entity's identity or the 15 | > characteristics necessary for a particular event or transaction to occur. 16 | 17 | ## Trust Boundary 18 | The trust model also helps determine the location and direction of the trust boundaries where a 19 | [trust boundary](https://en.wikipedia.org/wiki/Trust_boundary) describes a location where 20 | program data or execution changes its level of "trust", or where two principals with different 21 | capabilities exchange data or commands. Specific to Confidential Containers is the trust 22 | boundary that corresponds to the boundary of the Trusted Execution Environment (TEE). The TEE 23 | side of the trust boundary will be hardened to prevent the violation of the trust 24 | boundary. 25 | 26 | ## Required Documentation 27 | In order to describe and understand particular threats we need to establish trust boundaries and 28 | trust models relating to the key aspects, components and actors involved in Cloud Native 29 | Confidential Computing. We explore trust using different orthogonal ways of considering cloud 30 | native approaches when they use an underlying TEE technology and 31 | identifying where there may be considerations to preserve the value of using a TEE. 32 | 33 | ### Trust Model Considerations 34 | - [Personas](./trust_model_personas.md) 35 | 36 | Further documentation will highlight specific [threat vectors](./threats_overview.md) in detail, 37 | considering risk, 38 | impact, mitigation etc as the project progresses. The Security Assurance section, Page 31, of 39 | Cloud Native Computing Foundation (CNCF) 40 | [Cloud Native Security Paper](https://github.com/cncf/tag-security/blob/main/security-whitepaper/v2/CNCF_cloud-native-security-whitepaper-May2022-v2.pdf) 41 | will guide this more detailed threat vector effort. 42 | 43 | ### Related Prior Effort 44 | 45 | Confidential Containers brings confidential computing into a cloud native context and should 46 | therefore refer to and build on trust and security models already defined. 47 | 48 | For example: 49 | 50 | - Confidential Computing Consortium (CCC) published 51 | "[A Technical Analysis of Confidential Computing](https://confidentialcomputing.io/wp-content/uploads/sites/85/2021/03/CCC-Tech-Analysis-Confidential-Computing-V1.pdf)" 52 | section 5 of which defines the threat model for confidential computing. 53 | - CNCF Security Technical Advisory Group published 54 | "[Cloud Native Security Whitepaper](https://github.com/cncf/tag-security/blob/main/security-whitepaper/v2/CNCF_cloud-native-security-whitepaper-May2022-v2.pdf)" 55 | - Kubernetes provides documentation : 56 | "[Overview of Cloud Native Security](https://kubernetes.io/docs/concepts/security/overview/)" 57 | - Open Web Application Security Project - 58 | "[Docker Security Threat Modeling](https://github.com/OWASP/Docker-Security/blob/main/001%20-%20Threats.md)" 59 | 60 | The commonality between confidential containers project and confidential computing is to reduce 61 | the ability for unauthorised access to data and code inside TEEs sufficiently such that this path 62 | is not an economically or logically viable attack during execution (5.1 Goal within the CCC 63 | publication 64 | [A Technical Analysis of Confidential Computing](https://confidentialcomputing.io/wp-content/uploads/sites/85/2021/03/CCC-Tech-Analysis-Confidential-Computing-V1.pdf)). 65 | 66 | This means our trust and threat modelling should 67 | - Focus on which aspects of code and data have integrity and/or confidentiality protections. 68 | - Focus on enhancing existing Cloud Native models in the context of exploiting TEEs. 69 | - Consider existing Cloud Native technologies and the role they can play for confidential containers. 70 | - Consider additional technologies to fulfil a role in Cloud Native exploitation of TEEs. 71 | 72 | ### Out of Scope 73 | 74 | The following items are considered out-of-scope for the trust/threat modelling within confidential 75 | containers : 76 | 77 | - Vulnerabilities within the application/code which has been requested to run inside a TEE. 78 | - Availability part of the Confidentiality/Integrity/Availability in CIA Triad. 79 | - Software TEEs. At this time we are focused on hardware TEEs. 80 | - Certain security guarantees are defined by the underlying TEE and these 81 | may vary between TEEs and generations of the same TEE. We take these guarantees at face value 82 | and will only highlight them where they become relevant to the trust model or threats we 83 | consider. 84 | 85 | ### Summary 86 | 87 | In practice, those deploying workloads into TEE environments may have varying levels of trust 88 | in the personas who have privileges regarding orchestration or hosting the workload. This trust 89 | may be based on factors such as the relationship with the owner or operator of the host, the 90 | software and hardware it comprises, and the likelihood of physical, software, or social 91 | engineering compromise. 92 | 93 | Confidential containers will have specific focus on preventing potential security threats at 94 | the TEE boundary and ensure privileges which are accepted within cloud native environment as 95 | crossing the boundary are mitigated from threats within the boundary. We cannot allow the 96 | security of the TEE to be under control of operations outside the TEE or from areas not trusted 97 | by the TEE. 98 | -------------------------------------------------------------------------------- /trust_model_personas.md: -------------------------------------------------------------------------------- 1 | # Trust Model Considerations - Personas 2 | 3 | ## Personas 4 | Otherwise referred to as actors or agents, these are individuals or groups capable of 5 | carrying out a particular threat. 6 | In identifying personas we consider : 7 | - The Runtime Environment, Figure 5, Page 19 of CNCF 8 | [Cloud Native Security Paper](https://github.com/cncf/tag-security/blob/main/security-whitepaper/v2/CNCF_cloud-native-security-whitepaper-May2022-v2.pdf). 9 | This highlights three layers, Cloud/Environment, Workload Orchestration, Application. 10 | - The Kubernetes 11 | [Overview of Cloud Native Security](https://kubernetes.io/docs/concepts/security/overview/) 12 | identifies the 4C's of Cloud Native 13 | Security as Cloud, Cluster, Container and Code. However data is core to confidential 14 | containers rather than code. 15 | - The Confidential Computing Consortium paper 16 | [A Technical Analysis of Confidential Computing v1.1](https://confidentialcomputing.io/wp-content/uploads/sites/85/2021/03/CCC-Tech-Analysis-Confidential-Computing-V1.pdf) 17 | defines Confidential Computing as the protection of data in use by performing computations in a 18 | hardware-based Trusted Execution Environment (TEE). 19 | 20 | In considering personas we recognise that a trust boundary exists between each persona and we 21 | explore how the least privilege principle (as described on Page 40 of 22 | [Cloud Native Security Paper](https://github.com/cncf/tag-security/blob/main/security-whitepaper/v2/CNCF_cloud-native-security-whitepaper-May2022-v2.pdf) 23 | ) should apply to any actions which cross these boundaries. 24 | 25 | Confidential containers can provide enhancements to ensure that the expected code/containers 26 | are the only code that can operate over the data. However any vulnerabilities within this code 27 | are not mitigated by using confidential containers, the Cloud Native Security Whitepaper 28 | details Lifecycle aspects that relate to the security of the code being placed into containers 29 | such as Static/Dynamic Analysis, Security Tests, Code Review etc which must still be followed. 30 | 31 | ![Personas Model](./images/persona_model.svg) 32 | 33 | Any of these personas could attempt to perform malicious actions: 34 | 35 | ### Infrastructure Operator 36 | This persona has privileges within the Cloud Infrastructure which includes the hardware and 37 | firmware used to provide compute, network and storage to the Cloud Native solution. 38 | They are responsible for availability of infrastructure used by the cloud native environment. 39 | - Have access to the physical hardware. 40 | - Have access to the processes involved in the deployment of compute/storage/memory used by any 41 | orchestration components and by the workload. 42 | - Have control over TEE hardware availability/type. 43 | - Responsibility for applying firmware updates to infrastructure including the TEE Technology. 44 | 45 | Example : Cloud Service Provider (CSP), Infrastructure as a Service (IaaS), Site Reliability Engineer, etc. 46 | (SRE) 47 | 48 | ### Orchestration Operator 49 | This persona has privileges within the Orchestration/Cluster. 50 | They are responsible for deploying a solution into a particular cloud native environment and 51 | managing the orchestration environment. 52 | For managed cluster this would also include the administration of the cluster control plane. 53 | - Control availability of service. 54 | - Control webhooks and deployment of workloads. 55 | - Control availability of cluster resources (data/networking/storage) and cluster 56 | services (Logging/Monitoring/Load Balancing) for the workloads. 57 | - Control the deployment of runtime artifacts initially required by the TEE during 58 | initialisation. 59 | These boot images once initialised will receive the workload. 60 | 61 | Example : A kubernetes administrator responsible for deploying pods to a cluster and 62 | maintaining the cluster. 63 | 64 | ### Workload Provider 65 | This persona designs and creates the orchestration objects comprising the solution (e.g. 66 | kubernetes pod descriptions etc). These objects reference containers published by Container Image Providers. 67 | In some cases the Workload and Container Image Providers may be the same entity. 68 | The solution defined is intended to provide the Application or Workload which in turn provides 69 | value to the data owners (customers and clients). 70 | The workload provider and data owner could be part of same company/organisation but 71 | following the least privilege principle the workload provider should not be able to view or 72 | manipulate end user data without informed consent. 73 | - Need to prove to customer aspects of compliance. 74 | - Defines what the solution requires in order to run and maintain compliance (resources, utility 75 | containers/services, storage). 76 | - Chooses the method of verifying the container images (from those supported by Container Image 77 | Provider) and obtains artifacts needed to allow verification to be completed within 78 | the TEE. 79 | - Provide the boot images initially required by the TEE during 80 | initialisation or designates a trusted party to do so. 81 | - Provide the attestation verification service, or designate a trusted party to provide the 82 | attestation verification service. 83 | 84 | Example : 3rd party software vendor, cloud solution provider 85 | 86 | ### Container Image Provider 87 | 88 | This persona is responsible for the part of the supply chain that builds container images and 89 | provides them for use by the solution. Since a workload can 90 | be composed of multiple containers, there may be multiple container image providers, some 91 | will be closely connected to the workload provider (business logic containers), others more 92 | independent to the workload provider (side car containers). The container image provider is expected 93 | to use a mechanism to allow provenance of container image to be established when a 94 | workload pulls in these images at deployment time. This can take the form of signing or encrypting 95 | the container images. 96 | - Builds container images. 97 | - Owner of business logic containers. These may contain proprietary algorithms, models or secrets. 98 | - Signs or encrypts the images. 99 | - Defines the methods available for verifying the container images to be used. 100 | - Publishes the signature verification key (public key). 101 | - Provides any decryption keys through a secure channel (generally to a key management system 102 | controlled by a Key Broker Service). 103 | - Provides other required verification artifacts (secure channel may be considered). 104 | - Protects the keys used to sign or encrypt the container images. 105 | 106 | It is recognised that hybrid options exist surrounding workload provider and container provider. 107 | For example the workload provider may choose to protect their supply chain by 108 | signing/encrypting their own container images after following the build patterns already 109 | established by the container image provider. 110 | 111 | Example : Sidecar Container Provider 112 | 113 | 114 | ### Data Owner 115 | Owner of data used, and manipulated by the application. 116 | - Concerned with visibility and integrity of their data. 117 | - Concerned with compliance and protection of their data. 118 | - Uses and shares data with solutions. 119 | - Wishes to ensure no visibility or manipulation of data is possible by 120 | Orchestration Operator or Cloud Operator personas. 121 | 122 | ## Discussion 123 | 124 | ### Data Owner vs. All Other Personas 125 | 126 | The key trust relationship here is between the Data Owner and the other personas. The Data Owner 127 | trusts the code in the form of container images chosen by the Workload Provider to operate across 128 | their data, however they do not trust the Orchestration Operator or Cloud Operator with their 129 | data and wish to ensure data confidentiality. 130 | 131 | 132 | ### Workload Provider vs. Container Image Provider 133 | 134 | The Workload Provider is free to choose Container Image Providers that will provide not only 135 | the images they need but also support the verification method they require. A key aspect to this 136 | relationship is the Workload Provider applying Supply Chain 137 | Security practices (as 138 | described on Page 42 of 139 | [Cloud Native Security Paper](https://github.com/cncf/tag-security/blob/main/security-whitepaper/v2/CNCF_cloud-native-security-whitepaper-May2022-v2.pdf) 140 | ) when considering Container 141 | Image Providers. So the Container Image Provider must support the Workload Providers 142 | ability to provide assurance to the Data Owner regarding integrity of the code. 143 | 144 | With Confidential Containers we match the TEE boundary to the most restrictive boundary which is 145 | between the Workload Provider and the Orchestration Operator. 146 | 147 | ### Orchestration Operator vs. Infrastructure Operator 148 | 149 | Outside the TEE we distinguish between the Orchestration Operator and the Infrastructure 150 | Operator due to nature of how they can impact the TEE and the concerns of Workload Provider and 151 | Data Owner. Direct threats exist from the Orchestration Operator as some orchestration actions 152 | must be permitted to cross the TEE boundary otherwise orchestration cannot occur. A key goal is to 153 | *deprivilege orchestration* and restrict the 154 | Orchestration Operators privileges across the boundary. However indirect threats exist 155 | from the Infrastructure Operator who would not be permitted to exercise orchestration APIs but 156 | could exploit the low level hardware or firmware capabilities to access or impact the contents 157 | of a TEE. 158 | 159 | ### Workload Provider vs. Data Owner 160 | 161 | Inside the TEE we need to be able to distinguish between the Workload Provider and Data Owner in recognition that 162 | the same workload (or parts such as logging/monitoring etc) can be re-used with different data 163 | sets to provide a service/solution. In the case of bespoke workload, the workload provider and 164 | Data Owner may be the same persona. As mentioned the Data Owner must have a level of 165 | trust in the Workload Provider to use and expose the data provided in an expected and approved 166 | manner. Page 10 of [A Technical Analysis of Confidential Computing v1.1](https://confidentialcomputing.io/wp-content/uploads/sites/85/2021/03/CCC-Tech-Analysis-Confidential-Computing-V1.pdf) 167 | , suggests some approaches to establish trust between them. 168 | 169 | The TEE boundary allows the introduction of secrets but just as we recognised the 170 | TEE does not provide protection from code vulnerabilities, we also recognised that a TEE cannot 171 | enforce complete distrust between Workload Provider and Data Owner. This means secrets within 172 | the TEE are at risk from both Workload Provider and Data Owner and trying to keep secrets 173 | which protect the workload (container encryption etc), separated from secrets to protect the 174 | data (data encryption) is not provided simply by using a TEE. 175 | 176 | Recognising that Data Owner and Workload Provider are separate personas helps us to 177 | identify threats to both data and workload independently and to recognise that any solution must 178 | consider the potential independent nature of these personas. 179 | Two examples of trust between Data Owner and Workload Provider are : 180 | - AI Models which are proprietary and protected requires the workload to be encrypted and not 181 | shared with the Data Owner. In this case secrets private to the Workload Provider are needed 182 | to access the workload, secrets requiring access to the data are provided by the Data Owner 183 | while trusting the workload/model without having direct access to how the workload functions. 184 | The Data Owner completely trusts the workload and Workload Provider, whereas the Workload 185 | Provider does not trust the Data Owner with the full details of their workload. 186 | - Data Owner verifies and approves certain versions of a workload, the workload provides the 187 | data owner with secrets in order to fulfil this. These secrets are available in the TEE for 188 | use by the Data Owner to verify the workload, once achieved the data owner will then provide 189 | secrets and data into the TEE for use by the workload in full confidence of what the workload 190 | will do with their data. The Data Owner will independently verify versions of the workload and 191 | will only trust specific versions of the workload with the data whereas the Workload Provider 192 | completely trusts the Data Owner. 193 | 194 | ### Data Owner vs. End User 195 | We do not draw a distinction between data owner and end user though we do recognise that in 196 | some cases these may not be identical. For example data may be provided to a workload to allow 197 | analysis and results to be made available to an end user. The original data is never provided 198 | directly to the end user but the derived data is, in this case the data owner can be different 199 | from the end user and may wish to protect this data from the end user. 200 | 201 | --------------------------------------------------------------------------------