├── README.md ├── proposals ├── NNN-skip-template │ └── README.md ├── 004-crd-domains │ └── README.md ├── 002-azure-repo │ └── README.md ├── 005-skip-approval-process │ └── README.md ├── 006-spinapp-invocation-limits │ └── README.md ├── 000-skips │ └── README.md └── 003-shim-runtime-options │ └── README.md └── LICENSE /README.md: -------------------------------------------------------------------------------- 1 | # SKIPs 2 | 3 | A SpinKube Improvement Proposal (SKIP) is a way to propose, communicate and 4 | coordinate changes for the SpinKube project. 5 | 6 | ## Quickstart 7 | 8 | 1. Socialize an idea with the relevant maintainers, you can reach out to people 9 | privately, via the #spinkube channel in the CNCF Slack, or by joining a SpinKube 10 | community meeting and adding it to the agenda. 11 | 2. Follow the process outlined in the [SKIP Template](proposals/NNN-skip-template/README.md). 12 | -------------------------------------------------------------------------------- /proposals/NNN-skip-template/README.md: -------------------------------------------------------------------------------- 1 | # SKIP 000 - SKIP NAME 2 | 3 | Summary: A one paragraph summary of your change 4 | 5 | Owner: The email of the proposer of the SKIP. 6 | 7 | Impacted Projects: 8 | 9 | - [ ] spin-operator 10 | - [ ] `spin kube` plugin 11 | - [ ] runtime-class-manager 12 | - [ ] containerd-shim-spin 13 | - [ ] Governance 14 | - [ ] Creates a new project 15 | 16 | Created: The date the SKIP was first written 17 | 18 | Updated: The date the SKIP was last refreshed 19 | 20 | ## Background 21 | 22 | What is the context for your change? Why do you want it? 23 | 24 | ## Proposal 25 | 26 | What is your change? What does it do? 27 | 28 | ## Alternatives Considered 29 | 30 | What else have you considered and why is the proposed solution a better fit? 31 | -------------------------------------------------------------------------------- /proposals/004-crd-domains/README.md: -------------------------------------------------------------------------------- 1 | # SKIP 004 - CRD Domains 2 | 3 | Summary: We should pivot to using the _spinkube.dev_ domain for all CRDs. 4 | 5 | Owner: Caleb Schoepp 6 | 7 | Impacted Projects: 8 | 9 | - [x] spin-operator 10 | - [ ] `spin kube` plugin 11 | - [x] runtime-class-manager 12 | - [ ] containerd-shim-spin 13 | - [ ] Governance 14 | - [ ] Creates a new project 15 | 16 | Created: 11/07/2024 17 | 18 | Updated: n/a 19 | 20 | ## Background 21 | 22 | Kubernetes resources are identified by a unique [group, version, and kind](https://book.kubebuilder.io/cronjob-tutorial/gvks). Furthermore, the resource must be namespaced under a valid domain name. Currently the Spin Operator project has two custom resource definitions (CRDs): `SpinApp` and `SpinAppExecutor`. These CRDs use the domain _spinoperator.dev_. 23 | 24 | ## Proposal 25 | 26 | I am proposing that we migrate from using _spinoperator.dev_ to _spinkube.dev_ as the domain for the Spin Operator CRDs. I also propose that going forward any new CRDs created by the Spin Operator project or any other SpinKube project use the _spinkube.dev_ domain. 27 | 28 | The benefit of this change include: 29 | 30 | - Consistency: All SpinKube projects will use the same domain for their CRDs. 31 | - Branding: The _spinkube.dev_ domain is more descriptive of the overall project than _spinoperator.dev_. 32 | - Future Proofing: The _spinkube.dev_ domain is more generic and allows for more flexibility in the future should anything about the Spin Operator change. 33 | 34 | Changing the domain of a CRD is a breaking change and will require a full upgrade process which involves uninstalling the old CRDs and reinstalling the new CRDs. This is suboptimal, but is only reasonably achievable early in the lifetime of project like we are right now. Breaking changes like this only get harder the longer we wait. 35 | 36 | ## Alternatives Considered 37 | 38 | ### Sticking with _spinoperator.dev_ 39 | 40 | Alternatively, we could stick with the status quo if we don't think the breaking change is worth it. 41 | -------------------------------------------------------------------------------- /proposals/002-azure-repo/README.md: -------------------------------------------------------------------------------- 1 | # SKIP 002 - SpinKube Azure 2 | 3 | Summary: A repository for all things Azure related in the SpinKube project 4 | 5 | Owner: Jiaxiao (Joe) Zhou 6 | 7 | Impacted Projects: 8 | 9 | - [ ] spin-operator 10 | - [ ] `spin kube` plugin 11 | - [ ] runtime-class-manager 12 | - [ ] containerd-shim-spin 13 | - [ ] Governance 14 | - [x] Creates a new project 15 | 16 | Created: June 4th, 2024 17 | 18 | Updated: June 4th, 2024 19 | 20 | ## Background 21 | 22 | I have worked on a project called [spinkube-azure](https://github.com/Mossaka/spinkube-azure) which essentially is a helm chart that deploys spinkube operator and its related components on Azure Kubernetes Service (AKS) cluster, with a focus on the Azure specific configurations and integrations. The goal of this project is to provide a one-stop shop for all things Azure related and to make it the eaisest way to deploy spinkube on AKS. Features may include: 23 | 24 | - AppInsights integration 25 | - Workload Identity integration 26 | - Azure Monitor integration 27 | - etc. 28 | 29 | It shares a lot of similarities with the [spinkube-oneclick](https://github.com/jpflueger/spinkube-oneclick) project, which is also not in the SpinKube organization. 30 | 31 | The intended usage of the project is twofold: 32 | 33 | 1. Provide an open-source Helm chart that users can easily install on their AKS clusters. 34 | 2. Package the Helm chart as an artifact for the Azure Marketplace, or as a Microsoft managed service such as AKS Extension. 35 | 36 | ## Proposal 37 | 38 | I am proposing to create a new repository under the SpinKube organization called `azure` to host the spinkube-azure project. This repository will be the home for all things Azure related in the SpinKube project. This will allow us to have a dedicated place to work on Azure specific features and configurations, and to make it easier for the community to contribute to the project. 39 | 40 | Specially I am proposing to move the [spinkube-azure](https://github.com/Mossaka/spinkube-azure) project to the new repository and rename it to `azure`. I will be the owner of the repository and will be responsible for maintaining it. 41 | 42 | A roadmap for the project will be created to outline the features and integrations that we want to add to the project. This will be a living document that will be updated as we make progress on the project. 43 | 44 | ## Alternatives Considered 45 | -------------------------------------------------------------------------------- /proposals/005-skip-approval-process/README.md: -------------------------------------------------------------------------------- 1 | # SKIP 005 - SKIP Approval Process 2 | 3 | Summary: A process for approving future SKIPs. 4 | 5 | Owner: Caleb Schoepp 6 | 7 | Impacted Projects: 8 | 9 | - [x] spin-operator 10 | - [x] `spin kube` plugin 11 | - [x] runtime-class-manager 12 | - [x] containerd-shim-spin 13 | - [x] Governance 14 | - [ ] Creates a new project 15 | 16 | Created: 30/10/2024 17 | 18 | Updated: n/a 19 | 20 | ## Background 21 | 22 | SKIPs were designed as a lightweight way to have discussions about changes that are more complex or affect multiple projects. They have worked well for this purpose, but we never codified how we should reach agreement around approving a SKIP. This proposal seeks to change that. 23 | 24 | ## Proposal 25 | 26 | The following conditions must be met for a SKIP to be considered approved: 27 | 28 | - The SKIP has been open for comment for at least two weeks and all comments are resolved. 29 | - The SKIP has been approved by at least two maintainers who are not an author. 30 | - The SKIP has been approved by at least one maintainer of each different impacted project[^1]. 31 | - The SKIP has been discussed as an agenda item in at least one SpinKube community meeting[^2]. 32 | 33 | Once a SKIP meets these conditions it is considered approved. The SKIP may now be merged at will by either the author or a reviewer. It is considered official once it has been merged. 34 | 35 | ### Disagreement 36 | 37 | Ideally the approval process of a SKIP will flow smoothly with disagreements being worked out in the SKIP PR. However, it is possible that a SKIP might encounter some blocking disagreement: 38 | 39 | - A maintainer is strongly opposed to the SKIP. 40 | - A maintainer disagrees with the impacted projects list. 41 | 42 | As SpinKube is still early in its life we shouldn't be too prescriptive about scenarios like this. If an impasse is reached it should be resolved by the SpinKube maintainers gathering to discuss and reach a consensus. 43 | 44 | ## Alternatives Considered 45 | 46 | ### Handling disagreement with a vote 47 | 48 | An alternative to solving disagreement through conversation and consensus was requiring an async vote from the maintainers with a super-majority. This process felt too rigid for where SpinKube is currently at and also introduces adversarial dynamics of who is included in the vote. 49 | 50 | [^1]: It is expected that the SKIP author accurately reflects which projects are impacted by the SKIP. If a reviewer believes a project is or is not impacted they may request the author to update the SKIP. 51 | [^2]: There are no quorum requirements. 52 | -------------------------------------------------------------------------------- /proposals/006-spinapp-invocation-limits/README.md: -------------------------------------------------------------------------------- 1 | # SKIP 006 - SpinApp invocation limits 2 | 3 | Summary: Add support for configuring per-invocation resource limits in `SpinApp` CRD 4 | 5 | Owner: Kate Goldenring 6 | 7 | Impacted Projects: 8 | 9 | - [x] spin-operator 10 | - [ ] `spin kube` plugin 11 | - [ ] runtime-class-manager 12 | - [ ] containerd-shim-spin 13 | - [ ] Governance 14 | - [ ] Creates a new project 15 | 16 | Created: 05-29-2025 17 | 18 | Updated: 05-29-2025 19 | 20 | ## Background 21 | 22 | Spin recently [added support for limiting instance memory](https://github.com/spinframework/spin/pull/3135) and has had an outstanding request for supporting [request timeouts](https://github.com/spinframework/spin/issues/30). The `SpinApp` CRD should support configuring per invocation limits. 23 | 24 | ## Proposal 25 | 26 | Add an `invocationLimits` section to the `SpinApp` CRD with a mapping of limits. The limits can be of two types: common and executor namespaced. The project will define common keys that executors *should* support, starting with `memory`. Executors can define additional limits that they support as namespaced keys; for example, the key for a `bar` limit specific to the Spintainer executor may be `spintainer.spinframework.dev/bar`. 27 | 28 | The Spin Operator will validate common limits. For `memory`, it will assert that the requested value does not exceed that of the `resources.limits.memory` section. Executors are responsible for documenting and validating their limits. One approach is to provide a validating webhook for the `SpinApp` resource to avoid unaccepted values being passed. 29 | 30 | The following is an example `SpinApp` that uses the `foo` executor, which supports the common `memory` limit and an executor specific `foo.dev/executionWallTime` limit: 31 | 32 | ```yaml 33 | apiVersion: core.spinkube.dev/v1alpha1 34 | kind: SpinApp 35 | metadata: 36 | name: hello 37 | spec: 38 | executor: foo 39 | image: "ghcr.io/spinframework/myapp:latest" 40 | replicas: 1 41 | invocationLimits: 42 | memory: "64Mi" 43 | foo.dev/executionWallTime: "30s" 44 | ``` 45 | 46 | For the Spin containerd shim executor, the Spin Operator can configure the per invocation memory limit by setting the (converted to bytes) value of the `memory` in the `SPIN_MAX_INSTANCE_MEMORY` environment variable on Deployments. This is similar to how OTEL settings [are configured](https://github.com/spinframework/spin-operator/blob/4900cf4bdfdc519c152e7d20f2529fccee7ba40f/internal/controller/deployment.go#L128). 47 | 48 | ```yaml 49 | apiVersion: core.spinkube.dev/v1alpha1 50 | kind: SpinApp 51 | metadata: 52 | name: hello 53 | spec: 54 | executor: containerd-shim-spin 55 | image: "ghcr.io/spinframework/myapp:latest" 56 | replicas: 1 57 | # Pod level resource limits 58 | resources: 59 | limits: 60 | memory: "128Mi" 61 | # Per-invocation limits 62 | invocationLimits: 63 | memory: "64Mi" 64 | ``` 65 | 66 | ## Alternatives Considered 67 | 68 | Instead of a generic mapping of limits, these limits could be defined to better validate `SpinApps`; however, a generic map enables executors to diverge in their support of limiting. 69 | 70 | Instead of adding an `invocationLimits` section, the `resources.limits` section could be overloaded with invocation limit specific resources. While this is an option, not all invocation limits map to the concept of a "resource", i.e. `executionWallTime`. Also, the `resources` must be Kubernetes resources, so these limits syntactically would be [Node extended resources](https://kubernetes.io/docs/tasks/administer-cluster/extended-resource-node/). There is concern that SpinApp overloading these concepts could lead to confusion. Finally, adding an distinct `invocationLimits` section highlights a key advantage of SpinKube: support of per invocation limits. 71 | -------------------------------------------------------------------------------- /proposals/000-skips/README.md: -------------------------------------------------------------------------------- 1 | # SKIP 000 - What is a SKIP? 2 | 3 | Summary: A lightweight process for coordinating changes to SpinKube 4 | 5 | Owner: Danielle Lancashire 6 | 7 | Impacted Projects: 8 | 9 | - [ ] spin-operator 10 | - [ ] `spin kube` plugin 11 | - [ ] runtime-class-manager 12 | - [ ] containerd-shim-spin 13 | - [x] Governance 14 | - [ ] Creates a new project 15 | 16 | Created: 18/04/2024 17 | 18 | Updated: 18/04/2024 19 | 20 | ## Background 21 | 22 | SpinKube is a new project that was the "coming together" of many seperate 23 | efforts under a common banner. We've quickly found that there are changes we 24 | wish to make that cross across multiple projects, and decisions that require 25 | broader scope to be understandable. We need a lightweight way to have those 26 | discussions and reach agreement about direction. 27 | 28 | ## Proposal 29 | 30 | A SKIP is a tool for recording a potential solution to a problem. It follows a 31 | specific format. There is a specific process for adopting a SKIP. And there is 32 | a process for recording when a SKIP is no longer the accepted practice. 33 | 34 | SKIPs are living documents. They change over time in both minor and major ways. 35 | The SKIP itself, combined with Git's change tracking, should indicate how it 36 | has changed. 37 | 38 | It is customary to refer to a SKIP by its number (”SKIP 004,” pronounced “skip 39 | oh-oh-four” or “skip four”). When necessary, it may also be referenced by its 40 | version (”SKIP 000 v1”) 41 | 42 | ### The Document Head 43 | 44 | Two pieces of information appear at the top of the document before the 45 | Background section: 46 | 47 | - **Overview:** This is one paragraph that summarizes the document. It should 48 | contain both a summary of the decision point (problem) and a 49 | summary of the proposed solution. 50 | 51 | - For any SKIPs exceeding a couple of pages, it is recommend that a table of 52 | contents also be added. In particular, with SKIPs that have many subheadings, 53 | the table of contents can serve as an outline of the structure of the proposal. 54 | 55 | ### The Background Section 56 | 57 | In the background section, clearly articulate the decision point that has 58 | necessitated this SKIP. Identify the key problem, a critical gap, or the 59 | motivating factor. A background section may be one paragraph or may be a few pages. 60 | The background section may be broken into multiple subsections (with subsections 61 | starting at Heading 3 level). 62 | 63 | On occasion, another document may clearly spell out the problem. In such cases, 64 | one is encouraged to point to that document as the definitive source, and merely 65 | summarize in the background section. 66 | 67 | However, if no such document does exist, the background section should provide 68 | ample material for understanding the issue. Links to other resources are 69 | encouraged, though the author should take into consideration the possibility 70 | that such resources may not always be available. 71 | 72 | ### The Proposal Section 73 | 74 | The Proposal section outlines a proposed solution, process, or practice that 75 | clearly answers the issue as outlined in the Background section. It will be one 76 | or more paragraphs, possibly divided up into subsections. Before a SKIP can be 77 | considered, the proposal must fully address the issue raised in the background. 78 | That is, a SKIP should not be used to elaborate a problem, but to provide a 79 | solution (or provide a partial solution) for a well-articulated problem. 80 | 81 | While a proposal may reference external documents, a proposal should stand on 82 | its own without requiring the reader to read the external documents. The purpose 83 | of this requirement is to ensure that SKIPs remain a digestible and 84 | authoritative source. 85 | 86 | ### The References Section 87 | 88 | The references section includes links to relevant sources of information. 89 | Not all links in the document must have an accompanying reference. This is not 90 | a works cited, but a bibliography that points to other related sources such as 91 | relevant SKIPs, external documentation, or witty XKCD comics. 92 | 93 | ## Alternatives Considered 94 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright (c) The Spin Framework Contributors. 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /proposals/003-shim-runtime-options/README.md: -------------------------------------------------------------------------------- 1 | # SKIP 003 - Configuring Runtime Options in the `containerd-shim-spin` 2 | 3 | Summary: Configuring Spin runtime options in the `containerd-shim-spin` 4 | 5 | Owner: Kate Goldenring 6 | 7 | Impacted Projects: 8 | 9 | - [ ] spin-operator 10 | - [ ] `spin kube` plugin 11 | - [x] runtime-class-manager 12 | - [x] containerd-shim-spin 13 | - [ ] Governance 14 | - [ ] Creates a new project 15 | 16 | Created: 30/05/2024 17 | 18 | ## Overview 19 | 20 | There is currently no defined way to configure the execution of the 21 | `containerd-shim-spin` SpinKube executor. This proposal aims to address this at 22 | two levels: Spin application execution configuration (Spin runtime and 23 | components) and Shim execution (containerd shim) The former is for the lifetime 24 | of a Spin application and the later is for the lifetime of a Spin executor. 25 | These configurations will be set via Custom Resource Definitions (CRDs) in 26 | Kubernetes, specifically extending the SpinApp and SpinAppExecutor CRDs. This 27 | approach will allow operators to easily configure both the application and the 28 | platform execution environments. 29 | 30 | ## Background 31 | 32 | Spin exposes various flags and environment variables in order to configure how 33 | an application is executed. Take, for example, the following command: 34 | 35 | ```sh 36 | OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 spin up --listen "127.0.0.1:3000" --disable-pooling --log-dir /var/logs --runtime-config-file /var/spin/config.toml 37 | ``` 38 | 39 | The previous command tells the Spin runtime to do the following: 40 | 41 | - Send traces of a Spin app to the exporter at `http://127.0.0.1:4318` 42 | - Listen for requests to the Spin app on IP address and port `127.0.0.1:3000` 43 | - Disable Wasmtime's pooling instance allocator 44 | - Use `/var/logs` as the log directory for the stdout and stderr of components 45 | - Use `/var/spin/config.toml` as the runtime configuration file. 46 | 47 | Currently, there is no way for configure any of the runtime flags through the 48 | `containerd-shim-spin` (also referred to as "the shim" in this document). On the 49 | other hand, environment variables can be exposed to the shim as container 50 | environment variables, since `youki` injects the container environment variables 51 | into the shim process before `run_wasi` is executed. For example, the 52 | `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable can be configured in a Spin 53 | application deployment as follows: 54 | 55 | ```yaml 56 | containers: 57 | - name: foo 58 | env: 59 | - name: NODE_IP 60 | valueFrom: 61 | fieldRef: 62 | fieldPath: status.hostIP 63 | - name: OTEL_EXPORTER_OTLP_ENDPOINT 64 | value: "http://$(NODE_IP):4318" 65 | ``` 66 | 67 | However, this experience is not SpinKube-native, as there are no fields for 68 | setting environment variables in the `SpinApp` CRD. Furthermore, Kubernetes 69 | users are used to container environment variables being used to configure an 70 | application not the runtime. This points to the need to distinguish between 71 | which environment variables are for the Spin runtime vs the Spin application 72 | components. 73 | 74 | The above discussed ways a user may wish to configure the an executor (namely 75 | the shim) for the lifetime of an application's execution. A user may also wish 76 | configuration to last for the lifetime of an executor (the lifetime of the 77 | `SpinAppExecutor` CRD). For example, a user may wish to disable precompilation 78 | for all applications. 79 | 80 | In summary, configuration can be broken out into two groups: 81 | 82 | - Spin application execution configuration - granular to the `SpinApp` CRD level 83 | - Spin runtime specific options (oftentimes in the form of CLI flags: `spin up 84 | --listen "127.0.0.1:3000"`) 85 | - Spin runtime environment variables (such as 86 | `OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 spin up` ) 87 | - Shim execution configuration - granular to the `SpinAppExecutor` level 88 | - Spin shim execution options (such as disabling pre-compilation) 89 | - Spin shim execution environment variables (such as `RUST LOG`) 90 | 91 | ## Terminology 92 | 93 | | Term | Definition | 94 | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | 95 | | Container runtime | A program that runs and manages the lifecycle of containers, or in this case Spin apps, i.e. Youki. | 96 | | CRD | Custom Resource Definition, a Kubernetes extension mechanism for defining custom resources | 97 | | Environment Variables | Variables that are part of the environment in which a process runs | 98 | | PodSpec | A Kubernetes object that describes the specification of a pod, including its containers, volumes, and other properties | 99 | | Shim | In this document, a shortened reference for the `containerd-shim-spin` containerd shim | 100 | | Spin | A CLI tool for scaffolding, building and running serverless Wasm components | 101 | | SpinApp | A custom resource definition (CRD) in Kubernetes that represents a Spin application | 102 | | SpinAppExecutor | A custom resource definition (CRD) in Kubernetes that represents a Spin executor (such as the Spin containerd shim) | 103 | | SpinKube | A Kubernetes operator for managing Spin applications | 104 | | Spin runtime | A Wasm execution engine that executes Wasm components using Wasmtime | 105 | | Spin Application Variables | Variables defined in a Spin application manifest (in the `[variables]` section) with values that can be dynamically updated by an [application variables provider](https://developer.fermyon.com/spin/v2/dynamic-configuration#application-variables-runtime-configuration). | 106 | | Wasmtime | The WebAssembly runtime that Spin uses to execute WebAssembly components | 107 | 108 | ## Proposal 109 | 110 | As laid out in the [background](#background), this SKIP aims to enable the 111 | following categories of configuration: 112 | 113 | - Spin application execution configuration 114 | - Executor configuration 115 | 116 | The first category pertains to the execution of a single Spin application, while 117 | the later discusses how to configure the platform. 118 | 119 | This SKIP focuses on how the `containerd-shim-spin` and `runwasi` will consume 120 | configuration. While this SKIP encourages the use of the SpinKube `SpinApp` and 121 | `SpinAppExecutor` CRDs to set configuration, it does not propose the structure 122 | of the changes to the CRDs to add configuration. That should be proposed in a 123 | future SKIP. 124 | 125 | ### Spin Application Execution Configuration 126 | 127 | Spin application execution can be configured via container environment 128 | variables, which are exposed to the shim. The following is a subset of 129 | configuration values that may be supported: 130 | 131 | | Key | Spin CLI | Example Value | 132 | | --------------------------- | ----------------------------------------------------------- | ----------------------- | 133 | | SPIN_HTTP_LISTEN_ADDR | `spin up --listen` | "0.0.0.0:3000" | 134 | | OTEL_EXPORTER_OTLP_ENDPOINT | `OTEL_EXPORTER_OTLP_ENDPOINT=http://123.4.5.6:4318 spin up` | "http://123.4.5.6:4318" | 135 | | SPIN_LOG_DIR | `spin up --log-dir /tmp/log` | "/tmp/log" | 136 | | SPIN_RUNTIME_CONFIG_FILE | `spin up --runtime-config-file /var/config.toml` | "/var/config.toml" | 137 | | AWS_DEFAULT_REGION | `AWS_DEFAULT_REGION="us-west-2" spin up` | "us-west-2" | 138 | | AWS_ACCESS_KEY_ID | `AWS_ACCESS_KEY_ID="ABC" spin up` | "ABC" | 139 | | AWS_SECRET_ACCESS_KEY | `AWS_SECRET_ACCESS_KEY="123" spin up` | "123" | 140 | | AWS_SESSION_TOKEN | `AWS_SESSION_TOKEN="token" spin up` | "token" | 141 | 142 | These values will be configured as environment variables in the Container spec. 143 | For example, the following sets the listen address for the HTTP trigger: 144 | 145 | ```sh 146 | spec 147 | containers: 148 | - name: myspinapp 149 | image: "ghcr.io/myuser:v1" 150 | env: 151 | - name: SPIN_HTTP_LISTEN_ADDR 152 | value: "0.0.0.0:3000" 153 | ... 154 | ``` 155 | 156 | The `containerd-shim-spin` instance will look for execution environment 157 | variables and configure execution accordingly. 158 | 159 | At first, this feels strange, since it differs from user's previous 160 | understanding of Linux container environment variables. Deploying a Spin app 161 | should not be compared to deploying a Linux container. A Linux container has one 162 | virtualized environment and listener. In a Linux container context, environment 163 | variables are set in the app execution and affect the business logic. On the 164 | other hand, a Spin app can have N many triggers and components. The Wasm runtime 165 | embedded in Spin (Wasmtime) will isolate each component in its own virtualized 166 | environment upon execution. While Spin does support setting environment 167 | variables in components, they are static. While the Spin CLI does support 168 | setting the same set of environment variables in all components of an app (`spin 169 | up --env`), it is more of a developer shortcut and not advised. It makes each 170 | sandbox bigger than necessary and doesn't emphasize the component model's 171 | strengths. See the [Spin Application Variables 172 | section](#spin-application-variables) for a discussion on how variables will be 173 | exposed to application components. 174 | 175 | #### Spin Application Variables 176 | 177 | Instead of configuring their applications with environment variables, SpinKube 178 | users should use [Spin application 179 | variables](https://developer.fermyon.com/spin/v2/variables#application-variables). 180 | 181 | These variables are declared in a Spin application manifest (`spin.toml`) and 182 | their values can be configured in the [`SpinApp` 183 | CRD](https://www.spinkube.dev/docs/spin-operator/reference/spin-app/#spinappspecvariablesindex) 184 | if using the environment variable provider. Application variables can also be 185 | dynamically updated using [Azure Key 186 | Vault](https://developer.fermyon.com/spin/v2/dynamic-configuration#azure-key-vault-application-variable-provider) 187 | or 188 | [Vault](https://developer.fermyon.com/spin/v2/dynamic-configuration#azure-key-vault-application-variable-provider). 189 | 190 | However, requiring that all variables be set by an administrator is likely too 191 | limiting for Kubernetes contexts where there are some cases in which the 192 | scheduler or webhooks inject environment variables into containers that 193 | components wish to access. For example, by default Kubernetes injects [cluster 194 | information environment 195 | variables](https://kubernetes.io/docs/concepts/containers/container-environment/#cluster-information) 196 | such as `KUBERNETES_SERVICE_HOST` and `KUBERNETES_SERVICE_PORT` into each 197 | container. Many Kubernetes libraries expect these to be available and would be 198 | incompatible otherwise. However, the shim should not simply inject all container 199 | environment variables into every component's environment. Instead, the shim will 200 | expose environment variables as application variable, by resetting them in the 201 | environment with the `SPIN_VARIABLE` prefix. It will only do this the 202 | application variables listed in the Spin manifest (`spin.toml`). This means that 203 | Spin components can get access to these variables if specifically configured in 204 | the Spin manifest to have access. For example, say a a component wants access to 205 | the `KUBERNETES_SERVICE_ADDRESS` container environment variable, it would 206 | configure this in the Spin.toml as follows: 207 | 208 | ```toml 209 | [variables] 210 | kubernetes_service_address = { required = true } 211 | 212 | [component.example] 213 | source = "target/wasm32-wasi/release/example.wasm" 214 | allowed_outbound_hosts = [] 215 | [component.example.variables] 216 | kubernetes_service_address = "{{ kubernetes_service_address }}" 217 | ``` 218 | 219 | Then, it can be accessed in the Spin app using the variables SDK: 220 | 221 | ```rust 222 | let addr = variables::get("kubernetes_service_address")?; 223 | ``` 224 | 225 | In documentation, we should clarify that Spin Wasm components can only be 226 | configured with Spin application variables. The variable names must be known and 227 | set in the Spin manifest, as in the above example. 228 | 229 | #### Spin Shim Executor Configuration 230 | 231 | Users may wish to configure settings for the executors at large, beyond the 232 | scope of a single application's execution. Containerd supports [configuring 233 | options for 234 | runtimes](https://github.com/containerd/containerd/blob/main/docs/man/containerd-config.toml.5.md#multiple-runtimes), 235 | such as the Spin shim. These options are set in the containerd configuration 236 | file (`config.toml`), which is where the runtime class manager notifies 237 | containerd of the existence of the shim. The Runtime Class Manager could also 238 | add shim options when setting up the shim. For example, it may make the 239 | following additions to a containerd configuration file: 240 | 241 | ```toml 242 | # Tell containerd where to find the shim (it parses this as /containerd/path/containerd-shim-spin-v2) 243 | [plugins."io.containerd.grpc.v1.cri".containerd.runtimes."spin"] 244 | runtime_type = "io.containerd.spin.v2" 245 | # Add options to each execution of the shim 246 | [plugins."io.containerd.grpc.v1.cri".containerd.runtimes."spin".options] 247 | Foo = "bar" 248 | DISABLE_PRECOMPILATION = "true" 249 | ``` 250 | 251 | > For an example implementation, see [gVisor's containerd runtime options 252 | > configuration 253 | > documentation](https://gvisor.dev/docs/user_guide/containerd/configuration/). 254 | 255 | When a container is executed with CRI, these options are sent to Runwasi in a 256 | [`CreateTaskRequest`](https://github.com/containerd/containerd/blob/main/core/runtime/v2/README.md#container-level-shim-configuration). 257 | However, right now Runwasi does not parse them. Runwasi should parse the 258 | options,and pass them to the shim `Engine` through a new `configure_options` 259 | method that should run prior to any other methods. Runwasi may also extract it's 260 | own configuration from these containerd runtime options. 261 | 262 | The following are Spin shim and Runwasi options that should be configurable: 263 | 264 | | Key | Example Value | Scope | 265 | | ------------------------------- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | 266 | | DISABLE_PRECOMPILATION | "true" | containerd-shim-spin or [Runwasi](https://github.com/containerd/runwasi/blob/7cc5e5a8025f1508c264dee1faf1c8c57cf645a1/crates/containerd-shim-wasm/src/sandbox/containerd/client.rs#L38) | 267 | | GRPC_MAX_WRITE_CHUNK_SIZE_BYTES | "15728640" | [Runwasi](https://github.com/containerd/runwasi/blob/7cc5e5a8025f1508c264dee1faf1c8c57cf645a1/crates/containerd-shim-wasm/src/sandbox/containerd/client.rs#L38) | 268 | 269 | Ideally, these options could be set in the `SpinAppExecutor` CRD. Then, the 270 | RuntimeClassManager can watch the resource and update and restart containerd as 271 | needed. 272 | 273 | ### Spin Shim Executor Environment Variables 274 | 275 | Since containerd shims inherit the environment variables of the calling 276 | containerd process, environment variables that need to be set on the shim 277 | process need to be set on containerd. Configuring containerd environment 278 | variables differs depending on how the service is executed. When directly 279 | invoking containerd, the log level of the shim could be adjusted as follows: 280 | 281 | ```sh 282 | RUST_LOG=info containerd --address /tmp/containerd.sock 283 | ``` 284 | 285 | If the process is managed by systemd, you can add an environment variable to the 286 | systemd service file. First, stop the service and bring up the service file: 287 | 288 | ```sh 289 | systemctl stop containerd 290 | systemctl edit containerd 291 | ``` 292 | 293 | > note, for k3s, edit the k3s service (`systemctl edit k3s`) 294 | 295 | At the top of the file, add the environment variable to the service: 296 | 297 | ```sh 298 | [Service] 299 | Environment="RUST_LOG=info" 300 | ``` 301 | 302 | Then the containerd service must be restarted to surface the change. 303 | 304 | ```sh 305 | systemctl restart containerd 306 | ``` 307 | 308 | To automate the above steps for a SpinKube user, a log level field could be 309 | added to the `SpinAppExecutor` and the Runtime Class Manager could handle 310 | configuring and restarting containerd. 311 | 312 | ## References 313 | 314 | - Runwasi issue for tracking containerd config options for shims: 315 | https://github.com/containerd/runwasi/issues/573 316 | - `containerd-shim-spin` issue tracking inability to run two Spin apps in the 317 | same container due to fixed listen address: 318 | https://github.com/spinkube/containerd-shim-spin/issues/52 319 | - `containerd-shim-spin` issue considering which `spin up` flags to support in 320 | the shim: https://github.com/spinkube/containerd-shim-spin/issues/130 321 | 322 | ## Alternatives Considered 323 | 324 | ### Configuring Spin Execution Alternatives 325 | 326 | The following are alternate strategies for configuring the Spin runtime in the 327 | `containerd-shim-spin`. 328 | 329 | 1. Spin execution options could be configured in in Pod annotations; however, it 330 | is not granular enough for the case of two Spin apps in a Pod, for example, 331 | [listen ports could still 332 | conflict](https://github.com/spinkube/containerd-shim-spin/issues/52). 333 | 334 | 2. As shown in the [Spin runtime configuration 335 | workaround](#wasm-runtime-specific-options-workaround), options could be set 336 | as container arguments, but setting flags in CRDs is less ergonomic than 337 | environment variables, and parsing logic would need to be added to the shim. 338 | However, taking this path would potentially enable reusing Spin's reconciling 339 | of the mutual exclusiveness of flags. 340 | 341 | 3. Spin Runtime config (`runtime-config.toml`) could be enhanced to support 342 | configuring triggers. Right now, the Spin runtime config file is used to 343 | configure host components. So far, it has been used the describe how to 344 | configure the host environment for the guest. That could encompass 345 | considering how the host should be triggered to invoke the guest. This may be 346 | the better long term solution; however, it will take design to consider the 347 | scope of Spin runtime config. 348 | 349 | 4. Alternatively, there may be places each of these configuration options fit 350 | outside of CLI flags. For example, the port for HTTP triggered apps could be 351 | configured in the trigger specification in the `spin.toml`. Log directory is 352 | already a part of [runtime 353 | config](https://github.com/fermyon/spin/blob/b3db535c9edb72278d4db3a201f0ed214e561354/crates/trigger/src/runtime_config.rs#L203-L204) 354 | despite the fact that users are more comfortable configuring it through the 355 | Spin CLI. 356 | 357 | 5. Having an extensible way to configure trigger execution will be important to 358 | trigger developers. Spin could consider supporting a trigger config to do 359 | this. 360 | 6. Flags for `spin up` could be exposed as container arguments rather than 361 | environment variables, such as the following in a Kubernetes PodSpec: 362 | ```sh 363 | containers: 364 | - name: spin-test 365 | image: ghcr.io/deislabs/containerd-wasm-shims/examples/spin-rust-hello:v0.10.0 366 | command: ["/"] 367 | args: ["--listen", "0.0.0.0:3000"] 368 | ``` 369 | While the shim currently has access to these arguments from the 370 | [`RuntimeContext` 371 | args](https://github.com/containerd/runwasi/blob/f5f497c4b21a5d55613095a3ae878c4ef4b83b91/crates/containerd-shim-wasm/src/container/context.rs#L14), 372 | the shim does not parse these arguments and would need to be enlightened in 373 | a similar manner as the Spin CLI to discern which are for which trigger. 374 | While this is approach is technically feasible, the user experience is poor. 375 | It unnecessarily requires the user to understand the Spin CLI in order to 376 | use the Spin runtime on Kubernetes. 377 | 7. Instead of setting container environment variables as Spin application 378 | variables, all container environment variables could be injected into all 379 | components of the Spin application by default. As explained in the [Spin 380 | Application Variables](#spin-application-variables), this expands the Wasm 381 | sandbox and does not fit with the objectives of the component model. 382 | --------------------------------------------------------------------------------