├── LICENSE ├── README.md ├── api ├── bootstrap.go ├── provider.go └── types.go └── contributing.md /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | Copyright 2015 Openstorage.org. 179 | 180 | Licensed under the Apache License, Version 2.0 (the "License"); 181 | you may not use this file except in compliance with the License. 182 | You may obtain a copy of the License at 183 | 184 | http://www.apache.org/licenses/LICENSE-2.0 185 | 186 | Unless required by applicable law or agreed to in writing, software 187 | distributed under the License is distributed on an "AS IS" BASIS, 188 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 189 | See the License for the specific language governing permissions and 190 | limitations under the License. 191 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Specifications for Container Storage Interfaces 2 | 3 | The purpose of this project is to define the various (vendor agnostic) interfaces between cloud native schedulers and persistent data services. 4 | 5 | ### Issues addressed by this spec 6 | 7 | 1. Deployment of the data service provider by the scheduler: The data service provider, perhaps packaged as a container should be deployed on the servers being managed by the scheduler. 8 | 9 | 2. Inline storage service provisioning: Users should be able to allocate the data service resources programatically via the scheduler interface. This obsoletes the need to do static out-of-band volume provisioning. 10 | 11 | 3. Data locality aware scheduling: The scheduler should be able to take into account the locality of a container's data, before it schedules it on a host. 12 | 13 | 4. Scheduler driven data life cycle management: The life cycle of a container and it's storage are different. The scheduler should be able to manage both, separately as independent entities. 14 | 15 | 5. Propagation of the data service properties via the scheduler: When a storage resource is created, the properties of the resource should be transparently passed through by the scheduler. This obsoletes the need for such information being provided out-of-band via other methods. 16 | 17 | 6. Common protocol of communication via the data service, scheduler and container runtime engine: The data service provider should be able to allocate resources and manage them using the same (or close to similar) protocol regardless of it being used by the scheduler agent or the container runtime engine, like Docker. 18 | 19 | 7. Application awareness facilitated to the data service layer: The data service layer should have broader context of the application that is being deployed. An application will comprise of many containers and having a broader context enables the data service layer to optimize and implement the correct HA features. As an example, consider the deployment of a Cassandra ring. Knowing the various containers that are part of the ring will help the data service provider to appropriately place the data in the correct failure domains. 20 | 21 | 8. Authentication of access to a data service facilitated via the scheduler: Prohibit a data service provider from allowing a container to use a service (such as a volume) that it is not authorized to use. 22 | 23 | 9. Monitoring - Alerts and Stats propagated via the scheduler: A common framework to get alerts and stats via the scheduler is desired. This prevents the need for external event correlation. 24 | 25 | ### Organization of the spec 26 | This spec covers two aspects of orchestrating the deployment of data services via a scheduler: 27 | 28 | 1. The bootstrap deployment of the data service container itself. 29 | 2. The runtime communication between a scheduler agent and the data service container. 30 | 31 | ## Bootstrap Deployment of Data Service Resources 32 | This section of the spec describes how data service providers are deployed by orchestration software. For example, these providers can be packaged as Linux Containers and they would need to be depoyed on the physical infrastructure by the orchestration software. This is specified in [api/bootstrap.go](api/bootstrap.go). 33 | 34 | ## Runtime communication between the scheduler and the data service 35 | Once the data service has been deployed, there are 4 specific interfaces that schedulers and data service providers need to implement. This is specified in [api/provider.go](api/provider.go). The scheduler and the provider could communicate via a runtime `UNIX sock` file on the agent host machine (TBD). 36 | 37 | ### 1. Discovery of Data Services 38 | Applications that rely on data services should be able to dynamically discover where the provisioned resources are available. The data service API should also be able to influence where and when these services should be scheduled based on the underlying constraints. 39 | 40 | ### 2. Provisioning and Instantiation of Data Services 41 | The allocation, use (read and write) and destruction (what used to be known as CRUD) needs to be orchestrated through this interface. 42 | 43 | ### 3. Lifecycle Operations on Data Services (TBD) 44 | Data state and its lifecycle, such as retention levels, version levels, access controls should be separated from the actual application that uses them. It should also be controlled by the scheduling software and it is the goal of this API to define how that is goverened. 45 | 46 | ### 4. Security (TBD) 47 | This defines a set of constraints around how a container can authenticate itself in order to operate on a storage service. This would prevent a container launched by a user from accessing a volume they do not have access to. 48 | 49 | ## Licensing 50 | `CNCF-CSI` is licensed under the Apache License, Version 2.0. See LICENSE for the full license text. 51 | 52 | ## Contributing 53 | Want to collaborate and add? Here are instructions to [get started contributing code](contributing.md) 54 | -------------------------------------------------------------------------------- /api/bootstrap.go: -------------------------------------------------------------------------------- 1 | package api 2 | 3 | /* 4 | #include 5 | #cgo LDFLAGS: -lcgroup 6 | */ 7 | import "C" 8 | 9 | // Cgroup is the structure describing one or more control groups. The structure 10 | // is opaque to applications. 11 | type Cgroup struct { 12 | g *C.struct_cgroup 13 | } 14 | 15 | // Device structure represents the type storage being provided to the 16 | // data service. 17 | type Device struct { 18 | // Type could be a string such as "block", "ebs", "nfs" etc. It is 19 | // up to the data service to interpret the device type. 20 | Type string 21 | 22 | // Metadata contains device type specific constraints and information. 23 | // For example, for an EBS volume type, it can contain the AWS access keys. 24 | Metadata map[string]string 25 | } 26 | 27 | // Geography physical location of the node. 28 | type Geography struct { 29 | Zone string 30 | Rack int 31 | } 32 | 33 | // Node contains details regarding a specific host. This information 34 | // will be provided to the service being deployed. This information is 35 | // provided as a file on the host in yaml format. 36 | type Node struct { 37 | 38 | // ID is unique node identifier. 39 | ID string 40 | 41 | // IPs is the list of IPs for this node. 42 | IPs []string 43 | 44 | // Geography is the physical location of the node. 45 | Geography Geogrpahy 46 | 47 | // Devices is the list of devices. 48 | Devices []Device 49 | 50 | // Constraints are cgroup restriuctions on the service container. 51 | Constraints Cgroup 52 | 53 | // ClusterID uniquely identifies the cluster that this 54 | // data service is part of. 55 | ClusterID string 56 | 57 | // Metadata provides arbitrary name value pairs. 58 | Metadata map[string]string 59 | } 60 | 61 | // Bootstrap contains information for the scheduler. It instructs the scheduler 62 | // to deploy a given service on a set of nodes from a container image. 63 | type Bootstrap struct { 64 | // DataServiceName data service name. 65 | DataServiceName string 66 | 67 | // Image name to execute on the host. 68 | Image string 69 | } 70 | 71 | // Installer will manage the provisioning of data services on a set of 72 | // machines as per the bootstrap and node information. 73 | type Installer interface { 74 | // Deploy data service to set of nodes. 75 | Deploy(b *Bootstrap, Nodes []Node) 76 | 77 | // Upgrade data service on set of nodes. 78 | Upgrade(b *Bootstrap, Nodes []Node) 79 | 80 | // Destroy data service on set of nodes. 81 | Destroy(b *Bootstrap, Nodes []Node) 82 | } 83 | -------------------------------------------------------------------------------- /api/provider.go: -------------------------------------------------------------------------------- 1 | package api 2 | 3 | import ( 4 | "net/url" 5 | "time" 6 | ) 7 | 8 | // Modified version from https://github.com/codedellemc/libstorage as an example. 9 | // This is a minimal definition. Ultimately, this will be the simplest and 10 | // most concise definition that consolidates the goodness from muliple 11 | // service management drivers. 12 | 13 | // Service definition. (TBD) 14 | type Service struct { 15 | } 16 | 17 | // ServiceSpec are options when creating a new data service. 18 | type ServiceSpec struct { 19 | AvailabilityZone *string 20 | IOPS *int64 21 | Size *int64 22 | Encrypted *bool 23 | EncryptionKey *string 24 | Options map[string]string 25 | } 26 | 27 | type Capability int 28 | 29 | const ( 30 | CapabilityEncryption Capability = iota 31 | CapabilityCompresssion 32 | CapabilityDeduplication 33 | CapabilityReplication 34 | CapabilityDR 35 | CapabilityMulitAZ 36 | CapabilityConverged 37 | ) 38 | 39 | type DataService struct { 40 | // ServiceType could be a string such as object, block, file. 41 | ServiceType string 42 | Size uint64 43 | Iops uint64 44 | Capabilities []Capability 45 | } 46 | 47 | type CreateOptions struct { 48 | // SrcID Create service from source ID 49 | SrcID string 50 | // LateBinding allow creation of volume but defers resource allocation 51 | // to when the service is instantiated. 52 | LateBinding bool 53 | } 54 | 55 | // Provder implements a data service provider. This interface implements the 56 | // union of the the data service's CRUD commands as well as it's 57 | // lifecycle operations. 58 | type Provider interface { 59 | // Type returns the type of storage the driver provides. 60 | Type() (string, error) 61 | 62 | // ServiceType advertises the type of service (block, object, file etc) 63 | // offered by this provider on a given node. 64 | ServiceType() (DataService, error) 65 | 66 | // SchedulerQuery returns a list of nodes (IPs) that are preferred nodes 67 | // to run a container on, given a set of opts. 68 | SchedulerQuery( 69 | opts map[string]string, 70 | ) ([]string, error) 71 | 72 | // Enumerate all services that satisfy contraints defined by opts. 73 | Enumerate(opts map[string]string) ([]*Service, error) 74 | 75 | // Inspect inspects a single service. 76 | Inspect( 77 | ID string, 78 | opts map[string]string, 79 | ) (*Service, error) 80 | 81 | // Create creates a new service. 82 | Create( 83 | name string, 84 | spec *ServiceSpec, 85 | createOpts *CreateOptions, 86 | opts map[string]string, 87 | ) (*Service, error) 88 | 89 | // Backup to provider. 90 | Backup(ID string, provider Provider) 91 | 92 | // Snapshot snapshots a service. 93 | Snapshot( 94 | ID, snapshotName string, 95 | opts map[string]string, 96 | ) (*Service, error) 97 | 98 | // Remove removes a service. 99 | Remove( 100 | ID string, 101 | opts map[string]string, 102 | ) error 103 | 104 | // Attach attaches a service and provides a token clients can use 105 | // to validate that device has appeared locally. 106 | Attach( 107 | ID string, 108 | opts map[string]string, 109 | ) (*Service, string, error) 110 | 111 | // Detach detaches a service. 112 | Detach( 113 | ID string, 114 | opts map[string]string, 115 | ) (*Service, error) 116 | 117 | // Mount service to specific path. 118 | Mount( 119 | ID, mountpoint string, 120 | opts map[string]string, 121 | ) error 122 | 123 | // Unmount service to specific path. 124 | Unmount( 125 | ID, mountpoint string, 126 | opts map[string]string, 127 | ) error 128 | 129 | // Stat returns the service and network statistics for this provider 130 | // on a given node. 131 | Stat() (ServiceStat, NetStat, error) 132 | 133 | // LogStats provides an logging URL for the provider dump 134 | // service stats to. An interval of 0 stops the logging. 135 | LogStats(url url.URL, interval time.Duration) error 136 | 137 | // Alerts returns the alerts for this provider on a given node. 138 | Alerts() ([]Alert, error) 139 | 140 | // LogAlerts provides an alerting URL for the provider dump 141 | // service alerts to. An interval of 0 stops the logging. 142 | LogAlerts(url url.URL, interval time.Duration) error 143 | } 144 | -------------------------------------------------------------------------------- /api/types.go: -------------------------------------------------------------------------------- 1 | package api 2 | 3 | import ( 4 | "time" 5 | ) 6 | 7 | type ServiceStat struct { 8 | BytesRead uint64 9 | NumReads uint64 10 | BytesWritten uint64 11 | NumWrites uint64 12 | IoDepth uint64 13 | QueueTime uint64 14 | } 15 | 16 | type NetStat struct { 17 | BytesSent uint64 18 | BytesReceived uint64 19 | } 20 | 21 | const ( 22 | HiAlert = iota 23 | MedAlert 24 | LowAlert 25 | ) 26 | 27 | type Alert struct { 28 | AlertType int 29 | AlertData []byte 30 | TimeStamp time.Time 31 | Id uint64 32 | } 33 | -------------------------------------------------------------------------------- /contributing.md: -------------------------------------------------------------------------------- 1 | # Contributing 2 | 3 | The specification and code is licensed under the Apache 2.0 license found in 4 | the `LICENSE` file of this repository. 5 | 6 | See the [Style Guide](STYLEGUIDE.md). 7 | 8 | ### Sign your work 9 | 10 | The sign-off is a simple line at the end of the explanation for the 11 | patch, which certifies that you wrote it or otherwise have the right to 12 | pass it on as an open-source patch. The rules are pretty simple: if you 13 | can certify the below (from 14 | [developercertificate.org](http://developercertificate.org/)): 15 | 16 | ``` 17 | Developer Certificate of Origin 18 | Version 1.1 19 | 20 | Copyright (C) 2004, 2006 The Linux Foundation and its contributors. 21 | 660 York Street, Suite 102, 22 | San Francisco, CA 94110 USA 23 | 24 | Everyone is permitted to copy and distribute verbatim copies of this 25 | license document, but changing it is not allowed. 26 | 27 | 28 | Developer's Certificate of Origin 1.1 29 | 30 | By making a contribution to this project, I certify that: 31 | 32 | (a) The contribution was created in whole or in part by me and I 33 | have the right to submit it under the open source license 34 | indicated in the file; or 35 | 36 | (b) The contribution is based upon previous work that, to the best 37 | of my knowledge, is covered under an appropriate open source 38 | license and I have the right under that license to submit that 39 | work with modifications, whether created in whole or in part 40 | by me, under the same open source license (unless I am 41 | permitted to submit under a different license), as indicated 42 | in the file; or 43 | 44 | (c) The contribution was provided directly to me by some other 45 | person who certified (a), (b) or (c) and I have not modified 46 | it. 47 | 48 | (d) I understand and agree that this project and the contribution 49 | are public and that a record of the contribution (including all 50 | personal information I submit with it, including my sign-off) is 51 | maintained indefinitely and may be redistributed consistent with 52 | this project or the open source license(s) involved. 53 | ``` 54 | 55 | then you just add a line to every git commit message: 56 | 57 | Signed-off-by: Joe Smith 58 | 59 | using your real name (sorry, no pseudonyms or anonymous contributions.) 60 | 61 | You can add the sign off when creating the git commit via `git commit -s`. 62 | --------------------------------------------------------------------------------