├── .gitignore ├── LICENSE ├── README.md ├── client_quick_setup.md ├── client_quick_setup_openshift.md ├── deploy ├── client-certificate-ds.yaml ├── client-ds.yaml ├── config.yaml ├── data-ds.yaml ├── example-pod.yaml ├── fio-benchmark-ds.yaml ├── ingress.yaml ├── legacy-client-ds.yaml ├── metadata-ds.yaml ├── qmgmt-pod.yaml ├── quobyte-ns.yaml ├── quobyte-services.yaml ├── registry-ds.yaml └── webconsole-deployment.yaml ├── operator ├── README.md └── deploy │ ├── client-config.yaml │ ├── client.yaml │ ├── operator.yaml │ ├── qmgmt-pod.yaml │ ├── quobyte-config.yaml │ ├── quobyte-ns.yaml │ ├── services-config.yaml │ └── services.yaml ├── server_quick_setup.md ├── tools ├── qbootstrap ├── qmkdev └── setup-remote ├── using_quobyte_volumes.md └── volumes ├── claim.json ├── example-pod.yaml ├── pv.yaml ├── quobyte-admin-secret.yaml └── storageclass.yaml /.gitignore: -------------------------------------------------------------------------------- 1 | disks 2 | .vagrant 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2017-2018, Quobyte Inc. 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are met: 6 | 7 | * Redistributions of source code must retain the above copyright notice, this 8 | list of conditions and the following disclaimer. 9 | 10 | * Redistributions in binary form must reproduce the above copyright notice, 11 | this list of conditions and the following disclaimer in the documentation 12 | and/or other materials provided with the distribution. 13 | 14 | * Neither the name of quobyte-automation nor the names of its 15 | contributors may be used to endorse or promote products derived from 16 | this software without specific prior written permission. 17 | 18 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 19 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 20 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 21 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 22 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 23 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 24 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 25 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 26 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 27 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 28 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Scripts and tools for deploying Quobyte on Kubernetes 2 | ===================================================== 3 | 4 | **DEPRECATED in favour of [Quobyte CSI](https://github.com/quobyte/quobyte-csi)** 5 | 6 | Currently contains: 7 | * **operator** The recommended way to run Quobyte clients and servers in your k8s cluster. 8 | * **deploy** Deployment files for Quobyte on Kubernetes 9 | * **volumes** Examples how to use PersistentVolumes, PersistentVolumeClaims, and StorageClassses with Quobyte 10 | * **tools** device initialization tools: **qbootstrap** and **qmkdev** 11 | 12 | Please see the quick start guides for instructions: 13 | 14 | * Manual deployment of Quobyte [clients](client_quick_setup.md) and [services](server_quick_setup.md). 15 | * [Operator based setup for Quobyte clients and services](operator/README.md) 16 | 17 | These blog posts will give you some more insights and ideas how to benefit from these integrations: 18 | * [“Quobernetes” or How to Enable Simple and Efficient Storage Operations for Kubernetes](https://www.quobyte.com/blog/2018/08/27/quobernetes-is-kubernetes-with-quobyte-software-storage/) 19 | * [The State of Secure Storage Access in Container Infrastructures](https://www.quobyte.com/blog/2017/03/17/the-state-of-secure-storage-access-in-container-infrastructures/) 20 | -------------------------------------------------------------------------------- /client_quick_setup.md: -------------------------------------------------------------------------------- 1 | # Quick Start Guide to use Quobyte from Kubernetes 2 | 3 | ## Prerequisites 4 | If you use Docker 1.12 or older, you need to add the MountFlags parameter. 5 | For newer versions of Docker, this is not required anymore. 6 | 7 | Ensure MountFlags are shared or not set: 8 | ```bash 9 | $ systemctl cat docker.service | grep MountFlags=shared 10 | ``` 11 | 12 | If not you can set the MountFlags as shared with the following commands: 13 | ```bash 14 | $ cat << EOF > /etc/systemd/system/docker.service.d/slave-mount-flags.conf 15 | [Service] 16 | MountFlags=shared 17 | EOF 18 | 19 | $ systemctl daemon-reload && systemctl restart docker 20 | ``` 21 | 22 | Also make sure your Kubernetes cluster runs with the `MountPropagation` feature 23 | gate enabled for kubelets as well as kube-apiservers. Refer to a 24 | [feature support matrix](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) 25 | for details. 26 | 27 | ## Client setup 28 | This guide assumes that you have a dedicated Quobyte server running and you 29 | want to provide access to Quobyte volumes to pods running in Kubernetes. 30 | 31 | To access a Quobyte volume a pod has to run on a Kubernetes node which has a 32 | Quobyte client running. The client runs inside of a Pod and makes the Quobyte 33 | storage accessible to other pods. 34 | 35 | We provide two different DaemonSet definitions: the [recommended one](deploy/client-ds.yaml) 36 | and [legacy one](deploy/legacy-client-ds.yaml). The former can be used only with 37 | explicit user and group mapping. The latter allows you to resolve user and group 38 | identity using a host Name Service Switch (NSS), which we do not recommend as 39 | this approach poses certain security and system stability risks. We provide the 40 | legacy DaemonSet configuration for backward compatibility reasons only. If using 41 | identity mapping feature is not suitable for your deployment scenario we 42 | recommend you to create a customized version of our client Docker image. This 43 | way you can apply necessary changes in the container's NSS facility to meet your 44 | requirements. 45 | 46 | Quobyte clients run in the quobyte namespace. 47 | ```bash 48 | $ cd deploy 49 | $ kubectl create -f quobyte-ns.yaml 50 | ``` 51 | 52 | To connect to Quobyte, the client needs to resolve the address of the registry. 53 | It is configured in the client-ds.yaml DaemonSet definition: 54 | ```yaml 55 | env: 56 | - name: QUOBYTE_REGISTRY value: registry.quobyte ``` If you have a QNS 57 | (Quobyte Name Service), the value for QUOBYTE_REGISTRY might look like 58 | `.myquobyte.net` 59 | 60 | If you have a certificate for the client, it is stored as a Secret and 61 | mounted into the client Pod as client.cfg. 62 | 63 | First create a file that contains only the certificate information 64 | (, , and blocks) and store it as a secret. 65 | ```bash 66 | $ kubectl -n quobyte create secret generic client-config --from-file /tmp/client.cfg 67 | ``` 68 | 69 | ```bash 70 | $ kubectl -n quobyte create -f client-ds.yaml 71 | or 72 | $ kubectl -n quobyte create -f client-certificate-ds.yaml 73 | ``` 74 | 75 | The deployed DaemonSet starts client pods on all nodes marked as 76 | `quobyte_client`. This can either be done manually, or by using the Quobyte 77 | operator. 78 | 79 | ```bash 80 | $ kubectl label nodes quobyte_client="true" 81 | ``` 82 | 83 | When the client pod is up and running, you should see a mount point on the Kubernetes node 84 | at `/var/lib/kubelet/plugins/kubernetes.io~quobyte`. 85 | 86 | Please note: Only those nodes labeled with the 'quobyte_client' label and hence 87 | are running the Quobyte client can provide access to Quobyte storage to other pods 88 | running on the node. 89 | 90 | ##Benchmarking 91 | 92 | For easy testing and benchmarking we provide a fio-container which uses 93 | Quobyte volumes. By default, it will start writing to volume `fio-test` 94 | 95 | ```bash 96 | $ kubectl create -f fio-benchmark-ds.yaml 97 | ``` 98 | This will start a single Pod on a node which is marked as quobyte_client. 99 | The container is designed to put load on the volume, so you can scale it: 100 | 101 | ```bash 102 | $ kubectl scale --replicas=100 deployment fio-benchmark 103 | ``` 104 | 105 | # Volume Access 106 | See [using_quobyte_volumes.md](using_quobyte_volumes.md) 107 | -------------------------------------------------------------------------------- /client_quick_setup_openshift.md: -------------------------------------------------------------------------------- 1 | # Quick Start Guide to use Quobyte from OpenShift Origin 2 | 3 | Accessing Quobyte volumes from kubernetes is currently partly supported. 4 | We fully support mounting PersistentVolumes, but dynamic provisioning with storage 5 | classes is not yet supported and requires an external storage 6 | provisioner as described 7 | [here](https://docs.openshift.org/latest/install_config/provisioners.html). 8 | 9 | ## Client setup 10 | 11 | In an OpenShift environment, you need to create a service 12 | account which runs the client container. 13 | 14 | ```bash 15 | $ oc create serviceaccount quobyteclientsrv -n quobyte 16 | $ oc adm policy add-scc-to-user privileged -n quobyte -z quobyteclientsrv 17 | ``` 18 | Then, uncomment the line 19 | ```yaml 20 | serviceAccountName: quobyteclientsrv 21 | ``` 22 | in client-ds.yaml or client-certificate-ds.yaml. 23 | 24 | The remaining steps are the same as found in the [client quick setup](client_quick_setup.md). 25 | -------------------------------------------------------------------------------- /deploy/client-certificate-ds.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1beta2 2 | kind: DaemonSet 3 | metadata: 4 | name: client 5 | namespace: quobyte 6 | spec: 7 | selector: 8 | matchLabels: 9 | role: client 10 | template: 11 | metadata: 12 | annotations: 13 | prometheus.io/scrape: 'true' 14 | prometheus.io/path: '/prometheus' 15 | prometheus.io/port: '55000' 16 | labels: 17 | role: client 18 | version: "2" 19 | spec: 20 | containers: 21 | - name: quobyte-client 22 | image: quay.io/quobyte/quobyte-client:2 23 | imagePullPolicy: Always 24 | env: 25 | - name: QUOBYTE_CLIENT_LOG_LEVEL 26 | value: INFO 27 | - name: QUOBYTE_REGISTRY 28 | value: registry.quobyte 29 | - name: QUOBYTE_MOUNT_POINT 30 | # Note that the mount point has to be a subdir of the volume(Mount) 31 | value: /mnt/kubernetes.io~quobyte 32 | - name: NODENAME 33 | valueFrom: 34 | fieldRef: 35 | fieldPath: spec.nodeName 36 | ports: 37 | - name: http-port 38 | containerPort: 55000 39 | hostPort: 55000 40 | protocol: TCP 41 | readinessProbe: 42 | timeoutSeconds: 5 43 | httpGet: 44 | port: 55000 45 | path: / 46 | livenessProbe: 47 | initialDelaySeconds: 30 48 | timeoutSeconds: 5 49 | httpGet: 50 | port: 55000 51 | path: / 52 | command: 53 | - /bin/bash 54 | - -xec 55 | - | 56 | if cut -d" " -f2 /proc/self/mounts | grep -q ${QUOBYTE_MOUNT_POINT}; then 57 | umount -l ${QUOBYTE_MOUNT_POINT} 58 | fi 59 | 60 | mkdir -p /root/.quobyte ${QUOBYTE_MOUNT_POINT} 61 | 62 | if find "$QUOBYTE_MOUNT_POINT" -mindepth 1 -print -quit 2>/dev/null | grep -q .; then 63 | echo "POLLUTED MOUNT POINT DETECTED! Cannot use $QUOBYTE_MOUNT_POINT as a mount point." 64 | echo "Please remove all files and directories from $QUOBYTE_MOUNT_POINT and " 65 | echo "run 'chattr +i $QUOBYTE_MOUNT_POINT' to prevent future mount point pollution." 66 | else 67 | # set the mount point immutable. As long as mount.quobyte does not run, 68 | # other processes cannot write data to this dir. 69 | chattr +i ${QUOBYTE_MOUNT_POINT} || \ 70 | echo "WARNING: The local filesystem does not support IMMUTABLE flag. Mount point pollution is possible." 71 | mkdir -p /root/.quobyte 72 | cp /quobytecert/client.cfg /root/.quobyte/client.cfg 73 | /usr/bin/mount.quobyte --hostname ${NODENAME} \ 74 | -c ./root/.quobyte/client.cfg \ 75 | --allow-usermapping-in-volumename --http-port 55000 -f \ 76 | -d ${QUOBYTE_CLIENT_LOG_LEVEL} -l /dev/stdout ${OPTS} \ 77 | --minidump-path /tmp/minidumps --allow-minidump-upload \ 78 | ${QUOBYTE_REGISTRY}/ ${QUOBYTE_MOUNT_POINT} 79 | fi 80 | 81 | 82 | securityContext: 83 | privileged: true 84 | volumeMounts: 85 | - name: k8s-plugin-dir 86 | mountPath: /mnt 87 | mountPropagation: Bidirectional 88 | - name: etcfs 89 | mountPath: /etcfs 90 | - name: configs 91 | mountPath: /quobytecert 92 | - name: minidumps-dir 93 | mountPath: /tmp/minidumps 94 | lifecycle: 95 | preStop: 96 | exec: 97 | command: ["/bin/bash", "-xc", "umount -l ${QUOBYTE_MOUNT_POINT}"] 98 | hostPID: true 99 | nodeSelector: 100 | quobyte_client: "true" 101 | volumes: 102 | - name: k8s-plugin-dir 103 | hostPath: 104 | path: /var/lib/kubelet/plugins/ 105 | - name: etcfs 106 | hostPath: 107 | path: /etc 108 | - name: minidumps-dir 109 | hostPath: 110 | path: /var/lib/quobyte/.minidumps 111 | - name: configs 112 | secret: 113 | defaultMode: 420 114 | items: 115 | - key: client.cfg 116 | path: client.cfg 117 | secretName: client-config 118 | -------------------------------------------------------------------------------- /deploy/client-ds.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: DaemonSet 3 | metadata: 4 | name: client 5 | namespace: quobyte 6 | spec: 7 | selector: 8 | matchLabels: 9 | role: client 10 | template: 11 | metadata: 12 | annotations: 13 | prometheus.io/scrape: 'true' 14 | prometheus.io/path: '/prometheus' 15 | prometheus.io/port: '55000' 16 | labels: 17 | role: client 18 | version: "2" 19 | spec: 20 | containers: 21 | - name: quobyte-client 22 | image: quay.io/quobyte/quobyte-client:2 23 | imagePullPolicy: Always 24 | env: 25 | - name: QUOBYTE_CLIENT_LOG_LEVEL 26 | value: INFO 27 | - name: QUOBYTE_REGISTRY 28 | value: registry.quobyte 29 | - name: QUOBYTE_MOUNT_POINT 30 | # Note that the mount point has to be a subdir of the volume(Mount) 31 | value: /mnt/kubernetes.io~quobyte 32 | - name: NODENAME 33 | valueFrom: 34 | fieldRef: 35 | fieldPath: spec.nodeName 36 | ports: 37 | - name: http-port 38 | containerPort: 55000 39 | hostPort: 55000 40 | protocol: TCP 41 | readinessProbe: 42 | timeoutSeconds: 5 43 | httpGet: 44 | port: 55000 45 | path: / 46 | livenessProbe: 47 | initialDelaySeconds: 30 48 | timeoutSeconds: 5 49 | httpGet: 50 | port: 55000 51 | path: / 52 | command: 53 | - /bin/bash 54 | - -xec 55 | - | 56 | if cut -d" " -f2 /proc/self/mounts | grep -q ${QUOBYTE_MOUNT_POINT}; then 57 | umount -l ${QUOBYTE_MOUNT_POINT} 58 | fi 59 | 60 | mkdir -p /root/.quobyte ${QUOBYTE_MOUNT_POINT} 61 | 62 | if find "$QUOBYTE_MOUNT_POINT" -mindepth 1 -print -quit 2>/dev/null | grep -q .; then 63 | echo "POLLUTED MOUNT POINT DETECTED! Cannot use $QUOBYTE_MOUNT_POINT as a mount point." 64 | echo "Please remove all files and directories from $QUOBYTE_MOUNT_POINT and " 65 | echo "run 'chattr +i $QUOBYTE_MOUNT_POINT' to prevent future mount point pollution." 66 | else 67 | # set the mount point immutable. As long as mount.quobyte does not run, 68 | # other processes cannot write data to this dir. 69 | chattr +i ${QUOBYTE_MOUNT_POINT} || \ 70 | echo "WARNING: The local filesystem does not support IMMUTABLE flag. Mount point pollution is possible." 71 | 72 | /usr/bin/mount.quobyte --hostname ${NODENAME} \ 73 | --allow-usermapping-in-volumename --http-port 55000 -f \ 74 | -d ${QUOBYTE_CLIENT_LOG_LEVEL} -l /dev/stdout ${OPTS} \ 75 | --minidump-path /tmp/minidumps --allow-minidump-upload \ 76 | ${QUOBYTE_REGISTRY}/ ${QUOBYTE_MOUNT_POINT} 77 | fi 78 | 79 | 80 | securityContext: 81 | privileged: true 82 | volumeMounts: 83 | - name: k8s-plugin-dir 84 | mountPath: /mnt 85 | mountPropagation: Bidirectional 86 | - name: minidumps-dir 87 | mountPath: /tmp/minidumps 88 | lifecycle: 89 | preStop: 90 | exec: 91 | command: ["/bin/bash", "-xc", "umount -l ${QUOBYTE_MOUNT_POINT}"] 92 | nodeSelector: 93 | quobyte_client: "true" 94 | volumes: 95 | - name: k8s-plugin-dir 96 | hostPath: 97 | path: /var/lib/kubelet/plugins/ 98 | - name: minidumps-dir 99 | hostPath: 100 | path: /var/lib/quobyte/.minidumps 101 | -------------------------------------------------------------------------------- /deploy/config.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | name: quobyte-config 5 | namespace: quobyte 6 | data: 7 | registry.min_mem: 64m 8 | registry.max_mem: 512m 9 | metadata.min_mem: 256m 10 | metadata.max_mem: 1024m 11 | data.min_mem: 256m 12 | data.max_mem: 1024m 13 | api.min_mem: 64m 14 | api.max_mem: 300m 15 | webconsole.min_mem: 64m 16 | webconsole.max_mem: 512m 17 | -------------------------------------------------------------------------------- /deploy/data-ds.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: DaemonSet 3 | metadata: 4 | name: data 5 | namespace: quobyte 6 | spec: 7 | template: 8 | metadata: 9 | annotations: 10 | prometheus.io/scrape: 'true' 11 | prometheus.io/path: '/prometheus' 12 | prometheus.io/port: '7873' 13 | labels: 14 | role: data 15 | version: "2" 16 | spec: 17 | containers: 18 | - name: quobyte-data 19 | image: quay.io/quobyte/quobyte-server:2 20 | # imagePullPolicy: Always 21 | # resources: 22 | # requests: 23 | # memory: "512Mi" 24 | # cpu: "200m" 25 | 26 | # priviliged is required for full device inspector support 27 | securityContext: 28 | privileged: true 29 | env: 30 | - name: QUOBYTE_SERVICE 31 | value: data 32 | - name: QUOBYTE_REGISTRY 33 | value: registry 34 | - name: MAX_MEM 35 | valueFrom: 36 | configMapKeyRef: 37 | name: quobyte-config 38 | key: data.max_mem 39 | - name: MIN_MEM 40 | valueFrom: 41 | configMapKeyRef: 42 | name: quobyte-config 43 | key: data.min_mem 44 | command: 45 | - /bin/bash 46 | - -xec 47 | - | 48 | sed "s/.*MIN_MEM_DATA=.*/MIN_MEM_DATA=${MIN_MEM}/" -i /etc/default/quobyte 49 | sed "s/.*MAX_MEM_DATA=.*/MAX_MEM_DATA=${MAX_MEM}/" -i /etc/default/quobyte 50 | exec /bin/bash -x /opt/main.sh 51 | volumeMounts: 52 | - mountPath: /run/udev 53 | name: host-udev 54 | - mountPath: /lib/modules 55 | name: lib-modules 56 | - mountPath: /mnt/quobyte 57 | name: mnt-quobyte 58 | - mountPath: /var/lib/quobyte 59 | name: var-lib-quobyte 60 | mountPropagation: Bidirectional 61 | ports: 62 | - name: rpc-tcp 63 | containerPort: 7873 64 | protocol: TCP 65 | readinessProbe: 66 | timeoutSeconds: 5 67 | httpGet: 68 | port: 7873 69 | path: / 70 | livenessProbe: 71 | initialDelaySeconds: 30 72 | timeoutSeconds: 5 73 | httpGet: 74 | port: 7873 75 | path: / 76 | nodeSelector: 77 | quobyte_data: "true" 78 | volumes: 79 | - name: host-udev 80 | hostPath: 81 | path: /run/udev 82 | - name: lib-modules 83 | hostPath: 84 | path: /lib/modules 85 | - name: mnt-quobyte 86 | hostPath: 87 | path: /mnt/quobyte 88 | - name: var-lib-quobyte 89 | hostPath: 90 | path: /var/lib/quobyte 91 | -------------------------------------------------------------------------------- /deploy/example-pod.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Pod 3 | metadata: 4 | name: example 5 | spec: 6 | containers: 7 | - name: busybox 8 | image: busybox:1.25 9 | volumeMounts: 10 | - name: quobytevolume 11 | mountPath: /persistent 12 | command: 13 | - /bin/sh 14 | - -ec 15 | - | 16 | if [ -f /persistent/log ]; then 17 | echo -n "Found old state starting at "; head -n1 /persistent/log 18 | else 19 | echo -n "Starting with a fresh state" 20 | fi 21 | while sleep 5; do date | tee -a /persistent/log; done 22 | volumes: 23 | - name: quobytevolume 24 | quobyte: 25 | registry: ignored:7861 26 | volume: testVolume 27 | readOnly: false 28 | user: root 29 | group: root 30 | -------------------------------------------------------------------------------- /deploy/fio-benchmark-ds.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1beta2 2 | kind: Deployment 3 | metadata: 4 | name: fio-benchmark 5 | labels: 6 | app: fio-benchmark 7 | spec: 8 | replicas: 1 9 | selector: 10 | matchLabels: 11 | app: fio-benchmark 12 | template: 13 | metadata: 14 | labels: 15 | app: fio-benchmark 16 | spec: 17 | containers: 18 | - name: fio 19 | image: datawiseio/fio:latest 20 | volumeMounts: 21 | - name: quobytevolume 22 | mountPath: /data 23 | command: ["/usr/bin/fio"] 24 | args: ["--name=k8sfio", 25 | "--rw=write", 26 | "--numjobs=1", 27 | "--iodepth=1", 28 | "--filesize=1G", 29 | "--runtime=120s", 30 | "--time_based", 31 | "--filename_format=$(NODE_NAME)-$jobnum", 32 | "--directory=/data/", 33 | "--output-format=json", 34 | "--output=/data/$(NODE_NAME).out"] 35 | env: 36 | - name: NODE_NAME 37 | valueFrom: 38 | fieldRef: 39 | fieldPath: spec.nodeName 40 | nodeSelector: 41 | quobyte_client: "true" 42 | volumes: 43 | - name: quobytevolume 44 | quobyte: 45 | registry: LBIP:7861 46 | volume: fio-test 47 | readOnly: false 48 | user: root 49 | group: root 50 | -------------------------------------------------------------------------------- /deploy/ingress.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: Ingress 3 | metadata: 4 | name: quobyte 5 | namespace: quobyte 6 | spec: 7 | rules: 8 | - http: 9 | paths: 10 | - path: /api 11 | backend: 12 | serviceName: api 13 | servicePort: 80 14 | - path: / 15 | backend: 16 | serviceName: webconsole 17 | servicePort: 80 18 | -------------------------------------------------------------------------------- /deploy/legacy-client-ds.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: DaemonSet 3 | metadata: 4 | name: client 5 | namespace: quobyte 6 | spec: 7 | selector: 8 | matchLabels: 9 | role: client 10 | template: 11 | metadata: 12 | annotations: 13 | prometheus.io/scrape: 'true' 14 | prometheus.io/path: '/prometheus' 15 | prometheus.io/port: '55000' 16 | labels: 17 | role: client 18 | version: "2" 19 | spec: 20 | containers: 21 | - name: quobyte-client 22 | image: quay.io/quobyte/quobyte-client:2 23 | imagePullPolicy: Always 24 | env: 25 | - name: QUOBYTE_CLIENT_LOG_LEVEL 26 | value: INFO 27 | - name: QUOBYTE_REGISTRY 28 | value: registry.quobyte 29 | - name: QUOBYTE_MOUNT_POINT 30 | # Note that the mount point has to be a subdir of the volume(Mount) 31 | value: /var/lib/kubelet/plugins/kubernetes.io~quobyte 32 | - name: NODENAME 33 | valueFrom: 34 | fieldRef: 35 | fieldPath: spec.nodeName 36 | ports: 37 | - name: http-port 38 | containerPort: 55000 39 | hostPort: 55000 40 | protocol: TCP 41 | readinessProbe: 42 | timeoutSeconds: 5 43 | httpGet: 44 | port: 55000 45 | path: / 46 | livenessProbe: 47 | initialDelaySeconds: 30 48 | timeoutSeconds: 5 49 | httpGet: 50 | port: 55000 51 | path: / 52 | command: 53 | - /bin/bash 54 | - -xec 55 | - | 56 | if [[ ! -f /etcfs/fuse.conf ]]; then 57 | echo "Copy fuse config to host" 58 | { echo -e '# Copied from Quobyte Client Container\n'; cat /etc/fuse.conf; } > /etcfs/fuse.conf 59 | fi 60 | if [[ $(grep "^[^#]" /etcfs/fuse.conf | grep -c "user_allow_other") -eq 0 ]]; then 61 | echo "user_allow_other" >> /etcfs/fuse.conf 62 | fi 63 | if cut -d" " -f2 /etcfs/mtab | grep -q ${QUOBYTE_MOUNT_POINT}; then 64 | echo "found existing mount point on host. Trying to umount ${QUOBYTE_MOUNT_POINT}" 65 | if ! /bin/nsenter -t 1 -m -- /bin/umount -f ${QUOBYTE_MOUNT_POINT}; then 66 | /bin/nsenter -t 1 -m -- /bin/umount -l ${QUOBYTE_MOUNT_POINT} 67 | fi 68 | sleep 1 # give os the chance to clean up everything. 69 | else 70 | if ! [[ $(ls `dirname ${QUOBYTE_MOUNT_POINT}`|egrep "^`basename ${QUOBYTE_MOUNT_POINT}`$") ]]; then 71 | echo "mount point ${QUOBYTE_MOUNT_POINT} does not exist, creating it..." 72 | mkdir -p ${QUOBYTE_MOUNT_POINT} 73 | fi 74 | fi 75 | 76 | # set the mount point immutable. As long as mount.quobyte does not run, 77 | # other processes cannot write data to this dir. 78 | chattr +i ${QUOBYTE_MOUNT_POINT} 79 | 80 | # Until k8s mountPropagation is stable, nsenter is used for the client 81 | # to mount Quobyte into the host's mount namespace. By using nsenter, the client process 82 | # will also use the host's dns resolve mechanisms, which most probably will not include 83 | # the k8s internal name resolution. So the client is not able to resolve k8s internal 'registry.quobyte' anymore. 84 | # Hence, for k8s internal Quobyte clusters, we need to pre-resolve the current ips, and let the client 85 | # work with ips and not dns names. 86 | if [ "$QUOBYTE_REGISTRY" == "registry" ] || [ "$QUOBYTE_REGISTRY" == "registry.quobyte" ] ; then 87 | registry_ips= 88 | for ip in $(nslookup ${QUOBYTE_REGISTRY} | grep ${QUOBYTE_REGISTRY} -A2 | grep Address | cut -d':' -f2 | awk '{print $1}'); do 89 | registry_ips="$ip $registry_ips" 90 | done 91 | ADDR=$(echo $registry_ips | tr ' ' ',') 92 | echo "Assuming to connect to k8s hosted Quobyte cluster. Resolved DNS to: '$ADDR'" 93 | else 94 | ADDR=${QUOBYTE_REGISTRY} 95 | echo "Assuming to connect to an external Quobyte cluster. Client will resolve DNS for: '$ADDR'" 96 | fi 97 | 98 | /bin/nsenter -t 1 --wd=. -m -- \ 99 | lib/ld-linux-x86-64.so.2 \ 100 | --library-path ./lib \ 101 | /usr/bin/mount.quobyte --hostname ${NODENAME} --allow-usermapping-in-volumename \ 102 | --http-port 55000 -f -l /dev/stdout -d ${QUOBYTE_CLIENT_LOG_LEVEL} ${OPTS} \ 103 | ${ADDR}/ ${QUOBYTE_MOUNT_POINT} 104 | 105 | securityContext: 106 | privileged: true 107 | volumeMounts: 108 | - name: k8s-plugin-dir 109 | mountPath: /var/lib/kubelet/plugins/ 110 | - name: etcfs 111 | mountPath: /etcfs 112 | lifecycle: 113 | preStop: 114 | exec: 115 | command: 116 | - /bin/bash 117 | - -xc 118 | - | 119 | if ! /bin/nsenter -t 1 -m -- /bin/umount -f ${QUOBYTE_MOUNT_POINT}; then 120 | /bin/nsenter -t 1 -m -- /bin/umount -l ${QUOBYTE_MOUNT_POINT} 121 | fi 122 | hostPID: true 123 | nodeSelector: 124 | quobyte_client: "true" 125 | volumes: 126 | - name: k8s-plugin-dir 127 | hostPath: 128 | path: /var/lib/kubelet/plugins/ 129 | - name: etcfs 130 | hostPath: 131 | path: /etc 132 | -------------------------------------------------------------------------------- /deploy/metadata-ds.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: DaemonSet 3 | metadata: 4 | name: metadata 5 | namespace: quobyte 6 | spec: 7 | template: 8 | metadata: 9 | annotations: 10 | prometheus.io/scrape: 'true' 11 | prometheus.io/path: '/prometheus' 12 | prometheus.io/port: '7872' 13 | labels: 14 | role: metadata 15 | version: "2" 16 | spec: 17 | containers: 18 | - name: quobyte-metadata 19 | image: quay.io/quobyte/quobyte-server:2 20 | # imagePullPolicy: Always 21 | # resources: 22 | # limits: 23 | # memory: "1212Mi" 24 | # cpu: "200m" 25 | 26 | # priviliged is required for full device inspector support 27 | securityContext: 28 | privileged: true 29 | env: 30 | - name: QUOBYTE_SERVICE 31 | value: metadata 32 | - name: QUOBYTE_REGISTRY 33 | value: registry 34 | - name: MAX_MEM 35 | valueFrom: 36 | configMapKeyRef: 37 | name: quobyte-config 38 | key: metadata.max_mem 39 | - name: MIN_MEM 40 | valueFrom: 41 | configMapKeyRef: 42 | name: quobyte-config 43 | key: metadata.min_mem 44 | command: 45 | - /bin/bash 46 | - -xec 47 | - | 48 | sed "s/.*MIN_MEM_METADATA=.*/MIN_MEM_METADATA=${MIN_MEM}/" -i /etc/default/quobyte 49 | sed "s/.*MAX_MEM_METADATA=.*/MAX_MEM_METADATA=${MAX_MEM}/" -i /etc/default/quobyte 50 | exec /bin/bash -x /opt/main.sh 51 | volumeMounts: 52 | - mountPath: /run/udev 53 | name: host-udev 54 | - mountPath: /lib/modules 55 | name: lib-modules 56 | - mountPath: /mnt/quobyte 57 | name: mnt-quobyte 58 | - mountPath: /var/lib/quobyte 59 | name: var-lib-quobyte 60 | mountPropagation: Bidirectional 61 | ports: 62 | - name: rpc-tcp 63 | containerPort: 7872 64 | protocol: TCP 65 | readinessProbe: 66 | timeoutSeconds: 5 67 | httpGet: 68 | port: 7872 69 | path: / 70 | livenessProbe: 71 | initialDelaySeconds: 30 72 | timeoutSeconds: 5 73 | httpGet: 74 | port: 7872 75 | path: / 76 | nodeSelector: 77 | quobyte_metadata: "true" 78 | volumes: 79 | - name: host-udev 80 | hostPath: 81 | path: /run/udev 82 | - name: lib-modules 83 | hostPath: 84 | path: /lib/modules 85 | - name: mnt-quobyte 86 | hostPath: 87 | path: /mnt/quobyte 88 | - name: var-lib-quobyte 89 | hostPath: 90 | path: /var/lib/quobyte 91 | -------------------------------------------------------------------------------- /deploy/qmgmt-pod.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Pod 3 | metadata: 4 | labels: 5 | role: qmgmt-pod 6 | version: "2" 7 | name: qmgmt-pod 8 | namespace: quobyte 9 | spec: 10 | containers: 11 | - name: qmgmt-pod 12 | image: quay.io/quobyte/quobyte-server:2 13 | command: 14 | - /bin/bash 15 | - -xec 16 | - | 17 | tail -f /dev/null 18 | -------------------------------------------------------------------------------- /deploy/quobyte-ns.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Namespace 3 | metadata: 4 | name: quobyte 5 | -------------------------------------------------------------------------------- /deploy/quobyte-services.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: registry 5 | namespace: quobyte 6 | annotations: 7 | prometheus.io/scrape: 'true' 8 | prometheus.io/path: '/prometheus' 9 | prometheus.io/port: '7871' 10 | spec: 11 | clusterIP: "None" # headless service => kube-dns will return pod IPs 12 | ports: 13 | - name: quobyte # available as _quobyte._tcp.registry via DNS 14 | port: 7861 15 | protocol: TCP 16 | - name: rpc-udp 17 | port: 7861 18 | protocol: UDP 19 | - name: http 20 | port: 7871 21 | protocol: TCP 22 | selector: 23 | role: registry 24 | --- 25 | apiVersion: v1 26 | kind: Service 27 | metadata: 28 | name: api 29 | namespace: quobyte 30 | spec: 31 | ports: 32 | - name: api80 33 | targetPort: 7860 34 | port: 80 35 | protocol: TCP 36 | - name: api 37 | port: 7860 38 | protocol: TCP 39 | selector: 40 | role: webconsole 41 | --- 42 | apiVersion: v1 43 | kind: Service 44 | metadata: 45 | name: webconsole 46 | namespace: quobyte 47 | spec: 48 | ports: 49 | - name: web80 50 | targetPort: 8080 51 | port: 80 52 | protocol: TCP 53 | - name: web 54 | port: 8080 55 | protocol: TCP 56 | selector: 57 | role: webconsole 58 | -------------------------------------------------------------------------------- /deploy/registry-ds.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: DaemonSet 3 | metadata: 4 | name: registry 5 | namespace: quobyte 6 | spec: 7 | template: 8 | metadata: 9 | labels: 10 | role: registry 11 | version: "2" 12 | spec: 13 | containers: 14 | - name: quobyte-registry 15 | image: quay.io/quobyte/quobyte-server:2 16 | # imagePullPolicy: Always 17 | # resources: 18 | # limits: 19 | # memory: "512Mi" 20 | # cpu: "100m" 21 | 22 | # priviliged is required for full device inspector support 23 | securityContext: 24 | privileged: true 25 | env: 26 | - name: QUOBYTE_SERVICE 27 | value: registry 28 | - name: QUOBYTE_REGISTRY 29 | value: registry 30 | - name: QUOBYTE_EXTRA_SERVICE_CONFIG 31 | value: > 32 | constants.automation.manage_registry_replicas=true 33 | - name: MAX_MEM 34 | valueFrom: 35 | configMapKeyRef: 36 | name: quobyte-config 37 | key: registry.max_mem 38 | - name: MIN_MEM 39 | valueFrom: 40 | configMapKeyRef: 41 | name: quobyte-config 42 | key: registry.min_mem 43 | - name: CURRENT_NODE 44 | valueFrom: 45 | fieldRef: 46 | fieldPath: spec.nodeName 47 | - name: QUOBYTE_BOOTSTRAP_NODE 48 | valueFrom: 49 | configMapKeyRef: 50 | name: quobyte-config 51 | key: registry.bootstrap_node 52 | command: 53 | - /bin/bash 54 | - -xec 55 | - | 56 | if [ "${QUOBYTE_BOOTSTRAP_NODE}" = "${CURRENT_NODE}" ]; then 57 | sed "s/.*MIN_MEM_REGISTRY=.*/MIN_MEM_REGISTRY=${MIN_MEM}/" -i /etc/default/quobyte 58 | sed "s/.*MAX_MEM_REGISTRY=.*/MAX_MEM_REGISTRY=${MAX_MEM}/" -i /etc/default/quobyte 59 | mkdir -p /var/lib/quobyte/devices/registry-bootstrap 60 | if [ ! -f /var/lib/quobyte/devices/registry-bootstrap/QUOBYTE_DEV_SETUP ]; then 61 | mkdir -p /devices/dev1 62 | cat > /var/lib/quobyte/devices/registry-bootstrap/QUOBYTE_DEV_SETUP <> /var/lib/quobyte/devices/registry-bootstrap/QUOBYTE_DEV_SETUP 69 | fi 70 | fi 71 | 72 | if [ ! -f /var/lib/quobyte/devices/registry-bootstrap/UUID ]; then 73 | echo uuid=$(uuidgen) >> /var/lib/quobyte/devices/registry-bootstrap/UUID 74 | fi 75 | cat /var/lib/quobyte/devices/registry-bootstrap/UUID >> /etc/quobyte/$QUOBYTE_SERVICE.cfg 76 | fi 77 | exec /bin/bash -x /opt/main.sh 78 | lifecycle: 79 | preStop: 80 | exec: 81 | command: 82 | - /bin/bash 83 | - -xe 84 | - | 85 | if [ "${QUOBYTE_BOOTSTRAP_NODE}" = "${CURRENT_NODE}" ]; then 86 | qmgmt -u api registry remove $(grep device.id= /var/lib/quobyte/devices/registry-bootstrap/QUOBYTE_DEV_ID | cut -d= -f2) 87 | rm -rf /var/lib/quobyte/devices/registry-bootstrap/ 88 | fi 89 | ports: 90 | - name: rpc-tcp 91 | containerPort: 7861 92 | protocol: TCP 93 | - name: rpc-udp 94 | containerPort: 7861 95 | protocol: UDP 96 | - name: http 97 | containerPort: 7871 98 | protocol: TCP 99 | volumeMounts: 100 | - mountPath: /run/udev 101 | name: host-udev 102 | - mountPath: /lib/modules 103 | name: libmodules 104 | - mountPath: /var/lib/quobyte 105 | name: var-lib-quobyte 106 | mountPropagation: Bidirectional 107 | - mountPath: /mnt/quobyte 108 | name: mnt-quobyte 109 | readinessProbe: 110 | timeoutSeconds: 5 111 | httpGet: 112 | port: 7871 113 | path: / 114 | livenessProbe: 115 | initialDelaySeconds: 30 116 | timeoutSeconds: 5 117 | httpGet: 118 | port: 7871 119 | path: / 120 | nodeSelector: 121 | quobyte_registry: "true" 122 | volumes: 123 | - name: host-udev 124 | hostPath: 125 | path: /run/udev 126 | - name: libmodules 127 | hostPath: 128 | path: /lib/modules 129 | - name: mnt-quobyte 130 | hostPath: 131 | path: /mnt/quobyte 132 | - name: var-lib-quobyte 133 | hostPath: 134 | path: /var/lib/quobyte 135 | -------------------------------------------------------------------------------- /deploy/webconsole-deployment.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: Deployment 3 | metadata: 4 | name: webconsole 5 | namespace: quobyte 6 | spec: 7 | template: 8 | metadata: 9 | labels: 10 | role: webconsole 11 | version: "2" 12 | spec: 13 | containers: 14 | - name: quobyte-webconsole 15 | image: quay.io/quobyte/quobyte-server:2 16 | # imagePullPolicy: Always 17 | # resources: 18 | # limits: 19 | # memory: "300Mi" 20 | # cpu: "100m" 21 | env: 22 | - name: QUOBYTE_SERVICE 23 | value: "webconsole" 24 | - name: QUOBYTE_WEBCONSOLE_PORT 25 | value: "8080" 26 | - name: QUOBYTE_REGISTRY 27 | value: registry 28 | - name: QUOBYTE_API 29 | value: api 30 | - name: MAX_MEM 31 | valueFrom: 32 | configMapKeyRef: 33 | name: quobyte-config 34 | key: webconsole.max_mem 35 | - name: MIN_MEM 36 | valueFrom: 37 | configMapKeyRef: 38 | name: quobyte-config 39 | key: webconsole.min_mem 40 | command: 41 | - /bin/bash 42 | - -xec 43 | - | 44 | sed "s/MIN_MEM_WEBCONSOLE=.*/MIN_MEM_WEBCONSOLE=${MIN_MEM}/" -i /etc/default/quobyte 45 | sed "s/MAX_MEM_WEBCONSOLE=.*/MAX_MEM_WEBCONSOLE=${MAX_MEM}/" -i /etc/default/quobyte 46 | exec /bin/bash -x /opt/main.sh 47 | ports: 48 | - name: http 49 | containerPort: 8080 50 | protocol: TCP 51 | - name: quobyte-api 52 | image: quay.io/quobyte/quobyte-server:2 53 | env: 54 | - name: QUOBYTE_SERVICE 55 | value: api 56 | - name: QUOBYTE_API_PORT 57 | value: "7860" 58 | - name: QUOBYTE_REGISTRY 59 | value: registry 60 | - name: QUOBYTE_LOG_LEVEL 61 | value: DEBUG 62 | - name: MAX_MEM 63 | valueFrom: 64 | configMapKeyRef: 65 | name: quobyte-config 66 | key: api.max_mem 67 | - name: MIN_MEM 68 | valueFrom: 69 | configMapKeyRef: 70 | name: quobyte-config 71 | key: api.min_mem 72 | ports: 73 | - name: api 74 | containerPort: 7860 75 | protocol: TCP 76 | command: 77 | - /bin/bash 78 | - -xec 79 | - | 80 | sed "s/.*MIN_MEM_API=.*/MIN_MEM_API=${MIN_MEM}/" -i /etc/default/quobyte 81 | sed "s/.*MAX_MEM_API=.*/MAX_MEM_API=${MAX_MEM}/" -i /etc/default/quobyte 82 | exec /bin/bash -x /opt/main.sh 83 | resources: 84 | limits: 85 | memory: "300Mi" 86 | cpu: "100m" 87 | readinessProbe: 88 | timeoutSeconds: 5 89 | tcpSocket: 90 | port: 7860 91 | livenessProbe: 92 | initialDelaySeconds: 30 93 | timeoutSeconds: 10 94 | tcpSocket: 95 | port: 7860 96 | -------------------------------------------------------------------------------- /operator/README.md: -------------------------------------------------------------------------------- 1 | # Bootstrap and Operate a Quobyte Cluster in Kubernetes with the Operator 2 | This guide shows you how to bootstrap a Quobyte cluster on a set of kubernetes nodes 3 | which have some empty storage devices ready to be used for a distributed storage system. The operator simplifies the cluster bootstrap and management, and the latest Quobyte features let you format and set up the cluster from the web console. 4 | 5 | For a quick overview, we have prepared a high level blog post: [“Quobernetes” or How to Enable Simple and Efficient Storage Operations for Kubernetes](https://www.quobyte.com/blog/2018/08/27/quobernetes-is-kubernetes-with-quobyte-software-storage/) 6 | 7 | ## Prerequisites 8 | - Kubernetes 1.9 is fully supported by the operator 9 | - To use Quobyte 2.0 features like automatically mounting Quobyte devices, 10 | or the formatting and preparation of unformatted devices requires the *mountPropagation* kubernetes feature, which is a gated feature in kubernetes 1.9 11 | and comes in beta in kubernetes 1.10. This guide assumes that you have the *mountPropagation* feature enabled on your cluster. 12 | - A cluster which consists of at least 4 nodes, with 2 unformatted devices each. 13 | In this guide we will refer to the nodes as node1 to node4. 14 | 15 | ## Deploy Operator 16 | Create the *quobyte* namespace. This is where the operator and the cluster lives. 17 | ``` bash 18 | kubectl create -f quobyte-ns.yaml 19 | ``` 20 | 21 | Once the namespace is created, deploy required RBAC and the operator. 22 | ``` bash 23 | kubectl -n quobyte create -f operator.yaml 24 | ``` 25 | 26 | ## Configure Quobyte Services with the Operator 27 | Quobyte runs best with 3 replicas of the registry, where we require 1 bootstrapped registry. To make the setup as easy as possible, we defined an ephemeral registry, which is used to bootstrap the cluster. The final cluster will have registry devices on nodes 2, 3, and 4, so we will use node1 for bootstrap. 28 | 29 | The `quobyte-config.yaml` file provides a `registry.bootstrap_node` option and allows to fine tune the memory limits for the services. Edit the file to point to your bootstrap registry. 30 | 31 | ``` yaml 32 | registry.bootstrap_node: "node1" 33 | ``` 34 | 35 | Now create the configurations and services daemonsets. 36 | ``` bash 37 | kubectl -n quobyte create -f quobyte-config.yaml 38 | kubectl -n quobyte create -f services.yaml 39 | ``` 40 | 41 | To run the Quobyte services in kubernetes, first edit the services-config.yaml file, replace the node1-node4 entries with a list of your nodes 42 | and determine which node should run which services. The number of nodes is arbitrary but we recommend using 4+ nodes in order to be able handle 43 | outage scenarios, etc. 44 | 45 | We chose node1 to be the bootstrap registry, but we need to define 3 other nodes to persist the fully replicated cluster. We also recommend to start at least 3 metadata services and data services on all nodes which contain devices which should store your valuable information. Edit the `services-config.yaml` to match your cluster: 46 | 47 | ```yaml 48 | apiVersion: quobyte.com/v1 49 | kind: QuobyteService 50 | metadata: 51 | name: quobyte-services 52 | spec: 53 | registry: 54 | daemonSetName: registry 55 | nodes: 56 | - node1 # will become the ephemeral bootstrap device 57 | metadata: 58 | daemonSetName: metadata 59 | nodes: 60 | - node1 61 | - node2 62 | - node3 63 | - node4 64 | data: 65 | daemonSetName: data 66 | nodes: 67 | - node1 68 | - node2 69 | - node3 70 | - node4 71 | ``` 72 | 73 | When the services-config created,the operator will start to deploy the services to the target nodes. 74 | ``` 75 | kubectl -n quobyte create -f services-config.yaml 76 | kubectl -n quobyte create -f qmgmt-pod.yaml 77 | 78 | kubectl -n quobyte get pods -o wide -w 79 | ``` 80 | Now you should see all configured pods running. 81 | You will also see a *qmgmt-pod* and a *webconsole* running. 82 | The qmgmt-pod gives you full cli access to the cluster. Lets check that the 83 | primary registry is running: 84 | 85 | ```bash 86 | kubectl -n quobyte exec -it qmgmt-pod -- qmgmt -u api registry list 87 | ``` 88 | 89 | Now let's list all unformatted devices which the data and metadata services found 90 | and could be formatted now. 91 | 92 | ```bash 93 | kubectl -n quobyte exec -it qmgmt-pod -- qmgmt -u api device list-unformatted 94 | ``` 95 | 96 | You can either proceed to set up the devices with qmgmt or jump over to the webconsole to get some visual support. 97 | 98 | Unless you already have set up an ingress to access the service, you can acceess the console with a port forward 99 | ``` 100 | kubectl -n quobyte port-forward "$(kubectl get po -owide -n quobyte | grep webconsole | awk '{print $1}')" 8080:8080 101 | ``` 102 | Then point your browser to http://localhost:8080 and follow the setup wizard. 103 | 104 | The [Devices tab](http://localhost:8080/#DeviceListState:) will show you all unformatted devices. Please note that even if multiple services are running on the same node, only one of the Quobyte pods will be responsible to mount and format devices. 105 | 106 | ## Make the Quobyte Cluster Persistent 107 | The ephemereal registry is great for bootstrapping and trying out a Quobyte cluster. 108 | So if you're just interested in a demo, you can safely skip this chapter and just 109 | use a single registry to run your cluster. But please note, once the registry pod is terminated, 110 | the Quobyte cluster becomes unsuable. 111 | 112 | To make the cluster persistent, we first need to create 3 registry devices. 113 | There must be only one registry device per registry service, so choose one device from each of the other services and create registry devices on them. 114 | A maintenance task will run and format and set up the devices. Give the webconsole some seconds to retrieve the last system state and the devices will show up as 115 | unassociated devices. 116 | 117 | Now let's spin up our three target registries. 118 | Edit the services-config.yaml again and add nodes 2 to 4 as registries. 119 | 120 | ```yaml 121 | Edit the `services-config.yaml` to match your cluster 122 | 123 | ```yaml 124 | registry: 125 | daemonSetName: registry 126 | nodes: 127 | - node1 # will become the ephemeral bootstrap node 128 | - node2 129 | - node3 130 | - node4 131 | ``` 132 | 133 | An update to the services-config CRD triggers the operator, which will start the 134 | registries then. 135 | 136 | ```bash 137 | kubectl -n quobyte apply -f services-config.yaml 138 | ``` 139 | 140 | Wait until the pods are running and check that Quobyte found the devices. 141 | 142 | ```bash 143 | kubectl -n quobyte exec -it qmgmt-pod -- qmgmt -u api device list 144 | 145 | Id Host Mode Disk Used Disk Avail Services LED Mode Tags 146 | 1 registry-vfz59 ONLINE 4 GB 40 GB REGISTRY OFF hdd 147 | 2 registry-f7szc ONLINE 34 MB 21 GB REGISTRY OFF hdd 148 | 3 registry-zq2vh ONLINE 34 MB 21 GB REGISTRY OFF hdd 149 | 4 registry-4xxkk ONLINE 34 MB 21 GB REGISTRY OFF hdd 150 | ``` 151 | 152 | We see a total of 4 registry devices, but the registry will only use 3 of them. 153 | ```bash 154 | kubectl -n quobyte exec -it qmgmt-pod -- qmgmt -u api registry list 155 | 156 | Primary Id Host Mode 157 | - 3 registry-zq2vh ONLINE 158 | - 4 registry-4xxkk ONLINE 159 | 1 1 registry-vfz59 ONLINE 160 | ``` 161 | 162 | If we see 3 ONLINE registries, the work of the ephemeral bootstrap node is done 163 | and it is safe to delete it. So remove it from the `services-config.yaml` 164 | ```yaml 165 | registry: 166 | daemonSetName: registry 167 | nodes: 168 | - node2 169 | - node3 170 | - node4 171 | ``` 172 | and update the service-config. The operator will then terminate the ephemeral registry. 173 | ```bash 174 | kubectl -n quobyte apply -f services-config.yaml 175 | ``` 176 | Wait some seconds and check that all 3 persisted registries are ONLINE. 177 | ```bash 178 | kubectl -n quobyte exec -it qmgmt-pod -- qmgmt -u api registry list 179 | P 180 | rimary Id Host Mode 181 | - 1 registry-vfz59 ONLINE 182 | - 3 registry-zq2vh ONLINE 183 | 1 4 registry-4xxkk ONLINE 184 | ``` 185 | 186 | As a last step, you should decommission the ephemeral device, since it will never come back. 187 | ```bash 188 | kubectl -n quobyte exec -it qmgmt-pod -- qmgmt -u api device update status 1 DECOMMISSIONED 189 | ``` 190 | 191 | ## Create Data and Metadata Devices 192 | Now we need some data and metadata devices to actually store data. 193 | From the webconsole, either format the remaining devices according to your needs, 194 | or choose a device and *Set devices types* to add Data or Metadata contents to the device. 195 | 196 | Now you have a fully working Quobyte cluster. For further configuration and creation of volumes, please refer to the Quobyte documentation. 197 | 198 | # Deploy Quobyte Clients 199 | The operator can deploy and manage Quobyte clients - which serve the volumes to your application pods. Every kubernetes node which should provide access to Quobyte storage, has to run a Quobyte client pod. 200 | 201 | If the operator finds a client CRD, it will start to deploy the according pods. 202 | First edit the `client-config.yaml` 203 | 204 | ```yaml 205 | spec: 206 | rollingUpdatesEnabled: true 207 | daemonSetName: client 208 | nodes: 209 | - node1 210 | - node2 211 | - node3 212 | - node4 213 | ``` 214 | Nodes are optional. If not given, operator queries nodes with the `nodeSelector` in the `daemonSetName: client` and does only updates. 215 | 216 | ```bash 217 | kubectl -n quobyte create -f client.yaml 218 | kubectl -n quobyte create -f client-config.yaml 219 | ``` 220 | 221 | Once the client-config is created, you should see pods being started on the 222 | desired hosts. 223 | 224 | If you add or remove clients, edit the client-config.yaml and update it with 225 | 226 | ``` 227 | kubectl -n quobyte apply -f client-config.yaml 228 | ``` 229 | 230 | When the clients are ready, you can start using Quobyte volumes in your pods. 231 | Please have a look at [Volume Access](../using_quobyte_volumes.md) for examples. 232 | 233 | # Rolling Updates 234 | The operator supports rolling updates only for **clients**. To trigger rolling update of client, please change the container image of daemonset configured in the client-config.yaml. Set `rollingUpdatesEnabled: true` , the operator will upgrade one node after the other. 235 | 236 | Quobyte service containers are updated with careful timing between pod restarts, to always ensure availability of the Quobyte services. 237 | 238 | All pods from all other namespaces can access Quobyte volumes which are managed by the client. Since a client update requires a 239 | pod restart, all other pods on the same node, which currently access a Quobyte volume, need to be stopped. 240 | It's not a good idea to give an operator full permission to drain a full node, we decided to go for a defensive mode. 241 | For every node to upgrade, the operator checks for other pods with Quobyte volumes mounted. If no pods are found, the client is restarted immediately. 242 | If pods are found, they are listed on the operator's status page. The operator also supports to retrieve its status as json. 243 | The administrator will then need to manually stop or drain the pods. 244 | 245 | The operator comes with a service and a status page for clients. With kubectl, you can reach it on http://localhost:7878 246 | ```bash 247 | kubectl -n quobyte port-forward quobyte-operator-xzy 7878:7878 248 | ``` 249 | **Services rolling updates** should follow standard daemonset updates. One way to trigger rolling update for services is to set the new image for daemonset container as shown below 250 | ``` 251 | kubectl set image ds/ = 252 | ``` 253 | # Uninstall Quobyte with Operator 254 | If you want to remove all services or clients, remove the config files, before 255 | you delete the deployments or the operator. This will terminate the scheduled pods and remove the all labels, which the operator applied to any nodes. 256 | ``` 257 | kubectl -n quobyte delete -f services-config.yaml 258 | or 259 | kubectl -n quobyte delete -f client-config.yaml 260 | ``` 261 | 262 | # Build Operator from Source 263 | 264 | ## Requirements 265 | 1. golang 1.10+ 266 | 2. glide for package management 267 | 3. docker 268 | 269 | ## Build 270 | 1. Clone the repository. 271 | ``` 272 | git clone git@github.com:quobyte/k8s-operator.git github.com/quobyte/k8s-operator 273 | ``` 274 | 2. Compile and build binary from source. 275 | ``` 276 | cd github.com/quobyte/k8s-operator 277 | export GOPATH=$(pwd) 278 | ./build #build the operator binary 279 | ``` 280 | If you're building for the first time after clone run ``glide install --strip-vendor`` to get the dependencies. 281 | 282 | 3. To run operator outside cluster (skip to 4 to run operator inside cluster) 283 | ``` 284 | ./operator --kubeconfig 285 | ``` 286 | Follow [Deploy clients](#deploy-clients), and you can skip step 3 of deploy clients. 287 | 288 | 4. Build the container and push it to repository 289 | `` 290 | ./build # push the built image to the container repository-url 291 | `` 292 | 5. Edit ``operator.yaml`` and point ``quobyte-operator`` container image to the docker image. 293 | -------------------------------------------------------------------------------- /operator/deploy/client-config.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: quobyte.com/v1 2 | kind: QuobyteClient 3 | metadata: 4 | name: quobyte-client 5 | namespace: quobyte 6 | spec: 7 | rollingUpdatesEnabled: true 8 | daemonSetName: client 9 | nodes: 10 | - node1 11 | - node2 12 | - node3 13 | - node4 14 | -------------------------------------------------------------------------------- /operator/deploy/client.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: DaemonSet 3 | metadata: 4 | name: client 5 | namespace: quobyte 6 | spec: 7 | # operator requires 'OnDelete' strategy 8 | updateStrategy: 9 | type: 'OnDelete' 10 | selector: 11 | matchLabels: 12 | # operator currently supports only single label 13 | role: client 14 | template: 15 | metadata: 16 | annotations: 17 | prometheus.io/scrape: 'true' 18 | prometheus.io/path: '/prometheus' 19 | prometheus.io/port: '55000' 20 | # operator requires this flag. 21 | quobyteClient: 'true' 22 | labels: 23 | role: client 24 | version: "2" 25 | spec: 26 | # operator uses tain while upgrading the client image 27 | tolerations: 28 | - key: "quobyte-upgrade" 29 | operator: "Equal" 30 | value: "true" 31 | effect: "NoSchedule" 32 | containers: 33 | - name: quobyte-client 34 | image: quay.io/quobyte/quobyte-client:2 35 | imagePullPolicy: Always 36 | env: 37 | - name: QUOBYTE_CLIENT_LOG_LEVEL 38 | value: INFO 39 | - name: QUOBYTE_REGISTRY 40 | value: registry.quobyte 41 | - name: QUOBYTE_MOUNT_POINT 42 | # Note that the mount point has to be a subdir of the volume(Mount) 43 | value: /mnt/kubernetes.io~quobyte 44 | - name: NODENAME 45 | valueFrom: 46 | fieldRef: 47 | fieldPath: spec.nodeName 48 | ports: 49 | - name: http-port 50 | containerPort: 55000 51 | hostPort: 55000 52 | protocol: TCP 53 | readinessProbe: 54 | timeoutSeconds: 5 55 | httpGet: 56 | port: 55000 57 | path: / 58 | livenessProbe: 59 | initialDelaySeconds: 30 60 | timeoutSeconds: 5 61 | httpGet: 62 | port: 55000 63 | path: / 64 | command: 65 | - /bin/bash 66 | - -xec 67 | - | 68 | if cut -d" " -f2 /proc/self/mounts | grep -q ${QUOBYTE_MOUNT_POINT}; then 69 | umount -l ${QUOBYTE_MOUNT_POINT} 70 | fi 71 | 72 | mkdir -p /root/.quobyte ${QUOBYTE_MOUNT_POINT} 73 | 74 | # set the mount point immutable. As long as mount.quobyte does not run, 75 | # other processes cannot write data to this dir. 76 | chattr +i ${QUOBYTE_MOUNT_POINT} || \ 77 | echo "WARNING: The local filesystem does not support IMMUTABLE flag" 78 | 79 | /usr/bin/mount.quobyte --hostname ${NODENAME} \ 80 | --allow-usermapping-in-volumename --http-port 55000 -f \ 81 | -d ${QUOBYTE_CLIENT_LOG_LEVEL} -l /dev/stdout ${OPTS} \ 82 | ${QUOBYTE_REGISTRY}/ ${QUOBYTE_MOUNT_POINT} 83 | securityContext: 84 | privileged: true 85 | volumeMounts: 86 | - name: k8s-plugin-dir 87 | mountPath: /mnt 88 | mountPropagation: Bidirectional 89 | lifecycle: 90 | preStop: 91 | exec: 92 | command: ["/bin/bash", "-xc", "umount -l ${QUOBYTE_MOUNT_POINT}"] 93 | nodeSelector: 94 | # operator currently supports only single selector 95 | quobyte_client: "true" 96 | volumes: 97 | - name: k8s-plugin-dir 98 | hostPath: 99 | path: /var/lib/kubelet/plugins/ 100 | -------------------------------------------------------------------------------- /operator/deploy/operator.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: rbac.authorization.k8s.io/v1beta1 2 | kind: ClusterRoleBinding 3 | metadata: 4 | name: quobyte-operator 5 | roleRef: 6 | apiGroup: rbac.authorization.k8s.io 7 | kind: ClusterRole 8 | name: quobyte-operator 9 | subjects: 10 | - kind: ServiceAccount 11 | name: quobyte-operator 12 | namespace: quobyte 13 | --- 14 | apiVersion: rbac.authorization.k8s.io/v1beta1 15 | kind: RoleBinding 16 | metadata: 17 | name: quobyte-operator 18 | namespace: quobyte 19 | roleRef: 20 | apiGroup: rbac.authorization.k8s.io 21 | kind: Role 22 | name: quobyte-operator 23 | subjects: 24 | - kind: ServiceAccount 25 | name: quobyte-operator 26 | namespace: quobyte 27 | --- 28 | apiVersion: rbac.authorization.k8s.io/v1beta1 29 | kind: ClusterRole 30 | metadata: 31 | name: quobyte-operator 32 | rules: 33 | - apiGroups: 34 | - apiextensions.k8s.io 35 | resources: 36 | - customresourcedefinitions 37 | verbs: 38 | - "*" 39 | - apiGroups: 40 | - quobyte.com 41 | resources: 42 | - quobyteclients 43 | - quobyteservices 44 | verbs: 45 | - "*" 46 | - apiGroups: [""] 47 | resources: 48 | - persistentvolumeclaims 49 | verbs: ["get"] 50 | - apiGroups: [""] 51 | resources: 52 | - pods 53 | verbs: ["list"] 54 | - apiGroups: ["storage.k8s.io"] 55 | resources: 56 | - storageclasses 57 | verbs: ["get"] 58 | - apiGroups: [""] 59 | resources: 60 | - nodes 61 | verbs: ["list","get","patch"] 62 | - apiGroups: [""] 63 | resources: 64 | - namespaces 65 | verbs: ["get"] 66 | - apiGroups: ["apps"] 67 | resources: 68 | - daemonsets 69 | verbs: ["get","list","watch"] 70 | --- 71 | apiVersion: rbac.authorization.k8s.io/v1beta1 72 | kind: Role 73 | metadata: 74 | namespace: quobyte 75 | name: quobyte-operator 76 | rules: 77 | - apiGroups: [""] 78 | resources: 79 | - pods 80 | verbs: ["list", "get", "delete", "create", "watch"] 81 | - apiGroups: ["apps"] 82 | resources: 83 | - daemonsets 84 | verbs: ["get","list","watch"] 85 | --- 86 | apiVersion: v1 87 | kind: ServiceAccount 88 | metadata: 89 | name: quobyte-operator 90 | namespace: quobyte 91 | --- 92 | apiVersion: apps/v1beta1 93 | kind: Deployment 94 | metadata: 95 | name: quobyte-operator 96 | namespace: quobyte 97 | spec: 98 | replicas: 1 99 | template: 100 | metadata: 101 | labels: 102 | name: quobyte-operator 103 | role: quobyte-operator 104 | spec: 105 | serviceAccountName: quobyte-operator 106 | containers: 107 | - name: operator 108 | image: quay.io/quobyte/k8s-operator:2 109 | imagePullPolicy: Always 110 | ports: 111 | - containerPort: 7878 112 | protocol: TCP 113 | --- 114 | apiVersion: v1 115 | kind: Service 116 | metadata: 117 | name: quobyte-operator 118 | namespace: quobyte 119 | spec: 120 | ports: 121 | - name: http 122 | port: 7878 123 | protocol: TCP 124 | selector: 125 | role: quobyte-operator 126 | -------------------------------------------------------------------------------- /operator/deploy/qmgmt-pod.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Pod 3 | metadata: 4 | labels: 5 | role: qmgmt-pod 6 | version: "2" 7 | name: qmgmt-pod 8 | namespace: quobyte 9 | spec: 10 | containers: 11 | - name: qmgmt-pod 12 | image: quay.io/quobyte/quobyte-server:2 13 | command: 14 | - /bin/bash 15 | - -xec 16 | - | 17 | tail -f /dev/null 18 | -------------------------------------------------------------------------------- /operator/deploy/quobyte-config.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | name: quobyte-config 5 | namespace: quobyte 6 | data: 7 | # The specified node will become the initial ephemeral bootstrap node 8 | registry.bootstrap_node: "node1" 9 | 10 | # Adjust these settings according to your workloads and recommendations from Quobyte documentation 11 | registry.min_mem: 64m 12 | registry.max_mem: 512m 13 | 14 | metadata.min_mem: 256m 15 | metadata.max_mem: 1024m 16 | 17 | data.min_mem: 256m 18 | data.max_mem: 1024m 19 | 20 | api.min_mem: 64m 21 | api.max_mem: 300m 22 | 23 | webconsole.min_mem: 64m 24 | webconsole.max_mem: 512m 25 | 26 | # DNS name of registry for the clients to connect to 27 | client.registry: registry 28 | client.log_level: INFO 29 | -------------------------------------------------------------------------------- /operator/deploy/quobyte-ns.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Namespace 3 | metadata: 4 | name: quobyte 5 | -------------------------------------------------------------------------------- /operator/deploy/services-config.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: quobyte.com/v1 2 | kind: QuobyteService 3 | metadata: 4 | name: quobyte-services 5 | namespace: quobyte 6 | spec: 7 | registry: 8 | daemonSetName: registry 9 | nodes: 10 | - node1 11 | metadata: 12 | daemonSetName: metadata 13 | nodes: 14 | - node1 15 | - node2 16 | - node3 17 | - node4 18 | data: 19 | daemonSetName: data 20 | nodes: 21 | - node1 22 | - node2 23 | - node3 24 | - node4 25 | -------------------------------------------------------------------------------- /operator/deploy/services.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: registry 5 | namespace: quobyte 6 | annotations: 7 | prometheus.io/scrape: 'true' 8 | prometheus.io/path: '/prometheus' 9 | prometheus.io/port: '7871' 10 | spec: 11 | clusterIP: "None" # headless service => kube-dns will return pod IPs 12 | ports: 13 | - name: quobyte # available as _quobyte._tcp.registry via DNS 14 | port: 7861 15 | protocol: TCP 16 | - name: rpc-udp 17 | port: 7861 18 | protocol: UDP 19 | - name: http 20 | port: 7871 21 | protocol: TCP 22 | selector: 23 | role: registry 24 | --- 25 | apiVersion: v1 26 | kind: Service 27 | metadata: 28 | name: api 29 | namespace: quobyte 30 | spec: 31 | ports: 32 | - name: api80 33 | targetPort: 7860 34 | port: 80 35 | protocol: TCP 36 | - name: api 37 | port: 7860 38 | protocol: TCP 39 | selector: 40 | role: webconsole 41 | --- 42 | apiVersion: v1 43 | kind: Service 44 | metadata: 45 | name: webconsole 46 | namespace: quobyte 47 | spec: 48 | ports: 49 | - name: web80 50 | targetPort: 8080 51 | port: 80 52 | protocol: TCP 53 | - name: web 54 | port: 8080 55 | protocol: TCP 56 | selector: 57 | role: webconsole 58 | --- 59 | apiVersion: apps/v1 60 | kind: DaemonSet 61 | metadata: 62 | name: registry 63 | namespace: quobyte 64 | spec: 65 | selector: 66 | matchLabels: 67 | role: registry 68 | template: 69 | metadata: 70 | labels: 71 | role: registry 72 | spec: 73 | containers: 74 | - name: quobyte-registry 75 | image: quay.io/quobyte/quobyte-server:2 76 | imagePullPolicy: Always 77 | # resources: 78 | # limits: 79 | # memory: "512Mi" 80 | # cpu: "100m" 81 | 82 | # priviliged is required for full device inspector support 83 | securityContext: 84 | privileged: true 85 | env: 86 | - name: QUOBYTE_SERVICE 87 | value: registry 88 | - name: QUOBYTE_REGISTRY 89 | value: registry 90 | - name: QUOBYTE_EXTRA_SERVICE_CONFIG 91 | value: > 92 | constants.automation.manage_registry_replicas=true 93 | - name: MAX_MEM 94 | valueFrom: 95 | configMapKeyRef: 96 | name: quobyte-config 97 | key: registry.max_mem 98 | - name: MIN_MEM 99 | valueFrom: 100 | configMapKeyRef: 101 | name: quobyte-config 102 | key: registry.min_mem 103 | - name: CURRENT_NODE 104 | valueFrom: 105 | fieldRef: 106 | fieldPath: spec.nodeName 107 | - name: QUOBYTE_BOOTSTRAP_NODE 108 | valueFrom: 109 | configMapKeyRef: 110 | name: quobyte-config 111 | key: registry.bootstrap_node 112 | command: 113 | - /bin/bash 114 | - -xec 115 | - | 116 | if [ "${QUOBYTE_BOOTSTRAP_NODE}" = "${CURRENT_NODE}" ]; then 117 | sed "s/.*MIN_MEM_REGISTRY=.*/MIN_MEM_REGISTRY=${MIN_MEM}/" -i /etc/default/quobyte 118 | sed "s/.*MAX_MEM_REGISTRY=.*/MAX_MEM_REGISTRY=${MAX_MEM}/" -i /etc/default/quobyte 119 | mkdir -p /var/lib/quobyte/devices/registry-bootstrap 120 | if [ ! -f /var/lib/quobyte/devices/registry-bootstrap/QUOBYTE_DEV_SETUP ]; then 121 | mkdir -p /devices/dev1 122 | cat > /var/lib/quobyte/devices/registry-bootstrap/QUOBYTE_DEV_SETUP <> /var/lib/quobyte/devices/registry-bootstrap/QUOBYTE_DEV_SETUP 129 | fi 130 | fi 131 | 132 | if [ ! -f /var/lib/quobyte/devices/registry-bootstrap/UUID ]; then 133 | echo uuid=$(uuidgen) >> /var/lib/quobyte/devices/registry-bootstrap/UUID 134 | fi 135 | cat /var/lib/quobyte/devices/registry-bootstrap/UUID >> /etc/quobyte/$QUOBYTE_SERVICE.cfg 136 | fi 137 | exec /bin/bash -x /opt/main.sh 138 | lifecycle: 139 | preStop: 140 | exec: 141 | command: 142 | - /bin/bash 143 | - -xe 144 | - | 145 | if [ "${QUOBYTE_BOOTSTRAP_NODE}" = "${CURRENT_NODE}" ]; then 146 | qmgmt -u api registry remove $(grep device.id= /var/lib/quobyte/devices/registry-bootstrap/QUOBYTE_DEV_ID | cut -d= -f2) 147 | rm -rf /var/lib/quobyte/devices/registry-bootstrap/ 148 | fi 149 | ports: 150 | - name: rpc-tcp 151 | containerPort: 7861 152 | protocol: TCP 153 | - name: rpc-udp 154 | containerPort: 7861 155 | protocol: UDP 156 | - name: http 157 | containerPort: 7871 158 | protocol: TCP 159 | volumeMounts: 160 | - mountPath: /run/udev 161 | name: host-udev 162 | - mountPath: /lib/modules 163 | name: libmodules 164 | - mountPath: /var/lib/quobyte 165 | name: var-lib-quobyte 166 | mountPropagation: Bidirectional 167 | - mountPath: /mnt/quobyte 168 | name: mnt-quobyte 169 | readinessProbe: 170 | timeoutSeconds: 5 171 | httpGet: 172 | port: 7871 173 | path: / 174 | livenessProbe: 175 | initialDelaySeconds: 30 176 | timeoutSeconds: 5 177 | httpGet: 178 | port: 7871 179 | path: / 180 | nodeSelector: 181 | quobyte_registry: "true" 182 | volumes: 183 | - name: host-udev 184 | hostPath: 185 | path: /run/udev 186 | - name: libmodules 187 | hostPath: 188 | path: /lib/modules 189 | - name: var-lib-quobyte 190 | hostPath: 191 | path: /var/lib/quobyte 192 | - name: mnt-quobyte 193 | hostPath: 194 | path: /mnt/quobyte 195 | --- 196 | apiVersion: apps/v1 197 | kind: DaemonSet 198 | metadata: 199 | name: data 200 | namespace: quobyte 201 | spec: 202 | selector: 203 | matchLabels: 204 | role: data 205 | template: 206 | metadata: 207 | annotations: 208 | prometheus.io/scrape: 'true' 209 | prometheus.io/path: '/prometheus' 210 | prometheus.io/port: '7873' 211 | labels: 212 | role: data 213 | spec: 214 | containers: 215 | - name: quobyte-data 216 | image: quay.io/quobyte/quobyte-server:2 217 | imagePullPolicy: Always 218 | # resources: 219 | # requests: 220 | # memory: "512Mi" 221 | # cpu: "200m" 222 | 223 | # priviliged is required for full device inspector support 224 | securityContext: 225 | privileged: true 226 | env: 227 | - name: QUOBYTE_SERVICE 228 | value: data 229 | - name: QUOBYTE_REGISTRY 230 | value: registry 231 | - name: MAX_MEM 232 | valueFrom: 233 | configMapKeyRef: 234 | name: quobyte-config 235 | key: data.max_mem 236 | - name: MIN_MEM 237 | valueFrom: 238 | configMapKeyRef: 239 | name: quobyte-config 240 | key: data.min_mem 241 | command: 242 | - /bin/bash 243 | - -xec 244 | - | 245 | sed "s/.*MIN_MEM_DATA=.*/MIN_MEM_DATA=${MIN_MEM}/" -i /etc/default/quobyte 246 | sed "s/.*MAX_MEM_DATA=.*/MAX_MEM_DATA=${MAX_MEM}/" -i /etc/default/quobyte 247 | exec /bin/bash -x /opt/main.sh 248 | lifecycle: 249 | preStop: 250 | exec: 251 | command: 252 | - /bin/bash 253 | - -xe 254 | - | 255 | if grep -q uuid /etc/quobyte/data.cfg; then 256 | UUID=`grep "uuid=" /etc/quobyte/data.cfg | cut -d= -f 2` 257 | echo "Deregistering service $UUID" 258 | qmgmt -u api service deregister $UUID 259 | fi 260 | volumeMounts: 261 | - mountPath: /run/udev 262 | name: host-udev 263 | - mountPath: /lib/modules 264 | name: lib-modules 265 | - mountPath: /var/lib/quobyte 266 | name: var-lib-quobyte 267 | mountPropagation: Bidirectional 268 | - mountPath: /mnt/quobyte 269 | name: mnt-quobyte 270 | ports: 271 | - name: rpc-tcp 272 | containerPort: 7873 273 | protocol: TCP 274 | readinessProbe: 275 | timeoutSeconds: 5 276 | httpGet: 277 | port: 7873 278 | path: / 279 | livenessProbe: 280 | initialDelaySeconds: 30 281 | timeoutSeconds: 5 282 | httpGet: 283 | port: 7873 284 | path: / 285 | nodeSelector: 286 | quobyte_data: "true" 287 | volumes: 288 | - name: host-udev 289 | hostPath: 290 | path: /run/udev 291 | - name: lib-modules 292 | hostPath: 293 | path: /lib/modules 294 | - name: var-lib-quobyte 295 | hostPath: 296 | path: /var/lib/quobyte 297 | - name: mnt-quobyte 298 | hostPath: 299 | path: /mnt/quobyte 300 | --- 301 | apiVersion: apps/v1 302 | kind: DaemonSet 303 | metadata: 304 | name: metadata 305 | namespace: quobyte 306 | spec: 307 | selector: 308 | matchLabels: 309 | role: metadata 310 | template: 311 | metadata: 312 | annotations: 313 | prometheus.io/scrape: 'true' 314 | prometheus.io/path: '/prometheus' 315 | prometheus.io/port: '7872' 316 | labels: 317 | role: metadata 318 | spec: 319 | containers: 320 | - name: quobyte-metadata 321 | image: quay.io/quobyte/quobyte-server:2 322 | imagePullPolicy: Always 323 | # resources: 324 | # limits: 325 | # memory: "1212Mi" 326 | # cpu: "200m" 327 | 328 | # priviliged is required for full device inspector support 329 | securityContext: 330 | privileged: true 331 | env: 332 | - name: QUOBYTE_SERVICE 333 | value: metadata 334 | - name: QUOBYTE_REGISTRY 335 | value: registry 336 | - name: MAX_MEM 337 | valueFrom: 338 | configMapKeyRef: 339 | name: quobyte-config 340 | key: metadata.max_mem 341 | - name: MIN_MEM 342 | valueFrom: 343 | configMapKeyRef: 344 | name: quobyte-config 345 | key: metadata.min_mem 346 | command: 347 | - /bin/bash 348 | - -xec 349 | - | 350 | sed "s/.*MIN_MEM_METADATA=.*/MIN_MEM_METADATA=${MIN_MEM}/" -i /etc/default/quobyte 351 | sed "s/.*MAX_MEM_METADATA=.*/MAX_MEM_METADATA=${MAX_MEM}/" -i /etc/default/quobyte 352 | exec /bin/bash -x /opt/main.sh 353 | lifecycle: 354 | preStop: 355 | exec: 356 | command: 357 | - /bin/bash 358 | - -xe 359 | - | 360 | if grep -q uuid /etc/quobyte/metadata.cfg; then 361 | UUID=`grep "uuid=" /etc/quobyte/data.cfg | cut -d= -f 2` 362 | echo "Deregistering service $UUID" 363 | qmgmt -u api service deregister $UUID 364 | fi 365 | volumeMounts: 366 | - mountPath: /run/udev 367 | name: host-udev 368 | - mountPath: /lib/modules 369 | name: lib-modules 370 | - mountPath: /var/lib/quobyte 371 | name: var-lib-quobyte 372 | mountPropagation: Bidirectional 373 | - mountPath: /mnt/quobyte 374 | name: mnt-quobyte 375 | ports: 376 | - name: rpc-tcp 377 | containerPort: 7872 378 | protocol: TCP 379 | readinessProbe: 380 | timeoutSeconds: 5 381 | httpGet: 382 | port: 7872 383 | path: / 384 | livenessProbe: 385 | initialDelaySeconds: 30 386 | timeoutSeconds: 5 387 | httpGet: 388 | port: 7872 389 | path: / 390 | nodeSelector: 391 | quobyte_metadata: "true" 392 | volumes: 393 | - name: host-udev 394 | hostPath: 395 | path: /run/udev 396 | - name: lib-modules 397 | hostPath: 398 | path: /lib/modules 399 | - name: var-lib-quobyte 400 | hostPath: 401 | path: /var/lib/quobyte 402 | - name: mnt-quobyte 403 | hostPath: 404 | path: /mnt/quobyte 405 | --- 406 | apiVersion: apps/v1 407 | kind: Deployment 408 | metadata: 409 | name: webconsole 410 | namespace: quobyte 411 | spec: 412 | selector: 413 | matchLabels: 414 | role: webconsole 415 | template: 416 | metadata: 417 | labels: 418 | role: webconsole 419 | version: "2" 420 | spec: 421 | containers: 422 | - name: quobyte-webconsole 423 | image: quay.io/quobyte/quobyte-server:2 424 | imagePullPolicy: Always 425 | # resources: 426 | # limits: 427 | # memory: "300Mi" 428 | # cpu: "100m" 429 | env: 430 | - name: QUOBYTE_SERVICE 431 | value: "webconsole" 432 | - name: QUOBYTE_WEBCONSOLE_PORT 433 | value: "8080" 434 | - name: QUOBYTE_REGISTRY 435 | value: registry 436 | - name: QUOBYTE_API 437 | value: api 438 | - name: MAX_MEM 439 | valueFrom: 440 | configMapKeyRef: 441 | name: quobyte-config 442 | key: webconsole.max_mem 443 | - name: MIN_MEM 444 | valueFrom: 445 | configMapKeyRef: 446 | name: quobyte-config 447 | key: webconsole.min_mem 448 | command: 449 | - /bin/bash 450 | - -xec 451 | - | 452 | sed "s/MIN_MEM_WEBCONSOLE=.*/MIN_MEM_WEBCONSOLE=${MIN_MEM}/" -i /etc/default/quobyte 453 | sed "s/MAX_MEM_WEBCONSOLE=.*/MAX_MEM_WEBCONSOLE=${MAX_MEM}/" -i /etc/default/quobyte 454 | exec /bin/bash -x /opt/main.sh 455 | lifecycle: 456 | preStop: 457 | exec: 458 | command: 459 | - /bin/bash 460 | - -xe 461 | - | 462 | if grep -q uuid /etc/quobyte/webconsole.cfg; then 463 | UUID=`grep "uuid=" /etc/quobyte/data.cfg | cut -d= -f 2` 464 | echo "Deregistering service $UUID" 465 | qmgmt -u api service deregister $UUID 466 | fi 467 | ports: 468 | - name: http 469 | containerPort: 8080 470 | protocol: TCP 471 | - name: quobyte-api 472 | image: quay.io/quobyte/quobyte-server:2 473 | env: 474 | - name: QUOBYTE_SERVICE 475 | value: api 476 | - name: QUOBYTE_API_PORT 477 | value: "7860" 478 | - name: QUOBYTE_REGISTRY 479 | value: registry 480 | - name: QUOBYTE_LOG_LEVEL 481 | value: DEBUG 482 | - name: MAX_MEM 483 | valueFrom: 484 | configMapKeyRef: 485 | name: quobyte-config 486 | key: api.max_mem 487 | - name: MIN_MEM 488 | valueFrom: 489 | configMapKeyRef: 490 | name: quobyte-config 491 | key: api.min_mem 492 | ports: 493 | - name: api 494 | containerPort: 7860 495 | protocol: TCP 496 | command: 497 | - /bin/bash 498 | - -xec 499 | - | 500 | sed "s/.*MIN_MEM_API=.*/MIN_MEM_API=${MIN_MEM}/" -i /etc/default/quobyte 501 | sed "s/.*MAX_MEM_API=.*/MAX_MEM_API=${MAX_MEM}/" -i /etc/default/quobyte 502 | exec /bin/bash -x /opt/main.sh 503 | lifecycle: 504 | preStop: 505 | exec: 506 | command: 507 | - /bin/bash 508 | - -xe 509 | - | 510 | if grep -q uuid /etc/quobyte/api.cfg; then 511 | UUID=`grep "uuid=" /etc/quobyte/data.cfg | cut -d= -f 2` 512 | echo "Deregistering service $UUID" 513 | qmgmt -u api service deregister $UUID 514 | fi 515 | resources: 516 | limits: 517 | memory: "300Mi" 518 | cpu: "100m" 519 | readinessProbe: 520 | timeoutSeconds: 5 521 | tcpSocket: 522 | port: 7860 523 | livenessProbe: 524 | initialDelaySeconds: 30 525 | timeoutSeconds: 10 526 | tcpSocket: 527 | port: 7860 528 | -------------------------------------------------------------------------------- /server_quick_setup.md: -------------------------------------------------------------------------------- 1 | # Quick Start Guide to run Quobyte in Kubernetes 2 | 3 | ## Prerequisites 4 | 5 | Quobyte is designed to run on dedicated disk drives which are formatted with ext4 or xfs and 6 | are initialized as a Quobyte device. 7 | Quobyte 2.0 features the Device Inspector, which can detect unformatted devices 8 | on a node and format and mount them for use in Quobyte. The Device Inspector 9 | requires the [feature gate](https://kubernetes.io/docs/reference/feature-gates/) "MountPropagation=true" to be set. 10 | This feature is in alpha State on Kubernetes 1.8 and requires manual interaction to enable. 11 | If it's not set, the Device Inspector is able to detect and format devices, but they will not be mounted. 12 | 13 | If you plan to create the cluster with the help of the Device Inspector, you can 14 | start a directory based ephemeral registry, or prepare and mount a registry device. 15 | 16 | Instead of creating any devices, just schedule an ephemeral bootstrap registry 17 | first, start registries, data, and metadata services on the target pods 18 | and let the device inspector create the devices. 19 | Afterwards, it is safe to delete the bootstrap registry again. 20 | 21 | Quobyte setup requires bootstrap node. Update `registry.bootstrap_node` in `deploy/config.yaml` with bootstrap node. 22 | 23 | Label your bootstrap node as registry. 24 | ```bash 25 | kubectl label node quobyte_registry="true" 26 | ``` 27 | 28 | Create the quobyte namespace, set up config and services. 29 | 30 | ```bash 31 | kubectl create -f quobyte-ns.yaml 32 | kubectl -n quobyte create -f config.yaml 33 | kubectl -n quobyte create -f quobyte-services.yaml 34 | 35 | kubectl -n quobyte create -f registry-ds.yaml 36 | kubectl -n quobyte create -f data-ds.yaml 37 | kubectl -n quobyte create -f metadata-ds.yaml 38 | ``` 39 | 40 | As soon as the registry is up, start the webconsole, api, and qmgmt pods: 41 | 42 | ```bash 43 | kubectl create -f webconsole-deployment.yaml 44 | kubectl create -f qmgmt-pod.yaml 45 | ``` 46 | 47 | When all pods are up, you should be able to log in to your initial cluster with your preferred browser: 48 | ```bash 49 | kubectl port-forward 8080:8080 50 | ``` 51 | 52 | To schedule other registries, data, and metadata servers, label the nodes accordingly. 53 | The services will show up in the webconsole and the Device Inspector will help 54 | to set up the devices. 55 | 56 | ```bash 57 | kubectl label nodes quobyte_data="true" 58 | kubectl label nodes quobyte_registry="true" 59 | kubectl label nodes quobyte_metadata="true" 60 | ``` 61 | 62 | Once you have three other registries set up on physical devices, it's safe 63 | to remove the `quobyte_registry` label, delete the registry pod on the bootstrap node and update `registry.bootstrap_node` in `deploy/config.yaml` with empty string`""`. 64 | -------------------------------------------------------------------------------- /tools/qbootstrap: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Copyright 2017-2018 Quobyte Inc. 3 | 4 | LANG=en_US.utf-8 5 | DEVICE_ID_FILE="QUOBYTE_DEV_SETUP" 6 | test -z $SMARTCTL && SMARTCTL=/usr/sbin/smartctl 7 | 8 | function print_usage { 9 | if [[ "$(basename $0)" == qbootstrap* ]]; then 10 | printf "Usage: %s [-f] [-t type] [-s serno] mount_point\n" $(basename $0) 11 | echo "Creates a new Quobyte bootstrap device on the disk/file system mounted at ." 12 | echo "-f overwrites any existing device information (dangerous!)." 13 | echo "-s serial_number serial number for a device." 14 | echo "-y do not ask for confirmation." 15 | else 16 | printf "Usage: %s [-f] [-t type] [-s serno] mount_point\n" $(basename $0) 17 | echo "Creates a new Quobyte device on the disk/file system mounted at ." 18 | echo "-f overwrites any existing device information (dangerous!)." 19 | echo "-t DATA|METADATA|REGISTRY optional device type" 20 | echo " this device should be used as." 21 | echo "-s serial_number serial number for a device." 22 | fi 23 | echo 24 | exit 2 25 | } 26 | 27 | force_flag= 28 | type_flag= 29 | serial_number= 30 | yes_mode= 31 | bootstrap_mode= 32 | while getopts "fyBhs:t:" name 33 | do 34 | case $name in 35 | f) force_flag=1;; 36 | t) type_flag="$OPTARG";; 37 | s) serial_number="$OPTARG";; 38 | y) yes_mode=1;; 39 | ?) print_usage;; 40 | esac 41 | done 42 | shift $(($OPTIND - 1)) 43 | 44 | if [[ "$(basename $0)" == qbootstrap* ]] 45 | then 46 | bootstrap_mode=1 47 | fi 48 | 49 | if [ $# -ne 1 ]; then 50 | print_usage 51 | fi 52 | 53 | if [ -n "$EUID" -a $EUID -ne 0 ]; then 54 | echo "Error: This programm must be executed as root (requires access to smartctl/lshw to read device identity)." 55 | exit 4 56 | fi 57 | 58 | MOUNT_POINT="$(readlink -f "$1")" 59 | 60 | if [[ $bootstrap_mode == 1 ]]; then 61 | if [[ "$type_flag" != "" ]];then 62 | echo "Cannot specify type when creating bootstrap registry device." 63 | exit 11 64 | fi 65 | type_flag="R" 66 | 67 | if [ -z "$yes_mode" ]; then 68 | echo "Attention: a bootstrap device must only be created once per Quobyte installation." 69 | echo "If you have already created a bootstrap device on any machine, please use qmkdev -t to create further devices." 70 | read -p "Are you sure you that you want to make ${MOUNT_POINT} your installation's bootstrap device (y/n)? " -n 1 -r 71 | echo # (optional) move to a new line 72 | if [[ ! $REPLY =~ ^[Yy]$ ]] 73 | then 74 | echo "Cancelled." 75 | exit 8 76 | fi 77 | else 78 | echo "Skipping confirmation." 79 | fi 80 | fi 81 | 82 | echo "Device mount point: ${MOUNT_POINT}" 83 | type_flag=$(echo "$type_flag"|awk '{print toupper($0)}') 84 | if [[ "$type_flag" != "" && "$type_flag" != "REGISTRY" && \ 85 | "$type_flag" != "METADATA" && "$type_flag" != "DATA" &&\ 86 | "$type_flag" != "D" && "$type_flag" != "M" && "$type_flag" != "R" ]]; then 87 | echo "Error: Invalid device type." 88 | exit 10 89 | fi 90 | 91 | DEVICE_FILE="${MOUNT_POINT}/${DEVICE_ID_FILE}" 92 | if [ -e "${DEVICE_FILE}" ]; then 93 | if [ -z "$force_flag" ]; then 94 | echo "Error: Selected mount point is already a Quobyte device." 95 | echo " Use -f to overwrite the current device information." 96 | echo "Exiting." 97 | exit 3 98 | else 99 | echo "WARNING: Device information will be overwritten. All data on this device is lost." 100 | mv ${DEVICE_FILE} ${DEVICE_FILE}.bak 101 | fi 102 | fi 103 | 104 | 105 | escaped="${MOUNT_POINT// /\\040}" 106 | # Ignore any rootfs mounts. 107 | lines="$(fgrep " ${escaped} " /proc/mounts | fgrep -v " rootfs ")" 108 | if [[ "$(wc -l <<<"${lines}")" -gt 1 ]]; then 109 | echo "Error: Ambiguous mount point. Multiple matches for ' ${escaped} ' in /proc/mounts." 110 | exit 11 111 | fi 112 | if [[ "$(cut -d " " -f 2 <<<"${lines}")" != "${escaped}" ]]; then 113 | echo "Error: Invalid mount point. ' ${escaped} ' not found in second column of /proc/mounts." 114 | exit 11 115 | fi 116 | DEVICE="$(cut -d " " -f 1 <<<"${lines}")" 117 | FSTYPE="$(cut -d " " -f 3 <<<"${lines}")" 118 | 119 | if [ -z "${DEVICE}" ]; then 120 | echo "Error: Invalid mount point. ' ${escaped} ' not found in /proc/mounts." 121 | exit 11 122 | fi 123 | if [ ! -b "${DEVICE}" ]; then 124 | echo "Error: Invalid mount point. ${DEVICE} is no block device." 125 | exit 12 126 | fi 127 | # NOTE(quobyte): Keep this list in sync with the one in the data service. 128 | SUPPORTED_FS_TYPES="ext4 xfs" 129 | supported_fs_found=0 130 | for expected_fs in $SUPPORTED_FS_TYPES 131 | do 132 | if [[ $expected_fs == $FSTYPE ]]; then 133 | supported_fs_found=1 134 | break 135 | fi 136 | done 137 | if [ $supported_fs_found -eq 0 ]; then 138 | echo "Error: Device file system not supported (${FSTYPE}). Supported file system types are: ${SUPPORTED_FS_TYPES}" 139 | exit 13 140 | fi 141 | 142 | echo "Linux device: ${DEVICE}" 143 | 144 | smartctl_available() { 145 | test -x "$(type -P "${SMARTCTL}")" 146 | } 147 | 148 | set_device_model_smartctl() { 149 | local DEVICE="$1" 150 | DEVICE_MODEL="$("${SMARTCTL}" --info "${DEVICE}" | \ 151 | sed -n 's/Device Model:\s*\(.*\)$/\1/p')" 152 | if [ -z "$DEVICE_MODEL" ]; then 153 | DEVICE_MODEL="$("${SMARTCTL}" --info "${DEVICE}" | \ 154 | sed -n 's/Product:\s*\([-\._a-zA-Z0-9]*\).*/\1/p')" 155 | fi 156 | } 157 | 158 | set_device_serial_smartctl() { 159 | local DEVICE="$1" 160 | DEVICE_SERIAL="$("${SMARTCTL}" --info "${DEVICE}" | \ 161 | sed -n 's/Serial.*:\s*\([-\._a-zA-Z0-9]*\).*/\1/p')" 162 | } 163 | 164 | set_device_model_and_serial_lshw() { 165 | local device="$1" 166 | local lshw="$(type -P "${LSHW:-lshw}" /usr/sbin/lshw /usr/bin/lshw | head -n1)" 167 | 168 | # Parse XML as "lshw -short" output does not contain serial number... 169 | local line="$("${lshw}" -quiet -class disk -xml | awk ' 170 | BEGIN { 171 | node_level = 0 172 | disk_node_entered_at = -1 173 | 174 | logicalname = "" 175 | serial = "" 176 | product = "" 177 | } 178 | 179 | /^ */ { 188 | if (node_level == disk_node_entered_at) { 189 | product = gensub("^ *([^<]+)", "\\1", 1) 190 | } 191 | next 192 | } 193 | /^ */ { 194 | if (node_level == disk_node_entered_at) { 195 | logicalname = gensub("^ *([^<]+)", "\\1", 1) 196 | } 197 | next 198 | } 199 | /^ */ { 200 | if (node_level == disk_node_entered_at) { 201 | serial = gensub("^ *([^<]+)", "\\1", 1) 202 | } 203 | next 204 | } 205 | 206 | /^ *<\/node>/ { 207 | if (node_level == disk_node_entered_at) { 208 | disk_node_entered_at = -1 209 | printf("%s,%s,%s\n", logicalname, serial, product) 210 | 211 | logicalname = "" 212 | serial = "" 213 | product = "" 214 | } 215 | node_level -= 1 216 | } 217 | ' | fgrep "${device}," | head -n1)" 218 | 219 | DEVICE_MODEL="$(cut -d',' -f 3 <<<"${line}")" 220 | DEVICE_SERIAL="$(cut -d',' -f 2 <<<"${line}")" 221 | } 222 | 223 | 224 | if smartctl_available; then 225 | set_device_model_smartctl "${DEVICE}" 226 | else 227 | echo "INFO: smartctl not available, falling back to lshw" 228 | set_device_model_and_serial_lshw "${DEVICE}" 229 | fi 230 | if [ -z "$DEVICE_MODEL" ]; then 231 | DEVICE_MODEL="unknown" 232 | echo "WARNING: Cannot determine device model." 233 | fi 234 | 235 | generate_random_serial=0 236 | if [ -n "$serial_number" ]; then 237 | echo "Override Serial Number: ${serial_number}" 238 | DEVICE_SERIAL="${serial_number}" 239 | else 240 | if [ $generate_random_serial -eq 0 ]; then 241 | if [[ "$DEVICE_MODEL" =~ ^QEMU.* ]]; then 242 | echo "Ignoring pseudo serial number from QEMU device. Generating uuid." 243 | generate_random_serial=1 244 | elif [[ "$DEVICE_MODEL" =~ ^VBOX.* ]]; then 245 | echo "Ignoring pseudo serial number from VBOX device. Generating uuid." 246 | generate_random_serial=1 247 | else 248 | smartctl_available && set_device_serial_smartctl "${DEVICE}" 249 | if [ -z "${DEVICE_SERIAL}" ]; then 250 | echo "Cannot determine device serial number. Generating uuid." 251 | generate_random_serial=1 252 | fi 253 | fi 254 | fi 255 | fi 256 | if [ $generate_random_serial -eq 0 -a ${#DEVICE_SERIAL} -lt 5 ]; then 257 | echo "Device serial number too short. Generating uuid." 258 | generate_random_serial=1 259 | fi 260 | if [ $generate_random_serial -eq 1 ]; then 261 | DEVICE_SERIAL=`uuidgen 2> /dev/null` 262 | if [ -z "$DEVICE_SERIAL" ]; then 263 | echo "Cannot determine device serial number and uuidgen failed." 264 | echo "Please provide a serial number using the -s switch." 265 | exit 7 266 | fi 267 | fi 268 | 269 | if [[ "$type_flag" == "DATA" || "$type_flag" == "D" ]]; then 270 | AVAIL_SPACE=`df -B1 $MOUNT_POINT | grep -v Avail | awk '{ print $4 "\t" }'` 271 | AVAIL_SPACE_HR=`df -h $MOUNT_POINT | grep -v Avail | awk '{ print $4 "\t" }' | tr -d '[[:space:]]'` 272 | if [ $AVAIL_SPACE -lt 21474836480 ]; then 273 | echo "WARNING: $MOUNT_POINT has less than 20G of available space ($AVAIL_SPACE_HR)." 274 | echo "Quobyte may refuse to create new files if such devices are used as data devices." 275 | echo "We highly recommend to use a device with more capacity!" 276 | fi 277 | fi 278 | 279 | CREATION_DATE=`date` 280 | umask 0022 281 | echo "# Quobyte device identifier file" > "${DEVICE_FILE}" 282 | echo "# Created ${CREATION_DATE}" >> "${DEVICE_FILE}" 283 | echo "# Hostname: ${HOSTNAME}" >> "${DEVICE_FILE}" 284 | echo "device.serial=${DEVICE_SERIAL}" >> "${DEVICE_FILE}" 285 | echo "device.model=${DEVICE_MODEL}" >> "${DEVICE_FILE}" 286 | if [ -n "$type_flag" ]; then 287 | if [[ "$type_flag" == "METADATA" || "$type_flag" == "M" ]]; then 288 | echo "device.type=METADATA_DEVICE" >> "${DEVICE_FILE}" 289 | elif [[ "$type_flag" == "DATA" || "$type_flag" == "D" ]]; then 290 | echo "device.type=DATA_DEVICE" >> "${DEVICE_FILE}" 291 | elif [[ "$type_flag" == "REGISTRY" || "$type_flag" == "R" ]]; then 292 | if [[ $bootstrap_mode == 1 ]]; then 293 | echo "device.bootstrap=true" >> "${DEVICE_FILE}" 294 | fi 295 | echo "device.type=DIR_DEVICE" >> "${DEVICE_FILE}" 296 | fi 297 | else 298 | echo "No type specified. To use the device, please assign one or more content" 299 | echo "types using the qmgmt tool." 300 | fi 301 | chown quobyte "${DEVICE_FILE}" 302 | chown quobyte "${MOUNT_POINT}" 303 | if [[ $? != 0 ]]; then 304 | echo "WARNING: Cannot change ownership of mount point, please make sure Quobyte services" 305 | echo " can write to the mount point!" 306 | fi 307 | 308 | if [ "$FSTYPE" = "ext4" ]; then 309 | tune2fs >/dev/null -m 0 "${DEVICE}" 310 | fi 311 | 312 | if [[ $bootstrap_mode == 1 ]]; then 313 | echo "Bootstrap device successfully initialized." 314 | else 315 | echo "Device successfully initialized." 316 | fi 317 | echo 318 | echo "If this device is mounted on a host with a running Quobyte service, " 319 | echo "the device will be registered automatically." 320 | -------------------------------------------------------------------------------- /tools/qmkdev: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Copyright 2017-2018 Quobyte Inc. 3 | 4 | LANG=en_US.utf-8 5 | DEVICE_ID_FILE="QUOBYTE_DEV_SETUP" 6 | test -z $SMARTCTL && SMARTCTL=/usr/sbin/smartctl 7 | 8 | function print_usage { 9 | if [[ "$(basename $0)" == qbootstrap* ]]; then 10 | printf "Usage: %s [-f] [-t type] [-s serno] mount_point\n" $(basename $0) 11 | echo "Creates a new Quobyte bootstrap device on the disk/file system mounted at ." 12 | echo "-f overwrites any existing device information (dangerous!)." 13 | echo "-s serial_number serial number for a device." 14 | echo "-y do not ask for confirmation." 15 | else 16 | printf "Usage: %s [-f] [-t type] [-s serno] mount_point\n" $(basename $0) 17 | echo "Creates a new Quobyte device on the disk/file system mounted at ." 18 | echo "-f overwrites any existing device information (dangerous!)." 19 | echo "-t DATA|METADATA|REGISTRY optional device type" 20 | echo " this device should be used as." 21 | echo "-s serial_number serial number for a device." 22 | fi 23 | echo 24 | exit 2 25 | } 26 | 27 | force_flag= 28 | type_flag= 29 | serial_number= 30 | yes_mode= 31 | bootstrap_mode= 32 | while getopts "fyBhs:t:" name 33 | do 34 | case $name in 35 | f) force_flag=1;; 36 | t) type_flag="$OPTARG";; 37 | s) serial_number="$OPTARG";; 38 | y) yes_mode=1;; 39 | ?) print_usage;; 40 | esac 41 | done 42 | shift $(($OPTIND - 1)) 43 | 44 | if [[ "$(basename $0)" == qbootstrap* ]] 45 | then 46 | bootstrap_mode=1 47 | fi 48 | 49 | if [ $# -ne 1 ]; then 50 | print_usage 51 | fi 52 | 53 | if [ -n "$EUID" -a $EUID -ne 0 ]; then 54 | echo "Error: This programm must be executed as root (requires access to smartctl/lshw to read device identity)." 55 | exit 4 56 | fi 57 | 58 | MOUNT_POINT="$(readlink -f "$1")" 59 | 60 | if [[ $bootstrap_mode == 1 ]]; then 61 | if [[ "$type_flag" != "" ]];then 62 | echo "Cannot specify type when creating bootstrap registry device." 63 | exit 11 64 | fi 65 | type_flag="R" 66 | 67 | if [ -z "$yes_mode" ]; then 68 | echo "Attention: a bootstrap device must only be created once per Quobyte installation." 69 | echo "If you have already created a bootstrap device on any machine, please use qmkdev -t to create further devices." 70 | read -p "Are you sure you that you want to make ${MOUNT_POINT} your installation's bootstrap device (y/n)? " -n 1 -r 71 | echo # (optional) move to a new line 72 | if [[ ! $REPLY =~ ^[Yy]$ ]] 73 | then 74 | echo "Cancelled." 75 | exit 8 76 | fi 77 | else 78 | echo "Skipping confirmation." 79 | fi 80 | fi 81 | 82 | echo "Device mount point: ${MOUNT_POINT}" 83 | type_flag=$(echo "$type_flag"|awk '{print toupper($0)}') 84 | if [[ "$type_flag" != "" && "$type_flag" != "REGISTRY" && \ 85 | "$type_flag" != "METADATA" && "$type_flag" != "DATA" &&\ 86 | "$type_flag" != "D" && "$type_flag" != "M" && "$type_flag" != "R" ]]; then 87 | echo "Error: Invalid device type." 88 | exit 10 89 | fi 90 | 91 | DEVICE_FILE="${MOUNT_POINT}/${DEVICE_ID_FILE}" 92 | if [ -e "${DEVICE_FILE}" ]; then 93 | if [ -z "$force_flag" ]; then 94 | echo "Error: Selected mount point is already a Quobyte device." 95 | echo " Use -f to overwrite the current device information." 96 | echo "Exiting." 97 | exit 3 98 | else 99 | echo "WARNING: Device information will be overwritten. All data on this device is lost." 100 | mv ${DEVICE_FILE} ${DEVICE_FILE}.bak 101 | fi 102 | fi 103 | 104 | 105 | escaped="${MOUNT_POINT// /\\040}" 106 | # Ignore any rootfs mounts. 107 | lines="$(fgrep " ${escaped} " /proc/mounts | fgrep -v " rootfs ")" 108 | if [[ "$(wc -l <<<"${lines}")" -gt 1 ]]; then 109 | echo "Error: Ambiguous mount point. Multiple matches for ' ${escaped} ' in /proc/mounts." 110 | exit 11 111 | fi 112 | if [[ "$(cut -d " " -f 2 <<<"${lines}")" != "${escaped}" ]]; then 113 | echo "Error: Invalid mount point. ' ${escaped} ' not found in second column of /proc/mounts." 114 | exit 11 115 | fi 116 | DEVICE="$(cut -d " " -f 1 <<<"${lines}")" 117 | FSTYPE="$(cut -d " " -f 3 <<<"${lines}")" 118 | 119 | if [ -z "${DEVICE}" ]; then 120 | echo "Error: Invalid mount point. ' ${escaped} ' not found in /proc/mounts." 121 | exit 11 122 | fi 123 | if [ ! -b "${DEVICE}" ]; then 124 | echo "Error: Invalid mount point. ${DEVICE} is no block device." 125 | exit 12 126 | fi 127 | # NOTE(quobyte): Keep this list in sync with the one in the data service. 128 | SUPPORTED_FS_TYPES="ext4 xfs" 129 | supported_fs_found=0 130 | for expected_fs in $SUPPORTED_FS_TYPES 131 | do 132 | if [[ $expected_fs == $FSTYPE ]]; then 133 | supported_fs_found=1 134 | break 135 | fi 136 | done 137 | if [ $supported_fs_found -eq 0 ]; then 138 | echo "Error: Device file system not supported (${FSTYPE}). Supported file system types are: ${SUPPORTED_FS_TYPES}" 139 | exit 13 140 | fi 141 | 142 | echo "Linux device: ${DEVICE}" 143 | 144 | smartctl_available() { 145 | test -x "$(type -P "${SMARTCTL}")" 146 | } 147 | 148 | set_device_model_smartctl() { 149 | local DEVICE="$1" 150 | DEVICE_MODEL="$("${SMARTCTL}" --info "${DEVICE}" | \ 151 | sed -n 's/Device Model:\s*\(.*\)$/\1/p')" 152 | if [ -z "$DEVICE_MODEL" ]; then 153 | DEVICE_MODEL="$("${SMARTCTL}" --info "${DEVICE}" | \ 154 | sed -n 's/Product:\s*\([-\._a-zA-Z0-9]*\).*/\1/p')" 155 | fi 156 | } 157 | 158 | set_device_serial_smartctl() { 159 | local DEVICE="$1" 160 | DEVICE_SERIAL="$("${SMARTCTL}" --info "${DEVICE}" | \ 161 | sed -n 's/Serial.*:\s*\([-\._a-zA-Z0-9]*\).*/\1/p')" 162 | } 163 | 164 | set_device_model_and_serial_lshw() { 165 | local device="$1" 166 | local lshw="$(type -P "${LSHW:-lshw}" /usr/sbin/lshw /usr/bin/lshw | head -n1)" 167 | 168 | # Parse XML as "lshw -short" output does not contain serial number... 169 | local line="$("${lshw}" -quiet -class disk -xml | awk ' 170 | BEGIN { 171 | node_level = 0 172 | disk_node_entered_at = -1 173 | 174 | logicalname = "" 175 | serial = "" 176 | product = "" 177 | } 178 | 179 | /^ */ { 188 | if (node_level == disk_node_entered_at) { 189 | product = gensub("^ *([^<]+)", "\\1", 1) 190 | } 191 | next 192 | } 193 | /^ */ { 194 | if (node_level == disk_node_entered_at) { 195 | logicalname = gensub("^ *([^<]+)", "\\1", 1) 196 | } 197 | next 198 | } 199 | /^ */ { 200 | if (node_level == disk_node_entered_at) { 201 | serial = gensub("^ *([^<]+)", "\\1", 1) 202 | } 203 | next 204 | } 205 | 206 | /^ *<\/node>/ { 207 | if (node_level == disk_node_entered_at) { 208 | disk_node_entered_at = -1 209 | printf("%s,%s,%s\n", logicalname, serial, product) 210 | 211 | logicalname = "" 212 | serial = "" 213 | product = "" 214 | } 215 | node_level -= 1 216 | } 217 | ' | fgrep "${device}," | head -n1)" 218 | 219 | DEVICE_MODEL="$(cut -d',' -f 3 <<<"${line}")" 220 | DEVICE_SERIAL="$(cut -d',' -f 2 <<<"${line}")" 221 | } 222 | 223 | 224 | if smartctl_available; then 225 | set_device_model_smartctl "${DEVICE}" 226 | else 227 | echo "INFO: smartctl not available, falling back to lshw" 228 | set_device_model_and_serial_lshw "${DEVICE}" 229 | fi 230 | if [ -z "$DEVICE_MODEL" ]; then 231 | DEVICE_MODEL="unknown" 232 | echo "WARNING: Cannot determine device model." 233 | fi 234 | 235 | generate_random_serial=0 236 | if [ -n "$serial_number" ]; then 237 | echo "Override Serial Number: ${serial_number}" 238 | DEVICE_SERIAL="${serial_number}" 239 | else 240 | if [ $generate_random_serial -eq 0 ]; then 241 | if [[ "$DEVICE_MODEL" =~ ^QEMU.* ]]; then 242 | echo "Ignoring pseudo serial number from QEMU device. Generating uuid." 243 | generate_random_serial=1 244 | elif [[ "$DEVICE_MODEL" =~ ^VBOX.* ]]; then 245 | echo "Ignoring pseudo serial number from VBOX device. Generating uuid." 246 | generate_random_serial=1 247 | else 248 | smartctl_available && set_device_serial_smartctl "${DEVICE}" 249 | if [ -z "${DEVICE_SERIAL}" ]; then 250 | echo "Cannot determine device serial number. Generating uuid." 251 | generate_random_serial=1 252 | fi 253 | fi 254 | fi 255 | fi 256 | if [ $generate_random_serial -eq 0 -a ${#DEVICE_SERIAL} -lt 5 ]; then 257 | echo "Device serial number too short. Generating uuid." 258 | generate_random_serial=1 259 | fi 260 | if [ $generate_random_serial -eq 1 ]; then 261 | DEVICE_SERIAL=`uuidgen 2> /dev/null` 262 | if [ -z "$DEVICE_SERIAL" ]; then 263 | echo "Cannot determine device serial number and uuidgen failed." 264 | echo "Please provide a serial number using the -s switch." 265 | exit 7 266 | fi 267 | fi 268 | 269 | if [[ "$type_flag" == "DATA" || "$type_flag" == "D" ]]; then 270 | AVAIL_SPACE=`df -B1 $MOUNT_POINT | grep -v Avail | awk '{ print $4 "\t" }'` 271 | AVAIL_SPACE_HR=`df -h $MOUNT_POINT | grep -v Avail | awk '{ print $4 "\t" }' | tr -d '[[:space:]]'` 272 | if [ $AVAIL_SPACE -lt 21474836480 ]; then 273 | echo "WARNING: $MOUNT_POINT has less than 20G of available space ($AVAIL_SPACE_HR)." 274 | echo "Quobyte may refuse to create new files if such devices are used as data devices." 275 | echo "We highly recommend to use a device with more capacity!" 276 | fi 277 | fi 278 | 279 | CREATION_DATE=`date` 280 | umask 0022 281 | echo "# Quobyte device identifier file" > "${DEVICE_FILE}" 282 | echo "# Created ${CREATION_DATE}" >> "${DEVICE_FILE}" 283 | echo "# Hostname: ${HOSTNAME}" >> "${DEVICE_FILE}" 284 | echo "device.serial=${DEVICE_SERIAL}" >> "${DEVICE_FILE}" 285 | echo "device.model=${DEVICE_MODEL}" >> "${DEVICE_FILE}" 286 | if [ -n "$type_flag" ]; then 287 | if [[ "$type_flag" == "METADATA" || "$type_flag" == "M" ]]; then 288 | echo "device.type=METADATA_DEVICE" >> "${DEVICE_FILE}" 289 | elif [[ "$type_flag" == "DATA" || "$type_flag" == "D" ]]; then 290 | echo "device.type=DATA_DEVICE" >> "${DEVICE_FILE}" 291 | elif [[ "$type_flag" == "REGISTRY" || "$type_flag" == "R" ]]; then 292 | if [[ $bootstrap_mode == 1 ]]; then 293 | echo "device.bootstrap=true" >> "${DEVICE_FILE}" 294 | fi 295 | echo "device.type=DIR_DEVICE" >> "${DEVICE_FILE}" 296 | fi 297 | else 298 | echo "No type specified. To use the device, please assign one or more content" 299 | echo "types using the qmgmt tool." 300 | fi 301 | chown quobyte "${DEVICE_FILE}" 302 | chown quobyte "${MOUNT_POINT}" 303 | if [[ $? != 0 ]]; then 304 | echo "WARNING: Cannot change ownership of mount point, please make sure Quobyte services" 305 | echo " can write to the mount point!" 306 | fi 307 | 308 | if [ "$FSTYPE" = "ext4" ]; then 309 | tune2fs >/dev/null -m 0 "${DEVICE}" 310 | fi 311 | 312 | if [[ $bootstrap_mode == 1 ]]; then 313 | echo "Bootstrap device successfully initialized." 314 | else 315 | echo "Device successfully initialized." 316 | fi 317 | echo 318 | echo "If this device is mounted on a host with a running Quobyte service, " 319 | echo "the device will be registered automatically." 320 | -------------------------------------------------------------------------------- /tools/setup-remote: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #set -x 3 | 4 | case "$1" in 5 | list) 6 | ssh $2 lsblk 7 | ;; 8 | 9 | download) 10 | HOST=$2 11 | ssh $HOST "curl https://raw.githubusercontent.com/quobyte/kubernetes/master/tools/qmkdev -o /tmp/qmkdev; \ 12 | chmod +x /tmp/qmkdev" 13 | ;; 14 | 15 | makedata) 16 | HOST=$2 17 | DEVICE=$3 18 | ssh $HOST "sudo /tmp/qmkdev -f -s $(uuidgen) -t DATA /mnt/quobyte/device_${DEVICE}" 19 | ;; 20 | 21 | makemetadata) 22 | HOST=$2 23 | DEVICE=$3 24 | ssh $HOST "sudo /tmp/qmkdev -f -s $(uuidgen) -t METADATA /mnt/quobyte/device_${DEVICE}" 25 | ;; 26 | 27 | makeregistry) 28 | HOST=$2 29 | DEVICE=$3 30 | ssh $HOST "sudo /tmp/qmkdev -f -s $(uuidgen) -t REGISTRY /mnt/quobyte/device_${DEVICE}" 31 | ;; 32 | 33 | bootstrap-device) 34 | HOST=$2 35 | DEVICE=$3 36 | ssh $HOST "curl https://raw.githubusercontent.com/quobyte/kubernetes/master/tools/qbootstrap -o /tmp/qbootstrap; \ 37 | chmod +x /tmp/qbootstrap" 38 | ssh $HOST "yes y | sudo /tmp/qbootstrap /mnt/quobyte/device_${DEVICE}" 39 | ;; 40 | 41 | bootstrap-dir) 42 | HOST=$2 43 | DEVICEFILE=/var/lib/quobyte/devices/registry-bootstrap/QUOBYTE_DEV_SETUP 44 | ssh $HOST "mkdir -p /var/lib/quobyte/devices/registry-bootstrap" 45 | echo "# Quobyte device identifier file" | ssh $HOST "sudo cat > ${DEVICEFILE}" 46 | echo "# Hostname: ${HOST}" | ssh $HOST "sudo cat >> ${DEVICEFILE}" 47 | echo "device.serial=$(uuidgen)" | ssh $HOST "sudo cat >> ${DEVICEFILE}" 48 | echo "device.model=unknown" | ssh $HOST "sudo cat >> ${DEVICEFILE}" 49 | echo "device.type=DIR_DEVICE" | ssh $HOST "sudo cat >> ${DEVICEFILE}" 50 | ;; 51 | 52 | format) 53 | HOST=$2 54 | DEVICE=$3 55 | echo "creating DATA on $HOST device $DEVICE" 56 | ssh -t $2 \ 57 | "sudo mkfs.xfs -f -isize=1024 /dev/${DEVICE}; 58 | sudo mkdir -p /mnt/quobyte/device_${DEVICE} 59 | echo "/dev/${DEVICE} /mnt/quobyte/device_$DEVICE xfs relatime,nodiscard 0 0" | sudo tee --append /etc/fstab 60 | sudo mount -a 61 | " 62 | ;; 63 | 64 | reformat) 65 | HOST=$2 66 | DEVICE=$3 67 | ssh -t $2 \ 68 | "sudo umount /mnt/quobyte/device_$DEVICE; \ 69 | sudo mkfs.xfs -f -isize=1024 /dev/${DEVICE}1; 70 | sudo mount -a 71 | " 72 | ;; 73 | 74 | *) 75 | echo "NOTE: This tool is a quick hack and not a recommended or supported way for setup." 76 | echo $"Usage: $0 " 77 | echo $"$0 download | Downloads qmked to the host" 78 | echo $"$0 format | formats the device " 79 | echo $"$0 reformat | unmounts and reformats" 80 | echo $"$0 bootstrap-device | call qmkev on the device" 81 | echo $"$0 bootstrap-dir | create a dir based bootstrap" 82 | echo $"$0 makedata | marks the formatted and mounted device as DATA" 83 | echo $"$0 makemetadata | marks the formatted and mounted device as METADATA" 84 | echo $"$0 makeregistry | marks the formatted and mounted device as REGISTRY" 85 | exit 2 86 | esac 87 | -------------------------------------------------------------------------------- /using_quobyte_volumes.md: -------------------------------------------------------------------------------- 1 | # Volume Access 2 | 3 | Quobyte volumes can be accessed from Pods in multiple ways. Either, via 4 | the QuobyteVolumeSource like in `deploy/example-pod.yaml` 5 | 6 | ```yaml 7 | volumes: 8 | - name: quobytevolume 9 | quobyte: 10 | registry: ignored:7861 # Unused string required for API compatibility 11 | volume: testVolume 12 | readOnly: false 13 | user: username 14 | group: groupname 15 | ``` 16 | 17 | or as a PersistentVolumeClaim, which is defined in the same namespace, where 18 | the accessing pod is running. The claim (volumes/claim.json) 19 | 20 | ```json 21 | { 22 | "kind": "PersistentVolumeClaim", 23 | "apiVersion": "v1", 24 | "metadata": { 25 | "name": "test" 26 | }, 27 | "spec": { 28 | "accessModes": [ 29 | "ReadWriteOnce" 30 | ], 31 | "resources": { 32 | "requests": { 33 | "storage": "3Gi" 34 | } 35 | }, 36 | "storageClassName": "base" 37 | } 38 | } 39 | ``` 40 | 41 | ```bash 42 | $ kubectl -n quobyte create -f volumes/claim.json 43 | ``` 44 | 45 | is mounted in the Pod like (see volumes/example-pod.yaml): 46 | 47 | ```yaml 48 | volumes: 49 | - name: quobytepvc 50 | persistentVolumeClaim: 51 | claimName: test 52 | ``` 53 | 54 | The PersistentVolumeClaim is bound to a PersistentVolume which is created by an 55 | administrator for existing Quobyte Volumes or dynamically by the StorageClass 56 | (volumes/storageclass.yaml). 57 | 58 | If a Quobyte Volume already exists, the administrator can make it available 59 | as a PersistentVolume by creating a resource (volumes/pv.yaml). 60 | Please note that Quobyte currently ignores the required capacity.storage field, 61 | since its using internal quota mechanisms. 62 | 63 | ```yaml 64 | kind: PersistentVolume 65 | apiVersion: v1 66 | metadata: 67 | name: test 68 | labels: 69 | type: quobyte 70 | spec: 71 | capacity: 72 | storage: 1Gi 73 | accessModes: 74 | - ReadWriteOnce 75 | storageClassName: "base" 76 | quobyte: 77 | registry: ignored:7861 # Unused string required for API compatibility 78 | volume: test 79 | readOnly: false 80 | user: username 81 | group: groupname 82 | ``` 83 | 84 | ```bash 85 | $ kubectl -n quobyte create -f volumes/pv.yaml 86 | ``` 87 | 88 | For dynamic provisioning, a StorageClass is created, which manages the 89 | lifecycle of Quobyte volumes. 90 | Each provisioner is bound to a Quobyte tenant, which is specified in the 'quobyteTenant' field. 91 | The UUID of the tenant can be found in the Webconsole. 92 | For Kubernetes > 1.10, the 'quobyteTenant' can specify the name of the tenant. 93 | 94 | ```yaml 95 | apiVersion: storage.k8s.io/v1 96 | kind: StorageClass 97 | metadata: 98 | name: base 99 | provisioner: kubernetes.io/quobyte 100 | parameters: 101 | quobyteAPIServer: "http://api.quobyte:7860" 102 | registry: "registry.quobyte:7861" 103 | adminSecretName: "quobyte-admin-secret" 104 | adminSecretNamespace: "kube-system" 105 | user: "username" 106 | group: "groupname" 107 | quobyteConfig: "BASE" 108 | quobyteTenant: "uuid of tenant" 109 | createQuota: "False" 110 | ``` 111 | 112 | To enable it, first create the Quobyte admin secret according to your API 113 | credentials in volumes/quobyte-admin-secret.yaml (defaults to admin:quobyte): 114 | ```yaml 115 | apiVersion: v1 116 | kind: Secret 117 | metadata: 118 | name: quobyte-admin-secret 119 | type: "kubernetes.io/quobyte" 120 | data: 121 | password: cXVvYnl0ZQ== 122 | user: YWRtaW4= 123 | type: kubernetes.io/quobyte 124 | ``` 125 | 126 | The password and user strings are base64 encoded and can be created like 127 | `echo -n "quobyte" | base64`. 128 | 129 | ```bash 130 | $ kubectl -n kube-system create -f volumes/quobyte-admin-secret.yaml 131 | $ kubectl -n quobyte create -f volumes/storageclass.yaml 132 | ``` 133 | 134 | ## Quobyte Tenants and Kubernetes Namespaces 135 | 136 | Quobyte supports multiple tenants and provides a secure mapping of containers to 137 | users known in Quobyte. 138 | For a longer read, please see the article on the Quobyte blog: 139 | [The State of Secure Storage Access in Container Infrastructures](https://www.quobyte.com/blog/2017/03/17/the-state-of-secure-storage-access-in-container-infrastructures/) 140 | 141 | The Kubernetes deployments of the Quobyte client use the `--allow-usermapping-in-volumename`, which allows to map all accesses to the 142 | volume to a particular user/group, independent of the accessing user. 143 | If you specify `user#group@volume_name` instead of just the volumename, 144 | the Quobyte client will map all storage accesses to the `user:group`. 145 | For example, if your container runs internally with id 0, or you have multiple 146 | arbitrary user ids in your containers, all files in the Quobyte volume will be 147 | owned by `user:group`. 148 | 149 | ## Further Readings 150 | 151 | - https://github.com/kubernetes/examples/tree/master/staging/volumes/quobyte 152 | 153 | - https://kubernetes.io/docs/tasks/configure-pod-container/configure-persistent-volume-storage/#create-a-persistentvolume 154 | 155 | - https://kubernetes.io/docs/concepts/storage/storage-classes/#quobyte 156 | -------------------------------------------------------------------------------- /volumes/claim.json: -------------------------------------------------------------------------------- 1 | { 2 | "kind": "PersistentVolumeClaim", 3 | "apiVersion": "v1", 4 | "metadata": { 5 | "name": "test" 6 | }, 7 | "spec": { 8 | "accessModes": [ 9 | "ReadWriteOnce" 10 | ], 11 | "resources": { 12 | "requests": { 13 | "storage": "1Gi" 14 | } 15 | }, 16 | "storageClassName": "base" 17 | } 18 | } 19 | -------------------------------------------------------------------------------- /volumes/example-pod.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ReplicationController 3 | metadata: 4 | name: server 5 | spec: 6 | replicas: 1 7 | selector: 8 | role: server 9 | template: 10 | metadata: 11 | labels: 12 | role: server 13 | spec: 14 | containers: 15 | - name: server 16 | image: nginx 17 | volumeMounts: 18 | - mountPath: /var/lib/www/html 19 | name: quobytepvc 20 | volumes: 21 | - name: quobytepvc 22 | persistentVolumeClaim: 23 | claimName: test 24 | -------------------------------------------------------------------------------- /volumes/pv.yaml: -------------------------------------------------------------------------------- 1 | kind: PersistentVolume 2 | apiVersion: v1 3 | metadata: 4 | name: test 5 | labels: 6 | type: quobyte 7 | spec: 8 | capacity: 9 | storage: 1Gi 10 | accessModes: 11 | - ReadWriteOnce 12 | storageClassName: "base" 13 | quobyte: 14 | # the registry parameter was an required parameter, but is not used anymore. 15 | # It's here for API compatibility. 16 | registry: ignored:7861 17 | volume: test 18 | readOnly: false 19 | user: root 20 | group: root 21 | -------------------------------------------------------------------------------- /volumes/quobyte-admin-secret.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Secret 3 | metadata: 4 | name: quobyte-admin-secret 5 | type: "kubernetes.io/quobyte" 6 | data: 7 | password: cXVvYnl0ZQ== 8 | user: YWRtaW4= 9 | type: kubernetes.io/quobyte 10 | -------------------------------------------------------------------------------- /volumes/storageclass.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: storage.k8s.io/v1 2 | kind: StorageClass 3 | metadata: 4 | name: base 5 | provisioner: kubernetes.io/quobyte 6 | parameters: 7 | quobyteAPIServer: "http://api.quobyte:7860" 8 | registry: "registry.quobyte:7861" 9 | adminSecretName: "quobyte-admin-secret" 10 | adminSecretNamespace: "kube-system" 11 | user: "root" 12 | group: "root" 13 | quobyteConfig: "BASE" 14 | # For kubernetes < 1.10.2, quobyteTenant needs point to the tenant's uuid. 15 | # Later versions are able to resolve the tenant's name. 16 | quobyteTenant: "" 17 | createQuota: "False" 18 | reclaimPolicy: Retain 19 | --------------------------------------------------------------------------------