├── BE-pv.yaml ├── FE-pv.yaml ├── FE-deployment.yaml ├── BE-deployment.yaml └── README.md /BE-pv.yaml: -------------------------------------------------------------------------------- 1 | #pv 2 | apiVersion: v1 3 | kind: PersistentVolume 4 | metadata: 5 | name: volume-doris-be-node-010000000234 6 | spec: 7 | storageClassName: volume-doris-be 8 | capacity: 9 | storage: 50Gi 10 | accessModes: 11 | - ReadWriteOnce 12 | local: 13 | path: "/data/doris-be" 14 | nodeAffinity: 15 | required: 16 | nodeSelectorTerms: 17 | - matchExpressions: 18 | - key: kubernetes.io/hostname 19 | operator: In 20 | values: 21 | - node-010000000234 22 | persistentVolumeReclaimPolicy: Recycle 23 | --- 24 | #sc 25 | apiVersion: storage.k8s.io/v1 26 | kind: StorageClass 27 | metadata: 28 | name: volume-doris-be 29 | provisioner: kubernetes.io/no-provisioner 30 | reclaimPolicy: Retain 31 | --- 32 | #pvc 33 | apiVersion: v1 34 | kind: PersistentVolumeClaim 35 | metadata: 36 | name: pvc-doris-be 37 | spec: 38 | accessModes: 39 | - ReadWriteOnce 40 | resources: 41 | requests: 42 | storage: 20Gi 43 | storageClassName: volume-doris-be 44 | volumeMode: Filesystem 45 | -------------------------------------------------------------------------------- /FE-pv.yaml: -------------------------------------------------------------------------------- 1 | #pv 2 | apiVersion: v1 3 | kind: PersistentVolume 4 | metadata: 5 | name: volume-doris-fe-node-010000000233 6 | spec: 7 | storageClassName: volume-doris-fe 8 | capacity: 9 | storage: 10Gi 10 | accessModes: 11 | - ReadWriteOnce 12 | local: 13 | path: "/data/doris-fe" 14 | nodeAffinity: 15 | required: 16 | nodeSelectorTerms: 17 | - matchExpressions: 18 | - key: kubernetes.io/hostname 19 | operator: In 20 | values: 21 | - node-010000000233 22 | persistentVolumeReclaimPolicy: Recycle 23 | volumeMode: Filesystem 24 | --- 25 | #sc 26 | apiVersion: storage.k8s.io/v1 27 | kind: StorageClass 28 | metadata: 29 | name: volume-doris-fe 30 | provisioner: kubernetes.io/no-provisioner 31 | reclaimPolicy: Retain 32 | --- 33 | #pvc 34 | apiVersion: v1 35 | kind: PersistentVolumeClaim 36 | metadata: 37 | name: pvc-doris-fe 38 | spec: 39 | accessModes: 40 | - ReadWriteOnce 41 | resources: 42 | requests: 43 | storage: 10Gi 44 | storageClassName: volume-doris-fe 45 | volumeMode: Filesystem -------------------------------------------------------------------------------- /FE-deployment.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | kind: Deployment 3 | apiVersion: apps/v1 4 | metadata: 5 | name: doris-fe 6 | labels: 7 | app: doris-fe 8 | spec: 9 | replicas: 1 10 | selector: 11 | matchLabels: 12 | app: doris-fe 13 | template: 14 | metadata: 15 | labels: 16 | app: doris-fe 17 | spec: 18 | hostNetwork: true 19 | dnsPolicy: ClusterFirstWithHostNet 20 | containers: 21 | - name: doris-fe 22 | env: 23 | - name: FE_IPADDRESS 24 | valueFrom: 25 | fieldRef: 26 | fieldPath: status.podIP 27 | image: 359261016/apache-doris-fe:v0.15 28 | command: [ "/bin/bash", "-ce", "/opt/fe/bin/start_fe.sh --daemon;tail -f /dev/null" ] 29 | volumeMounts: 30 | - mountPath: /opt/fe/doris-meta 31 | name: volume-fe 32 | livenessProbe: 33 | httpGet: 34 | path: /api/bootstrap 35 | port: 8030 36 | initialDelaySeconds: 300 37 | periodSeconds: 30 38 | timeoutSeconds: 5 39 | failureThreshold: 3 40 | ports: 41 | - containerPort: 8030 42 | protocol: TCP 43 | - containerPort: 9020 44 | protocol: TCP 45 | - containerPort: 9030 46 | protocol: TCP 47 | - containerPort: 9010 48 | protocol: TCP 49 | resources: 50 | limits: 51 | cpu: 2 52 | memory: 4G 53 | requests: 54 | cpu: 200m 55 | memory: 1G 56 | volumes: 57 | - name: volume-fe 58 | persistentVolumeClaim: 59 | claimName: pvc-doris-fe 60 | -------------------------------------------------------------------------------- /BE-deployment.yaml: -------------------------------------------------------------------------------- 1 | kind: Deployment 2 | apiVersion: apps/v1 3 | metadata: 4 | name: doris-be 5 | labels: 6 | app: doris-be 7 | spec: 8 | replicas: 1 9 | selector: 10 | matchLabels: 11 | app: doris-be 12 | template: 13 | metadata: 14 | labels: 15 | app: doris-be 16 | spec: 17 | hostNetwork: true 18 | dnsPolicy: ClusterFirstWithHostNet 19 | containers: 20 | - name: doris-be 21 | env: 22 | - name: BE_IPADDRESS 23 | valueFrom: 24 | fieldRef: 25 | fieldPath: status.podIP 26 | image: 359261016/apache-doris-be:v0.15 27 | command: [ "/bin/bash", "-ce", "/opt/be/bin/start_be.sh --daemon;tail -f /dev/null" ] 28 | livenessProbe: 29 | httpGet: 30 | path: /api/health 31 | port: 8040 32 | initialDelaySeconds: 300 33 | periodSeconds: 30 34 | timeoutSeconds: 5 35 | failureThreshold: 3 36 | ports: 37 | - containerPort: 9060 38 | protocol: TCP 39 | - containerPort: 9070 40 | protocol: TCP 41 | - containerPort: 8040 42 | protocol: TCP 43 | - containerPort: 9050 44 | protocol: TCP 45 | - containerPort: 8060 46 | protocol: TCP 47 | resources: 48 | limits: 49 | cpu: 2 50 | memory: 2G 51 | requests: 52 | cpu: 200m 53 | memory: 1G 54 | volumeMounts: 55 | - mountPath: /opt/be/storage 56 | name: volume-be 57 | volumes: 58 | - name: volume-be 59 | persistentVolumeClaim: 60 | claimName: pvc-doris-be 61 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # doris-on-k8s 2 | 3 | # 前言 4 | 5 | 当前使用**hostNetWork**方式部署。包括：1个FE，1个BE。 6 | 7 | 说明： 8 | 9 | - StatefulSet+SVC方式（已舍弃） 10 | 11 | 优点：常见的部署方式，一个节点可部署多个BE。 12 | 13 | 缺点：Doris服务连续性缺失。测试过程中发现，因为FE记录的是podIP，所以每次重启BE-pod后，podIP变化，需手动重新ADD BACKEND。 14 | 15 | - hostNetwork方式（目前部署方式） 16 | 17 | 优点：PodIP都是节点IP，不用每次ADD BACKEND。保证了Doris服务的连续性。 18 | 19 | 缺点：每个节点只能启动一个BE，因为对于同Deployment下的hostNetwork: true启动的Pod，每个node上只能启动一个。 20 | 21 | 22 | 23 | ## 文件说明 24 | 25 | * Doris-FE 26 | 27 | FE-pv.yaml：创建FE需使用的存储，包括pv、sc、pvc。 28 | 29 | FE-deployment.yaml：FE部署脚本 30 | 31 | * Doris-BE 32 | 33 | FE-pv.yaml：创建BE需使用的存储，包括pv、sc、pvc。 34 | 35 | FE-deployment.yaml：BE部署脚本 36 | 37 | ## 镜像说明 38 | 39 | * docker pull 359261016/apache-doris-be:v0.15 40 | * docker pull 359261016/apache-doris-fe:v0.15 41 | 42 | 也可自己参照官方文档构建镜像 43 | http://doris.incubator.apache.org/zh-CN/installing/compilation.html#%E4%BD%BF%E7%94%A8-docker-%E5%BC%80%E5%8F%91%E9%95%9C%E5%83%8F%E7%BC%96%E8%AF%91-%E6%8E%A8%E8%8D%90 44 | 45 | # 部署步骤 46 | 47 | ## 1.修改pv.yaml 48 | 49 | 下载四个yaml，**将FE-pv.yaml，BE-pv.yaml中的节点亲和性，修改成自己集群的node**。或删除节点亲和性相关配置。 50 | 51 | ``` 52 | nodeAffinity: 53 | required: 54 | nodeSelectorTerms: 55 | - matchExpressions: 56 | - key: kubernetes.io/hostname 57 | operator: In 58 | values: 59 | - node-010000000233 60 | ``` 61 | 62 | 修改后执行命令即可完成FE、BE的部署 63 | 64 | ```shell 65 | kubectl apply -f FE-pv.yaml 66 | kubectl apply -f FE-deployment.yaml 67 | kubectl apply -f BE-pv.yaml 68 | kubectl apply -f BE-deployment.yaml 69 | ``` 70 | 71 | 72 | ## 2. 添加BE到FE 73 | 74 | 执行完查看doris服务，启动正常 75 | 76 | ![img](https://intranetproxy.alipay.com/skylark/lark/0/2021/png/13456375/1640153315318-55e7f8aa-062d-4cf0-a99a-a1fd03645094.png) 77 | 78 | 添加BE到FE 79 | 80 | ```yaml 81 | #连接FE，172.16.0.161换成自己的节点IP 82 | mysql -h172.16.0.161 -P9030 -uroot 83 | SHOW PROC '/backends'; 84 | #添加BE，如果Alive=true，添加成功 85 | ALTER SYSTEM ADD BACKEND "172.16.0.161:9050"; 86 | ``` 87 | 88 | ![img](https://intranetproxy.alipay.com/skylark/lark/0/2021/png/13456375/1640153257035-4725cc76-c171-4f83-9c51-ca563b66fae4.png) 89 | 90 | BE已添加，可正常使用Doris 91 | 92 | **至此doris on k8s部署结束**。后续使用请参考官方文档：https://doris.apache.org/master/zh-CN/installing/compilation.html 93 | 94 | 95 | 96 | ## 3. 简单使用 97 | 98 | ```sql 99 | #简单样例 100 | CREATE DATABASE example_db; 101 | USE example_db; 102 | CREATE TABLE table1 103 | ( 104 | siteid INT DEFAULT '10', 105 | citycode SMALLINT, 106 | username VARCHAR(32) DEFAULT '', 107 | pv BIGINT SUM DEFAULT '0' 108 | ) 109 | AGGREGATE KEY(siteid, citycode, username) 110 | DISTRIBUTED BY HASH(siteid) BUCKETS 10 111 | PROPERTIES("replication_num" = "1"); 112 | 113 | insert into table1 values(1,1,"test",2); 114 | select * from table1; 115 | ``` 116 | 117 | ![img](https://intranetproxy.alipay.com/skylark/lark/0/2021/png/13456375/1640153489339-deed85b3-0341-4e75-9be4-0fdfbc28fec6.png) 118 | 119 | # 附问题记录 120 | 121 | ## 1.每次BE重启后都需手动ADD BACKEND 122 | 123 | 现象：原本使用StatefulSet+SVC方式部署BE，执行ALTER SYSTEM ADD BACKEND "doris-be-0.doris-be-svc:9050"; BE添加到集群，当前podIP是10.108.2.107。 124 | 125 | ![img](https://intranetproxy.alipay.com/skylark/lark/0/2021/png/13456375/1639988922169-2a92a32d-2b4f-4e26-9e0d-4c5d34356c9f.png) 126 | 127 | 手动delete BE pod后SHOW PROC '/backends';后，新起的BE pod无法使用 128 | 129 | ![img](https://intranetproxy.alipay.com/skylark/lark/0/2021/png/13456375/1639988941548-3635bdf0-4fb4-4656-9689-48359707aac1.png) 130 | 原因：ADD BACKEND命令记录的是IP，而不是host。所以当拉起新的BE pod后，即使host不变，但是PodIP已经变化。所以检查失败。 131 | 132 | 解决：不使用StatefulSet+SVC。改用Deployment+hostNetwork方式，将PodIP固定为NodeIP。 133 | 134 | 演示：手动删除并启动新BE，Doris服务如下，无需手动ADD BACKEND。 135 | 136 | ![img](https://intranetproxy.alipay.com/skylark/lark/0/2021/png/13456375/1639990207541-0fdb9a0e-4db8-48b4-8955-cebb2295fa8f.png) 137 | 138 | 139 | 140 | ## 2.重启FE，每次都新clusterId，导致原BE无法加入集群 141 | 142 | 场景：StatefulSet+SVC方式部署场景下，BE与FE运行正常，此时BE中记录当前的clusterId。此时手动delete FE，新起的FE的clusterId会变化，导致执行ADD BACKEND不成功，报错invalid cluster id。 143 | 144 | ![img](https://intranetproxy.alipay.com/skylark/lark/0/2021/png/13456375/1639991310239-f64ca104-c8f4-4053-93d6-eafce4b8c6a6.png) 145 | 146 | ![img](https://intranetproxy.alipay.com/skylark/lark/0/2021/png/13456375/1639990632935-21c2810f-7833-4f76-a95b-a26f1ecf4f95.png) 147 | 148 | 原因： 149 | 150 | ![img](https://intranetproxy.alipay.com/skylark/lark/0/2021/png/13456375/1639991198029-5be7a4bb-d6b3-475e-9864-a036bd220e3a.png) 151 | 152 | 解决：改用Deployment+hostNetwork方式，将FE的PodIP固定为NodeIP。 153 | 154 | 演示：手动删除并启动新FE，Doris服务如下，无需手动ADD BACKEND。 155 | 156 | ![img](https://intranetproxy.alipay.com/skylark/lark/0/2021/png/13456375/1639990825586-1673bf43-40ef-466f-9a67-21bdbb0b5a77.png) 157 | 158 | 159 | 160 | ## 3.hostNetWork方式下，Doris服务连续性验证 161 | 162 | 手动重启BE，重启过程中提示backend does not exist or not alive，启动后恢复正常 163 | 164 | ![img](https://intranetproxy.alipay.com/skylark/lark/0/2021/png/13456375/1640153928156-97b8b4f4-09c2-4509-a711-ee66b27d2af8.png) 165 | 166 | 167 | 168 | 手动重启FE，重启过程中提示Can't connect to MySQL server，启动后恢复正常 169 | 170 | ![img](https://intranetproxy.alipay.com/skylark/lark/0/2021/png/13456375/1640154218000-8d2d396a-1f4a-4964-a05c-542ef7a92e64.png) 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | --------------------------------------------------------------------------------