├── LICENSE ├── README.md ├── deploy.sh ├── deployment ├── service.yaml └── statefulset.yaml ├── environments └── default.yaml └── templates ├── service.yaml └── statefulset.yaml /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 Alex Jones 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # _deprecated_ kubernetes-nifi-cluster _please use https://github.com/AlexsJones/nifi_ 2 | 3 | ![forks](https://img.shields.io/github/forks/AlexsJones/kubernetes-nifi-cluster.svg) 4 | ![stars](https://img.shields.io/github/stars/AlexsJones/kubernetes-nifi-cluster.svg) 5 | ![license](https://img.shields.io/github/license/AlexsJones/kubernetes-nifi-cluster.svg) 6 | ![issues](https://img.shields.io/github/issues/AlexsJones/kubernetes-nifi-cluster.svg) 7 | 8 | 9 | A nifi cluster running in kubernetes 10 | 11 | ## Helm chart now available 12 | 13 | This repo is now being deprecated in favour of a Helm compatible [chart](https://github.com/AlexsJones/nifi) 14 | 15 | _I am going to appeal to add it into helm/charts as soon as my dependent zookeeper chart is added_ 16 | 17 | Thank you! 18 | 19 | ### Prerequisites 20 | 21 | - This example is using Google Cloud Platform persistent volumes for its backing store (easy to convert to AWS). 22 | 23 | - There is no longer a requirement to install [vortex](https://github.com/AlexsJones/vortex) for interpolation. 24 | - For setting up the environment run: 25 | 26 | ``` 27 | docker run -v $PWD:/tmp tibbar/vortex:v1 -template /tmp/templates -output /tmp/deployment -varpath /tmp/environments/default.yaml 28 | ``` 29 | 30 | This will generate your default environment 31 | 32 | - Requires [zookeeper](https://github.com/AlexsJones/kubernetes-zookeeper-cluster) 33 | ``` 34 | # Assumes you've checked it out next to the nifi cluster... 35 | cd kubernetes-zookeeper-cluster 36 | docker run -v $PWD:/tmp tibbar/vortex:v1 -template /tmp/templates -output /tmp/deployment -varpath /tmp/environments/small.yaml 37 | kubectl create -f deployment/ 38 | cd ../kubernetes-nifi-cluster 39 | ``` 40 | 41 | Now zookeeper is setup with three nodes on the zk namespace you are ready! 42 | 43 | ## Deploy Nifi 44 | 45 | _Make sure zk is running to avoid headaches_ 46 | 47 | ``` 48 | # Please create the nifi namespace first 49 | # Ensure you have run docker vortex build step mentioned above^ 50 | 51 | kubectl create ns nifi 52 | 53 | kubectl create -f deployment/ 54 | ``` 55 | 56 | Once running you should see... 57 | ``` 58 | kubectl get pods -n nifi 59 | NAME READY STATUS RESTARTS AGE 60 | nifi-0 1/1 Running 0 25m 61 | nifi-1 1/1 Running 0 25m 62 | nifi-2 1/1 Running 0 25m 63 | 64 | kubectl get pvc -n nifi 65 | NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE 66 | contentrepository-nifi-0 Bound pvc-c00b39d5-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d 67 | contentrepository-nifi-1 Bound pvc-c0116c25-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d 68 | contentrepository-nifi-2 Bound pvc-c019d7ee-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d 69 | databaserepository-nifi-0 Bound pvc-c00a3682-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d 70 | databaserepository-nifi-1 Bound pvc-c00f87a8-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d 71 | databaserepository-nifi-2 Bound pvc-c017dbe4-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d 72 | flowfilerepository-nifi-0 Bound pvc-c0096aac-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d 73 | flowfilerepository-nifi-1 Bound pvc-c00df6bb-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d 74 | flowfilerepository-nifi-2 Bound pvc-c016020d-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d 75 | provenancerepository-nifi-0 Bound pvc-c008b6bd-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d 76 | provenancerepository-nifi-1 Bound pvc-c0132c86-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d 77 | provenancerepository-nifi-2 Bound pvc-c01aec6b-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d 78 | ``` 79 | 80 | --- 81 | 82 | ## UI 83 | 84 | Port forward to the UX: 85 | ``` 86 | kubectl port-forward nifi-0 8080:8080 -n nifi 87 | ``` 88 | 89 | `http://:8080/nifi/` 90 | 91 | ## Scaling 92 | 93 | `kubectl scale --replicas sts/nifi` 94 | 95 | ## Configuration 96 | 97 | The `environments/` folder can have new files added and used with `build_environment.sh ` and those values interpolated 98 | For more informatino on this please see [here](https://github.com/AlexsJones/vortex/blob/master/README.md) 99 | 100 | ### Production readiness 101 | 102 | A checklist for consideration I would think about before you want to run this in prod 103 | - Use an identity aware proxy infront of the load balancer 104 | - Recovery procedure from the PVC's 105 | - Scaling, liveliness & readiness probes 106 | -------------------------------------------------------------------------------- /deploy.sh: -------------------------------------------------------------------------------- 1 | kubectl create ns nifi 2 | kubectl create -f deployment/service.yaml --namespace=nifi 3 | kubectl create -f deployment/statefulset.yaml --namespace=nifi 4 | -------------------------------------------------------------------------------- /deployment/service.yaml: -------------------------------------------------------------------------------- 1 | kind: Service 2 | apiVersion: v1 3 | metadata: 4 | name: nifi-http 5 | spec: 6 | type: LoadBalancer 7 | selector: 8 | app: nifi 9 | ports: 10 | - protocol: TCP 11 | port: 8080 12 | targetPort: 8080 13 | name: nifi 14 | 15 | --- 16 | 17 | kind: Service 18 | apiVersion: v1 19 | metadata: 20 | name: nifi 21 | spec: 22 | clusterIP: None 23 | selector: 24 | app: nifi 25 | ports: 26 | - protocol: TCP 27 | port: 8081 28 | targetPort: 8081 29 | name: nifi-listen-http 30 | - protocol: TCP 31 | port: 2881 32 | targetPort: 2881 33 | name: nifi-site-protocol 34 | - protocol: TCP 35 | port: 2882 36 | targetPort: 2882 37 | name: nifi-node-protocol 38 | -------------------------------------------------------------------------------- /deployment/statefulset.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1beta1 2 | kind: StatefulSet 3 | metadata: 4 | name: nifi 5 | spec: 6 | serviceName: nifi 7 | replicas: 3 8 | podManagementPolicy: Parallel 9 | updateStrategy: 10 | type: RollingUpdate 11 | template: 12 | metadata: 13 | labels: 14 | app: nifi 15 | spec: 16 | affinity: 17 | podAntiAffinity: 18 | requiredDuringSchedulingIgnoredDuringExecution: 19 | - labelSelector: 20 | matchExpressions: 21 | - key: "app" 22 | operator: In 23 | values: 24 | - nifi 25 | topologyKey: "kubernetes.io/hostname" 26 | containers: 27 | - name: nifi 28 | command: ["/bin/sh","-c"] 29 | args: ['sed -e "s/^search/& $(dnsdomainname)/" /etc/resolv.conf > /tmp/resolv.conf;cat /tmp/resolv.conf > /etc/resolv.conf; cd /opt/nifi && ./start_nifi.sh'] 30 | image: xemuliam/nifi:latest 31 | imagePullPolicy: Always 32 | ports: 33 | - containerPort: 8080 34 | - containerPort: 8081 35 | - containerPort: 8443 36 | - containerPort: 2882 37 | env: 38 | - name: ZK_NODES_LIST 39 | # Requires zookeeper running 40 | value: "zk-0.zk-hs.zk.svc.cluster.local,zk-1.zk-hs.zk.svc.cluster.local,zk-2.zk-hs.zk.svc.cluster.local" 41 | - name: ZK_MONITOR_PORT 42 | value: "2888" 43 | - name: IS_CLUSTER_NODE 44 | value: "yes" 45 | - name: BANNER_TEXT 46 | value: "Nifi Cluster" 47 | -------------------------------------------------------------------------------- /environments/default.yaml: -------------------------------------------------------------------------------- 1 | replicas: 3 2 | image: xemuliam/nifi:latest 3 | -------------------------------------------------------------------------------- /templates/service.yaml: -------------------------------------------------------------------------------- 1 | kind: Service 2 | apiVersion: v1 3 | metadata: 4 | name: nifi-http 5 | spec: 6 | type: LoadBalancer 7 | selector: 8 | app: nifi 9 | ports: 10 | - protocol: TCP 11 | port: 8080 12 | targetPort: 8080 13 | name: nifi 14 | 15 | --- 16 | 17 | kind: Service 18 | apiVersion: v1 19 | metadata: 20 | name: nifi 21 | spec: 22 | clusterIP: None 23 | selector: 24 | app: nifi 25 | ports: 26 | - protocol: TCP 27 | port: 8081 28 | targetPort: 8081 29 | name: nifi-listen-http 30 | - protocol: TCP 31 | port: 2881 32 | targetPort: 2881 33 | name: nifi-site-protocol 34 | - protocol: TCP 35 | port: 2882 36 | targetPort: 2882 37 | name: nifi-node-protocol 38 | -------------------------------------------------------------------------------- /templates/statefulset.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1beta1 2 | kind: StatefulSet 3 | metadata: 4 | name: nifi 5 | spec: 6 | serviceName: nifi 7 | replicas: {{.replicas}} 8 | podManagementPolicy: Parallel 9 | updateStrategy: 10 | type: RollingUpdate 11 | template: 12 | metadata: 13 | labels: 14 | app: nifi 15 | spec: 16 | affinity: 17 | podAntiAffinity: 18 | requiredDuringSchedulingIgnoredDuringExecution: 19 | - labelSelector: 20 | matchExpressions: 21 | - key: "app" 22 | operator: In 23 | values: 24 | - nifi 25 | topologyKey: "kubernetes.io/hostname" 26 | containers: 27 | - name: nifi 28 | command: ["/bin/sh","-c"] 29 | args: ['sed -e "s/^search/& $(dnsdomainname)/" /etc/resolv.conf > /tmp/resolv.conf;cat /tmp/resolv.conf > /etc/resolv.conf; cd /opt/nifi && ./start_nifi.sh'] 30 | image: {{.image}} 31 | imagePullPolicy: Always 32 | ports: 33 | - containerPort: 8080 34 | - containerPort: 8081 35 | - containerPort: 8443 36 | - containerPort: 2882 37 | env: 38 | - name: ZK_NODES_LIST 39 | # Requires zookeeper running 40 | value: "zk-0.zk-hs.zk.svc.cluster.local,zk-1.zk-hs.zk.svc.cluster.local,zk-2.zk-hs.zk.svc.cluster.local" 41 | - name: ZK_MONITOR_PORT 42 | value: "2888" 43 | - name: IS_CLUSTER_NODE 44 | value: "yes" 45 | - name: BANNER_TEXT 46 | value: "Nifi Cluster" 47 | --------------------------------------------------------------------------------