├── README.md ├── advanced.md ├── exoplanets.md └── style.css /README.md: -------------------------------------------------------------------------------- 1 | # 1. Labels # 2 | 3 | `oc label node node1 env=dev`
4 | `oc label node node2 env=prod`
5 | `oc label node node1 env=test --overwrite`
6 | `oc get nodes --show-labels`
7 | `oc label node node1 env-`
8 | 9 | # 2. Node Selector # 10 | 11 | `oc patch deployment/myapp \`
12 |   `--patch '{"spec":{"template":{"spec":{"nodeSelector":{"env":"dev"}}}}}'` 13 | 14 | `oc adm new-project demo --node-selector "env=dev"`
15 | `oc annotate namespace demo openshift.io/node-selector "env=dev" --overwrite` 16 | 17 | `oc patch namespace demo \`
18 |   `--patch '{"metadata":{"annotations":{"openshift.io/node-selector": "env=dev"}}}'` 19 | 20 | # 3. Taints # 21 | 22 | `oc adm taint nodes node1 dedicated=foo:NoSchedule -o json --dry-run=client |\`
23 |   `jq .spec.taints` 24 | 25 | ``` 26 | [ 27 | { 28 | "effect": "NoSchedule", 29 | "key": "dedicated", 30 | "value": "foo" 31 | } 32 | ] 33 | ``` 34 | 35 | `oc describe node node1 | grep -A2 Taint` 36 | 37 | ``` 38 | Taints: dedicated=foo:NoSchedule 39 | test=foo:NoSchedule 40 | 41 | ``` 42 | delete node taint
43 | `oc adm taint node node1 dedicated-` 44 | 45 | delete the other key
46 | `oc adm taint node node1 test-` 47 | 48 | list current node taints on the master node
49 | `oc get node/master0.nms.cp.fyre.ibm.com -o json |jq .spec.taints` 50 | ``` 51 | [ 52 | { 53 | "effect": "NoSchedule", 54 | "key": "node-role.kubernetes.io/master" 55 | } 56 | ] 57 | ``` 58 | Add a new taint foo:NoSchedule
59 | `oc adm taint nodes master0.nms.cp.fyre.ibm.com foo:NoSchedule` 60 | 61 | list the current taints, you can see the added taint
62 | `oc get node/master0.nms.cp.fyre.ibm.com -o json |jq .spec.taints` 63 | ``` 64 | [ 65 | { 66 | "effect": "NoSchedule", 67 | "key": "foo" 68 | }, 69 | { 70 | "effect": "NoSchedule", 71 | "key": "node-role.kubernetes.io/master" 72 | } 73 | ] 74 | ``` 75 | lets delete the taint
76 | `oc adm taint nodes master0.nms.cp.fyre.ibm.com foo-` 77 | 78 | list the current taints - should be back to orignal starting point
79 | `oc get node/master0.nms.cp.fyre.ibm.com -o json |jq .spec.taints` 80 | ``` 81 | [ 82 | { 83 | "effect": "NoSchedule", 84 | "key": "node-role.kubernetes.io/master" 85 | } 86 | ] 87 | ``` 88 | 89 | # 4. OAuth # 90 | 91 | Install htpasswd command line utility
92 | `sudo yum install httpd-tools` 93 | 94 | Create a new file for htpass with users and their passwords
95 | `htpasswd -c -b /tmp/htpass user1 password1`
96 | `htpasswd -b /tmp/htpass user2 password2` 97 | 98 | Copy existing oauth config to a file for editing
99 | `oc get oauth cluster -o yaml > /tmp/oauth.yaml` 100 | 101 | Add httpasswd to oauth.yaml 102 | ``` 103 | spec: 104 | identityProviders: 105 | - name: localusers 106 | type: HTPasswd 107 | mappingMethod: claim 108 | htpasswd: 109 | fileData: 110 | name: htpass-secret 111 | ``` 112 | Create a new secret which will hold the users and password file
113 | `oc create secret generic htpass-secret \`
114 |   `--from-file htpasswd=/tmp/htpass -n openshift-config` 115 | 116 | Now merge/replace existing oauth with edited version
117 | `oc replace -f /tmp/oauth.yaml` 118 | 119 | Watch the pods being replaced for the new config to take into affect
120 | `oc get po -w -n openshift-authentication` 121 | 122 | Lets delete user2 from htpass file
123 | `htpasswd -D /tmp/htpass user2` 124 | 125 | Update the secret with new htpass
126 | `oc set data secret/htpass-secret --from-file htpasswd=/tmp/htpass -n openshift-config` 127 | 128 | delete the user and identity from the system
129 | `oc delete user user2`
130 | `oc delete identity localusers:user2` 131 | 132 | delete all users and identities defined in OAuth
133 | `oc delete user --all`
134 | `oc delete identity --all` 135 | 136 | # 5. Users, Groups, and Authentication # 137 | 138 | Assign cluster admin role to user
139 | `oc adm policy add-cluster-role-to-user cluster-admin user-name` 140 | 141 | Assign role at project/namespace level
142 | `oc policy add-role-to-user role-name user-name -n project-name ` 143 | 144 | Get current clusterrolebindings configured for self-provisioners
145 | `oc get clusterrolebindings -o wide |grep -E "NAME|self-provisioners"` 146 | 147 | Describe the clusterrolebindings and clusterrole
148 | `oc describe clusterrolebindings self-provisioners`
149 | `oc describe clusterrole self-provisioner`
150 | 151 | > **clusterrolebindings = self-provisioners**
152 | > **clusterrole = self-provisioner** 153 | 154 | Remove self-provisioner role from system such that authenticated users can't create projects
155 | `oc adm policy remove-cluster-role-from-group self-provisioner \`
156 |   `system:authenticated:oauth` 157 | 158 | Restore self-provisioners back to cluster as original
159 | `oc adm policy add-cluster-role-to-group \`
160 |   `--rolebinding-name self-provisioners self-provisioner system:authenticated:oauth` 161 | 162 | Creating new groups
163 | `oc adm group new dev-users dev1 dev2`
164 | `oc adm group new qa-users qa1 qa2` 165 | 166 | Assign roles at namespace/project level. You will need admin role to assign users.
167 | `oc policy add-role-to-group edit dev-users -n namespace`
168 | `oc policy add-role-to-group view qa-users -n namespace`
169 | `oc policy add-role-to-user admin user1 -n namespace` 170 | 171 | Get all the rolebindings for the current namespace
172 | `oc get rolebindings -o wide` 173 | 174 | # 6. Remove kubeadmin from the system # 175 | 176 | Make sure you have assigned cluster-admin to someone else before doing this!
177 | `oc adm policy add-cluster-role-to-user cluster-admin user-name` 178 | 179 | Instead of removing/deleting the user, we will remove the password from the system.
180 | `oc delete secret kubeadmin -n kube-system` 181 | 182 | # 7. Secrets and ConfigMaps # 183 | 184 | Create secrets from literals and apply to a deployment
185 | `oc create secret generic secretname --from-literal key1=value1 \`
186 |   `--from-literal key2=value2`
187 | `oc set env deployment/hello --from secret/secretname` 188 | 189 | Mount the secret file into the pod filesystem
190 | `oc set volume deployment/demo --add --type secret --secret-name demo-secret \`
191 |   `--mount-path /app-secrets` 192 | 193 | Example of setting env variables that contain sensitive data
194 | `oc create secret generic mysql \`
195 |   `--from-literal user=dba \`
196 |   `--from-literal password=redhat123 \`
197 |   `--from-literal database=test \`
198 |   `--from-literal hostname=mysql` 199 | 200 | `oc new-app --name mysql \`
201 |   `--docker-image registry.access.redhat.com/rhscl/mysql-57-rhel7:5.7-47` 202 | 203 | `oc set env deployment/mysql --prefix MYSQL_ --from secret/mysql` 204 | 205 | To use a private image in quay.io using secrets stored in files
206 | `podman login -u quay-username quay.io` 207 | 208 | `oc create secret generic quayio \`
209 |   `--from-file .dockerconfigjson=${XDG_RUNTIME_DIR}/containers/auth.json \`
210 |   `--type kubernetes.io/dockerconfigjson` 211 | 212 | `oc secrets link default quayio --for pull` 213 | 214 | `oc import-image php --from quay.io/quay-username/php-70-rhel7 --confirm` 215 | 216 | # 8. Secure Routes # 217 | 218 | Using openssl generate a private key and a public key
219 | `openssl req -x509 -newkey rsa:2048 -nodes -keyout cert.key -out cert.crt \`
220 |   `-subj "/C=US/ST=FL/L=Tampa/O=IBM/CN=*.apps.acme.com" -days 365` 221 | 222 | If additional subjectAtlName DNS extensions are needed
223 | `openssl req -x509 -newkey rsa:2048 -nodes -keyout cert.key -out cert.crt \`
224 |   `-subj "/C=US/ST=FL/L=Tampa/O=IBM/CN=*.apps.acme.com" -days 365 \`
225 |   `-addext "subjectAltName = DNS:host1.apps.acme.com,DNS:host2.apps.acme.com"` 226 | 227 | 228 | Using the key and cert create a TLS secret
229 | `oc create secret tls demo-certs --cert cert.crt --key cert.key` 230 | 231 | Mount the tls certs into the pod (using deployment)
232 | `oc set volume deployment/demo --add --type=secret --secret-name demo-tls \`
233 |   `--mount-path /usr/local/etc/ssl/certs --name tls-mount` 234 | 235 | Now create a passthrough route
236 | `oc create route passthrough demo-https --service demo-https --port 8443 \`
237 |   `--hostname demo-https.apps.ocp4.example.com` 238 | 239 | Using edge route with same certs
240 | `oc create route edge demo-https --service api-frontend --hostname api.apps.acme.com \`
241 |   `--key cert.key --cert cert.crt` 242 | 243 | Export the router cert in case we need to use it as a ca-cert
244 | `oc extract secrets/router-ca --keys tls.crt -n openshift-ingress-operator --to /tmp/` 245 | 246 | # 9. Security Contexts (SCC) and ServiceAccounts # 247 | 248 | Get the current SCC roles defined
249 | `oc get scc` 250 | 251 | Get details of scc anyuid
252 | `oc describe scc anyuid` 253 | 254 | Create a service account in the current project and assign the anyuid priviledges to the service account.
255 | `oc create serviceaccount svc-name`
256 | `oc adm policy add-scc-to-user anyuid -z svc-name -n namespace`
257 | `oc set serviceaccount deployment/demo svc-name` 258 | 259 | review the scc priviledges needed for a pod
260 | `oc get po/podname-756ff-9cjbj -o yaml | oc adm policy scc-subject-review -f -` 261 | 262 | Example of gitlab being run as anyuid using serviceaccount
263 | `oc new-app --name gitlab --docker-image quay.io/redhattraiing/gitlab-ce:8.4.3-ce.0`
264 | `oc get po/gitlab-6c5b5c5d55-gzkct -o yaml | oc adm policy scc-subject-review -f -`
265 | `oc create sa gitlab-sa`
266 | `oc adm policy add-scc-to-user anyuid -z gitlab-sa`
267 | `oc set sa deployment/gitlab gitlab-sa` 268 | 269 | # 10. Limits, Quotas, and LimitRanges # 270 | 271 | Check the node resources
272 | `oc describe node node1` 273 | 274 | Set resources on a deployment. Request limits are how much each request is allowed, and limit is the max allowed
275 | `oc set resources deployment hello-world-nginx \`
276 |   `--requests cpu=10m,memory=20Mi --limits cpu=180m,memory=100Mi` 277 | 278 | Scheduling decisions are made based on the request to ensure that a node has enough capacity available to meet the requested value. If a container specifies limits, but omits requests, the requests are defaulted to the limits. A container is not able to exceed the specified limit on the node. The enforcement of limits is dependent upon the compute resource type. If a container makes no request or limit, the container is scheduled to a node with no resource guarantees. 279 | 280 | Quota is project level resources available
281 | `oc create quota dev-quota --hard services=10,cpu=1300m,memory=1.5Gi` 282 | 283 | Quota including requests and limits at namespace level. Remember if you don't provide requests values, then same limits values are used for both requests and limits
284 | `oc create quota my-quota --hard pods=10,requests.cpu=2,requests.memory=1Gi,limits.cpu=4,limits.memory=2Gi` 285 | 286 | Cluster Quota is resources available across multiple projects
287 | `oc create clusterquota env-qa \`
288 |   `--project-annotation-selector.openshift.io/requester=qa \`
289 |   `--hard pods=12,secrets=20,services=5` 290 | 291 | Show all project annotations and labels
292 | `oc describe namespace demo` 293 | 294 | Limit ranges in a yaml file are defined as follows 295 | ``` 296 | apiVersion: "v1" 297 | kind: "LimitRange" 298 | metadata: 299 | name: "nms-limits" 300 | spec: 301 | limits: 302 | - type: "Pod" 303 | max: 304 | cpu: "2" 305 | memory: "1Gi" 306 | min: 307 | cpu: "200m" 308 | memory: "6Mi" 309 | - type: "Container" 310 | max: 311 | cpu: "2" 312 | memory: "1Gi" 313 | min: 314 | cpu: "100m" 315 | memory: "4Mi" 316 | default: # default if not specified in the Pod spec 317 | cpu: "300m" 318 | memory: "200Mi" 319 | defaultRequest: # default request if not specified in the Pod spec 320 | cpu: "200m" 321 | memory: "100Mi" 322 | ``` 323 | oc create -f limitranges.yaml -n net-ingress 324 | 325 | oc describe limits nms-limits -n net-ingress 326 | ``` 327 | Name: nms-limits 328 | Namespace: net-ingress 329 | Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio 330 | ---- -------- --- --- --------------- ------------- ----------------------- 331 | Pod cpu 200m 2 - - - 332 | Pod memory 6Mi 1Gi - - - 333 | Container cpu 100m 2 200m 300m - 334 | Container memory 4Mi 1Gi 100Mi 200Mi - 335 | 336 | ``` 337 | # 11. Scaling and AutoScaler # 338 | 339 | `oc scale --replicas 3 deployment/demo` 340 | 341 | `oc autoscale dc/demo --min 1 --max 10 --cpu-percent 80` 342 | 343 | `oc get hpa` 344 | 345 | # 12. Readiness and Liveness # 346 | 347 | - Readiness: How long before container is ready to serve requests. If probe fails Openshift removes the IP from the services endpoints, such that no traffic is forwarded to the container. 348 | - Liveness: Probe to test if container is in healthy state? If probe fails, Openshift kills the conatiner and redeploys. 349 | 350 | Probes have the following configuration settings: 351 | - __initialDelaySeconds__: required, default value=0, how long to wait to start the probe after container starts 352 | - __timeoutSeconds__: required, default value=1, how long to wait for the probe to finish 353 | - __periodSeconds__: not required, default value=10, how often to probe 354 | - __successThreshold__: not required, default value=1, min consecutive successes after failure to consider success 355 | - __failureThreshold__: not required, default value=3, min consecutive failures to be considered a failure 356 | 357 | Examples of setting probes:
358 | `oc set probe dc/demo --readiness --initial-delay-seconds 20` 359 | 360 | `oc set probe dc/webapp --readiness --get-url=http://:8080/healthz \`
361 |   `--period-seconds 10 --timeout-seconds 1 --initial-delay-seconds 30` 362 | 363 | `oc set probe dc/mq --liveness --open-tcp 1414 --period-seconds 3 \`
364 |   `--timeout-seconds 2 --failure-threshold 3 --initial-delay-seconds 30` 365 | 366 | `oc set probe dc/ace --liveness --get-url http://:7600/healthz \`
367 |   `--initial-delay-seconds 30 --period-seconds 10 --dry-run=client -o json | \`
368 |     `jq .spec.template.spec.containers[].livenessProbe` 369 | ``` 370 | { 371 | "httpGet": { 372 | "path": "/healthz", 373 | "port": 7600, 374 | "scheme": "HTTP" 375 | }, 376 | "initialDelaySeconds": 30, 377 | "periodSeconds": 10 378 | } 379 | ``` 380 | `oc set probe dc/mq --liveness --open-tcp 1414 --period-seconds 3 \`
381 |   `--timeout-seconds 2 --failure-threshold 3 --initial-delay-seconds 30 \`
382 |   `--dry-run=client -o json | jq .spec.template.spec.containers[].livenessProbe` 383 | ``` 384 | { 385 | "tcpSocket": { 386 | "port": 1414 387 | }, 388 | "initialDelaySeconds": 30, 389 | "timeoutSeconds": 2, 390 | "periodSeconds": 3, 391 | "failureThreshold": 3 392 | } 393 | ``` 394 | 395 | Removing probes:
396 | `oc set probe dc/demo --remove --readiness --liveness` 397 | 398 | # 13. Image Registry # 399 | 400 | Built in registry is housed in openshift-image-registry namespace
401 | `oc -n openshift-image-registry get svc` 402 | 403 | cluster wide svc naming scheme is service-name.namespace.svc
404 | `image-registry.openshift-image-registry.svc` 405 | 406 | using stored images from openshift project 407 | `oc get images`
408 | `oc get is -n openshift | grep httpd` 409 | 410 | `oc new-app --name nms --docker-image \`
411 |   `image-registry.openshift-image-registry.svc:5000/openshift/httpd:latest` 412 | 413 | OR 414 | 415 | `oc new-app --name nms \`
416 |   `image-registry.openshift-image-registry.svc:5000/openshift/httpd:latest` 417 | 418 | OR 419 | 420 | `oc new-app --name nms --image \`
421 |   `image-registry.openshift-image-registry.svc:5000/openshift/httpd:latest` 422 | 423 | create a service
424 | `oc expose deployment/nms --port 8080 --target-port 8080` 425 | 426 | # 14. Deployment Strategy # 427 | 428 | Using a docker strategy using local build directory
429 | `oc new-app --strategy docker --binary --name myapp`
430 | `oc start-build myapp --from-dir . --follow`
431 | `oc expose deployment myapp --target-port 8080 --port 80`
435 | `oc new-app --binary --image-stream nodejs --name nodejs`
436 | `oc start-build nodejs --from-dir . --follow`
437 | `oc expose svc/nodejs` 438 | 439 | # 15. General Troubleshooting # 440 | 441 | Following are high level issues that are highlighted in the course 442 | + Limits or Quotas are exceeded 443 | + Taints on node (NoSchedule) 444 | + Route is misconfigured 445 | + Permissions missing, assign SA to deployment 446 | 447 | Useful commands. 448 | ``` 449 | oc logs pod-name 450 | oc adm top pods 451 | oc adm top nodes 452 | oc get events --field-selector type=Warning 453 | oc debug pod 454 | oc debug node/nodename 455 | oc adm taint node node-name key- 456 | ``` 457 | 458 | Connect to a service using a debug image
459 | `oc get svc/mysql -o jsonpath="{.spec.clusterIP}{'\n'}"`
460 | `oc debug -t deployment/app --image registry.access.redhat.com/ubi8/ubi:8.0`
461 | `curl -v telnet://:` 462 | 463 | After fixing the deployment (yaml or the issue at hand), the deployment might have timed out by the time the issue was fixed. In order to push the deployment a new "rollout" might be needed.
464 | 465 | `oc rollout latest dc/demo` 466 | 467 | or 468 | 469 | `oc rollout latest deployment/demo` 470 | 471 | An example app was deployed where the endpoint wasn't working. After troubleshooting it was found the name of the service was defined with app tagname was mis spelled. Had to fix the typo to get the service working again. 472 | 473 | Another case was where instead of Route an Ingress with the wrong hostname was defined. When you delete the route, the route was re-created by openshift with a different route-name but still having the wrong hostname. To fix, we had to edit the Ingress configuration with the correct hostname. Once that was done, the correct route was generated and started working! 474 | -------------------------------------------------------------------------------- /advanced.md: -------------------------------------------------------------------------------- 1 | # Upgrade Cluster to a newer Channel # 2 | 3 | Downgrading is not supported! So be very careful if you decide to do this! 4 | 5 | Test your changes first 6 | 7 | `oc patch clusterversion/version --type merge --patch '{"spec":{"channel":"stable-4.7"}}' --dry-run=client -o json | jq .spec.channel` 8 | 9 | ``` 10 | oc patch clusterversion version --type="merge" -p '{"spec":{"channel":"stable-4.6"}}' 11 | 12 | oc adm upgrade 13 | 14 | oc adm upgrade --to-latest=true 15 | ``` 16 | -------------------------------------------------------------------------------- /exoplanets.md: -------------------------------------------------------------------------------- 1 | # 1. Install DB Cockroach Operator 2 | Create a namespace (project). Install operator into this namespace. Click on Operators -> OperatorHub 3 | Select Database, to display the list of database operators. In the filters select "Certified" operators and then click on Cockroach Operator. 4 | Install the operator by selecting the following options: "Update Channel = stable" and "Installation Mode = A specific namespace on the cluster" 5 | and enter the namespace above and then hit install. 6 | 7 | # 2. Create DB cluster 8 | Once operator is successfully installed, click on Operators -> Installed Operators and click on the installed operator. 9 | Next select create instance of the db cluster. Select all defaults except turn off TLS and name as crdb-example, hit create. 10 | Wait for the db cluster (3 nodes) to start. 11 | 12 | `oc get po -w` 13 | 14 | # 3. Check the service name 15 | check the name of the svc that contains the ClusterIP for db-cluster 16 | 17 | `oc get svc` 18 | 19 | Copy the cluster name `crdb-example-public` and use it for DB_HOST env variable 20 | 21 | # 4. Create the deployment 22 | ``` 23 | apiVersion: apps/v1 24 | kind: Deployment 25 | metadata: 26 | annotations: 27 | labels: 28 | app: exoplanets 29 | name: exoplanets 30 | spec: 31 | replicas: 1 32 | selector: 33 | matchLabels: 34 | app: exoplanets 35 | template: 36 | metadata: 37 | labels: 38 | app: exoplanets 39 | spec: 40 | containers: 41 | - env: 42 | - name: DB_HOST 43 | value: crdb-example-public 44 | - name: DB_NAME 45 | value: postgres 46 | - name: DB_PORT 47 | value: "26257" 48 | - name: DB_USER 49 | value: root 50 | image: quay.io/redhattraining/exoplanets:v1.0 51 | name: exoplanets 52 | ports: 53 | - containerPort: 8080 54 | protocol: TCP 55 | readinessProbe: 56 | httpGet: 57 | path: /healthz 58 | port: 8080 59 | ``` 60 | `oc create -f exoplanets.yaml` 61 | 62 | The above can also be accomplished by running the following commands 63 | ``` 64 | oc new-app --name exoplanets --docker-image quay.io/redhattraining/exoplanets:v1.0 \ 65 | -e DB_HOST=crdb-example-public -e DB_PORT=26257 -e DB_USER=root -e DB_NAME=postgres 66 | 67 | oc set probe deployment/exoplanets --readiness --get-url http://:8080/healthz 68 | ``` 69 | 70 | # 5. create svc and route 71 | 72 | `oc expose deployment/exoplanets --port 8080` 73 | 74 | `oc expose svc/exoplanets --port 8080` 75 | 76 | # 6. Test out the route using a browser 77 | -------------------------------------------------------------------------------- /style.css: -------------------------------------------------------------------------------- 1 | 24 | --------------------------------------------------------------------------------