├── .gitignore ├── README.md ├── cr-use-psp-static.yml ├── cr-use-psp.yml ├── crb-use-psp-static.yml ├── pod-basic.yml ├── pod-privileged.yml ├── pod-with-psp-access.yml ├── psp-nopriv.yml ├── psp-static.yml ├── rb-nonadmin-edit.yml └── rb-nonprivileged.yml /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Automating Kubernetes Security 2 | 3 | This is a walkthrough guide for the live demo performed during the `Automating Kubernetes Security` webinar. 4 | 5 | ## Cluster Setup 6 | Set up a Kubernetes cluster with `kubeadm`, or use an existing one. 7 | 8 | Demo environment info: 9 | - OS: `Ubuntu 18.04 Bionic Beaver LTS` 10 | - Container Runtime: `Docker 19.03.12` 11 | - Kubernetes version: `1.19.2` 12 | - Networking Plugin: `Calico` 13 | 14 | ## Set Up a Namespace and Non-Admin ServiceAccount to Use for Testing 15 | Create a Namespace. 16 | 17 | ``` 18 | kubectl create ns development 19 | ``` 20 | 21 | Create the ServiceAccount. 22 | 23 | ``` 24 | kubectl create sa nonadmin -n development 25 | ``` 26 | 27 | Create a RoleBinding allowing the account to edit objects in the `development` Namespace. 28 | 29 | ``` 30 | vi rb-nonadmin-edit.yml 31 | ``` 32 | 33 | ``` 34 | apiVersion: rbac.authorization.k8s.io/v1 35 | kind: RoleBinding 36 | metadata: 37 | name: rb-nonadmin-edit 38 | namespace: development 39 | roleRef: 40 | kind: ClusterRole 41 | name: edit 42 | apiGroup: rbac.authorization.k8s.io 43 | subjects: 44 | - kind: ServiceAccount 45 | name: nonadmin 46 | namespace: development 47 | ``` 48 | 49 | ``` 50 | kubectl create -f rb-nonadmin-edit.yml --save-config 51 | ``` 52 | 53 | Verify that you can run commands as the `nonadmin` user. You should see the message `No resources found in development namespace.` since no Pods have been created in the Namespace. 54 | 55 | ``` 56 | kubectl --as=system:serviceaccount:development:nonadmin get pods -n development 57 | ``` 58 | 59 | ## Create a PodSecurityPolicy 60 | Create a new PodSecurityPolicy that will prevent pods from using privileged mode. 61 | 62 | ``` 63 | vi psp-nopriv.yml 64 | ``` 65 | 66 | ``` 67 | apiVersion: policy/v1beta1 68 | kind: PodSecurityPolicy 69 | metadata: 70 | name: psp-nopriv 71 | spec: 72 | privileged: false 73 | runAsUser: 74 | rule: RunAsAny 75 | fsGroup: 76 | rule: RunAsAny 77 | seLinux: 78 | rule: RunAsAny 79 | supplementalGroups: 80 | rule: RunAsAny 81 | volumes: 82 | - configMap 83 | - downwardAPI 84 | - emptyDir 85 | - persistentVolumeClaim 86 | - secret 87 | - projected 88 | ``` 89 | 90 | Create the PodSecurityPolicy. 91 | 92 | ``` 93 | kubectl create -f psp-nopriv.yml --save-config 94 | ``` 95 | 96 | ## Set Up RBAC to Authorize the Use of the PodSecurityPolicy for a New ServiceAccount 97 | Create a ClusterRole that allows the use of our PodSecurityPolicy. 98 | 99 | ``` 100 | vi cr-use-psp.yml 101 | ``` 102 | 103 | ``` 104 | apiVersion: rbac.authorization.k8s.io/v1 105 | kind: ClusterRole 106 | metadata: 107 | name: cr-use-psp 108 | rules: 109 | - apiGroups: ['policy'] 110 | resources: ['podsecuritypolicies'] 111 | verbs: ['use'] 112 | resourceNames: 113 | - psp-nopriv 114 | ``` 115 | 116 | Create the ClusterRole. 117 | 118 | ``` 119 | kubectl create -f cr-use-psp.yml --save-config 120 | ``` 121 | 122 | Create a ServiceAccount. 123 | 124 | ``` 125 | kubectl create sa nonprivileged -n development 126 | ``` 127 | 128 | Create a RoleBinding that allows the `nonprivileged` ServiceAccount to use the PodSecurityPolicy in the `default` Namespace. 129 | 130 | ``` 131 | vi rb-nonprivileged.yml 132 | ``` 133 | 134 | ``` 135 | apiVersion: rbac.authorization.k8s.io/v1 136 | kind: RoleBinding 137 | metadata: 138 | name: rb-nonprivileged 139 | namespace: development 140 | roleRef: 141 | kind: ClusterRole 142 | name: cr-use-psp 143 | apiGroup: rbac.authorization.k8s.io 144 | subjects: 145 | - kind: ServiceAccount 146 | name: nonprivileged 147 | namespace: development 148 | ``` 149 | 150 | Create the RoleBinding. 151 | 152 | ``` 153 | kubectl create -f rb-nonprivileged.yml --save-config 154 | ``` 155 | 156 | ## Turn on the PodSecurityPolicy Admission Controller 157 | 158 | ``` 159 | sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml 160 | ``` 161 | 162 | Locate the line for the `--enable-admission-plugins` and add `PodSecurityPolicy` to the list. 163 | 164 | ``` 165 | - --enable-admission-plugins=NodeRestriction,PodSecurityPolicy 166 | ``` 167 | 168 | Run a command to make sure the API Server is still responding after the change to the manifest file (it may take a few moments to come back up). 169 | 170 | ``` 171 | kubectl get nodes 172 | ``` 173 | 174 | ## Create a Pod 175 | Create a basic Pod, using the `nonadmin` ServiceAccount created earlier. 176 | 177 | ``` 178 | vi pod-basic.yml 179 | ``` 180 | 181 | ``` 182 | apiVersion: v1 183 | kind: Pod 184 | metadata: 185 | name: pod-basic 186 | namespace: development 187 | spec: 188 | containers: 189 | - name: nginx 190 | image: nginx 191 | ``` 192 | 193 | ``` 194 | kubectl create -f pod-basic.yml --as=system:serviceaccount:development:nonadmin --save-config 195 | ``` 196 | 197 | This operation will fail, since the Pod does not have access to any PodSecurityPolicies that would allow it to be created. 198 | 199 | Let's try again, this time ensuring the Pod has access to our PodSecurityPolicy by giving it the `nonprivileged` ServiceAccount. 200 | 201 | ``` 202 | vi pod-with-psp-access.yml 203 | ``` 204 | 205 | ``` 206 | apiVersion: v1 207 | kind: Pod 208 | metadata: 209 | name: pod-with-psp-access 210 | namespace: development 211 | spec: 212 | serviceAccountName: nonprivileged 213 | containers: 214 | - name: nginx 215 | image: nginx 216 | ``` 217 | 218 | ``` 219 | kubectl create -f pod-with-psp-access.yml --as=system:serviceaccount:development:nonadmin --save-config 220 | ``` 221 | 222 | This time, it should succeed since the Pod's ServiceAccount has access to the PodSecurityPolicy, and the Pod does not violate any policies. 223 | 224 | Examine the Pod with `kubectl describe`. 225 | 226 | ``` 227 | kubectl describe pod pod-with-psp-access -n development 228 | ``` 229 | 230 | Note that there an annotation which lists the PodSecurityPolicy that allowed the Pod. 231 | 232 | Let's try to create a Pod that does violate the policy by requesting privileged access. 233 | 234 | ``` 235 | vi pod-privileged.yml 236 | ``` 237 | 238 | ``` 239 | apiVersion: v1 240 | kind: Pod 241 | metadata: 242 | name: pod-privileged 243 | namespace: development 244 | spec: 245 | serviceAccountName: nonprivileged 246 | containers: 247 | - name: nginx 248 | image: nginx 249 | securityContext: 250 | privileged: true 251 | ``` 252 | 253 | ``` 254 | kubectl create -f pod-privileged.yml --as=system:serviceaccount:development:nonadmin --save-config 255 | ``` 256 | 257 | This should fail, since the privileged mode containers are not allowed by the PodSecurityPolicy. 258 | 259 | ## Fix Mirror Pod Creation for Static Pods 260 | List your system Pods. 261 | 262 | ``` 263 | kubectl get pods -n kube-system 264 | ``` 265 | 266 | You may notice that the kube-apiserver mirror Pod is missing. While the API Server itself is in fact running since it is managed by kubelet and bypasses PodSecurityPolicies, the PodSecurityPolicies are preventing the mirror Pod from being created. This makes these system Pods invisible in the Kubernetes API. Let's fix it! 267 | 268 | Create a new PodSecurityPolicy that will allow static mirror Pods. 269 | 270 | ``` 271 | vi psp-static.yml 272 | ``` 273 | 274 | ``` 275 | apiVersion: policy/v1beta1 276 | kind: PodSecurityPolicy 277 | metadata: 278 | name: psp-static 279 | spec: 280 | allowPrivilegeEscalation: true 281 | fsGroup: 282 | rule: RunAsAny 283 | hostNetwork: true 284 | runAsUser: 285 | rule: RunAsAny 286 | seLinux: 287 | rule: RunAsAny 288 | supplementalGroups: 289 | rule: RunAsAny 290 | volumes: 291 | - configMap 292 | - downwardAPI 293 | - emptyDir 294 | - persistentVolumeClaim 295 | - secret 296 | - projected 297 | - hostPath 298 | ``` 299 | 300 | ``` 301 | kubectl create -f psp-static.yml --save-config 302 | ``` 303 | 304 | Create a ClusterRole with access to the policy. 305 | 306 | ``` 307 | vi cr-use-psp-static.yml 308 | ``` 309 | 310 | ``` 311 | apiVersion: rbac.authorization.k8s.io/v1 312 | kind: ClusterRole 313 | metadata: 314 | name: cr-use-psp-static 315 | rules: 316 | - apiGroups: ['policy'] 317 | resources: ['podsecuritypolicies'] 318 | verbs: ['use'] 319 | resourceNames: 320 | - psp-static 321 | ``` 322 | 323 | ``` 324 | kubectl create -f cr-use-psp-static.yml --save-config 325 | ``` 326 | 327 | Create a ClusterRoleBinding to allow Kubernetes Nodes to use the policy. 328 | 329 | ``` 330 | vi crb-use-psp-static.yml 331 | ``` 332 | 333 | ``` 334 | apiVersion: rbac.authorization.k8s.io/v1 335 | kind: ClusterRoleBinding 336 | metadata: 337 | name: crb-use-psp-static 338 | roleRef: 339 | kind: ClusterRole 340 | name: cr-use-psp-static 341 | apiGroup: rbac.authorization.k8s.io 342 | subjects: 343 | - kind: Group 344 | name: system:nodes 345 | ``` 346 | 347 | ``` 348 | kubectl create -f crb-use-psp-static.yml --save-config 349 | ``` 350 | 351 | List system Pods again. 352 | 353 | ``` 354 | kubectl get pods -n kube-system 355 | ``` 356 | 357 | You should see a `kube-apiserver` Pod appear. If it doesn't, you may need to wait a minute or two and try again. 358 | -------------------------------------------------------------------------------- /cr-use-psp-static.yml: -------------------------------------------------------------------------------- 1 | apiVersion: rbac.authorization.k8s.io/v1 2 | kind: ClusterRole 3 | metadata: 4 | name: cr-use-psp-static 5 | rules: 6 | - apiGroups: ['policy'] 7 | resources: ['podsecuritypolicies'] 8 | verbs: ['use'] 9 | resourceNames: 10 | - psp-static 11 | -------------------------------------------------------------------------------- /cr-use-psp.yml: -------------------------------------------------------------------------------- 1 | apiVersion: rbac.authorization.k8s.io/v1 2 | kind: ClusterRole 3 | metadata: 4 | name: cr-use-psp 5 | rules: 6 | - apiGroups: ['policy'] 7 | resources: ['podsecuritypolicies'] 8 | verbs: ['use'] 9 | resourceNames: 10 | - psp-nopriv 11 | -------------------------------------------------------------------------------- /crb-use-psp-static.yml: -------------------------------------------------------------------------------- 1 | apiVersion: rbac.authorization.k8s.io/v1 2 | kind: ClusterRoleBinding 3 | metadata: 4 | name: crb-use-psp-static 5 | roleRef: 6 | kind: ClusterRole 7 | name: cr-use-psp-static 8 | apiGroup: rbac.authorization.k8s.io 9 | subjects: 10 | - kind: Group 11 | name: system:nodes 12 | -------------------------------------------------------------------------------- /pod-basic.yml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Pod 3 | metadata: 4 | name: pod-basic 5 | namespace: development 6 | spec: 7 | containers: 8 | - name: nginx 9 | image: nginx 10 | -------------------------------------------------------------------------------- /pod-privileged.yml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Pod 3 | metadata: 4 | name: pod-privileged 5 | namespace: development 6 | spec: 7 | serviceAccountName: nonprivileged 8 | containers: 9 | - name: nginx 10 | image: nginx 11 | securityContext: 12 | privileged: true 13 | -------------------------------------------------------------------------------- /pod-with-psp-access.yml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Pod 3 | metadata: 4 | name: pod-with-psp-access 5 | namespace: development 6 | spec: 7 | serviceAccountName: nonprivileged 8 | containers: 9 | - name: nginx 10 | image: nginx 11 | -------------------------------------------------------------------------------- /psp-nopriv.yml: -------------------------------------------------------------------------------- 1 | apiVersion: policy/v1beta1 2 | kind: PodSecurityPolicy 3 | metadata: 4 | name: psp-nopriv 5 | spec: 6 | privileged: false 7 | runAsUser: 8 | rule: RunAsAny 9 | fsGroup: 10 | rule: RunAsAny 11 | seLinux: 12 | rule: RunAsAny 13 | supplementalGroups: 14 | rule: RunAsAny 15 | volumes: 16 | - configMap 17 | - downwardAPI 18 | - emptyDir 19 | - persistentVolumeClaim 20 | - secret 21 | - projected 22 | -------------------------------------------------------------------------------- /psp-static.yml: -------------------------------------------------------------------------------- 1 | apiVersion: policy/v1beta1 2 | kind: PodSecurityPolicy 3 | metadata: 4 | name: psp-static 5 | spec: 6 | allowPrivilegeEscalation: true 7 | fsGroup: 8 | rule: RunAsAny 9 | hostNetwork: true 10 | runAsUser: 11 | rule: RunAsAny 12 | seLinux: 13 | rule: RunAsAny 14 | supplementalGroups: 15 | rule: RunAsAny 16 | volumes: 17 | - configMap 18 | - downwardAPI 19 | - emptyDir 20 | - persistentVolumeClaim 21 | - secret 22 | - projected 23 | - hostPath 24 | -------------------------------------------------------------------------------- /rb-nonadmin-edit.yml: -------------------------------------------------------------------------------- 1 | apiVersion: rbac.authorization.k8s.io/v1 2 | kind: RoleBinding 3 | metadata: 4 | name: rb-nonadmin-edit 5 | namespace: development 6 | roleRef: 7 | kind: ClusterRole 8 | name: edit 9 | apiGroup: rbac.authorization.k8s.io 10 | subjects: 11 | - kind: ServiceAccount 12 | name: nonadmin 13 | namespace: development 14 | -------------------------------------------------------------------------------- /rb-nonprivileged.yml: -------------------------------------------------------------------------------- 1 | apiVersion: rbac.authorization.k8s.io/v1 2 | kind: RoleBinding 3 | metadata: 4 | name: rb-nonprivileged 5 | namespace: development 6 | roleRef: 7 | kind: ClusterRole 8 | name: cr-use-psp 9 | apiGroup: rbac.authorization.k8s.io 10 | subjects: 11 | - kind: ServiceAccount 12 | name: nonprivileged 13 | namespace: development 14 | --------------------------------------------------------------------------------