├── assets ├── tmux_cssh_vm.png ├── falco_audit_log2.png ├── falcosidekick_ui.png └── everything_should_fine.png └── README.md /assets/tmux_cssh_vm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/developer-guy/falco-analyze-audit-log-from-k3s-cluster/HEAD/assets/tmux_cssh_vm.png -------------------------------------------------------------------------------- /assets/falco_audit_log2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/developer-guy/falco-analyze-audit-log-from-k3s-cluster/HEAD/assets/falco_audit_log2.png -------------------------------------------------------------------------------- /assets/falcosidekick_ui.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/developer-guy/falco-analyze-audit-log-from-k3s-cluster/HEAD/assets/falcosidekick_ui.png -------------------------------------------------------------------------------- /assets/everything_should_fine.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/developer-guy/falco-analyze-audit-log-from-k3s-cluster/HEAD/assets/everything_should_fine.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ![falco_audit_log](./assets/falco_audit_log2.png) 2 | 3 | # Analyze Kubernetes Audit logs using Falco 4 | There is concept called "Event Sources" in Falco, these "Event Sources" defines where Falco can consume events, and apply rules to these events to detect abnormal behavior. Currently Falco supports thefollowing event sources: 5 | 6 | * System Calls (syscall) via the [drivers](https://falco.org/docs/event-sources/drivers) 7 | * [Kubernetes Audit Events](https://falco.org/docs/event-sources/kubernetes-audit) (k8s_audit) 8 | 9 | In this guide, we are going to use "Kubernetes Audit Logs" as an event source that Falco can consume. 10 | 11 | 12 | 13 | 14 | # Table of Contents 15 | - 🧰 [Prerequisites](#prerequisites) 16 | 17 | - 🎁 [What is in it for you ?](#what-is-in-it-for-you-) 18 | 19 | -

[What are the meanings of Audit Log in Kubernetes ?](#what-are-the-meanings-of-audit-log-in-kubernetes-) 20 | -

[How can you enable Audit Log feature in Kubernetes ?](#how-can-you-enable-audit-log-feature-in-kubernetes-) 21 | -

[How can you enable embedded web server available within the Falco project ?](#how-can-you-enable-embedded-web-server-available-within-the-falco-project-) 22 | -

[What is the purpose of Falcosidekick project ?](#what-is-the-purpose-of-falcosidekick-project-) 23 | 24 | - 👨‍💻 [Hands On](#hands-on) 25 | - 👀 [References](#references) 26 | 27 | 28 | 29 | # Prerequisites 30 | *

Kubernetes Cluster v1.20.2 31 | *

tmux-cssh (or another tool to connect multiple SSH servers) 32 | *

kubectl v1.20.2 33 | *

Helm v3.5.1 34 | 35 | # What is in it for you ? 36 | You are going to learn: 37 | * the meaning of audit logs in Kubernetes. 38 | * how you can enable Audit log feature in Kubernetes. 39 | * how you can enable embedded web server available within the Falco project. 40 | * the purpose of the Falcosidekick project. 41 | 42 | # What are the meanings of Audit Log in Kubernetes ? 43 | Kubernetes auditing provides a security-relevant, chronological set of records documenting the sequence of actions in a cluster. The cluster audits the activities generated by users, by applications that use the Kubernetes API, and by the control plane itself. 44 | 45 | Auditing allows cluster administrators to answer the following questions: 46 | 47 | * what happened? 48 | * when did it happen? 49 | * who initiated it? 50 | * on what did it happen? 51 | * where was it observed? 52 | * from where was it initiated? 53 | * to where was it going? 54 | 55 | Audit records begin their lifecycle inside the kube-apiserver component. Each request on each stage of its execution generates an audit event, which is then pre-processed according to a certain policyand written to a backend. The policy determines what's recorded and the backends persist the records. The current backend implementations include logs files and webhooks. 56 | 57 | > Credit: https://kubernetes.io/docs/tasks/debug-application-cluster/audit 58 | 59 | # How can you enable Audit Log feature in Kubernetes ? 60 | In this demo, we are going to use Webhook backend mode for Audit Log to be able to send audit logs to the webserver that is available within the Falco project. 61 | 62 | The webhook audit backend sends audit events to a remote web API, which is assumed to be a form of the Kubernetes API, including means of authentication. You can configure a webhook audit backend using the following kube-apiserver flags: 63 | 64 | * --audit-webhook-config-file specifies the path to a file with a webhook configuration. The webhook configuration is effectively a specialized kubeconfig. 65 | * --audit-webhook-initial-backoff specifies the amount of time to wait after the first failed request before retrying. Subsequent requests are retried with exponential backoff. 66 | The webhook config file uses the kubeconfig format to specify the remote address of the service and credentials used to connect to it. 67 | 68 | > Credit: https://kubernetes.io/docs/tasks/debug-application-cluster/audit/#webhook-backend 69 | 70 | # How can you enable embedded web server available within the Falco project ? 71 | In this demo, we are going to using Helm to install the Falco, so, there is value that we can enableor disable to audit log feature called "auditLog.enabled", if we enable this, the embedded webserveris going to start within the Falco to consume audit events at port *8765* and behind the *k8s-audit* endpoint. 72 | 73 | # What is the purpose of Falcosidekick project ? 74 | By default, Falco has 5 outputs for its events: stdout, file, gRPC, shell and http.Even if they're convenient, we can quickly be limited to integrating Falco with other components. Here comes falcosidekick, a little daemon that extends that number of possible outputs. 75 | 76 | The current list of available falcosidekick outputs (version 2.13.0) is: 77 | 78 | * Slack 79 | * Rocketchat 80 | * Mattermost 81 | * Teams 82 | * Datadog 83 | * AlertManager 84 | * Elasticsearch 85 | * Loki 86 | * NATS 87 | * Influxdb 88 | * AWS Lambda 89 | * AWS SQS 90 | * SMTP (email) 91 | * Opsgenie 92 | * Webhook 93 | 94 | Beyond that, it provides metrics about the number of events and let you add custom fields in events, for example environment, region, etc 95 | 96 | # Hands On 97 | In this demo, we are going provision our k3s cluster using Terraform and Ansible with HA support. To get more detail about the installation of the k3s cluster please follow the [link.](https://github.com/developer-guy/kubernetes-cluster-setup-using-terraform-and-k3s-on-digitalocean) 98 | First, we are going to install Falco and the Falcosidekick with UI enabled. In order to do that, we'll use the Helm. 99 | ```bash 100 | $ helm install falco --namespace falco --set falco.jsonOutput=true --set falco.httpOutput.enabled=true --set falco.httpOutput.url=http://falcosidekick:2801 --set falcosidekick.enabled=true --set falcosidekick.webui.enabled=true falcosecurity/falco --set auditLog.enabled=true --create-namespace 101 | NAME: falco 102 | LAST DEPLOYED: Sun Feb 21 23:53:05 2021 103 | NAMESPACE: falco 104 | STATUS: deployed 105 | REVISION: 1 106 | NOTES: 107 | Falco agents are spinning up on each node in your cluster. After a few 108 | seconds, they are going to start monitoring your containers looking for 109 | security issues. 110 | 111 | 112 | No further action should be required. 113 | ``` 114 | Lets verify the installation. 115 | ```bash 116 | $ kubectl --namespace=falco get pods 117 | Found existing alias for "kubectl". You should use: "k" 118 | NAME READY STATUS RESTARTS AGE 119 | falco-falcosidekick-ui-5d59c77944-7htbc 1/1 Running 0 44s 120 | falco-ckgqp 1/1 Running 0 45s 121 | falco-falcosidekick-6c5df8d54-6rb5h 1/1 Running 0 44s 122 | falco-74vkg 1/1 Running 0 45s 123 | falco-falcosidekick-6c5df8d54-7tprb 1/1 Running 0 44s 124 | falco-tc78l 1/1 Running 0 45s 125 | falco-48vk2 1/1 Running 0 45s 126 | falco-blkzl 1/1 Running 0 45s 127 | ``` 128 | 129 | Then, we should expose embedded webserver to be able send logs to it from the master nodes. 130 | ```bash 131 | $ kubectl patch svc falco --namespace=falco --type='json' -p '[{"op":"replace","path":"/spec/type","value":"NodePort"}]' 132 | Found existing alias for "kubectl". You should use: "k" 133 | service/falco patched 134 | ``` 135 | 136 | Get the NodePort ip of the Falco service to reach embedded server. 137 | ```bash 138 | $ kubectl --namespace=falco get services Found existing alias for "kubectl". You should use: "k" 139 | NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE 140 | falco-falcosidekick-ui ClusterIP 10.43.221.136 2802/TCP 2m52s 141 | falco-falcosidekick ClusterIP 10.43.179.95 2801/TCP 2m52s 142 | falco NodePort 10.43.114.83 8765:30958/TCP 2m52s 143 | ``` 144 | 145 | After you set up the k3s cluster, lets enable Audit Log feature on the k3s master nodes. In order to do that, we'll use tmux-cssh to connect to the VMs. Copy your VM'S IP addresses first then run the following command: 146 | ```bash 147 | $ tmux-cssh -o '-i ~/.ssh/digitalocean_rsa' root@161.35.209.51 root@167.71.44.217 # <-- This command will open a new tmux session and divide panes for each of the VM and syncronize commnds between them. 148 | ``` 149 | 150 | After you run the command you should see similar screen below: 151 | 152 |

153 | 154 | After that, you should run the following commands to enable Audit Log, once you type command in one screen, it will appear in the other pane immediately. 155 | ```bash 156 | $ sudo mkdir -p /var/lib/rancher/audit 157 | $ wget https://raw.githubusercontent.com/falcosecurity/evolution/master/examples/k8s_audit_config/audit-policy.yaml 158 | $ sudo cp audit-policy.yaml /var/lib/rancher/audit/ 159 | $ export IP=161.35.217.182 160 | $ cat << EOF | sudo tee /var/lib/rancher/audit/webhook-config.yaml 161 | apiVersion: v1 162 | kind: Config 163 | clusters: 164 | - name: falco 165 | cluster: 166 | server: http://$IP:30958/k8s-audit 167 | contexts: 168 | - context: 169 | cluster: falco 170 | user: "" 171 | name: default-context 172 | current-context: default-context 173 | preferences: {} 174 | users: [] 175 | EOF 176 | ``` 177 | Once you prepare the the files, we should edit our k3s.service in the VMs. 178 | ```bash 179 | $ vim /etc/systemd/system/k3s.service 180 | # add these lines to after the k3s server command. 181 | # --kube-apiserver-arg=audit-log-path=/var/lib/rancher/audit/audit.log 182 | # --kube-apiserver-arg=audit-policy-file=/var/lib/rancher/audit/audit-policy.yaml 183 | # --kube-apiserver-arg=audit-webhook-config-file=/var/lib/rancher/audit/webhook-config.yaml 184 | ``` 185 | 186 | This Unit file should looks like the following above: 187 | ```Unit 188 | [Unit] 189 | Description=Lightweight Kubernetes 190 | Documentation=https://k3s.io 191 | Wants=network-online.target 192 | After=network-online.target 193 | 194 | [Service] 195 | Type=notify 196 | ExecStartPre=-/sbin/modprobe br_netfilter 197 | ExecStartPre=-/sbin/modprobe overlay 198 | ExecStart=/usr/local/bin/k3s server --kube-apiserver-arg=audit-log-path=/var/lib/rancher/audit/audit.log --kube-apiserver-arg=audit-policy-file=/var/lib/rancher/audit/audit-policy.yaml --kube-apiserver-arg=audit-webhook-config-file=/var/lib/rancher/audit/webhook-config.yaml --config /etc/rancher/k3s/config.yaml 199 | KillMode=process 200 | Delegate=yes 201 | LimitNOFILE=1048576 202 | LimitNPROC=infinity 203 | LimitCORE=infinity 204 | TasksMax=infinity 205 | TimeoutStartSec=0 206 | Restart=always 207 | RestartSec=5s 208 | 209 | [Install] 210 | WantedBy=multi-user.target 211 | ``` 212 | 213 | After you edit the Unit file, you should reload the daemon and the k3s service. 214 | ```bash 215 | $ sudo systemctl daemon-reload && \ 216 | sudo systemctl restart k3s 217 | ``` 218 | 219 | If everything goes fine, you should see similar output like the following image below: 220 | 221 |

222 | 223 | In order to test this scenario, we need to create an event that Falco provides alert for it. 224 | ```bash 225 | $ kubectl create configmap aws-creds \ 226 | --from-literal aws_access_key_id=AKES20LNOA 227 | ``` 228 | 229 | Once you created this, we should see the alert at the Falcosidekick UI. 230 | ```bash 231 | $ kubectl --namespace=falco port-forward svc/falco-falcosidekick-ui 2802 232 | Found existing alias for "kubectl". You should use: "k" 233 | Forwarding from 127.0.0.1:2802 -> 2802 234 | Forwarding from [::1]:2802 -> 2802 235 | ``` 236 | 237 | Open the browser, go to the localhost:2802, you should see similar screen following below: 238 | ![falco_sidekick_ui](./assets/falcosidekick_ui.png) 239 | 240 | # References 241 | * https://blog.alexellis.io/falco-at-the-edge-arm64/ 242 | * https://falco.org/blog/extend-falco-outputs-with-falcosidekick/ 243 | * https://sysdig.com/blog/kubernetes-audit-log-falco 244 | 245 | --------------------------------------------------------------------------------