├── kube-dns.yml ├── README.md └── egress.yml /kube-dns.yml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | name: kube-dns 5 | namespace: kube-system 6 | data: 7 | stubDomains: | 8 | {"egress.local": ["coredns-coredns.coredns.svc.cluster.local"]} 9 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Overview 2 | 3 | This repo contains a proof-of-concept to enable Linkerd's metrics, retries, and other features for external (off-cluster or 3rd-party) HTTP calls. For example, with this POC, all calls to `api.github.com` can be monitored for success rate, retried automatically, and so on. 4 | 5 | # Background 6 | 7 | Calls to third-party APIs are typically TLS'd by the application directly (e.g. the client connects to `httpS://api.github.com`). However, Linkerd can only provide metrics and reliability features for *non-TLS'd* HTTP connections (e.g. `http://api.github.com`)--which it will then add TLS to, on behalf of the application. Thus, if we want metrics, we must provide non-TLS'd connections to Linkerd. 8 | 9 | However, simply changing all third-party HTTP calls from HTTPS to plaintext HTTP is risky: if these connections are ever accidentally made in a non-Linkerd-enabled context they may traverse the open internet in plaintext, potentially exposing sensitive data. 10 | 11 | Thus, the purpose of this POC is to provide a *safe* mechanism by which your application can initiate a plaintext HTTP call to a third party API (to be TLS'd by Linkerd), by ensuring that these calls will fail unless they are run in the production context. 12 | 13 | ## How it works 14 | 15 | This POC installs an egress proxy and DNS rules such that all calls to `FOO.egress.local` resolve to this proxy, which in turn encrypts the connection and proxies it to `FOO`. For example, a call to `api.github.com.egress.local` will be proxied to `api.github.com`. Linkerd will mTLS the connection from the application to the egress proxy; the egress proxy will TLS the connection to the third party. 16 | 17 | Then, in application code, all calls in the production context *only* to `httpS://FOO` are replacted with calls to `http://FOO.egress.local`. 18 | 19 | Critically, this approach "fails safely": if the application is not running in the production context (with Linkerd and the egress proxy), the `.egress.local` domain will not resolve and this unencrypted call will fail. Additionally, this provides a convenient point for enforcing egress control. 20 | 21 | ## Installation 22 | 23 | ```bash 24 | kubectl apply -f egress.yml 25 | kubectl apply -f kube-dns.yml 26 | ``` 27 | 28 | Note: If you're already using coredns, just add the following to your configmap: 29 | 30 | ``` 31 | rewrite name regex (.*)\.egress.local proxy.egress.svc.cluster.local 32 | ``` 33 | 34 | ## Usage 35 | 36 | In your application code, when running in the production context *only*, replace calls to `httpS://FOO` with calls to `http://FOO.egress.local`. For example, calling of addressing `httpS://api.github.com`, call `http://api.github.com.egress.local`. 37 | 38 | Metrics for calls to `FOO.egress.local` will then be available alongside all other Linkerd metrics. 39 | 40 | # Future work 41 | 42 | - Restricting outbound traffic to specific egress pods. 43 | - Inspecting HTTP metrics at a central location. 44 | - Applying egress policy. 45 | -------------------------------------------------------------------------------- /egress.yml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Namespace 3 | metadata: 4 | name: egress 5 | --- 6 | apiVersion: v1 7 | kind: Service 8 | metadata: 9 | name: proxy 10 | namespace: egress 11 | labels: 12 | app.kubernetes.io/instance: nginx 13 | app.kubernetes.io/name: egress-proxy 14 | spec: 15 | selector: 16 | app.kubernetes.io/instance: nginx 17 | app.kubernetes.io/name: egress-proxy 18 | ports: 19 | - name: http 20 | port: 80 21 | targetPort: 8080 22 | - name: proxy 23 | port: 8080 24 | targetPort: 8080 25 | --- 26 | apiVersion: v1 27 | kind: ConfigMap 28 | metadata: 29 | name: nginx 30 | namespace: egress 31 | labels: 32 | app.kubernetes.io/instance: nginx 33 | app.kubernetes.io/name: egress-proxy 34 | data: 35 | nginx.conf: |- 36 | events { 37 | } 38 | http { 39 | include /etc/nginx/resolvers-data/resolvers.conf; 40 | 41 | server { 42 | listen 8080; 43 | 44 | location / { 45 | if ($host ~* "(.*)\.egress\.local") { 46 | set $real_host $1; 47 | } 48 | 49 | proxy_pass https://$real_host; 50 | proxy_set_header Host $real_host; 51 | } 52 | } 53 | } 54 | --- 55 | apiVersion: apps/v1 56 | kind: Deployment 57 | metadata: 58 | name: nginx 59 | namespace: egress 60 | labels: 61 | app.kubernetes.io/instance: nginx 62 | app.kubernetes.io/name: egress-proxy 63 | spec: 64 | replicas: 1 65 | selector: 66 | matchLabels: 67 | app.kubernetes.io/instance: nginx 68 | app.kubernetes.io/name: egress-proxy 69 | template: 70 | metadata: 71 | annotations: 72 | linkerd.io/inject: enabled 73 | labels: 74 | app.kubernetes.io/instance: nginx 75 | app.kubernetes.io/name: egress-proxy 76 | spec: 77 | volumes: 78 | - name: config 79 | configMap: 80 | name: nginx 81 | - name: resolvers-data 82 | emptyDir: {} 83 | initContainers: 84 | - name: setup-resolvers 85 | image: busybox:1.28 86 | command: ["sh", "-c", "echo resolver $(awk 'BEGIN{ORS=\" \"} $1==\"nameserver\" {print $2}' /etc/resolv.conf)\";\" > /etc/nginx/resolvers-data/resolvers.conf"] 87 | volumeMounts: 88 | - name: resolvers-data 89 | mountPath: /etc/nginx/resolvers-data 90 | containers: 91 | - name: nginx 92 | image: nginx:1.17 93 | ports: 94 | - name: http 95 | containerPort: 8080 96 | volumeMounts: 97 | - name: config 98 | mountPath: /etc/nginx 99 | - name: resolvers-data 100 | mountPath: /etc/nginx/resolvers-data 101 | --- 102 | apiVersion: v1 103 | kind: ConfigMap 104 | metadata: 105 | name: coredns 106 | namespace: egress 107 | labels: 108 | app.kubernetes.io/instance: coredns 109 | app.kubernetes.io/name: coredns 110 | data: 111 | Corefile: |- 112 | .:53 { 113 | errors 114 | health { 115 | lameduck 5s 116 | } 117 | ready 118 | kubernetes cluster.local in-addr.arpa ip6.arpa { 119 | pods insecure 120 | fallthrough in-addr.arpa ip6.arpa 121 | ttl 30 122 | } 123 | prometheus 0.0.0.0:9153 124 | forward . /etc/resolv.conf 125 | cache 30 126 | loop 127 | reload 128 | loadbalance 129 | rewrite name regex (.*)\.egress.local proxy.egress.svc.cluster.local 130 | log 131 | } 132 | --- 133 | apiVersion: v1 134 | kind: Service 135 | metadata: 136 | name: coredns 137 | namespace: egress 138 | labels: 139 | app.kubernetes.io/instance: coredns 140 | app.kubernetes.io/name: coredns 141 | annotations: 142 | prometheus.io/port: "9153" 143 | prometheus.io/scrape: "true" 144 | spec: 145 | selector: 146 | app.kubernetes.io/instance: coredns 147 | app.kubernetes.io/name: coredns 148 | ports: 149 | - port: 53 150 | protocol: UDP 151 | name: udp-53 152 | - port: 53 153 | protocol: TCP 154 | name: tcp-53 155 | type: ClusterIP 156 | --- 157 | apiVersion: apps/v1 158 | kind: Deployment 159 | metadata: 160 | name: coredns 161 | namespace: egress 162 | labels: 163 | app.kubernetes.io/instance: coredns 164 | app.kubernetes.io/name: coredns 165 | spec: 166 | replicas: 1 167 | strategy: 168 | type: RollingUpdate 169 | rollingUpdate: 170 | maxUnavailable: 1 171 | maxSurge: 10% 172 | selector: 173 | matchLabels: 174 | app.kubernetes.io/instance: coredns 175 | app.kubernetes.io/name: coredns 176 | template: 177 | metadata: 178 | labels: 179 | app.kubernetes.io/name: coredns 180 | app.kubernetes.io/instance: coredns 181 | spec: 182 | serviceAccountName: default 183 | containers: 184 | - name: "coredns" 185 | image: "coredns/coredns:1.6.9" 186 | imagePullPolicy: IfNotPresent 187 | args: [ "-conf", "/etc/coredns/Corefile" ] 188 | volumeMounts: 189 | - name: config-volume 190 | mountPath: /etc/coredns 191 | resources: 192 | limits: 193 | cpu: 100m 194 | memory: 128Mi 195 | requests: 196 | cpu: 100m 197 | memory: 128Mi 198 | ports: 199 | - containerPort: 53 200 | protocol: UDP 201 | name: udp-53 202 | - containerPort: 53 203 | protocol: TCP 204 | name: tcp-53 205 | 206 | livenessProbe: 207 | httpGet: 208 | path: /health 209 | port: 8080 210 | scheme: HTTP 211 | initialDelaySeconds: 60 212 | timeoutSeconds: 5 213 | successThreshold: 1 214 | failureThreshold: 5 215 | readinessProbe: 216 | httpGet: 217 | path: /ready 218 | port: 8181 219 | scheme: HTTP 220 | initialDelaySeconds: 10 221 | timeoutSeconds: 5 222 | successThreshold: 1 223 | failureThreshold: 5 224 | volumes: 225 | - name: config-volume 226 | configMap: 227 | name: coredns-coredns 228 | items: 229 | - key: Corefile 230 | path: Corefile 231 | --------------------------------------------------------------------------------