├── .gitignore ├── tunnel-router ├── start.sh ├── Dockerfile ├── change.py └── router.py ├── example-svc.yaml ├── tunnel-router.yaml ├── README.md └── static └── cluster-layout.svg /.gitignore: -------------------------------------------------------------------------------- 1 | .*.swp 2 | *.pyc 3 | -------------------------------------------------------------------------------- /tunnel-router/start.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | echo "Disabling reverse path filtering" 4 | for i in /proc/sys/net/ipv4/conf/*/rp_filter 5 | do 6 | echo 0 > $i 7 | done 8 | 9 | cd /router/ 10 | exec env XTABLES_LIBDIR=/usr/lib/xtables/ python3 -u ./router.py 11 | -------------------------------------------------------------------------------- /example-svc.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | annotations: 5 | # This is the IP that will be tunneled into the pod. 6 | # The pod will also get this IP added to a pod-local interface. 7 | cmd.nu/tunnel: 31.31.164.211 8 | labels: 9 | k8s-app: test 10 | name: test 11 | spec: 12 | ports: 13 | - port: 3000 14 | protocol: TCP 15 | targetPort: 3000 16 | selector: 17 | k8s-app: test 18 | -------------------------------------------------------------------------------- /tunnel-router/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3-alpine 2 | 3 | RUN apk --update add gcc linux-headers musl-dev iptables 4 | RUN pip3 install python-iptables pyroute2 pykube docker 5 | 6 | ADD start.sh / 7 | ADD *.py /router/ 8 | 9 | # TODO: python-iptables seems to have trouble finding the libraries 10 | RUN ln -sf /usr/lib/libxtables.so.*.*.* /usr/lib//libxtables.so 11 | RUN ln -sf /usr/lib/libiptc.so.*.*.* /usr/lib/libiptc.so 12 | RUN ln -sf /usr/lib /usr/lib64 13 | 14 | CMD ["/start.sh"] 15 | -------------------------------------------------------------------------------- /tunnel-router.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: DaemonSet 3 | metadata: 4 | labels: 5 | k8s-app: tunnel-router 6 | name: tunnel-router 7 | namespace: kube-system 8 | spec: 9 | template: 10 | metadata: 11 | labels: 12 | k8s-app: tunnel-router 13 | spec: 14 | hostNetwork: true 15 | # Needed to inject network interfaces into pods 16 | hostPID: true 17 | # Needed to communicate with docker for network NS resolution 18 | hostIPC: true 19 | containers: 20 | - image: bluecmd/tunnel-router 21 | env: 22 | - name: TUNNEL_ROUTER_MODE 23 | value: 'gre' 24 | imagePullPolicy: Always 25 | name: router 26 | securityContext: 27 | privileged: true 28 | volumeMounts: 29 | - name: run 30 | mountPath: /var/run/docker.sock 31 | volumes: 32 | - name: run 33 | hostPath: 34 | path: /var/run/docker.sock 35 | terminationGracePeriodSeconds: 5 36 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # kube-service-tunnel 2 | 3 | Goal: Have externally routable IPs end up as normal IP packets inside pods. 4 | 5 | Implementation: Using tunnels on ingressing traffic torward these IPs, tunnel the traffic to inside the pods. 6 | 7 | State: 8 | - GRE verified working w/ Calico (caveat https://github.com/projectcalico/felix/issues/1248) 9 | - MPLS does not configure pods yet (pending https://github.com/coreos/bugs/issues/1730). 10 | 11 | ## Getting Started 12 | 13 | 1. `kubectl create -f tunnel-router.yaml` 14 | 2. Annotate a service with `cmd.nu/tunnel: $ip` where $ip is an IP that is routed towards your nodes. 15 | 16 | Done! You should now be able to access the pod normally through the tunnel'd IP. If your pod listens to `0.0.0.0`/`::0` everything should just work. 17 | 18 | ## Modes of Operation 19 | 20 | kube-service-tunnel preferred mode uses lwtunnels to avoid creating a massive 21 | amount of tunnel interfaces. Instead, encapsulation is specified on a per-route 22 | basis. For this to work you need a kernel with CONFIG\_LWTUNNEL (newer than 4.3). 23 | 24 | Alternatively kube-service-tunnel can work using GRE tunnels using ordinary 25 | tunnel interfaces. 26 | 27 | Currently, as far as I know, the only supported encapsulation for lwtunnels 28 | is MPLS, which limits the usability of this mode to node-local encapsulation. 29 | If you have other router infrastructure set up to deliver traffic to your nodes 30 | (like [`svc-bgp`](https://github.com/dhtech/kubernetes/tree/master/svc-bgp)) 31 | this will work just fine. Otherwise you might want to use the GRE mode. 32 | 33 | Due to the restrictions above the following applies: 34 | 35 | * In MPLS mode you are responsible to get the packet to a serving node. 36 | 37 | * In GRE mode a lot of interfaces will be created (one per endpoint). 38 | 39 | **If anybody knows more about lwtunnels and how to use it to encapsulate inside 40 | an IP packet, please let me know.** 41 | 42 | ### Load Balancing 43 | 44 | `TODO: Explain this in more detail` 45 | 46 | Using iptables' `HMARK` target, an incoming packet receives a hash that is used to select among a configurable number of routing tables. Each routing table contains a route for every active tunnel IP. One endpoint will most likely be in multiple buckets, and if you have more endpoints than buckets you will have no traffic to the excess part of your endpoints. 47 | 48 | ### Configuration of Endpoints 49 | 50 | When a new endpoint is discovered that belongs to a service with a tunnel IP, that endpoint's pod must be reconfigured to have the tunnel IP available. This is done in a container runtime dependent way (due to network namespace ID lookup), and currently only Docker is supported. 51 | 52 | If the mode of operation is GRE: A GRE interface is created inside the network namespace and the tunnel IP is attached to it. 53 | 54 | If the mode of operation is MPLS: An MPLS decapsulation rule is added to pop the label `100` and deliver those packages locally. The tunnel IP is added to the looback inteface. 55 | 56 | ### Example Cluster Layout 57 | 58 | ![Example Cluster Layout](https://storage.googleapis.com/bluecmd/kube-service-tunnel/cluster-layout.svg) 59 | 60 | An example cluster would have a sane number of nodes marked to handle ingress traffic, here shown with a `node=tunnel` label. Those nodes would announce themselves to the DC routers in a suitable manner, for exmaple using BGP + BFD. 61 | 62 | The cluser ingress nodes would then be responsible for handling routing of incoming packets towards a suitable endpoint. The routing should be done such a way that when a tunnel node fails, the next tunnel node should pick the same routing path as the failed node. 63 | 64 | Note: In the current version the `HMARK` seed is random, which voids the previous paragraph. It is on the road map to store routing decisions in the service objects themselves to allow for hitless resumption of routing. 65 | 66 | ## Example Service 67 | 68 | ``` 69 | apiVersion: v1 70 | kind: Service 71 | metadata: 72 | annotations: 73 | # This is the IP that will be tunneled into the pod. 74 | # The pod will also get this IP added to a pod-local interface. 75 | cmd.nu/tunnel: 1.2.3.4 76 | labels: 77 | k8s-app: test 78 | name: test 79 | spec: 80 | ports: 81 | - port: 3000 82 | protocol: TCP 83 | targetPort: 3000 84 | selector: 85 | k8s-app: test 86 | ``` 87 | 88 | ## Roadmap 89 | 90 | Features planned: 91 | 92 | - Tests :-) 93 | - Full MPLS support 94 | - Draining of endpoints 95 | - Hit-less failover 96 | 97 | Ideas that may come to the roadmap if anyone needs them: 98 | 99 | - Network policy support 100 | - Accounting 101 | 102 | ## FAQ 103 | 104 | Frequently asked questions and answers. 105 | 106 | ### Why not use ... 107 | 108 | There are many ways one could solve the problem discussed here. Let's address why those solutions didn't work for me. 109 | 110 | #### NodePort 111 | 112 | For a while this is what I used. However, it is bulky when used in production. It lacks isolation between services - i.e. a service cannot use the same port, and most obviously it doesn't support the well-known ports. Having the end user having to type `https://cmd.nu:30000` would not be a good end user story. 113 | 114 | Lastly, it doesn't scale. You will run out of ports soon enough. 115 | 116 | #### LoadBalancer 117 | 118 | Same issue as above, but now requires an unknown external agent to forward the TCP connections to the assigned NodePort. Due to this forwarding you will lose the original TCP connection and the metadata about it - such as source-IP. 119 | 120 | #### ClusterIP 121 | 122 | Having the Cluster IP routable from the Internet is something we tried during Dreamhack Winter 2016. It worked reasonably well, and is the closest match to what I would consider production ready yet. Sadly, there are [assumptions](https://github.com/kubernetes/kubernetes/issues/7433) around how cluster IPs work and they are not easy to work around. 123 | 124 | You would also find yourself wasting a lot of routable IPv4 addresses for things that are only cluster internal. 125 | 126 | #### Directly Routed IPs 127 | 128 | Basically, remove the tunnel interfaces and just route the traffic via normal routing. 129 | 130 | This would theoretically work, but a lot of fancy operations that you would like to have - like client affinity, draining, and such would be borderline impossible to implement. The normal multipath routing support in Linux is quite wild-west. 131 | -------------------------------------------------------------------------------- /tunnel-router/change.py: -------------------------------------------------------------------------------- 1 | import binascii 2 | import collections 3 | import errno 4 | import iptc 5 | import os 6 | import pyroute2 7 | import socket 8 | 9 | 10 | # Prefix tunnel interfaces with this string 11 | TUNNEL_PREFIX = os.environ.get('TUNNEL_ROUTER_TUNNEL_PREFIX', 'ts') 12 | BUCKETS = int(os.environ.get('TUNNEL_ROUTER_BUCKETS', '2')) 13 | MODE = os.environ.get('TUNNEL_ROUTER_MODE', 'mpls') 14 | 15 | Interface = collections.namedtuple('Interface', ('ifx', 'internal')) 16 | 17 | 18 | class AddService(object): 19 | """Start ingressing a tunnel IP for a service.""" 20 | 21 | def __init__(self, service): 22 | self.service = service 23 | 24 | def enact(self, service_map, filter_chain, ingress_chain): 25 | print('ADD', self.service) 26 | 27 | rule = iptc.Rule() 28 | rule.dst = self.service.tunnel_ip 29 | t = rule.create_target(ingress_chain.name) 30 | m = rule.create_match("comment") 31 | m.comment = "Tunnel ingress for (%s, %s)" % ( 32 | self.service.name, self.service.namespace) 33 | filter_chain.insert_rule(rule) 34 | 35 | service_map[self.service] = rule 36 | 37 | 38 | class RemoveService(object): 39 | """Stop ingressing a tunnel IP for a service.""" 40 | 41 | def __init__(self, service): 42 | self.service = service 43 | 44 | def enact(self, service_map, filter_chain, ingress_chain): 45 | print('REMOVE', self.service) 46 | rule = service_map[self.service] 47 | filter_chain.delete_rule(rule) 48 | del service_map[self.service] 49 | 50 | 51 | class RefreshEndpoints(object): 52 | """Recalculate all the routing buckets for a service.""" 53 | 54 | def __init__(self, service): 55 | self.service = service 56 | 57 | def enact(self, endpoint_map, ip): 58 | print('REFRESH', self.service) 59 | 60 | # TODO research what the state of per-route encap is. that would be 61 | # extremely nice to use here instead of having a lot of GRE interfaces. 62 | #ip.route('add', dst=self.service.tunnel_ip, oif=2, encap={'type': 'mpls', 'labels': '200/300'}) 63 | 64 | dst = self.service.tunnel_ip + '/32' 65 | 66 | # TODO: only apply the actual changes we need 67 | for table in range(BUCKETS): 68 | try: 69 | ip.route('del', table=(table+1), dst=dst) 70 | except pyroute2.netlink.exceptions.NetlinkError: 71 | pass 72 | 73 | endpoints = endpoint_map[self.service] 74 | if not endpoints: 75 | del endpoint_map[self.service] 76 | return 77 | 78 | # TODO: do actual balancing 79 | endpoint = list(endpoints.keys())[0] 80 | iface = endpoints[endpoint][0] 81 | for table in range(BUCKETS): 82 | if MODE == 'gre': 83 | ip.route('add', table=(table+1), dst=dst, oif=iface.ifx) 84 | if MODE == 'mpls': 85 | ip.route('add', table=(table+1), dst=dst, gateway=endpoint, 86 | encap={'type': 'mpls', 'labels': 100}) 87 | 88 | 89 | class AddEndpoint(object): 90 | """Set up a new tunnel to the new endpoint.""" 91 | 92 | def __init__(self, service, endpoint): 93 | self.service = service 94 | self.endpoint = endpoint 95 | 96 | def enact(self, endpoint_map, ip): 97 | print('NEW_TUNNEL', self.service, self.endpoint) 98 | ifs = [] 99 | 100 | # Open network namespace inside the endpoint if we have it. 101 | # If the pod is not local, we do not have it - but another tunnel 102 | # router will. 103 | netns = (pyroute2.NetNS(self.endpoint.networkNs) 104 | if self.endpoint.networkNs else None) 105 | 106 | if MODE == 'gre': 107 | ifname = TUNNEL_PREFIX + str(binascii.hexlify( 108 | socket.inet_aton(self.endpoint.ip)), 'utf-8') 109 | try: 110 | ip.link('add', ifname=ifname, kind='gre', 111 | gre_remote=self.endpoint.ip) 112 | except pyroute2.netlink.exceptions.NetlinkError as e: 113 | if e.code != errno.EEXIST: 114 | raise 115 | ifx = ip.link_lookup(ifname=ifname)[0] 116 | ifs.append(Interface(ifx, internal=False)) 117 | ip.link('set', state='up', index=ifx) 118 | 119 | if netns: 120 | print('NEW_POD_TUNNEL', self.service, self.endpoint) 121 | ifname = self.service.name 122 | try: 123 | netns.link('add', ifname=ifname, kind='gre', 124 | gre_local=self.endpoint.ip) 125 | except pyroute2.netlink.exceptions.NetlinkError as e: 126 | if e.code != errno.EEXIST: 127 | raise 128 | ifx = netns.link_lookup(ifname=ifname)[0] 129 | ifs.append(Interface(ifx, internal=True)) 130 | try: 131 | netns.addr('add', address=self.service.tunnel_ip, 132 | prefixlen=32, index=ifx) 133 | except pyroute2.netlink.exceptions.NetlinkError as e: 134 | if e.code != errno.EEXIST: 135 | raise 136 | netns.link('set', state='up', index=ifx) 137 | if netns: 138 | netns.close() 139 | endpoint_map[self.service][self.endpoint] = ifs 140 | 141 | 142 | class RemoveEndpoint(object): 143 | """Remove tunnel to an old endpoint.""" 144 | 145 | def __init__(self, service, endpoint): 146 | self.service = service 147 | self.endpoint = endpoint 148 | 149 | def enact(self, endpoint_map, ip): 150 | print('REMOVE_TUNNEL', self.service, self.endpoint) 151 | 152 | # Open network namespace inside the endpoint if we have it. 153 | # If the pod is not local, we do not have it - but another tunnel 154 | # router will. 155 | netns = None 156 | try: 157 | netns = (pyroute2.NetNS(self.endpoint.networkNs) 158 | if self.endpoint.networkNs else None) 159 | except FileNotFoundError: 160 | # If the namespace has gone away the interface is also gone 161 | pass 162 | 163 | for iface in endpoint_map[self.service][self.endpoint]: 164 | if iface.internal and netns: 165 | print('REMOVE_POD_IFACE', self.service, self.endpoint, iface) 166 | netns.link('delete', index=iface.ifx) 167 | else: 168 | print('REMOVE_HOST_IFACE', self.service, self.endpoint, iface) 169 | ip.link('delete', index=iface.ifx) 170 | 171 | if netns: 172 | netns.close() 173 | del endpoint_map[self.service][self.endpoint] 174 | -------------------------------------------------------------------------------- /tunnel-router/router.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import collections 4 | import docker 5 | import iptc 6 | import os 7 | import pykube 8 | import pyroute2 9 | import random 10 | import time 11 | import traceback 12 | 13 | import change 14 | 15 | 16 | INGRESS_CHAIN = os.environ.get( 17 | 'TUNNEL_ROUTER_INGRESS_CHAIN', 'TUNNEL-INGRESS') 18 | FILTER_CHAIN = os.environ.get( 19 | 'TUNNEL_ROUTER_FILTER_CHAIN', 'TUNNEL-FILTER') 20 | TUNNEL_ANNOTATION = os.environ.get( 21 | 'TUNNEL_ROUTER_TUNNEL_ANNOTATION', 'cmd.nu/tunnel') 22 | TABLE_OFFSET = os.environ.get( 23 | 'TUNNEL_ROUTER_TABLE_OFFSET', '1') 24 | 25 | 26 | Service = collections.namedtuple('Service', ('name', 'namespace', 'tunnel_ip')) 27 | Endpoint = collections.namedtuple('Endpoint', ('ip', 'networkNs')) 28 | 29 | 30 | def create_ingress_chain(): 31 | """Ingress chain marks all packets for tunnel ingress.""" 32 | rule = iptc.Rule() 33 | t = rule.create_target('HMARK') 34 | 35 | t.hmark_tuple = 'src,dst,sport,dport' 36 | t.hmark_mod = str(change.BUCKETS) 37 | t.hmark_offset = TABLE_OFFSET 38 | t.hmark_rnd = str(random.randint(1, 65535)) 39 | 40 | mangle_table = iptc.Table(iptc.Table.MANGLE) 41 | ingress_chain = iptc.Chain(mangle_table, INGRESS_CHAIN) 42 | if mangle_table.is_chain(INGRESS_CHAIN): 43 | ingress_chain.flush() 44 | else: 45 | ingress_chain = mangle_table.create_chain(INGRESS_CHAIN) 46 | ingress_chain.insert_rule(rule) 47 | return ingress_chain 48 | 49 | 50 | def create_ingress_filter_chain(): 51 | """Ingress filter chain matches the tunnel VIPs.""" 52 | mangle_table = iptc.Table(iptc.Table.MANGLE) 53 | filter_chain = iptc.Chain(mangle_table, FILTER_CHAIN) 54 | if mangle_table.is_chain(FILTER_CHAIN): 55 | filter_chain.flush() 56 | else: 57 | filter_chain = mangle_table.create_chain(FILTER_CHAIN) 58 | return filter_chain 59 | 60 | 61 | def register_ingress(): 62 | """Insert rules for packets to move through the ingress filter.""" 63 | for c in ['PREROUTING', 'OUTPUT']: 64 | chain = iptc.Chain(iptc.Table(iptc.Table.MANGLE), c) 65 | for rule in chain.rules: 66 | if rule.target.name == FILTER_CHAIN: 67 | # Already registered 68 | break 69 | else: 70 | rule = iptc.Rule() 71 | t = rule.create_target(FILTER_CHAIN) 72 | chain.insert_rule(rule) 73 | 74 | 75 | def get_services(api): 76 | """Return set of services.""" 77 | filters = set() 78 | for svc in pykube.Service.objects(api).filter(namespace=pykube.all): 79 | annotations = svc.metadata.get('annotations', {}) 80 | tunnel_ip = annotations.get(TUNNEL_ANNOTATION, None) 81 | if tunnel_ip is None: 82 | continue 83 | filters.add((Service(svc.name, svc.metadata['namespace'], tunnel_ip))) 84 | return filters 85 | 86 | 87 | def docker_container_to_netns(container_id): 88 | try: 89 | docker_id = container_id[len('docker://'):] 90 | client = docker.from_env() 91 | container = client.containers.get(docker_id) 92 | pid = container.attrs['State']['Pid'] 93 | return '/proc/%d/ns/net' % pid 94 | except docker.errors.NotFound: 95 | # Container is not local 96 | return None 97 | 98 | 99 | def container_to_netns(container_id): 100 | if container_id.startswith('docker://'): 101 | return docker_container_to_netns(container_id) 102 | else: 103 | print('Unknown container family:', container_id) 104 | return None 105 | 106 | 107 | def get_endpoints(api, services): 108 | """Return map of (service) = set(Endpoint).""" 109 | 110 | # Create a fast-lookup for (svc, ns) -> Service object 111 | lookup_map = {(x.name, x.namespace): x for x in services} 112 | 113 | pods = {} 114 | for pod in pykube.Pod.objects(api).filter(namespace=pykube.all): 115 | pods[pod.metadata['uid']] = pod 116 | 117 | endpoints = {} 118 | for endp in pykube.Endpoint.objects(api).filter(namespace=pykube.all): 119 | svc = lookup_map.get( 120 | (endp.metadata['name'], endp.metadata['namespace']), None) 121 | if svc is None: 122 | continue 123 | ips = set() 124 | subsets = endp.obj['subsets'] 125 | for s in subsets: 126 | for address in s['addresses']: 127 | pod_uid = address['targetRef']['uid'] 128 | pod = pods[pod_uid] 129 | container_status = pod.obj['status']['containerStatuses'][0] 130 | netns = container_to_netns(container_status['containerID']) 131 | ips.add(Endpoint(address['ip'], netns)) 132 | endpoints[svc] = ips 133 | return endpoints 134 | 135 | 136 | def calculate_filter_changes(api, service_map): 137 | # Calculate filter changes 138 | new_services = get_services(api) 139 | current_services = set(service_map.keys()) 140 | removed_services = current_services - new_services 141 | added_services = new_services - current_services 142 | for svc in added_services: 143 | yield change.AddService(svc) 144 | for svc in removed_services: 145 | yield change.RemoveService(svc) 146 | 147 | 148 | def calculate_routing_changes(api, endpoint_map, service_filter): 149 | # Calculate routing balancing changes 150 | new_endpoints_map = get_endpoints(api, service_filter) 151 | 152 | # Endppint changes in already known, or new, services 153 | for svc, new_endpoints in new_endpoints_map.items(): 154 | current_endpoints = set(endpoint_map.get(svc, dict()).keys()) 155 | added_endpoints = new_endpoints - current_endpoints 156 | removed_endpoints = current_endpoints - new_endpoints 157 | for endpoint in added_endpoints: 158 | yield change.AddEndpoint(svc, endpoint) 159 | for endpoint in removed_endpoints: 160 | yield change.RemoveEndpoint(svc, endpoint) 161 | if current_endpoints != new_endpoints: 162 | yield change.RefreshEndpoints(svc) 163 | 164 | # Purge empty endpoint services 165 | removed_services = set(endpoint_map.keys()) - set(new_endpoints_map.keys()) 166 | for svc in removed_services: 167 | for endpoint in endpoint_map[svc].keys(): 168 | yield change.RemoveEndpoint(svc, endpoint) 169 | yield change.RefreshEndpoints(svc) 170 | 171 | 172 | def purge_old_tunnels(): 173 | ip = pyroute2.IPRoute() 174 | for link in ip.get_links(): 175 | ifname = link.get_attr('IFLA_IFNAME') 176 | if ifname.startswith(change.TUNNEL_PREFIX): 177 | ip.link('del', ifname=ifname) 178 | 179 | 180 | def create_iproute_rules(): 181 | ip = pyroute2.IPRoute() 182 | for i in range(change.BUCKETS): 183 | try: 184 | ip.rule('add', table=(i+1), fwmark=(i+1)) 185 | except pyroute2.netlink.exceptions.NetlinkError: 186 | # Assume it already exists 187 | pass 188 | 189 | 190 | def loop(ingress_chain, filter_chain, service_map, endpoint_map): 191 | print('Starting poll loop for Kubernetes services') 192 | kube_creds = None 193 | if 'KUBECONFIG' in os.environ: 194 | kube_creds = pykube.KubeConfig.from_file(os.environ['KUBECONFIG']) 195 | else: 196 | kube_creds = pykube.KubeConfig.from_service_account() 197 | api = pykube.HTTPClient(kube_creds) 198 | 199 | ip = pyroute2.IPRoute() 200 | while True: 201 | filter_changes = calculate_filter_changes(api, service_map) 202 | for c in filter_changes: 203 | c.enact(service_map, filter_chain, ingress_chain) 204 | 205 | routing_changes = calculate_routing_changes( 206 | api, endpoint_map, service_map.keys()) 207 | 208 | for c in routing_changes: 209 | c.enact(endpoint_map, ip) 210 | 211 | time.sleep(1) 212 | 213 | 214 | if __name__ == '__main__': 215 | print('Creating ingress chain') 216 | ingress_chain = create_ingress_chain() 217 | 218 | print('Creating ingress filter chain') 219 | filter_chain = create_ingress_filter_chain() 220 | 221 | print('Registering ingress') 222 | register_ingress() 223 | 224 | print('Purging old tunnels') 225 | purge_old_tunnels() 226 | 227 | print('Creating iproute rules') 228 | create_iproute_rules() 229 | 230 | # Map 1: Used to filter on IPs to ingress in the tunnels 231 | # Stored as (service, tunnel-ip) = iptc.Rule 232 | service_map = {} 233 | 234 | # Map 2: Used to balance among endpoints (pods) 235 | # Stored as (service) = {pod: tunnel} 236 | # On changes on the above, recalculate the route maps 237 | endpoint_map = collections.defaultdict(dict) 238 | 239 | while True: 240 | try: 241 | loop(ingress_chain, filter_chain, service_map, endpoint_map) 242 | except KeyboardInterrupt: 243 | break 244 | except: 245 | print('Exception in main loop:') 246 | traceback.print_exc() 247 | time.sleep(1) 248 | -------------------------------------------------------------------------------- /static/cluster-layout.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | --------------------------------------------------------------------------------