├── .gitignore ├── README.md └── sbulb ├── __init__.py ├── __main__.py ├── bpf ├── checksum.c ├── ipv4.c ├── ipv6.c └── loadbalancer.c ├── tests ├── __init__.py ├── ipv4.cfg ├── ipv6.cfg ├── test_config.py ├── test_loadbalancer.py └── test_util.py └── util.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.ini 2 | /tmp/ 3 | /.project 4 | /.pydevproject 5 | /.settings 6 | /**/__pycache/**/* 7 | __pycache__/ -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # udploadbalancer 2 | An UDP load-balancer prototype using bcc (XDP/Bpf) 3 | 4 | ``` 5 | usage: sbulb [-h] -vs VIRTUAL_SERVER 6 | (-rs REAL_SERVER [REAL_SERVER ...] | -cfg CONFIG_FILE) -p PORT 7 | [PORT ...] [-d {0,1,2,3,4}] 8 | [-l {CRITICAL,ERROR,WARNING,INFO,DEBUG,TRACE}] [-mp MAX_PORTS] 9 | [-mrs MAX_REALSERVERS] [-ma MAX_ASSOCIATIONS] 10 | ifnet 11 | 12 | positional arguments: 13 | ifnet network interface to load balance (e.g. eth0) 14 | 15 | optional arguments: 16 | -h, --help show this help message and exit 17 | -vs VIRTUAL_SERVER, --virtual_server VIRTUAL_SERVER 18 | Virtual server address (e.g. 10.40.0.1) 19 | -rs REAL_SERVER [REAL_SERVER ...], --real_server REAL_SERVER [REAL_SERVER ...] 20 | Real server address(es) (e.g. 10.40.0.2 10.40.0.3) 21 | -cfg CONFIG_FILE, --config_file CONFIG_FILE 22 | a path to a file containing real server address(es). 23 | File will be polled each second for modification and configuration 24 | updated dynamically. A file content example : 25 | 26 | [Real Servers] 27 | 10.0.0.4 28 | 10.0.0.2 29 | 10.0.0.6 30 | 31 | -p PORT [PORT ...], --port PORT [PORT ...] 32 | UDP port(s) to load balance 33 | -d {0,1,2,3,4}, --debug {0,1,2,3,4} 34 | Use to set bpf verbosity, 0 is minimal. (default: 0) 35 | -l {CRITICAL,ERROR,WARNING,INFO,DEBUG,TRACE}, --loglevel {CRITICAL,ERROR,WARNING,INFO,DEBUG,TRACE} 36 | Use to set logging verbosity. (default: ERROR) 37 | -mp MAX_PORTS, --max_ports MAX_PORTS 38 | Set the maximum number of port to load balance. (default: 16) 39 | -mrs MAX_REALSERVERS, --max_realservers MAX_REALSERVERS 40 | Set the maximum number of real servers. (default: 32) 41 | -ma MAX_ASSOCIATIONS, --max_associations MAX_ASSOCIATIONS 42 | Set the maximum number of associations. (default: 1048576) 43 | This defined the maximum number of foreign peers supported at the same time. 44 | ``` 45 | Eg : `sudo python3 -m sbulb eth0 -vs 10.188.7.99 -rs 10.188.100.163 10.188.100.230 -p 5683 5684` 46 | 47 | # Behavior 48 | This load balancer can be considered as a Layer-4 NAT load-balancer as it only modifies IP address. 49 | 50 | For ingress traffic : 51 | - we search if we have a `clientip:port/realserverip` association. 52 | - if yes, we modify destination address (**dest NAT**) replacing **virtual IP address** by the **real server IP one**. 53 | - if no, we pick a real server and create a new association, and do dest NAT as above. 54 | 55 | For egress traffic : 56 | - we search if we have an `clientip:port/realserverip` association. 57 | - if yes and packet comes from the associated real server , we modify source address (**source NAT**) replacing real server address by the **virtual server ip address**. 58 | - if yes and packet comes from "not associated" real server, we drop the packet 59 | - if no, we create a new association using the source IP address(**real server IP address**) and modifying source address(**source NAT**) by the **virtual server ip address**. 60 | 61 | We keep this association is a large LRU map as long as possible, meaning the oldest association is only removed if LRU map is full and new association must be created. 62 | 63 | The algorithm used is [a simple round-robin](https://github.com/sbernard31/udploadbalancer/issues/8). 64 | 65 | :warning: All packets from the realservers to the client must go through the udp load-balancer machine/director ([like with LVS-NAT](http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-NAT.html#NAT_default_gw)). 66 | 67 | # Why create a new load balancer ? 68 | In a cluster, generally the good practice is to share states between each server instances, but sometime some states can not be shared... 69 | E.g. a cluster of servers which can not share DTLS connection, in this case you want to always send packet from a given client to the same server to limit the number of handshakes. 70 | To do that you need to create **a long-lived association** between the client and the server, but most of the UDP loadbalancer are thougth to have ephemere association. Most of the time this association lifetime can be configured and you can set a large value, but here thanks to the LRU map we can keep the association as long as we can. 71 | 72 | The other point is **server initiated communication**. We want to be able to initiate communication from a server exactly as if communication was initiated by a client. Meaning same association table is used. 73 | 74 | # Limitation 75 | This is a simple load-balancer and so it have some limitations : 76 | 77 | - All traffic (ingress and egress) should be handled by the same network interface. 78 | - All traffic should go to the same ethernet gateway (which is the case most of the time). 79 | - Does not support IP fragmentation. 80 | - Does not support IP packet with header options for now. Meaning IP header size (number of 32 bits word) must be set to 5. 81 | 82 | # Requirements & dependencies 83 | You need : 84 | - a recent linux kernel to be able to launch xdp/bpf code. (currently tested with 4.19.x package) 85 | - [bcc](https://github.com/iovisor/bcc) installed. (currently tested with v0.8 : package [python3-bpfcc](https://packages.debian.org/search?suite=all§ion=all&arch=any&searchon=names&keywords=python3-bpfcc) on debian) 86 | - linux-headers installed to allow bcc to compile bpf code. 87 | 88 | # Usage with systemd 89 | 90 | Sbulb supports the sd_notify(3) mechanism, but does not require systemd or any systemd library to run. This allows sbulb to notify systemd when it is ready to accept connections. To use this feature, you can write a service like this: 91 | 92 | [Unit] 93 | Description=UDP Load Balancer 94 | Wants=network-online.target 95 | [Service] 96 | Type=notify 97 | NotifyAccess=all 98 | Environment=PYTHONUNBUFFERED=1 99 | ExecStart=/usr/bin/python3 -m slulb args... 100 | [Install] 101 | WantedBy=multi-user.target 102 | 103 | # Performance 104 | See [our wiki page](https://github.com/AirVantage/sbulb/wiki/Benchmark) about that. 105 | 106 | # Unit Tests 107 | To launch unit tests : 108 | 109 | ``` 110 | sudo python3 -m unittest # all tests 111 | sudo python3 -m unittest sbulb.tests.IPv4TestCase # only 1 test case 112 | sudo python3 -m unittest sbulb.tests.IPv4TestCase.test_lru # only 1 test 113 | ``` 114 | Tests needs [bcc](https://github.com/iovisor/bcc) v0.14 ([python3-bpfcc](https://packages.debian.org/buster-backports/python3-bpfcc)) and [scapy](https://scapy.net/) ([python3-scapy](https://packages.debian.org/buster/python3-scapy)). 115 | 116 | # XDP/Bpf 117 | 118 | Why XDP/Bpf ? [Why is the kernel community replacing iptables with BPF?](https://cilium.io/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables/). 119 | 120 | Read about XDP/Bpf : [Dive into BPF: a list of reading material](https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf/). 121 | 122 | Inspirations : 123 | - [bcc example / xdp drop_count](https://github.com/iovisor/bcc/blob/master/examples/networking/xdp/xdp_drop_count.py) 124 | - [Netronome / l4lb demo](https://github.com/Netronome/bpf-samples/tree/master/l4lb) 125 | - [Facebook / katran](https://github.com/facebookincubator/katran) 126 | - [xdp_load_balancer proto](https://gist.github.com/summerwind/080750455a396a1b1ba78938b3178f6b) 127 | - [Cilium / cilium](https://github.com/cilium/cilium) 128 | - [Polycube-network / polycube ](https://github.com/polycube-network/polycube) (see [loadbalancer services](https://github.com/polycube-network/polycube/tree/master/src/services)) 129 | 130 | Documentations : 131 | - [bcc](https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md) 132 | - [kernel version](https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md) 133 | - [bpf and xdp reference guide from cilium](https://cilium.readthedocs.io/en/v1.5/bpf/) 134 | 135 | Thesis : 136 | - [High-performance Load Balancer in IOVisor](https://webthesis.biblio.polito.it/7498/1/tesi.pdf) 137 | 138 | -------------------------------------------------------------------------------- /sbulb/__init__.py: -------------------------------------------------------------------------------- 1 | import atexit 2 | from bcc import BPF 3 | from builtins import staticmethod 4 | import configparser 5 | import ctypes 6 | from dataclasses import dataclass, field 7 | from enum import Enum 8 | from ipaddress import _IPAddressBase # @UnusedImport 9 | import ipaddress 10 | import logging 11 | import os 12 | import signal 13 | import socket 14 | import subprocess 15 | from typing import List # @UnusedImport 16 | 17 | from sbulb.util import ip_mac_tostr, ip_strton, ip_ntostr 18 | 19 | # Define pythons log level 20 | _log_level_name = ["CRITICAL", "ERROR", "WARNING", "INFO", "DEBUG", "TRACE"] 21 | logging.addLevelName(5, "TRACE") # add TRACE level 22 | 23 | 24 | @dataclass 25 | class Cfg: 26 | # network interface to attach xdp program 27 | ifnet: str = "" 28 | # virtual server IP address 29 | virtual_server_ip: _IPAddressBase = None 30 | # ports to load balanced 31 | ports: List[int] = field(default_factory=list) 32 | # list of real server IP addresses 33 | real_server_ips: List[_IPAddressBase] = field(default_factory=list) 34 | # Name of file containing list of real servers 35 | config_file :str = None 36 | config_file_mtime :int = 0 37 | # the current ip version used (4 or 6) 38 | ip_version: int = 4 39 | # bpf verbosity 40 | debug: int = 0 41 | # log level to used 42 | loglevel: str = "ERROR" 43 | # maximum number of load balanced port 44 | max_ports: int = 5 45 | # maximum number of real servers 46 | max_realservers: int = 5 47 | # maximum number of associations 48 | max_associations: int = 10000 49 | 50 | def validate(self, apply_config_file=False): 51 | # TODO can we check ifnet exits? 52 | # check ip version 53 | if self.ip_version not in (4, 6): 54 | raise ValueError( 55 | "Invalid config : Invalid ip verison {} should be 4 or 6)", 56 | self.ip_version) 57 | 58 | # check virtual server 59 | if self.virtual_server_ip.version is not self.ip_version : 60 | raise ValueError( 61 | "Virtual server ip {} seems to be ipv{} address. ipv{} is expected" 62 | .format(self.virtual_server_ip, 63 | self.virtual_server_ip.version, 64 | self.ip_version)) 65 | 66 | # check real server 67 | if self.config_file : 68 | if apply_config_file : 69 | print ("\nLoading real servers from {} file ..."\ 70 | .format(self.config_file)) 71 | self.real_server_ips = self.load_real_server() 72 | # update last load time (see config_file_changed) 73 | self.config_file_mtime = os.stat(self.config_file).st_mtime 74 | for s in self.real_server_ips: 75 | print(" {}".format(s)) 76 | print ("...real servers loaded.") 77 | else: 78 | self.load_real_server() 79 | else: 80 | self._check_real_server(self.real_server_ips) 81 | 82 | # check ports 83 | for p in self.ports: 84 | if p not in range(0, 65535): 85 | raise ValueError( 86 | "Invalid port {} should be in range 0..65535." \ 87 | .format(p)) 88 | 89 | if len(self.ports) > self.max_ports: 90 | raise ValueError( 91 | "Too many ports, {} ports configured, {} maximum allowed." \ 92 | .format(len(self.ports), self.max_ports)) 93 | # check debug 94 | if self.debug not in range(0, 5): 95 | raise ValueError("Invalid debug value {}, must be between 0..5" \ 96 | .format(self.debug)) 97 | 98 | # check log level 99 | if self.loglevel not in _log_level_name: 100 | raise ValueError("Invalid loglevel value {}, must be {}" \ 101 | .format(self.loglevel, _log_level_name)) 102 | 103 | def config_file_changed(self): 104 | new_mtime = os.stat(self.config_file).st_mtime 105 | changed = new_mtime != self.config_file_mtime 106 | self.config_file_mtime = new_mtime 107 | return changed 108 | 109 | def load_real_server(self): 110 | config = configparser.ConfigParser(allow_no_value=True, 111 | delimiters=("=")) 112 | with open(self.config_file, 'r') as f : 113 | config.read_file(f) 114 | 115 | rs = [] 116 | for ip in config["Real Servers"]: 117 | rs.append(ipaddress.ip_address(ip)) 118 | if len(rs) == 0: 119 | raise ValueError ("real server list must not be empty") 120 | self._check_real_server(rs) 121 | return rs 122 | 123 | def _check_real_server(self, real_servers): 124 | if len(real_servers) > self.max_realservers: 125 | raise ValueError("too many real servers, \ 126 | {} real servers configured,\ 127 | {} maximum allowed" \ 128 | .format(len(real_servers), 129 | self.max_realservers)) 130 | for ip in real_servers: 131 | if ip.version is not self.ip_version: 132 | raise ValueError("Real server ip {} seems to be ipv{} address. \ 133 | ipv{} is expected".format(ip, ip.version, 134 | self.ip_version)) 135 | 136 | 137 | class LoadBalancer: 138 | """An UDP Loadbalancer based on bcc(xdp)""" 139 | bpf: BPF = None # bpf pointer 140 | func = None # xdp function 141 | cfg: Cfg = None # config 142 | # Bpf Maps 143 | virtual_server_map = None 144 | ports_map = None 145 | real_servers_array = None 146 | real_servers_map = None 147 | 148 | def __init__(self, cfg:Cfg): 149 | self.cfg = cfg 150 | cfg.validate(apply_config_file=True); 151 | 152 | # Build C flags 153 | cflags = LogCode.toMacros() 154 | for levelName in _log_level_name: 155 | cflags.append("-D{}={}".format(levelName, 156 | logging.getLevelName(levelName))) 157 | cflags.append("-D{}={}".format("LOGLEVEL", 158 | logging.getLevelName(cfg.loglevel))) 159 | cflags.append("-D{}={}".format("MAX_PORTS", cfg.max_ports)) 160 | cflags.append("-D{}={}".format("MAX_REALSERVERS", cfg.max_realservers)) 161 | cflags.append("-D{}={}".format("MAX_ASSOCIATIONS", 162 | cfg.max_associations)) 163 | if cfg.ip_version is 6: 164 | cflags.append("-DIPV6=true"); 165 | 166 | # Compile bpf program 167 | print("\nCompiling bpf code & apply config ...") 168 | print("log level : {}".format(cfg.loglevel)) 169 | print("max ports : {}".format(cfg.max_ports)) 170 | print("max realservers : {}".format(cfg.max_realservers)) 171 | print("max associations : {}".format(cfg.max_associations)) 172 | self.bpf = BPF(src_file=b"sbulb/bpf/loadbalancer.c", 173 | debug=cfg.debug, cflags=cflags) 174 | self.func = self.bpf.load_func(b"xdp_prog", BPF.XDP) 175 | 176 | # Apply config to bpf maps 177 | self.virtual_server_map = self.bpf[b"virtualServer"] 178 | self.ports_map = self.bpf[b"ports"] 179 | self.real_servers_array = self.bpf[b"realServersArray"] 180 | self.real_servers_map = self.bpf[b"realServersMap"] 181 | self.virtual_server_map[self.virtual_server_map.Key(0)] = \ 182 | ip_strton(self.cfg.virtual_server_ip) 183 | for port in self.cfg.ports: 184 | self.ports_map[ self.ports_map.Key(socket.htons(port))] = \ 185 | self.ports_map.Leaf(True) 186 | self._update_real_server([], self.cfg.real_server_ips) 187 | 188 | # Display current map state 189 | self._dump_real_server_map() 190 | print("... compilation succeed & configuration applied.") 191 | 192 | def _update_real_server(self, old_server_ips, new_server_ips): 193 | """Update 'real server' bpf map content.""" 194 | nbOld = len(old_server_ips) 195 | nbNew = len(new_server_ips) 196 | 197 | for i in range(max(nbOld, nbNew)): 198 | if i >= nbOld: 199 | # addition 200 | new_server_ip = new_server_ips[i] 201 | new_server_nip = ip_strton(new_server_ip) 202 | self.real_servers_map[new_server_nip] = new_server_nip 203 | self.real_servers_array[self.real_servers_array.Key(i)] = \ 204 | new_server_nip 205 | print("Add {} at index {}".format(new_server_ip, i)) 206 | elif i >= nbNew: 207 | # deletion 208 | old_server_ip = old_server_ips[i] 209 | if old_server_ip in new_server_ips: 210 | print ("don't remove {} from map".format(old_server_ip)) 211 | else: 212 | old_server_nip = ip_strton(old_server_ip) 213 | del self.real_servers_map[old_server_nip] 214 | del self.real_servers_array[self.real_servers_array.Key(i)] 215 | print("delete {} at index {}".format(old_server_ip, i)) 216 | else: 217 | # update 218 | new_server_ip = new_server_ips[i] 219 | old_server_ip = old_server_ips[i] 220 | if new_server_ip == old_server_ip: 221 | print ("No change for {} at index {}" \ 222 | .format(new_server_ip, i)) 223 | else: 224 | new_server_nip = ip_strton(new_server_ip) 225 | self.real_servers_map[new_server_nip] = new_server_nip 226 | self.real_servers_array[self.real_servers_array.Key(i)] = \ 227 | new_server_nip 228 | if old_server_ip in new_server_ips: 229 | print ("don't remove {} from map".format(old_server_ip)) 230 | else: 231 | old_server_nip = ip_strton(old_server_ip) 232 | del self.real_servers_map[old_server_nip] 233 | print("Update {} to {} at index {}" \ 234 | .format(old_server_ip, new_server_ip, i)) 235 | 236 | def _dump_real_server_map(self): 237 | """Dump 'real servers' bpf map content.""" 238 | for i, v in self.real_servers_array.iteritems(): 239 | print ("[{}]={}".format(i.value, ip_ntostr(v))) 240 | for i, v in self.real_servers_map.iteritems(): 241 | print ("[{}]={}".format(ip_ntostr(i), ip_ntostr(v))) 242 | 243 | def attach(self, detach_on_exit=True, notify_systemd=True): 244 | print("\nAttaching bpf code ...") 245 | self.bpf.attach_xdp(self.cfg.ifnet, self.func) 246 | if detach_on_exit: 247 | atexit.register(self.detach) 248 | 249 | # Support systemd notify services. 250 | if notify_systemd and 'NOTIFY_SOCKET' in os.environ: 251 | try: 252 | subprocess.call("systemd-notify --ready", shell=True) 253 | except: 254 | pass 255 | print("...bpf code attached.") 256 | 257 | def loop(self): 258 | print("\nLoop watching logs buffer and config file ...") 259 | # Program loop 260 | self._open_log_buffer() 261 | stopper = Stopper() 262 | while not stopper.isStopped(): 263 | # read and log perf_buffer 264 | self.bpf.perf_buffer_poll(1000) 265 | # watch if config file changed 266 | if self.cfg.config_file: 267 | if self.cfg.config_file_changed(): 268 | # load real server from config 269 | new_real_server_ips = None 270 | try: 271 | new_real_server_ips = self.cfg.load_real_server() 272 | except Exception as e: 273 | new_real_server_ips = None 274 | print ("Unable to load config {} file : {}"\ 275 | .format(self.config_file, e)) 276 | print ("Old Config is keeping : {}".\ 277 | format(self.cfg.real_server_ips)) 278 | 279 | # if succeed try to update bpf map 280 | if new_real_server_ips is not None: 281 | print("Apply new config ...") 282 | self._update_real_server(self.cfg.real_server_ips, 283 | new_real_server_ips) 284 | self.cfg.real_server_ips = new_real_server_ips 285 | self._dump_real_server_map() 286 | print("... new config applied.") 287 | # DEBUG STUFF 288 | # (task, pid, cpu, flags, ts, msg) = 289 | # b.trace_fields(nonblocking = True) 290 | # while msg: 291 | # print("%s \n" % (msg)) 292 | # (task, pid, cpu, flags, ts, msg) = 293 | # b.trace_fields(nonblocking = True) 294 | print("... watching stopped.") 295 | 296 | def _open_log_buffer(self): 297 | ip_version = self.cfg.ip_version 298 | 299 | # Shared structure used for "logs" perf_buffer 300 | class LogEvent(ctypes.Structure): 301 | _fields_ = [ 302 | # code identied the kind of events 303 | ("code", ctypes.c_uint), 304 | # old/original packet addresses 305 | ("odmac", ctypes.c_ubyte * 6), 306 | ("osmac", ctypes.c_ubyte * 6), 307 | ("odaddr", ctypes.c_ubyte * 16 if ip_version is 6 308 | else ctypes.c_uint), 309 | ("osaddr", ctypes.c_ubyte * 16 if ip_version is 6 310 | else ctypes.c_uint), 311 | # new/modified packet addresses 312 | ("ndmac", ctypes.c_ubyte * 6), 313 | ("nsmac", ctypes.c_ubyte * 6), 314 | ("ndaddr", ctypes.c_ubyte * 16 if ip_version is 6 315 | else ctypes.c_uint), 316 | ("nsaddr", ctypes.c_ubyte * 16 if ip_version is 6 317 | else ctypes.c_uint), 318 | ] 319 | 320 | # Define Utility function to print log 321 | def print_event(cpu, data, size): 322 | event = ctypes.cast(data, ctypes.POINTER(LogEvent)).contents 323 | LogCode(event.code).log(event, ip_version) 324 | 325 | # Open perf buffer dedicated to logs 326 | self.bpf["logs"].open_perf_buffer(print_event) 327 | 328 | def detach(self): 329 | print ("\n Detaching bpf code ...") 330 | self.bpf.remove_xdp(self.cfg.ifnet) 331 | print (" ... code detached.") 332 | 333 | def flush_log_buffer(self): 334 | # read and log perf_buffer 335 | self.bpf.perf_buffer_poll(1000) 336 | 337 | 338 | # Handle Signal 339 | class Stopper: 340 | _stop = False 341 | 342 | def __init__(self): 343 | signal.signal(signal.SIGTERM, self.stop) 344 | signal.signal(signal.SIGINT, self.stop) # keyboard interruption 345 | 346 | def stop(self, signum, frame): # @UnusedVariable 347 | print ("\n... stopping by signal {}({}) ..." \ 348 | .format(signal.Signals(signum).name, signum)); 349 | self._stop = True 350 | 351 | def isStopped(self): 352 | return self._stop 353 | 354 | 355 | # Define log code constant 356 | class Direction(Enum): 357 | INGRESS = 1, 358 | EGRESS = 2, 359 | UNKNOWN = 3, 360 | 361 | 362 | class Kind(Enum): 363 | NOTIP = 1, 364 | UNCHANGED = 2, 365 | NAT = 3, 366 | 367 | 368 | class LogCode(Enum): 369 | # NOT IP (message with out address) 370 | INVALID_ETH_SIZE = "{} <-> {} Invalid size for ethernet packet", \ 371 | Direction.UNKNOWN, Kind.NOTIP 372 | NOT_IP_V4 = "{} <-> {} Not IPv4 packet", Direction.UNKNOWN, Kind.NOTIP 373 | NOT_IP_V6 = "{} <-> {} Not IPv6 packet", Direction.UNKNOWN, Kind.NOTIP 374 | UNEXPECTED_IPHDR_PARSING_ERR = \ 375 | "{} <-> {} Unexpected error return by ip header parsing", \ 376 | Direction.UNKNOWN, Kind.NOTIP 377 | 378 | # UNCHANGED (message with origin address only) 379 | INVALID_IP_SIZE = "{} <─> {} Invalid size for IP packet", \ 380 | Direction.UNKNOWN, Kind.UNCHANGED 381 | TOO_SMALL_IP_HEADER = "{} <─> {} Too small IP header", \ 382 | Direction.UNKNOWN, Kind.UNCHANGED 383 | NOT_UDP = "{} <─> {} Not UDP packet", Direction.UNKNOWN, Kind.UNCHANGED 384 | TOO_BIG_IP_HEADER = "{} <─> {} Too big IP header", \ 385 | Direction.UNKNOWN, Kind.UNCHANGED 386 | FRAGMENTED_IP_PACKET = "{} <─> {} Fragmented IP packet", \ 387 | Direction.UNKNOWN, Kind.UNCHANGED 388 | INVALID_UDP_SIZE = "{} <─> {} Invalid size for UDP packet", \ 389 | Direction.UNKNOWN, Kind.UNCHANGED 390 | NO_VIRTUAL_SERVER = "{} <─> {} No virtual server configured", \ 391 | Direction.UNKNOWN, Kind.UNCHANGED 392 | UNHANDLED_TRAFFIC = "{} <─> {} Unhandled traffic", \ 393 | Direction.UNKNOWN, Kind.UNCHANGED 394 | LIFETIME_EXPIRED = "{} <-> {} TTL or hoplimit expired", \ 395 | Direction.UNKNOWN, Kind.UNCHANGED 396 | 397 | INGRESS_NOT_HANDLED_PORT = "{} ──> {} Unhandled port", \ 398 | Direction.INGRESS, Kind.UNCHANGED 399 | INGRESS_CANNOT_CREATE_ASSO = "{} ──> {} Unable to create association", \ 400 | Direction.INGRESS, Kind.UNCHANGED 401 | INGRESS_CANNOT_CREATE_ASSO2 = \ 402 | "{} ──> {} Unable to create association (MUST not happened)", \ 403 | Direction.INGRESS, Kind.UNCHANGED 404 | 405 | EGRESS_NOT_HANDLED_PORT = "{} <── {} Unhandled port", \ 406 | Direction.EGRESS, Kind.UNCHANGED 407 | EGRESS_CANNOT_CREATE_ASSO = "{} <── {} Unable to create association", \ 408 | Direction.EGRESS, Kind.UNCHANGED 409 | EGRESS_NOT_AUTHORIZED = "{} <── {} Not associated real server", \ 410 | Direction.EGRESS, Kind.UNCHANGED 411 | 412 | # NAT (message with origin an destination addresses) 413 | INGRESS_NEW_NAT = \ 414 | "{} ─┐ {} Destination NAT\n{} └> {} (NEW ASSOCIATION)", \ 415 | Direction.INGRESS, Kind.NAT 416 | INGRESS_REUSED_NAT = \ 417 | "{} ─┐ {} Destination NAT\n{} └> {} (REUSED ASSOCIATION)", \ 418 | Direction.INGRESS, Kind.NAT 419 | 420 | EGRESS_NEW_NAT = "{} ┌ {} Source NAT\n{} <─┘ {} (NEW ASSOCIATION)" , \ 421 | Direction.EGRESS, Kind.NAT 422 | EGRESS_REUSED_NAT = \ 423 | "{} ┌ {} Source NAT\n{} <─┘ {} (REUSED ASSOCIATION)", \ 424 | Direction.EGRESS, Kind.NAT 425 | 426 | def __new__(cls, msg, direction, kind): 427 | value = len(cls.__members__) + 1 428 | obj = object.__new__(cls) 429 | obj._value_ = value 430 | obj.msg = msg 431 | obj.direction = direction 432 | obj.kind = kind 433 | return obj 434 | 435 | def log(self, event, ip_version): 436 | """Print log message.""" 437 | mac_ip_str_size = 57 if ip_version is 6 else 33 438 | if self.kind is Kind.NAT: 439 | if self.direction is Direction.INGRESS: 440 | print(self.msg.format( 441 | ip_mac_tostr(event.osmac, 442 | event.osaddr).rjust(mac_ip_str_size), 443 | ip_mac_tostr(event.odmac, 444 | event.odaddr).ljust(mac_ip_str_size), 445 | " "*mac_ip_str_size, 446 | ip_mac_tostr(event.ndmac, 447 | event.ndaddr).ljust(mac_ip_str_size))) 448 | elif self.direction is Direction.EGRESS: 449 | print(self.msg.format( 450 | " "*mac_ip_str_size, 451 | ip_mac_tostr(event.osmac, 452 | event.osaddr).ljust(mac_ip_str_size), 453 | ip_mac_tostr(event.ndmac, 454 | event.ndaddr).rjust(mac_ip_str_size), 455 | ip_mac_tostr(event.nsmac, 456 | event.nsaddr).ljust(mac_ip_str_size))) 457 | else: 458 | print("Invalid direction for NAT log event:{}" \ 459 | .format(self.direction)) 460 | elif self.kind is Kind.UNCHANGED: 461 | if self.direction is Direction.INGRESS \ 462 | or self.direction is Direction.UNKNOWN : 463 | print(self.msg.format( 464 | ip_mac_tostr(event.osmac, 465 | event.osaddr).rjust(mac_ip_str_size), 466 | ip_mac_tostr(event.odmac, 467 | event.odaddr).ljust(mac_ip_str_size))) 468 | elif self.direction is Direction.EGRESS: 469 | print(self.msg.format( 470 | ip_mac_tostr(event.odmac, 471 | event.odaddr).rjust(mac_ip_str_size), 472 | ip_mac_tostr(event.osmac, 473 | event.osaddr).ljust(mac_ip_str_size))) 474 | else: 475 | print("Invalid direction of UNCHANGED log event : {}" \ 476 | .format(self.direction)) 477 | elif self.kind is Kind.NOTIP: 478 | if self.direction is Direction.UNKNOWN: 479 | print(self.msg.format( 480 | " "*mac_ip_str_size, 481 | " "*mac_ip_str_size)) 482 | else: 483 | print("Invalid direction of NOT IP log event : {}"\ 484 | .format(self.direction)) 485 | else: 486 | print("Invalid kind of log event : {}".format(self.kind)) 487 | 488 | @staticmethod 489 | def toMacros(): 490 | """Export all logCode as C macro list.""" 491 | macros = [] 492 | for code in LogCode: 493 | macros.append("-D{}={}".format(code.name, code.value)) 494 | return macros 495 | -------------------------------------------------------------------------------- /sbulb/__main__.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | import argparse 4 | import ctypes 5 | import ipaddress 6 | from sbulb import Cfg, LoadBalancer 7 | 8 | 9 | # Custom argument parser 10 | def ip_parser(s): 11 | try: 12 | return ipaddress.ip_address(s) 13 | except Exception as e: 14 | raise argparse.ArgumentTypeError("Invalid IP address '{}' : {}" \ 15 | .format(s, str(e))) 16 | 17 | 18 | def positive_int(s): 19 | if not s.isdigit(): 20 | raise argparse.ArgumentTypeError("{} is not a valid positive int" \ 21 | .format(s)) 22 | try: 23 | i = int(s) 24 | except Exception as e: 25 | raise argparse.ArgumentTypeError("{} is not a valid positive int : {}" \ 26 | .format(s, str(e))) 27 | if i < 0: 28 | raise argparse.ArgumentTypeError("{} is not a valid positive int" \ 29 | .format(s)) 30 | # TODO It is not clear what is current mapsize limit for map allowed by BPF. 31 | # so for now just check it is an unsigned int... 32 | max_long_value = ctypes.c_uint(-1) 33 | if i > max_long_value.value : 34 | raise argparse.ArgumentTypeError(\ 35 | "{} is not a valid positive int, max value is {}"\ 36 | .format(s, max_long_value.value)) 37 | return i 38 | 39 | 40 | # Parse Arguments 41 | parser = argparse.ArgumentParser(prog="sbulb", 42 | formatter_class=argparse.RawTextHelpFormatter) 43 | 44 | parser.add_argument("ifnet", 45 | help="network interface to load balance (e.g. eth0)") 46 | parser.add_argument("-vs", "--virtual_server", type=ip_parser, required=True, 47 | help=" Virtual server address (e.g. 10.40.0.1)") 48 | group = parser.add_mutually_exclusive_group(required=True) 49 | group.add_argument("-rs", "--real_server", type=ip_parser, nargs='+', 50 | help=" Real server address(es) (e.g. 10.40.0.2 10.40.0.3)") 51 | group.add_argument("-cfg", "--config_file", type=argparse.FileType('r'), 52 | help=''' a path to a file containing real server address(es). 53 | File will be polled each second for modification and configuration 54 | updated dynamically. A file content example : 55 | 56 | [Real Servers] 57 | 10.0.0.4 58 | 10.0.0.2 59 | 10.0.0.6 60 | 61 | ''') 62 | parser.add_argument("-p", "--port", type=int, nargs='+', required=True, 63 | help=" UDP port(s) to load balance") 64 | parser.add_argument("-d", "--debug", type=int, 65 | choices=[0, 1, 2, 3, 4], default=0, 66 | help="Use to set bpf verbosity, 0 is minimal. (default: %(default)s)") 67 | parser.add_argument("-l", "--loglevel", default="ERROR", 68 | choices=["CRITICAL", "ERROR", "WARNING", "INFO", "DEBUG", "TRACE"], 69 | help="Use to set logging verbosity. (default: %(default)s)") 70 | parser.add_argument("-mp", "--max_ports", type=positive_int, default=16, 71 | help="Set the maximum number of port to load balance. (default: %(default)s)") 72 | parser.add_argument("-mrs", "--max_realservers", type=positive_int, default=32, 73 | help="Set the maximum number of real servers. (default: %(default)s)") 74 | parser.add_argument("-ma", "--max_associations", 75 | type=positive_int, default=1048576, 76 | help="Set the maximum number of associations. (default: %(default)s)\n\ 77 | This defined the maximum number of foreign peers supported at the same time.") 78 | 79 | args = parser.parse_args() 80 | 81 | # Get configuration from Arguments 82 | cfg = Cfg(); 83 | cfg.ifnet = args.ifnet 84 | cfg.virtual_server_ip = args.virtual_server 85 | cfg.ip_version = cfg.virtual_server_ip.version 86 | cfg.ports = args.port 87 | cfg.debug = args.debug 88 | cfg.loglevel = args.loglevel 89 | cfg.max_ports = args.max_ports 90 | cfg.max_realservers = args.max_realservers 91 | cfg.max_associations = args.max_associations 92 | cfg.real_server_ips = args.real_server 93 | if args.config_file: 94 | cfg.config_file = args.config_file.name 95 | 96 | try: 97 | cfg.validate() 98 | except ValueError as e: 99 | print("Invalid argument : {}".format(e)) 100 | exit() 101 | 102 | # Create load balancer 103 | loadbalancer = LoadBalancer(cfg); 104 | 105 | # Attach it to XDP 106 | loadbalancer.attach() 107 | 108 | # Launch it 109 | loadbalancer.loop() 110 | -------------------------------------------------------------------------------- /sbulb/bpf/checksum.c: -------------------------------------------------------------------------------- 1 | // Checksum utilities 2 | __attribute__((__always_inline__)) 3 | static inline __u16 csum_fold_helper(__u64 csum) { 4 | int i; 5 | #pragma unroll 6 | for (i = 0; i < 4; i ++) { 7 | if (csum >> 16) 8 | csum = (csum & 0xffff) + (csum >> 16); 9 | } 10 | return ~csum; 11 | } 12 | 13 | // Update checksum following RFC 1624 (Eqn. 3): https://tools.ietf.org/html/rfc1624 14 | // HC' = ~(~HC + ~m + m') 15 | // Where : 16 | // HC - old checksum in header 17 | // HC' - new checksum in header 18 | // m - old value 19 | // m' - new value 20 | __attribute__((__always_inline__)) 21 | static inline void update_csum(__u64 *csum, __be32 old_addr,__be32 new_addr ) { 22 | // ~HC 23 | *csum = ~*csum; 24 | *csum = *csum & 0xffff; 25 | // + ~m 26 | __u32 tmp; 27 | tmp = ~old_addr; 28 | *csum += tmp; 29 | // + m 30 | *csum += new_addr; 31 | // then fold and complement result ! 32 | *csum = csum_fold_helper(*csum); 33 | } 34 | -------------------------------------------------------------------------------- /sbulb/bpf/ipv4.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include "sbulb/bpf/checksum.c" 3 | 4 | typedef __be32 ip_addr; 5 | 6 | /* 0x3FFF mask to check for fragment offset field */ 7 | #define IP_FRAGMENTED 65343 8 | 9 | __attribute__((__always_inline__)) 10 | static inline bool compare_ip_addr(ip_addr *ipA, ip_addr *ipB) { 11 | return (* ipA) == (* ipB); 12 | } 13 | 14 | __attribute__((__always_inline__)) 15 | static inline void copy_ip_addr(ip_addr * dest, ip_addr * src) { 16 | (* dest) = (*src); 17 | } 18 | 19 | __attribute__((__always_inline__)) 20 | static inline int parse_ip_header(struct ethhdr * eth, void * data_end, struct udphdr **udp, ip_addr ** saddr, ip_addr ** daddr) { 21 | 22 | // Handle only IPv4 packets 23 | if (eth->h_proto != bpf_htons(ETH_P_IP)) { 24 | return NOT_IP_V4; 25 | } 26 | 27 | // https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/include/uapi/linux/ip.h 28 | struct iphdr *iph; 29 | iph = (struct iphdr *) (eth + 1); 30 | if ((void *) (iph + 1) > data_end) { 31 | return INVALID_IP_SIZE; 32 | } 33 | 34 | // Extract ip address 35 | (* saddr) = &iph->saddr; 36 | (* daddr) = &iph->daddr; 37 | 38 | // Minimum valid header length value is 5. 39 | // see (https://tools.ietf.org/html/rfc791#section-3.1) 40 | if (iph->ihl < 5) { 41 | return TOO_SMALL_IP_HEADER; 42 | } 43 | 44 | // Handle only UDP traffic 45 | if (iph->protocol != IPPROTO_UDP) { 46 | return NOT_UDP; 47 | } 48 | 49 | // IP header size is variable because of options field. 50 | // see (https://tools.ietf.org/html/rfc791#section-3.1) 51 | // TODO #16 support IP header with variable size ? 52 | if (iph->ihl != 5) { 53 | return TOO_BIG_IP_HEADER; 54 | } 55 | 56 | // Do not support fragmented packets 57 | if (iph->frag_off & IP_FRAGMENTED) { 58 | return FRAGMENTED_IP_PACKET; 59 | } 60 | 61 | // handle packet lifetime : https://tools.ietf.org/html/rfc791 62 | if (iph->ttl <= 0) 63 | return LIFETIME_EXPIRED; 64 | // TODO #15 we should maybe send an ICMP packet 65 | 66 | // https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/include/uapi/linux/udp.h 67 | // Extract UDP header 68 | (*udp) = (struct udphdr *) (iph + 1); 69 | return 0; 70 | } 71 | 72 | __attribute__((__always_inline__)) 73 | static inline void update_ip_checksum(struct ethhdr * eth, void * data_end, ip_addr old_addr, ip_addr new_addr) { 74 | struct iphdr *iph; 75 | iph = (struct iphdr *) (eth + 1); 76 | __u64 cs = iph->check; 77 | update_csum(&cs, old_addr, new_addr); 78 | iph->check = cs; 79 | } 80 | 81 | __attribute__((__always_inline__)) 82 | static inline int update_udp_checksum(__u64 cs, ip_addr old_addr, ip_addr new_addr) { 83 | update_csum(&cs , old_addr, new_addr); 84 | return cs; 85 | } 86 | 87 | __attribute__((__always_inline__)) 88 | static inline void decrease_packet_lifetime(struct ethhdr * eth){ 89 | struct iphdr *iph; 90 | iph = (struct iphdr *) (eth + 1); 91 | 92 | // from include/net/ip.h 93 | u32 check = (__force u32)iph->check; 94 | check += (__force u32)htons(0x0100); 95 | iph->check = (__force __sum16)(check + (check >= 0xFFFF)); 96 | 97 | --iph->ttl; 98 | } 99 | -------------------------------------------------------------------------------- /sbulb/bpf/ipv6.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include "sbulb/bpf/checksum.c" 3 | 4 | typedef struct in6_addr ip_addr; 5 | 6 | __attribute__((__always_inline__)) 7 | static inline bool compare_ip_addr(ip_addr *ipA, ip_addr *ipB) { 8 | for(int i = 0; i < 16 ; ++i) 9 | if (ipA->s6_addr[i] != ipB->s6_addr[i]) 10 | return false; 11 | return true; 12 | } 13 | 14 | __attribute__((__always_inline__)) 15 | static inline void copy_ip_addr(ip_addr * dest, ip_addr * src) { 16 | memcpy(dest,src,16); 17 | } 18 | 19 | __attribute__((__always_inline__)) 20 | static inline int parse_ip_header(struct ethhdr * eth, void * data_end, struct udphdr **udp, ip_addr ** saddr, ip_addr ** daddr) { 21 | // Handle only IPv6 packets 22 | if (eth->h_proto != bpf_htons(ETH_P_IPV6)) { 23 | return NOT_IP_V6; 24 | } 25 | 26 | // https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/include/uapi/linux/ipv6.h 27 | struct ipv6hdr *iph; 28 | iph = (struct ipv6hdr *) (eth + 1); 29 | if ((void *) (iph + 1) > data_end) { 30 | return INVALID_IP_SIZE; 31 | } 32 | 33 | // Extract ip address 34 | (* saddr) = &iph->saddr; 35 | (* daddr) = &iph->daddr; 36 | 37 | // Handle only UDP traffic 38 | if (iph->nexthdr != IPPROTO_UDP) { 39 | return NOT_UDP; 40 | } 41 | 42 | // handle packet lifetime : https://tools.ietf.org/html/rfc8200#section-3 43 | if (iph->hop_limit <= 0) 44 | return LIFETIME_EXPIRED; 45 | // TODO #15 we should maybe send an ICMP packet 46 | 47 | // https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/include/uapi/linux/udp.h 48 | // Extract UDP header 49 | (*udp) = (struct udphdr *) (iph + 1); 50 | return 0; 51 | } 52 | 53 | __attribute__((__always_inline__)) 54 | static inline void update_ip_checksum(struct ethhdr * eth, void * data_end, ip_addr old_addr, ip_addr new_addr) { 55 | // no ip checksum in ipv6 56 | } 57 | 58 | __attribute__((__always_inline__)) 59 | static inline int update_udp_checksum(__u64 cs, ip_addr old_addr, ip_addr new_addr) { 60 | for(int i = 0; i < 4 ; ++i) 61 | update_csum(&cs , old_addr.s6_addr32[i], new_addr.s6_addr32[i]); 62 | return cs; 63 | } 64 | 65 | __attribute__((__always_inline__)) 66 | static inline void decrease_packet_lifetime(struct ethhdr * eth) { 67 | struct ipv6hdr *iph; 68 | iph = (struct ipv6hdr *) (eth + 1); 69 | --iph->hop_limit; 70 | } 71 | -------------------------------------------------------------------------------- /sbulb/bpf/loadbalancer.c: -------------------------------------------------------------------------------- 1 | #define KBUILD_MODNAME "foo" 2 | #include 3 | #include 4 | #include 5 | #include 6 | 7 | // include ip handling util 8 | #ifdef IPV6 9 | #include "sbulb/bpf/ipv6.c" 10 | #else 11 | #include "sbulb/bpf/ipv4.c" 12 | #endif 13 | 14 | // keys for association table. 15 | struct associationKey { 16 | ip_addr ipAddr; 17 | __be16 port; 18 | }; 19 | 20 | // load balancer state. 21 | struct state { 22 | int nextRS; // new real server index 23 | }; 24 | 25 | // Log structure 26 | struct logEvent { 27 | // code identifing the kind of events 28 | int code; 29 | // old/original packet addresses 30 | unsigned char odmac[ETH_ALEN]; 31 | unsigned char osmac[ETH_ALEN]; 32 | ip_addr odaddr; 33 | ip_addr osaddr; 34 | // new/modified packet addresses 35 | unsigned char ndmac[ETH_ALEN]; 36 | unsigned char nsmac[ETH_ALEN]; 37 | ip_addr ndaddr; 38 | ip_addr nsaddr; 39 | }; 40 | BPF_PERF_OUTPUT(logs); 41 | // Logging function 42 | __attribute__((__always_inline__)) 43 | static inline void log(unsigned char level, struct xdp_md *ctx, unsigned char code, struct logEvent * logEvent) { 44 | if (level >= LOGLEVEL) { 45 | logEvent->code = code; 46 | logs.perf_submit(ctx, logEvent, sizeof(*logEvent)); 47 | } 48 | } 49 | 50 | // A map which contains virtual server IP address 51 | BPF_HASH(virtualServer, int, ip_addr, 1); 52 | // A map which contains port to redirect 53 | BPF_HASH(ports, __be16, int, MAX_PORTS); 54 | // maps which contains real server IP addresses 55 | BPF_HASH(realServersArray, int, ip_addr, MAX_REALSERVERS); 56 | BPF_HASH(realServersMap, ip_addr, ip_addr, MAX_REALSERVERS); 57 | // association tables : link a foreign peer to a real server IP address 58 | BPF_TABLE("lru_hash", struct associationKey, ip_addr, associationTable, MAX_ASSOCIATIONS); 59 | // load balancer state 60 | BPF_HASH(lbState, int, struct state, 1); 61 | 62 | __attribute__((__always_inline__)) 63 | static inline ip_addr * new_association(struct associationKey * k) { 64 | // Get index of real server which must handle this peer 65 | int zero = 0; 66 | struct state * state = lbState.lookup(&zero); 67 | int rsIndex = 0; 68 | if (state != NULL) 69 | rsIndex = state->nextRS; 70 | 71 | // Get real server from this index. 72 | ip_addr * rsIp = realServersArray.lookup(&rsIndex); 73 | if (rsIp == NULL) { 74 | rsIndex = 0; // probably index out of bound so we restart from 0 75 | rsIp = realServersArray.lookup(&rsIndex); 76 | if (rsIp == NULL) 77 | return NULL; // XDP_ABORTED ? 78 | } 79 | 80 | // Update state (increment real server index) 81 | struct state newState = {}; 82 | newState.nextRS = rsIndex + 1; 83 | lbState.update(&zero, &newState); 84 | 85 | // Create new association 86 | if (associationTable.update(k, rsIp)) 87 | return NULL; // XDP_ABORTED ? 88 | 89 | return rsIp; 90 | } 91 | 92 | int xdp_prog(struct xdp_md *ctx) { 93 | 94 | void *data = (void *)(long)ctx->data; // begin of the packet 95 | void *data_end = (void *)(long)ctx->data_end; // end of the packet 96 | struct logEvent logEvent = {}; // stucture used to log 97 | 98 | // https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/include/uapi/linux/if_ether.h 99 | struct ethhdr * eth = data; 100 | if ((void *) (eth + 1) > data_end) { 101 | log(WARNING, ctx, INVALID_ETH_SIZE, &logEvent); 102 | return XDP_DROP; 103 | } 104 | 105 | // Parse IP header : extract ip address & udp header 106 | struct udphdr * udp = NULL; 107 | ip_addr * saddr = NULL; 108 | ip_addr * daddr = NULL; 109 | int res = parse_ip_header(eth, data_end, &udp, &saddr, &daddr); 110 | // Store packet addresses for logging 111 | if (saddr != NULL && daddr != NULL) { 112 | memcpy(&logEvent.odmac, eth->h_dest, ETH_ALEN); 113 | memcpy(&logEvent.osmac, eth->h_source, ETH_ALEN); 114 | copy_ip_addr(&logEvent.odaddr, daddr); 115 | copy_ip_addr(&logEvent.osaddr, saddr); 116 | } 117 | // Handle ip header error 118 | if (udp == NULL){ 119 | switch(res) { 120 | case NOT_IP_V4 : 121 | case NOT_IP_V6 : 122 | case NOT_UDP : 123 | log(TRACE, ctx, res, &logEvent); 124 | return XDP_PASS; 125 | case FRAGMENTED_IP_PACKET: 126 | case TOO_BIG_IP_HEADER: 127 | log(INFO, ctx, res, &logEvent); 128 | return XDP_PASS; 129 | case INVALID_IP_SIZE : 130 | case TOO_SMALL_IP_HEADER: 131 | case LIFETIME_EXPIRED: 132 | log(WARNING, ctx, res, &logEvent); 133 | return XDP_DROP; 134 | default : 135 | log(ERROR, ctx, res, &logEvent); 136 | return XDP_PASS; 137 | } 138 | } 139 | 140 | // https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/include/uapi/linux/udp.h 141 | if ((void *) (udp + 1) > data_end) { 142 | log(WARNING, ctx, INVALID_UDP_SIZE, &logEvent); 143 | return XDP_DROP; 144 | } 145 | 146 | // Get virtual server 147 | int zero = 0; 148 | ip_addr * vsIp = virtualServer.lookup(&zero); 149 | if (vsIp == NULL) { 150 | log(ERROR, ctx, NO_VIRTUAL_SERVER, &logEvent); 151 | return XDP_PASS; 152 | } 153 | 154 | // Store ip address modification for checksum incremental update 155 | ip_addr old_addr; 156 | __builtin_memset(&old_addr, 0, sizeof(old_addr)); 157 | ip_addr new_addr; 158 | __builtin_memset(&new_addr, 0, sizeof(new_addr)); 159 | 160 | // Is it ingress traffic ? destination IP == VIP 161 | if (compare_ip_addr(daddr, vsIp)) { 162 | // do not handle traffic on ports we don't want to redirect 163 | if (!ports.lookup(&(udp->dest))) { 164 | log(TRACE, ctx, INGRESS_NOT_HANDLED_PORT, &logEvent); 165 | return XDP_PASS; 166 | } else { 167 | // Handle ingress traffic 168 | // Find real server associated 169 | struct associationKey k = {}; 170 | copy_ip_addr(&k.ipAddr, saddr); 171 | k.port = udp->source; 172 | ip_addr * rsIp = associationTable.lookup(&k); 173 | // Create association if no real server associated 174 | // (or if real server associated does not exist anymore) 175 | if (rsIp == NULL || realServersMap.lookup(rsIp) == NULL) { 176 | rsIp = new_association(&k); 177 | if (rsIp == NULL) { 178 | log(ERROR, ctx, INGRESS_CANNOT_CREATE_ASSO, &logEvent); 179 | return XDP_DROP; // XDP_ABORTED ? 180 | } 181 | logEvent.code = INGRESS_NEW_NAT; 182 | } else { 183 | logEvent.code = INGRESS_REUSED_NAT; 184 | } 185 | // Should not happened, mainly needed to make verfier happy 186 | if (rsIp == NULL) { 187 | log(CRITICAL, ctx, INGRESS_CANNOT_CREATE_ASSO2, &logEvent); 188 | return XDP_DROP; // XDP_ABORTED ? 189 | } 190 | 191 | // Eth address swapping 192 | unsigned char dmac[ETH_ALEN]; 193 | memcpy(dmac, eth->h_dest, ETH_ALEN); 194 | // Use virtual server MAC address (so packet destination) as source 195 | memcpy(eth->h_dest, eth->h_source, ETH_ALEN); 196 | // Use source ethernet address as destination, 197 | // as we supose all ethernet traffic goes through this gateway. 198 | // (currently we support use case with only 1 ethernet gateway) 199 | memcpy(eth->h_source, dmac, ETH_ALEN); 200 | 201 | // Update IP address (DESTINATION NAT) 202 | copy_ip_addr(&old_addr,daddr); 203 | copy_ip_addr(&new_addr, rsIp); 204 | copy_ip_addr(daddr, rsIp); // use real server IP address as destination 205 | 206 | decrease_packet_lifetime(eth); 207 | } 208 | } else { 209 | // Is it egress traffic ? source ip == a real server IP 210 | ip_addr * rsIp = realServersMap.lookup(saddr); 211 | if (rsIp != NULL) { 212 | // do not handle traffic on ports we don't want to redirect 213 | if (!ports.lookup(&(udp->source))) { 214 | log(TRACE, ctx, EGRESS_NOT_HANDLED_PORT, &logEvent); 215 | return XDP_PASS; 216 | } else { 217 | // Handle egress traffic 218 | // Find real server associated to this foreign peer 219 | struct associationKey k = {}; 220 | copy_ip_addr(&k.ipAddr, daddr); 221 | k.port = udp->dest; 222 | ip_addr * currentRsIp = associationTable.lookup(&k); 223 | // Create association if no real server associated 224 | // (or if real server associated does not exist anymore) 225 | if (currentRsIp == NULL || realServersMap.lookup(currentRsIp) == NULL ) { 226 | if (associationTable.update(&k, rsIp)) { 227 | log(ERROR, ctx, EGRESS_CANNOT_CREATE_ASSO, &logEvent); 228 | return XDP_DROP; // XDP_ABORTED ? 229 | } 230 | logEvent.code = EGRESS_NEW_NAT; 231 | } else if (!compare_ip_addr(currentRsIp, rsIp)) { 232 | // If there is an association 233 | // only associated server is allow to send packet 234 | log(INFO, ctx, EGRESS_NOT_AUTHORIZED, &logEvent); 235 | return XDP_DROP; 236 | } else { 237 | logEvent.code = EGRESS_REUSED_NAT; 238 | } 239 | 240 | // Eth address swapping 241 | unsigned char dmac[ETH_ALEN]; 242 | memcpy(dmac, eth->h_dest, ETH_ALEN); 243 | // Use virtual server MAC address (so packet destination) as source 244 | memcpy(eth->h_dest, eth->h_source, ETH_ALEN); 245 | // Use source ethernet address as destination, 246 | // as we supose all ethernet traffic goes through this gateway. 247 | // (currently we support use case with only 1 ethernet gateway) 248 | memcpy(eth->h_source, dmac, ETH_ALEN); 249 | 250 | // Update IP address (SOURCE NAT) 251 | copy_ip_addr(&old_addr,saddr); 252 | copy_ip_addr(&new_addr,vsIp); 253 | copy_ip_addr(saddr,vsIp); // use virtual server IP address as source 254 | 255 | decrease_packet_lifetime(eth); 256 | } 257 | } else { 258 | // neither ingress(destIP=VirtualServerIP) nor egress(sourceIP=RealServerIP) traffic 259 | log(TRACE, ctx, UNHANDLED_TRAFFIC, &logEvent); 260 | return XDP_PASS; 261 | } 262 | } 263 | 264 | // Update IP checksum 265 | update_ip_checksum(eth, data_end, old_addr, new_addr); 266 | 267 | // Update UDP checksum 268 | udp->check = update_udp_checksum(udp->check, old_addr, new_addr); 269 | 270 | // Log address translation 271 | // Store new addresses 272 | memcpy(&logEvent.ndmac, eth->h_dest, ETH_ALEN); 273 | memcpy(&logEvent.nsmac, eth->h_source, ETH_ALEN); 274 | copy_ip_addr(&logEvent.ndaddr, daddr); 275 | copy_ip_addr(&logEvent.nsaddr, saddr); 276 | log(DEBUG, ctx, logEvent.code, &logEvent); 277 | 278 | return XDP_TX; 279 | } 280 | -------------------------------------------------------------------------------- /sbulb/tests/__init__.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | 3 | from sbulb.tests.test_config import ConfigTestCase 4 | from sbulb.tests.test_loadbalancer import IPv4TestCase 5 | from sbulb.tests.test_loadbalancer import IPv6TestCase 6 | 7 | 8 | def suite(): 9 | suite = unittest.TestSuite() 10 | suite.addTest(ConfigTestCase()) 11 | suite.addTest(IPv4TestCase()) 12 | suite.addTest(IPv6TestCase()) 13 | return suite 14 | 15 | 16 | if __name__ == '__main__': 17 | runner = unittest.TextTestRunner() 18 | runner.run(suite()) 19 | -------------------------------------------------------------------------------- /sbulb/tests/ipv4.cfg: -------------------------------------------------------------------------------- 1 | [Real Servers] 2 | 192.0.0.1 3 | 192.0.0.2 4 | -------------------------------------------------------------------------------- /sbulb/tests/ipv6.cfg: -------------------------------------------------------------------------------- 1 | [Real Servers] 2 | 2a01:e0a:3a8:d8c0:8809:cda:9245:4311 3 | 2a01:e0a:3a8:d8c0:8809:cda:9245:4322 4 | 2a01:e0a:3a8:d8c0:8809:cda:9245:4333 5 | -------------------------------------------------------------------------------- /sbulb/tests/test_config.py: -------------------------------------------------------------------------------- 1 | from ipaddress import IPv6Address, IPv4Address 2 | import ipaddress 3 | import unittest 4 | 5 | from sbulb import Cfg 6 | 7 | 8 | class ConfigTestCase(unittest.TestCase): 9 | 10 | def test_ipv4(self): 11 | cfg = Cfg(); 12 | cfg.ifnet = "test" 13 | cfg.virtual_server_ip = ipaddress.ip_address("10.0.0.1") 14 | cfg.ip_version = cfg.virtual_server_ip.version 15 | cfg.ports = [5683] 16 | cfg.config_file = "./sbulb/tests/ipv4.cfg" 17 | 18 | self.assertEqual([], cfg.real_server_ips) 19 | cfg.validate(True); 20 | self.assertEqual(2, len(cfg.real_server_ips)) 21 | 22 | servers = cfg.load_real_server(); 23 | self.assertEqual(cfg.real_server_ips, servers) 24 | self.assertTrue( 25 | all((type(s) is IPv4Address) for s in servers)) 26 | 27 | self.assertFalse(cfg.config_file_changed()) 28 | 29 | def test_ipv6(self): 30 | cfg = Cfg(); 31 | cfg.ifnet = "test" 32 | cfg.virtual_server_ip = ipaddress.ip_address(\ 33 | "2222:0000:0000:0000:0000:0000:0000:0001") 34 | cfg.ip_version = cfg.virtual_server_ip.version 35 | cfg.ports = [5683] 36 | cfg.config_file = "./sbulb/tests/ipv6.cfg" 37 | 38 | self.assertEqual([], cfg.real_server_ips) 39 | cfg.validate(True); 40 | self.assertEqual(3, len(cfg.real_server_ips)) 41 | 42 | servers = cfg.load_real_server(); 43 | self.assertEqual(cfg.real_server_ips, servers) 44 | self.assertTrue( 45 | all((type(s) is IPv6Address) for s in servers)) 46 | 47 | self.assertFalse(cfg.config_file_changed()) 48 | 49 | 50 | if __name__ == '__main__': 51 | unittest.main() 52 | -------------------------------------------------------------------------------- /sbulb/tests/test_loadbalancer.py: -------------------------------------------------------------------------------- 1 | from abc import ABC, abstractmethod 2 | import ipaddress 3 | from scapy.layers.inet import IP, UDP 4 | from scapy.layers.inet6 import IPv6 5 | from scapy.layers.l2 import Ether 6 | import unittest 7 | 8 | from sbulb import Cfg, LoadBalancer 9 | from sbulb.tests.test_util import run_test, assert_dropped, \ 10 | assert_redirect_from_real_server_ipv6, assert_redirect_to_real_server_ipv6, \ 11 | assert_redirect_from_real_server_ipv4, assert_redirect_to_real_server_ipv4 12 | 13 | 14 | class AbstractTestCase(ABC): 15 | load_balancer = None 16 | ports = [5683] 17 | 18 | # defined in subclasses : 19 | cli_ip = [] # client IP addresses 20 | vs_ip = None # virtual server IP address 21 | rs_ip = [] # real servers IP addresses 22 | IP # class used to create IP layer 23 | 24 | def setUp(self): 25 | cfg = Cfg(); 26 | cfg.ifnet = "test" 27 | cfg.loglevel = "DEBUG" 28 | cfg.max_associations = 4 29 | cfg.virtual_server_ip = ipaddress.ip_address(self.vs_ip) 30 | cfg.ip_version = cfg.virtual_server_ip.version 31 | cfg.ports = self.ports 32 | cfg.real_server_ips = [ipaddress.ip_address(self.rs_ip[0]), \ 33 | ipaddress.ip_address(self.rs_ip[1])] 34 | 35 | self.load_balancer = LoadBalancer(cfg) 36 | self.load_balancer._open_log_buffer() 37 | 38 | def tearDown(self): 39 | self.load_balancer.flush_log_buffer() 40 | unittest.TestCase.tearDown(self) 41 | 42 | @abstractmethod 43 | def assert_redirect_from_real_server(self): 44 | ... 45 | 46 | @abstractmethod 47 | def assert_redirect_to_real_server(self): 48 | ... 49 | 50 | def test_client_initiated(self): 51 | # from client 0 associating to real server 0 52 | packet_in = Ether() \ 53 | / self.IP(src=self.cli_ip[0], dst=self.vs_ip) \ 54 | / UDP(dport=self.ports[0]) 55 | res = run_test(self, self.load_balancer.func, packet_in) 56 | self.assert_redirect_to_real_server(self.rs_ip[0], packet_in, res) 57 | 58 | # from client 0 STILL associated to real server 0 59 | packet_in = Ether() \ 60 | / self.IP(src=self.cli_ip[0], dst=self.vs_ip) \ 61 | / UDP(dport=self.ports[0]) 62 | res = run_test(self, self.load_balancer.func, packet_in) 63 | self.assert_redirect_to_real_server(self.rs_ip[0], packet_in, res) 64 | 65 | # answer to client 0 from real server 0 66 | packet_in = Ether() \ 67 | / self.IP(src=self.rs_ip[0], dst=self.cli_ip[0]) \ 68 | / UDP(sport=self.ports[0]) 69 | res = run_test(self, self.load_balancer.func, packet_in) 70 | self.assert_redirect_from_real_server(self.vs_ip, packet_in, res) 71 | 72 | # server initiated from real server 1 to client 0 : dropped 73 | packet_in = Ether() \ 74 | / self.IP(src=self.rs_ip[1], dst=self.cli_ip[0]) \ 75 | / UDP(sport=self.ports[0]) 76 | res = run_test(self, self.load_balancer.func, packet_in) 77 | assert_dropped(self, res) 78 | 79 | def test_server_initiated(self): 80 | # from server 1 associating to client 0 81 | packet_in = Ether() \ 82 | / self.IP(src=self.rs_ip[1], dst=self.cli_ip[0]) \ 83 | / UDP(sport=self.ports[0]) 84 | res = run_test(self, self.load_balancer.func, packet_in) 85 | self.assert_redirect_from_real_server(self.vs_ip, packet_in, res) 86 | 87 | # from server 1 STILL associated to client 0 88 | packet_in = Ether() \ 89 | / self.IP(src=self.rs_ip[1], dst=self.cli_ip[0]) \ 90 | / UDP(sport=self.ports[0]) 91 | res = run_test(self, self.load_balancer.func, packet_in) 92 | self.assert_redirect_from_real_server(self.vs_ip, packet_in, res) 93 | 94 | # server initiated from real server 0 to client 0 : dropped 95 | packet_in = Ether() \ 96 | / self.IP(src=self.rs_ip[0], dst=self.cli_ip[0]) \ 97 | / UDP(sport=self.ports[0]) 98 | res = run_test(self, self.load_balancer.func, packet_in) 99 | assert_dropped(self, res) 100 | 101 | # from client 0 associated to real server 1 102 | packet_in = Ether() \ 103 | / self.IP(src=self.cli_ip[0], dst=self.vs_ip) \ 104 | / UDP(dport=self.ports[0]) 105 | res = run_test(self, self.load_balancer.func, packet_in) 106 | self.assert_redirect_to_real_server(self.rs_ip[1], packet_in, res) 107 | 108 | def test_round_robin(self): 109 | # from client 0 associating to real server 0 110 | packet_in = Ether() \ 111 | / self.IP(src=self.cli_ip[0], dst=self.vs_ip) \ 112 | / UDP(dport=self.ports[0]) 113 | res = run_test(self, self.load_balancer.func, packet_in) 114 | self.assert_redirect_to_real_server(self.rs_ip[0], packet_in, res) 115 | 116 | # from client 1 associating to real server 1 117 | packet_in = Ether() \ 118 | / self.IP(src=self.cli_ip[1], dst=self.vs_ip) \ 119 | / UDP(dport=self.ports[0]) 120 | res = run_test(self, self.load_balancer.func, packet_in) 121 | self.assert_redirect_to_real_server(self.rs_ip[1], packet_in, res) 122 | 123 | # from client 2 associating to real server 0 124 | packet_in = Ether() \ 125 | / self.IP(src=self.cli_ip[2], dst=self.vs_ip) \ 126 | / UDP(dport=self.ports[0]) 127 | res = run_test(self, self.load_balancer.func, packet_in) 128 | self.assert_redirect_to_real_server(self.rs_ip[0], packet_in, res) 129 | 130 | # from client 3 associating to real server 1 131 | packet_in = Ether() \ 132 | / self.IP(src=self.cli_ip[3], dst=self.vs_ip) \ 133 | / UDP(dport=self.ports[0]) 134 | res = run_test(self, self.load_balancer.func, packet_in) 135 | self.assert_redirect_to_real_server(self.rs_ip[1], packet_in, res) 136 | 137 | def test_round_robin_sameip(self): 138 | # from client 0 associating to real server 0 139 | packet_in = Ether() \ 140 | / self.IP(src=self.cli_ip[0], dst=self.vs_ip) \ 141 | / UDP(sport=1000, dport=self.ports[0]) 142 | res = run_test(self, self.load_balancer.func, packet_in) 143 | self.assert_redirect_to_real_server(self.rs_ip[0], packet_in, res) 144 | 145 | # from client 1 associating to real server 1 146 | packet_in = Ether() \ 147 | / self.IP(src=self.cli_ip[0], dst=self.vs_ip) \ 148 | / UDP(sport=1001, dport=self.ports[0]) 149 | res = run_test(self, self.load_balancer.func, packet_in) 150 | self.assert_redirect_to_real_server(self.rs_ip[1], packet_in, res) 151 | 152 | # from client 2 associating to real server 0 153 | packet_in = Ether() \ 154 | / self.IP(src=self.cli_ip[0], dst=self.vs_ip) \ 155 | / UDP(sport=1003, dport=self.ports[0]) 156 | res = run_test(self, self.load_balancer.func, packet_in) 157 | self.assert_redirect_to_real_server(self.rs_ip[0], packet_in, res) 158 | 159 | # from client 3 associating to real server 1 160 | packet_in = Ether() \ 161 | / self.IP(src=self.cli_ip[0], dst=self.vs_ip) \ 162 | / UDP(sport=1004, dport=self.ports[0]) 163 | res = run_test(self, self.load_balancer.func, packet_in) 164 | self.assert_redirect_to_real_server(self.rs_ip[1], packet_in, res) 165 | 166 | def test_remove_first_real_server(self): 167 | # from client 0 associating to real server 0 168 | packet_in = Ether() \ 169 | / self.IP(src=self.cli_ip[0], dst=self.vs_ip) \ 170 | / UDP(dport=self.ports[0]) 171 | res = run_test(self, self.load_balancer.func, packet_in) 172 | self.assert_redirect_to_real_server(self.rs_ip[0], packet_in, res) 173 | 174 | # server initiated from real server 1 to client 0 : dropped 175 | packet_in = Ether() \ 176 | / self.IP(src=self.rs_ip[1], dst=self.cli_ip[0]) \ 177 | / UDP(sport=self.ports[0]) 178 | res = run_test(self, self.load_balancer.func, packet_in) 179 | assert_dropped(self, res) 180 | 181 | # remove real server 0 182 | self.load_balancer.flush_log_buffer() 183 | self.load_balancer._update_real_server( 184 | self.load_balancer.cfg.real_server_ips, 185 | self.load_balancer.cfg.real_server_ips[1:]) 186 | 187 | # from server 1 associating to client 0 188 | packet_in = Ether() \ 189 | / self.IP(src=self.rs_ip[1], dst=self.cli_ip[0]) \ 190 | / UDP(sport=self.ports[0]) 191 | res = run_test(self, self.load_balancer.func, packet_in) 192 | self.assert_redirect_from_real_server(self.vs_ip, packet_in, res) 193 | 194 | def test_remove_last_real_server(self): 195 | # from client 0 associating to real server 0 196 | packet_in = Ether() \ 197 | / self.IP(src=self.cli_ip[0], dst=self.vs_ip) \ 198 | / UDP(dport=self.ports[0]) 199 | res = run_test(self, self.load_balancer.func, packet_in) 200 | self.assert_redirect_to_real_server(self.rs_ip[0], packet_in, res) 201 | 202 | # from client 1 associating to real server 2 203 | packet_in = Ether() \ 204 | / self.IP(src=self.cli_ip[1], dst=self.vs_ip) \ 205 | / UDP(dport=self.ports[0]) 206 | res = run_test(self, self.load_balancer.func, packet_in) 207 | self.assert_redirect_to_real_server(self.rs_ip[1], packet_in, res) 208 | 209 | # server initiated from real server 0 to client 1 : dropped 210 | packet_in = Ether() \ 211 | / self.IP(src=self.rs_ip[0], dst=self.cli_ip[1]) \ 212 | / UDP(sport=self.ports[0]) 213 | res = run_test(self, self.load_balancer.func, packet_in) 214 | assert_dropped(self, res) 215 | 216 | # remove real server 0 217 | self.load_balancer.flush_log_buffer() 218 | self.load_balancer._update_real_server( 219 | self.load_balancer.cfg.real_server_ips, 220 | self.load_balancer.cfg.real_server_ips[:1]) 221 | 222 | # from server 0 associating to client 1 223 | packet_in = Ether() \ 224 | / self.IP(src=self.rs_ip[0], dst=self.cli_ip[1]) \ 225 | / UDP(sport=self.ports[0]) 226 | res = run_test(self, self.load_balancer.func, packet_in) 227 | self.assert_redirect_from_real_server(self.vs_ip, packet_in, res) 228 | 229 | def test_add_real_server(self): 230 | # from client 0 associating to real server 0 231 | packet_in = Ether() \ 232 | / self.IP(src=self.cli_ip[0], dst=self.vs_ip) \ 233 | / UDP(dport=self.ports[0]) 234 | res = run_test(self, self.load_balancer.func, packet_in) 235 | self.assert_redirect_to_real_server(self.rs_ip[0], packet_in, res) 236 | 237 | # from client 1 associating to real server 1 238 | packet_in = Ether() \ 239 | / self.IP(src=self.cli_ip[1], dst=self.vs_ip) \ 240 | / UDP(dport=self.ports[0]) 241 | res = run_test(self, self.load_balancer.func, packet_in) 242 | self.assert_redirect_to_real_server(self.rs_ip[1], packet_in, res) 243 | 244 | # add real server 245 | self.load_balancer.flush_log_buffer() 246 | new_rs_ips = self.load_balancer.cfg.real_server_ips.copy() 247 | new_rs_ips.append(ipaddress.ip_address(self.rs_ip[2])) 248 | self.load_balancer._update_real_server( 249 | self.load_balancer.cfg.real_server_ips, new_rs_ips) 250 | 251 | # from client 2 associating to real server 2 252 | packet_in = Ether() \ 253 | / self.IP(src=self.cli_ip[2], dst=self.vs_ip) \ 254 | / UDP(dport=self.ports[0]) 255 | res = run_test(self, self.load_balancer.func, packet_in) 256 | self.assert_redirect_to_real_server(self.rs_ip[2], packet_in, res) 257 | 258 | def test_update_real_server(self): 259 | # from client 0 associating to real server 0 260 | packet_in = Ether() \ 261 | / self.IP(src=self.cli_ip[0], dst=self.vs_ip) \ 262 | / UDP(dport=self.ports[0]) 263 | res = run_test(self, self.load_balancer.func, packet_in) 264 | self.assert_redirect_to_real_server(self.rs_ip[0], packet_in, res) 265 | 266 | # from client 1 associating to real server 1 267 | packet_in = Ether() \ 268 | / self.IP(src=self.cli_ip[1], dst=self.vs_ip) \ 269 | / UDP(dport=self.ports[0]) 270 | res = run_test(self, self.load_balancer.func, packet_in) 271 | self.assert_redirect_to_real_server(self.rs_ip[1], packet_in, res) 272 | 273 | # add real server 274 | self.load_balancer.flush_log_buffer() 275 | new_rs_ips = self.load_balancer.cfg.real_server_ips.copy() 276 | new_rs_ips[0] = ipaddress.ip_address(self.rs_ip[2]) 277 | new_rs_ips[1] = ipaddress.ip_address(self.rs_ip[3]) 278 | self.load_balancer._update_real_server( 279 | self.load_balancer.cfg.real_server_ips, new_rs_ips) 280 | 281 | # from client 0 associating to real server 2 282 | packet_in = Ether() \ 283 | / self.IP(src=self.cli_ip[0], dst=self.vs_ip) \ 284 | / UDP(dport=self.ports[0]) 285 | res = run_test(self, self.load_balancer.func, packet_in) 286 | self.assert_redirect_to_real_server(self.rs_ip[2], packet_in, res) 287 | 288 | # from client 0 associating to real server 3 289 | packet_in = Ether() \ 290 | / self.IP(src=self.cli_ip[1], dst=self.vs_ip) \ 291 | / UDP(dport=self.ports[0]) 292 | res = run_test(self, self.load_balancer.func, packet_in) 293 | self.assert_redirect_to_real_server(self.rs_ip[3], packet_in, res) 294 | 295 | def test_lru(self): 296 | # test is configure with 4 association maximum 297 | # from client 0 associating to real server 0 298 | packet_in = Ether() \ 299 | / self.IP(src=self.cli_ip[0], dst=self.vs_ip) \ 300 | / UDP(dport=self.ports[0]) 301 | res = run_test(self, self.load_balancer.func, packet_in) 302 | self.assert_redirect_to_real_server(self.rs_ip[0], packet_in, res) 303 | 304 | # from client 1 associating to real server 1 305 | packet_in = Ether() \ 306 | / self.IP(src=self.cli_ip[1], dst=self.vs_ip) \ 307 | / UDP(dport=self.ports[0]) 308 | res = run_test(self, self.load_balancer.func, packet_in) 309 | self.assert_redirect_to_real_server(self.rs_ip[1], packet_in, res) 310 | 311 | # from client 2 associating to real server 0 312 | packet_in = Ether() \ 313 | / self.IP(src=self.cli_ip[2], dst=self.vs_ip) \ 314 | / UDP(dport=self.ports[0]) 315 | res = run_test(self, self.load_balancer.func, packet_in) 316 | self.assert_redirect_to_real_server(self.rs_ip[0], packet_in, res) 317 | 318 | # from client 3 associating to real server 1 319 | packet_in = Ether() \ 320 | / self.IP(src=self.cli_ip[3], dst=self.vs_ip) \ 321 | / UDP(dport=self.ports[0]) 322 | res = run_test(self, self.load_balancer.func, packet_in) 323 | self.assert_redirect_to_real_server(self.rs_ip[1], packet_in, res) 324 | 325 | # from client 0 still associated to real server 0 326 | packet_in = Ether() \ 327 | / self.IP(src=self.cli_ip[0], dst=self.vs_ip) \ 328 | / UDP(dport=self.ports[0]) 329 | res = run_test(self, self.load_balancer.func, packet_in) 330 | self.assert_redirect_to_real_server(self.rs_ip[0], packet_in, res) 331 | 332 | # from client 1 still associated to real server 1 333 | packet_in = Ether() \ 334 | / self.IP(src=self.cli_ip[1], dst=self.vs_ip) \ 335 | / UDP(dport=self.ports[0]) 336 | res = run_test(self, self.load_balancer.func, packet_in) 337 | self.assert_redirect_to_real_server(self.rs_ip[1], packet_in, res) 338 | 339 | # from client 4 associating to real server 0 340 | packet_in = Ether() \ 341 | / self.IP(src=self.cli_ip[4], dst=self.vs_ip) \ 342 | / UDP(dport=self.ports[0]) 343 | res = run_test(self, self.load_balancer.func, packet_in) 344 | self.assert_redirect_to_real_server(self.rs_ip[0], packet_in, res) 345 | 346 | # from client 2 lost its association with 0 (least recently used) 347 | # and should now be associating to real server 1 348 | packet_in = Ether() \ 349 | / self. IP(src=self.cli_ip[2], dst=self.vs_ip) \ 350 | / UDP(dport=self.ports[0]) 351 | res = run_test(self, self.load_balancer.func, packet_in) 352 | self.assert_redirect_to_real_server(self.rs_ip[1], packet_in, res) 353 | 354 | 355 | class IPv6TestCase(AbstractTestCase, unittest.TestCase): 356 | # client IP addresses 357 | cli_ip = ["5555:0000:0000:0000:0000:0000:0000:0001", 358 | "5555:0000:0000:0000:0000:0000:0000:0002", 359 | "5555:0000:0000:0000:0000:0000:0000:0003", 360 | "5555:0000:0000:0000:0000:0000:0000:0004", 361 | "5555:0000:0000:0000:0000:0000:0000:0005"] 362 | # virtual server IP address 363 | vs_ip = "1111:0000:0000:0000:0000:0000:0000:0001" 364 | # real servers IP addresses 365 | rs_ip = ["4444:0000:0000:0000:0000:0000:0000:0001", 366 | "4444:0000:0000:0000:0000:0000:0000:0002", 367 | "4444:0000:0000:0000:0000:0000:0000:0003", 368 | "4444:0000:0000:0000:0000:0000:0000:0004"] 369 | # class used to create IP layer 370 | IP = IPv6 371 | 372 | def assert_redirect_from_real_server(self, virtual_server_ip, 373 | packet_in, result): 374 | assert_redirect_from_real_server_ipv6(self, virtual_server_ip, 375 | packet_in, result) 376 | 377 | def assert_redirect_to_real_server(self, real_server_ip, 378 | packet_in, result): 379 | assert_redirect_to_real_server_ipv6(self, real_server_ip, 380 | packet_in, result) 381 | 382 | 383 | class IPv4TestCase(AbstractTestCase, unittest.TestCase): 384 | # client IP addresses 385 | cli_ip = ["55.0.0.1", "55.0.0.2", "55.0.0.3", "55.0.0.4", "55.0.0.5"] 386 | # virtual server IP address 387 | vs_ip = "11.0.0.1" 388 | # real servers IP addresses 389 | rs_ip = ["44.0.0.1", "44.0.0.2", "44.0.0.3", "44.0.0.4"] 390 | # class used to create IP layer 391 | IP = IP 392 | 393 | def assert_redirect_from_real_server(self, virtual_server_ip, 394 | packet_in, result): 395 | assert_redirect_from_real_server_ipv4(self, virtual_server_ip, 396 | packet_in, result) 397 | 398 | def assert_redirect_to_real_server(self, real_server_ip, 399 | packet_in, result): 400 | assert_redirect_to_real_server_ipv4(self, real_server_ip, 401 | packet_in, result) 402 | 403 | 404 | if __name__ == '__main__': 405 | unittest.main() 406 | -------------------------------------------------------------------------------- /sbulb/tests/test_util.py: -------------------------------------------------------------------------------- 1 | from bcc import libbcc, BPF 2 | import ctypes 3 | from dataclasses import dataclass 4 | from scapy.compat import raw 5 | from scapy.layers.inet import IP, UDP 6 | from scapy.layers.inet6 import IPv6 7 | from scapy.layers.l2 import Ether 8 | from scapy.packet import Packet 9 | 10 | 11 | def assert_redirect_to_real_server_ipv4(test, real_server_ip, 12 | packet_in, result): 13 | test.assertEqual(BPF.XDP_TX, result.retval) 14 | # client -> virtual server 15 | # become 16 | # client -> given real server 17 | expected = packet_in.copy() 18 | expected[Ether].dst = packet_in[Ether].src 19 | expected[Ether].src = packet_in[Ether].dst 20 | expected[IP].dst = real_server_ip 21 | expected[IP].ttl = packet_in[IP].ttl - 1 22 | # force checksum calculation 23 | del expected[IP].chksum 24 | del expected[UDP].chksum 25 | expected = Ether(raw(expected)) 26 | 27 | test.assertEqual(expected, result.packet) 28 | 29 | 30 | def assert_redirect_to_real_server_ipv6(test, real_server_ip, 31 | packet_in, result): 32 | test.assertEqual(BPF.XDP_TX, result.retval) 33 | # client -> virtual server 34 | # become 35 | # client -> given real server 36 | expected = packet_in.copy() 37 | expected[Ether].dst = packet_in[Ether].src 38 | expected[Ether].src = packet_in[Ether].dst 39 | expected[IPv6].dst = real_server_ip 40 | expected[IPv6].hlim = packet_in[IPv6].hlim - 1 41 | # force checksum calculation 42 | del expected[UDP].chksum 43 | expected = Ether(raw(expected)) 44 | 45 | test.assertEqual(expected, result.packet) 46 | 47 | 48 | def assert_redirect_from_real_server_ipv4(test, virtual_server_ip, 49 | packet_in, result): 50 | test.assertEqual(BPF.XDP_TX, result.retval) 51 | 52 | # real server -> client 53 | # become 54 | # virtual server -> client 55 | expected = packet_in.copy() 56 | expected[Ether].dst = packet_in[Ether].src 57 | expected[Ether].src = packet_in[Ether].dst 58 | expected[IP].src = virtual_server_ip 59 | expected[IP].ttl = packet_in[IP].ttl - 1 60 | # force checksum calculation 61 | del expected[IP].chksum 62 | del expected[UDP].chksum 63 | expected = Ether(raw(expected)) 64 | 65 | test.assertEqual(expected, result.packet) 66 | 67 | 68 | def assert_redirect_from_real_server_ipv6(test, virtual_server_ip, 69 | packet_in, result): 70 | test.assertEqual(BPF.XDP_TX, result.retval) 71 | 72 | # real server -> client 73 | # become 74 | # virtual server -> client 75 | expected = packet_in.copy() 76 | expected[Ether].dst = packet_in[Ether].src 77 | expected[Ether].src = packet_in[Ether].dst 78 | expected[IPv6].src = virtual_server_ip 79 | expected[IPv6].hlim = packet_in[IPv6].hlim - 1 80 | # force checksum calculation 81 | del expected[UDP].chksum 82 | expected = Ether(raw(expected)) 83 | 84 | test.assertEqual(expected, result.packet) 85 | 86 | 87 | def assert_dropped(test, result): 88 | test.assertEqual(BPF.XDP_DROP, result.retval) 89 | 90 | 91 | def run_test(test, func, data, data_out_len=1514): 92 | size = len(data) 93 | data = ctypes.create_string_buffer(raw(data), size) 94 | data_out = ctypes.create_string_buffer(data_out_len) 95 | size_out = ctypes.c_uint32() 96 | retval = ctypes.c_uint32() 97 | duration = ctypes.c_uint32() 98 | repeat = 1 99 | 100 | ret = libbcc.lib.bpf_prog_test_run(func.fd, repeat, 101 | ctypes.byref(data), size, 102 | ctypes.byref(data_out), 103 | ctypes.byref(size_out), 104 | ctypes.byref(retval), 105 | ctypes.byref(duration)) 106 | test.assertEqual(ret, 0) 107 | 108 | return Result(retval.value, Ether(data_out[:size_out.value])) 109 | 110 | 111 | @dataclass 112 | class Result: 113 | retval: ctypes.c_int32 114 | packet: Packet 115 | -------------------------------------------------------------------------------- /sbulb/util.py: -------------------------------------------------------------------------------- 1 | import ipaddress 2 | import socket 3 | import ctypes as ct 4 | 5 | 6 | # Utils 7 | def ip_strton(ip_address): 8 | addr = ipaddress.ip_address(ip_address) 9 | if addr.version == 4: 10 | return ct.c_uint(socket.htonl((int) (addr))) 11 | else: 12 | return (ct.c_ubyte * 16)(*list(addr.packed)) 13 | 14 | 15 | def ip_ntostr(ip_address): 16 | # handle ipv4 17 | if isinstance(ip_address, ct.c_uint): 18 | ip_address = ip_address.value; 19 | if isinstance(ip_address, int): 20 | return str(ipaddress.IPv4Address(socket.ntohl(ip_address))) 21 | # handle ipv6 22 | if "in6_addr" in str(type(ip_address)): 23 | ip_address = ip_address.in6_u.u6_addr8 24 | if isinstance(ip_address, ct.c_ubyte * 16): 25 | ip_address = bytes(bytearray(ip_address)) 26 | return str(ipaddress.IPv6Address(ip_address)) 27 | 28 | 29 | def ipversion(ip_address): 30 | if isinstance(ip_address, ct.c_uint): 31 | return 4 32 | if isinstance(ip_address, ct.c_ubyte * 16): 33 | return 6 34 | raise ValueError("unable to guess ip version of {} (type{})".format(ip_address, type(ip_address))) 35 | 36 | 37 | def mac_btostr(mac_address): 38 | bytestr = bytes(mac_address).hex() 39 | return ':'.join(bytestr[i:i + 2] for i in range(0, 12, 2)) 40 | 41 | 42 | def ip_mac_tostr(mac_address, ip_address): 43 | return "{}/{}".format(mac_btostr(mac_address), ip_ntostr(ip_address)) 44 | 45 | 46 | def ips_tostr(ips): 47 | return ", ".join(map(ip_ntostr, ips)) 48 | 49 | 50 | def ips_ton(ips): 51 | return map(ip_strton, ips) 52 | --------------------------------------------------------------------------------