├── LICENSE ├── README.md ├── cmd ├── Makefile ├── balancer.go ├── config.pl ├── config.sample.yaml ├── go.mod ├── go.sum ├── main.go ├── tests │ ├── Makefile │ ├── bgp01.json │ ├── bgp01.yaml │ ├── bgp02.json │ ├── bgp02.yaml │ ├── compare.pl │ ├── services01.json │ ├── services01.yaml │ ├── services02.json │ ├── services02.yaml │ ├── services03.json │ ├── services03.yaml │ ├── services04.json │ └── services04.yaml └── xvs.go ├── config.go ├── doc ├── NOTES.md ├── README.md ├── advanced.md ├── basic.md ├── bgp.md ├── console.jpg ├── domain.md ├── ipv6-l3.yaml ├── kibana.jpg ├── servers.md ├── services.md ├── vc5.drawio.png └── vc5.drawio.xml ├── elastic.go ├── go.mod ├── go.sum ├── logging.go ├── manager.go ├── prometheus.go ├── static ├── favicon.ico ├── index.css ├── index.html └── index.js └── status.go /README.md: -------------------------------------------------------------------------------- 1 | # VC5 2 | 3 | 4 | 5 | 6 | 7 | **This README is currently being updated to reflect recent changes - 8 | some information may not reflect the current codebase. This 9 | iteration of code is not yet battle ready - use a v0.2 release for 10 | production** 11 | 12 | A horizontally scalable Direct Server Return 13 | ([DSR](https://www.loadbalancer.org/blog/direct-server-return-is-simply-awesome-and-heres-why/)) 14 | layer 4 load balancer (L4LB) for Linux using [XDP/eBPF](https://www.datadoghq.com/blog/xdp-intro/). 15 | 16 | If you think that this may be useful or have any 17 | questions/suggestions, feel free to contact me at vc5lb@proton.me or 18 | raise a GitHub issue. 19 | 20 | **Now supports IPv6 and distribution at layer 3 (AKA 21 | tunnelling)**! The XVS library has been updated to include these 22 | features, and also does away with the need to run health checks from 23 | a network namespace, considerably simplifying the code. This will end 24 | the requirement that all backends share a VLAN with the load balancer. 25 | 26 | Code restrictions currently mean that enabling tunnelling on a 27 | per-service basis is not supported. Using the `-tunnel` option allows a 28 | layer 3 tunnelling to be globally enabled using a single scheme 29 | (IP-in-IP, GRE, FOU or GUE). Going forward, the code will be updated 30 | to allow for tunnelling to be configured at the service level. 31 | 32 | Layer 2 load balancing will continue to be supported - the primary 33 | reason for starting the project was because of the lack of layer 2 34 | support by [Facebook's 35 | Katran](https://github.com/facebookincubator/katran) load 36 | balancer. 37 | 38 | A [sample IPv6/L3 configuration file](doc/ipv6-l3.yaml) is included - 39 | better documentation to follow. 40 | 41 | 42 | ## About 43 | 44 | VC5 is a network load balancer designed to work as replacement for 45 | legacy hardware appliances. It allows services with virtual IP 46 | addresses (VIPs) to be distributed to sets of backend ("real") 47 | servers. Real servers might run the services themselves or act as 48 | proxies for another layer of servers (eg. HAProxy serving as a layer 7 49 | HTTP router/SSL offload when application layer decisions need to 50 | made). The only requirement being that VIPs need to be configured on a 51 | loopback device on each real server, eg.: `ip addr add 52 | 192.168.101.1/32 dev lo` 53 | 54 | Services and real servers are specified in a configuration file, along 55 | with health check definitions. When the backend servers pass checks 56 | and enough are available to provide a service, then virtual IP 57 | addresses are advertised to routers via BGP. 58 | 59 | Distributing traffic at both layer 2 and layer 3 is now 60 | supported. Layer 2 distribution requires that real servers share a 61 | VLAN with the load balancer; upon receiving a packet to be 62 | distributed, the load balancer updates the ethernet hardware addresses 63 | in the packet to use the real server's MAC address as the destination 64 | and its own MAC address as the source, and forwards the packet via the 65 | appropriate interface, updating the 802.1Q VLAN ID if packets are VLAN 66 | tagged. 67 | 68 | Layer 3 distribution requires packets to be encapsulated in a 69 | tunneling protocol addressed to the real server IP and forwarded via a 70 | router (unless the server and laod balancer share a VLAN). If, when 71 | encapsulated, a packet exceeds the network maximum trasmission size 72 | then an ICMP message is sent to the source with advice as to the 73 | appropriate MTU to use. Backend servers only need to decapsulate 74 | packets - bidirectional tunneling with load balancers is not required. 75 | 76 | One server with a 10Gbit/s network interface should be capable of 77 | supporting an HTTP service in excess of 100Gbit/s egress bandwidth due 78 | to the asymmetric nature of most internet traffic. For smaller 79 | services a modest virtual machine or two will likely handle a service 80 | generating a number of gigabit/s of egress traffic. 81 | 82 | If one instance is not sufficient then more servers may be added to 83 | horizontally scale capacity (and provide redundancy) using your 84 | router's ECMP feature. 802.3ad bonded interfaces and 802.1Q VLAN 85 | trunking is supported (see [examples/](examples/) directory). 86 | 87 | No kernel modules or complex setups are required, although for best 88 | performance a network card driver with XDP native mode support is 89 | recommended (eg.: mlx4, mlx5, i40e, ixgbe, ixgbevf, nfp, bnxt, thunder, 90 | dpaa2, qede). A full list is availble at [The XDP Project's driver 91 | support page](https://github.com/xdp-project/xdp-project/blob/master/areas/drivers/README.org). 92 | 93 | ## Goals/status 94 | 95 | * ✅ Simple deployment with a single binary 96 | * ✅ Stable backend selection with the Maglev hashing algorithm 97 | * ✅ Route health injection handled automatically; no need to run other software such as ExaBGP 98 | * ✅ Minimally invasive; does not require any modification of iptables rules on balancer 99 | * ✅ No modification of backend servers beyond adding the VIP to a loopback device/tunnel termination with L3 distribution 100 | * ✅ Health checks are run against the VIP on backend servers, not their real addresses 101 | * ✅ HTTP/HTTPS, half-open SYN probe and UDP/TCP DNS health checks built in 102 | * ✅ In-kernel packet switching with eBPF/XDP; native mode drivers avoid sk_buff allocation 103 | * ✅ Multiple VLAN support 104 | * ✅ Multiple NIC support for lower bandwidth/development applications 105 | * ✅ Tagged/bonded network devices to support high-availibility/high-bandwidth 106 | * ✅ Observability via a web console, Elasticsearch logging (in development) and Prometheus metrics 107 | * ✅ IPv6 support and ability to mix IPv4 and IPv6 backends with either type of VIP. 108 | * ✅ Layer 3 traffic distribution with IP-in-IP, GRE, FOU and GUE support. 109 | 110 | ## Quickstart 111 | 112 | For best results you should disable/uninstall irqbalance. 113 | 114 | You will need to select a primary IP to pass to the balancer. This is 115 | used for the BGP router ID. 116 | 117 | A simple example on a server with a single, untagged ethernet interface: 118 | 119 | * `apt-get install git make libelf-dev golang-1.20 libyaml-perl libjson-perl ethtool` (or your distro's equivalent) 120 | * `ln -s /usr/lib/go-1.20/bin/go /usr/local/bin/go` (ensure that the Go binary is in your path) 121 | * `git clone https://github.com/davidcoles/vc5.git` 122 | * `cd vc5/cmd` 123 | * `cp config.sample.yaml config.yaml` (edit config.yaml to match your requirements) 124 | * `make` (pulls down the [libbpf](https://github.com/libbpf/libbpf) library, builds the binary and JSON config file) 125 | * `./vc5 10.1.10.100 config.json eth0` (amend to use your server's IP address and ethernet interface) 126 | * A web console will be on your load balancer server's port 80 by default 127 | * Add your VIP to the loopback device on your backend servers (eg.: `ip addr add 192.168.101.1/32 dev lo`) 128 | * Configure your network/client to send traffic for your VIP to the load balancer, either via BGP (see config file) or static routing 129 | 130 | It is almost certainly easier to use the binary from the latest Github 131 | release (compiled for x86-64). This will have been tested in 132 | production so should be reliable. Ensure that your configuration is 133 | compatible with this version by using the config.pl script from the 134 | tagged release (or, of course, you can build your own JSON config 135 | however you prefer). 136 | 137 | If you update the YAML config file and regenerate the JSON (`make 138 | config.json`) you can reload the new configuration by sending an a 139 | SIGINT (Ctrl-C) or SIGUSR2 to the process. SIGQUIT (Ctrl-\\) or SIGTERM 140 | will cause the process to gracefully shut down BGP connections and 141 | exit. 142 | 143 | A more complex example with an LACP bonded ethernet device consisting 144 | of two (10Gbps Intel X520 on my test server) interfaces, with native 145 | XDP driver mode enabled and tagged VLANs: 146 | 147 | `config.yaml` vlans entry: 148 | 149 | ``` 150 | vlans: 151 | 10: 10.1.10.0/24 152 | 20: 10.1.20.0/24 153 | 30: 10.1.30.0/24 154 | ``` 155 | 156 | Command line: 157 | 158 | `./vc5 -n 10.1.10.100 config.json enp130s0f0 enp130s0f1` 159 | 160 | The binary will detect your VLAN interfaces by looking for devices 161 | with IP addreses which are contained in the VLAN prefixes in the 162 | configuration file. If you use separate untagged physical interfaces 163 | then this should now work transparently without any extra 164 | configuration, just list all of the interfaces on the command line so 165 | that the eBPF code is loaded into each of them. 166 | 167 | Because connection state is tracked on a per-core basis 168 | (BPF_MAP_TYPE_LRU_PERCPU_HASH), you should ensure that RSS ([Receive 169 | Side 170 | Scaling](https://www.kernel.org/doc/Documentation/networking/scaling.txt)) 171 | will consistently route packets for a flow to the same CPU core in the 172 | event of your switch slecting a different interface when the LACP 173 | topology changes. Disable irqbalance, ensure that channel settings are 174 | the same on each interface (ethtool -l/-L) and that RSS flow hash 175 | indirection matches (ethtool -x/-X). 176 | 177 | The setup can be tested by starting a long running connection 178 | (eg. using iperf with the -t option) to a set of backend servers, then 179 | [disabling the chosen backend with an asterisk after the IP address in 180 | the config file](doc/servers.md), determining which interface is 181 | receiving the flow on the load balancer (eg., `watch -d 'cat 182 | /proc/interrupts | grep enp130s0f'` and look for the rapidly 183 | increasing IRQ counter) and then dropping this interface out of LACP 184 | (`ifenslave -d bond0 enp130s0f0`). You should see the flow move to the 185 | other network interface but still hit the same core. 186 | 187 | When using backends in multiple subnets, for best performance you 188 | should ensure that all VLANs are tagged on a single trunk interface 189 | (LACP bonded if you have more than one physical interface) with 190 | subnet/VLAN ID mappings specified in the `vlans` section of the config 191 | file. 192 | 193 | If this is not possible (for example creating trunked interfaces on 194 | vSphere is not simple), then you can assign each subnet to a different 195 | untagged interface: 196 | 197 | `./vc5 10.1.10.100 config.json eth0 eth1 eth2` 198 | 199 | 200 | ## Background/more info 201 | 202 | A good summary of the concepts in use are discussed in [Patrick 203 | Shuff's "Building a Billion User Load Balancer" 204 | talk](https://www.youtube.com/watch?v=bxhYNfFeVF4&t=1060s) and [Nitika 205 | Shirokov's Katran talk](https://www.youtube.com/watch?v=da9Qw7v5qLM) 206 | 207 | A basic web console and Prometheus metrics server is included: ![Console screenshot](doc/console.jpg) 208 | 209 | Experimental elasticsearch support for logging (direct to your 210 | cluster, no need to scrape system logs) is now included. Every probe 211 | to backend servers is logged, so if one goes down you can see 212 | precisely what error was returned, as well all sorts of other 213 | conditions. This will require a lot of refinement and more sensible 214 | naming of log parameters, etc. (if you've got any insights please get 215 | in touch), but it should lead to being able to get some good insights 216 | into what is going on with the system - my very inept first attempt 217 | creating a Kibana dashboard as an example: ![Kibana screenshot](doc/kibana.jpg) 218 | 219 | 220 | ## Performance 221 | 222 | This has mostly been tested using Icecast backend servers with clients 223 | pulling a mix of low and high bitrate streams (48kbps - 192kbps). 224 | 225 | It seems that a VMWare guest (4 core, 8GB) using the XDP generic 226 | driver will support 100K concurrent clients, 380Mbps/700Kpps through 227 | the load balancer and 8Gbps of traffic from the backends directly to 228 | the clients. 229 | 230 | On a single (non-virtualised) Intel Xeon Gold 6314U CPU (2.30GHz 32 231 | physical cores, with hyperthreading enabled for 64 logical cores) and 232 | an Intel 10G 4P X710-T4L-t ethernet card, I was able to run 700K 233 | streams at 2Gbps/3.8Mpps ingress traffic and 46.5Gbps egress. The 234 | server was more than 90% idle. Unfortunately I did not have the 235 | resources available to create more clients/servers. 236 | 237 | 238 | ## Operation 239 | 240 | There are three modes of operation, simple, VLAN, and multi-NIC 241 | based. In simple mode all hosts must be on the same subnet as the 242 | primary address of the load balancer. In VLAN mode (enabled by 243 | declaring entries under the "vlans" section of the YAML/JSON config 244 | file), server entries must match a VLAN/CIDR subnet entry. VLAN tagged 245 | interfaces need to be created in the OS and have an IP address 246 | assigned within the subnet. In multi-NIC mode subnets are given IDs in 247 | the same manner as VLANs, but bpf_redirect() is used to send traffic 248 | out of the appropriately configured interface (rather than changing 249 | the VLAN ID and using XDP_TX). 250 | 251 | In VLAN mode, all traffic for the load balancer needs to be on a tagged VLAN (no 252 | pushing or popping of 802.1Q is done - yet). 253 | 254 | 255 | -------------------------------------------------------------------------------- /cmd/Makefile: -------------------------------------------------------------------------------- 1 | # If you already have libbpf installed elsewhere on your system then 2 | # specify the path like so: 3 | # CGO_CFLAGS=-I/path/to/libbpf CGO_LDFLAGS=-L/path/to/libbpf go build 4 | 5 | BPFVER ?= v1.5.0 6 | LIBBPF ?= $(PWD)/libbpf 7 | GOLANG ?= 1.21 8 | 9 | default: vc5 config.json 10 | 11 | race: 12 | $(MAKE) default FLAGS=-race 13 | 14 | vc5: libbpf/bpf/libbpf.a *.go 15 | CGO_CFLAGS="-I$(LIBBPF)" CGO_LDFLAGS="-L$(LIBBPF)/bpf" go build $(FLAGS) -o $@ 16 | 17 | # avoid clobbering an existing config file when make is run with -B 18 | config.yaml: 19 | if [ ! -e $@ ]; then cp config.sample.yaml $@; fi 20 | 21 | config.json: config.pl config.yaml 22 | ./config.pl config.yaml >$@- && mv $@- $@ 23 | 24 | clean: 25 | rm -f vc5 config.json 26 | 27 | distclean: clean 28 | rm -rf libbpf 29 | 30 | libbpf: 31 | git clone -b $(BPFVER) --depth 1 https://github.com/libbpf/libbpf 32 | 33 | libbpf/bpf: libbpf 34 | cd libbpf && ln -s src bpf 35 | 36 | libbpf/bpf/libbpf.a: 37 | $(MAKE) libbpf/bpf 38 | cd libbpf/bpf && $(MAKE) 39 | 40 | # just for quick setup on testing machines 41 | focal: apt-install /usr/local/bin/go 42 | jammy: apt-install /usr/local/bin/go 43 | bookworm: apt-install; $(MAKE) /usr/local/bin/go GOLANG=1.19 44 | 45 | apt-install:; apt install -y libelf-dev libyaml-perl libjson-perl 46 | 47 | /usr/local/bin/go: 48 | apt install -y golang-$(GOLANG) 49 | ln -s /usr/lib/go-$(GOLANG)/bin/go /usr/local/bin/go 50 | 51 | # alternately: cloc --match-f='\.go$' --not-match-f='_test.go' .. 52 | cloc: 53 | cloc *.go ../*.go 54 | 55 | 56 | -------------------------------------------------------------------------------- /cmd/balancer.go: -------------------------------------------------------------------------------- 1 | /* 2 | * VC5 load balancer. Copyright (C) 2021-present David Coles 3 | * 4 | * This program is free software; you can redistribute it and/or modify 5 | * it under the terms of the GNU General Public License as published by 6 | * the Free Software Foundation; either version 2 of the License, or 7 | * (at your option) any later version. 8 | * 9 | * This program is distributed in the hope that it will be useful, 10 | * but WITHOUT ANY WARRANTY; without even the implied warranty of 11 | * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 12 | * GNU General Public License for more details. 13 | * 14 | * You should have received a copy of the GNU General Public License along 15 | * with this program; if not, write to the Free Software Foundation, Inc., 16 | * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 17 | */ 18 | 19 | package main 20 | 21 | import ( 22 | "errors" 23 | "fmt" 24 | "net" 25 | 26 | "github.com/davidcoles/xvs" 27 | "vc5" 28 | ) 29 | 30 | type Client = xvs.Client 31 | type ServiceExtended = xvs.ServiceExtended 32 | type DestinationExtended = xvs.DestinationExtended 33 | type Service = xvs.Service 34 | type Destination = xvs.Destination 35 | type Protocol = xvs.Protocol 36 | 37 | // Implement the vc5.Balancer interface; used to retrieve stats and configure the data-plane 38 | 39 | type Balancer struct { 40 | Client Client 41 | Logger vc5.Logger 42 | //tunnel xvs.TunnelType 43 | //port uint16 44 | } 45 | 46 | // func (b *Balancer) Stats() map[vc5.Instance]vc5.Stats { 47 | func (b *Balancer) Stats() (summary vc5.Summary, stats map[vc5.Instance]vc5.Stats) { 48 | 49 | info, _ := b.Client.Info() 50 | summary.Latency = info.Latency 51 | /* 52 | //summary.Dropped = info.Dropped 53 | //summary.Blocked = info.Blocked 54 | //summary.TooBig = info.TooBig 55 | //summary.NotQueued = info.NotQueued 56 | summary.IngressOctets = info.Stats.Octets 57 | summary.IngressPackets = info.Stats.Packets 58 | //summary.EgressOctets = 0 // Not available in DSR 59 | //summary.EgressPackets = 0 // Not available in DSR 60 | summary.Flows = info.Stats.Flows 61 | summary.Current = info.Stats.Current 62 | */ 63 | 64 | summary.IngressOctets = info.Stats.IncomingBytes 65 | summary.IngressPackets = info.Stats.IncomingPackets 66 | summary.Flows = info.Stats.Connections 67 | //summary.Current = info.Stats.Current 68 | 69 | summary.DSR = true 70 | summary.VC5 = true 71 | 72 | stats = map[vc5.Instance]vc5.Stats{} 73 | services, _ := b.Client.Services() 74 | 75 | for _, s := range services { 76 | protocol := vc5.Protocol(s.Service.Protocol) 77 | service := vc5.Service{Address: s.Service.Address, Port: s.Service.Port, Protocol: protocol} 78 | 79 | destinations, _ := b.Client.Destinations(s.Service) 80 | 81 | for _, d := range destinations { 82 | 83 | destination := vc5.Destination{Address: d.Destination.Address, Port: s.Service.Port} 84 | 85 | instance := vc5.Instance{ 86 | Service: service, 87 | Destination: destination, 88 | } 89 | 90 | stats[instance] = vc5.Stats{ 91 | /* 92 | IngressOctets: d.Stats.Octets, 93 | IngressPackets: d.Stats.Packets, 94 | EgressOctets: 0, // Not available in DSR 95 | EgressPackets: 0, // Not available in DSR 96 | Flows: d.Stats.Flows, 97 | Current: d.Stats.Current, 98 | */ 99 | 100 | //EgressOctets: 0, // Not available in DSR 101 | //EgressPackets: 0, // Not available in DSR 102 | IngressOctets: d.Stats.IncomingBytes, 103 | IngressPackets: d.Stats.IncomingPackets, 104 | Flows: d.Stats.Connections, 105 | MAC: net.HardwareAddr(d.MAC[:]).String(), 106 | Current: d.ActiveConnections, 107 | } 108 | } 109 | } 110 | 111 | return 112 | } 113 | 114 | // Synchronise the manifest of services from the director/manager to the xvs client 115 | func (b *Balancer) Configure(manifests []vc5.Manifest) error { 116 | 117 | from_xvs := func(s Service) vc5.Service { 118 | return vc5.Service{Address: s.Address, Port: s.Port, Protocol: vc5.Protocol(s.Protocol)} 119 | } 120 | 121 | services := map[vc5.Service]vc5.Manifest{} 122 | 123 | // create a map of desired services and check that DSR restrictions are followed: 124 | for _, s := range manifests { 125 | services[s.Service()] = s 126 | 127 | for _, d := range s.Destinations { 128 | if s.Port != d.Port { 129 | return errors.New("Destination ports must match service ports for DSR") 130 | } 131 | } 132 | } 133 | 134 | // iterate through a list of active services and remove if no longer needed (desn't exist in the 'services' map): 135 | svcs, _ := b.Client.Services() 136 | for _, s := range svcs { 137 | if _, wanted := services[from_xvs(s.Service)]; !wanted { 138 | b.Client.RemoveService(s.Service) 139 | } 140 | } 141 | 142 | // for each desired service create the necessary xvs configuration (service description and list of backends) and apply: 143 | for _, s := range services { 144 | 145 | service := Service{Address: s.Address, Port: s.Port, Protocol: Protocol(s.Protocol), Sticky: s.Sticky} 146 | 147 | var dsts []Destination 148 | 149 | var tunnelType xvs.TunnelType = xvs.NONE 150 | 151 | switch s.TunnelType { 152 | case "gre": 153 | tunnelType = xvs.GRE 154 | case "ipip": 155 | tunnelType = xvs.IPIP 156 | case "fou": 157 | tunnelType = xvs.FOU 158 | case "gue": 159 | tunnelType = xvs.GUE 160 | } 161 | 162 | for _, d := range s.Destinations { 163 | if d.Port == s.Port { 164 | dsts = append(dsts, Destination{ 165 | Address: d.Address, 166 | Disable: d.HealthyWeight() == 0, 167 | TunnelType: tunnelType, 168 | TunnelPort: s.TunnelPort, 169 | TunnelEncapNoChecksum: s.TunnelEncapNoChecksum, 170 | }) 171 | } 172 | } 173 | 174 | b.Client.SetService(service, dsts...) 175 | } 176 | 177 | return nil 178 | } 179 | 180 | func (b *Balancer) Metrics() (n []string, metrics []string) { 181 | 182 | client := b.Client 183 | 184 | names := map[string]bool{} 185 | 186 | info, _ := client.Info() 187 | 188 | names["global_latency"] = true 189 | metrics = append(metrics, fmt.Sprintf("global_latency %d", info.Latency)) 190 | 191 | for k, v := range info.Metrics { 192 | name := "global_" + k 193 | inst := "" 194 | line := fmt.Sprintf("%s%s %d", name, inst, v) 195 | names[name] = k == "current" 196 | metrics = append(metrics, line) 197 | } 198 | 199 | services, _ := client.Services() 200 | 201 | for _, service := range services { 202 | 203 | s, _ := client.Service(service.Service) 204 | 205 | serv := s.Service 206 | proto := "" 207 | switch serv.Protocol { 208 | case xvs.TCP: 209 | proto = "tcp" 210 | case xvs.UDP: 211 | proto = "udp" 212 | default: 213 | proto = fmt.Sprint(proto) 214 | } 215 | snam := fmt.Sprintf("%s:%d:%s", serv.Address, serv.Port, proto) 216 | 217 | for k, v := range s.Metrics { 218 | inst := fmt.Sprintf(`{address="%s",port="%d",protocol="%s",service="%s"}`, serv.Address, serv.Port, proto, snam) 219 | name := "service_" + k 220 | line := fmt.Sprintf("%s%s %d", name, inst, v) 221 | names[name] = k == "current" 222 | metrics = append(metrics, line) 223 | } 224 | 225 | destinations, _ := client.Destinations(serv) 226 | 227 | for _, d := range destinations { 228 | dest := d.Destination 229 | 230 | for k, v := range d.Metrics { 231 | inst := fmt.Sprintf(`{address="%s",port="%d",protocol="%s",destination="%s",service="%s"}`, serv.Address, serv.Port, proto, dest.Address, snam) 232 | name := "destination_" + k 233 | line := fmt.Sprintf("%s%s %d", name, inst, v) 234 | names[name] = k == "current" 235 | metrics = append(metrics, line) 236 | } 237 | } 238 | } 239 | 240 | vips, _ := client.VIPs() 241 | 242 | for _, vip := range vips { 243 | for k, v := range vip.Metrics { 244 | inst := fmt.Sprintf(`{address="%s"}`, vip.Address) 245 | name := "virtual_" + k 246 | line := fmt.Sprintf("%s%s %d", name, inst, v) 247 | names[name] = k == "current" 248 | metrics = append(metrics, line) 249 | } 250 | } 251 | 252 | for k, t := range names { 253 | if t { 254 | n = append(n, k+" gauge") 255 | } else { 256 | n = append(n, k+" counter") 257 | } 258 | } 259 | 260 | return 261 | } 262 | -------------------------------------------------------------------------------- /cmd/config.pl: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env perl 2 | use strict; 3 | use YAML; 4 | use JSON; 5 | use Socket; 6 | use Getopt::Std; 7 | use feature qw(switch); 8 | no warnings qw(experimental::smartmatch); 9 | 10 | # This script is designed to take a comapct, human-readable YAML 11 | # configuration file and translate it into a more verbose, explicit 12 | # JSON format that the load balancer daemon can ingest. 13 | 14 | my $TRUE = JSON::true; 15 | my $FALSE = JSON::false; 16 | 17 | getopts('nhx', \my %opt); 18 | 19 | # -n NAT mode (allow port mappings) 20 | # -x Dump example config on stdout 21 | 22 | if($opt{'x'}) { 23 | print ; 24 | exit; 25 | } 26 | 27 | my $conf = YAML::Load(join('', <>)); 28 | my $json = {}; 29 | 30 | my $policy = $conf->{'policy'}; 31 | my $servers = $conf->{'servers'}; 32 | my $services = $conf->{'services'}; 33 | my %defaults; 34 | 35 | my $scheduler = $conf->{'scheduler'} if exists $conf->{'scheduler'}; 36 | 37 | $json->{'services'} = services($scheduler, $services, \%defaults, $servers, $policy); 38 | $json->{'bgp'} = new_rhi($conf->{'bgp'}, $conf->{'prefixes'}); 39 | $conf->{'learn'}+=0 if defined $conf->{'learn'}; 40 | 41 | foreach(qw(vlans vlans6 multicast webserver webroot defcon logging address interfaces native untagged host_id)) { 42 | $json->{$_} = $conf->{$_} if exists $conf->{$_}; 43 | } 44 | 45 | if(defined $conf->{'native'}) { 46 | $json->{'native'} = jsonbool($conf->{'native'}); 47 | } 48 | 49 | if(defined $conf->{'untagged'}) { 50 | $json->{'untagged'} = jsonbool($conf->{'untagged'}); 51 | } 52 | 53 | if(defined $conf->{'bgp'} && jsonbool($conf->{'bgp'}->{'listen'})) { 54 | $json->{'listen'} = $TRUE; 55 | } 56 | 57 | if(defined $conf->{'bgp'} && $conf->{'bgp'}->{'learn'} > 0) { 58 | $json->{'learn'} = $conf->{'bgp'}->{'learn'} + 0; 59 | } 60 | 61 | if(defined $json->{'logging'}) { 62 | $json->{'logging'}->{'alert'}+=0; 63 | 64 | # if(defined $json->{'logging'}->{'elasticsearch'}) { 65 | # my $val = jsonbool($json->{'logging'}->{'elasticsearch'}->{'data_stream'}); 66 | # $json->{'logging'}->{'elasticsearch'}->{'data_stream'} = $val; 67 | # } 68 | } 69 | 70 | if(defined $json->{'logging'}) { 71 | $json->{'logging'}->{'syslog'} = jsonbool($json->{'logging'}->{'syslog'}); 72 | } 73 | 74 | if(defined $json->{'defcon'}) { 75 | $json->{'defcon'}+=0; 76 | given($json->{'defcon'}) { 77 | when(1) {} 78 | when(2) {} 79 | when(3) {} 80 | when(4) {} 81 | when(5) {} 82 | default { die "defcon setting needs to be between an integer between 1 and 5"; } 83 | } 84 | } 85 | 86 | print to_json($json, {pretty => 1, canonical => 1}); 87 | 88 | exit; 89 | 90 | sub services { 91 | my($scheduler, $services, $defaults, $servers, $policy) = @_; 92 | my %defaults = %$defaults; 93 | my %out; 94 | 95 | my %disabled; 96 | 97 | foreach my $s (@$services) { 98 | 99 | $defaults{_host} = key($s, 'host', undef); # checks 100 | $defaults{_path} = key($s, 'path', undef); # checks 101 | $defaults{_meth} = key($s, 'method', undef); # checks 102 | $defaults{_expc} = key($s, 'expect', undef); # checks 103 | $defaults{_name} = key($s, 'name', undef); 104 | $defaults{_desc} = key($s, 'description', undef); 105 | $defaults{_prio} = key($s, 'priority', undef); 106 | $defaults{_ttyp} = key($s, 'tunnel-type', undef); 107 | $defaults{_tpor} = key($s, 'tunnel-port', 0)+0; 108 | $defaults{_need} = key($s, 'need', 1)+0; 109 | $defaults{_stic} = key($s, 'sticky', JSON::false); 110 | $defaults{_rest} = key($s, 'reset', JSON::false); 111 | $defaults{_schd} = key($s, 'scheduler', $scheduler); 112 | $defaults{_pers} = key($s, 'persist', undef); 113 | 114 | my @virtual; 115 | my @servers; 116 | my %policy; 117 | 118 | given(ref($s->{'virtual'})) { 119 | when('ARRAY') { @virtual = @{$s->{'virtual'}}} 120 | when('') { @virtual = ($s->{'virtual'}) } 121 | default { die } 122 | } 123 | 124 | given(ref($s->{'servers'})) { 125 | when('ARRAY') { @servers = @{$s->{'servers'}}} 126 | when('') { 127 | my $n = $s->{'servers'}; 128 | die "Server list '$n' does not exist\n" unless exists $servers->{$n}; 129 | @servers = @{$servers->{$n}}; 130 | 131 | } 132 | default { die } 133 | } 134 | 135 | given(ref($s->{'policy'})) { 136 | when('HASH') { %policy = %{$s->{'policy'}} } 137 | when('') { 138 | my $n = $s->{'policy'}; 139 | die "Policy '$n' does not exist\n" unless exists $policy->{$n}; 140 | %policy = %{$policy->{$n}}; 141 | } 142 | default { die } 143 | } 144 | 145 | my %servers; 146 | foreach(@servers) { 147 | die "bad server: $_\n" unless /^(\d+\.\d+\.\d+\.\d+|[0-9a-f:]+)(\*|)$/; 148 | $servers{$1} = {_dsbl => $2 eq '' ? 0 : 1}; 149 | } 150 | 151 | my @policy = policy(\%policy, \%defaults); 152 | 153 | foreach my $v (@virtual) { 154 | 155 | if($v =~ /^(.*)\*$/) { 156 | $v = $1; 157 | $disabled{$v} = 1; 158 | } 159 | 160 | foreach my $p (@policy) { 161 | my $l4 = $p->{_prot} . ':' . $p->{_port}; 162 | my @p = %$p; 163 | 164 | my $svc = { 'need' => $p->{_need}+0 }; 165 | 166 | $svc->{'name'} = $p->{_name} if defined $p->{_name}; 167 | $svc->{'description'} = $p->{_desc} if defined $p->{_desc}; 168 | $svc->{'priority'} = $p->{_prio} if defined $p->{_prio}; 169 | $svc->{'tunnel-type'} = $p->{_ttyp} if defined $p->{_ttyp}; 170 | $svc->{'tunnel-port'} = $p->{_tpor} if defined $p->{_tpor} && $p->{_tpor} != 0; 171 | $svc->{'scheduler'} = $p->{_schd} if defined $p->{_schd}; 172 | $svc->{'persist'} = $p->{_pers}+0 if defined $p->{_pers}; 173 | $svc->{'sticky'} = jsonbool($p->{_stic}) if defined $p->{_stic}; 174 | $svc->{'reset'} = jsonbool($p->{_rest}) if defined $p->{_rest}; 175 | 176 | if (defined $svc->{'tunnel-type'} && $svc->{'tunnel-type'} !~ /^(none|ipip|gre|fou|gue)$/) { 177 | my $ttype = $svc->{'tunnel-type'}; 178 | die "unsupported tunnel type '$ttype'\n"; 179 | } 180 | 181 | my %rips; 182 | 183 | my $checks = checklist(@{$p->{_chks}}); 184 | my $bind = $p->{_bind}+0; 185 | 186 | if($bind != 0 && $bind < 1 || $bind > 65535) { 187 | die "bind: $bind\n"; 188 | } 189 | 190 | if(!defined $opt{'n'} && $bind != $p->{_port}) { 191 | die "port mismatch! enable port mapping for non DSR with -n"; 192 | } 193 | 194 | if(defined $svc->{'priority'} && $svc->{'priority'} !~ /^(critical|high|medium|low)$/) { 195 | die "Invalid priority: ".$svc->{'priority'}."\n"; 196 | } 197 | 198 | foreach my $s (sort keys %servers) { 199 | $rips{$s.":$bind"} = { 200 | 'checks' => $checks, 201 | 'disabled' => $servers{$s}->{_dsbl} ? JSON::true : JSON::false, 202 | 'weight' => $servers{$s}->{_dsbl} ? 0 : 1, 203 | } 204 | } 205 | 206 | $svc->{'reals'} = \%rips; 207 | 208 | $out{$v.":".$p->{_port}.":".$p->{_prot}} = $svc; 209 | } 210 | } 211 | } 212 | 213 | foreach(keys %out) { 214 | $out{$_}->{'disabled'} = $TRUE if /^([^:]+):/ && $disabled{$1}; 215 | } 216 | 217 | return \%out; 218 | } 219 | 220 | sub checklist { 221 | my(@c) = @_; 222 | my @ret; 223 | foreach my $c (@c) { 224 | my $t = $c->{_type}; 225 | my $p = $c->{_port}+0; 226 | my %c; 227 | 228 | $c{'type'} = $t; 229 | $c{'port'} = $p if $p > 0; 230 | 231 | given($t) { 232 | when('dns') { 233 | #if($opt{m}) { 234 | $c{'method'} = $c->{_meth} if defined $c->{_meth}; 235 | #} else { 236 | # $c{'method'} = $c->{_meth} eq "tcp" ? $TRUE : $FALSE if defined $c->{_meth}; 237 | #} 238 | } 239 | 240 | when(/^(http|https)$/) { 241 | $c{'host'} = $c->{_host} if defined $c->{_host}; 242 | $c{'path'} = $c->{_path} if defined $c->{_path}; 243 | $c{'expect'} = expect($c->{_expc}) if defined $c->{_expc}; 244 | #if($opt{m}) { 245 | $c{'method'} = $c->{_meth} if defined $c->{_meth}; 246 | #} else { 247 | # $c{'method'} = $c->{_meth} eq "HEAD" ? $TRUE : $FALSE if defined $c->{_meth}; 248 | #} 249 | } 250 | } 251 | 252 | push @ret, \%c; 253 | } 254 | 255 | return [ @ret ]; 256 | } 257 | 258 | sub expect { 259 | my($expect) = @_; 260 | my @expect; 261 | 262 | return [ 0 ] if $expect eq 'any'; 263 | 264 | foreach (split(/\s+/, $expect)) { 265 | my @val; 266 | 267 | if(/([1-9][0-9][0-9])-([1-9][0-9][0-9])$/) { 268 | if($1 > $2) { 269 | @val = $2..$1; 270 | } else { 271 | @val = $1..$2; 272 | } 273 | } else { 274 | die unless /^[1-9][0-9][0-9]$/; 275 | @val = ($_+0); 276 | } 277 | 278 | push @expect, @val; 279 | } 280 | 281 | return [ @expect ]; 282 | } 283 | 284 | sub policy { 285 | my($policy, $defaults) = @_; 286 | my @policy; 287 | my %policy = %$policy; 288 | 289 | foreach my $p (sort keys %policy) { 290 | my $v = $policy{$p}; 291 | $v = {} unless defined $v; # policy may be void - eg. all defaults 292 | 293 | given(ref($v)) { 294 | when ('HASH') {} 295 | when ('') { 296 | given ($v) { 297 | when (/^[1-9][0-9]*$/) { $v = {'bind' => $v} } 298 | default { die "$v" } 299 | } 300 | } 301 | default { die ref($v) } 302 | } 303 | 304 | my $def = 1; 305 | my $tcp = 1; 306 | my $port = 0; 307 | my $type = "none"; 308 | 309 | if($p =~ /^(.*)\*$/) { 310 | $p = $1; 311 | $v->{'checks'} = []; 312 | $def = 0; 313 | } 314 | 315 | given ($p) { 316 | when (/^[1-9][0-9]*$/) { $port = $p; $type = "syn"; } 317 | when (m'^([1-9][0-9]*)/tcp$') { $port = $1; $type = "syn"; } 318 | when (m'^([1-9][0-9]*)/udp$') { $port = $1; $tcp = 0; } 319 | 320 | when (m'^(([1-9][0-9]*)/|)http$') { $port = $2 eq '' ? 80 : $2+0; $type = "http"; } 321 | when (m'^(([1-9][0-9]*)/|)https$') { $port = $2 eq '' ? 443 : $2+0; $type = "https"; } 322 | when (m'^(([1-9][0-9]*)/|)domain$') { $port = $2 eq '' ? 53 : $2+0; $type = "domain"; } 323 | 324 | when ('domain/tcp') { $port = 53; $type = "dns"; $tcp = 1 } 325 | when ('domain/udp') { $port = 53; $type = "dns"; $tcp = 0 } 326 | 327 | when ('ftp') { $port = 21; $type = "syn"; } 328 | when ('smtp') { $port = 25; $type = "syn"; } 329 | when ('ssh') { $port = 22; $type = "syn"; } 330 | when ('telnet') { $port = 23; $type = "syn"; } 331 | when ('pop2') { $port = 109; $type = "syn"; } 332 | when ('pop3') { $port = 110; $type = "syn"; } 333 | when ('imap') { $port = 143; $type = "syn"; } 334 | when ('imaps') { $port = 993; $type = "syn"; } 335 | 336 | default { die "policy: $p\n" } 337 | } 338 | 339 | $port = int($port)+0; 340 | die "port: $port\n" if $port < 1 || $port > 65535; 341 | 342 | $type = "none" if !$def; 343 | 344 | given ($type) { 345 | when ("domain") { 346 | push @policy, service('dns', 1, $port, $v, $defaults); 347 | push @policy, service('dns', 0, $port, $v, $defaults); 348 | } 349 | 350 | default { push @policy, service($type, $tcp, $port, $v, $defaults) } 351 | } 352 | } 353 | 354 | return @policy; 355 | } 356 | 357 | sub service() { 358 | my($type, $tcp, $port, $policy, $defaults) = @_; 359 | my $protocol = $tcp ? "tcp" : "udp"; 360 | 361 | my %defaults = %$defaults if defined $defaults; # SUSPECT 362 | 363 | $defaults{_host} = $policy->{'host'} if exists $policy->{'host'}; 364 | $defaults{_path} = $policy->{'path'} if exists $policy->{'path'}; 365 | $defaults{_meth} = $policy->{'method'} if exists $policy->{'method'}; 366 | $defaults{_expc} = $policy->{'expect'} if exists $policy->{'expect'}; 367 | 368 | my @checks = @{$policy->{'checks'}} if defined $policy->{'checks'}; # SUSPECT 369 | 370 | return { 371 | _prot => $protocol, 372 | _port => $port, 373 | 374 | _pers => key($policy, 'persist', $defaults->{_pers}), 375 | _schd => key($policy, 'scheduler', $defaults->{_schd}), 376 | _stic => key($policy, 'sticky', $defaults->{_stic}), 377 | _rest => key($policy, 'reset', $defaults->{_rest}), 378 | _need => key($policy, 'need', $defaults->{_need}), 379 | _name => key($policy, 'name', $defaults->{_name}), 380 | _ttyp => key($policy, 'tunnel-type', $defaults->{_ttyp}), 381 | _tpor => key($policy, 'tunnel-port', $defaults->{_tpor}), 382 | _desc => key($policy, 'description', $defaults->{_desc}), 383 | _prio => key($policy, 'priority', $defaults->{_prio}), 384 | _bind => key($policy, 'bind', $port)+0, 385 | _chks => [ checks($tcp, $port, $type, $policy, \%defaults, @checks) ], 386 | }; 387 | } 388 | 389 | sub checks() { 390 | my($tcp, $port, $type, $policy, $defaults, @checks) = @_; 391 | my %d = %$defaults; 392 | my @c; 393 | 394 | if(scalar(@checks) == 0) { 395 | given ($type) { 396 | when ('none') { } 397 | when (/^http|htts$/) { 398 | push @c, { 399 | _type => $type, 400 | _host => $d{_host}, 401 | _path => $d{_path}, 402 | _meth => $d{_meth}, 403 | _expc => $d{_expc}, 404 | }; 405 | } 406 | 407 | when('dns') { 408 | my $meth = $d{_meth}; 409 | $meth = $defaults->{_meth} if !defined $meth && defined $defaults->{_meth}; 410 | $meth = $tcp ? "tcp" : "udp" if (!defined $meth || $meth !~ /^(tcp|udp)$/i ); 411 | 412 | push @c, { 413 | _type => $type, 414 | _meth => $meth, 415 | }; 416 | } 417 | 418 | when ('syn') { push @c, { _type => $type } } 419 | 420 | default { die "$type\n" } 421 | } 422 | } else { 423 | foreach my $c (@checks) { 424 | my $type = $c->{"type"}; 425 | my $port = key($c, 'port', 0)+0; 426 | 427 | given ($type) { 428 | when (/^http|htts$/) { 429 | push @c, { 430 | _type => $type, 431 | _host => key($c, 'host', $d{_host}), 432 | _path => key($c, 'path', $d{_path}), 433 | _meth => key($c, 'method', $d{_meth}), 434 | _expc => key($c, 'expect', $d{_expc}), 435 | _port => $port, 436 | }; 437 | } 438 | 439 | when ('dns') { 440 | my $meth = $c->{"method"}; 441 | $meth = $defaults->{_meth} if !defined $meth && defined $defaults->{_meth}; 442 | $meth = undef unless $meth =~ /^(tcp|udp)$/; 443 | $meth = $tcp ? "tcp" : "udp" unless defined $meth; 444 | push @c, { 445 | _type => $type, 446 | _meth => $meth, 447 | _port => $port, 448 | }; 449 | } 450 | 451 | when ('syn') { push @c, { _type => $type, _port => $port } } 452 | 453 | default { die "$type\n" } 454 | } 455 | } 456 | } 457 | 458 | return @c; 459 | } 460 | 461 | sub key { 462 | my($a, $k, $d) = @_; 463 | 464 | my $ret = defined $a->{$k} ? $a->{$k} : $d; 465 | 466 | return undef unless defined $ret; 467 | 468 | die "Name '$ret' isn't valid\n" if $k eq 'name' && $ret !~ /^[a-z0-9][-a-z0-9]*$/i; 469 | #die "Expect '$ret' isn't valid\n" if $k eq 'expect' && $ret !~ /^[1-9][0-9][0-9]$/; 470 | die "Method '$ret' isn't valid\n" if $k eq 'method' && $ret !~ /^(HEAD|GET)$/; 471 | 472 | return $ret; 473 | } 474 | 475 | sub jsonbool { 476 | my($v) = @_; 477 | return $v eq 'true' ? JSON::true : JSON::false; 478 | } 479 | 480 | sub yamlbool { 481 | my($v) = @_; 482 | return $v =~ /^(true|yes|on)$/i ? JSON::true : JSON::false; 483 | } 484 | 485 | 486 | ###################################################################### 487 | 488 | sub filter { 489 | my($m, $n) = @_; 490 | return "0.0.0.0/0" if $n eq 'any'; 491 | return $n unless defined $m && exists $m->{$n}; 492 | return @{$m->{$n}}; 493 | } 494 | 495 | 496 | sub new_rhi { 497 | my($rhi, $map) = @_; 498 | 499 | my $default = params($rhi); 500 | my %peers = map { $_ => $default } @{$rhi->{'peers'}} if defined $rhi->{'peers'}; # SUSPECT 501 | 502 | 503 | if(defined $rhi->{'groups'}) { 504 | foreach my $g (@{$rhi->{'groups'}}) { 505 | #my @accept = map { filter($map, $_) } @{$g->{'accept'}} if defined $g->{'accept'}; 506 | #my @reject = map { filter($map, $_) } @{$g->{'reject'}} if defined $g->{'reject'}; 507 | my @accept; 508 | my @reject; 509 | @accept = map { filter($map, $_) } @{$g->{'accept'}} if defined $g->{'accept'}; 510 | @reject = map { filter($map, $_) } @{$g->{'reject'}} if defined $g->{'reject'}; 511 | 512 | my $d = params($g, %$default); 513 | 514 | if(defined $g->{'peers'}) { 515 | foreach my $p (@{$g->{'peers'}}) { 516 | die "ASN not set for $p\n" unless $d->{'as_number'} > 0; 517 | $d->{'accept'} = \@accept; 518 | $d->{'reject'} = \@reject; 519 | $peers{$p} = $d; 520 | } 521 | } 522 | } 523 | } 524 | 525 | return \%peers; 526 | } 527 | 528 | sub params { 529 | my($o, %p) = @_; 530 | 531 | $p{'communities'} = $o->{'communities'} if defined $o->{'communities'}; 532 | $p{'source_ip'} = $o->{'source_ip'} if defined $o->{'source_ip'}; 533 | $p{'as_number'} = $o->{'as_number'}+0 if defined $o->{'as_number'}; 534 | $p{'hold_time'} = $o->{'hold_time'}+0 if defined $o->{'hold_time'}; 535 | $p{'local_pref'} = $o->{'local_pref'}+0 if defined $o->{'local_pref'}; 536 | $p{'med'} = $o->{'med'}+0 if defined $o->{'med'}; 537 | if (defined $o->{'next_hop_6'}) { 538 | $p{'multiprotocol'} = $TRUE; 539 | $p{'next_hop_6'} = $o->{'next_hop_6'}; 540 | } 541 | return \%p; 542 | } 543 | 544 | 545 | __END__; 546 | --- 547 | 548 | #webserver: :80 549 | #webroot: /var/local/vc5 550 | #multicast: 224.0.0.1:12345 551 | 552 | #native: false 553 | #untagged: false 554 | #address: 10.1.10.100 # load balancer server's primary ip 555 | #interfaces: 556 | # - ens192 557 | # - ens224 558 | 559 | bgp: 560 | as_number: 65000 561 | peers: 562 | - 10.1.10.200 563 | - 10.1.10.201 564 | 565 | # If Teams or Slack webhook URLs are set then messages of level (default 0) or lower wil be sent to the channel. 566 | # If elasticsearch/index is set then all logs will be written to elasticsearch 567 | # Other setting are optional, and the usual Elasticsearch environment variables will be consulted by the library 568 | 569 | #logging: 570 | # #alert: 4 # 0:EMERG, 1:ALERT, 2:CRIT, 3:ERR, 4:WARNING, 5:NOTICE, 6:INFO, 7:DEBUG 571 | # #teams: https://myorganisation.webhook.office.com/webhookb2/.... 572 | # #slack: https://hooks.slack.com/services/.... 573 | # elasticsearch: 574 | # index: vc5 575 | # #addresses: 576 | # # - http://10.1.2.31/ 577 | # # - http://10.1.2.32/ 578 | # #username: elastic 579 | # #password: Xg5nRkc9RA3hALMiBw8X 580 | 581 | #vlans: 582 | # 10: 10.1.10.0/24 583 | # 20: 10.1.20.0/24 584 | # 30: 10.1.30.0/24 585 | # 40: 10.1.40.0/24 586 | 587 | services: 588 | 589 | - name: nginx 590 | virtual: 591 | - 192.168.101.1 592 | servers: 593 | - 10.1.10.10 594 | - 10.1.10.11 595 | - 10.1.10.12 596 | - 10.1.10.13 597 | need: 1 598 | path: /alive 599 | policy: 600 | http: 601 | 602 | # - name: bind 603 | # description: DNS servers on a different VLAN 604 | # virtual: 605 | # - 192.168.101.2 606 | # servers: 607 | # - 10.1.20.10 608 | # - 10.1.20.12 609 | # - 10.1.20.13 610 | # - 10.1.20.14 611 | # policy: 612 | # domain: 613 | -------------------------------------------------------------------------------- /cmd/config.sample.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | bgp: 3 | as_number: 65000 4 | peers: 5 | - 10.1.10.200 6 | - 10.1.10.201 7 | 8 | # If Teams or Slack webhook URLs are set then messages of level (default 0) or lower wil be sent to the channel. 9 | # If elasticsearch/index is set then all logs will be written to elasticsearch 10 | # Other setting are optional, and the usual Elasticsearch environment variables will be consulted by the library 11 | 12 | #logging: 13 | # #alert: 4 # 0:EMERG, 1:ALERT, 2:CRIT, 3:ERR, 4:WARNING, 5:NOTICE, 6:INFO, 7:DEBUG 14 | # #teams: https://myorganisation.webhook.office.com/webhookb2/.... 15 | # #slack: https://hooks.slack.com/services/.... 16 | # elasticsearch: 17 | # index: vc5 18 | # #addresses: 19 | # # - http://10.1.2.31/ 20 | # # - http://10.1.2.32/ 21 | # #username: elastic 22 | # #password: Xg5nRkc9RA3hALMiBw8X 23 | 24 | #vlans: 25 | # 10: 10.1.10.0/24 26 | # 20: 10.1.20.0/24 27 | # 30: 10.1.30.0/24 28 | # 40: 10.1.40.0/24 29 | 30 | services: 31 | 32 | - name: nginx 33 | virtual: 34 | - 192.168.101.1 35 | servers: 36 | - 10.1.10.10 37 | - 10.1.10.11 38 | - 10.1.10.12 39 | - 10.1.10.13 40 | need: 1 41 | path: /alive 42 | policy: 43 | http: 44 | 45 | # - name: bind 46 | # description: DNS servers on a different VLAN 47 | # virtual: 48 | # - 192.168.101.2 49 | # servers: 50 | # - 10.1.20.10 51 | # - 10.1.20.12 52 | # - 10.1.20.13 53 | # - 10.1.20.14 54 | # policy: 55 | # domain: 56 | -------------------------------------------------------------------------------- /cmd/go.mod: -------------------------------------------------------------------------------- 1 | module main 2 | 3 | go 1.19 4 | 5 | require ( 6 | github.com/davidcoles/cue v0.1.4 7 | github.com/davidcoles/xvs v0.2.10 8 | vc5 v0.0.0 9 | ) 10 | 11 | require ( 12 | github.com/davidcoles/bgp v0.0.4 // indirect 13 | github.com/elastic/go-elasticsearch/v7 v7.17.10 // indirect 14 | ) 15 | 16 | replace vc5 => ../. 17 | 18 | //replace github.com/davidcoles/cue => ../../cue 19 | //replace github.com/davidcoles/xvs => ../../xvs 20 | //replace github.com/davidcoles/bgp => ../../bgp 21 | -------------------------------------------------------------------------------- /cmd/go.sum: -------------------------------------------------------------------------------- 1 | github.com/davidcoles/bgp v0.0.4 h1:K63kH+vnDnFFrhKqftekNELB1C4lOxxmmXFzlJKI5VA= 2 | github.com/davidcoles/bgp v0.0.4/go.mod h1:d9tqNdWCrF0N8LWfOEqrA/GSeMj/XycB60WBZ4uDvEo= 3 | github.com/davidcoles/cue v0.1.4 h1:KrH2hhOLwVKxbEkQL0a6zjXhwZSCzyt0vkRkq0fO7ZM= 4 | github.com/davidcoles/cue v0.1.4/go.mod h1:26FTBytVHJ1XQWOGC+Cfnx4Q9xV8k1xr6K5uZ7s/EBw= 5 | github.com/davidcoles/xvs v0.2.10 h1:aBcwheQMPcjUuW5FLBsi36ljlLJlLdnK4X33ZpsKzOU= 6 | github.com/davidcoles/xvs v0.2.10/go.mod h1:NQg6Ob9zLr49qH3I6E6zqnGp4tVBu4cT/3vXUB2XA8w= 7 | github.com/elastic/go-elasticsearch/v7 v7.17.10 h1:TCQ8i4PmIJuBunvBS6bwT2ybzVFxxUhhltAs3Gyu1yo= 8 | github.com/elastic/go-elasticsearch/v7 v7.17.10/go.mod h1:OJ4wdbtDNk5g503kvlHLyErCgQwwzmDtaFC4XyOxXA4= 9 | -------------------------------------------------------------------------------- /cmd/main.go: -------------------------------------------------------------------------------- 1 | /* 2 | * VC5 load balancer. Copyright (C) 2021-present David Coles 3 | * 4 | * This program is free software; you can redistribute it and/or modify 5 | * it under the terms of the GNU General Public License as published by 6 | * the Free Software Foundation; either version 2 of the License, or 7 | * (at your option) any later version. 8 | * 9 | * This program is distributed in the hope that it will be useful, 10 | * but WITHOUT ANY WARRANTY; without even the implied warranty of 11 | * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 12 | * GNU General Public License for more details. 13 | * 14 | * You should have received a copy of the GNU General Public License along 15 | * with this program; if not, write to the Free Software Foundation, Inc., 16 | * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 17 | */ 18 | 19 | package main 20 | 21 | import ( 22 | "context" 23 | "encoding/json" 24 | "flag" 25 | "fmt" 26 | "log" 27 | "net" 28 | "net/http" 29 | "net/netip" 30 | "os" 31 | "os/exec" 32 | "os/signal" 33 | "syscall" 34 | "time" 35 | 36 | "github.com/davidcoles/xvs" 37 | "vc5" 38 | ) 39 | 40 | type KV = map[string]any 41 | type leveler struct{} 42 | 43 | func main() { 44 | 45 | const FACILITY = "vc5" 46 | 47 | // XVS specific 48 | learn := flag.Uint("l", 0, "Learn; wait for this many seconds before advertising VIPs (for multicast flow state adverts)") 49 | native := flag.Bool("n", false, "Use native mode XDP; better performance on network cards that support it") 50 | multicast := flag.String("m", "", "Multicast address used to share flow state between instances") 51 | flows := flag.Uint("F", 0, "Set maximum number of flows (per-core)") 52 | delay := flag.Uint("D", 0, "Delay between initialisaton of interfaces (to prevent bond from flapping)") 53 | //cmd_path := flag.String("C", "", "Command channel path") 54 | 55 | // common with stayinalived 56 | listen := flag.Bool("b", false, "Enable BGP listener on port 179") 57 | webroot := flag.String("r", "", "Webserver root directory to override built-in documents") 58 | webserver := flag.String("w", ":80", "Webserver listen address") 59 | asn := flag.Uint("A", 0, "Autonomous System Number to enable loopback BGP") 60 | hardfail := flag.Bool("H", false, "Hard fail on balancer configuration error") 61 | closeidle := flag.Bool("c", false, "Close idle HTTP connections") 62 | hostid := flag.String("I", "", "Host ID for logging") 63 | 64 | flag.Bool("toobig", false, "dummy") 65 | 66 | timeout := flag.Uint("timeout", 0, "Timeout program after this many minutes - fail safe for testing") 67 | test := flag.Bool("test", false, "test mode - debug logging") 68 | 69 | // Changing number of flows will only work on newer kernels 70 | // Not supported: 5.4.0-171-generic 71 | // Supported: 5.15.0-112-generic, 6.6.28+rpt-rpi-v7 72 | 73 | flag.Parse() 74 | 75 | args := flag.Args() 76 | 77 | addr := args[0] 78 | file := args[1] 79 | nics := args[2:] 80 | 81 | config, err := vc5.Load(file) 82 | 83 | if err != nil { 84 | log.Fatal("Couldn't load config file:", config, err) 85 | } 86 | 87 | if *hostid == "" { 88 | *hostid = addr 89 | } 90 | 91 | logs := vc5.NewLogger(*hostid, config.LoggingConfig()) 92 | 93 | if len(nics) < 1 { 94 | logs.Fatal(FACILITY, "args", KV{"error.message": "No interfaces defined"}) 95 | } 96 | 97 | address, err := netip.ParseAddr(addr) 98 | 99 | if err != nil { 100 | logs.Fatal(FACILITY, "args", KV{"error.message": "Invalid address"}) 101 | } 102 | 103 | if !address.Is4() { 104 | logs.Fatal(FACILITY, "args", KV{"error.message": "Address is not IPv4: " + address.String()}) 105 | } 106 | 107 | routerID := address.As4() 108 | 109 | var webListener net.Listener 110 | 111 | // Before making any changes to the state of the system (loading 112 | // XDP, etc) we attempt to listen on the webserver port. This 113 | // should prevent multiple instances running at the same time and 114 | // interfering with each other. 115 | if *webserver != "" { 116 | webListener, err = net.Listen("tcp", *webserver) 117 | if err != nil { 118 | log.Fatal(err) 119 | } 120 | } 121 | 122 | // If BGP peers do not support a "passive" option (eg. ExtremeXOS) 123 | // then we may need to listen on port 179 to prevent the session 124 | // getting into an error state - the manager will accept the 125 | // connection but then quietly drop it after ten seconds or 126 | // so. This seems to keep the peer happy. 127 | if *listen { 128 | err = bgpListener(logs) 129 | if err != nil { 130 | log.Fatal(err) 131 | } 132 | } 133 | 134 | // Open a UNIX domain socket for receiving commands whilst 135 | // running. Currently used to re-attach XDP code to an interface 136 | // as a mitigation for some badly behaved network cards. 137 | // var cmd_sock net.Listener 138 | //if *cmd_path != "" { 139 | // os.Remove(*cmd_path) 140 | // if cmd_sock, err = net.Listen("unix", *cmd_path); err != nil { 141 | // log.Fatal(err) 142 | // } 143 | //} 144 | 145 | // Run ethtool against the network interfaces to disable various 146 | // offload parameters which seem to interfere with XDP operations 147 | ethtool(nics) 148 | 149 | // Initialise the load balancing library which will deal with the 150 | // data-plane - this is what actually switches incoming packets 151 | 152 | opts := xvs.Options{ 153 | IPv4VLANs: config.Prefixes(), 154 | IPv6VLANs: config.Prefixes6(), 155 | DriverMode: *native, 156 | FlowsPerCPU: uint32(*flows), 157 | InterfaceInitDelay: uint8(*delay), 158 | Bonding: false, 159 | } 160 | 161 | if *test { 162 | opts.Logger = logs 163 | } 164 | 165 | client, err := xvs.NewWithOptions(opts, nics...) 166 | 167 | if err != nil { 168 | logs.Fatal(FACILITY, "client", KV{"error.message": "Couldn't start client: " + err.Error()}) 169 | } 170 | 171 | info, err := client.Info() 172 | 173 | if err != nil { 174 | logs.Fatal(FACILITY, "client", KV{"error.message": "Couldn't get client info: " + err.Error()}) 175 | } 176 | 177 | inside := info.IPv4 178 | monitor, err := vc5.Monitor(inside, false) 179 | 180 | if *timeout > 0 { 181 | go func() { 182 | time.Sleep(time.Minute * time.Duration(*timeout)) 183 | log.Fatal("timeout") 184 | }() 185 | } 186 | 187 | // Short delay to let interfaces quiesce after loading XDP 188 | //time.Sleep(5 * time.Second) 189 | 190 | // Add a short delay on return to allow BGP, etc to cleanly exit 191 | defer time.Sleep(5 * time.Second) 192 | 193 | // Create a balancer instance - this implements interface methods 194 | // (configuration changes, stats requests, etc). which are called 195 | // by the manager object (which handles the main event loop) 196 | balancer := &Balancer{ 197 | Client: client, 198 | Logger: logs.Sub("balancer"), 199 | //tunnel: tunnel, 200 | //port: uint16(*tunnelPort), 201 | } 202 | 203 | // Run services to perform healthchecks in network namespace, handle 204 | // commands from UNIX socket and share flow info via multicast 205 | //services(os.Args[0], *closeidle, client, *socket, cmd_sock, *multicast, balancer.Logger) 206 | services(os.Args[0], *closeidle, client, *multicast, balancer.Logger) 207 | 208 | // Add some custom HTTP endpoints to the default mux to handle 209 | // requests specific to this type of load balancer client 210 | httpEndpoints(client, balancer, logs) 211 | 212 | // context to use for shutting down services when we're about to exit 213 | ctx, shutdown := context.WithCancel(context.Background()) 214 | defer shutdown() 215 | 216 | // The manager handles the main event loop, healthchecks, requests 217 | // for the console/metrics, sets up BGP sessions, etc. 218 | manager := vc5.Manager{ 219 | Balancer: balancer, 220 | Logs: logs, 221 | Learn: *learn, // Number of seconds to wait before advertising any VIPs 222 | NAT: nat(client), // We use a NAT method and a custom probe function 223 | Prober: prober(client, monitor), // to run checks from the inside network namespace 224 | RouterID: routerID, // BGP router ID to use to speak to peers 225 | WebRoot: *webroot, // Serve static files from this directory 226 | WebListener: webListener, // Listen for incoming web connections if not nil 227 | BGPLoopback: uint16(*asn), // If non-zero then loopback BGP is enabled 228 | Interval: 2, // Delay in seconds between updating statistics 229 | HardFail: *hardfail, // Exit if apply (not load) of config fails, when set 230 | } 231 | 232 | if err := manager.Manage(ctx, config); err != nil { 233 | logs.Fatal(FACILITY, "manager", KV{"error.message": "Couldn't start manager: " + err.Error()}) 234 | } 235 | 236 | // We are succesfully up and running, so send a high priority 237 | // alert to let the world know - perhaps we crashed previously and 238 | // were restarted by the service manager 239 | logs.Alert(vc5.ALERT, FACILITY, "initialised", KV{}, "Initialised") 240 | 241 | // We now wait for signals to tell us to reload the configuration file or exit 242 | sig := make(chan os.Signal, 10) 243 | signal.Notify(sig, syscall.SIGUSR2, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT) 244 | 245 | for { 246 | switch <-sig { 247 | case syscall.SIGINT: 248 | fallthrough 249 | case syscall.SIGUSR2: 250 | logs.Alert(vc5.NOTICE, FACILITY, "reload", KV{}, "Reload signal received") 251 | conf, err := vc5.Load(file) 252 | if err == nil { 253 | config = conf 254 | c := xvs.Config{IPv4VLANs: config.Prefixes(), IPv6VLANs: config.Prefixes6()} 255 | err := client.SetConfig(c) 256 | log.Println(err) 257 | manager.Configure(conf) 258 | } else { 259 | text := "Couldn't load config file " + file + " :" + err.Error() 260 | logs.Alert(vc5.ALERT, FACILITY, "config", KV{"file.path": file, "error.message": err.Error()}, text) 261 | } 262 | 263 | case syscall.SIGTERM: 264 | fallthrough 265 | case syscall.SIGQUIT: 266 | logs.Alert(vc5.ALERT, FACILITY, "exiting", KV{}, "Exiting") 267 | return 268 | } 269 | } 270 | } 271 | 272 | func httpEndpoints(client Client, balancer *Balancer, logs vc5.Logger) { 273 | 274 | /* 275 | http.HandleFunc("/prefixes.json", func(w http.ResponseWriter, r *http.Request) { 276 | t := time.Now() 277 | p := client.Prefixes() 278 | milliseconds := time.Now().Sub(t) / time.Millisecond 279 | logs.Event(6, "web", "prefixes", KV{"milliseconds": milliseconds}) 280 | js, err := json.Marshal(&p) 281 | if err != nil { 282 | w.WriteHeader(http.StatusInternalServerError) 283 | return 284 | } 285 | js = append(js, 0x0a) // add a newline for readability 286 | w.Header().Set("Content-Type", "application/json") 287 | w.Write(js) 288 | }) 289 | */ 290 | 291 | http.HandleFunc("/xxmetrics", func(w http.ResponseWriter, r *http.Request) { 292 | names, metrics := balancer.Metrics() 293 | w.Header().Set("Content-Type", "text/plain") 294 | 295 | for _, n := range names { 296 | w.Write([]byte(fmt.Sprintf("# TYPE xvs_%s counter\n", n))) 297 | } 298 | 299 | for _, m := range metrics { 300 | w.Write([]byte(fmt.Sprintln("xvs_" + m))) 301 | } 302 | }) 303 | 304 | http.HandleFunc("/lb.json", func(w http.ResponseWriter, r *http.Request) { 305 | var ret []interface{} 306 | type status struct { 307 | Service ServiceExtended 308 | Destinations []DestinationExtended 309 | } 310 | svcs, _ := client.Services() 311 | for _, se := range svcs { 312 | dsts, _ := client.Destinations(se.Service) 313 | ret = append(ret, status{Service: se, Destinations: dsts}) 314 | } 315 | js, err := json.MarshalIndent(&ret, " ", " ") 316 | if err != nil { 317 | w.WriteHeader(http.StatusInternalServerError) 318 | return 319 | } 320 | js = append(js, 0x0a) // add a newline for readability 321 | w.Header().Set("Content-Type", "application/json") 322 | w.Write(js) 323 | }) 324 | } 325 | 326 | func bgpListener(logs vc5.Logger) error { 327 | F := "bgp.listener" 328 | 329 | l, err := net.Listen("tcp", ":179") 330 | 331 | if err == nil { 332 | go func() { 333 | for { 334 | conn, err := l.Accept() 335 | 336 | if err != nil { 337 | logs.Event(vc5.ERR, F, "accept", KV{"error.message": err.Error()}) 338 | } else { 339 | go func(c net.Conn) { 340 | defer c.Close() 341 | logs.Event(vc5.INFO, F, "accept", KV{"client.address": conn.RemoteAddr().String()}) 342 | time.Sleep(time.Second * 10) 343 | }(conn) 344 | } 345 | } 346 | }() 347 | } 348 | 349 | return err 350 | } 351 | 352 | // func services(binary string, closeidle bool, client Client, socket string, cmd_sock net.Listener, multicast string, logger vc5.Logger) { 353 | func services(binary string, closeidle bool, client Client, multicast string, logger vc5.Logger) { 354 | /* 355 | cmd := []string{binary} 356 | 357 | if closeidle { 358 | cmd = append(cmd, "-I") 359 | } 360 | 361 | cmd = append(cmd, "-P", socket, client.NamespaceAddress()) 362 | 363 | go spawn(logger, client.Namespace(), cmd...) 364 | */ 365 | 366 | //go readCommands(cmd_sock, client, logger) 367 | 368 | if multicast != "" { 369 | go multicast_send(client, multicast) 370 | go multicast_recv(client, multicast) 371 | } 372 | } 373 | 374 | func ethtool(nics []string) { 375 | for _, i := range nics { 376 | exec.Command("ethtool", "-K", i, "rx", "off").Output() 377 | exec.Command("ethtool", "-K", i, "tx", "off").Output() 378 | exec.Command("ethtool", "-K", i, "rxvlan", "off").Output() 379 | exec.Command("ethtool", "-K", i, "txvlan", "off").Output() 380 | } 381 | } 382 | -------------------------------------------------------------------------------- /cmd/tests/Makefile: -------------------------------------------------------------------------------- 1 | # regression tests for the config parser 2 | 3 | SERVICES != (ls services*.yaml | sed 's/yaml$$/test/') 4 | BGP != (ls bgp*.yaml | sed 's/yaml$$/test/') 5 | 6 | tests: services bgp 7 | 8 | services: $(SERVICES) 9 | bgp: $(BGP) 10 | 11 | 12 | %.test:; ../config.pl -n $*.yaml | ./compare.pl $*.json 13 | ref:; for json in $$(ls *.yaml | sed 's/yaml$$/json/'); do $(MAKE) $$json; done 14 | %.json: %.yaml; ../config.pl -n $< > $@- && mv $@- $@ 15 | reset:; rm -f -- *.json 16 | 17 | -------------------------------------------------------------------------------- /cmd/tests/bgp01.json: -------------------------------------------------------------------------------- 1 | { 2 | "bgp" : { 3 | "10.1.2.252" : { 4 | "as_number" : 65001 5 | }, 6 | "10.1.2.253" : { 7 | "as_number" : 65001 8 | } 9 | }, 10 | "learn" : 20, 11 | "services" : {} 12 | } 13 | -------------------------------------------------------------------------------- /cmd/tests/bgp01.yaml: -------------------------------------------------------------------------------- 1 | 2 | bgp: 3 | learn: 20 4 | as_number: 65001 5 | peers: 6 | - 10.1.2.252 7 | - 10.1.2.253 8 | 9 | 10 | -------------------------------------------------------------------------------- /cmd/tests/bgp02.json: -------------------------------------------------------------------------------- 1 | { 2 | "bgp" : { 3 | "10.1.2.252" : { 4 | "accept" : [ 5 | "192.168.123.0/24", 6 | "192.168.124.0/24" 7 | ], 8 | "as_number" : 65001, 9 | "hold_time" : 10, 10 | "reject" : [ 11 | "0.0.0.0/0" 12 | ] 13 | }, 14 | "10.1.2.253" : { 15 | "accept" : [ 16 | "192.168.123.0/24", 17 | "192.168.124.0/24" 18 | ], 19 | "as_number" : 65001, 20 | "hold_time" : 10, 21 | "reject" : [ 22 | "0.0.0.0/0" 23 | ] 24 | }, 25 | "10.1.3.252" : { 26 | "accept" : [], 27 | "as_number" : 65002, 28 | "communities" : [ 29 | "65002:12345" 30 | ], 31 | "hold_time" : 10, 32 | "reject" : [ 33 | "192.168.123.0/24", 34 | "192.168.124.0/24" 35 | ] 36 | }, 37 | "10.1.3.253" : { 38 | "accept" : [], 39 | "as_number" : 65002, 40 | "communities" : [ 41 | "65002:12345" 42 | ], 43 | "hold_time" : 10, 44 | "reject" : [ 45 | "192.168.123.0/24", 46 | "192.168.124.0/24" 47 | ] 48 | } 49 | }, 50 | "learn" : 120, 51 | "listen" : true, 52 | "services" : {} 53 | } 54 | -------------------------------------------------------------------------------- /cmd/tests/bgp02.yaml: -------------------------------------------------------------------------------- 1 | 2 | bgp: 3 | learn: 120 4 | listen: true 5 | hold_time: 10 6 | groups: 7 | 8 | - name: group-a 9 | as_number: 65001 10 | accept: 11 | - group-a 12 | reject: 13 | - any 14 | peers: 15 | - 10.1.2.252 16 | - 10.1.2.253 17 | 18 | - name: group-b 19 | as_number: 65002 20 | reject: 21 | - group-a 22 | peers: 23 | - 10.1.3.252 24 | - 10.1.3.253 25 | communities: 26 | - 65002:12345 27 | 28 | prefixes: 29 | group-a: 30 | - 192.168.123.0/24 31 | - 192.168.124.0/24 32 | -------------------------------------------------------------------------------- /cmd/tests/compare.pl: -------------------------------------------------------------------------------- 1 | #!/usr/bin/perl 2 | use strict; 3 | use YAML; 4 | use JSON; 5 | use Test::Deep; 6 | use Test::Builder; 7 | 8 | my $file = shift or die; 9 | 10 | die unless open(JSON, '<', $file); 11 | my $ref = from_json(join('', )); 12 | my $test = from_json(join('', )); 13 | 14 | my $Test = Test::Builder->new; 15 | cmp_deeply($ref, $test, $file); 16 | $Test->done_testing(); 17 | -------------------------------------------------------------------------------- /cmd/tests/services01.json: -------------------------------------------------------------------------------- 1 | { 2 | "bgp" : {}, 3 | "services" : { 4 | "192.168.123.45:109:tcp" : { 5 | "description" : "Basic named services example", 6 | "name" : "services01", 7 | "need" : 1, 8 | "reals" : { 9 | "10.1.2.100:109" : { 10 | "checks" : [ 11 | { 12 | "type" : "syn" 13 | } 14 | ], 15 | "disabled" : false, 16 | "weight" : 1 17 | }, 18 | "10.1.2.101:109" : { 19 | "checks" : [ 20 | { 21 | "type" : "syn" 22 | } 23 | ], 24 | "disabled" : false, 25 | "weight" : 1 26 | }, 27 | "10.1.2.102:109" : { 28 | "checks" : [ 29 | { 30 | "type" : "syn" 31 | } 32 | ], 33 | "disabled" : false, 34 | "weight" : 1 35 | }, 36 | "10.1.2.103:109" : { 37 | "checks" : [ 38 | { 39 | "type" : "syn" 40 | } 41 | ], 42 | "disabled" : false, 43 | "weight" : 1 44 | } 45 | }, 46 | "reset" : false, 47 | "sticky" : false 48 | }, 49 | "192.168.123.45:110:tcp" : { 50 | "description" : "Basic named services example", 51 | "name" : "services01", 52 | "need" : 1, 53 | "reals" : { 54 | "10.1.2.100:110" : { 55 | "checks" : [ 56 | { 57 | "type" : "syn" 58 | } 59 | ], 60 | "disabled" : false, 61 | "weight" : 1 62 | }, 63 | "10.1.2.101:110" : { 64 | "checks" : [ 65 | { 66 | "type" : "syn" 67 | } 68 | ], 69 | "disabled" : false, 70 | "weight" : 1 71 | }, 72 | "10.1.2.102:110" : { 73 | "checks" : [ 74 | { 75 | "type" : "syn" 76 | } 77 | ], 78 | "disabled" : false, 79 | "weight" : 1 80 | }, 81 | "10.1.2.103:110" : { 82 | "checks" : [ 83 | { 84 | "type" : "syn" 85 | } 86 | ], 87 | "disabled" : false, 88 | "weight" : 1 89 | } 90 | }, 91 | "reset" : false, 92 | "sticky" : false 93 | }, 94 | "192.168.123.45:143:tcp" : { 95 | "description" : "Basic named services example", 96 | "name" : "services01", 97 | "need" : 1, 98 | "reals" : { 99 | "10.1.2.100:143" : { 100 | "checks" : [ 101 | { 102 | "type" : "syn" 103 | } 104 | ], 105 | "disabled" : false, 106 | "weight" : 1 107 | }, 108 | "10.1.2.101:143" : { 109 | "checks" : [ 110 | { 111 | "type" : "syn" 112 | } 113 | ], 114 | "disabled" : false, 115 | "weight" : 1 116 | }, 117 | "10.1.2.102:143" : { 118 | "checks" : [ 119 | { 120 | "type" : "syn" 121 | } 122 | ], 123 | "disabled" : false, 124 | "weight" : 1 125 | }, 126 | "10.1.2.103:143" : { 127 | "checks" : [ 128 | { 129 | "type" : "syn" 130 | } 131 | ], 132 | "disabled" : false, 133 | "weight" : 1 134 | } 135 | }, 136 | "reset" : false, 137 | "sticky" : false 138 | }, 139 | "192.168.123.45:22:tcp" : { 140 | "description" : "Basic named services example", 141 | "name" : "services01", 142 | "need" : 1, 143 | "reals" : { 144 | "10.1.2.100:22" : { 145 | "checks" : [ 146 | { 147 | "type" : "syn" 148 | } 149 | ], 150 | "disabled" : false, 151 | "weight" : 1 152 | }, 153 | "10.1.2.101:22" : { 154 | "checks" : [ 155 | { 156 | "type" : "syn" 157 | } 158 | ], 159 | "disabled" : false, 160 | "weight" : 1 161 | }, 162 | "10.1.2.102:22" : { 163 | "checks" : [ 164 | { 165 | "type" : "syn" 166 | } 167 | ], 168 | "disabled" : false, 169 | "weight" : 1 170 | }, 171 | "10.1.2.103:22" : { 172 | "checks" : [ 173 | { 174 | "type" : "syn" 175 | } 176 | ], 177 | "disabled" : false, 178 | "weight" : 1 179 | } 180 | }, 181 | "reset" : false, 182 | "sticky" : false 183 | }, 184 | "192.168.123.45:23:tcp" : { 185 | "description" : "Basic named services example", 186 | "name" : "services01", 187 | "need" : 1, 188 | "reals" : { 189 | "10.1.2.100:23" : { 190 | "checks" : [ 191 | { 192 | "type" : "syn" 193 | } 194 | ], 195 | "disabled" : false, 196 | "weight" : 1 197 | }, 198 | "10.1.2.101:23" : { 199 | "checks" : [ 200 | { 201 | "type" : "syn" 202 | } 203 | ], 204 | "disabled" : false, 205 | "weight" : 1 206 | }, 207 | "10.1.2.102:23" : { 208 | "checks" : [ 209 | { 210 | "type" : "syn" 211 | } 212 | ], 213 | "disabled" : false, 214 | "weight" : 1 215 | }, 216 | "10.1.2.103:23" : { 217 | "checks" : [ 218 | { 219 | "type" : "syn" 220 | } 221 | ], 222 | "disabled" : false, 223 | "weight" : 1 224 | } 225 | }, 226 | "reset" : false, 227 | "sticky" : false 228 | }, 229 | "192.168.123.45:25:tcp" : { 230 | "description" : "Basic named services example", 231 | "name" : "services01", 232 | "need" : 1, 233 | "reals" : { 234 | "10.1.2.100:25" : { 235 | "checks" : [ 236 | { 237 | "type" : "syn" 238 | } 239 | ], 240 | "disabled" : false, 241 | "weight" : 1 242 | }, 243 | "10.1.2.101:25" : { 244 | "checks" : [ 245 | { 246 | "type" : "syn" 247 | } 248 | ], 249 | "disabled" : false, 250 | "weight" : 1 251 | }, 252 | "10.1.2.102:25" : { 253 | "checks" : [ 254 | { 255 | "type" : "syn" 256 | } 257 | ], 258 | "disabled" : false, 259 | "weight" : 1 260 | }, 261 | "10.1.2.103:25" : { 262 | "checks" : [ 263 | { 264 | "type" : "syn" 265 | } 266 | ], 267 | "disabled" : false, 268 | "weight" : 1 269 | } 270 | }, 271 | "reset" : false, 272 | "sticky" : false 273 | }, 274 | "192.168.123.45:443:tcp" : { 275 | "description" : "Basic named services example", 276 | "name" : "services01", 277 | "need" : 1, 278 | "reals" : { 279 | "10.1.2.100:443" : { 280 | "checks" : [ 281 | { 282 | "type" : "https" 283 | } 284 | ], 285 | "disabled" : false, 286 | "weight" : 1 287 | }, 288 | "10.1.2.101:443" : { 289 | "checks" : [ 290 | { 291 | "type" : "https" 292 | } 293 | ], 294 | "disabled" : false, 295 | "weight" : 1 296 | }, 297 | "10.1.2.102:443" : { 298 | "checks" : [ 299 | { 300 | "type" : "https" 301 | } 302 | ], 303 | "disabled" : false, 304 | "weight" : 1 305 | }, 306 | "10.1.2.103:443" : { 307 | "checks" : [ 308 | { 309 | "type" : "https" 310 | } 311 | ], 312 | "disabled" : false, 313 | "weight" : 1 314 | } 315 | }, 316 | "reset" : false, 317 | "sticky" : false 318 | }, 319 | "192.168.123.45:53:tcp" : { 320 | "description" : "Basic named services example", 321 | "name" : "services01", 322 | "need" : 1, 323 | "reals" : { 324 | "10.1.2.100:53" : { 325 | "checks" : [ 326 | { 327 | "method" : "tcp", 328 | "type" : "dns" 329 | } 330 | ], 331 | "disabled" : false, 332 | "weight" : 1 333 | }, 334 | "10.1.2.101:53" : { 335 | "checks" : [ 336 | { 337 | "method" : "tcp", 338 | "type" : "dns" 339 | } 340 | ], 341 | "disabled" : false, 342 | "weight" : 1 343 | }, 344 | "10.1.2.102:53" : { 345 | "checks" : [ 346 | { 347 | "method" : "tcp", 348 | "type" : "dns" 349 | } 350 | ], 351 | "disabled" : false, 352 | "weight" : 1 353 | }, 354 | "10.1.2.103:53" : { 355 | "checks" : [ 356 | { 357 | "method" : "tcp", 358 | "type" : "dns" 359 | } 360 | ], 361 | "disabled" : false, 362 | "weight" : 1 363 | } 364 | }, 365 | "reset" : false, 366 | "sticky" : false 367 | }, 368 | "192.168.123.45:53:udp" : { 369 | "description" : "Basic named services example", 370 | "name" : "services01", 371 | "need" : 1, 372 | "reals" : { 373 | "10.1.2.100:53" : { 374 | "checks" : [ 375 | { 376 | "method" : "udp", 377 | "type" : "dns" 378 | } 379 | ], 380 | "disabled" : false, 381 | "weight" : 1 382 | }, 383 | "10.1.2.101:53" : { 384 | "checks" : [ 385 | { 386 | "method" : "udp", 387 | "type" : "dns" 388 | } 389 | ], 390 | "disabled" : false, 391 | "weight" : 1 392 | }, 393 | "10.1.2.102:53" : { 394 | "checks" : [ 395 | { 396 | "method" : "udp", 397 | "type" : "dns" 398 | } 399 | ], 400 | "disabled" : false, 401 | "weight" : 1 402 | }, 403 | "10.1.2.103:53" : { 404 | "checks" : [ 405 | { 406 | "method" : "udp", 407 | "type" : "dns" 408 | } 409 | ], 410 | "disabled" : false, 411 | "weight" : 1 412 | } 413 | }, 414 | "reset" : false, 415 | "sticky" : false 416 | }, 417 | "192.168.123.45:80:tcp" : { 418 | "description" : "Basic named services example", 419 | "name" : "services01", 420 | "need" : 1, 421 | "reals" : { 422 | "10.1.2.100:80" : { 423 | "checks" : [ 424 | { 425 | "type" : "http" 426 | } 427 | ], 428 | "disabled" : false, 429 | "weight" : 1 430 | }, 431 | "10.1.2.101:80" : { 432 | "checks" : [ 433 | { 434 | "type" : "http" 435 | } 436 | ], 437 | "disabled" : false, 438 | "weight" : 1 439 | }, 440 | "10.1.2.102:80" : { 441 | "checks" : [ 442 | { 443 | "type" : "http" 444 | } 445 | ], 446 | "disabled" : false, 447 | "weight" : 1 448 | }, 449 | "10.1.2.103:80" : { 450 | "checks" : [ 451 | { 452 | "type" : "http" 453 | } 454 | ], 455 | "disabled" : false, 456 | "weight" : 1 457 | } 458 | }, 459 | "reset" : false, 460 | "sticky" : false 461 | }, 462 | "192.168.123.45:993:tcp" : { 463 | "description" : "Basic named services example", 464 | "name" : "services01", 465 | "need" : 1, 466 | "reals" : { 467 | "10.1.2.100:993" : { 468 | "checks" : [ 469 | { 470 | "type" : "syn" 471 | } 472 | ], 473 | "disabled" : false, 474 | "weight" : 1 475 | }, 476 | "10.1.2.101:993" : { 477 | "checks" : [ 478 | { 479 | "type" : "syn" 480 | } 481 | ], 482 | "disabled" : false, 483 | "weight" : 1 484 | }, 485 | "10.1.2.102:993" : { 486 | "checks" : [ 487 | { 488 | "type" : "syn" 489 | } 490 | ], 491 | "disabled" : false, 492 | "weight" : 1 493 | }, 494 | "10.1.2.103:993" : { 495 | "checks" : [ 496 | { 497 | "type" : "syn" 498 | } 499 | ], 500 | "disabled" : false, 501 | "weight" : 1 502 | } 503 | }, 504 | "reset" : false, 505 | "sticky" : false 506 | } 507 | } 508 | } 509 | -------------------------------------------------------------------------------- /cmd/tests/services01.yaml: -------------------------------------------------------------------------------- 1 | 2 | services: 3 | 4 | - name: services01 5 | description: Basic named services example 6 | virtual: 192.168.123.45 7 | servers: 8 | - 10.1.2.100 9 | - 10.1.2.101 10 | - 10.1.2.102 11 | - 10.1.2.103 12 | policy: 13 | http: 14 | https: 15 | domain: 16 | smtp: 17 | ssh: 18 | telnet: 19 | pop2: 20 | pop3: 21 | imap: 22 | imaps: 23 | -------------------------------------------------------------------------------- /cmd/tests/services02.json: -------------------------------------------------------------------------------- 1 | { 2 | "bgp" : {}, 3 | "services" : { 4 | "192.168.123.1:443:tcp" : { 5 | "description" : "Four webservers", 6 | "name" : "services02-1", 7 | "need" : 1, 8 | "reals" : { 9 | "10.1.1.100:443" : { 10 | "checks" : [ 11 | { 12 | "type" : "https" 13 | } 14 | ], 15 | "disabled" : false, 16 | "weight" : 1 17 | }, 18 | "10.1.1.101:443" : { 19 | "checks" : [ 20 | { 21 | "type" : "https" 22 | } 23 | ], 24 | "disabled" : false, 25 | "weight" : 1 26 | }, 27 | "10.1.1.102:443" : { 28 | "checks" : [ 29 | { 30 | "type" : "https" 31 | } 32 | ], 33 | "disabled" : false, 34 | "weight" : 1 35 | }, 36 | "10.1.1.103:443" : { 37 | "checks" : [ 38 | { 39 | "type" : "https" 40 | } 41 | ], 42 | "disabled" : false, 43 | "weight" : 1 44 | } 45 | }, 46 | "reset" : false, 47 | "sticky" : false 48 | }, 49 | "192.168.123.1:80:tcp" : { 50 | "description" : "Four webservers", 51 | "name" : "services02-1", 52 | "need" : 1, 53 | "reals" : { 54 | "10.1.1.100:80" : { 55 | "checks" : [ 56 | { 57 | "type" : "http" 58 | } 59 | ], 60 | "disabled" : false, 61 | "weight" : 1 62 | }, 63 | "10.1.1.101:80" : { 64 | "checks" : [ 65 | { 66 | "type" : "http" 67 | } 68 | ], 69 | "disabled" : false, 70 | "weight" : 1 71 | }, 72 | "10.1.1.102:80" : { 73 | "checks" : [ 74 | { 75 | "type" : "http" 76 | } 77 | ], 78 | "disabled" : false, 79 | "weight" : 1 80 | }, 81 | "10.1.1.103:80" : { 82 | "checks" : [ 83 | { 84 | "type" : "http" 85 | } 86 | ], 87 | "disabled" : false, 88 | "weight" : 1 89 | } 90 | }, 91 | "reset" : false, 92 | "sticky" : false 93 | }, 94 | "192.168.123.2:443:tcp" : { 95 | "description" : "Four other webservers with a different VIP", 96 | "name" : "services02-2", 97 | "need" : 1, 98 | "reals" : { 99 | "10.1.2.100:443" : { 100 | "checks" : [ 101 | { 102 | "type" : "https" 103 | } 104 | ], 105 | "disabled" : false, 106 | "weight" : 1 107 | }, 108 | "10.1.2.101:443" : { 109 | "checks" : [ 110 | { 111 | "type" : "https" 112 | } 113 | ], 114 | "disabled" : false, 115 | "weight" : 1 116 | }, 117 | "10.1.2.102:443" : { 118 | "checks" : [ 119 | { 120 | "type" : "https" 121 | } 122 | ], 123 | "disabled" : false, 124 | "weight" : 1 125 | }, 126 | "10.1.2.103:443" : { 127 | "checks" : [ 128 | { 129 | "type" : "https" 130 | } 131 | ], 132 | "disabled" : false, 133 | "weight" : 1 134 | } 135 | }, 136 | "reset" : false, 137 | "sticky" : false 138 | }, 139 | "192.168.123.2:80:tcp" : { 140 | "description" : "Four other webservers with a different VIP", 141 | "name" : "services02-2", 142 | "need" : 1, 143 | "reals" : { 144 | "10.1.2.100:80" : { 145 | "checks" : [ 146 | { 147 | "type" : "http" 148 | } 149 | ], 150 | "disabled" : false, 151 | "weight" : 1 152 | }, 153 | "10.1.2.101:80" : { 154 | "checks" : [ 155 | { 156 | "type" : "http" 157 | } 158 | ], 159 | "disabled" : false, 160 | "weight" : 1 161 | }, 162 | "10.1.2.102:80" : { 163 | "checks" : [ 164 | { 165 | "type" : "http" 166 | } 167 | ], 168 | "disabled" : false, 169 | "weight" : 1 170 | }, 171 | "10.1.2.103:80" : { 172 | "checks" : [ 173 | { 174 | "type" : "http" 175 | } 176 | ], 177 | "disabled" : false, 178 | "weight" : 1 179 | } 180 | }, 181 | "reset" : false, 182 | "sticky" : false 183 | }, 184 | "192.168.123.3:443:tcp" : { 185 | "description" : "Two servers disabled", 186 | "name" : "services02-3", 187 | "need" : 1, 188 | "reals" : { 189 | "10.1.3.100:443" : { 190 | "checks" : [ 191 | { 192 | "type" : "https" 193 | } 194 | ], 195 | "disabled" : true, 196 | "weight" : 0 197 | }, 198 | "10.1.3.101:443" : { 199 | "checks" : [ 200 | { 201 | "type" : "https" 202 | } 203 | ], 204 | "disabled" : true, 205 | "weight" : 0 206 | }, 207 | "10.1.3.102:443" : { 208 | "checks" : [ 209 | { 210 | "type" : "https" 211 | } 212 | ], 213 | "disabled" : false, 214 | "weight" : 1 215 | }, 216 | "10.1.3.103:443" : { 217 | "checks" : [ 218 | { 219 | "type" : "https" 220 | } 221 | ], 222 | "disabled" : false, 223 | "weight" : 1 224 | } 225 | }, 226 | "reset" : false, 227 | "sticky" : false 228 | }, 229 | "192.168.123.3:80:tcp" : { 230 | "description" : "Two servers disabled", 231 | "name" : "services02-3", 232 | "need" : 1, 233 | "reals" : { 234 | "10.1.3.100:80" : { 235 | "checks" : [ 236 | { 237 | "type" : "http" 238 | } 239 | ], 240 | "disabled" : true, 241 | "weight" : 0 242 | }, 243 | "10.1.3.101:80" : { 244 | "checks" : [ 245 | { 246 | "type" : "http" 247 | } 248 | ], 249 | "disabled" : true, 250 | "weight" : 0 251 | }, 252 | "10.1.3.102:80" : { 253 | "checks" : [ 254 | { 255 | "type" : "http" 256 | } 257 | ], 258 | "disabled" : false, 259 | "weight" : 1 260 | }, 261 | "10.1.3.103:80" : { 262 | "checks" : [ 263 | { 264 | "type" : "http" 265 | } 266 | ], 267 | "disabled" : false, 268 | "weight" : 1 269 | } 270 | }, 271 | "reset" : false, 272 | "sticky" : false 273 | }, 274 | "HTTP/HTTPS checks for services not running on default ports:8080:tcp" : { 275 | "description" : "Two servers disabled", 276 | "name" : "services02-4", 277 | "need" : 1, 278 | "reals" : { 279 | "10.1.3.100:8080" : { 280 | "checks" : [ 281 | { 282 | "type" : "http" 283 | } 284 | ], 285 | "disabled" : false, 286 | "weight" : 1 287 | }, 288 | "10.1.3.101:8080" : { 289 | "checks" : [ 290 | { 291 | "type" : "http" 292 | } 293 | ], 294 | "disabled" : false, 295 | "weight" : 1 296 | }, 297 | "10.1.3.102:8080" : { 298 | "checks" : [ 299 | { 300 | "type" : "http" 301 | } 302 | ], 303 | "disabled" : false, 304 | "weight" : 1 305 | }, 306 | "10.1.3.103:8080" : { 307 | "checks" : [ 308 | { 309 | "type" : "http" 310 | } 311 | ], 312 | "disabled" : false, 313 | "weight" : 1 314 | } 315 | }, 316 | "reset" : false, 317 | "sticky" : false 318 | }, 319 | "HTTP/HTTPS checks for services not running on default ports:8443:tcp" : { 320 | "description" : "Two servers disabled", 321 | "name" : "services02-4", 322 | "need" : 1, 323 | "reals" : { 324 | "10.1.3.100:8443" : { 325 | "checks" : [ 326 | { 327 | "type" : "https" 328 | } 329 | ], 330 | "disabled" : false, 331 | "weight" : 1 332 | }, 333 | "10.1.3.101:8443" : { 334 | "checks" : [ 335 | { 336 | "type" : "https" 337 | } 338 | ], 339 | "disabled" : false, 340 | "weight" : 1 341 | }, 342 | "10.1.3.102:8443" : { 343 | "checks" : [ 344 | { 345 | "type" : "https" 346 | } 347 | ], 348 | "disabled" : false, 349 | "weight" : 1 350 | }, 351 | "10.1.3.103:8443" : { 352 | "checks" : [ 353 | { 354 | "type" : "https" 355 | } 356 | ], 357 | "disabled" : false, 358 | "weight" : 1 359 | } 360 | }, 361 | "reset" : false, 362 | "sticky" : false 363 | } 364 | } 365 | } 366 | -------------------------------------------------------------------------------- /cmd/tests/services02.yaml: -------------------------------------------------------------------------------- 1 | 2 | services: 3 | 4 | - name: services02-1 5 | description: Four webservers 6 | virtual: 192.168.123.1 7 | servers: 8 | - 10.1.1.100 9 | - 10.1.1.101 10 | - 10.1.1.102 11 | - 10.1.1.103 12 | policy: 13 | http: 14 | https: 15 | 16 | - name: services02-2 17 | description: Four other webservers with a different VIP 18 | virtual: 192.168.123.2 19 | servers: 20 | - 10.1.2.100 21 | - 10.1.2.101 22 | - 10.1.2.102 23 | - 10.1.2.103 24 | policy: 25 | http: 26 | https: 27 | 28 | - name: services02-3 29 | description: Two servers disabled 30 | virtual: 192.168.123.3 31 | servers: 32 | - 10.1.3.100* 33 | - 10.1.3.101* 34 | - 10.1.3.102 35 | - 10.1.3.103 36 | policy: 37 | http: 38 | https: 39 | 40 | - name: services02-4 41 | description: Two servers disabled 42 | virtual: HTTP/HTTPS checks for services not running on default ports 43 | servers: 44 | - 10.1.3.100 45 | - 10.1.3.101 46 | - 10.1.3.102 47 | - 10.1.3.103 48 | policy: 49 | 8080/http: 50 | 8443/https: 51 | 52 | -------------------------------------------------------------------------------- /cmd/tests/services03.json: -------------------------------------------------------------------------------- 1 | { 2 | "bgp" : {}, 3 | "services" : { 4 | "192.168.123.45:443:tcp" : { 5 | "description" : "Different health checks/settings for HTTPS", 6 | "name" : "services03-1", 7 | "need" : 2, 8 | "reals" : { 9 | "10.1.2.100:443" : { 10 | "checks" : [ 11 | { 12 | "expect" : [ 13 | 302 14 | ], 15 | "host" : "ssl.example.com", 16 | "path" : "/redirect", 17 | "type" : "https" 18 | } 19 | ], 20 | "disabled" : false, 21 | "weight" : 1 22 | }, 23 | "10.1.2.101:443" : { 24 | "checks" : [ 25 | { 26 | "expect" : [ 27 | 302 28 | ], 29 | "host" : "ssl.example.com", 30 | "path" : "/redirect", 31 | "type" : "https" 32 | } 33 | ], 34 | "disabled" : false, 35 | "weight" : 1 36 | } 37 | }, 38 | "reset" : false, 39 | "sticky" : false 40 | }, 41 | "192.168.123.45:80:tcp" : { 42 | "description" : "Different health checks/settings for HTTPS", 43 | "name" : "services03-1", 44 | "need" : 1, 45 | "reals" : { 46 | "10.1.2.100:80" : { 47 | "checks" : [ 48 | { 49 | "host" : "foo.example.com", 50 | "path" : "/health", 51 | "type" : "http" 52 | } 53 | ], 54 | "disabled" : false, 55 | "weight" : 1 56 | }, 57 | "10.1.2.101:80" : { 58 | "checks" : [ 59 | { 60 | "host" : "foo.example.com", 61 | "path" : "/health", 62 | "type" : "http" 63 | } 64 | ], 65 | "disabled" : false, 66 | "weight" : 1 67 | } 68 | }, 69 | "reset" : false, 70 | "sticky" : true 71 | }, 72 | "192.168.123.67:443:tcp" : { 73 | "description" : "Custom health check for HTTPS - check port is open (SYN) but also run http check against port 80", 74 | "name" : "services03-2", 75 | "need" : 1, 76 | "reals" : { 77 | "10.1.3.100:443" : { 78 | "checks" : [ 79 | { 80 | "type" : "syn" 81 | }, 82 | { 83 | "host" : "bar.example.com", 84 | "path" : "/health", 85 | "port" : 80, 86 | "type" : "http" 87 | } 88 | ], 89 | "disabled" : false, 90 | "weight" : 1 91 | }, 92 | "10.1.3.101:443" : { 93 | "checks" : [ 94 | { 95 | "type" : "syn" 96 | }, 97 | { 98 | "host" : "bar.example.com", 99 | "path" : "/health", 100 | "port" : 80, 101 | "type" : "http" 102 | } 103 | ], 104 | "disabled" : false, 105 | "weight" : 1 106 | } 107 | }, 108 | "reset" : false, 109 | "sticky" : true 110 | }, 111 | "192.168.123.67:80:tcp" : { 112 | "description" : "Custom health check for HTTPS - check port is open (SYN) but also run http check against port 80", 113 | "name" : "services03-2", 114 | "need" : 1, 115 | "reals" : { 116 | "10.1.3.100:80" : { 117 | "checks" : [ 118 | { 119 | "host" : "bar.example.com", 120 | "path" : "/health", 121 | "type" : "http" 122 | } 123 | ], 124 | "disabled" : false, 125 | "weight" : 1 126 | }, 127 | "10.1.3.101:80" : { 128 | "checks" : [ 129 | { 130 | "host" : "bar.example.com", 131 | "path" : "/health", 132 | "type" : "http" 133 | } 134 | ], 135 | "disabled" : false, 136 | "weight" : 1 137 | } 138 | }, 139 | "reset" : false, 140 | "sticky" : true 141 | } 142 | } 143 | } 144 | -------------------------------------------------------------------------------- /cmd/tests/services03.yaml: -------------------------------------------------------------------------------- 1 | 2 | services: 3 | 4 | - name: services03-1 5 | description: Different health checks/settings for HTTPS 6 | virtual: 192.168.123.45 7 | servers: 8 | - 10.1.2.100 9 | - 10.1.2.101 10 | host: foo.example.com 11 | path: /health 12 | sticky: true 13 | policy: 14 | http: 15 | https: 16 | path: /redirect 17 | host: ssl.example.com 18 | expect: 302 19 | sticky: false 20 | need: 2 21 | 22 | - name: services03-2 23 | description: Custom health check for HTTPS - check port is open (SYN) but also run http check against port 80 24 | virtual: 192.168.123.67 25 | servers: 26 | - 10.1.3.100 27 | - 10.1.3.101 28 | host: bar.example.com 29 | path: /health 30 | sticky: true 31 | policy: 32 | http: 33 | https: 34 | checks: 35 | - type: syn 36 | - type: http 37 | port: 80 38 | 39 | ## FIXME: this doesn't work as I would expect (run check against a different port to that of the service) - maybe a bug 40 | # policy: 41 | # http: 42 | # port: 8080 43 | -------------------------------------------------------------------------------- /cmd/tests/services04.json: -------------------------------------------------------------------------------- 1 | { 2 | "bgp" : {}, 3 | "services" : { 4 | "192.168.123.1:80:tcp" : { 5 | "description" : "Map service port to different port on real servers (NAT)", 6 | "name" : "services04-1", 7 | "need" : 1, 8 | "reals" : { 9 | "10.1.1.100:8080" : { 10 | "checks" : [ 11 | { 12 | "type" : "http" 13 | } 14 | ], 15 | "disabled" : false, 16 | "weight" : 1 17 | }, 18 | "10.1.1.101:8080" : { 19 | "checks" : [ 20 | { 21 | "type" : "http" 22 | } 23 | ], 24 | "disabled" : false, 25 | "weight" : 1 26 | }, 27 | "10.1.1.102:8080" : { 28 | "checks" : [ 29 | { 30 | "type" : "http" 31 | } 32 | ], 33 | "disabled" : false, 34 | "weight" : 1 35 | }, 36 | "10.1.1.103:8080" : { 37 | "checks" : [ 38 | { 39 | "type" : "http" 40 | } 41 | ], 42 | "disabled" : false, 43 | "weight" : 1 44 | } 45 | }, 46 | "reset" : false, 47 | "scheduler" : "roundrobin", 48 | "sticky" : false 49 | }, 50 | "192.168.123.2:80:tcp" : { 51 | "description" : "Disable all health checks for service", 52 | "name" : "services04-2", 53 | "need" : 1, 54 | "reals" : { 55 | "10.1.2.100:80" : { 56 | "checks" : [], 57 | "disabled" : false, 58 | "weight" : 1 59 | }, 60 | "10.1.2.101:80" : { 61 | "checks" : [], 62 | "disabled" : false, 63 | "weight" : 1 64 | }, 65 | "10.1.2.102:80" : { 66 | "checks" : [], 67 | "disabled" : false, 68 | "weight" : 1 69 | }, 70 | "10.1.2.103:80" : { 71 | "checks" : [], 72 | "disabled" : false, 73 | "weight" : 1 74 | } 75 | }, 76 | "reset" : false, 77 | "scheduler" : "roundrobin", 78 | "sticky" : false 79 | }, 80 | "192.168.123.3:80:tcp" : { 81 | "description" : "Specify a different scheduler with sticky and reset - to prevent IPVS sending sticky sessions to dead servers", 82 | "name" : "services04-4", 83 | "need" : 1, 84 | "reals" : { 85 | "10.1.4.100:80" : { 86 | "checks" : [ 87 | { 88 | "type" : "http" 89 | } 90 | ], 91 | "disabled" : false, 92 | "weight" : 1 93 | }, 94 | "10.1.4.101:80" : { 95 | "checks" : [ 96 | { 97 | "type" : "http" 98 | } 99 | ], 100 | "disabled" : false, 101 | "weight" : 1 102 | }, 103 | "10.1.4.102:80" : { 104 | "checks" : [ 105 | { 106 | "type" : "http" 107 | } 108 | ], 109 | "disabled" : false, 110 | "weight" : 1 111 | }, 112 | "10.1.4.103:80" : { 113 | "checks" : [ 114 | { 115 | "type" : "http" 116 | } 117 | ], 118 | "disabled" : false, 119 | "weight" : 1 120 | } 121 | }, 122 | "reset" : true, 123 | "scheduler" : "maglev", 124 | "sticky" : true 125 | } 126 | } 127 | } 128 | -------------------------------------------------------------------------------- /cmd/tests/services04.yaml: -------------------------------------------------------------------------------- 1 | scheduler: roundrobin 2 | 3 | services: 4 | 5 | - name: services04-1 6 | description: Map service port to different port on real servers (NAT) 7 | virtual: 192.168.123.1 8 | servers: 9 | - 10.1.1.100 10 | - 10.1.1.101 11 | - 10.1.1.102 12 | - 10.1.1.103 13 | policy: 14 | http: 8080 15 | 16 | - name: services04-2 17 | description: Disable all health checks for service 18 | virtual: 192.168.123.2 19 | servers: 20 | - 10.1.2.100 21 | - 10.1.2.101 22 | - 10.1.2.102 23 | - 10.1.2.103 24 | policy: 25 | http*: 26 | 27 | - name: services04-3 28 | description: Specify a different scheduler 29 | virtual: 192.168.123.3 30 | servers: 31 | - 10.1.3.100 32 | - 10.1.3.101 33 | - 10.1.3.102 34 | - 10.1.3.103 35 | scheduler: leastconn 36 | policy: 37 | http: 38 | 39 | - name: services04-4 40 | description: Specify a different scheduler with sticky and reset - to prevent IPVS sending sticky sessions to dead servers 41 | virtual: 192.168.123.3 42 | servers: 43 | - 10.1.4.100 44 | - 10.1.4.101 45 | - 10.1.4.102 46 | - 10.1.4.103 47 | scheduler: maglev 48 | sticky: true 49 | reset: true 50 | policy: 51 | http: 52 | 53 | -------------------------------------------------------------------------------- /cmd/xvs.go: -------------------------------------------------------------------------------- 1 | /* 2 | * VC5 load balancer. Copyright (C) 2021-present David Coles 3 | * 4 | * This program is free software; you can redistribute it and/or modify 5 | * it under the terms of the GNU General Public License as published by 6 | * the Free Software Foundation; either version 2 of the License, or 7 | * (at your option) any later version. 8 | * 9 | * This program is distributed in the hope that it will be useful, 10 | * but WITHOUT ANY WARRANTY; without even the implied warranty of 11 | * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 12 | * GNU General Public License for more details. 13 | * 14 | * You should have received a copy of the GNU General Public License along 15 | * with this program; if not, write to the Free Software Foundation, Inc., 16 | * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 17 | */ 18 | 19 | package main 20 | 21 | import ( 22 | "fmt" 23 | "log" 24 | "net" 25 | "net/netip" 26 | "time" 27 | 28 | "github.com/davidcoles/cue/mon" 29 | "vc5" 30 | ) 31 | 32 | // xvs specific routines 33 | 34 | func mac(m [6]byte) string { 35 | return fmt.Sprintf("%02x:%02x:%02x:%02x:%02x:%02x", m[0], m[1], m[2], m[3], m[4], m[5]) 36 | } 37 | 38 | const maxDatagramSize = 1500 39 | 40 | func multicast_send(c Client, address string) { 41 | 42 | addr, err := net.ResolveUDPAddr("udp", address) 43 | 44 | if err != nil { 45 | log.Fatal(err) 46 | } 47 | 48 | conn, err := net.DialUDP("udp", nil, addr) 49 | 50 | if err != nil { 51 | log.Fatal(err) 52 | } 53 | 54 | conn.SetWriteBuffer(maxDatagramSize * 100) 55 | 56 | ticker := time.NewTicker(time.Millisecond * 10) 57 | 58 | var buff [maxDatagramSize]byte 59 | 60 | for { 61 | select { 62 | case <-ticker.C: 63 | n := 0 64 | 65 | read_flow: 66 | f := c.ReadFlow() 67 | if len(f) > 0 { 68 | buff[n] = uint8(len(f)) 69 | 70 | copy(buff[n+1:], f[:]) 71 | n += 1 + len(f) 72 | if n < maxDatagramSize-100 { 73 | goto read_flow 74 | } 75 | } 76 | 77 | if n > 0 { 78 | conn.Write(buff[:n]) 79 | } 80 | } 81 | } 82 | } 83 | 84 | func multicast_recv(c Client, address string) { 85 | udp, err := net.ResolveUDPAddr("udp", address) 86 | 87 | if err != nil { 88 | log.Fatal(err) 89 | } 90 | 91 | conn, err := net.ListenMulticastUDP("udp", nil, udp) 92 | 93 | conn.SetReadBuffer(maxDatagramSize * 1000) 94 | 95 | buff := make([]byte, maxDatagramSize) 96 | 97 | for { 98 | nread, _, err := conn.ReadFromUDP(buff) 99 | if err == nil { 100 | for n := 0; n+1 < nread; { 101 | l := int(buff[n]) 102 | o := n + 1 103 | n = o + l 104 | if l > 0 && n <= nread { 105 | c.WriteFlow(buff[o:n]) 106 | } 107 | } 108 | } 109 | } 110 | } 111 | 112 | // return a function which will translate a vip/rip pair to a nat address - used by the manager to log destination.nat.ip 113 | func nat(client Client) func(vip, rip netip.Addr) (netip.Addr, bool) { 114 | return func(vip, rip netip.Addr) (netip.Addr, bool) { return client.NAT(vip, rip), true } 115 | } 116 | 117 | // return a function which will relay probe requests to the network namespace healtchcheck proxy (which run against the nat address) 118 | func prober(client Client, monitor *mon.Mon) func(netip.Addr, netip.Addr, vc5.Check) (ok bool, diagnostic string) { 119 | 120 | return func(vip, addr netip.Addr, check vc5.Check) (ok bool, diagnostic string) { 121 | if check.Host == "" { 122 | if vip.Is6() { 123 | check.Host = "[" + vip.String() + "]" 124 | } else { 125 | check.Host = vip.String() 126 | } 127 | 128 | } 129 | return monitor.Probe(addr, check) 130 | } 131 | } 132 | -------------------------------------------------------------------------------- /config.go: -------------------------------------------------------------------------------- 1 | /* 2 | * VC5 load balancer. Copyright (C) 2021-present David Coles 3 | * 4 | * This program is free software; you can redistribute it and/or modify 5 | * it under the terms of the GNU General Public License as published by 6 | * the Free Software Foundation; either version 2 of the License, or 7 | * (at your option) any later version. 8 | * 9 | * This program is distributed in the hope that it will be useful, 10 | * but WITHOUT ANY WARRANTY; without even the implied warranty of 11 | * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 12 | * GNU General Public License for more details. 13 | * 14 | * You should have received a copy of the GNU General Public License along 15 | * with this program; if not, write to the Free Software Foundation, Inc., 16 | * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 17 | */ 18 | 19 | package vc5 20 | 21 | import ( 22 | "embed" 23 | "encoding/json" 24 | "errors" 25 | "fmt" 26 | "io/ioutil" 27 | "net" 28 | "net/netip" 29 | "os" 30 | "regexp" 31 | "strconv" 32 | //"time" 33 | 34 | "github.com/davidcoles/bgp" 35 | "github.com/davidcoles/cue" 36 | "github.com/davidcoles/cue/mon" 37 | ) 38 | 39 | //go:embed static/* 40 | var STATIC embed.FS 41 | 42 | type Priority = priority 43 | type Protocol = protocol 44 | 45 | const ( 46 | TCP = 0x06 47 | UDP = 0x11 48 | ) 49 | 50 | type Real struct { 51 | Checks []mon.Check `json:"checks,omitempty"` 52 | Disabled bool `json:"disabled,omitempty"` 53 | Weight uint8 `json:"weight,omitempty"` 54 | } 55 | 56 | // Describes a Layer 4 service 57 | type ServiceDefinition struct { 58 | // The service name - should be a short identifier, suitable for using as a Prometheus label value 59 | Name string `json:"name,omitempty"` 60 | 61 | // A short description of the service 62 | Description string `json:"description,omitempty"` 63 | 64 | // critical, high, medium or low (default is critical) 65 | Priority priority `json:"priority"` 66 | 67 | // The minimum number of real servers which need to be healthy to consider this service viable 68 | Need uint8 `json:"need,omitempty"` 69 | 70 | // Backend servers and corresponding health checks 71 | Destinations map[Destination]Real `json:"reals,omitempty"` 72 | 73 | // If set to true, the backend selection algorithm will not include layer 4 port numbers 74 | Sticky bool `json:"sticky,omitempty"` 75 | 76 | Scheduler string `json:"scheduler"` 77 | Persist uint32 `json:"persist"` // used in IPVS version 78 | Reset bool `json:"reset,omitempty"` // used in IPVS version 79 | 80 | TunnelType string `json:"tunnel-type,omitempty"` 81 | TunnelPort uint16 `json:"tunnel-port,omitempty"` 82 | TunnelEncapNoChecksum bool `json:"tunnel-no-checksum,omitempty"` 83 | } 84 | 85 | type services map[Service]ServiceDefinition 86 | 87 | // Load balancer configuration 88 | type Config struct { 89 | Services services `json:"services,omitempty"` 90 | VLANs map[uint16]netip.Prefix `json:"vlans,omitempty"` // VLAN ID to subnet mappings 91 | VLANs6 map[uint16]netip.Prefix `json:"vlans6,omitempty"` // VLAN ID to subnet mappings 92 | BGP map[string]bgp.Parameters `json:"bgp,omitempty"` // BGP peers 93 | Logging Logging_ `json:"logging,omitempty"` 94 | } 95 | 96 | func (c *Config) Prefixes() map[uint16]netip.Prefix { 97 | return c.VLANs 98 | } 99 | 100 | func (c *Config) Prefixes6() map[uint16]netip.Prefix { 101 | return c.VLANs6 102 | } 103 | 104 | func (c *Config) LoggingConfig() Logging { 105 | return c.Logging.Logging() 106 | } 107 | 108 | func (c *Config) Bgp(asn uint16, mp bool) map[string]bgp.Parameters { 109 | if asn > 0 { 110 | return map[string]bgp.Parameters{"127.0.0.1": bgp.Parameters{ASNumber: asn, HoldTime: 4, Multiprotocol: mp}} 111 | } 112 | 113 | return c.BGP 114 | } 115 | 116 | func (c *Config) Priorities() map[netip.Addr]priority { 117 | 118 | priorities := map[netip.Addr]priority{} 119 | 120 | for k, v := range c.Services { 121 | p, ok := priorities[k.Address] 122 | 123 | if !ok { 124 | p = LOW 125 | priorities[k.Address] = p 126 | } 127 | 128 | if v.Priority < p { 129 | priorities[k.Address] = v.Priority 130 | } 131 | } 132 | 133 | return priorities 134 | } 135 | 136 | // Reads the load-balancer configuration from a JSON file. Returns a 137 | // pointer to the Config object on success, and sets the error to 138 | // non-nil on failure. 139 | func Load(file string) (*Config, error) { 140 | 141 | f, err := os.Open(file) 142 | 143 | if err != nil { 144 | return nil, err 145 | } 146 | 147 | defer f.Close() 148 | 149 | b, err := ioutil.ReadAll(f) 150 | 151 | if err != nil { 152 | return nil, err 153 | } 154 | 155 | var config Config 156 | 157 | err = json.Unmarshal(b, &(config)) 158 | 159 | if err != nil { 160 | return nil, err 161 | } 162 | 163 | return &config, nil 164 | } 165 | 166 | type Prefix net.IPNet 167 | 168 | func (p *Prefix) String() string { 169 | return (*net.IPNet)(p).String() 170 | } 171 | 172 | func (p *Prefix) Contains(i netip.Addr) bool { 173 | var ip net.IP 174 | if i.Is4() { 175 | t := i.As4() 176 | ip = t[:] 177 | } else if i.Is6() { 178 | t := i.As16() 179 | ip = t[:] 180 | } 181 | 182 | return (*net.IPNet)(p).Contains(ip) 183 | } 184 | 185 | func (p *Prefix) UnmarshalJSON(data []byte) error { 186 | 187 | l := len(data) 188 | 189 | if l < 3 || data[0] != '"' || data[l-1] != '"' { 190 | return errors.New("CIDR address should be a string: " + string(data)) 191 | } 192 | 193 | cidr := string(data[1 : l-1]) 194 | 195 | //ip, ipnet, err := net.ParseCIDR(cidr) 196 | _, ipnet, err := net.ParseCIDR(cidr) 197 | 198 | if err != nil { 199 | return err 200 | } 201 | 202 | //if ip.String() != ipnet.IP.String() { 203 | //return errors.New("CIDR address contains host portion: " + cidr) 204 | //} 205 | 206 | *p = Prefix(*ipnet) 207 | 208 | return nil 209 | } 210 | 211 | func (c *Config) Parse() []cue.Service { 212 | 213 | var services []cue.Service 214 | 215 | for ipp, svc := range c.Services { 216 | 217 | service := cue.Service{ 218 | Address: ipp.Address, 219 | Port: ipp.Port, 220 | Protocol: uint8(ipp.Protocol), 221 | Required: svc.Need, 222 | Scheduler: svc.Scheduler, 223 | Persist: svc.Persist, 224 | Sticky: svc.Sticky, 225 | Reset: svc.Reset, 226 | } 227 | 228 | for ap, dst := range svc.Destinations { 229 | 230 | destination := cue.Destination{ 231 | Address: ap.Address, 232 | Port: ap.Port, 233 | Weight: dst.Weight, 234 | Disabled: dst.Disabled, 235 | Checks: append([]mon.Check{}, dst.Checks...), 236 | } 237 | 238 | service.Destinations = append(service.Destinations, destination) 239 | 240 | } 241 | 242 | services = append(services, service) 243 | } 244 | 245 | return services 246 | } 247 | 248 | type protocol uint8 249 | 250 | func (p protocol) MarshalText() ([]byte, error) { 251 | switch p { 252 | case TCP: 253 | return []byte("tcp"), nil 254 | case UDP: 255 | return []byte("udp"), nil 256 | } 257 | return nil, errors.New("Invalid protocol") 258 | } 259 | 260 | func (p protocol) string() string { 261 | switch p { 262 | case TCP: 263 | return "tcp" 264 | case UDP: 265 | return "udp" 266 | } 267 | return fmt.Sprintf("%d", p) 268 | } 269 | 270 | func (p protocol) String() string { 271 | switch p { 272 | case TCP: 273 | return "tcp" 274 | case UDP: 275 | return "udp" 276 | } 277 | return fmt.Sprintf("%d", p) 278 | } 279 | 280 | type priority uint8 281 | 282 | const CRITICAL = 0 283 | const HIGH = 1 284 | const MEDIUM = 2 285 | const LOW = 3 286 | 287 | func (p *priority) UnmarshalText(data []byte) error { 288 | 289 | text := string(data) 290 | 291 | switch text { 292 | case "critical": 293 | *p = CRITICAL 294 | case "high": 295 | *p = HIGH 296 | case "medium": 297 | *p = MEDIUM 298 | case "low": 299 | *p = LOW 300 | default: 301 | return errors.New("Invalid prority") 302 | } 303 | return nil 304 | } 305 | 306 | func (p priority) MarshalText() ([]byte, error) { 307 | switch p { 308 | case CRITICAL: 309 | return []byte("critcal"), nil 310 | case HIGH: 311 | return []byte("high"), nil 312 | case MEDIUM: 313 | return []byte("medium"), nil 314 | case LOW: 315 | return []byte("low"), nil 316 | } 317 | return nil, errors.New("Invalid prority") 318 | } 319 | 320 | type Service struct { 321 | Address netip.Addr 322 | Port uint16 323 | Protocol Protocol 324 | } 325 | 326 | func (s Service) String() string { 327 | return fmt.Sprintf("%s:%d:%s", s.Address, s.Port, s.Protocol) 328 | } 329 | 330 | func (t Service) MarshalText() ([]byte, error) { 331 | return []byte(fmt.Sprintf("%s:%d:%s", t.Address, t.Port, t.Protocol)), nil 332 | } 333 | 334 | func (t *Service) UnmarshalText(data []byte) error { 335 | 336 | text := string(data) 337 | 338 | //re := regexp.MustCompile(`^(\d+\.\d+\.\d+\.\d+):(\d+):(tcp|udp)$`) 339 | re := regexp.MustCompile(`^([0-9-a-f:\.]+):(\d+):(tcp|udp)$`) 340 | 341 | m := re.FindStringSubmatch(text) 342 | 343 | if len(m) != 4 { 344 | return errors.New("Badly formed ip:port:protocol - " + text) 345 | } 346 | 347 | ip, err := netip.ParseAddr(m[1]) 348 | 349 | if err != nil { 350 | return err 351 | } 352 | 353 | t.Address = ip 354 | 355 | port, err := strconv.Atoi(m[2]) 356 | if err != nil { 357 | return err 358 | } 359 | 360 | if port < 0 || port > 65535 { 361 | return errors.New("Badly formed ip:port:protocol, port out of rance 0-65535 - " + text) 362 | } 363 | 364 | t.Port = uint16(port) 365 | 366 | switch m[3] { 367 | case "tcp": 368 | t.Protocol = TCP 369 | case "udp": 370 | t.Protocol = UDP 371 | default: 372 | return errors.New("Badly formed ip:port:protocol, tcp/udp - " + text) 373 | } 374 | 375 | return nil 376 | } 377 | 378 | type Destination struct { 379 | Address netip.Addr 380 | Port uint16 381 | } 382 | 383 | func (d Destination) String() string { 384 | return fmt.Sprintf("%s:%d", d.Address, d.Port) 385 | } 386 | 387 | func (i Destination) MarshalText() ([]byte, error) { 388 | return []byte(fmt.Sprintf("%s:%d", i.Address, i.Port)), nil 389 | } 390 | 391 | func (i *Destination) UnmarshalText(data []byte) error { 392 | 393 | //re := regexp.MustCompile(`^(\d+\.\d+\.\d+\.\d+)(|:(\d+))$`) 394 | re := regexp.MustCompile(`^([0-9-a-f:\.]+?)(|:(\d+))$`) 395 | 396 | m := re.FindStringSubmatch(string(data)) 397 | 398 | if len(m) != 4 { 399 | return errors.New("Badly formed ip:port") 400 | } 401 | 402 | ip, err := netip.ParseAddr(m[1]) 403 | 404 | if err != nil { 405 | return err 406 | } 407 | 408 | i.Address = ip 409 | 410 | if m[3] != "" { 411 | 412 | port, err := strconv.Atoi(m[3]) 413 | if err != nil { 414 | return err 415 | } 416 | 417 | if port < 0 || port > 65535 { 418 | return errors.New("Badly formed ip:port") 419 | } 420 | 421 | i.Port = uint16(port) 422 | } 423 | 424 | return nil 425 | } 426 | -------------------------------------------------------------------------------- /doc/NOTES.md: -------------------------------------------------------------------------------- 1 | # Notes 2 | 3 | My scribblings - likely not useful to anyone else ... 4 | 5 | 6 | https://wiki.nftables.org/wiki-nftables/index.php/Performing_Network_Address_Translation_(NAT) 7 | 8 | https://unix.stackexchange.com/questions/429077/how-to-do-nat-based-on-port-number-in-stateless-nat 9 | 10 | 11 | Set destination IP address on real server by DSCP - for L3 DSR 12 | 13 | * `nft add table raw` 14 | * `nft add chain raw prerouting { type filter hook prerouting priority raw \; }` 15 | * `nft add rule raw prerouting ip dscp 0x04 ip daddr set 192.168.101.4 notrack` 16 | 17 | https://lpc.events/event/11/contributions/950/attachments/889/1704/lpc_from_xdp_to_socket_fb.pdf 18 | 19 | https://github.com/xdp-project/xdp-tutorial.git 20 | 21 | https://lpc.events/event/2/contributions/71/attachments/17/9/presentation-lpc2018-xdp-tutorial.pdf 22 | 23 | https://yhbt.net/lore/xdp-newbies/CANLN0e5_HtYC1XQ=Z=JRLe-+3bTqoEWdbHJEGhbF7ZT=gz=ynQ@mail.gmail.com/T/ 24 | 25 | 26 | Intel Xeon Gold 6314U CPU @ 2.30GHz 27 | Intel Ethernet 10G 4P X710-T4L-t OCP 28 | using percpu hash - not using LACP: 29 | 30 | * 550K 1.5Gbps 3Mpps 1190ns 36Gbps 31 | * 600K 1.7Gbps 3.25Mpps 1177ns 40Gbps >90% idle 32 | * 650K 1.8Gbps 3.4Mpps 1145ns 42.7Gbps >90% idle 33 | * 675K 1.9Gbps 3.6Mpps 1303ns 44.7Gbps >90% idle 34 | * 700K 2.0Gbps 3.8Mpps 1295ns 46.5Gbps >90% idle 35 | 36 | ## TODOs 37 | 38 | * IPIP/GRE/DSCP L3 support 39 | * Multicast status to help last conns check 40 | * More complete BGP4 implementation 41 | * BFD implementation (maybe no need for this with 3s hold time) 42 | 43 | ## Elasticsearch 44 | 45 | wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - 46 | 47 | echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | \ 48 | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list 49 | 50 | apt update 51 | apt install elasticsearch kibana 52 | 53 | cat >>/etc/elasticsearch/elasticsearch.yml < index patterns -> create index pattern 198 | ... 199 | roles -> create role -> load-balancer : index perms on my-index-name 200 | users -> create user -> load-balancer : roles->load-balancer 201 | 202 | edit logging params ... 203 | 204 | logging: 205 | elasticsearch: 206 | addresses: 207 | - http://10.9.8.7:9200/ 208 | # add more addresses if you have a cluster 209 | index: my-index-name 210 | username: load-balancer 211 | password: Rarkensh1droj 212 | 213 | 214 | 215 | echo "deb [signed-by=/etc/apt/keyrings/elastic.gpg] https://artifacts.elastic.co/packages/7.x/apt stable main" >/etc/apt/sources.list.d/elastic-7.x.list 216 | 217 | cat >/etc/apt/keyrings/elastic.gpg < 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | -------------------------------------------------------------------------------- /elastic.go: -------------------------------------------------------------------------------- 1 | package vc5 2 | 3 | import ( 4 | "bytes" 5 | "context" 6 | 7 | "github.com/elastic/go-elasticsearch/v7" 8 | "github.com/elastic/go-elasticsearch/v7/esapi" 9 | ) 10 | 11 | // https://pkg.go.dev/github.com/elastic/go-elasticsearch/v7 12 | // https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html 13 | 14 | type Elasticsearch struct { 15 | Index string `json:"index,omitempty"` 16 | Addresses []string `json:"addresses,omitempty"` 17 | Username secret `json:"username,omitempty"` 18 | Password secret `json:"password,omitempty"` 19 | 20 | client *elasticsearch.Client 21 | } 22 | 23 | func (e *Elasticsearch) start() (err error) { 24 | 25 | e.client, err = elasticsearch.NewClient(elasticsearch.Config{ 26 | Addresses: e.Addresses, 27 | Username: string(e.Username), 28 | Password: string(e.Password), 29 | }) 30 | 31 | if err != nil { 32 | return err 33 | } 34 | 35 | return nil 36 | } 37 | 38 | func (e *Elasticsearch) log(host string, id uint64, body []byte) bool { 39 | if e.client == nil { 40 | return false 41 | } 42 | 43 | ctx := context.Background() 44 | req := esapi.IndexRequest{ 45 | Index: e.Index, 46 | //DocumentID: fmt.Sprintf("%s-%d", host, id), // don't think that this was ever really needed ... 47 | Body: bytes.NewReader(body), 48 | Refresh: "true", 49 | } 50 | 51 | res, err := req.Do(ctx, e.client) 52 | 53 | if err != nil { 54 | return false 55 | } 56 | 57 | defer res.Body.Close() 58 | 59 | //if res.StatusCode != 201 { 60 | // log.Println(err, res.StatusCode, string(body)) 61 | //} 62 | 63 | return res.StatusCode == 201 64 | } 65 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module vc5 2 | 3 | go 1.19 4 | 5 | require ( 6 | github.com/davidcoles/bgp v0.0.4 7 | github.com/davidcoles/cue v0.1.4 8 | github.com/elastic/go-elasticsearch/v7 v7.17.10 9 | ) 10 | 11 | //replace github.com/davidcoles/cue => ../cue 12 | //replace github.com/davidcoles/bgp => ../bgp 13 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/davidcoles/bgp v0.0.4 h1:K63kH+vnDnFFrhKqftekNELB1C4lOxxmmXFzlJKI5VA= 2 | github.com/davidcoles/bgp v0.0.4/go.mod h1:d9tqNdWCrF0N8LWfOEqrA/GSeMj/XycB60WBZ4uDvEo= 3 | github.com/davidcoles/cue v0.1.4 h1:KrH2hhOLwVKxbEkQL0a6zjXhwZSCzyt0vkRkq0fO7ZM= 4 | github.com/davidcoles/cue v0.1.4/go.mod h1:26FTBytVHJ1XQWOGC+Cfnx4Q9xV8k1xr6K5uZ7s/EBw= 5 | github.com/elastic/go-elasticsearch/v7 v7.17.10 h1:TCQ8i4PmIJuBunvBS6bwT2ybzVFxxUhhltAs3Gyu1yo= 6 | github.com/elastic/go-elasticsearch/v7 v7.17.10/go.mod h1:OJ4wdbtDNk5g503kvlHLyErCgQwwzmDtaFC4XyOxXA4= 7 | -------------------------------------------------------------------------------- /logging.go: -------------------------------------------------------------------------------- 1 | /* 2 | * VC5 load balancer. Copyright (C) 2021-present David Coles 3 | * 4 | * This program is free software; you can redistribute it and/or modify 5 | * it under the terms of the GNU General Public License as published by 6 | * the Free Software Foundation; either version 2 of the License, or 7 | * (at your option) any later version. 8 | * 9 | * This program is distributed in the hope that it will be useful, 10 | * but WITHOUT ANY WARRANTY; without even the implied warranty of 11 | * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 12 | * GNU General Public License for more details. 13 | * 14 | * You should have received a copy of the GNU General Public License along 15 | * with this program; if not, write to the Free Software Foundation, Inc., 16 | * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 17 | */ 18 | 19 | package vc5 20 | 21 | import ( 22 | "bytes" 23 | "encoding/json" 24 | "fmt" 25 | "log" 26 | "log/syslog" 27 | "net/http" 28 | "os" 29 | //"runtime" 30 | "sort" 31 | "strings" 32 | "sync/atomic" 33 | "time" 34 | 35 | "github.com/davidcoles/bgp" 36 | ) 37 | 38 | const ( 39 | EMERG = 0 40 | ALERT = 1 41 | CRIT = 2 42 | ERR = 3 43 | WARNING = 4 44 | NOTICE = 5 45 | INFO = 6 46 | DEBUG = 7 47 | ) 48 | 49 | // old config, to be deprecated 50 | type Logging_ = logging 51 | type logging struct { 52 | Syslog bool `json:"syslog,omitempty"` 53 | Slack secret `json:"slack,omitempty"` 54 | Teams secret `json:"teams,omitempty"` 55 | Alert level `json:"alert,omitempty"` 56 | Elasticsearch Elasticsearch `json:"elasticsearch,omitempty"` 57 | } 58 | 59 | func (l *logging) Logging() Logging { 60 | logging := Logging{ 61 | Elasticsearch: l.Elasticsearch, 62 | Syslog: l.Syslog, 63 | } 64 | 65 | logging.Webhooks = map[secret]Webhook{} 66 | 67 | if l.Teams != "" { 68 | logging.Webhooks[l.Teams] = Webhook{Level: l.Alert, Type: "teams"} 69 | } 70 | 71 | if l.Slack != "" { 72 | logging.Webhooks[l.Slack] = Webhook{Level: l.Alert, Type: "slack"} 73 | } 74 | 75 | return logging 76 | } 77 | 78 | type KV = map[string]any 79 | 80 | type secret string 81 | 82 | func (s secret) MarshalText() ([]byte, error) { return []byte("************"), nil } 83 | func (s *secret) String() string { return "************" } 84 | 85 | type level uint8 86 | 87 | func (l level) String() string { 88 | a := []string{"EMERG", "ALERT", "CRIT", "ERR", "WARNING", "NOTICE", "INFO", "DEBUG"} 89 | 90 | if int(l) < len(a) { 91 | return a[l] 92 | } 93 | 94 | return "UNKNOWN" 95 | } 96 | 97 | type Webhook struct { 98 | Level level `json:"level,omitempty"` 99 | Type string `json:"type,omitempty"` 100 | ent chan ent 101 | } 102 | 103 | type Logging struct { 104 | Elasticsearch Elasticsearch `json:"elasticsearch,omitempty"` 105 | Webhooks map[secret]Webhook `json:"webhooks,omitempty"` 106 | Syslog bool `json:"syslog,omitempty"` 107 | } 108 | 109 | type entry struct { 110 | Indx uint64 `json:"indx"` 111 | Time int64 `json:"time"` 112 | Text string `json:"text"` 113 | } 114 | 115 | type ent struct { 116 | id uint64 117 | host string 118 | 119 | level level 120 | facility string 121 | text string 122 | json []byte 123 | time time.Time 124 | 125 | es *Elasticsearch 126 | typ string 127 | 128 | history []entry 129 | get chan bool 130 | start uint64 131 | alert bool 132 | } 133 | 134 | func NewLogger(hostid string, l Logging) *Sink { 135 | logs := &Sink{HostID: hostid} 136 | logs.Start(l) 137 | return logs 138 | } 139 | 140 | type Sink = sink 141 | type sink struct { 142 | HostID string 143 | 144 | e chan *ent 145 | l chan Logging 146 | host string 147 | 148 | webhook atomic.Uint64 149 | elastic atomic.Uint64 150 | } 151 | 152 | func (s *sink) Debug(msg string, list ...any) { 153 | //if msg == "nat" { 154 | // fmt.Println(list) 155 | //} 156 | m := map[string]any{"msg": msg} 157 | for n := 0; n+1 < len(list); n += 2 { 158 | if key, ok := list[n].(string); ok { 159 | m[key] = list[n+1] 160 | } 161 | } 162 | s.Event(DEBUG, "balancer", "debug", m) 163 | } 164 | 165 | func (s *sink) Sub(f string) *sub { return &sub{parent: s, facility: f} } 166 | func (s *sink) sub(f string) *sub { return &sub{parent: s, facility: f} } 167 | func (s *sink) Event(n uint8, f, a string, e map[string]any) { s.event(n, f, a, e) } 168 | func (s *sink) Alert(n uint8, f, a string, e map[string]any, t ...string) { s.alert(n, f, a, e, t...) } 169 | func (s *sink) State(f, a string, e map[string]any) { s.state(f, a, e) } 170 | 171 | func (s *sink) Fatal(f string, a string, e map[string]any) { 172 | s.Alert(EMERG, f, a, e) 173 | time.Sleep(time.Second) // give log entry the chance to flush 174 | log.Fatal(fmt.Sprint(f, a, e)) 175 | } 176 | 177 | type LogStats struct { 178 | ElasticsearchErrors uint64 `json:"elasticsearch_errors"` 179 | WebhookErrors uint64 `json:"webhook_errors"` 180 | } 181 | 182 | func (s *sink) Stats() (_ LogStats) { 183 | if s == nil { 184 | return 185 | } 186 | return LogStats{ 187 | ElasticsearchErrors: s.elastic.Load(), 188 | WebhookErrors: s.webhook.Load(), 189 | } 190 | } 191 | 192 | const _EVENT = 0 193 | const _STATE = 1 194 | const _ALERT = 2 195 | 196 | func (s *sink) state(facility string, action string, event map[string]any) { 197 | s._event(_STATE, DEBUG, facility, action, event) 198 | } 199 | 200 | func (s *sink) event(lev uint8, facility string, action string, event map[string]any) { 201 | s._event(_EVENT, lev, facility, action, event) 202 | } 203 | 204 | func (s *sink) alert(lev uint8, facility string, action string, event map[string]any, t ...string) { 205 | s._event(_ALERT, lev, facility, action, event, t...) 206 | } 207 | 208 | func (s *sink) _event(kind uint8, lev uint8, facility string, action string, event map[string]any, hrt ...string) { 209 | if s == nil { 210 | return 211 | } 212 | 213 | var alert bool 214 | 215 | level := level(lev) 216 | 217 | now := time.Now() 218 | 219 | reason, ok := event["reason"] 220 | 221 | if ok { 222 | delete(event, "reason") 223 | event["event.reason"] = reason.(string) 224 | } 225 | 226 | event["host.id"] = s.host 227 | event["date"] = now.UnixNano() / int64(time.Millisecond) 228 | event["@timestamp"] = now.UnixNano() / int64(time.Millisecond) 229 | event["level"] = level.String() 230 | event["event.module"] = facility 231 | event["event.action"] = action 232 | event["event.severity"] = uint8(level) 233 | 234 | switch kind { 235 | case _ALERT: 236 | event["event.kind"] = "alert" 237 | alert = true 238 | case _STATE: 239 | event["event.kind"] = "state" 240 | default: 241 | event["event.kind"] = "event" 242 | } 243 | 244 | var t []string 245 | for k, v := range event { 246 | switch k { 247 | case "date": 248 | case "@timestamp": 249 | default: 250 | t = append(t, fmt.Sprintf("%s:%v", k, v)) 251 | } 252 | } 253 | sort.Strings(t) 254 | text := strings.Join(t, " ") 255 | 256 | if len(hrt) > 0 { 257 | text = hrt[0] 258 | } 259 | 260 | js, _ := json.Marshal(event) 261 | 262 | s.e <- &ent{text: text, json: js, level: level, facility: facility, time: now, alert: alert} 263 | } 264 | 265 | func (s *sink) Get(start uint64) (h []entry) { 266 | if s == nil { 267 | return 268 | } 269 | 270 | l := &ent{get: make(chan bool), start: start} 271 | s.e <- l 272 | <-l.get 273 | //return l.history 274 | h = l.history 275 | // reverse h ... simpler to do this in Javascript, perhaps? 276 | for i, j := 0, len(h)-1; i < j; i, j = i+1, j-1 { 277 | h[i], h[j] = h[j], h[i] 278 | } 279 | return h 280 | } 281 | 282 | func (s *sink) Configure(l Logging) { 283 | if s == nil { 284 | return 285 | } 286 | s.l <- l 287 | } 288 | 289 | func (s *sink) Start(l Logging) { 290 | s.e = make(chan *ent, 1000) 291 | s.l = make(chan Logging, 1000) 292 | 293 | { 294 | host := s.HostID 295 | 296 | if host == "" { 297 | host, _ = os.Hostname() 298 | } 299 | 300 | if host == "" { 301 | host = fmt.Sprintf("%d", time.Now().UnixNano()) 302 | } 303 | 304 | s.host = host 305 | } 306 | 307 | go func() { 308 | 309 | // Not using full UnixNano here because large integers cause an 310 | // overflow in jq(1) which I often use for highlighting JSON 311 | // and it confuses me when the numbers are wrong! 312 | id := uint64(time.Now().UnixNano() / 1000000) 313 | 314 | webhooks := map[secret]Webhook{} 315 | var elastic chan ent 316 | var syslog chan ent 317 | console := history() 318 | 319 | config := func(l Logging) { 320 | 321 | if l.Elasticsearch.Index == "" { 322 | if elastic != nil { 323 | close(elastic) 324 | elastic = nil 325 | } 326 | } else { 327 | if elastic == nil { 328 | elastic = elasticSink(l.Elasticsearch, &(s.elastic)) 329 | } else { 330 | select { 331 | case elastic <- ent{es: &(l.Elasticsearch)}: 332 | default: // get rid of blocking channel 333 | close(elastic) 334 | elastic = elasticSink(l.Elasticsearch, &(s.elastic)) 335 | } 336 | } 337 | } 338 | 339 | for k, v := range l.Webhooks { 340 | if x, ok := webhooks[k]; !ok { 341 | // does not exist 342 | v.ent = webhookSink(string(k), &(s.webhook)) 343 | webhooks[k] = v 344 | } else { 345 | // does exist - update 346 | x.Type = v.Type 347 | x.Level = v.Level 348 | webhooks[k] = x 349 | } 350 | } 351 | 352 | for k, v := range webhooks { 353 | if _, ok := l.Webhooks[k]; !ok { 354 | close(v.ent) 355 | delete(webhooks, k) 356 | } 357 | } 358 | 359 | if l.Syslog { 360 | if syslog == nil { 361 | syslog = syslogSink() 362 | } 363 | } else { 364 | if syslog != nil { 365 | close(syslog) 366 | syslog = nil 367 | } 368 | } 369 | 370 | } 371 | 372 | config(l) 373 | 374 | for { 375 | select { 376 | case l := <-s.l: 377 | config(l) 378 | 379 | case e := <-s.e: 380 | e.host = s.host 381 | e.id = id 382 | id++ 383 | 384 | // send to console 385 | if console != nil && ((e.alert && e.level < DEBUG) || e.get != nil) { 386 | select { 387 | case console <- e: 388 | default: 389 | } 390 | } 391 | 392 | if e.get != nil { 393 | break // e.get is only used to get history info for the console - it's not a real log 394 | } 395 | 396 | // send to webhooks 397 | for _, v := range webhooks { 398 | if e.alert && e.level <= v.Level { 399 | e.typ = v.Type 400 | select { 401 | case v.ent <- *e: // copy by value, .typ won't get modified later 402 | default: 403 | s.webhook.Add(1) 404 | } 405 | } 406 | } 407 | 408 | // send to syslog 409 | if syslog != nil { 410 | select { 411 | case syslog <- *e: 412 | default: 413 | } 414 | } 415 | 416 | // send to elasticsearch 417 | if elastic != nil { 418 | select { 419 | case elastic <- *e: 420 | default: 421 | s.elastic.Add(1) 422 | } 423 | } 424 | } 425 | } 426 | }() 427 | } 428 | 429 | func simpleMessage(lines ...string) ([]byte, error) { 430 | type slack struct { 431 | Text string `json:"text"` 432 | } 433 | m := strings.Join(lines, "\n") 434 | 435 | return json.Marshal(&slack{Text: m}) 436 | } 437 | 438 | func adaptiveCard(lines ...string) ([]byte, error) { 439 | 440 | body := []any{} 441 | 442 | for _, text := range lines { 443 | body = append(body, map[string]any{"type": "TextBlock", "text": text, "wrap": true}) 444 | } 445 | 446 | return json.MarshalIndent(map[string]any{ 447 | "type": "message", 448 | "attachments": []any{ 449 | map[string]any{ 450 | "contentType": "application/vnd.microsoft.card.adaptive", 451 | "contentUrl": nil, 452 | "content": map[string]any{ 453 | "$schema": "http://adaptivecards.io/schemas/adaptive-card.json", 454 | "type": "AdaptiveCard", 455 | "version": "1.2", 456 | "body": body, 457 | }, 458 | }, 459 | }, 460 | }, " ", " ") 461 | } 462 | 463 | func payload(l ent) []byte { 464 | switch l.typ { 465 | case "teams": 466 | js, _ := adaptiveCard(fmt.Sprintf("%s %s[%s]:", l.host, l.facility, level(l.level)), l.text) 467 | return js 468 | case "slack": 469 | fallthrough 470 | default: 471 | js, _ := simpleMessage(fmt.Sprintf("%s %s[%s]: %s", l.host, l.facility, level(l.level), l.text)) 472 | return js 473 | } 474 | 475 | return nil 476 | } 477 | 478 | func deliver(dest string, js []byte) bool { 479 | 480 | res, err := http.Post(string(dest), "application/json", bytes.NewReader(js)) 481 | 482 | if err != nil { 483 | return false 484 | } 485 | 486 | defer res.Body.Close() 487 | 488 | // Slack returns 200, Teams returns 202 - return true if either of these 489 | return res.StatusCode == 200 || res.StatusCode == 202 490 | } 491 | 492 | func webhookSink(url string, fail *atomic.Uint64) chan ent { 493 | c := make(chan ent, 1000) 494 | go func() { 495 | for l := range c { 496 | // could batch here 497 | m := payload(l) 498 | if !deliver(url, m) { 499 | fail.Add(1) 500 | } 501 | } 502 | }() 503 | return c 504 | } 505 | 506 | func syslogSink() chan ent { 507 | 508 | s, err := syslog.New(syslog.LOG_WARNING|syslog.LOG_DAEMON, "") 509 | 510 | if err != nil { 511 | return nil 512 | } 513 | 514 | c := make(chan ent, 1000) 515 | 516 | go func() { 517 | // This is probably overkill, but remember hearing something about syslog blocking threads. 518 | /* 519 | orig := runtime.GOMAXPROCS(0) 520 | 521 | runtime.GOMAXPROCS(orig + 1) 522 | 523 | procs := runtime.GOMAXPROCS(0) 524 | 525 | fmt.Println("procs", orig, procs) 526 | 527 | runtime.LockOSThread() 528 | 529 | defer func() { 530 | runtime.UnlockOSThread() 531 | defer runtime.GOMAXPROCS(orig) 532 | defer fmt.Println("EXITING") 533 | }() 534 | */ 535 | 536 | for e := range c { 537 | switch e.level { 538 | case EMERG: 539 | err = s.Emerg(e.text) 540 | case ALERT: 541 | err = s.Alert(e.text) 542 | case CRIT: 543 | err = s.Crit(e.text) 544 | case ERR: 545 | err = s.Err(e.text) 546 | case WARNING: 547 | err = s.Warning(e.text) 548 | case NOTICE: 549 | err = s.Notice(e.text) 550 | case INFO: 551 | err = s.Info(e.text) 552 | case DEBUG: 553 | err = s.Debug(e.text) 554 | } 555 | } 556 | }() 557 | 558 | return c 559 | } 560 | 561 | func elasticSink(es Elasticsearch, f *atomic.Uint64) chan ent { 562 | c := make(chan ent, 1000) 563 | 564 | err := es.start() 565 | 566 | if err != nil { 567 | return nil 568 | } 569 | 570 | go func() { 571 | for e := range c { 572 | 573 | if e.es != nil { // reconfigure 574 | es = *(e.es) 575 | es.start() 576 | } else { 577 | if !es.log(e.host, e.id, e.json) { 578 | f.Add(1) 579 | } 580 | } 581 | } 582 | }() 583 | 584 | return c 585 | } 586 | 587 | func history() chan *ent { 588 | c := make(chan *ent, 1000) 589 | 590 | go func() { 591 | var history []entry // oldest log entry first 592 | 593 | for e := range c { 594 | if e.get != nil { 595 | // find entries newer than l.start 596 | var s []entry 597 | //for n := len(history) - 1; n > 0; n-- { // FIXME n >= 0 maybe ? 598 | for n := len(history) - 1; n >= 0; n-- { 599 | h := history[n] 600 | if h.Indx <= e.start { 601 | break 602 | } 603 | s = append(s, h) 604 | } 605 | e.history = s // s will be in order of newest to oldest 606 | close(e.get) 607 | } else { 608 | history = append(history, entry{Indx: e.id, Text: e.text, Time: e.time.Unix()}) 609 | for len(history) > 1000 { 610 | history = history[1:] 611 | } 612 | } 613 | } 614 | }() 615 | 616 | return c 617 | } 618 | 619 | type Logger interface { 620 | State(f, a string, e map[string]any) 621 | Event(l uint8, f, a string, e map[string]any) 622 | Alert(l uint8, f, a string, e map[string]any, text ...string) // a single text arg is used for human readable log lines if present 623 | } 624 | 625 | type parent interface { 626 | state(f, a string, e map[string]any) 627 | event(l uint8, f, a string, e map[string]any) 628 | alert(l uint8, f, a string, e map[string]any, t ...string) 629 | } 630 | 631 | type Sub = sub 632 | type sub struct { 633 | parent parent 634 | facility string 635 | } 636 | 637 | func (l *sub) State(f, a string, e map[string]any) { l.parent.state(l.facility+"."+f, a, e) } 638 | func (l *sub) Event(n uint8, f, a string, e map[string]any) { 639 | l.parent.event(n, l.facility+"."+f, a, e) 640 | } 641 | func (l *sub) Alert(n uint8, f, a string, e map[string]any, t ...string) { 642 | l.parent.alert(n, l.facility+"."+f, a, e, t...) 643 | } 644 | 645 | func (l *sub) BGPPeer(peer string, params bgp.Parameters, add bool) { 646 | F := "peer" 647 | if add { 648 | //l.NOTICE("add", KV{"peer": peer}) 649 | l.Event(NOTICE, F, "add", KV{"server.address": peer}) 650 | } else { 651 | //l.NOTICE("remove", KV{"peer": peer}) 652 | l.Event(NOTICE, F, "remove", KV{"server.address": peer}) 653 | } 654 | } 655 | 656 | func (l *sub) BGPSession(peer string, local bool, reason string) { 657 | F := "session" 658 | text := fmt.Sprintf("BGP session with %s: %s", peer, reason) 659 | if local { 660 | //l.NOTICE("local", KV{"peer": peer, "reason": reason}) 661 | l.Alert(NOTICE, F, "local", KV{"server.address": peer, "error.message": reason}, text) 662 | } else { 663 | //l.ERR("remote", KV{"peer": peer, "reason": reason}) 664 | l.Alert(ERR, F, "remote", KV{"server.address": peer, "error.message": reason}, text) 665 | } 666 | } 667 | -------------------------------------------------------------------------------- /manager.go: -------------------------------------------------------------------------------- 1 | /* 2 | * VC5 load balancer. Copyright (C) 2021-present David Coles 3 | * 4 | * This program is free software; you can redistribute it and/or modify 5 | * it under the terms of the GNU General Public License as published by 6 | * the Free Software Foundation; either version 2 of the License, or 7 | * (at your option) any later version. 8 | * 9 | * This program is distributed in the hope that it will be useful, 10 | * but WITHOUT ANY WARRANTY; without even the implied warranty of 11 | * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 12 | * GNU General Public License for more details. 13 | * 14 | * You should have received a copy of the GNU General Public License along 15 | * with this program; if not, write to the Free Software Foundation, Inc., 16 | * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 17 | */ 18 | 19 | package vc5 20 | 21 | import ( 22 | "bytes" 23 | "compress/gzip" 24 | "context" 25 | "encoding/json" 26 | "fmt" 27 | //"log" 28 | "net" 29 | "net/http" 30 | "net/netip" 31 | "runtime/debug" 32 | "strconv" 33 | "strings" 34 | "sync" 35 | "time" 36 | 37 | "github.com/davidcoles/bgp" 38 | "github.com/davidcoles/cue" 39 | "github.com/davidcoles/cue/mon" 40 | ) 41 | 42 | type Manager struct { 43 | config *Config 44 | Director *cue.Director 45 | Balancer Balancer 46 | Logs *Sink 47 | WebRoot string 48 | BGPLoopback uint16 49 | NAT func(netip.Addr, netip.Addr) (netip.Addr, bool) 50 | Prober func(netip.Addr, netip.Addr, Check) (bool, string) 51 | RouterID [4]byte 52 | WebListener net.Listener 53 | Interval uint8 54 | Learn uint 55 | 56 | Address netip.Addr 57 | SNI bool 58 | 59 | HardFail bool 60 | 61 | pool *bgp.Pool 62 | mutex sync.Mutex 63 | services servicemap 64 | summary Summary 65 | vip map[netip.Addr]state 66 | rib []netip.Addr 67 | } 68 | 69 | type Check = mon.Check 70 | 71 | func Monitor(addr netip.Addr, cic bool) (*mon.Mon, error) { 72 | m := &mon.Mon{ 73 | IPv4: addr, // for SYN probes 74 | CloseIdleConnections: cic, 75 | } 76 | 77 | err := m.Init(nil) 78 | 79 | if err != nil { 80 | return nil, err 81 | } 82 | 83 | return m, nil 84 | } 85 | 86 | func (m *Manager) Probe(_ *mon.Mon, i mon.Instance, check mon.Check) (ok bool, diagnostic string) { 87 | 88 | vip := i.Service.Address 89 | rip := i.Destination.Address 90 | 91 | if m.NAT != nil { 92 | // if we're using NAT to reach the backend then map the VIP/RIP tuple to the NAT address 93 | rip, ok = m.NAT(vip, rip) 94 | if !ok { 95 | return false, "NAT lookup failed" 96 | } 97 | } 98 | 99 | return m.Prober(vip, rip, check) 100 | } 101 | 102 | func (m *Manager) Manage(ctx context.Context, cfg *Config) error { 103 | // mostly lifted from main.go - probably need a bit of rationalising 104 | 105 | m.config = cfg 106 | 107 | learn := time.Duration(m.Learn) 108 | if learn == 0 { 109 | learn = 1 110 | } 111 | 112 | start := time.Now() 113 | F := "vc5" 114 | 115 | m.Director = &cue.Director{ 116 | Notifier: m, 117 | SNI: m.SNI, 118 | Address: m.Address, 119 | } 120 | 121 | if m.Prober != nil { 122 | m.Director.Prober = m 123 | } 124 | 125 | routerID := m.RouterID 126 | 127 | // If loopback BGP mode is enabled (m.BGPLoopback is the ASN that we should use) then override router ID 128 | if m.BGPLoopback > 0 { 129 | routerID = [4]byte{127, 0, 0, 1} // no sensible BGP daemon would ever use this, surely! 130 | } 131 | 132 | m.pool = bgp.NewPool(routerID, m.config.Bgp(m.BGPLoopback, true), nil, m.Logs.Sub("bgp")) 133 | 134 | if m.pool == nil { 135 | return fmt.Errorf("BGP pool fail") 136 | } 137 | 138 | if err := m.Director.Start(m.config.Parse()); err != nil { 139 | return err 140 | } 141 | 142 | old := map[Instance]Stats{} 143 | m.vip = map[netip.Addr]state{} 144 | m.summary, m.services, old = serviceStatus(m.config, m.Balancer, m.Director, nil) 145 | 146 | // Collect stats 147 | go func() { 148 | 149 | interval := m.Interval 150 | 151 | if interval == 0 { 152 | interval = 10 153 | } 154 | 155 | ticker := time.NewTicker(time.Duration(interval) * time.Second) 156 | defer ticker.Stop() 157 | 158 | for { 159 | var summary Summary 160 | m.mutex.Lock() 161 | summary, m.services, old = serviceStatus(m.config, m.Balancer, m.Director, old) 162 | m.summary.Update(summary, start) 163 | m.mutex.Unlock() 164 | select { 165 | case <-ticker.C: 166 | case <-ctx.Done(): 167 | return 168 | } 169 | } 170 | }() 171 | 172 | // advertise VIPs via BGP 173 | go func() { 174 | timer := time.NewTimer(learn * time.Second) 175 | ticker := time.NewTicker(5 * time.Second) 176 | services := m.Director.Status() 177 | 178 | defer func() { 179 | ticker.Stop() 180 | timer.Stop() 181 | m.pool.RIB(nil) 182 | time.Sleep(2 * time.Second) 183 | m.pool.Close() 184 | }() 185 | 186 | var initialised bool 187 | for { 188 | select { 189 | case <-ctx.Done(): // shuting down 190 | return 191 | case <-ticker.C: // check for matured VIPs 192 | //m.mutex.Lock() 193 | //vipmap = VipLog(director.Status(), vipmap, config.Priorities(), logs) 194 | //m.mutex.Unlock() 195 | case <-m.Director.C: // a backend has changed state 196 | m.mutex.Lock() 197 | services = m.Director.Status() 198 | var manifests []Manifest 199 | for _, s := range services { 200 | key := Service{Address: s.Address, Port: s.Port, Protocol: Protocol(s.Protocol)} 201 | service, ok := m.config.Services[key] 202 | 203 | if ok { 204 | //manifests = append(manifests, Manifest(s)) 205 | manifests = append(manifests, toManifest(s, service)) 206 | } 207 | } 208 | err := m.Balancer.Configure(manifests) 209 | m.mutex.Unlock() 210 | if err != nil { 211 | // if the configuration failed then the system is in an corrupt state 212 | // so the best thing to do is exit? (set HardFail) - not default yet 213 | text := "Couldn't apply config: " + err.Error() 214 | if m.HardFail { 215 | m.Logs.Fatal(F, "manager", KV{"error.message": text}) 216 | } else { 217 | m.Logs.Alert(ERR, F, "manager", KV{"error.message": text}, text) 218 | } 219 | } 220 | 221 | case <-timer.C: 222 | m.Logs.Alert(NOTICE, F, "learn-timer-expired", KV{}, "Learn timer expired") 223 | initialised = true 224 | } 225 | 226 | m.mutex.Lock() 227 | m.vip = vipState(services, m.vip, m.config.Priorities(), m.Logs, initialised) 228 | m.rib = adjRIBOut(m.vip, initialised) 229 | m.mutex.Unlock() 230 | 231 | m.pool.RIB(m.rib) 232 | } 233 | }() 234 | 235 | static := http.FS(STATIC) 236 | var fs http.FileSystem 237 | 238 | webroot := m.WebRoot 239 | 240 | if webroot != "" { 241 | fs = http.FileSystem(http.Dir(webroot)) 242 | } 243 | 244 | http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { 245 | 246 | if fs != nil { 247 | file := r.URL.Path 248 | if file == "/" { 249 | file = "/index.html" 250 | } 251 | 252 | if f, err := fs.Open(file); err == nil { 253 | f.Close() 254 | http.FileServer(fs).ServeHTTP(w, r) 255 | return 256 | } 257 | } 258 | 259 | r.URL.Path = "static/" + r.URL.Path 260 | http.FileServer(static).ServeHTTP(w, r) 261 | }) 262 | 263 | http.HandleFunc("/log/", func(w http.ResponseWriter, r *http.Request) { 264 | 265 | start, _ := strconv.ParseUint(r.URL.Path[5:], 10, 64) 266 | logs := m.Logs.Get(start) 267 | js, err := json.MarshalIndent(&logs, " ", " ") 268 | if err != nil { 269 | w.WriteHeader(http.StatusInternalServerError) 270 | return 271 | } 272 | w.Header().Set("Content-Type", "application/json") 273 | w.Write(js) 274 | w.Write([]byte("\n")) 275 | }) 276 | 277 | http.HandleFunc("/build.json", func(w http.ResponseWriter, r *http.Request) { 278 | info, ok := debug.ReadBuildInfo() 279 | if !ok { 280 | w.WriteHeader(http.StatusNotFound) 281 | return 282 | } 283 | 284 | js, err := json.MarshalIndent(info, " ", " ") 285 | if err != nil { 286 | w.WriteHeader(http.StatusInternalServerError) 287 | return 288 | } 289 | w.Header().Set("Content-Type", "application/json") 290 | w.Write(js) 291 | w.Write([]byte("\n")) 292 | }) 293 | 294 | http.HandleFunc("/cue.json", func(w http.ResponseWriter, r *http.Request) { 295 | js, err := m.Cue() 296 | if err != nil { 297 | w.WriteHeader(http.StatusInternalServerError) 298 | return 299 | } 300 | w.Header().Set("Content-Type", "application/json") 301 | w.Write(js) 302 | w.Write([]byte("\n")) 303 | }) 304 | 305 | http.HandleFunc("/status.json", func(w http.ResponseWriter, r *http.Request) { 306 | js, err := m.JSONStatus() 307 | 308 | if err != nil { 309 | w.WriteHeader(http.StatusInternalServerError) 310 | return 311 | } 312 | w.Header().Set("Content-Type", "application/json") 313 | js = append(js, 0x0a) // add a newline 314 | w.Write(js) 315 | }) 316 | 317 | http.HandleFunc("/status.json.gz", func(w http.ResponseWriter, r *http.Request) { 318 | js, err := m.JSONStatus() 319 | 320 | if err != nil { 321 | w.WriteHeader(http.StatusInternalServerError) 322 | return 323 | } 324 | 325 | w.Header().Set("Content-Type", "application/json") 326 | w.Header().Set("Content-Encoding", "gzip") 327 | js = append(js, 0x0a) // add a newline 328 | 329 | var b bytes.Buffer 330 | gz := gzip.NewWriter(&b) 331 | gz.Write(js) 332 | gz.Close() 333 | 334 | w.Write(b.Bytes()) 335 | }) 336 | 337 | http.HandleFunc("/metrics", func(w http.ResponseWriter, r *http.Request) { 338 | 339 | types, metrics := m.Balancer.Metrics() 340 | 341 | w.Header().Set("Content-Type", "text/plain") 342 | 343 | for _, n := range types { 344 | w.Write([]byte(fmt.Sprintf("# TYPE xvs_%s\n", n))) 345 | } 346 | 347 | w.Write([]byte(strings.Join(m.Prometheus("vc5"), "\n") + "\n")) 348 | 349 | for _, m := range metrics { 350 | w.Write([]byte(fmt.Sprintln("xvs_" + m))) 351 | } 352 | 353 | }) 354 | 355 | http.HandleFunc("/config.json", func(w http.ResponseWriter, r *http.Request) { 356 | m.mutex.Lock() 357 | js, err := json.MarshalIndent(m.config, " ", " ") 358 | m.mutex.Unlock() 359 | 360 | if err != nil { 361 | w.WriteHeader(http.StatusInternalServerError) 362 | return 363 | } 364 | w.Header().Set("Content-Type", "application/json") 365 | w.Write(js) 366 | w.Write([]byte("\n")) 367 | }) 368 | 369 | listener := m.WebListener 370 | if listener != nil { 371 | go func() { 372 | for { 373 | server := http.Server{} 374 | err := server.Serve(listener) 375 | m.Logs.Alert(ALERT, F, "webserver", KV{"error.message": err.Error()}, "Webserver exited: "+err.Error()) 376 | time.Sleep(10 * time.Second) 377 | } 378 | }() 379 | } 380 | 381 | return nil 382 | } 383 | 384 | func (m *Manager) Configure(config *Config) { 385 | m.mutex.Lock() 386 | defer m.mutex.Unlock() 387 | m.Director.Configure(config.Parse()) 388 | m.pool.Configure(config.Bgp(m.BGPLoopback, false)) 389 | m.Logs.Configure(config.LoggingConfig()) 390 | m.config = config 391 | } 392 | 393 | func (m *Manager) Cue() ([]byte, error) { 394 | m.mutex.Lock() 395 | defer m.mutex.Unlock() 396 | return json.MarshalIndent(m.Director.Status(), " ", " ") 397 | } 398 | 399 | func (m *Manager) JSONStatus() ([]byte, error) { 400 | m.mutex.Lock() 401 | defer m.mutex.Unlock() 402 | return jsonStatus(m.summary, m.services, m.vip, m.pool, m.rib, m.Logs.Stats()) 403 | } 404 | 405 | func (m *Manager) Prometheus(s string) []string { 406 | m.mutex.Lock() 407 | defer m.mutex.Unlock() 408 | return Prometheus(s, m.services, m.summary, m.vip) 409 | } 410 | 411 | func jsonStatus(summary Summary, services servicemap, vips map[netip.Addr]state, pool *bgp.Pool, rib []netip.Addr, logstats LogStats) ([]byte, error) { 412 | return json.MarshalIndent(struct { 413 | Summary Summary `json:"summary"` 414 | Services servicemap `json:"services"` 415 | BGP map[string]bgp.Status `json:"bgp"` 416 | VIP []vipstats `json:"vip"` 417 | RIB []netip.Addr `json:"rib"` 418 | Logging LogStats `json:"logging"` 419 | }{ 420 | Summary: summary, 421 | Services: services, 422 | BGP: pool.Status(), 423 | VIP: vipStatus(services, vips), 424 | RIB: rib, 425 | Logging: logstats, 426 | }, " ", " ") 427 | } 428 | 429 | /**********************************************************************/ 430 | // notifications 431 | /**********************************************************************/ 432 | 433 | func ms(s mon.Service) Service { 434 | return Service{Address: s.Address, Port: s.Port, Protocol: Protocol(s.Protocol)} 435 | } 436 | 437 | func md(d mon.Destination) Destination { 438 | return Destination{Address: d.Address, Port: d.Port} 439 | } 440 | 441 | // interface method called by mon when a destination's health status transitions up or down 442 | func (m *Manager) Notify(instance mon.Instance, state bool) { 443 | text := fmt.Sprintf("Backend %s for service %s went %s", md(instance.Destination), ms(instance.Service), upDown(state)) 444 | m.Logs.Alert(5, "healthcheck", "state", notifyLog(instance, state), text) 445 | } 446 | 447 | // interface method called by mon every time a round of checks for a service on a destination is completed 448 | func (m *Manager) Result(instance mon.Instance, state bool, diagnostic string) { 449 | //m.Logs.Event(7, "healthcheck", "state", resultLog(instance, state, diagnostic)) 450 | m.Logs.State("healthcheck", "state", resultLog(instance, state, diagnostic)) 451 | } 452 | 453 | func (m *Manager) Check(instance mon.Instance, check string, round uint64, state bool, diagnostic string) { 454 | entry := checkLog(instance, state, diagnostic, check, round) 455 | if m.NAT != nil { 456 | nat, _ := m.NAT(instance.Service.Address, instance.Destination.Address) 457 | entry["destination.nat.ip"] = nat 458 | } 459 | m.Logs.Event(7, "healthcheck", "check", entry) 460 | } 461 | 462 | func notifyLog(instance mon.Instance, state bool) map[string]any { 463 | // https://www.elastic.co/guide/en/ecs/current/ecs-base.html 464 | // https://github.com/elastic/ecs/blob/main/generated/csv/fields.csv 465 | 466 | s := Service{Address: instance.Service.Address, Port: instance.Service.Port, Protocol: Protocol(instance.Service.Protocol)} 467 | d := Destination{Address: instance.Destination.Address, Port: instance.Destination.Port} 468 | 469 | return map[string]any{ 470 | "service.state": upDown(state), 471 | "service.ip": s.Address.String(), 472 | "service.port": instance.Service.Port, 473 | "service.protocol": s.Protocol.String(), 474 | "service.address": s.String(), 475 | "destination.ip": d.Address.String(), 476 | "destination.port": d.Port, 477 | "destination.address": d.String(), 478 | } 479 | } 480 | 481 | func resultLog(instance mon.Instance, status bool, diagnostic string) map[string]any { 482 | r := notifyLog(instance, status) 483 | r["diagnostic"] = diagnostic 484 | return r 485 | } 486 | 487 | func checkLog(instance mon.Instance, status bool, diagnostic string, check string, round uint64) map[string]any { 488 | r := resultLog(instance, status, diagnostic) 489 | r["check"] = check 490 | r["round"] = round 491 | return r 492 | } 493 | 494 | func adjRIBOut(vip map[netip.Addr]state, initialised bool) (r []netip.Addr) { 495 | for v, s := range vip { 496 | if initialised && s.up && time.Now().Sub(s.time) > time.Second*5 { 497 | r = append(r, v) 498 | } 499 | } 500 | return 501 | } 502 | 503 | /* 504 | func VipMap(services []cue.Service) map[netip.Addr]bool { 505 | 506 | m := map[netip.Addr]bool{} 507 | 508 | for _, v := range cue.AllVIPs(services) { 509 | m[v] = false 510 | } 511 | 512 | for _, v := range cue.HealthyVIPs(services) { 513 | m[v] = true 514 | } 515 | 516 | fmt.Println(m) 517 | 518 | return m 519 | } 520 | 521 | func VipLog(services []cue.Service, old map[netip.Addr]bool, priorities map[netip.Addr]priority, logs Logger) map[netip.Addr]bool { 522 | 523 | f := "vip" 524 | 525 | m := VipMap(services) 526 | 527 | for vip, state := range m { 528 | 529 | severity := p2s(priorities[vip]) 530 | 531 | if was, exists := old[vip]; exists { 532 | if state != was { 533 | logs.Alert(severity, f, "state", KV{"service.ip": vip, "service.state": upDown(state)}) 534 | } 535 | } else { 536 | logs.Alert(INFO, f, "added", KV{"service.ip": vip, "service.state": upDown(state)}) 537 | } 538 | 539 | logs.Event(DEBUG, f, "state", KV{"service.ip": vip, "service.state": upDown(state)}) 540 | } 541 | 542 | for vip, _ := range old { 543 | if _, exists := m[vip]; !exists { 544 | logs.Alert(6, f, "removed", KV{"service.ip": vip}) 545 | } 546 | } 547 | 548 | return m 549 | } 550 | 551 | */ 552 | -------------------------------------------------------------------------------- /prometheus.go: -------------------------------------------------------------------------------- 1 | /* 2 | * VC5 load balancer. Copyright (C) 2021-present David Coles 3 | * 4 | * This program is free software; you can redistribute it and/or modify 5 | * it under the terms of the GNU General Public License as published by 6 | * the Free Software Foundation; either version 2 of the License, or 7 | * (at your option) any later version. 8 | * 9 | * This program is distributed in the hope that it will be useful, 10 | * but WITHOUT ANY WARRANTY; without even the implied warranty of 11 | * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 12 | * GNU General Public License for more details. 13 | * 14 | * You should have received a copy of the GNU General Public License along 15 | * with this program; if not, write to the Free Software Foundation, Inc., 16 | * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 17 | */ 18 | 19 | package vc5 20 | 21 | import ( 22 | "fmt" 23 | "net/netip" 24 | "strings" 25 | "time" 26 | ) 27 | 28 | func Prometheus(p string, services servicemap, summary Summary, vips map[netip.Addr]state) []string { 29 | r := []string{help(p)} 30 | 31 | var defcon uint8 32 | 33 | r = append(r, fmt.Sprintf(p+`_uptime %d`, summary.Uptime)) 34 | r = append(r, fmt.Sprintf(p+`_defcon %d`, defcon)) 35 | r = append(r, fmt.Sprintf(p+`_latency %d`, summary.Latency)) 36 | r = append(r, fmt.Sprintf(p+`_sessions %d`, summary.Current)) 37 | r = append(r, fmt.Sprintf(p+`_session_total %d`, summary.Flows)) 38 | r = append(r, fmt.Sprintf(p+`_toobig %d`, summary.TooBig)) 39 | r = append(r, fmt.Sprintf(p+`_rx_packets %d`, summary.IngressPackets)) 40 | r = append(r, fmt.Sprintf(p+`_rx_octets %d`, summary.IngressOctets)) 41 | r = append(r, fmt.Sprintf(p+`_tx_packets %d`, summary.EgressPackets)) 42 | r = append(r, fmt.Sprintf(p+`_tx_octets %d`, summary.EgressOctets)) 43 | 44 | zeroone := func(u bool) uint8 { 45 | if u { 46 | return 1 47 | } 48 | return 0 49 | } 50 | 51 | updown := func(u bool) string { 52 | if u { 53 | return "up" 54 | } 55 | return "down" 56 | } 57 | 58 | now := time.Now() 59 | 60 | for vip, s := range vips { 61 | r = metric(r, p+`_vip_status{vip="%s"} %d`, vip, zeroone(s.Up())) 62 | r = metric(r, p+`_vip_status_duration{vip="%s",status="%s"} %d`, vip, updown(s.Up()), now.Sub(s.Time())/time.Second) 63 | } 64 | 65 | for _, x := range services { 66 | for _, s := range x { 67 | serv := fmt.Sprintf("%s:%s:%d", s.Address, s.Protocol, s.Port) 68 | name := s.Name 69 | stat := s.Stats 70 | up := zeroone(s.Up) 71 | 72 | name = strings.ReplaceAll(name, `\`, `\\`) 73 | name = strings.ReplaceAll(name, `"`, `\"`) 74 | 75 | r = metric(r, p+`_service_sessions{service="%s",name="%s"} %d`, serv, name, stat.Current) 76 | r = metric(r, p+`_service_sessions_total{service="%s",name="%s"} %d`, serv, name, stat.Flows) 77 | r = metric(r, p+`_service_rx_packets{service="%s",name="%s"} %d`, serv, name, stat.IngressPackets) 78 | r = metric(r, p+`_service_rx_octets{service="%s",name="%s"} %d`, serv, name, stat.IngressOctets) 79 | r = metric(r, p+`_service_tx_packets{service="%s",name="%s"} %d`, serv, name, stat.EgressPackets) 80 | r = metric(r, p+`_service_tx_octets{service="%s",name="%s"} %d`, serv, name, stat.EgressOctets) 81 | r = metric(r, p+`_service_status{service="%s",name="%s"} %d`, serv, name, up) 82 | r = metric(r, p+`_service_status_duration{service="%s",name="%s",status="%s"} %d`, serv, name, updown(s.Up), s.For) 83 | r = metric(r, p+`_service_reserves_used{service="%s",name="%s"} %d`, serv, name, 666) 84 | 85 | for _, d := range s.Destinations { 86 | real := fmt.Sprintf("%s:%d", d.Address, d.Port) 87 | stat := d.Stats 88 | up := zeroone(d.Up) 89 | 90 | r = metric(r, p+`_backend_sessions{service="%s",name="%s",backend="%s"} %d`, serv, name, real, stat.Current) 91 | r = metric(r, p+`_backend_sessions_total{service="%s",name="%s",backend="%s"} %d`, serv, name, real, stat.Flows) 92 | r = metric(r, p+`_backend_rx_packets{service="%s",name="%s",backend="%s"} %d`, serv, name, real, stat.IngressPackets) 93 | r = metric(r, p+`_backend_rx_octets{service="%s",name="%s",backend="%s"} %d`, serv, name, real, stat.IngressOctets) 94 | r = metric(r, p+`_backend_tx_packets{service="%s",name="%s",backend="%s"} %d`, serv, name, real, stat.EgressPackets) 95 | r = metric(r, p+`_backend_tx_octets{service="%s",name="%s",backend="%s"} %d`, serv, name, real, stat.EgressOctets) 96 | r = metric(r, p+`_backend_status{service="%s",name="%s",backend="%s"} %d`, serv, name, real, up) 97 | r = metric(r, p+`_backend_status_duration{service="%s",name="%s",backend="%s",status="%s"} %d`, serv, name, real, updown(d.Up), d.For) 98 | r = metric(r, p+`_backend_reserves_used{service="%s",name="%s",backend="%s"} %d`, serv, name, real, 666) 99 | 100 | } 101 | } 102 | } 103 | 104 | //return strings.Join(r, "\n") + "\n" 105 | return r 106 | } 107 | 108 | func metric(l []string, f string, a ...any) []string { 109 | return append(l, fmt.Sprintf(f, a...)) 110 | } 111 | 112 | func help(p string) string { 113 | return `# TYPE ` + p + `_uptime counter 114 | # TYPE ` + p + `_defcon gauge 115 | # TYPE ` + p + `_latency gauge 116 | # TYPE ` + p + `_sessions gauge 117 | # TYPE ` + p + `_session_total counter 118 | # TYPE ` + p + `_rx_packets counter 119 | # TYPE ` + p + `_rx_octets counter 120 | # TYPE ` + p + `_tx_packets counter 121 | # TYPE ` + p + `_tx_octets counter 122 | # TYPE ` + p + `_vip_status gauge 123 | # TYPE ` + p + `_vip_status_duration gauge 124 | # TYPE ` + p + `_service_sessions gauge 125 | # TYPE ` + p + `_service_sessions_total counter 126 | # TYPE ` + p + `_service_rx_packets counter 127 | # TYPE ` + p + `_service_rx_octets counter 128 | # TYPE ` + p + `_service_tx_packets counter 129 | # TYPE ` + p + `_service_tx_octets counter 130 | # TYPE ` + p + `_service_status gauge 131 | # TYPE ` + p + `_service_status_duration gauge 132 | # TYPE ` + p + `_service_reserves_used gauge 133 | # TYPE ` + p + `_backend_sessions gauge 134 | # TYPE ` + p + `_backend_sessions_total counter 135 | # TYPE ` + p + `_backend_rx_packets counter 136 | # TYPE ` + p + `_backend_rx_octets counter 137 | # TYPE ` + p + `_backend_tx_packets counter 138 | # TYPE ` + p + `_backend_tx_octets counter 139 | # TYPE ` + p + `_backend_status gauge 140 | # TYPE ` + p + `_backend_status_duration gauge 141 | # HELP ` + p + `_uptime Uptime in seconds 142 | # HELP ` + p + `_defcon Readiness level 143 | # HELP ` + p + `_latency Average packet processing latency in nanoseconds 144 | # HELP ` + p + `_sessions Estimated number of current active sessions 145 | # HELP ` + p + `_session_total Total number of new sessions written to state tracking table 146 | # HELP ` + p + `_rx_packets Total number of incoming packets 147 | # HELP ` + p + `_rx_octets Total number incoming bytes 148 | # HELP ` + p + `_tx_packets Total number of outgoing packets 149 | # HELP ` + p + `_tx_octets Total number outgoing bytes 150 | # HELP ` + p + `_vip_status gauge 151 | # HELP ` + p + `_vip_status_duration gauge 152 | # HELP ` + p + `_service_sessions gauge 153 | # HELP ` + p + `_service_sessions_total counter 154 | # HELP ` + p + `_service_rx_packets counter 155 | # HELP ` + p + `_service_rx_octets counter 156 | # HELP ` + p + `_service_tx_packets counter 157 | # HELP ` + p + `_service_tx_octets counter 158 | # HELP ` + p + `_service_status gauge 159 | # HELP ` + p + `_service_status_duration gauge 160 | # HELP ` + p + `_service_reserves_used gauge 161 | # HELP ` + p + `_backend_sessions gauge 162 | # HELP ` + p + `_backend_sessions_total counter 163 | # HELP ` + p + `_backend_rx_packets counter 164 | # HELP ` + p + `_backend_rx_octets counter 165 | # HELP ` + p + `_backend_tx_packets counter 166 | # HELP ` + p + `_backend_tx_octets counter 167 | # HELP ` + p + `_backend_status gauge 168 | # HELP ` + p + `_backend_status_duration gauge` 169 | } 170 | -------------------------------------------------------------------------------- /static/favicon.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/davidcoles/vc5/fce36d4f4733d19bf76ea59f6661bedbdbea0584/static/favicon.ico -------------------------------------------------------------------------------- /static/index.css: -------------------------------------------------------------------------------- 1 | body { 2 | font-family: arial, helvetica, sans-serif; 3 | font-size: 12px; 4 | font-weight: normal; 5 | color: black; 6 | background: white; 7 | } 8 | 9 | th,td { font-size: 10px;} 10 | 11 | #console { 12 | height: 150px; 13 | font-size: 10px; 14 | background: #e0e0e0; 15 | overflow-x: hidden; 16 | overflow-x: auto; 17 | } 18 | table, th, td { 19 | border: 1px solid black; 20 | border-collapse: collapse; 21 | text-align: center; 22 | } 23 | th, td { 24 | padding-top: 5px; 25 | padding-bottom: 5px; 26 | padding-left: 10px; 27 | padding-right: 10px; 28 | } 29 | 30 | td.d1 { background: white; } 31 | td.d2 { background: #ff7373; } 32 | td.d3 { background: #fdff73; } 33 | td.d4 { background: #00d199; } 34 | td.d5 { background: #008dff; } 35 | td.ip { background: #b00040; color: white; font-weight: bold; } 36 | th.ip { background: #b00040; color: white; font-weight: bold; } 37 | td.up { background: #a0f0a0; color: black; font-weight: bold; } 38 | td.dn { background: #f0a0a0; color: black; font-weight: bold; } 39 | 40 | tr.hd { background: #a0a0f0; } 41 | tr.up { background: #a0f0a0; } 42 | tr.dn { background: #f0a0a0; } 43 | tr.fb { background: #a0f0f0; } 44 | tr.ds { background: #888888; } 45 | 46 | th.al, td.al { text-align: left; } 47 | th.ac, td.ac { text-align: center; } 48 | th.ar, td.ar { text-align: right; } 49 | -------------------------------------------------------------------------------- /static/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 10 | 11 | 12 | 13 | config | 14 | status | 15 | metrics 16 | 17 |
18 | 19 |
20 | 21 |
22 | 23 |
24 | 25 |
26 | 27 |
28 | 29 |
30 | 31 |
32 | 33 | 34 | 35 | 36 | 37 | 38 | -------------------------------------------------------------------------------- /static/index.js: -------------------------------------------------------------------------------- 1 | var currents = {}; 2 | var lastms = 0; 3 | 4 | function sleep(ms) { 5 | return new Promise(resolve => setTimeout(resolve, ms)); 6 | } 7 | 8 | var getJSON = function(url, callback) { 9 | var xhr = new XMLHttpRequest(); 10 | xhr.open('GET', url, true); 11 | xhr.responseType = 'json'; 12 | xhr.onload = function() { 13 | var status = xhr.status; 14 | if (status === 200) { 15 | callback(null, xhr.response); 16 | } else { 17 | callback(status, xhr.response); 18 | } 19 | }; 20 | xhr.send(); 21 | }; 22 | 23 | function dhms(s) { // s - seconds 24 | var days = Math.floor(s / 86400); 25 | var hours = Math.floor((s % 86400) / 3600); 26 | var minutes = Math.floor((s % 3600) / 60); 27 | var seconds = Math.floor(s % 60); 28 | var start = false 29 | var dhms = "" 30 | 31 | if(days > 0) { 32 | dhms += days + "d" 33 | start = true 34 | } 35 | 36 | if(start || hours > 0) { 37 | dhms += hours + "h" 38 | if(start) { 39 | return dhms 40 | } 41 | start = true 42 | } 43 | 44 | if(start || minutes > 0) { 45 | dhms += minutes + "m" 46 | if(start) { 47 | return dhms 48 | } 49 | } 50 | 51 | dhms += seconds + "s" 52 | return dhms 53 | } 54 | 55 | 56 | function append(p, type, html, c) { 57 | var e = document.createElement(type) 58 | if(c !== undefined) { 59 | e.setAttribute("class", c) 60 | } 61 | if(html !== undefined && html !== null) { 62 | e.innerHTML = html 63 | } 64 | p.appendChild(e) 65 | return e 66 | } 67 | 68 | function esc(s) { 69 | return s 70 | .replace(/&/g, "&") 71 | .replace(//g, ">") 73 | .replace(/"/g, """) 74 | .replace(/'/g, "'"); 75 | } 76 | 77 | function spc(x) { 78 | return x.toString().replace(/\B(?=(\d{3})+(?!\d))/g, " "); 79 | } 80 | 81 | function tsf(num) { 82 | if(num === undefined) { 83 | return "XXX"; 84 | } 85 | var suffix = ['', 'K', 'M', 'G', 'T', 'P', 'E', 'Z', 'Y']; 86 | 87 | if(num < 1000) { 88 | return num.toString() + " "; 89 | } 90 | 91 | for(; num > 1000 && suffix.length > 1;) { 92 | num /= 1000 93 | suffix.shift() 94 | } 95 | 96 | if(num >= 100) { 97 | return Math.round(num).toString() + " " + suffix[0] 98 | } 99 | 100 | if(num > 10) { 101 | return num.toFixed(1) + " " + suffix[0] 102 | } 103 | 104 | return num.toFixed(2) + " " + suffix[0] 105 | } 106 | 107 | 108 | function updateStatus(url) { 109 | getJSON(url, function(err, data) { 110 | if (err !== null) { 111 | console.log('Something went wrong: ' + err) 112 | } else { 113 | 114 | var summary = summary_t(data.summary, data.bgp) 115 | 116 | var services = document.createElement("div"); 117 | 118 | var vips = document.createElement("div"); 119 | 120 | var rib = {} 121 | 122 | if(data.rib != undefined) { 123 | for(var vip of data.rib) { 124 | rib[vip] = true 125 | } 126 | } 127 | 128 | vips.appendChild(vips_t(data.vip, data.summary.dsr)) 129 | 130 | //for(var vip in data.services) { 131 | for(var v of data.vip) { 132 | var vip = v.vip 133 | services.appendChild(serv(v, vip, data.services[vip], rib[vip] ? true : false, data.summary.dsr)) 134 | append(services, "div", " ") 135 | } 136 | 137 | document.getElementById("vips").replaceWith(vips); 138 | vips.id = "vips"; 139 | 140 | document.getElementById("summary").replaceWith(summary); 141 | summary.id = "summary"; 142 | 143 | document.getElementById("services").replaceWith(services); 144 | services.id = "services"; 145 | } 146 | }) 147 | } 148 | 149 | function vips_t(vips, dsr) { 150 | var t = document.createElement("table"); 151 | var th = append(t, "tr", null, "hd") 152 | append(th, "th", "VIP") 153 | append(th, "th", "State") 154 | append(th, "th", "Traffic").setAttribute("colspan", dsr ? 1 : 2) 155 | append(th, "th", "Packets").setAttribute("colspan", dsr ? 1 : 2) 156 | append(th, "th", "Rate") 157 | append(th, "th", "Active") 158 | 159 | 160 | var row = function(v) { 161 | var tr = append(t, "tr", null, v.up ? "up" : "dn") 162 | append(tr, "td", ``+v.vip+``) 163 | append(tr, "td", dhms(v.for) + " " + (v.up ? "UP" : "DOWN")) 164 | append(tr, "td", tsf(v.stats.ingress_octets_per_second*8)+"bits/s in") 165 | if(!dsr) append(tr, "td", tsf(v.stats.egress_octets_per_second*8)+"bits/s out") 166 | append(tr, "td", tsf(v.stats.ingress_packets_per_second)+"packets/s in") 167 | if(!dsr) append(tr, "td", tsf(v.stats.egress_packets_per_second)+"packets/s out") 168 | append(tr, "td", tsf(v.stats.flows_per_second)+"conns/s") 169 | append(tr, "td", spc(v.stats.current), "ar") 170 | } 171 | 172 | for(var v of vips) { 173 | if(!v.up) row(v) 174 | } 175 | 176 | for(var v of vips) { 177 | if(v.up) row(v) 178 | } 179 | 180 | return t 181 | } 182 | 183 | function summary_t(s, bgp) { 184 | var div = document.createElement("div"); 185 | var t = append(div, "table") 186 | var hd = append(t, "tr", null, "hd") 187 | var tr = append(t, "tr", null, "up") 188 | 189 | if(s.vc5) { 190 | append(hd, "th", "Latency") 191 | append(tr, "td", s.latency_ns+"ns") 192 | } 193 | 194 | //append(hd, "th", "Dropped") 195 | //append(tr, "td", tsf(s.dropped_per_second)+"packets/s") 196 | 197 | //append(hd, "th", "Blocked") 198 | //append(tr, "td", tsf(s.blocked_per_second)+"packets/s") 199 | 200 | if(s.dsr) { 201 | append(hd, "th", "Traffic") 202 | append(tr, "td", tsf(s.ingress_octets_per_second*8)+"bits/s in") 203 | 204 | append(hd, "th", "Packets") 205 | append(tr, "td", tsf(s.ingress_packets_per_second)+"packets/s in") 206 | } else { 207 | append(hd, "th", "Traffic").setAttribute("colspan", 2) 208 | append(tr, "td", tsf(s.ingress_octets_per_second*8)+"bits/s in") 209 | append(tr, "td", tsf(s.egress_octets_per_second*8)+"bits/s out") 210 | 211 | append(hd, "th", "Packets").setAttribute("colspan", 2) 212 | append(tr, "td", tsf(s.ingress_packets_per_second)+"packets/s in") 213 | append(tr, "td", tsf(s.egress_packets_per_second)+"packets/s out") 214 | } 215 | 216 | 217 | append(hd, "th", "Connection rate") 218 | append(tr, "td", tsf(s.flows_per_second)+"conns/s") 219 | 220 | append(hd, "th", "Active connections") 221 | append(tr, "td", spc(s.current), "ar") 222 | 223 | var peers = Object.keys(bgp).sort() 224 | 225 | for(var peer of peers) { 226 | var conn = bgp[peer]; 227 | append(hd, "th", "BGP: " + peer) 228 | append(tr, "td", conn.state + " " + dhms(conn.duration_s), conn.state == "ESTABLISHED" ? "up" : "dn") 229 | } 230 | 231 | return div 232 | } 233 | 234 | function serv(v, _vip, list, up, dsr) { 235 | var vip = v.vip 236 | 237 | var div = document.createElement("div"); 238 | var t = append(div, "table") 239 | var tr = append(t, "tr", null, v.up ? "up" : "dn") 240 | 241 | append(tr, "th", v.vip, "ip") 242 | append(tr, "th", dhms(v.for) + " " + (v.up ? "UP" : "DOWN")) 243 | 244 | append(tr, "th", tsf(v.stats.ingress_octets_per_second*8)+"bits/s in") 245 | if(!dsr) append(tr, "th", tsf(v.stats.egress_octets_per_second*8)+"bits/s out") 246 | 247 | append(tr, "th", tsf(v.stats.ingress_packets_per_second)+"packets/s in") 248 | if(!dsr) append(tr, "th", tsf(v.stats.egress_packets_per_second)+"packets/s out") 249 | 250 | append(tr, "th", tsf(v.stats.flows_per_second)+"conns/s") 251 | append(tr, "th", spc(v.stats.current), "ar") 252 | 253 | append(div, "div", " ") 254 | 255 | t.setAttribute("id", vip) 256 | 257 | for(var s of list) { 258 | var t = append(div, "table") 259 | var tr = append(t, "tr", null, "hd") 260 | 261 | var title = esc(s.description) + " [" + s.available + "/" +s.destinations.length + " available - " + s.required + " required] " 262 | 263 | if(s.scheduler !== undefined) { 264 | title += s.scheduler 265 | } 266 | 267 | if(s.sticky !== undefined && s.sticky) { 268 | title += "(sticky)" 269 | } 270 | 271 | append(tr, "th", esc(s.name)) 272 | append(tr, "th", esc(title)).setAttribute("colspan", dsr ? 4 : 6) 273 | append(tr, "th", "Active") 274 | 275 | 276 | tr = append(t, "tr", null, s.available >= s.required ? "up" : "dn") 277 | append(tr, "th", s.address+":"+s.port+":"+s.protocol) 278 | append(tr, "th", dhms(s.for) + " " + (s.up ? "UP" : "DOWN")) 279 | append(tr, "th", tsf(s.stats.ingress_octets_per_second*8)+"bits/s in") 280 | if(!dsr) append(tr, "th", tsf(s.stats.egress_octets_per_second*8)+"bits/s out") 281 | append(tr, "th", tsf(s.stats.ingress_packets_per_second)+"packets/s in") 282 | if(!dsr) append(tr, "th", tsf(s.stats.egress_packets_per_second)+"packets/s out") 283 | append(tr, "th", tsf(s.stats.flows_per_second)+"conns/s") 284 | append(tr, "th", spc(s.stats.current), "ar") 285 | 286 | for(var d of s.destinations) { 287 | var c = d.disabled ? "ds" : d.up ? "up" : "dn" 288 | 289 | var address = document.createElement("span") 290 | address.setAttribute("title", d.mac) 291 | address.innerHTML = d.address+":"+d.port 292 | 293 | var status = document.createElement("span") 294 | status.setAttribute("title", "Last check: " + d.diagnostic) 295 | status.innerHTML = dhms(d.for) + " " + (d.up ? "UP" : "DOWN") + " ("+d.took+"ms)" 296 | 297 | var tr = append(t, "tr", null, c) 298 | append(tr, "td").appendChild(address) 299 | append(tr, "td").appendChild(status) 300 | append(tr, "td", spc(d.stats.ingress_octets_per_second*8), "ar") 301 | if(!dsr) append(tr, "td", spc(d.stats.egress_octets_per_second*8), "ar") 302 | append(tr, "td", spc(d.stats.ingress_packets_per_second), "ar") 303 | if(!dsr) append(tr, "td", spc(d.stats.egress_packets_per_second), "ar") 304 | append(tr, "td", spc(d.stats.flows_per_second), "ar") 305 | append(tr, "td", spc(d.stats.current), "ar") 306 | } 307 | 308 | append(div, "div", " ") 309 | } 310 | 311 | return div 312 | } 313 | 314 | 315 | 316 | var lastlog = 0; 317 | 318 | function lb() { 319 | 320 | //var url = window.location.href + 'stats.json'; 321 | var url = '/status.json.gz'; 322 | var log = '/log/'; 323 | //var log = window.location.href + 'log/'; 324 | 325 | console.log(url) 326 | function refresh() { 327 | updateStatus(url); 328 | updateLogs(log); 329 | setTimeout(refresh, 2000); 330 | } 331 | 332 | setTimeout(refresh, 100); 333 | } 334 | 335 | window.lb = lb; 336 | 337 | 338 | 339 | 340 | function addMessage(msg) { 341 | var consoleElement = document.querySelectorAll('#console')[0]; 342 | var messageElement = document.createElement('div'); 343 | messageElement.innerHTML = msg; 344 | if(consoleElement.childElementCount > 1000 ) { 345 | consoleElement.removeChild(consoleElement.lastChild); 346 | } 347 | consoleElement.insertBefore(messageElement, consoleElement.firstChild); 348 | } 349 | 350 | function updateLogs(url) { 351 | 352 | getJSON(url+lastlog, function(err, data) { 353 | if (err !== null) { 354 | //alert('Something went wrong: ' + err); 355 | } else { 356 | if(data !== null ) { 357 | data.forEach(function(item, index) { 358 | //console.log(index); 359 | //if(item.Level < 7) { 360 | lastlog = item.indx; 361 | var date = new Date(item.time*1000); 362 | var time = date.toLocaleString(); 363 | addMessage(time + ": " + item.text); 364 | //} 365 | }) 366 | } 367 | 368 | } 369 | }) 370 | } 371 | -------------------------------------------------------------------------------- /status.go: -------------------------------------------------------------------------------- 1 | /* 2 | * VC5 load balancer. Copyright (C) 2021-present David Coles 3 | * 4 | * This program is free software; you can redistribute it and/or modify 5 | * it under the terms of the GNU General Public License as published by 6 | * the Free Software Foundation; either version 2 of the License, or 7 | * (at your option) any later version. 8 | * 9 | * This program is distributed in the hope that it will be useful, 10 | * but WITHOUT ANY WARRANTY; without even the implied warranty of 11 | * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 12 | * GNU General Public License for more details. 13 | * 14 | * You should have received a copy of the GNU General Public License along 15 | * with this program; if not, write to the Free Software Foundation, Inc., 16 | * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 17 | */ 18 | 19 | package vc5 20 | 21 | import ( 22 | "fmt" 23 | "net/netip" 24 | "sort" 25 | "time" 26 | 27 | "github.com/davidcoles/cue" 28 | ) 29 | 30 | type servicemap map[netip.Addr][]serv 31 | 32 | type serv struct { 33 | Name string `json:"name,omitempty"` 34 | Description string `json:"description"` 35 | Address netip.Addr `json:"address"` 36 | Port uint16 `json:"port"` 37 | Protocol protocol `json:"protocol"` 38 | Required uint8 `json:"required"` 39 | Available uint8 `json:"available"` 40 | Stats Stats `json:"stats"` 41 | Destinations []dest `json:"destinations,omitempty"` 42 | Up bool `json:"up"` 43 | For uint64 `json:"for"` 44 | Last uint64 `json:"last"` 45 | Sticky bool `json:"sticky"` 46 | Scheduler string `json:"scheduler"` 47 | } 48 | 49 | type dest struct { 50 | Address netip.Addr `json:"address"` 51 | Port uint16 `json:"port"` 52 | Stats Stats `json:"stats"` 53 | Weight uint8 `json:"weight"` 54 | Disabled bool `json:"disabled"` 55 | Up bool `json:"up"` 56 | For uint64 `json:"for"` 57 | Took uint64 `json:"took"` 58 | When uint64 `json:"when"` 59 | Last uint64 `json:"last"` 60 | Diagnostic string `json:"diagnostic"` 61 | MAC string `json:"mac"` 62 | } 63 | 64 | type state struct { 65 | up bool 66 | time time.Time 67 | } 68 | 69 | func (s *state) Up() bool { return s.up } 70 | func (s *state) Time() time.Time { return s.time } 71 | 72 | type Stats struct { 73 | IngressOctets uint64 `json:"ingress_octets"` 74 | IngressPackets uint64 `json:"ingress_packets"` 75 | EgressOctets uint64 `json:"egress_octets"` 76 | EgressPackets uint64 `json:"egress_packets"` 77 | Flows uint64 `json:"flows"` 78 | Current uint32 `json:"current"` 79 | 80 | IngressOctetsPerSecond uint64 `json:"ingress_octets_per_second"` 81 | IngressPacketsPerSecond uint64 `json:"ingress_packets_per_second"` 82 | EgressOctetsPerSecond uint64 `json:"egress_octets_per_second"` 83 | EgressPacketsPerSecond uint64 `json:"egress_packets_per_second"` 84 | FlowsPerSecond uint64 `json:"flows_per_second"` 85 | time time.Time 86 | MAC string `json:"mac,omitempty"` 87 | } 88 | 89 | type Summary struct { 90 | Uptime uint64 `json:"uptime"` 91 | Latency uint64 `json:"latency_ns"` 92 | Current uint32 `json:"current"` 93 | 94 | Dropped uint64 `json:"dropped"` 95 | DroppedPerSecond uint64 `json:"dropped_per_second"` 96 | Blocked uint64 `json:"blocked"` 97 | BlockedPerSecond uint64 `json:"blocked_per_second"` 98 | NotQueued uint64 `json:"notqueued"` 99 | NotQueuedPerSecond uint64 `json:"notqueued_per_second"` 100 | TooBig uint64 `json:"toobig"` 101 | TooBigPerSecond uint64 `json:"toobig_per_second"` 102 | 103 | IngressOctets uint64 `json:"ingress_octets"` 104 | IngressOctetsPerSecond uint64 `json:"ingress_octets_per_second"` 105 | IngressPackets uint64 `json:"ingress_packets"` 106 | IngressPacketsPerSecond uint64 `json:"ingress_packets_per_second"` 107 | Flows uint64 `json:"flows"` 108 | FlowsPerSecond uint64 `json:"flows_per_second"` 109 | EgressOctets uint64 `json:"egress_octets"` 110 | EgressOctetsPerSecond uint64 `json:"egress_octets_per_second"` 111 | EgressPackets uint64 `json:"egress_packets"` 112 | EgressPacketsPerSecond uint64 `json:"egress_packets_per_second"` 113 | 114 | DSR bool `json:"dsr"` 115 | VC5 bool `json:"vc5"` 116 | 117 | time time.Time 118 | } 119 | 120 | func (s *Stats) add(x Stats) { 121 | s.IngressOctets += x.IngressOctets 122 | s.IngressPackets += x.IngressPackets 123 | s.Flows += x.Flows 124 | s.Current += x.Current 125 | s.IngressOctetsPerSecond += x.IngressOctetsPerSecond 126 | s.IngressPacketsPerSecond += x.IngressPacketsPerSecond 127 | s.FlowsPerSecond += x.FlowsPerSecond 128 | 129 | s.EgressOctets += x.EgressOctets 130 | s.EgressPackets += x.EgressPackets 131 | s.EgressOctetsPerSecond += x.EgressOctetsPerSecond 132 | s.EgressPacketsPerSecond += x.EgressPacketsPerSecond 133 | 134 | } 135 | 136 | type vipstats struct { 137 | VIP netip.Addr `json:"vip"` 138 | Up bool `json:"up"` 139 | Stats Stats `json:"stats"` 140 | For uint64 `json:"for"` 141 | } 142 | 143 | func vipStatus(in servicemap, state map[netip.Addr]state) (out []vipstats) { 144 | 145 | for vip, list := range in { 146 | var stats Stats 147 | for _, s := range list { 148 | stats.add(s.Stats) 149 | } 150 | 151 | r, ok := state[vip] 152 | if !ok { 153 | r.time = time.Now() 154 | } 155 | 156 | out = append(out, vipstats{VIP: vip, Stats: stats, Up: r.up, For: uint64(time.Now().Sub(r.time) / time.Second)}) 157 | } 158 | 159 | sort.SliceStable(out, func(i, j int) bool { 160 | return out[i].VIP.Compare(out[j].VIP) < 0 161 | }) 162 | 163 | return 164 | } 165 | 166 | func vipState(services []cue.Service, old map[netip.Addr]state, priorities map[netip.Addr]priority, logs Logger, mature bool) map[netip.Addr]state { 167 | F := "vip" 168 | 169 | rib := map[netip.Addr]bool{} 170 | new := map[netip.Addr]state{} 171 | 172 | for _, v := range cue.HealthyVIPs(services) { 173 | rib[v] = true 174 | } 175 | 176 | for _, v := range cue.AllVIPs(services) { 177 | 178 | up, _ := rib[v] 179 | severity := priorityToSeverity(priorities[v]) 180 | 181 | if o, ok := old[v]; ok { 182 | 183 | if o.up != up { 184 | new[v] = state{up: up, time: time.Now()} 185 | text := fmt.Sprintf("VIP %s went %s", v, upDown(up)) 186 | if mature { 187 | logs.Alert(severity, F, "state", KV{"service.ip": v, "service.state": upDown(up)}, text) 188 | } 189 | } else { 190 | new[v] = o 191 | } 192 | 193 | } else { 194 | new[v] = state{up: rib[v], time: time.Now()} 195 | if mature { 196 | logs.Event(DEBUG, F, "added", KV{"service.ip": v, "service.state": upDown(up)}) 197 | } 198 | } 199 | 200 | if mature { 201 | logs.State(F, "state", KV{"service.ip": v, "service.state": upDown(up)}) 202 | } 203 | } 204 | 205 | for vip, _ := range old { 206 | if _, exists := new[vip]; !exists { 207 | if mature { 208 | logs.Event(DEBUG, F, "removed", KV{"service.ip": vip}) 209 | } 210 | } 211 | } 212 | 213 | return new 214 | } 215 | 216 | func upDown(b bool) string { 217 | if b { 218 | return "up" 219 | } 220 | return "down" 221 | } 222 | 223 | func priorityToSeverity(p priority) uint8 { 224 | switch p { 225 | case CRITICAL: 226 | return ERR 227 | case HIGH: 228 | return WARNING 229 | case MEDIUM: 230 | return NOTICE 231 | case LOW: 232 | return INFO 233 | } 234 | return ERR 235 | } 236 | 237 | func (s *Summary) Update(n Summary, start time.Time) { 238 | 239 | o := *s 240 | *s = n 241 | 242 | s.Uptime = uint64(time.Now().Sub(start) / time.Second) 243 | s.time = time.Now() 244 | 245 | if o.time.Unix() != 0 { 246 | diff := uint64(s.time.Sub(o.time) / time.Millisecond) 247 | 248 | if diff != 0 { 249 | s.DroppedPerSecond = (1000 * (s.Dropped - o.Dropped)) / diff 250 | s.BlockedPerSecond = (1000 * (s.Blocked - o.Blocked)) / diff 251 | s.NotQueuedPerSecond = (1000 * (s.NotQueued - o.NotQueued)) / diff 252 | s.TooBigPerSecond = (1000 * (s.TooBig - o.TooBig)) / diff 253 | 254 | s.IngressPacketsPerSecond = (1000 * (s.IngressPackets - o.IngressPackets)) / diff 255 | s.IngressOctetsPerSecond = (1000 * (s.IngressOctets - o.IngressOctets)) / diff 256 | s.EgressPacketsPerSecond = (1000 * (s.EgressPackets - o.EgressPackets)) / diff 257 | s.EgressOctetsPerSecond = (1000 * (s.EgressOctets - o.EgressOctets)) / diff 258 | s.FlowsPerSecond = (1000 * (s.Flows - o.Flows)) / diff 259 | } 260 | } 261 | } 262 | 263 | type Instance struct { 264 | Service Service 265 | Destination Destination 266 | } 267 | 268 | type _Manifest cue.Service 269 | 270 | type Manifest struct { 271 | Address netip.Addr 272 | Port uint16 273 | Protocol Protocol 274 | Destinations []cue.Destination 275 | 276 | Sticky bool 277 | Scheduler string 278 | Persist uint32 279 | Reset bool 280 | //Required uint8 281 | //available uint8 282 | //Up bool 283 | //When time.Time 284 | TunnelType string 285 | TunnelPort uint16 286 | TunnelEncapNoChecksum bool 287 | } 288 | 289 | func toManifest(s cue.Service, d ServiceDefinition) (m Manifest) { 290 | m.Address = s.Address 291 | m.Port = s.Port 292 | m.Protocol = Protocol(s.Protocol) 293 | m.Destinations = s.Destinations 294 | 295 | m.Sticky = s.Sticky 296 | m.Scheduler = d.Scheduler 297 | m.Persist = d.Persist 298 | m.Reset = d.Reset 299 | //m.Required = s.Required 300 | //m.Up = s.Up 301 | //m.When = s.When 302 | 303 | m.TunnelType = d.TunnelType 304 | m.TunnelPort = d.TunnelPort 305 | m.TunnelEncapNoChecksum = d.TunnelEncapNoChecksum 306 | return 307 | } 308 | 309 | func (s _Manifest) Service() Service { return s.Instance().Service } 310 | func (s _Manifest) Instance() Instance { return instance(cue.Service(s), cue.Destination{}) } 311 | 312 | func (s Manifest) Service() Service { 313 | return Service{Address: s.Address, Port: s.Port, Protocol: Protocol(s.Protocol)} 314 | } 315 | 316 | type Balancer interface { 317 | Stats() (Summary, map[Instance]Stats) 318 | Configure([]Manifest) error 319 | Metrics() ([]string, []string) 320 | } 321 | 322 | func calculateRate(s Stats, o Stats) Stats { 323 | 324 | s.time = time.Now() 325 | 326 | if o.time.Unix() != 0 { 327 | diff := uint64(s.time.Sub(o.time) / time.Millisecond) 328 | 329 | if diff != 0 { 330 | s.EgressPacketsPerSecond = (1000 * (s.EgressPackets - o.EgressPackets)) / diff 331 | s.EgressOctetsPerSecond = (1000 * (s.EgressOctets - o.EgressOctets)) / diff 332 | s.IngressPacketsPerSecond = (1000 * (s.IngressPackets - o.IngressPackets)) / diff 333 | s.IngressOctetsPerSecond = (1000 * (s.IngressOctets - o.IngressOctets)) / diff 334 | s.FlowsPerSecond = (1000 * (s.Flows - o.Flows)) / diff 335 | } 336 | } 337 | 338 | return s 339 | } 340 | 341 | func serviceStatus(config *Config, balancer Balancer, director *cue.Director, old map[Instance]Stats) (Summary, servicemap, map[Instance]Stats) { 342 | 343 | var current uint32 344 | 345 | stats := map[Instance]Stats{} 346 | status := map[netip.Addr][]serv{} 347 | summary, allstats := balancer.Stats() 348 | //summary := balancer.Summary() 349 | 350 | for _, svc := range director.Status() { 351 | cnf, _ := config.Services[_Manifest(svc).Service()] 352 | //key := serviceInstance(svc) 353 | key := _Manifest(svc).Instance() 354 | lbs := map[Destination]Stats{} 355 | 356 | for k, v := range allstats { 357 | if k.Service == key.Service { 358 | lbs[k.Destination] = v 359 | } 360 | } 361 | 362 | var sum Stats 363 | for _, s := range lbs { 364 | sum.add(s) 365 | } 366 | 367 | serv := serv{ 368 | Name: cnf.Name, 369 | Description: cnf.Description, 370 | Address: svc.Address, 371 | Port: svc.Port, 372 | Protocol: protocol(svc.Protocol), 373 | Required: svc.Required, 374 | Available: svc.Available(), 375 | Up: svc.Up, 376 | For: uint64(time.Now().Sub(svc.When) / time.Second), 377 | Sticky: svc.Sticky, 378 | Scheduler: svc.Scheduler, 379 | Stats: calculateRate(sum, old[key]), 380 | } 381 | 382 | for _, dst := range svc.Destinations { 383 | foo := lbs[destination(dst)] 384 | 385 | key := instance(svc, dst) 386 | dest := dest{ 387 | Address: dst.Address, 388 | Port: dst.Port, 389 | Disabled: dst.Disabled, 390 | Up: dst.Status.OK, 391 | For: uint64(time.Now().Sub(dst.Status.When) / time.Second), 392 | Took: uint64(dst.Status.Took / time.Millisecond), 393 | Diagnostic: dst.Status.Diagnostic, 394 | Weight: dst.Weight, 395 | Stats: calculateRate(lbs[destination(dst)], old[key]), 396 | MAC: lbs[destination(dst)].MAC, 397 | } 398 | 399 | current += foo.Current 400 | stats[key] = dest.Stats 401 | serv.Destinations = append(serv.Destinations, dest) 402 | } 403 | 404 | stats[key] = serv.Stats 405 | 406 | sort.SliceStable(serv.Destinations, func(i, j int) bool { 407 | return serv.Destinations[i].Address.Compare(serv.Destinations[j].Address) < 0 408 | }) 409 | 410 | status[svc.Address] = append(status[svc.Address], serv) 411 | } 412 | 413 | summary.Current = current 414 | 415 | return summary, status, stats 416 | } 417 | 418 | func destination(d cue.Destination) Destination { return Destination{Address: d.Address, Port: d.Port} } 419 | func instance(s cue.Service, d cue.Destination) (i Instance) { 420 | i.Service = Service{Address: s.Address, Port: s.Port, Protocol: Protocol(s.Protocol)} 421 | i.Destination = Destination{Address: d.Address, Port: d.Port} 422 | return 423 | } 424 | 425 | //func manifests(c []cue.Service) (m []Manifest) { 426 | // for _, s := range x { 427 | // m = append(m, s) 428 | // } 429 | // return 430 | //} 431 | --------------------------------------------------------------------------------