CSLB - Client Side Load Balancing

Start Time	{{.StartTime.Format "2006-01-02T15:04:05Z07:00"}}
Up time	{{.Uptime}}
DialContext Intercepts	{{.DialContext}}
Time In Intercept	{{.Duration}}
Status Server Address	http://{{.StatusServerAddress}}
Executable	{{.Executable}}

Start Time

Up time

DialContext Intercepts

Time In Intercept

Status Server Address

http://{{.StatusServerAddress}}

Executable

PrintDialContext	Print entry into cslb.DialContext	{{.PrintDialContext}}
PrintHCResults	Print results of Health Check	{{.PrintHCResults}}
PrintIntercepts	Print each domain to Target intercept	{{.PrintIntercepts}}
PrintSRVLookup	Print results of SRV Lookups	{{.PrintSRVLookup}}
DisableInterception	Turn off Interception	{{.DisableInterception}}
DisableHealthChecks	Turn off Health Checks	{{.DisableHealthChecks}}
AllowNumericServices	Allow Numeric Service SRV lookups	{{.AllowNumericServices}}
HealthCheckTXTPrefix	Forms part of TXT qName	{{.HealthCheckTXTPrefix}}
HealthCheckContentOk	strings.Contains in health check body	"{{.HealthCheckContentOk}}"
HealthCheckFrequency	Time between health checks	{{.HealthCheckFrequency}}
InterceptTimeout	Maximum time to try targets	{{.InterceptTimeout}}
DialVetoDuration	Ignore downed targets for this duration	{{.DialVetoDuration}}
NotFoundSRVTTL	Cache lifetime for SRV NXDomain	{{.NotFoundSRVTTL}}
FoundSRVTTL	Cache lifetime for SRV found	{{.FoundSRVTTL}}
HealthTTL	Cache lifetime for SRV Target	{{.HealthTTL}}

PrintDialContext

Print entry into cslb.DialContext

PrintHCResults

Print results of Health Check

PrintIntercepts

Print each domain to Target intercept

PrintSRVLookup

Print results of SRV Lookups

DisableInterception

Turn off Interception

DisableHealthChecks

Turn off Health Checks

AllowNumericServices

Allow Numeric Service SRV lookups

HealthCheckTXTPrefix

Forms part of TXT qName

HealthCheckContentOk

strings.Contains in health check body

"{{.HealthCheckContentOk}}"

HealthCheckFrequency

Time between health checks

InterceptTimeout

Maximum time to try targets

DialVetoDuration

Ignore downed targets for this duration

NotFoundSRVTTL

Cache lifetime for SRV NXDomain

FoundSRVTTL

Cache lifetime for SRV found

HealthTTL

Cache lifetime for SRV Target

Intercepted calls to DialContext	{{.DialContext}}
Host or service don't match or interception disabled	{{.MissHostService}}
Times SRV lookup returned zero targets	{{.NoSRV}}
Calls to bestTarget()	{{.BestTarget}}
Times when all targets failed	{{.DupesStopped}}
system DialContext returned a good connection	{{.GoodDials}}
system DialContext returned an error	{{.FailedDials}}
Times intercept deadline expired	{{.Deadline}}

Intercepted calls to DialContext

Host or service don't match or interception disabled

Times SRV lookup returned zero targets

Calls to bestTarget()

Times when all targets failed

system DialContext returned a good connection

system DialContext returned an error

Times intercept deadline expired

CName	Expires	Lookups	Priority	Internal Weight	Port	Target	Good Dials	Failed Dials	IsGood
{{.CName}}	{{.Expires}}	{{.Lookups}}	{{.Priority}}	{{.Weight}}	{{.Port}}	{{.Target}}	{{.GoodDials}}	{{.FailedDials}}	{{.IsGood}}

CName

Expires

Lookups

Priority

Internal Weight

Port

Target

Good Dials

Failed Dials

IsGood

Target	Expires	Good Dials	Failed Dials	Next Dial Attempt	Last Dial Attempt	isGood	Last Dial Status	Last Health Check	Health Check URL	Last Health Status
{{.Key}}	{{.Expires}}	{{.GoodDials}}	{{.FailedDials}}	{{.NextDialAttempt}}	{{.LastDialAttempt}}	{{.IsGood}}	{{.LastDialStatus}}	{{.LastHealthCheck}}	{{.Url}}	{{.LastHealthCheckStatus}}

Target

Expires

Good Dials

Failed Dials

Next Dial
Attempt

Last Dial
Attempt

isGood

Last Dial
Status

Last Health
Check

Health Check URL

Last Health
Status

Client-Side Load Balancing {{.Version}} released on {{.ReleaseDate}}. Brought to you by 115 | https://github/markdingo/cslb at {{.RunAt}} 116 | 117 | ` 118 | ) 119 | 120 | type statusServer struct { 121 | cslb *cslb 122 | httpServer *http.Server 123 | allTmpl *template.Template 124 | trailerTmpl *template.Template 125 | } 126 | 127 | // newStatusServer creates the base status server ready for starting 128 | func newStatusServer(cslb *cslb) *statusServer { 129 | t := &statusServer{cslb: cslb} 130 | err := t.loadTemplates() 131 | if err != nil { 132 | log.Fatal(err) 133 | } 134 | t.httpServer = &http.Server{Addr: cslb.StatusServerAddress} 135 | mux := http.NewServeMux() 136 | mux.HandleFunc("/", t.generateStatus) 137 | t.httpServer.Handler = mux 138 | 139 | return t 140 | } 141 | 142 | // start is normally called as a separate go-routine since it calls the http listener which blocks. 143 | func (t *statusServer) start() { 144 | err := t.httpServer.ListenAndServe() 145 | if !strings.Contains(err.Error(), "http: Server closed") { // Good return? 146 | log.Fatal(err) 147 | } 148 | } 149 | 150 | // stop shuts down the http listener 151 | func (t *statusServer) stop(ctx context.Context) { 152 | t.httpServer.Shutdown(ctx) 153 | } 154 | 155 | // loadTemplates performs a one-time parse of all the internal templates needed for the status 156 | // page. It also attempts to "glob" load any template files found in the directory identified by the 157 | // "cslb_templates" environment variable. If the glob load fails it only causes a warning as the 158 | // default templates will still function.. 159 | func (t *statusServer) loadTemplates() error { 160 | t.allTmpl = template.New("") 161 | _, err := t.allTmpl.Parse(configStr) 162 | if err != nil { 163 | return err 164 | } 165 | _, err = t.allTmpl.Parse(cslbStr) 166 | if err != nil { 167 | return err 168 | } 169 | _, err = t.allTmpl.Parse(srvStr) 170 | if err != nil { 171 | return err 172 | } 173 | _, err = t.allTmpl.Parse(healthStr) 174 | if err != nil { 175 | return err 176 | } 177 | t.trailerTmpl, err = template.New("trailer").Parse(trailerStr) 178 | if err != nil { 179 | return err 180 | } 181 | 182 | if len(t.cslb.StatusServerTemplates) > 0 { // If an alternate template glob has been configured 183 | _, err = t.allTmpl.ParseGlob(t.cslb.StatusServerTemplates) 184 | if err != nil { 185 | log.Print("cslb Warning:", err) // Not fatal if replacement templates fail to load 186 | } 187 | } 188 | 189 | return nil 190 | } 191 | 192 | // Aggregate structs are conveniences so we can render derived values in a single template. 193 | 194 | type cslbAggConfig struct { 195 | StartTime time.Time 196 | Uptime time.Duration 197 | Duration time.Duration 198 | DialContext int 199 | Executable string 200 | config 201 | } 202 | 203 | type cslbAggTrailer struct { 204 | Version string 205 | ReleaseDate string 206 | RunAt string 207 | } 208 | 209 | // generateStatus writes the status page out. It's quite extensive because everything is on one page. 210 | func (t *statusServer) generateStatus(w http.ResponseWriter, req *http.Request) { 211 | var err error 212 | io.WriteString(w, header) 213 | 214 | var cac cslbAggConfig 215 | cac.config = t.cslb.config 216 | cas := t.cslb.cloneStats() // Take a copy to avoid holding a long mutex 217 | 218 | cac.StartTime = cas.StartTime 219 | cac.Uptime = time.Now().Sub(cas.StartTime).Truncate(time.Second) 220 | cac.Duration = cas.Duration.Truncate(time.Second) // Total time in intercepts 221 | cac.DialContext = cas.DialContext 222 | cac.Executable, _ = os.Executable() 223 | 224 | err = t.allTmpl.ExecuteTemplate(w, "config", &cac) 225 | if err != nil { 226 | log.Fatal(err) 227 | } 228 | err = t.allTmpl.ExecuteTemplate(w, "cslb", &cas) 229 | if err != nil { 230 | log.Fatal(err) 231 | } 232 | 233 | srvStats := t.cslb.srvStore.getStats(t.cslb.healthStore) // Clone all ceSRVs and ancillary data 234 | sort.Slice(srvStats.Srvs, func(i, j int) bool { // Sort for a low-flicker re-render 235 | return srvStats.Srvs[i].CName < srvStats.Srvs[j].CName 236 | }) 237 | sort.Slice(srvStats.nxDomains, func(i, j int) bool { // Sort for a low-flicker re-render 238 | return srvStats.nxDomains[i].CName < srvStats.nxDomains[j].CName 239 | }) 240 | 241 | // Place NXDomains at the end to reduce visual clutter. To further reduce clutter, remove 242 | // duplicate data which comes from the SRV cache. 243 | srvStats.Srvs = append(srvStats.Srvs, srvStats.nxDomains...) 244 | prevCName := "" 245 | for ix := 0; ix < len(srvStats.Srvs); ix++ { 246 | if prevCName == srvStats.Srvs[ix].CName { 247 | srvStats.Srvs[ix].CName = "" 248 | srvStats.Srvs[ix].Expires = "" 249 | srvStats.Srvs[ix].Lookups = "" 250 | } else { 251 | prevCName = srvStats.Srvs[ix].CName 252 | } 253 | } 254 | 255 | err = t.allTmpl.ExecuteTemplate(w, "srv", srvStats) 256 | if err != nil { 257 | log.Fatal(err) 258 | } 259 | 260 | healthStats := t.cslb.healthStore.getStats() // Clone all ceHealth entries 261 | sort.Slice(healthStats.Targets, func(i, j int) bool { // Sort for a low-flicker re-render 262 | return healthStats.Targets[i].Key < healthStats.Targets[j].Key 263 | }) 264 | err = t.allTmpl.ExecuteTemplate(w, "health", healthStats) 265 | if err != nil { 266 | log.Fatal(err) 267 | } 268 | 269 | tv := cslbAggTrailer{Version: Version, ReleaseDate: ReleaseDate, 270 | RunAt: time.Now().Format("2006-01-02T15:04:05Z07:00")} 271 | err = t.trailerTmpl.Execute(w, tv) 272 | if err != nil { 273 | log.Fatal(err) 274 | } 275 | } 276 | -------------------------------------------------------------------------------- /srv_test.go: -------------------------------------------------------------------------------- 1 | package cslb 2 | 3 | import ( 4 | "context" 5 | "fmt" 6 | "net" 7 | "strings" 8 | "sync" 9 | "testing" 10 | "time" 11 | ) 12 | 13 | type mockResolver struct { 14 | mu sync.Mutex // So go test -race doesn't complain 15 | srvs map[string][]*net.SRV 16 | txts map[string][]string 17 | lastSRV string 18 | lastTXT string 19 | } 20 | 21 | func newMockResolver() *mockResolver { 22 | return &mockResolver{srvs: make(map[string][]*net.SRV), txts: make(map[string][]string)} 23 | } 24 | 25 | // append the target to the srv. Last append is always at the end to tests can rely on position. 26 | func (t *mockResolver) appendSRV(service, proto, name string, target string, port, priority, weight int) { 27 | t.mu.Lock() 28 | defer t.mu.Unlock() 29 | 30 | cname := "" 31 | if len(service) != 0 || len(proto) != 0 { 32 | cname = "_" + service + "._" + proto + "." 33 | } 34 | cname += name 35 | ar, ok := t.srvs[cname] 36 | if !ok { 37 | ar = make([]*net.SRV, 0) 38 | } 39 | if len(target) > 0 { 40 | ar = append(ar, &net.SRV{Target: target, Port: uint16(port), 41 | Priority: uint16(priority), Weight: uint16(weight)}) 42 | t.srvs[cname] = ar 43 | } 44 | } 45 | 46 | func (t *mockResolver) LookupSRV(ctx context.Context, service, proto, name string) (cname string, srvs []*net.SRV, err error) { 47 | t.mu.Lock() 48 | defer t.mu.Unlock() 49 | 50 | qName := "" 51 | if len(service) != 0 || len(proto) != 0 { 52 | qName = "_" + service + "._" + proto + "." 53 | } 54 | qName += name 55 | t.lastSRV = qName 56 | srvs, ok := t.srvs[qName] 57 | if !ok { 58 | err = fmt.Errorf("mock LookupSRV not found for %s", qName) 59 | return 60 | } 61 | cname = qName 62 | 63 | return 64 | } 65 | 66 | func (t *mockResolver) appendTXT(target string, txts []string) { 67 | t.mu.Lock() 68 | defer t.mu.Unlock() 69 | 70 | t.txts[target] = txts 71 | } 72 | 73 | func (t *mockResolver) LookupTXT(ctx context.Context, qName string) (txts []string, err error) { 74 | t.mu.Lock() 75 | defer t.mu.Unlock() 76 | 77 | txts, ok := t.txts[qName] 78 | t.lastTXT = qName 79 | if !ok { 80 | err = fmt.Errorf("mock LookupTXT not found for %s", qName) 81 | } 82 | 83 | return 84 | } 85 | 86 | func makeMockResolver() *mockResolver { 87 | mr := newMockResolver() 88 | mr.appendSRV("http", "tcp", "example.net", "t1.example.net", 1, 10, 20) // port used to codify bestTarget() order 89 | mr.appendSRV("http", "tcp", "example.net", "t2.example.net", 1, 10, 20) 90 | mr.appendSRV("http", "tcp", "example.net", "t3.example.net", 1, 10, 30) 91 | mr.appendSRV("http", "tcp", "example.net", "t4.example.net", 1, 10, 40) 92 | mr.appendSRV("http", "tcp", "example.net", "t5.example.net", 2, 11, 1) 93 | mr.appendSRV("http", "tcp", "example.net", "t6.example.net", 2, 11, 1) 94 | mr.appendSRV("http", "tcp", "example.net", "t7.example.net", 2, 11, 2) 95 | mr.appendSRV("http", "tcp", "example.net", "t8.example.net", 2, 11, 10) 96 | mr.appendSRV("http", "tcp", "example.net", "t9.example.net", 3, 12, 20) 97 | mr.appendSRV("http", "tcp", "example.net", "t10.example.net", 3, 12, 30) 98 | mr.appendSRV("http", "tcp", "example.net", "t11.example.net", 3, 12, 40) 99 | mr.appendSRV("http", "tcp", "example.net", "t12.example.net", 3, 12, 50) 100 | mr.appendSRV("http", "tcp", "example.net", "t13.example.net", 3, 12, 60) 101 | mr.appendSRV("http", "tcp", "example.net", "t14.example.net", 3, 12, 70) 102 | mr.appendSRV("http", "tcp", "example.net", "t15.example.net", 3, 12, 80) 103 | mr.appendSRV("http", "tcp", "example.net", "t16.example.net", 4, 13, 90) 104 | mr.appendSRV("http", "tcp", "example.net", "t17.example.net", 4, 13, 91) 105 | mr.appendSRV("http", "tcp", "example.net", "t18.example.net", 4, 13, 92) 106 | mr.appendSRV("http", "tcp", "example.net", "t19.example.net", 4, 13, 93) 107 | mr.appendSRV("http", "tcp", "example.net", "t20.example.net", 4, 13, 94) 108 | 109 | mr.appendSRV("https", "udp", "example.com", "u1.example.com", 1443, 13, 10) 110 | mr.appendSRV("https", "udp", "example.com", "u2.example.com", 1444, 13, 20) 111 | mr.appendSRV("https", "udp", "example.com", "u3.example.com", 1444, 13, 30) 112 | mr.appendSRV("https", "udp", "example.com", "u4.example.com", 1444, 13, 0) 113 | mr.appendSRV("https", "udp", "example.com", "u5.example.com", 1444, 13, 0) 114 | mr.appendSRV("https", "udp", "example.com", "u6.example.com", 1444, 13, 0) 115 | mr.appendSRV("https", "udp", "example.com", "", 1444, 13, 100) // Should disappear completely 116 | mr.appendSRV("https", "udp", "example.com", "u7.example.com", 1444, 14, 0) 117 | 118 | mr.appendSRV("http", "tcp", "empty.example.org", "", 0, 0, 0) 119 | 120 | return mr 121 | } 122 | 123 | type srvTestCase struct { 124 | service, proto, domain string 125 | srvCount int 126 | target string 127 | bestCount int // How many calls to bestTarget before target expected to show up 128 | never string // target which should never be returned 129 | } 130 | 131 | var srvTestCases = []srvTestCase{ 132 | {"http", "tcp", "example.net", 20, "t1.example.net", 100, "t20.example.net"}, 133 | {"https", "udp", "example.com", 7, "u4.example.com", 6500, "u7.example.com"}, // 0.1% / 3 is the weight 134 | } 135 | 136 | func TestSRVPopulate(t *testing.T) { 137 | cslb := realInit() 138 | cslb.netResolver = makeMockResolver() 139 | cslb.start() 140 | 141 | for _, tc := range srvTestCases { 142 | t.Run(tc.service+"_"+tc.proto+"_"+tc.domain, func(t *testing.T) { 143 | cesrv := cslb.lookupSRV(context.Background(), time.Now(), tc.service, tc.proto, tc.domain) 144 | if cesrv.uniqueTargets() != tc.srvCount { 145 | t.Error("SRV Count mismatch. Expected", tc.srvCount, "got", cesrv.uniqueTargets()) 146 | } 147 | distrib := make(map[string]int) 148 | for ix := 0; ix < tc.bestCount; ix++ { 149 | srv := cslb.bestTarget(cesrv) 150 | if srv == nil { 151 | t.Fatal("bestTarget() returned nil for", tc.domain) 152 | } 153 | distrib[srv.Target]++ 154 | } 155 | if distrib[tc.target] == 0 { 156 | t.Error("Expected", tc.target, "to have been bestTarget() at least once") 157 | t.Log(cesrv) 158 | for k, v := range distrib { 159 | t.Log(k, v) 160 | } 161 | } 162 | if distrib[tc.never] > 0 { 163 | t.Error("Never expected", tc.never, "to be returned as best, but", distrib[tc.never]) 164 | t.Log(cesrv) 165 | for k, v := range distrib { 166 | t.Log(k, v) 167 | } 168 | } 169 | }) 170 | } 171 | } 172 | 173 | // Test that bestTarget() distributes targets according to their weights. At least roughly 174 | // proportionally within the limits of the PRNG. example.com SRV has u1=10, u2=20, u3=30 and u4-u7=0 175 | // thus u3 > u2 > u1 > (u4-u7). 176 | func TestSRVWeightDistribution(t *testing.T) { 177 | cslb := realInit() 178 | cslb.netResolver = makeMockResolver() 179 | cslb.start() 180 | 181 | cesrv := cslb.lookupSRV(context.Background(), time.Now(), "https", "udp", "example.com") 182 | distrib := make(map[string]int) 183 | for ix := 0; ix < 1000; ix++ { 184 | srv := cslb.bestTarget(cesrv) 185 | distrib[srv.Target]++ 186 | } 187 | u1 := distrib["u1.example.com"] 188 | u2 := distrib["u2.example.com"] 189 | u3 := distrib["u3.example.com"] 190 | u4 := distrib["u4.example.com"] 191 | u5 := distrib["u5.example.com"] 192 | u6 := distrib["u6.example.com"] 193 | u7 := distrib["u7.example.com"] 194 | if !(u3 > u2) { 195 | t.Error("Expected u3 GT u2", u3, u2) 196 | } 197 | if !(u2 > u1) { 198 | t.Error("Expected u2 GT u1", u2, u1) 199 | } 200 | if u4 > u1 || u5 > u1 || u6 > u1 || u7 > u1 { 201 | t.Error("Expected u1 GT U4-u7", u1, u4, u5, u6, u7) 202 | } 203 | } 204 | 205 | // Test that failed targets are not considered by lookupSRV. 206 | func TestSRVHealth(t *testing.T) { 207 | cslb := realInit() 208 | cslb.netResolver = makeMockResolver() 209 | cslb.randIntn = func(int) int { return 0 } 210 | cslb.start() 211 | 212 | cesrv := cslb.lookupSRV(context.Background(), time.Now(), "https", "udp", "example.com") 213 | srv := cslb.bestTarget(cesrv) // nextRand is zero so first weight should win 214 | if srv == nil { 215 | t.Fatal("Setup error") 216 | } 217 | if srv.Target != "u1.example.com" { 218 | t.Error("bestTarget should be u1, not", srv) 219 | } 220 | fakeNow := time.Now().Add(time.Minute) // Put it sufficiently in the future 221 | nowPlusOne := fakeNow.Add(time.Hour) 222 | cslb.setDialResult(nowPlusOne, "u1.example.com", int(srv.Port), fmt.Errorf("")) 223 | srv = cslb.bestTarget(cesrv) // nextRand is zero but first is down 224 | if srv == nil { 225 | t.Fatal("Setup error") 226 | } 227 | if srv.Target != "u2.example.com" { 228 | t.Error("bestTarget should now be u2, not", srv) 229 | } 230 | cslb.setDialResult(nowPlusOne, "u2.example.com", int(srv.Port), fmt.Errorf("")) 231 | srv = cslb.bestTarget(cesrv) 232 | if srv == nil { 233 | t.Fatal("Setup error") 234 | } 235 | if srv.Target != "u3.example.com" { 236 | t.Error("bestTarget should now be u3, not", srv) 237 | } 238 | 239 | cslb.setDialResult(nowPlusOne, "u3.example.com", int(srv.Port), fmt.Errorf("")) // Last of the highest priority targets 240 | srv = cslb.bestTarget(cesrv) 241 | if srv == nil { 242 | t.Fatal("Setup error") 243 | } 244 | if srv.Target != "u4.example.com" { // Should get next priority down as second choice 245 | t.Error("bestTarget should now be u4, not", srv) 246 | } 247 | 248 | cslb.setDialResult(nowPlusOne, "u4.example.com", int(srv.Port), fmt.Errorf("")) 249 | cslb.setDialResult(fakeNow, "u5.example.com", 1444, fmt.Errorf("")) // Closest to now 250 | cslb.setDialResult(nowPlusOne, "u6.example.com", 1444, fmt.Errorf("")) 251 | 252 | srv = cslb.bestTarget(cesrv) 253 | if srv == nil { 254 | t.Fatal("Setup error") 255 | } 256 | if srv.Target != "u7.example.com" { // Last second choice 257 | t.Error("bestTarget should now be u7, not", srv) 258 | } 259 | cslb.setDialResult(nowPlusOne, "u7.example.com", int(srv.Port), fmt.Errorf("")) // Every Target is now in bad health 260 | 261 | srv = cslb.bestTarget(cesrv) // Should now get least-worst 262 | if srv == nil { 263 | t.Fatal("Setup error") 264 | } 265 | if srv.Target != "u5.example.com" { // Closet to now 266 | t.Error("bestTarget should now be u5, not", srv) 267 | } 268 | } 269 | 270 | func TestSRVNoFind(t *testing.T) { 271 | cslb := realInit() 272 | mr := makeMockResolver() 273 | cslb.netResolver = mr 274 | cslb.start() 275 | 276 | cesrv := cslb.lookupSRV(context.Background(), time.Now(), "http", "tcp", "empty.example.org") 277 | srv := cslb.bestTarget(cesrv) // nextRand is zero so first weight should win 278 | if srv != nil { 279 | t.Error("Should have got nil for bestTarget as SRV is empty. Got", srv) 280 | } 281 | 282 | // Test Cache hit while we're here 283 | 284 | mr.appendSRV("http", "tcp", "empty.example.org", "e1.example.org", 0, 0, 0) // Now in DNS 285 | cesrv = cslb.lookupSRV(context.Background(), time.Now(), "http", "tcp", "empty.example.org") 286 | srv = cslb.bestTarget(cesrv) // nextRand is zero so first weight should win 287 | if srv != nil { 288 | t.Error("Should have still got nil for bestTarget as SRV is in cache. Got", srv) 289 | } 290 | 291 | } 292 | 293 | func TestSRVString(t *testing.T) { 294 | cslb := realInit() 295 | cslb.netResolver = makeMockResolver() 296 | cslb.start() 297 | cesrv := cslb.lookupSRV(context.Background(), time.Now(), "https", "udp", "example.com") 298 | s := cesrv.String() 299 | c := strings.Count(s, "tarw=") 300 | if c != cesrv.uniqueTargets() { 301 | t.Error("Expected", cesrv.uniqueTargets(), "tarw= patterns, but got", c) 302 | } 303 | } 304 | 305 | // Test that the cache cleaner is expiring entries 306 | func TestSRVcleaner(t *testing.T) { 307 | cslb := realInit() 308 | cslb.netResolver = makeMockResolver() 309 | now := time.Now() 310 | yesterday := now.AddDate(0, 0, -1) // Yesterday 311 | 312 | cslb.lookupSRV(context.Background(), now, "http", "tcp", "keep.expire.example.com") 313 | for ix := 0; ix < 99; ix++ { 314 | cslb.lookupSRV(context.Background(), yesterday, "http", "tcp", 315 | fmt.Sprintf("%d.expire.example.com", ix)) 316 | } 317 | cslb.srvStore.start(time.Second / 2) 318 | defer cslb.srvStore.stop() 319 | 320 | time.Sleep(time.Second) // Give cleaner time to run 321 | cslb.srvStore.RLock() 322 | origLen := len(cslb.srvStore.cache) 323 | cslb.srvStore.RUnlock() 324 | if origLen != 1 { 325 | t.Error("Expected one entry, not", origLen) 326 | } 327 | } 328 | -------------------------------------------------------------------------------- /doc.go: -------------------------------------------------------------------------------- 1 | /* 2 | Package cslb provides transparent HTTP/HTTPS Client Side Load Balancing for Go programs. 3 | 4 | Cslb intercepts "net/http" Dial Requests and re-directs them to a preferred set of target hosts 5 | based on the load balancing configuration expressed in DNS SRV and TXT Resource Records (RRs). 6 | 7 | Only one trivial change is required to client applications to benefit from cslb which is to import 8 | this package and (if needed) enabling it for non-default http.Transport instances. Cslb processing 9 | is triggered by the presence of SRV RRs. If no SRVs exist cslb is benign which means you can deploy 10 | your application with cslb and independently activate and deactivate cslb processing for each 11 | service at any time. 12 | 13 | No server-side changes are required at all - apart for possibly dispensing with your server-side 14 | load-balancers! 15 | 16 | # DEFAULT USAGE 17 | 18 | Importing cslb automatically enables interception for http.DefaultTransport. In this program 19 | snippet: 20 | 21 | import ( 22 | "net/http" 23 | _ "github.com/markdingo/cslb" 24 | ) 25 | 26 | func main() { 27 | resp, err := http.Get("http://example.net/resource") 28 | 29 | the Dial Request made by http.Get is intercepted and processed by cslb. 30 | 31 | # NON DEFAULT USAGE 32 | 33 | If the application uses its own http.Transport then cslb processing needs to be activated by calling 34 | the cslb.Enable() function, i.e.: 35 | 36 | import ( 37 | "net/http" 38 | "github.com/markdingo/cslb" 39 | ) 40 | 41 | func main() { 42 | myTransport := http.Transport{...} 43 | cslb.Enable(myTransport) 44 | client := &http.Client{Transport: myTransport} 45 | resp, err := client.Get("http://mydomain/resource") 46 | ... 47 | 48 | The cslb.Enable() function replaces http.Transport.DialContext with its own intercept function. 49 | 50 | # WHEN TO USE CSLB 51 | 52 | Server-side load-balancers are no panacea. They add deployment and diagnostic complexity, cost, 53 | throughput constraints and become an additional point of possible failure. 54 | 55 | Cslb can help you achieve good load-balancing and fail-over behaviour without the need for *any* 56 | server-side load-balancers. This is particularly useful in enterprise and micro-service deployments 57 | as well as smaller application deployments where configuring and managing load-balancers is a 58 | significant resource drain. 59 | 60 | Cslb can be used to load-balance across geographically dispersed targets or where "hot stand-by" 61 | systems are purposely deployed on diverse infrastructure. 62 | 63 | # DNS ACTIVATION 64 | 65 | When cslb intercepts a http.Transport Dial Request to port 80 or port 443 it looks up SRV RRs as 66 | prescribed by RFC2782. That is, _http._tcp.$domain and _https._tcp.$domain respectively. Cslb 67 | directs the Dial Request to the highest preference target based on the SRV algorithm. If that Dial 68 | Request fails, it tries the next lower preference target until a successful connection is returned 69 | or all unique targets fail or it runs out of time. 70 | 71 | Cslb caches the SRV RRs (or their non-existence) as well as the result of Dial Requests to the SRV 72 | targets to optimize subequent intercepted calls and the selection of preferred targets. If no SRV 73 | RRs exist, cslb passes the Dial Request on to net.DialContext. 74 | 75 | # RULES OF INTERCEPTION 76 | 77 | Cslb has specific rules about when interception occurs. It normally only considers intercepting port 78 | 80 and port 443 however if the "cslb_allports" environment variable is set, cslb intercepts 79 | non-standard HTTP ports and maps them to numeric service names. For example http://example.net:8080 80 | gets mapped to _8080._tcp.example.net as the SRV name to resolve. 81 | 82 | # ACTIVE HEALTH CHECKS 83 | 84 | While cslb runs passively by caching the results of previous Dial Requests, it can also run actively 85 | by periodically performing health checks on targets. This is useful as an administrator can control 86 | health check behaviour to move a target "in and out of rotation" without changing DNS entries and 87 | waiting for TTLs to age out. Health checks are also likely to make the application a little more 88 | responsive as they are less likely to make a dial attempt to a target that is not working. 89 | 90 | Active health checking is enabled by the presence of a TXT RR in the sub-domain "_$port._cslb" of 91 | the target. E.g. if the SRV target is "s1.example.net:80" then cslb looks for the TXT RR at 92 | "_80._cslb.s1.example.net". If that TXT RR contains a URL then it becomes the health check URL. If 93 | no TXT RR exists or the contents do not form a valid URL then no active health check is performed 94 | for that target. 95 | 96 | The health check URL does not have to be related to the target in any particular way. It could be a 97 | URL to a central monitoring system which performs complicated application level tests and 98 | performance monitoring. Or it could be a URL on the target system itself. 99 | 100 | A health check is considered successful when a GET of the URL returns a 200 status and the content 101 | contains the uppercase text "OK" somewhere in the body (See the "cslb_hc_ok" environment variable 102 | for how this can be modified). Unless both those conditions are met the target is considered 103 | unavailable. 104 | 105 | Active health checks cease once a target becomes idle for too long and health check Dial Requests 106 | are *not* get intercepted by cslb. 107 | 108 | # CONVERTING A SITE TO CSLB 109 | 110 | If your current service exists on a single server called "s1.example.net" and you want to spread the 111 | load across additional servers "s2.example.net" and "s3.example.net" and assuming you've added the 112 | "cslb" package to your application then the following DNS changes active cslb processing: 113 | 114 | Current DNS 115 | 116 | s1.example.net. IN A 172.16.254.1 117 | IN AAAA 2001:db8::1 118 | 119 | s2.example.net. IN A 172.16.254.2 120 | IN AAAA 2001:db8::2 121 | 122 | s3.example.net. IN A 172.16.254.3 123 | IN AAAA 2001:db8::3 124 | 125 | Additional DNS 126 | 127 | _http._tcp.s1.example.net. IN SRV 1 70 80 s1.example.net. 128 | IN SRV 1 30 80 s2.example.net. 129 | IN SRV 2 0 8080 s3.example.net. 130 | 131 | _80._cslb.s1.example.net. IN TXT "http://healthchecker.example.com/s1" 132 | _80._cslb.s2.example.net. IN TXT "http://healthchecker.example.com/s2" 133 | _8080._cslb.s3.example.net. IN TXT "http://s3.example.net/ok" 134 | 135 | A number of observations about this DNS setup: 136 | 137 | - "s1" and "s2" are the highest priority 138 | - "s3" is only ever considered if both "s1" and "s2" are not responding 139 | - On average 70 out of 100 requests will be directed to "s1" 140 | - Connections to "s3" are made on port 8080 141 | - The health check for "s3" is on the same system as the service 142 | - The heallth checks for "s1" and "s2" are on a centralized system 143 | 144 | # CACHE AGEING 145 | 146 | Cslb maintains a cache of SRV lookups and the health status of targets. Cache entries automatically 147 | age out as a form of garbage collection. Removed cache entries stop any associated active health 148 | checks. Unfortunately the cache ageing does not have access to the DNS TTLs associated with the SRV 149 | RRs so it makes a best-guess at reasonable time-to-live values. 150 | 151 | The important point to note is that *all* values get periodically refreshed from the DNS. Nothing 152 | persists internally forever regardless of the level of activity. This means you can be sure that any 153 | changes to your DNS will be noticed by cslb in due course. 154 | 155 | # STATUS WEB PAGE 156 | 157 | Cslb optional runs a web server which presents internal statistics on its performance and 158 | activity. This web service has *no* access controls so it's best to only run it on a loopback 159 | address. Setting the environment variable "cslb_listen" to a listen address activates the status 160 | server. E.g.: 161 | 162 | $ cslb_listen=127.0.0.1:8081 ./myProgram 163 | 164 | # RUN TIME CONTROLS 165 | 166 | On initialization the cslb package examines the "cslb_options" environment variable for single 167 | letter options which have the following meaning: 168 | 169 | 'd' - Debug print dialContext calls 170 | 'h' - Debug print Health Check results 171 | 'i' - Debug print intercepted Dial Requests 172 | 'r' - Debug print system Dial Context results 173 | 's' - Debug print SRV Lookups 174 | 175 | 'C' - Disable all Dial Request interception 176 | 'H' - Disable all health checks 177 | 'N' - Allow numeric service lookups for non-HTTP(S) ports 178 | 179 | An example of how this might by used from a shell: 180 | 181 | $ cslb_options=dh ./yourProgram -options ... 182 | 183 | Many internal configuration values can be over-ridden with environment variables as shown in this 184 | table: 185 | 186 | +----------------+----------------------------------------+---------+---------------+ 187 | | Variable Name | Description | Default | Format | 188 | +----------------+----------------------------------------+---------+---------------+ 189 | | cslb_dial_veto | Target veto period after dial fails | 1m | time.Duration | 190 | | cslb_hc_freq | Frequency of health checks per target | 50s | time.Duration | 191 | | cslb_hc_ok | strings.Contains in health check body | "OK" | String | 192 | | cslb_listen | Listen address for status server | | address:port | 193 | | cslb_nxd_ttl | Cache lifetime for NXDOMAIN SRVs | 20m | time.Duration | 194 | | cslb_srv_ttl | Cache lifetime for found SRVs | 5m | time.Duration | 195 | | cslb_tar_ttl | Cache lifetime for dial Targets | 5m | time.Duration | 196 | | cslb_templates | Alternate status server html/templates | | filepath.Glob | 197 | | cslb_timeout | Default intercept Dial duration | 1m | time.Duration | 198 | +----------------+----------------------------------------+---------+---------------+ 199 | 200 | Any values which are invalid or fall outside a reasonable range are ignored. 201 | 202 | # DETECTING A GOOD SERVICE 203 | 204 | Cslb only knows about the results of network connection attempts made by DialContext and the results 205 | of any configured health checks. If a service is accepting network connections but not responding to 206 | HTTP requests - or responding negatively - the client experiences failures but cslb will be unaware 207 | of these failures. The result is that cslb will continue to direct future Dial Requests to that 208 | faulty service in accordance with the SRV priorities. If your service is vulnerable to this 209 | scenario, active health checks are recommended. This could be something ss simple as an on-service 210 | health check which responds based on recent "200 OK" responses in the service log file. 211 | Alternatively an on-service monitor which closes the listen socket will also work. 212 | 213 | In general, defining a failing service is a complicated matter that only the application truly 214 | understands. For this reason health checks are used as an intermediary which does understand 215 | application level failures and converts them to simple language which cslb groks. 216 | 217 | # RECOMMENDED SETUP 218 | 219 | While every service is different there are a few general guidelines which apply to most services 220 | when using cslb. First of all, run simple health checks if you can and configure them for use by 221 | cslb. Second, have each target configured with both ipv4 and ipv6 addresses. This affords two 222 | potentially independent network paths to the targets. Furthermore, net.Dialer attempts both ipv4 and 223 | ipv6 connections simultaneously which maximizes responsiveness for the client. 224 | 225 | Third, consider a "canary" target as a low preference (highest numeric value SRV priority) 226 | target. If this "canary" target is accessed by cslb clients it tells you they are having trouble 227 | reaching their "real" targets. Being able to run a "canary" service is one of the side-benefits of 228 | cslb and SRVs. 229 | 230 | # CAVEATS 231 | 232 | Whan analyzing the Status Web Page or watching the Run Time Control output, observers need to be 233 | aware of caching by the http (and possibly other) packages. For example not every call to http.Get() 234 | results in a Dial Request as httpClient tries to re-use connections. 235 | 236 | In a similar vein if you change a DNS entry and don't believe cslb has noticed this change within an 237 | appropriate TTL amount of time, be aware that on some platforms the intervening recursive resolvers 238 | adjust TTLs as they see fit. For example some home-gamer routers are known to increase short TTLs to 239 | values they believe to be a more "appropriate" in an attempt to reduce their cache churn. 240 | 241 | Perhaps the biggest caveat of all is that cslb relies on being enabled for all http.Transports in 242 | use by your application. If you are importing a package (either directly or indirectly) which 243 | constructs its own http.Transports then you'll need to modify that package to call cslb.Enable() 244 | otherwise those http requests will not be intercepted. Of course if the package is making requests 245 | incidental to the core functionality of your application then maybe it doesn't matter and you can 246 | leave them be. Something to be aware of. 247 | 248 | ----- 249 | */ 250 | package cslb 251 | -------------------------------------------------------------------------------- /srv.go: -------------------------------------------------------------------------------- 1 | package cslb 2 | 3 | /* 4 | The srv structs cache SRV RRs. They are structured to make life easy for bestTarget() as that is 5 | presumed to be the most heavily used function in this package. The relationship between structs is: 6 | ceSRV->cePriority->ceTarget where cePriority contains all matching priorities and ceTarget contains 7 | all targets with that priority. 8 | */ 9 | 10 | import ( 11 | "context" 12 | "fmt" 13 | "net" 14 | "sort" 15 | "strings" 16 | "sync" 17 | "time" 18 | ) 19 | 20 | const ( 21 | smallChanceMultiplier = 1000 // Fraction of weight given to zero weighted targets 22 | ) 23 | 24 | type srvCache struct { 25 | sync.RWMutex // Protects everything within this struct 26 | done chan bool // Shuts down the cache cleaner 27 | cache map[string]*ceSRV // The cache key is ToLower(qName). 28 | } 29 | 30 | type ceSRV struct { 31 | expires time.Time // When this entry expire out of the cache 32 | lookups int // Includes initial lookup that creates the cache entry 33 | priorities []*cePriority // Slice of targets with equal priority 34 | uniqueTargetCount int // Count of all unique targets (host:port) 35 | } 36 | 37 | // String return a printable string of the cached Entry SRV 38 | func (t *ceSRV) String() (s string) { 39 | s = fmt.Sprintf("%s (%d):", t.expires, len(t.priorities)) 40 | for _, cep := range t.priorities { 41 | s += fmt.Sprintf("\n\tp=%d totw=%d (%d):", cep.priority, cep.totalWeight, len(cep.targets)) 42 | for _, cet := range cep.targets { 43 | s += fmt.Sprintf("\n\t\ttarw=%d %s:%d", cet.weight, cet.target, cet.port) 44 | } 45 | } 46 | return 47 | } 48 | 49 | type cePriority struct { 50 | priority int 51 | totalWeight int // Sum of all weights - used as an upper limit for the PRNG 52 | targets []*ceTarget // Slice of all targets within priority 53 | } 54 | 55 | type ceTarget struct { 56 | weight int 57 | port int 58 | target string 59 | } 60 | 61 | // healthStoreKey generates the lookup key for the healthStore. It's of the form host:port 62 | func (t *ceTarget) healthStoreKey() string { 63 | return makeHealthStoreKey(t.target, t.port) 64 | } 65 | 66 | func newSrvCache() *srvCache { 67 | return &srvCache{cache: make(map[string]*ceSRV), done: make(chan bool)} 68 | } 69 | 70 | func (t *srvCache) start(cacheInterval time.Duration) { 71 | go t.cleaner(cacheInterval) 72 | } 73 | 74 | func (t *srvCache) stop() { 75 | close(t.done) 76 | } 77 | 78 | // lookupSRV looks up the SRV RR for the domain. First it tries looking in the cache and if not 79 | // there, the DNS is consulted. The qName is of the form ToLower(_http._tcp.$domain) where "http" is 80 | // the service and "tcp" is the proto. The cache is updated with the results of the DNS lookup. 81 | // 82 | // lookupSRV returns a *ceSRV with an array of (possibly zero) net.SRV RRs even with an NXDomain 83 | // response (which comes back as an error). An empty list means the DNS lookup failed; normally this 84 | // means that the domain should not be considered to be under cslb control. 85 | // 86 | // We construct the full key/qName ourselves rather than rely on LookupSRV as we need the formed 87 | // qName for the cache lookup. I suppose we could have a different cache key and let LookupSRV do 88 | // it's thing but that would be a little confusing. Besides, the SRV name construction is well-known 89 | // and simple. 90 | func (t *cslb) lookupSRV(ctx context.Context, now time.Time, service, proto, domain string) *ceSRV { 91 | key := strings.ToLower("_" + service + "._" + proto + "." + domain) // rfc2782 format 92 | t.srvStore.RLock() 93 | cesrv := t.srvStore.cache[key] 94 | if cesrv != nil { // If cache entry exists we're done 95 | cesrv.lookups++ 96 | t.srvStore.RUnlock() 97 | return cesrv 98 | } 99 | t.srvStore.RUnlock() // Don't hold mutex across a possible DNS lookup 100 | 101 | cesrv = &ceSRV{expires: now.Add(t.NotFoundSRVTTL), lookups: 1} // Assume NXDomain 102 | _, srvList, _ := t.netResolver.LookupSRV(ctx, "", "", key) 103 | if len(srvList) > 0 { // Found something so transfer to the new ceSRV 104 | cesrv.expires = now.Add(t.FoundSRVTTL) 105 | cesrv.populate(srvList) 106 | } 107 | 108 | // Insert/over-write the cache entry. It's possible another go-routine snuck in while we 109 | // were off in DNS-land and created an entry with the same key. Oh well. The slowest 110 | // go-routine wins. We don't bother queuing all callers for the same cache key behind each 111 | // other and let one do the resolution. I suppose we could at some point in the future by 112 | // creating a sync.Cond and the resolving go-routine could Broadcast when done. Go makes 113 | // this quite easy but that's bound to be a premature optimization as a single program is 114 | // unlikely to be banging away at a single domain at the same instant in time without an 115 | // entry existing in our cache. 116 | 117 | targetKeys := cesrv.uniqueTargetKeys() 118 | cesrv.uniqueTargetCount = len(targetKeys) 119 | t.srvStore.Lock() 120 | t.srvStore.cache[key] = cesrv // cesrv is now read-only for the rest of its life 121 | t.srvStore.Unlock() 122 | t.populateHealthStore(now, targetKeys) 123 | 124 | return cesrv 125 | } 126 | 127 | // populate transfers the SRV RRs into the ceSRV. This means sorting the SRVs by priority order and 128 | // also calculating total weight so we can conveniently apply the SRV selection algorithm. 129 | // 130 | // The Golang resolver has pre-sorted the SRV RRs for us in priority/weight order but we don't rely 131 | // on that as there may be replacement resolvers or mock resolvers involved. 132 | // 133 | // RFC2782 says that "weights of 0 should have a very small chance of being selected" without 134 | // defining "very small". They way we achieve this as well as make our selection algorithm simple is 135 | // to give them collectively an effective weight of 0.1% of the total weights so that all zero 136 | // targets will on average get 1/1000th of traffic. That seems like a "very small chance" to me. 137 | func (t *ceSRV) populate(srvs []*net.SRV) { 138 | sort.Slice(srvs, func(i, j int) bool { return srvs[i].Priority < srvs[j].Priority }) 139 | var cep *cePriority // Ptr to current priority or nil if none yet 140 | for _, srv := range srvs { 141 | if len(srv.Target) == 0 { // RFC2782 says to ignore any zero length targets completely 142 | continue 143 | } 144 | if cep == nil || int(srv.Priority) != cep.priority { // Create a new higher priority? 145 | cep = &cePriority{priority: int(srv.Priority)} 146 | t.priorities = append(t.priorities, cep) 147 | } 148 | cet := &ceTarget{weight: int(srv.Weight) * smallChanceMultiplier, port: int(srv.Port), 149 | target: strings.ToLower(srv.Target)} 150 | cep.targets = append(cep.targets, cet) 151 | cep.totalWeight += cet.weight 152 | } 153 | 154 | // Assign a non-zero weight to targets with an SRV weight of zero 155 | 156 | for _, cep := range t.priorities { 157 | zeroWeightEntryCount := 0 158 | for _, cet := range cep.targets { 159 | if cet.weight == 0 { 160 | zeroWeightEntryCount++ 161 | } 162 | } 163 | 164 | if zeroWeightEntryCount == 0 { 165 | continue 166 | } 167 | 168 | verySmall := cep.totalWeight / smallChanceMultiplier 169 | verySmall = (verySmall + zeroWeightEntryCount - 1) / zeroWeightEntryCount 170 | if verySmall == 0 { // This can be true if totalWeight is very small or zero! 171 | verySmall = 1 172 | } 173 | for _, cet := range cep.targets { 174 | if cet.weight == 0 { 175 | cet.weight = verySmall 176 | cep.totalWeight += verySmall 177 | } 178 | } 179 | } 180 | } 181 | 182 | // bestTarget selects the "best" target to try and connect to based on the SRV selection algorithm 183 | // and the health of the targets. That is, pick the SRV set with the lowest-numerical priority 184 | // first. If all those targets are unavailable then pick the SRV set with the next lowest-numerical 185 | // priority. Within the selected priority weight is used to distribute load. E.g. a weight list of 186 | // a=1, b=2, c=3 would ideally have 12 requests distributed such that 2 go to a, 4 go to b and 6 go 187 | // to c. 188 | // 189 | // If all targets in all priorities are "bad" due to health checks or connection failures, pick the 190 | // the least-worst target which is the target with a nextDialAttempt closest to now. 191 | // 192 | // A synthesized SRV is always returned if there are any targets in the SRV. In the case of the 193 | // least-worst return, maybe the caller will get lucky and the connection will come good this time? 194 | // Or maybe they won't get lucky but at least they get to see a "connection failed" outcome and can 195 | // report it to something or someone. 196 | // 197 | // To summarize, the returned SRV will be one of the following in the order shown: 198 | // 199 | // - Highest Priority in good health in weight range - first choice within priority 200 | // - First target in highest Priority in good health - second choice within priority 201 | // - Same thing for each Priority down if none of the previous priorities are in good health 202 | // - Target with soonest next connection attempt regardless of priority or weight - least worst 203 | // 204 | // The caller should always check for a nil return, the other values in the returned SRV are mostly 205 | // returned as a convenience to the caller. They should not presume they are the exact same values 206 | // as retrieved from the DNS but they will be comparable. 207 | func (t *cslb) bestTarget(cesrv *ceSRV) (srv *net.SRV) { 208 | if len(cesrv.priorities) == 0 { // Either an NXDomain or SRV with zero length targets 209 | return nil 210 | } 211 | 212 | srv = &net.SRV{} // We will return something! 213 | now := time.Now() 214 | t.healthStore.RLock() // Apply Read lock across whole search rather than a nickle & dime approach 215 | defer t.healthStore.RUnlock() // whereby we may cycle the lock many times. 216 | 217 | // Search for the in-range weight but also note a target in good health in passing (called 218 | // our secondChoice) as the preferred weight may be in bad health in which case we'll take 219 | // any weight in the same priority as our second choice in preference to a lower priority. 220 | 221 | haveSecondChoice := false 222 | for _, cep := range cesrv.priorities { 223 | wix := t.randIntn(cep.totalWeight) // Select the weight value using a "cheap" RNG 224 | lower := 0 225 | upper := 0 226 | for _, cet := range cep.targets { 227 | ceh := t.healthStore.cache[cet.healthStoreKey()] 228 | upper += cet.weight 229 | if ceh == nil || ceh.isGood(now) { 230 | if wix >= lower && wix < upper { // Is this target in the weight range? 231 | srv.Target = cet.target 232 | srv.Port = uint16(cet.port) 233 | srv.Priority = uint16(cep.priority) 234 | srv.Weight = uint16(cet.weight) / smallChanceMultiplier 235 | return // This is expected to be the "happy path" 236 | } 237 | if !haveSecondChoice { // If we don't have a second choice yet, use this one 238 | haveSecondChoice = true 239 | srv.Target = cet.target 240 | srv.Port = uint16(cet.port) 241 | srv.Priority = uint16(cep.priority) 242 | srv.Weight = uint16(cet.weight) / smallChanceMultiplier 243 | } 244 | } 245 | lower = upper // Iterate over targets 246 | } 247 | if haveSecondChoice { // Preferred weight range was in bad health but we 248 | return // found a good health target in the preferred priority 249 | } 250 | } 251 | 252 | // Didn't find *any* healthy targets so search over *all* targets for least-worst. We don't 253 | // expect this to occur very often so the search loop is run a second time rather than add 254 | // complexity to the relatively simple search loop above. The least-worst target has the 255 | // soonest nextDialAttempt. Priority and weight are ignored as there is no point in 256 | // considering a higher priority target which has a nextDialAttempt way off into the future 257 | // as that means it's *just* failed whereas one that's a millisecond away from now has had 258 | // the longest time period to "come good". 259 | 260 | var smallestLeastWorst time.Time 261 | for _, cep := range cesrv.priorities { 262 | for _, cet := range cep.targets { 263 | ceh := t.healthStore.cache[cet.healthStoreKey()] 264 | nextAttempt := now 265 | if ceh != nil { // This could have changed underneath us, so be defensive 266 | nextAttempt = ceh.nextDialAttempt 267 | } 268 | 269 | // smallestLeastWorst starts life as IsZero() and nextAttempt is always 270 | // greater than zero so the first time thru this test always comes true and 271 | // sets a least-worst. 272 | 273 | if smallestLeastWorst.IsZero() || nextAttempt.Before(smallestLeastWorst) { 274 | srv.Target = cet.target 275 | srv.Port = uint16(cet.port) 276 | srv.Priority = uint16(cep.priority) 277 | srv.Weight = uint16(cet.weight) / smallChanceMultiplier 278 | smallestLeastWorst = nextAttempt 279 | } 280 | } 281 | } 282 | 283 | return 284 | } 285 | 286 | // uniqueTargetKeys returns a slice of all unique targets keys in the SRV (a key is host:port). The 287 | // SRV might actually have more targets than this count if some of the targets are identical. This 288 | // shouldn't occur in a single well-constructed SRV arrangement but targets might be shared across 289 | // other SRVs so we've generalized the need to cater for that such that a target is an independent 290 | // beast which just happens to be attached to one or more SRVs. 291 | func (t *ceSRV) uniqueTargetKeys() (tSlice []string) { 292 | dupes := make(map[string]bool) 293 | for _, cep := range t.priorities { 294 | for _, cet := range cep.targets { 295 | dupes[cet.healthStoreKey()] = true 296 | } 297 | } 298 | 299 | for k := range dupes { 300 | tSlice = append(tSlice, k) 301 | } 302 | 303 | return 304 | } 305 | 306 | // uniqueTargets returns the count of uniqueTargetKeys 307 | func (t *ceSRV) uniqueTargets() (count int) { 308 | return t.uniqueTargetCount 309 | } 310 | 311 | // cleaner periodically scans the srvStore for expired entries and deletes them. Normally run as a 312 | // go-routine. 313 | func (t *srvCache) cleaner(cleanInterval time.Duration) { 314 | ticker := time.NewTicker(cleanInterval) 315 | defer ticker.Stop() 316 | 317 | for { 318 | select { 319 | case <-t.done: 320 | return 321 | case now := <-ticker.C: 322 | t.clean(now) 323 | } 324 | } 325 | } 326 | 327 | func (t *srvCache) clean(now time.Time) { 328 | t.Lock() 329 | defer t.Unlock() 330 | 331 | for key, cesrv := range t.cache { 332 | if cesrv.expires.Before(now) { 333 | delete(t.cache, key) 334 | } 335 | } 336 | } 337 | 338 | // ceSrvAsStats is a clone of ceSRV (and related material) with exported variable for html.Template 339 | type ceSrvAsStats struct { 340 | CName string 341 | Expires string 342 | Lookups string 343 | Priority int 344 | Weight int 345 | Port int 346 | Target string 347 | GoodDials int // From healthStore 348 | FailedDials int 349 | IsGood bool 350 | } 351 | 352 | type srvStats struct { 353 | Srvs []ceSrvAsStats 354 | nxDomains []ceSrvAsStats 355 | } 356 | 357 | // getStats clones all the ceSRV entries into a struct suitable for the status service. This 358 | // shouldn't be too expensive as we don't expect a huge number of SRVs, but who knows? 359 | func (t *srvCache) getStats(hc *healthCache) *srvStats { 360 | now := time.Now() 361 | s := &srvStats{} 362 | t.RLock() 363 | defer t.RUnlock() 364 | 365 | s.Srvs = make([]ceSrvAsStats, 0, len(t.cache)*4) // Just guesses, but better than nothing and 366 | s.nxDomains = make([]ceSrvAsStats, 0, len(t.cache)) // over-sized is probably better than under-sized. 367 | for cname, cesrv := range t.cache { 368 | if len(cesrv.priorities) == 0 { // NXDomain? 369 | s.nxDomains = append(s.nxDomains, 370 | ceSrvAsStats{CName: cname, 371 | Expires: cesrv.expires.Sub(now).Truncate(time.Second).String(), 372 | Lookups: fmt.Sprintf("%d", cesrv.lookups), 373 | Target: "**NXDomain**"}) 374 | continue 375 | } 376 | for _, cep := range cesrv.priorities { 377 | for _, cet := range cep.targets { 378 | entry := ceSrvAsStats{CName: cname, 379 | Expires: cesrv.expires.Sub(now).Truncate(time.Second).String(), 380 | Lookups: fmt.Sprintf("%d", cesrv.lookups), 381 | Priority: cep.priority, 382 | Weight: cet.weight, 383 | Port: cet.port, 384 | Target: cet.target, 385 | IsGood: true} 386 | hc.RLock() 387 | ceh := hc.cache[cet.healthStoreKey()] 388 | if ceh != nil { 389 | entry.GoodDials = ceh.goodDials 390 | entry.FailedDials = ceh.failedDials 391 | entry.IsGood = ceh.isGood(now) 392 | } 393 | hc.RUnlock() 394 | 395 | s.Srvs = append(s.Srvs, entry) 396 | } 397 | } 398 | } 399 | 400 | return s 401 | } 402 | --------------------------------------------------------------------------------

CSLB Global State - which is completely empty but nonetheless functional

CSLB Global State

CSLB Config

CSLB Global Statistics

SRV DNS Cache

Target Health Cache