├── .gitignore ├── DRIVERS.md ├── README.md ├── driver_redis.go ├── examples ├── consumer.go └── provider.go └── gosd.go /.gitignore: -------------------------------------------------------------------------------- 1 | # Compiled Object files, Static and Dynamic libs (Shared Objects) 2 | *.o 3 | *.a 4 | *.so 5 | .source 6 | gosd 7 | 8 | # Folders 9 | _obj 10 | _test 11 | src 12 | bin 13 | 14 | # Architecture specific extensions/prefixes 15 | *.[568vq] 16 | [568vq].out 17 | 18 | *.cgo1.go 19 | *.cgo2.c 20 | _cgo_defun.c 21 | _cgo_gotypes.go 22 | _cgo_export.* 23 | 24 | _testmain.go 25 | 26 | *.exe 27 | *.test 28 | *.prof 29 | -------------------------------------------------------------------------------- /DRIVERS.md: -------------------------------------------------------------------------------- 1 | # Drivers 2 | 3 | Right now, I've only implemented a driver for Redis. 4 | Fork it and feel free to develop other drivers (etcd, memcache, custom apis...). 5 | 6 | ## How it works 7 | 8 | The driver shall implement the interface below. Simple like that. 9 | 10 | ``` 11 | type Driver interface { 12 | Start(name, url string) string 13 | Get() (map[string]string, error) 14 | Delete(currentName string) 15 | } 16 | ``` 17 | 18 | ### Methods and their returns 19 | 20 | Here are the details of each method: 21 | 22 | - Start(name, url string) string: 23 | - receives the service name (***"my-service-name"*** for example) and its URL (***https://...***). 24 | - responds an unique name for that service, note that it shall be formed with `-` for example: ***my-service-name-20160315122936.17256541*** 25 | 26 | 27 | - Get() (map[string]string, error) 28 | - responds with a key list of services unique names and their respective URLs, for example: 29 | 30 | ``` 31 | provider-20160315122936.17256541 | http://localhost:3333 32 | provider-20160315122713.15645582 | http://localhost:3334 33 | consumer-20160315122543.15645582 | http://localhost:3335 34 | ``` 35 | 36 | - Delete(currentName string) 37 | - receives a service unique name (like ***provider-20160315122936.17256541***) and removes it of it's database. The next time you run the command ***Get()*** that removed service data will not be retrieved. 38 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # gosd 2 | 3 | Scale your Go microservices in 5 minutes with this simple service discovery client for small architectures. 4 | 5 | It's ideal for starting projects because you don't need a separated network to run your service discovery. So it's very important you set your service discovery Database in a high available system, like Amazon ElastiCache. 6 | 7 | Today the only driver implemented works for Redis, but other are planned (etcd, SQLite, custom API...). 8 | 9 | ## Installation 10 | 11 | Just add the library to your project. 12 | 13 | ```go 14 | $ go get github.com/dalvaren/gosd 15 | ``` 16 | 17 | ## Configuration 18 | 19 | Following the [12 factor app rules](http://12factor.net/), the configurations for this library shall be done using environment variables. Below are the list with the needed configuration 20 | 21 | ```go 22 | # For the Redis Driver 23 | export gosdRedisDB=0 # Redis database 24 | export gosdRedisAddr="localhost:6379" # Redis host address 25 | export gosdRedisPassword="" # Redis password 26 | 27 | # For the Library 28 | export gosdTryRefreshAmount=3 # Number of times the library tries to reach the Service Discovery database for updating. 29 | export gosdTryFindServiceAmount=5 # Number of times the service tries to find the service URL in its cache. It's important to wait for other services to start. 30 | export gosdTryFindServiceDelay=3 # Delay in seconds between attempts of gosdTryFindServiceAmount 31 | ``` 32 | 33 | ## Usage 34 | 35 | First, you need to import it. 36 | 37 | ```go 38 | import gosd "github.com/dalvaren/gosd" 39 | ``` 40 | 41 | Start the Service Discovery client and choose the desired driver. Do it at your application startup. Basically, this command makes the microservice register itself in service discovery when it starts. 42 | 43 | ```go 44 | currentName := gosd.Start("my-service-name", "http://localhost:8885", gosd.DriverRedis{}) 45 | ``` 46 | 47 | ***currentName*** is the unique name of your service, used to close only it when this microservice closes. ***my-service-name*** is the desired service name. ***"http://localhost:8885"*** is an example of a reachable host for this service. You can implement some way to get it automatically. The service will unregister automatically with some problem occurs with the application. 48 | 49 | Now every time you want to get the most updated version of the registered services list you run: 50 | 51 | ```go 52 | gosd.Get() 53 | ``` 54 | 55 | And for the last, when you need some specific microservice URL (for an API request for example) you just call: 56 | 57 | ```go 58 | gosd.IterateServiceRoute("my-other-service-name") 59 | ``` 60 | 61 | This command iterates on local service list, using a Round Robin algorithm to give you the next service URL. 62 | 63 | And that's it. 64 | If you set right the configurations, you can start microservices on demand and they will be discovered by who is using it, after they call `gosd.Get()` or `gosd.UpdateByCron()` . 65 | 66 | You can see other (and important) features in next section. 67 | 68 | ## Advanced option 69 | 70 | 1. Update using GOSD cron. You can call this command on all microservice endpoints, put it in a middleware or something similar. 71 | 72 | ```go 73 | gosd.UpdateByCron() 74 | ``` 75 | 76 | 1. Delete some service manually, by its URL. It shall be done after you make some request and the service returns no response (you can perform 3 to 5 attempts before removing some server). Take care with this command, since you can unregister the own service: 77 | 78 | ```go 79 | gosd.DeleteServiceWithURL("http://localhost:8881") 80 | ``` 81 | 82 | 1. Sometimes it's interesting to register some services locally, for that you can use the command below. But remember to register again after each `gosd.Get()` or `gosd.UpdateByCron()` : 83 | 84 | ```go 85 | gosd.AddServiceManually("service-name", "http://localhost:8886") 86 | ``` 87 | 88 | 89 | ## Example 90 | 91 | As example let's run 2 identical services called "provider", where they responds with their unique IDs. I'm using Gin as framework. 92 | 93 | ```go 94 | // provider.go 95 | package main 96 | 97 | import "os" 98 | import "github.com/gin-gonic/gin" 99 | import "github.com/dalvaren/gosd" 100 | 101 | func main() { 102 | gosd.Start("provider", "http://localhost" + os.Args[1], gosd.DriverRedis{}) 103 | gosd.Get() 104 | 105 | router := gin.Default() 106 | router.GET("/ping", func(ginContext *gin.Context) { 107 | gosd.UpdateByCron() 108 | ginContext.JSON(200, "Provider: " + os.Args[2]) 109 | }) 110 | router.Run(os.Args[1]) 111 | } 112 | 113 | ``` 114 | 115 | - Open a terninal and run that with `go run provider.go :3333 1` 116 | - Open another terninal and run that with `go run provider.go :3334 2` 117 | 118 | And the "consumer", who makes 10 requests for the 2 services. Note that it does not need to know them individually and you also can start them some seconds later. 119 | 120 | ```go 121 | // consumer.go 122 | package main 123 | 124 | import "fmt" 125 | import "github.com/dalvaren/gosd" 126 | import "github.com/parnurzeal/gorequest" 127 | 128 | func main() { 129 | gosd.Start("consumer", "http://localhost:fake", gosd.DriverRedis{}) 130 | gosd.Get() 131 | 132 | for n:=0; n < 10; n++ { 133 | providerURL := gosd.IterateServiceRoute("provider") 134 | request := gorequest.New() 135 | _, body, _ := request.Get(providerURL + "/ping").End() 136 | fmt.Print(body) 137 | } 138 | } 139 | 140 | ``` 141 | 142 | - Open another terninal and run that with `go run consumer.go` ... it will print something like: 143 | 144 | ``` 145 | "Provider: 1" 146 | "Provider: 2" 147 | "Provider: 1" 148 | "Provider: 2" 149 | "Provider: 1" 150 | "Provider: 2" 151 | "Provider: 1" 152 | "Provider: 2" 153 | "Provider: 1" 154 | "Provider: 2" 155 | ``` 156 | 157 | ## Contribute creating drivers for it! 158 | 159 | Right now, I've only implemented a driver for Redis. 160 | Fork it and feel free to develop other drivers (etcd, memcache, custom apis...). 161 | The driver documentation can be found [here](https://github.com/dalvaren/gosd/blob/master/DRIVERS.md)! 162 | 163 | ## Author 164 | 165 | Daniel Campos 166 | -------------------------------------------------------------------------------- /driver_redis.go: -------------------------------------------------------------------------------- 1 | // To run: 2 | // go get github.com/githubnemo/CompileDaemon 3 | // CompileDaemon -command="./gervice" 4 | 5 | package gosd 6 | 7 | import "os" 8 | import "fmt" 9 | import "time" 10 | import "strconv" 11 | import redis "gopkg.in/redis.v3" 12 | 13 | type DriverRedis struct {} 14 | 15 | var RedisClient *redis.Client 16 | 17 | func (this DriverRedis) Start(name, url string) string { 18 | // start 19 | redisDB,_ := strconv.Atoi(os.Getenv("gosdRedisDB")) 20 | RedisClient = redis.NewClient(&redis.Options{ 21 | Addr: os.Getenv("gosdRedisAddr"), 22 | Password: os.Getenv("gosdRedisPassword"), // no password set 23 | DB: int64(redisDB), // use default DB 24 | }) 25 | _, err := RedisClient.Ping().Result() 26 | if err != nil { 27 | fmt.Println("Error connecting with Redis.") 28 | fmt.Println(err.Error()) 29 | return "standalone-" + name 30 | } 31 | 32 | // set 33 | currentName := registerService(name, url) 34 | return currentName 35 | } 36 | 37 | func (this DriverRedis) Get() (map[string]string, error) { 38 | return RedisClient.HGetAllMap("gosd").Result() 39 | } 40 | 41 | func (this DriverRedis) Delete(currentName string) { 42 | RedisClient.HDel("gosd", currentName) 43 | } 44 | 45 | 46 | func registerService(basicName, url string) string { 47 | finalServiceName := "" 48 | created := false 49 | for created != true { 50 | finalServiceName = basicName + "-" + time.Now().Format("20060102150405.99999999") 51 | created,_ = RedisClient.HSetNX("gosd", finalServiceName, url).Result() 52 | } 53 | return finalServiceName 54 | } 55 | -------------------------------------------------------------------------------- /examples/consumer.go: -------------------------------------------------------------------------------- 1 | // consumer.go 2 | package main 3 | 4 | import "fmt" 5 | import "github.com/dalvaren/gosd" 6 | import "github.com/parnurzeal/gorequest" 7 | 8 | func main() { 9 | gosd.Start("consumer", "http://localhost:fake", gosd.DriverRedis{}) 10 | gosd.Get() 11 | 12 | for n:=0; n < 10; n++ { 13 | providerURL := gosd.IterateServiceRoute("provider") 14 | request := gorequest.New() 15 | _, body, _ := request.Get(providerURL + "/ping").End() 16 | fmt.Print(body) 17 | } 18 | } 19 | -------------------------------------------------------------------------------- /examples/provider.go: -------------------------------------------------------------------------------- 1 | // provider.go 2 | package main 3 | 4 | import "os" 5 | import "github.com/gin-gonic/gin" 6 | import "github.com/dalvaren/gosd" 7 | 8 | func main() { 9 | gosd.Start("provider", "http://localhost" + os.Args[1], gosd.DriverRedis{}) 10 | gosd.Get() 11 | 12 | router := gin.Default() 13 | router.GET("/ping", func(ginContext *gin.Context) { 14 | gosd.UpdateByCron() 15 | ginContext.JSON(200, "Provider: " + os.Args[2]) 16 | }) 17 | router.Run(os.Args[1]) 18 | } 19 | -------------------------------------------------------------------------------- /gosd.go: -------------------------------------------------------------------------------- 1 | // To run: 2 | // go get github.com/githubnemo/CompileDaemon 3 | // CompileDaemon -command="./gosd" 4 | 5 | package gosd 6 | 7 | import "os" 8 | import "fmt" 9 | import "time" 10 | import "strings" 11 | import "strconv" 12 | import "syscall" 13 | import "os/signal" 14 | 15 | type Driver interface { 16 | Start(name, url string) string 17 | Get() (map[string]string, error) 18 | Delete(currentName string) 19 | } 20 | 21 | 22 | type ServiceCacheEntry struct { 23 | Name string 24 | URL string 25 | } 26 | 27 | type Updater struct { 28 | Name string 29 | State string // expired or updated 30 | TTL time.Time 31 | Driver Driver 32 | ServiceCacheEntries []ServiceCacheEntry 33 | } 34 | 35 | type ServiceMap struct { 36 | Index int 37 | URLs []string 38 | } 39 | 40 | type Settings struct { 41 | TryRefreshAmount int 42 | TryFindServiceAmount int 43 | TryFindServiceDelay time.Duration 44 | } 45 | 46 | var ServiceSettings Settings 47 | var ServiceMaps map[string]ServiceMap 48 | var ServiceUpdater Updater 49 | var LastCronTime time.Time 50 | 51 | func AddServiceManually(name, url string) { 52 | serviceCacheEntry := ServiceCacheEntry{ 53 | Name: name + "-" + time.Now().Format("20060102150405.99999999"), 54 | URL: url, 55 | } 56 | ServiceUpdater.ServiceCacheEntries = append(ServiceUpdater.ServiceCacheEntries, serviceCacheEntry) 57 | recalculateServiceMaps(ServiceUpdater) 58 | } 59 | 60 | func UpdateByCron() { 61 | if LastCronTime.Add(120 * time.Minute).Before(time.Now()) { 62 | Get() 63 | } 64 | } 65 | 66 | func WaitClosing() { 67 | c := make(chan os.Signal, 1) 68 | signal.Notify(c, os.Interrupt) 69 | signal.Notify(c, syscall.SIGTERM) 70 | go func() { 71 | <-c 72 | fmt.Println("Finishing service: " + ServiceUpdater.Name) 73 | Finish(ServiceUpdater.Name) 74 | os.Exit(1) 75 | }() 76 | } 77 | 78 | func IterateServiceRoute(serviceBaseName string) string { 79 | attemptsNumber := ServiceSettings.TryFindServiceAmount 80 | for attemptsNumber > 0 { 81 | url := getNextURLForService(serviceBaseName) 82 | if url != "" { 83 | return url 84 | } 85 | Get() 86 | attemptsNumber-- 87 | time.Sleep(ServiceSettings.TryFindServiceDelay) 88 | } 89 | return "" 90 | } 91 | 92 | func Finish(currentName string) { 93 | // delete 94 | Delete(currentName) 95 | } 96 | 97 | func Delete(currentName string) { 98 | ServiceUpdater.Driver.Delete(currentName) 99 | Get() 100 | } 101 | 102 | func DeleteServiceWithURL(url string) { 103 | for _,serviceCacheEntry := range ServiceUpdater.ServiceCacheEntries { 104 | if serviceCacheEntry.URL == url { 105 | Delete(serviceCacheEntry.Name) 106 | } 107 | } 108 | } 109 | 110 | func Start(name, url string, driver Driver) string { 111 | // start settings 112 | ServiceSettings.TryRefreshAmount = 3 113 | ServiceSettings.TryFindServiceAmount = 5 114 | ServiceSettings.TryFindServiceDelay = 3 * time.Second 115 | if os.Getenv("gosdTryRefreshAmount") != "" { 116 | param,_ := strconv.Atoi(os.Getenv("gosdTryRefreshAmount")) 117 | ServiceSettings.TryRefreshAmount = param 118 | } 119 | if os.Getenv("gosdTryFindServiceAmount") != "" { 120 | param,_ := strconv.Atoi(os.Getenv("gosdTryFindServiceAmount")) 121 | ServiceSettings.TryRefreshAmount = param 122 | } 123 | if os.Getenv("gosdTryFindServiceDelay") != "" { 124 | param,_ := strconv.Atoi(os.Getenv("gosdTryFindServiceDelay")) 125 | ServiceSettings.TryFindServiceDelay = time.Duration(param) * time.Second 126 | } 127 | 128 | // start cron 129 | LastCronTime = time.Now() 130 | 131 | // start 132 | currentName := driver.Start(name, url) 133 | ServiceUpdater = Updater{ 134 | Name: currentName, 135 | State: "expired", 136 | TTL: time.Now(), 137 | Driver: driver, 138 | } 139 | 140 | // force service to unregister on closing 141 | WaitClosing() 142 | 143 | return currentName 144 | } 145 | 146 | func Get() { 147 | val := tryRefreshForNTimes(ServiceSettings.TryRefreshAmount) 148 | ServiceUpdater.ServiceCacheEntries = []ServiceCacheEntry{} 149 | for key,value := range val { 150 | // populate Updater 151 | serviceCacheEntry := ServiceCacheEntry{ 152 | Name: key, 153 | URL: value, 154 | } 155 | ServiceUpdater.ServiceCacheEntries = append(ServiceUpdater.ServiceCacheEntries, serviceCacheEntry) 156 | } 157 | recalculateServiceMaps(ServiceUpdater) 158 | } 159 | 160 | func getNextURLForService(baseName string) string { 161 | serviceMaps, exists := ServiceMaps[baseName] 162 | if !exists { 163 | return "" 164 | } 165 | if serviceMaps.Index >= len(ServiceMaps[baseName].URLs) { 166 | serviceMaps.Index = 0 167 | } 168 | serviceMaps.Index++ 169 | ServiceMaps[baseName] = serviceMaps 170 | return ServiceMaps[baseName].URLs[(serviceMaps.Index - 1)] 171 | } 172 | 173 | func recalculateServiceMaps(updater Updater) { 174 | if len(updater.ServiceCacheEntries) == 0 { 175 | return 176 | } 177 | serviceMaps := map[string]ServiceMap{} 178 | for _,serviceCacheEntry := range updater.ServiceCacheEntries { 179 | serviceMap := getServiceMap(getServiceBaseName(serviceCacheEntry.Name), serviceMaps) 180 | serviceMap.URLs = append(serviceMap.URLs, serviceCacheEntry.URL) 181 | serviceMaps[getServiceBaseName(serviceCacheEntry.Name)] = serviceMap 182 | } 183 | ServiceMaps = serviceMaps 184 | } 185 | 186 | func getServiceMap(baseName string, serviceMaps map[string]ServiceMap) ServiceMap { 187 | for key,serviceMap := range serviceMaps { 188 | if key == baseName { 189 | return serviceMap 190 | } 191 | } 192 | return ServiceMap{ 193 | Index: 0, 194 | URLs: []string{}, 195 | } 196 | } 197 | 198 | func getServiceBaseName(name string) string { 199 | if index := strings.LastIndex(name, "-"); index > -1 { 200 | return name[0:index] 201 | } 202 | return name 203 | } 204 | 205 | func tryRefreshForNTimes(n int) map[string]string { 206 | for n > 0 { 207 | val, err := ServiceUpdater.Driver.Get() 208 | if err != nil { 209 | n-- 210 | } else { 211 | return val 212 | } 213 | } 214 | return map[string]string{} 215 | } 216 | --------------------------------------------------------------------------------