4 | 
5 | 
6 |
3 |
More details explanation on HashiCorp Vault, please check my Medium article : Secrets in Kubernetes and HashiCorp Vault
87 |Implement the Vault Envs in the pod deployment
89 | 90 | spec: 91 | serviceAccountName: vault 92 | containers: 93 | - name: go-cache-kubernetes-container-poc 94 | image: deeptiman1991/go-cache-kubernetes-v1:1.0.0 95 | env: 96 | - name: VAULT_ADDR 97 | value: "http://vault:8200" 98 | - name: JWT_PATH 99 | value: "/var/run/secrets/kubernetes.io/serviceaccount/token" 100 | - name: SERVICE_PORT 101 | value: "8080" 102 || YAML | 106 |go-cache-poc-app.yaml | 107 |
So, now SECRET_USERNAME & SECRET_PASSWORD environment variables can be used to connect to the MongoDB database from the application.
112 | 113 || Name | 121 |go-cache-poc-pvc | 122 |
| Kind | 125 |PersistentVolumeClaim | 126 |
| YAML | 129 |go-cache-poc-pvc.yaml | 130 |
| Name | 144 |go-cache-poc | 145 |
| Kind | 148 |Deployment | 149 |
| YAML | 152 |go-cache-poc-app.yaml | 153 |
| Name | 175 |go-cache-poc-service | 176 |
| Kind | 179 |Service | 180 |
| YAML | 183 |go-cache-poc-svc.yaml | 184 |
Kubernetes provides a feature that will allow us to create a stateful application in the cluster. There will be a storage class and services running under the cluster that will allow the databases to connect with services and store records in their persistent database.
199 |More details explanation on MongoDB StatefulSet, please check my Medium article : MongoDB StatefulSet in Kubernetes
201 || Name | 207 |mongodb-service | 208 |
| Kind | 211 |Service | 212 |
| YAML | 215 |mongodb-service.yaml | 216 |
| Name | 227 |mongod | 228 |
| Kind | 231 |StatefulSet | 232 |
| YAML | 235 |mongodb-statefulset.yaml | 236 |
Download Docker images for Redis
283 |
284 | $ docker run -p 6379:6379 redislabs/redismod
285 |
Redis Deployment
287 || Name | 291 |redismod | 292 |
| Kind | 295 |Deployment | 296 |
| YAML | 299 |redis-deployment.yaml | 300 |
$ kubectl apply -f redis-deployment.yaml
Redis Service
306 || Name | 310 |redis-service | 311 |
| Kind | 314 |Service | 315 |
| YAML | 318 |redis-service.yaml | 319 |
$ kubectl apply -f redis-service.yaml
Deploy the redismod image
324 |
325 | $ kubectl run redismod --image=redislabs/redismod --port=6379
326 |
Expose the deployment
328 |
329 | $ kubectl expose deployment redismod --type=NodePort
330 |
Now, check for the Redis Connection
332 |
333 | $ redis-cli -u $(minikube service --format "redis://{{.IP}}:{{.Port}}" --url redismod)
334 |
You must be getting an ip address with a port that can be used as a connection string for Redis
336 |
337 | | Name | 347 |zookeeper-app | 348 |
| Kind | 351 |Deployment | 352 |
| YAML | 355 |zookeeper-deployment.yaml | 356 |
| Name | 367 |zookeeper-service | 368 |
| Kind | 371 |Service | 372 |
| YAML | 375 |zookeeper-service.yaml | 376 |
| Name | 388 |kafka-service | 389 |
| Kind | 392 |Service | 393 |
| YAML | 396 |kafka-service.yaml | 397 |
| Name | 410 |kafka-repcon | 411 |
| Kind | 414 |Deployment | 415 |
| YAML | 418 |kafka-repcon.yaml | 419 |
https://github.com/go-swagger/go-swagger
456 | 457 | 458 | ## More Info 459 |This project is licensed under the MIT License
469 | -------------------------------------------------------------------------------- /database.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "id": 100, 4 | "name": "Employee1", 5 | "email": "employee1@gmail.com", 6 | "company": "company1", 7 | "occupation": "engineer1", 8 | "salary": "$10,000" 9 | }, 10 | { 11 | "id": 101, 12 | "name": "Employee2", 13 | "email": "employee2@gmail.com", 14 | "company": "company2", 15 | "occupation": "engineer2", 16 | "salary": "$20,000" 17 | }, 18 | { 19 | "id": 102, 20 | "name": "Employee2", 21 | "email": "employee2@gmail.com", 22 | "company": "company2", 23 | "occupation": "engineer2", 24 | "salary": "$30,000" 25 | }, 26 | { 27 | "id": 103, 28 | "name": "Employee3", 29 | "email": "employee3@gmail.com", 30 | "company": "company3", 31 | "occupation": "engineer3", 32 | "salary": "$40,000" 33 | }, 34 | { 35 | "id": 104, 36 | "name": "Employee4", 37 | "email": "employee4@gmail.com", 38 | "company": "company4", 39 | "occupation": "engineer4", 40 | "salary": "$40,000" 41 | }, 42 | { 43 | "id": 105, 44 | "name": "Employee5", 45 | "email": "employee5@gmail.com", 46 | "company": "company5", 47 | "occupation": "engineer5", 48 | "salary": "$50,000" 49 | }, 50 | { 51 | "id": 106, 52 | "name": "Employee6", 53 | "email": "employee6@gmail.com", 54 | "company": "company6", 55 | "occupation": "engineer6", 56 | "salary": "$60,000" 57 | }, 58 | { 59 | "id": 107, 60 | "name": "Employee7", 61 | "email": "employee7@gmail.com", 62 | "company": "company7", 63 | "occupation": "engineer7", 64 | "salary": "$70,000" 65 | }, 66 | { 67 | "id": 108, 68 | "name": "Employee8", 69 | "email": "employee8@gmail.com", 70 | "company": "company8", 71 | "occupation": "engineer8", 72 | "salary": "$80,000" 73 | }, 74 | { 75 | "id": 109, 76 | "name": "Employee9", 77 | "email": "employee9@gmail.com", 78 | "company": "company9", 79 | "occupation": "engineer9", 80 | "salary": "$90,000" 81 | }, 82 | { 83 | "id": 110, 84 | "name": "Employee10", 85 | "email": "employee10@gmail.com", 86 | "company": "company10", 87 | "occupation": "engineer10", 88 | "salary": "$100,000" 89 | }, 90 | { 91 | "id": 111, 92 | "name": "Employee11", 93 | "email": "employee11@gmail.com", 94 | "company": "company11", 95 | "occupation": "engineer11", 96 | "salary": "$110,000" 97 | }, 98 | { 99 | "id": 112, 100 | "name": "Employee12", 101 | "email": "employee12@gmail.com", 102 | "company": "company12", 103 | "occupation": "engineer12", 104 | "salary": "$120,000" 105 | }, 106 | { 107 | "id": 113, 108 | "name": "Employee13", 109 | "email": "employee13@gmail.com", 110 | "company": "company13", 111 | "occupation": "engineer13", 112 | "salary": "$130,000" 113 | }, 114 | { 115 | "id": 114, 116 | "name": "Employee14", 117 | "email": "employee14@gmail.com", 118 | "company": "company14", 119 | "occupation": "engineer14", 120 | "salary": "$140,000" 121 | }, 122 | { 123 | "id": 115, 124 | "name": "Employee15", 125 | "email": "employee15@gmail.com", 126 | "company": "company15", 127 | "occupation": "engineer15", 128 | "salary": "$150,000" 129 | }, 130 | { 131 | "id": 116, 132 | "name": "Employee16", 133 | "email": "employee16@gmail.com", 134 | "company": "company16", 135 | "occupation": "engineer16", 136 | "salary": "$160,000" 137 | }, 138 | { 139 | "id": 117, 140 | "name": "Employee17", 141 | "email": "employee17@gmail.com", 142 | "company": "company17", 143 | "occupation": "engineer17", 144 | "salary": "$170,000" 145 | }, 146 | { 147 | "id": 118, 148 | "name": "Employee18", 149 | "email": "employee18@gmail.com", 150 | "company": "company18", 151 | "occupation": "engineer18", 152 | "salary": "$180,000" 153 | }, 154 | { 155 | "id": 119, 156 | "name": "Employee19", 157 | "email": "employee19@gmail.com", 158 | "company": "company19", 159 | "occupation": "engineer19", 160 | "salary": "$190,000" 161 | }, 162 | { 163 | "id": 120, 164 | "name": "Employee20", 165 | "email": "employee20@gmail.com", 166 | "company": "company20", 167 | "occupation": "engineer20", 168 | "salary": "$200,000" 169 | } 170 | ] -------------------------------------------------------------------------------- /database/database_create_employees.go: -------------------------------------------------------------------------------- 1 | package database 2 | 3 | import ( 4 | "context" 5 | ) 6 | 7 | // CreateEmployee method 8 | func (e *EmployeeDB) CreateEmployee(employee *Employee) error { 9 | 10 | collection, err := e.GetCollection() 11 | if err != nil { 12 | e.log.Error("Unable to create collection", "error", err.Error()) 13 | return err 14 | } 15 | 16 | //check wheather employee exists in the db 17 | _, err = e.GetEmployeeByID(employee.ID) 18 | 19 | if err == ErrEmployeeNotFound { 20 | insertResult, err := collection.InsertOne(context.TODO(), employee) 21 | if err != nil { 22 | e.log.Error("Failed to create employee", "error", err.Error()) 23 | return err 24 | } 25 | e.log.Info("Employee Created", "success", insertResult) 26 | } else { 27 | return EmployeeAlreadyExists 28 | } 29 | 30 | return nil 31 | } 32 | -------------------------------------------------------------------------------- /database/database_delete_employee.go: -------------------------------------------------------------------------------- 1 | package database 2 | 3 | import ( 4 | "context" 5 | 6 | "go.mongodb.org/mongo-driver/bson" 7 | ) 8 | 9 | //DeleteEmployeeByID 10 | func (e *EmployeeDB) DeleteEmployeeByID(id int) error { 11 | 12 | collection, err := e.GetCollection() 13 | if err != nil { 14 | e.log.Error("Unable to create collection", "error", err.Error()) 15 | return err 16 | } 17 | 18 | _, err = collection.DeleteMany( 19 | context.TODO(), 20 | bson.M{MONGODB_COLLECTION_ID: id}) 21 | 22 | if err != nil { 23 | e.log.Error("Unable to delete employee", "error", err.Error()) 24 | return err 25 | } 26 | 27 | key := getKey(id) 28 | err = e.redisCache.Del(key) 29 | if err != nil { 30 | e.log.Error("Unable to delete from redis cache", "error", err.Error()) 31 | return err 32 | } 33 | 34 | return nil 35 | } 36 | -------------------------------------------------------------------------------- /database/database_get_employees.go: -------------------------------------------------------------------------------- 1 | package database 2 | 3 | import ( 4 | "context" 5 | 6 | "go.mongodb.org/mongo-driver/bson" 7 | "go.mongodb.org/mongo-driver/mongo/options" 8 | ) 9 | 10 | // GetAllEmployees method 11 | func (e *EmployeeDB) GetAllEmployees() ([]*Employee, error) { 12 | 13 | collection, err := e.GetCollection() 14 | if err != nil { 15 | e.log.Error("Unable to create collection", "error", err.Error()) 16 | return nil, err 17 | } 18 | 19 | findOptions := options.Find() 20 | findOptions.SetLimit(10) //currently supporting fetching 10 employee records 21 | 22 | records, err := collection.Find(context.TODO(), bson.D{{}}, findOptions) 23 | if err != nil { 24 | e.log.Error("Unable to fetch employee records", "error", err.Error()) 25 | return nil, err 26 | } 27 | 28 | var results []*Employee 29 | for records.Next(context.TODO()) { 30 | var employee Employee 31 | err := records.Decode(&employee) 32 | if err != nil { 33 | e.log.Error("Unable to read employee records", "error", err.Error()) 34 | return nil, err 35 | } 36 | results = append(results, &employee) 37 | } 38 | 39 | defer records.Close(context.TODO()) 40 | 41 | return results, nil 42 | } 43 | 44 | //GetEmployeeByID method 45 | func (e *EmployeeDB) GetEmployeeByID(id int) (*Employee, error) { 46 | 47 | var employee *Employee 48 | 49 | key := getKey(id) 50 | 51 | collection, err := e.GetCollection() 52 | if err != nil { 53 | e.log.Error("Unable to create collection", "error", err.Error()) 54 | return nil, err 55 | } 56 | 57 | filter := bson.D{{MONGODB_COLLECTION_ID, id}} 58 | cacheEmployee, err := e.redisCache.Get(key) 59 | 60 | if cacheEmployee == nil { 61 | e.log.Info("MongoDB ", "Employee", cacheEmployee) 62 | err = collection.FindOne(context.TODO(), filter).Decode(&employee) 63 | if err != nil { 64 | return nil, ErrEmployeeNotFound 65 | } 66 | e.redisCache.Set(key, employee) 67 | } else { 68 | e.log.Info("Redis Cache", "Employee", cacheEmployee) 69 | employee = cacheEmployee 70 | } 71 | 72 | if employee == nil { 73 | return nil, ErrEmployeeNotFound 74 | } 75 | 76 | return employee, nil 77 | } 78 | -------------------------------------------------------------------------------- /database/database_manager.go: -------------------------------------------------------------------------------- 1 | //Package classification of Employee API 2 | // 3 | //Documentation for Employee API 4 | // 5 | // 6 | // Schemes: http 7 | // BasePath: /api 8 | // Version: 1.0.0 9 | // 10 | // 11 | // Consumes: 12 | // - application/json 13 | // 14 | // Produces: 15 | // - application/json 16 | // 17 | //swagger:meta 18 | package database 19 | 20 | import ( 21 | "context" 22 | "encoding/json" 23 | "fmt" 24 | "io/ioutil" 25 | "net/http" 26 | "os" 27 | "strings" 28 | "time" 29 | 30 | "github.com/hashicorp/go-hclog" 31 | "go.mongodb.org/mongo-driver/mongo" 32 | "go.mongodb.org/mongo-driver/mongo/options" 33 | ) 34 | 35 | const ( 36 | VAULT_AUTH_LOGIN = "auth/kubernetes/login" 37 | VAULT_SECRET_DATA = "secret/data/webapp/config" 38 | VAULT_ROLE = "webapp" 39 | 40 | MONGODB_CONNECTION_STRING = "mongodb://mongod-0.mongodb-service.default.svc.cluster.local" 41 | MONGODB_DATABASE = "records" 42 | MONGODB_COLLECTION = "employees" 43 | 44 | MONGODB_COLLECTION_ID = "id" 45 | MONGODB_COLLECTION_NAME = "name" 46 | MONGODB_COLLECTION_EMAIL = "email" 47 | MONGODB_COLLECTION_COMPANY = "company" 48 | MONGODB_COLLECTION_OCCUPATION = "occupation" 49 | MONGODB_COLLECTION_SALARY = "salary" 50 | ) 51 | 52 | // Employee Schema structure 53 | type Employee struct { 54 | ID int `json:"id" validate:"required,gt=99"` 55 | Name string `json:"name" validate:"required"` 56 | Email string `json:"email" validate:"required,email"` 57 | Company string `json:"company" validate:"required"` 58 | Occupation string `json:"occupation" validate:"required"` 59 | Salary string `json:"salary" validate:"required"` 60 | } 61 | 62 | type EmployeeDB struct { 63 | redisCache *RedisCache 64 | log hclog.Logger 65 | } 66 | 67 | type Vault struct { 68 | Auth Auth 69 | } 70 | type Auth struct { 71 | Client_Token string 72 | } 73 | 74 | type VaultData struct { 75 | Data Data 76 | } 77 | 78 | type Data struct { 79 | Data SecretData 80 | } 81 | 82 | type SecretData struct { 83 | Username string 84 | Password string 85 | } 86 | 87 | var ErrEmployeeNotFound = fmt.Errorf("Employee not found") 88 | var EmployeeAlreadyExists = fmt.Errorf("Employee already exists") 89 | 90 | func InitializeDBManager(log hclog.Logger) *EmployeeDB { 91 | redisCache, _ := InitializeCacheClient() 92 | return &EmployeeDB{redisCache, log} 93 | } 94 | 95 | //GetMongoCredFromVault 96 | func (e *EmployeeDB) GetMongoCredFromVault() (*SecretData, error) { 97 | 98 | content_type := "application/json" 99 | VAULT_URL := os.Getenv("VAULT_ADDR") + "/v1/" 100 | JWT_TOKEN, err := ioutil.ReadFile(os.Getenv("JWT_PATH")) 101 | if err != nil { 102 | e.log.Error("Unable to read JWT File", "error", err.Error()) 103 | return nil, err 104 | } 105 | 106 | e.log.Info("VAULT_URL", "Prod", VAULT_URL) 107 | e.log.Info("Vault", "Auth URL", VAULT_URL+VAULT_AUTH_LOGIN) 108 | //Vault Request - 1: [Retrieve Client token from Vault] 109 | requestBody1 := strings.NewReader(` 110 | { 111 | "role": "` + VAULT_ROLE + `", 112 | "jwt": "` + string(JWT_TOKEN) + `" 113 | } 114 | `) 115 | res1, err := http.NewRequest("POST", VAULT_URL+VAULT_AUTH_LOGIN, requestBody1) 116 | if err != nil { 117 | e.log.Error("Unable to set Request", "error", err.Error()) 118 | return nil, err 119 | } 120 | res1.Header.Set("Content-Type", content_type) 121 | 122 | resp1, err := http.DefaultClient.Do(res1) 123 | if err != nil { 124 | e.log.Error("Unable to fetch Vault Client Token", "error", err.Error()) 125 | } 126 | defer resp1.Body.Close() 127 | data, _ := ioutil.ReadAll(resp1.Body) 128 | 129 | var vault Vault 130 | json.Unmarshal(data, &vault) 131 | 132 | e.log.Info("Vault", "Auth Response", string(data)) 133 | e.log.Info("Vault", "Secret URL", VAULT_URL+VAULT_SECRET_DATA) 134 | //Vault Request - 2: [Retrieve Secret Data from Vault] 135 | res2, err := http.NewRequest("GET", VAULT_URL+VAULT_SECRET_DATA, nil) 136 | if err != nil { 137 | e.log.Error("Unable to make Get request for Vault Data", "error", err.Error()) 138 | return nil, err 139 | } 140 | res2.Header.Set("Content-Type", content_type) 141 | res2.Header.Set("X-Vault-Token", vault.Auth.Client_Token) 142 | resp2, err := http.DefaultClient.Do(res2) 143 | if err != nil { 144 | e.log.Error("Unable to fetch Vault Data", "error", err.Error()) 145 | } 146 | defer resp2.Body.Close() 147 | vaultData, err := ioutil.ReadAll(resp2.Body) 148 | if err != nil { 149 | e.log.Error("Unable to read Vault Data", "error", err.Error()) 150 | return nil, err 151 | } 152 | 153 | var vData VaultData 154 | json.Unmarshal(vaultData, &vData) 155 | e.log.Info("Vault", "Response Secret Data", string(vaultData)) 156 | 157 | return &vData.Data.Data, nil 158 | } 159 | 160 | func (e *EmployeeDB) mongoClient() (*mongo.Client, error) { 161 | 162 | e.log.Info("MongoDB Server Connect") 163 | 164 | secretData, err := e.GetMongoCredFromVault() 165 | if err != nil { 166 | e.log.Error("Unable to read secret data from Vault", "error", err.Error()) 167 | return nil, err 168 | } 169 | 170 | client, err := mongo.NewClient(options.Client().ApplyURI(MONGODB_CONNECTION_STRING). 171 | SetAuth(options.Credential{ 172 | Username: secretData.Username, Password: secretData.Password, 173 | })) 174 | 175 | if err != nil { 176 | e.log.Error("Unable to create server mongo client", "error", err.Error()) 177 | return nil, err 178 | } 179 | 180 | return client, nil 181 | } 182 | 183 | func (e *EmployeeDB) ConnectDB() (*mongo.Client, error) { 184 | 185 | e.log.Info("Connect to MongoDB") 186 | 187 | client, err := e.mongoClient() 188 | if err != nil { 189 | e.log.Error("Unable to create mongo client", "error", err.Error()) 190 | return nil, err 191 | } 192 | 193 | ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) 194 | defer cancel() 195 | err = client.Connect(ctx) 196 | if err != nil { 197 | e.log.Error("Unable to create mongo client", "error", err.Error()) 198 | return nil, err 199 | } 200 | 201 | e.log.Info("MongoDB Connected Successfully") 202 | 203 | return client, nil 204 | } 205 | 206 | func (e *EmployeeDB) GetCollection() (*mongo.Collection, error) { 207 | 208 | client, err := e.ConnectDB() 209 | if err != nil { 210 | e.log.Error("MongoDB", "Error", err.Error()) 211 | return nil, err 212 | } 213 | employeeCollection := client.Database(MONGODB_DATABASE).Collection(MONGODB_COLLECTION) 214 | 215 | return employeeCollection, nil 216 | } 217 | -------------------------------------------------------------------------------- /database/database_test.go: -------------------------------------------------------------------------------- 1 | package database 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/assert" 7 | ) 8 | 9 | func TestEmployeeIDReturnsErr(t *testing.T) { 10 | e := Employee{ 11 | ID: 55, 12 | } 13 | err := Validate(e) 14 | assert.Len(t, err, 1) 15 | } 16 | 17 | func TestEmployeeEmailReturnsErr(t *testing.T) { 18 | e := Employee{ 19 | Email: "abc.com", 20 | } 21 | err := Validate(e) 22 | assert.Len(t, err, 1) 23 | } 24 | 25 | func TestEmployeeDetailsReturnsErr(t *testing.T) { 26 | e := Employee{ 27 | ID: 345, 28 | Name: "Michale", 29 | Email: "abc@abc.com", 30 | } 31 | err := Validate(e) 32 | assert.Len(t, err, 1) 33 | } 34 | -------------------------------------------------------------------------------- /database/database_update_employees.go: -------------------------------------------------------------------------------- 1 | package database 2 | 3 | import ( 4 | "context" 5 | 6 | "go.mongodb.org/mongo-driver/bson" 7 | ) 8 | 9 | //UpdateEmployee 10 | func (e *EmployeeDB) UpdateEmployee(employee *Employee) error { 11 | 12 | collection, err := e.GetCollection() 13 | if err != nil { 14 | e.log.Error("Unable to create collection", "error", err.Error()) 15 | return err 16 | } 17 | 18 | _, err = collection.UpdateMany( 19 | context.TODO(), 20 | bson.M{MONGODB_COLLECTION_ID: employee.Email}, 21 | bson.D{ 22 | {"$set", bson.D{ 23 | {MONGODB_COLLECTION_ID, employee.ID}, 24 | {MONGODB_COLLECTION_NAME, employee.Name}, 25 | {MONGODB_COLLECTION_EMAIL, employee.Email}, 26 | {MONGODB_COLLECTION_COMPANY, employee.Company}, 27 | {MONGODB_COLLECTION_OCCUPATION, employee.Occupation}, 28 | {MONGODB_COLLECTION_SALARY, employee.Salary}}}, 29 | }, 30 | ) 31 | if err != nil { 32 | e.log.Error("Unable to update employee", "error", err.Error()) 33 | return err 34 | } 35 | 36 | //Update Redis Cache 37 | key := getKey(employee.ID) 38 | e.redisCache.Set(key, employee) 39 | 40 | return nil 41 | } 42 | -------------------------------------------------------------------------------- /database/database_validate.go: -------------------------------------------------------------------------------- 1 | package database 2 | 3 | import ( 4 | "fmt" 5 | 6 | "github.com/go-playground/validator/v10" 7 | ) 8 | 9 | type Validation struct { 10 | validate *validator.Validate 11 | } 12 | 13 | type ValidationError struct { 14 | validator.FieldError 15 | } 16 | 17 | type ValidationErrors []ValidationError 18 | 19 | func (v ValidationError) Error() string { 20 | return fmt.Sprintf( 21 | "validation_error: '%s' '%s'", 22 | v.Namespace(), 23 | v.Type(), 24 | ) 25 | } 26 | 27 | func (v ValidationErrors) Errors() []string { 28 | errs := []string{} 29 | for _, err := range v { 30 | errs = append(errs, err.Error()) 31 | } 32 | return errs 33 | } 34 | 35 | var validate *validator.Validate 36 | 37 | func (e *EmployeeDB) RequestValidator(data interface{}) ValidationErrors { 38 | return Validate(data) 39 | } 40 | 41 | func Validate(data interface{}) ValidationErrors { 42 | 43 | validate = validator.New() 44 | 45 | validateRefErr := validate.Struct(data) 46 | if validateRefErr != nil { 47 | validErrs := validateRefErr.(validator.ValidationErrors) 48 | 49 | if len(validErrs) == 0 { 50 | return nil 51 | } 52 | 53 | var validationErrs []ValidationError 54 | for _, err := range validErrs { 55 | 56 | ve := ValidationError{err.(validator.FieldError)} 57 | validationErrs = append(validationErrs, ve) 58 | } 59 | return validationErrs 60 | } 61 | 62 | return nil 63 | } 64 | -------------------------------------------------------------------------------- /database/redis_cache_manger.go: -------------------------------------------------------------------------------- 1 | package database 2 | 3 | import ( 4 | "context" 5 | "encoding/json" 6 | "fmt" 7 | 8 | "github.com/go-redis/redis/v8" 9 | "github.com/hashicorp/go-hclog" 10 | ) 11 | 12 | type RedisCache struct { 13 | redisClient *redis.Client 14 | log hclog.Logger 15 | } 16 | 17 | const ( 18 | LOCAL_ADDR = "172.17.0.2:31594" 19 | SERVER_ADDR = "redis.default.svc.cluster.local:6379" 20 | ) 21 | 22 | var ctx = context.Background() 23 | 24 | func InitializeCacheClient() (*RedisCache, error) { 25 | 26 | log := hclog.Default() 27 | 28 | log.Info("Initialize Redis", "Connect", LOCAL_ADDR) 29 | 30 | rdsClient := redis.NewClient(&redis.Options{ 31 | Addr: LOCAL_ADDR, 32 | Password: "", 33 | DB: 0, 34 | }) 35 | 36 | _, err := rdsClient.Ping(ctx).Result() 37 | if err != nil { 38 | log.Error("Failed to connect", "Redis", err.Error()) 39 | return nil, fmt.Errorf("Failed to connect to redis client - %s", err.Error()) 40 | } 41 | 42 | log.Info("Redis Connected", "Success", LOCAL_ADDR) 43 | 44 | return &RedisCache{ 45 | redisClient: rdsClient, 46 | log: log, 47 | }, nil 48 | } 49 | 50 | func (c *RedisCache) Set(key string, value *Employee) error { 51 | 52 | c.log.Info("Set Redis Cache", "Key", key) 53 | 54 | json, err := json.Marshal(value) 55 | if err != nil { 56 | c.log.Error("Failed to Marshal cache data", "Key", key, "Error", err.Error()) 57 | return err 58 | } 59 | 60 | err = c.redisClient.Set(ctx, key, json, 0).Err() 61 | if err != nil { 62 | c.log.Error("Set Redis Cache", "Key", key, "Error", err.Error()) 63 | return err 64 | } 65 | return nil 66 | } 67 | 68 | func (c *RedisCache) Get(key string) (*Employee, error) { 69 | 70 | c.log.Info("Get Redis Cache", "Key", key) 71 | 72 | value, err := c.redisClient.Get(ctx, key).Result() 73 | if err != nil { 74 | c.log.Error("Get Redis Cache", "Key", key, "Error", err.Error()) 75 | return nil, err 76 | } 77 | 78 | employee := Employee{} 79 | err = json.Unmarshal([]byte(value), &employee) 80 | 81 | if err != nil { 82 | c.log.Error("Unable to Unmarshal Redis cache", "Key", key, "Error", err.Error()) 83 | return nil, err 84 | } 85 | return &employee, nil 86 | } 87 | 88 | func (c *RedisCache) Del(key string) error { 89 | 90 | c.log.Info("Del Redis Cache", "Key", key) 91 | 92 | err := c.redisClient.Del(ctx, key).Err() 93 | if err != nil { 94 | c.log.Error("Unable to Del Redis Cache", "Key", key, "Error", err.Error()) 95 | return err 96 | } 97 | 98 | return nil 99 | } 100 | 101 | func getKey(id int) string { 102 | return fmt.Sprintf("%s%d", "Key-", id) 103 | } 104 | -------------------------------------------------------------------------------- /deploy_kubernetes/go_cache_app/backup.txt: -------------------------------------------------------------------------------- 1 | env: 2 | - name: SECRET_USERNAME 3 | valueFrom: 4 | secretKeyRef: 5 | name: mongosecret 6 | key: username 7 | - name: SECRET_PASSWORD 8 | valueFrom: 9 | secretKeyRef: 10 | name: mongosecret 11 | key: password -------------------------------------------------------------------------------- /deploy_kubernetes/go_cache_app/go-cache-poc-app.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: Deployment 3 | metadata: 4 | name: go-cache-kubernetes-app-poc 5 | labels: 6 | app: go-cache-kubernetes-app-poc 7 | spec: 8 | replicas: 1 9 | strategy: 10 | type: Recreate 11 | selector: 12 | matchLabels: 13 | app: go-cache-kubernetes-app-poc 14 | template: 15 | metadata: 16 | creationTimestamp: null 17 | labels: 18 | app: go-cache-kubernetes-app-poc 19 | spec: 20 | serviceAccountName: vault 21 | containers: 22 | - name: go-cache-kubernetes-container-poc 23 | image: deeptiman1991/go-cache-kubernetes-v1:1.0.0 24 | imagePullPolicy: Always 25 | env: 26 | - name: VAULT_ADDR 27 | value: "http://vault:8200" 28 | - name: JWT_PATH 29 | value: "/var/run/secrets/kubernetes.io/serviceaccount/token" 30 | - name: SERVICE_PORT 31 | value: "8080" 32 | ports: 33 | - containerPort: 5000 34 | hostname: go-cache-kubernetes-app-poc 35 | restartPolicy: Always 36 | volumes: 37 | - name: go-cache-kubernetes-pvc-v1-poc 38 | persistentVolumeClaim: 39 | claimName: go-cache-kubernetes-pvc-v1-poc 40 | status: {} -------------------------------------------------------------------------------- /deploy_kubernetes/go_cache_app/go-cache-poc-pvc.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: PersistentVolumeClaim 3 | metadata: 4 | creationTimestamp: null 5 | labels: 6 | app: go-cache-kubernetes-pvc-v1-poc 7 | name: go-cache-kubernetes-pvc-v1-poc 8 | spec: 9 | accessModes: 10 | - ReadWriteOnce 11 | resources: 12 | requests: 13 | storage: 1Gi 14 | status: {} 15 | -------------------------------------------------------------------------------- /deploy_kubernetes/go_cache_app/go-cache-poc-svc.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: go-cache-kubernetes-svc-poc 5 | spec: 6 | ports: 7 | - name: "5000" 8 | port: 5000 9 | targetPort: 5000 10 | selector: 11 | app: go-cache-kubernetes-app-poc 12 | type: LoadBalancer 13 | status: 14 | loadBalancer: {} -------------------------------------------------------------------------------- /deploy_kubernetes/kafka/deploy/kafka-manager-deployment.yml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: Deployment 3 | metadata: 4 | name: kafka-manager 5 | namespace: kafka-ca1 6 | spec: 7 | replicas: 1 8 | selector: 9 | matchLabels: 10 | app: kafka-manager 11 | template: 12 | metadata: 13 | labels: 14 | app: kafka-manager 15 | spec: 16 | containers: 17 | - name: kafka-manager 18 | image: sheepkiller/kafka-manager@sha256:615f3b99d38aba2d5fdb3fb750a5990ba9260c8fb3fd29c7e776e8c150518b78 19 | ports: 20 | - containerPort: 9000 21 | env: 22 | - name: ZK_HOSTS 23 | value: "zookeeper-service:2181" -------------------------------------------------------------------------------- /deploy_kubernetes/kafka/deploy/kafka-manager-service.yml: -------------------------------------------------------------------------------- 1 | kind: Service 2 | apiVersion: v1 3 | metadata: 4 | name: kafka-manager 5 | namespace: kafka-ca1 6 | spec: 7 | type: NodePort 8 | selector: 9 | app: kafka-manager 10 | ports: 11 | - port: 9000 12 | protocol: TCP 13 | targetPort: 9000 -------------------------------------------------------------------------------- /deploy_kubernetes/kafka/deploy/kafka-service.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: Service 4 | metadata: 5 | name: kafka 6 | namespace: kafka-ca1 7 | labels: 8 | app: kafka 9 | spec: 10 | ports: 11 | - port: 9092 12 | name: plaintext 13 | - port: 9999 14 | name: jmx 15 | clusterIP: None 16 | selector: 17 | app: kafka -------------------------------------------------------------------------------- /deploy_kubernetes/kafka/deploy/kafka-statefulset.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: apps/v1 3 | kind: StatefulSet 4 | metadata: 5 | name: kafka 6 | namespace: kafka-ca1 7 | spec: 8 | selector: 9 | matchLabels: 10 | app: kafka 11 | serviceName: "kafka" 12 | replicas: 3 13 | podManagementPolicy: OrderedReady 14 | template: 15 | metadata: 16 | labels: 17 | app: kafka # has to match .spec.selector.matchLabels 18 | spec: 19 | containers: 20 | - name: kafka 21 | image: wurstmeister/kafka:2.11-2.0.0 22 | imagePullPolicy: IfNotPresent 23 | ports: 24 | - containerPort: 9092 25 | name: plaintext 26 | - containerPort: 9999 27 | name: jmx 28 | env: 29 | - name: KAFKA_ADVERTISED_PORT 30 | value: "9092" 31 | - name: BROKER_ID_COMMAND 32 | value: "hostname | cut -d'-' -f2" 33 | - name: KAFKA_ZOOKEEPER_CONNECT 34 | value: "zookeeper-service:2181" 35 | - name: KAFKA_LISTENERS 36 | value: "PLAINTEXT://:9092" 37 | - name: KAFKA_JMX_OPTS 38 | value: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=9999 -Djava.rmi.server.hostname=127.0.0.1" 39 | - name: JMX_PORT 40 | value: "9999" -------------------------------------------------------------------------------- /deploy_kubernetes/kafka/deploy/namespace.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: Namespace 4 | metadata: 5 | name: kafka-ca1 -------------------------------------------------------------------------------- /deploy_kubernetes/kafka/deploy/zookeeper-deployment.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: apps/v1 3 | kind: Deployment 4 | metadata: 5 | labels: 6 | app: zookeeper 7 | name: zookeeper 8 | namespace: kafka-ca1 9 | spec: 10 | replicas: 1 11 | selector: 12 | matchLabels: 13 | app: zookeeper 14 | template: 15 | metadata: 16 | labels: 17 | app: zookeeper 18 | spec: 19 | containers: 20 | - image: library/zookeeper:3.4.13 21 | imagePullPolicy: IfNotPresent 22 | name: zookeeper 23 | ports: 24 | - containerPort: 2181 25 | env: 26 | - name: ZOO_MY_ID 27 | value: "1" -------------------------------------------------------------------------------- /deploy_kubernetes/kafka/deploy/zookeeper-service.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: Service 4 | metadata: 5 | labels: 6 | app: zookeeper-service 7 | name: zookeeper-service 8 | namespace: kafka-ca1 9 | spec: 10 | type: NodePort 11 | ports: 12 | - name: zookeeper-port 13 | port: 2181 14 | # nodePort: 30181 15 | targetPort: 2181 16 | selector: 17 | app: zookeeper 18 | -------------------------------------------------------------------------------- /deploy_kubernetes/kafka/kafka-repcon.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ReplicationController 3 | metadata: 4 | name: kafka-repcon 5 | labels: 6 | app: kafka-repcon 7 | spec: 8 | replicas: 1 9 | selector: 10 | app: kafka-repcon 11 | template: 12 | metadata: 13 | labels: 14 | app: kafka-repcon 15 | spec: 16 | containers: 17 | - name: zookeeper-app-container 18 | image: wurstmeister/kafka 19 | command: 20 | - /bin/zookeeper-server-start.sh 21 | - /config/zookeeper.properties 22 | ports: 23 | - containerPort: 2181 24 | -------------------------------------------------------------------------------- /deploy_kubernetes/kafka/kafka-service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: kafka-service 5 | labels: 6 | name: kafka-service 7 | spec: 8 | ports: 9 | - name: "9092" 10 | port: 9092 11 | targetPort: 9092 12 | protocol: TCP 13 | - name: "2181" 14 | port: 2181 15 | targetPort: 2181 16 | selector: 17 | app: kafka-service 18 | type: LoadBalancer 19 | -------------------------------------------------------------------------------- /deploy_kubernetes/kafka/zookeeper-deployment.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: Deployment 3 | metadata: 4 | name: zookeeper-app 5 | spec: 6 | replicas: 1 7 | template: 8 | metadata: 9 | labels: 10 | app: zookeeper-app 11 | role: master 12 | tier: backend 13 | spec: 14 | containers: 15 | - name: zookeeper-app-container 16 | image: bitnami/zookeeper 17 | ports: 18 | - containerPort: 2181 19 | env: 20 | - name: ZOOKEEPER_ID 21 | value: "1" 22 | - name: ZOOKEEPER_SERVER_1 23 | value: zookeeper-app 24 | -------------------------------------------------------------------------------- /deploy_kubernetes/kafka/zookeeper-service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: zookeeper-service 5 | labels: 6 | app: zookeeper-service 7 | spec: 8 | ports: 9 | - name: client 10 | port: 2181 11 | protocol: TCP 12 | - name: follower 13 | port: 2888 14 | protocol: TCP 15 | - name: leader 16 | port: 3888 17 | protocol: TCP 18 | selector: 19 | app: zookeeper-service 20 | -------------------------------------------------------------------------------- /deploy_kubernetes/mongodb/mongodb-service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: mongodb-service 5 | labels: 6 | name: mongo 7 | spec: 8 | ports: 9 | - port: 27017 10 | targetPort: 27017 11 | clusterIP: None 12 | selector: 13 | role: mongo -------------------------------------------------------------------------------- /deploy_kubernetes/mongodb/mongodb-statefulset.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: StatefulSet 3 | metadata: 4 | name: mongod 5 | spec: 6 | serviceName: mongodb-service 7 | replicas: 3 8 | selector: 9 | matchLabels: 10 | role: mongo 11 | template: 12 | metadata: 13 | labels: 14 | role: mongo 15 | environment: test 16 | replicaset: MainRepSet 17 | spec: 18 | terminationGracePeriodSeconds: 10 19 | containers: 20 | - name: mongod-container 21 | image: mongo 22 | command: 23 | - "mongod" 24 | - "--bind_ip" 25 | - "0.0.0.0" 26 | - "--replSet" 27 | - "MainRepSet" 28 | resources: 29 | requests: 30 | cpu: 0.2 31 | memory: 200Mi 32 | ports: 33 | - containerPort: 27017 34 | volumeMounts: 35 | - name: mongodb-persistent-storage-claim 36 | mountPath: /data/db 37 | volumeClaimTemplates: 38 | - metadata: 39 | name: mongodb-persistent-storage-claim 40 | annotations: 41 | volume.beta.kubernetes.io/storage-class: "standard" 42 | spec: 43 | accessModes: [ "ReadWriteOnce" ] 44 | resources: 45 | requests: 46 | storage: 1Gi 47 | -------------------------------------------------------------------------------- /deploy_kubernetes/redis/redis-deployment.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: Deployment 3 | metadata: 4 | name: redismod 5 | spec: 6 | replicas: 1 7 | selector: 8 | matchLabels: 9 | app: redismod 10 | template: 11 | metadata: 12 | labels: 13 | app: redismod 14 | spec: 15 | containers: 16 | - name: redismod 17 | image: redislabs/redismod 18 | resources: 19 | requests: 20 | cpu: 100m 21 | memory: 100Mi 22 | ports: 23 | - containerPort: 6379 24 | securityContext: 25 | capabilities: 26 | add: 27 | - SYS_RESOURCE 28 | -------------------------------------------------------------------------------- /deploy_kubernetes/redis/redis-service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: redis-service 5 | labels: 6 | app: redis-service 7 | spec: 8 | ports: 9 | - port: 6379 10 | targetPort: 6379 11 | selector: 12 | app: redismod -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '1' 2 | 3 | services: 4 | 5 | go-cache-poc: 6 | container_name: go-cache-kubernetes-container-poc 7 | image: deeptiman1991/go-cache-kubernetes-v1:1.0.0 8 | hostname: go-cache-kubernetes-container-poc 9 | build: 10 | context: . 11 | dockerfile: Dockerfile 12 | environment: 13 | GET_HOST_FROM: dns 14 | networks: 15 | - go-cache-poc 16 | volumes: 17 | - .:/go/src/go-cache-kubernetes 18 | ports: 19 | - 5000:5000 20 | labels: 21 | kompose.service.type: LoadBalancer 22 | 23 | networks: 24 | go-cache-poc: 25 | driver: bridge 26 | -------------------------------------------------------------------------------- /handlers/handler_create_employee.go: -------------------------------------------------------------------------------- 1 | package handlers 2 | 3 | import ( 4 | "net/http" 5 | 6 | "go-cache-kubernetes/database" 7 | ) 8 | 9 | // swagger:route POST /employees create_employee 10 | // Request to create new employee 11 | // 12 | // responses: 13 | // 200: employeeResponse 14 | // 422: errorValidation 15 | // 501: errorResponse 16 | // POST request CreateEmployee 17 | func (h *Handlers) CreateEmployee(rw http.ResponseWriter, r *http.Request) { 18 | 19 | data := r.Context().Value(KeyEmp{}).(*database.Employee) 20 | 21 | h.log.Info("Insert Employee", "Create", data) 22 | 23 | err := h.database.CreateEmployee(data) 24 | if err != nil { 25 | h.respondJSON(rw, http.StatusBadRequest, &ServerError{Error: err.Error()}) 26 | return 27 | } 28 | 29 | h.respondJSON(rw, http.StatusCreated, map[string]string{"success": "Employee created successfully"}) 30 | } 31 | -------------------------------------------------------------------------------- /handlers/handler_delete_employee.go: -------------------------------------------------------------------------------- 1 | package handlers 2 | 3 | import ( 4 | "net/http" 5 | 6 | "go-cache-kubernetes/database" 7 | ) 8 | 9 | // swagger:route DELETE /api/{id} employees delete_employee 10 | // Delete the employee details using the id 11 | // responses: 12 | // 201: noContentResponse 13 | // 404: errorResponse 14 | // GET request GetEmployeeByID 15 | func (h *Handlers) DeleteEmployeeByID(rw http.ResponseWriter, r *http.Request) { 16 | 17 | id := getEmployeeID(r) 18 | 19 | h.log.Info("DeleteEmployeeByID", "id", id) 20 | 21 | _, err := h.database.GetEmployeeByID(id) 22 | 23 | switch err { 24 | case nil: 25 | case database.ErrEmployeeNotFound: 26 | h.log.Error("Unable to find employee", "error", err) 27 | h.respondJSON(rw, http.StatusNotFound, &ServerError{Error: err.Error()}) 28 | return 29 | default: 30 | h.log.Error("Error while find employee", "error", err) 31 | h.respondJSON(rw, http.StatusInternalServerError, &ServerError{Error: err.Error()}) 32 | return 33 | } 34 | 35 | err = h.database.DeleteEmployeeByID(id) 36 | if err != nil { 37 | h.respondJSON(rw, http.StatusBadRequest, &ServerError{Error: err.Error()}) 38 | return 39 | } 40 | 41 | h.log.Info("DeleteEmployeeByID", "Deleted", id) 42 | 43 | h.respondJSON(rw, http.StatusCreated, map[string]string{"success": "Employee deleted successfully"}) 44 | } 45 | -------------------------------------------------------------------------------- /handlers/handler_get_employees.go: -------------------------------------------------------------------------------- 1 | package handlers 2 | 3 | import ( 4 | "net/http" 5 | 6 | "go-cache-kubernetes/database" 7 | ) 8 | 9 | // swagger:route GET /api employees get_all_employees 10 | // Return the list of employees details 11 | // responses: 12 | // 200: employeeResponse 13 | // 404: errorResponse 14 | // GET request GetAllEmployees 15 | func (h *Handlers) GetAllEmployees(rw http.ResponseWriter, r *http.Request) { 16 | 17 | h.log.Info("GetAllEmployees Request") 18 | employees, err := h.database.GetAllEmployees() 19 | if err != nil { 20 | h.respondJSON(rw, http.StatusInternalServerError, &ServerError{Error: err.Error()}) 21 | return 22 | } 23 | h.respondJSON(rw, http.StatusOK, employees) 24 | } 25 | 26 | // swagger:route GET /api/{id} employees get_employee_by_id 27 | // Return the employee details for the id 28 | // responses: 29 | // 200: employeeResponse 30 | // 404: errorResponse 31 | // GET request GetEmployeeByID 32 | func (h *Handlers) GetEmployeeByID(rw http.ResponseWriter, r *http.Request) { 33 | 34 | id := getEmployeeID(r) 35 | 36 | h.log.Info("GetEmployeeByID", "id", id) 37 | 38 | employee, err := h.database.GetEmployeeByID(id) 39 | 40 | switch err { 41 | case nil: 42 | case database.ErrEmployeeNotFound: 43 | h.log.Error("Unable to find employee", "error", err) 44 | h.respondJSON(rw, http.StatusNotFound, &ServerError{Error: err.Error()}) 45 | return 46 | default: 47 | h.log.Error("Error while find employee", "error", err) 48 | h.respondJSON(rw, http.StatusInternalServerError, &ServerError{Error: err.Error()}) 49 | return 50 | } 51 | 52 | h.respondJSON(rw, http.StatusOK, employee) 53 | } 54 | -------------------------------------------------------------------------------- /handlers/handler_manager.go: -------------------------------------------------------------------------------- 1 | package handlers 2 | 3 | import ( 4 | "go-cache-kubernetes/database" 5 | 6 | "github.com/hashicorp/go-hclog" 7 | ) 8 | 9 | type Handlers struct { 10 | database *database.EmployeeDB 11 | log hclog.Logger 12 | } 13 | 14 | type KeyEmp struct{} 15 | 16 | // The structure containing server error msgs 17 | type ServerError struct { 18 | Error string `json:"error"` 19 | } 20 | 21 | // The structure containing validation error msgs 22 | type ValidationErrorMsg struct { 23 | Error []string `json:"error"` 24 | } 25 | 26 | func InitializeHandlers(database *database.EmployeeDB, log hclog.Logger) *Handlers { 27 | return &Handlers{database, log} 28 | } 29 | -------------------------------------------------------------------------------- /handlers/handler_response_validator.go: -------------------------------------------------------------------------------- 1 | package handlers 2 | 3 | import ( 4 | "context" 5 | "encoding/json" 6 | "io" 7 | "net/http" 8 | "strconv" 9 | 10 | "go-cache-kubernetes/database" 11 | 12 | "github.com/gorilla/mux" 13 | ) 14 | 15 | func (h *Handlers) ResponseValidator(request http.Handler) http.Handler { 16 | 17 | return http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) { 18 | 19 | var emp *database.Employee 20 | 21 | err := DecodeJSON(r.Body, &emp) 22 | if err != nil { 23 | h.respondJSON(rw, http.StatusBadRequest, &ServerError{Error: err.Error()}) 24 | return 25 | } 26 | 27 | //validate employee record 28 | validationErrors := h.database.RequestValidator(emp) 29 | if len(validationErrors) != 0 { 30 | h.respondJSON(rw, http.StatusUnprocessableEntity, &ValidationErrorMsg{Error: validationErrors.Errors()}) 31 | return 32 | } 33 | 34 | ctx := context.WithValue(r.Context(), KeyEmp{}, emp) 35 | r = r.WithContext(ctx) 36 | 37 | request.ServeHTTP(rw, r) 38 | }) 39 | } 40 | 41 | func (h *Handlers) respondJSON(rw http.ResponseWriter, status int, payload interface{}) { 42 | 43 | h.log.Info("respondJSON", "status", status) 44 | rw.Header().Add("Content-Type", "application/json") 45 | rw.WriteHeader(status) 46 | EncodeJSON(rw, payload) 47 | } 48 | 49 | func DecodeJSON(reader io.Reader, res interface{}) error { 50 | return json.NewDecoder(reader).Decode(res) 51 | } 52 | 53 | func EncodeJSON(writer io.Writer, res interface{}) error { 54 | return json.NewEncoder(writer).Encode(res) 55 | } 56 | 57 | func getEmployeeID(r *http.Request) int { 58 | vars := mux.Vars(r) 59 | id, _ := strconv.Atoi(vars["id"]) 60 | return id 61 | } 62 | -------------------------------------------------------------------------------- /handlers/handler_update_employee.go: -------------------------------------------------------------------------------- 1 | package handlers 2 | 3 | import ( 4 | "net/http" 5 | 6 | "go-cache-kubernetes/database" 7 | ) 8 | 9 | // swagger:route PUT /api/{id} employees update_employee 10 | // Return the updated employee details for the id 11 | // responses: 12 | // 200: employeeResponse 13 | // 404: errorResponse 14 | // PUT request UpdateEmployee 15 | func (h *Handlers) UpdateEmployee(rw http.ResponseWriter, r *http.Request) { 16 | 17 | data := r.Context().Value(KeyEmp{}).(*database.Employee) 18 | 19 | h.log.Info("UpdateEmployee", "id", data.ID) 20 | 21 | _, err := h.database.GetEmployeeByID(data.ID) 22 | 23 | switch err { 24 | case nil: 25 | case database.ErrEmployeeNotFound: 26 | h.log.Error("Unable to find employee", "error", err) 27 | h.respondJSON(rw, http.StatusNotFound, &ServerError{Error: err.Error()}) 28 | return 29 | default: 30 | h.log.Error("Error while find employee", "error", err) 31 | h.respondJSON(rw, http.StatusInternalServerError, &ServerError{Error: err.Error()}) 32 | return 33 | } 34 | 35 | h.log.Info("UpdateEmployee", "Employee", data) 36 | 37 | err = h.database.UpdateEmployee(data) 38 | if err != nil { 39 | h.log.Error("Unable to update employee", "error", err) 40 | h.respondJSON(rw, http.StatusBadRequest, &ServerError{Error: err.Error()}) 41 | return 42 | } 43 | 44 | h.respondJSON(rw, http.StatusCreated, map[string]string{"success": "Employee record updated successfully"}) 45 | } 46 | -------------------------------------------------------------------------------- /kafka/kafka_consumer.go: -------------------------------------------------------------------------------- 1 | package kafka 2 | 3 | import ( 4 | "encoding/json" 5 | "fmt" 6 | "net/http" 7 | 8 | "go-cache-kubernetes/database" 9 | 10 | "github.com/confluentinc/confluent-kafka-go/kafka" 11 | "github.com/hashicorp/go-hclog" 12 | ) 13 | 14 | const ( 15 | TOPIC = "EmployeesList" 16 | ) 17 | 18 | type KafkaConsumer struct { 19 | consumer *kafka.Consumer 20 | dbRef *database.EmployeeDB 21 | log hclog.Logger 22 | } 23 | 24 | func InitializeKafkaConsumer() *KafkaConsumer { 25 | 26 | c, _ := NewConsumer() 27 | log := hclog.Default() 28 | return &KafkaConsumer{ 29 | consumer: c, 30 | log: log, 31 | } 32 | } 33 | 34 | func NewConsumer() (*kafka.Consumer, error) { 35 | 36 | consumer, err := kafka.NewConsumer(&kafka.ConfigMap{ 37 | "bootstrap.servers": "localhost:9092", 38 | "broker.address.family": "v4", 39 | "group.id": "employees", 40 | "auto.offset.reset": "smallest", 41 | }) 42 | 43 | if err != nil { 44 | return nil, fmt.Errorf("Failed to initialize Kafka Consumer %s", err.Error()) 45 | } 46 | 47 | return consumer, nil 48 | } 49 | 50 | func (k *KafkaConsumer) ReadMessages(rw http.ResponseWriter, r *http.Request) { 51 | 52 | k.log.Info("Kafka Consumer", "Topic", TOPIC) 53 | 54 | k.consumer.SubscribeTopics([]string{TOPIC, "^aRegex.*[Tt]opic"}, nil) 55 | 56 | for { 57 | msg, err := k.consumer.ReadMessage(-1) 58 | if err == nil { 59 | var employee *database.Employee 60 | emp := []byte(string(msg.Value)) 61 | err = json.Unmarshal(emp, &employee) 62 | if err != nil { 63 | k.log.Error("Failed to read msg", "error", err.Error()) 64 | } 65 | k.log.Info("Kafka Consumer", "ReadMessage", string(msg.Value)) 66 | } else { 67 | // The client will automatically try to recover from all errors. 68 | fmt.Printf("Consumer error: %v (%v)\n", err, msg) 69 | k.log.Error("Kafka Consumer", "Error", err, "Msg", msg) 70 | } 71 | } 72 | } 73 | -------------------------------------------------------------------------------- /kafka/kafka_producer.go: -------------------------------------------------------------------------------- 1 | package kafka 2 | 3 | import ( 4 | "encoding/json" 5 | "fmt" 6 | "net/http" 7 | 8 | "go-cache-kubernetes/database" 9 | 10 | "github.com/confluentinc/confluent-kafka-go/kafka" 11 | "github.com/hashicorp/go-hclog" 12 | ) 13 | 14 | type KafkaProducer struct { 15 | producer *kafka.Producer 16 | dbRef *database.EmployeeDB 17 | log hclog.Logger 18 | } 19 | 20 | func InitializeKafkaProducer(database *database.EmployeeDB) *KafkaProducer { 21 | 22 | p, _ := NewProducer() 23 | log := hclog.Default() 24 | return &KafkaProducer{ 25 | producer: p, 26 | dbRef: database, 27 | log: log, 28 | } 29 | } 30 | 31 | func NewProducer() (*kafka.Producer, error) { 32 | 33 | producer, err := kafka.NewProducer(&kafka.ConfigMap{"bootstrap.servers": "localhost:9092"}) 34 | if err != nil { 35 | return nil, fmt.Errorf("Failed to initialize Kafka Producer %s", err.Error()) 36 | } 37 | 38 | return producer, nil 39 | } 40 | 41 | func (k *KafkaProducer) eventDelivery() { 42 | 43 | go func() { 44 | for event := range k.producer.Events() { 45 | switch ev := event.(type) { 46 | case *kafka.Message: 47 | if ev.TopicPartition.Error != nil { 48 | k.log.Error("Kafka Producer", "Delivery failed", ev.TopicPartition, "Error", ev.TopicPartition.Error) 49 | } else { 50 | k.log.Info("Kafka Producer", "Delivery message", ev.TopicPartition) 51 | } 52 | } 53 | } 54 | }() 55 | } 56 | 57 | func (k *KafkaProducer) ProduceMessages(rw http.ResponseWriter, r *http.Request) { 58 | 59 | topic := TOPIC 60 | 61 | k.log.Info("Kafka Producer", "Topic", TOPIC) 62 | 63 | employeesList, err := k.dbRef.GetAllEmployees() 64 | if err != nil { 65 | k.log.Error("Kafka Producer", "error", err.Error()) 66 | return 67 | } 68 | 69 | for _, employee := range employeesList { 70 | 71 | k.log.Info("Kafka Producer", "Employee", employee) 72 | 73 | json, err := json.Marshal(employee) 74 | if err != nil { 75 | k.log.Error("Kafka Producer", "error", err.Error()) 76 | return 77 | } 78 | 79 | k.producer.Produce(&kafka.Message{ 80 | TopicPartition: kafka.TopicPartition{ 81 | Topic: &topic, 82 | Partition: kafka.PartitionAny, 83 | }, 84 | Value: []byte(json), 85 | }, nil) 86 | } 87 | } 88 | -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "net/http" 5 | "os" 6 | "time" 7 | 8 | "go-cache-kubernetes/database" 9 | "go-cache-kubernetes/handlers" 10 | "go-cache-kubernetes/kafka" 11 | 12 | "github.com/go-openapi/runtime/middleware" 13 | gohandlers "github.com/gorilla/handlers" 14 | "github.com/gorilla/mux" 15 | "github.com/hashicorp/go-hclog" 16 | ) 17 | 18 | var address *string 19 | 20 | func main() { 21 | 22 | PORT := ":5000" 23 | address = &PORT 24 | log := hclog.Default() 25 | 26 | db := database.InitializeDBManager(log) 27 | kafkaProducer := kafka.InitializeKafkaProducer(db) 28 | kafkaConsumer := kafka.InitializeKafkaConsumer() 29 | handlers := handlers.InitializeHandlers(db, log) 30 | 31 | router := mux.NewRouter() 32 | 33 | createRequest := router.Methods(http.MethodPost).Subrouter() 34 | createRequest.HandleFunc("/api/create_employee", handlers.CreateEmployee) 35 | createRequest.Use(handlers.ResponseValidator) 36 | 37 | getRequest := router.Methods(http.MethodGet).Subrouter() 38 | getRequest.HandleFunc("/api/get_employee_by_id/{id:[0-9]+}", handlers.GetEmployeeByID) 39 | getRequest.HandleFunc("/api/get_all_employees", handlers.GetAllEmployees) 40 | 41 | updateRequest := router.Methods(http.MethodPut).Subrouter() 42 | updateRequest.HandleFunc("/api/update_employee", handlers.UpdateEmployee) 43 | updateRequest.Use(handlers.ResponseValidator) 44 | 45 | deleteRequest := router.Methods(http.MethodDelete).Subrouter() 46 | deleteRequest.HandleFunc("/api/delete_employee/{id:[0-9]+}", handlers.DeleteEmployeeByID) 47 | 48 | getRequest.HandleFunc("/kafka/producer", kafkaProducer.ProduceMessages) 49 | getRequest.HandleFunc("/kafka/consumer", kafkaConsumer.ReadMessages) 50 | 51 | op := middleware.RedocOpts{SpecURL: "/swagger.yaml"} 52 | redoc := middleware.Redoc(op, nil) 53 | 54 | getRequest.Handle("/docs", redoc) 55 | getRequest.Handle("/swagger.yaml", http.FileServer(http.Dir("./"))) 56 | 57 | cors := gohandlers.CORS(gohandlers.AllowedOrigins([]string{"*"})) 58 | 59 | server := http.Server{ 60 | Addr: PORT, 61 | Handler: cors(router), 62 | ErrorLog: log.StandardLogger(&hclog.StandardLoggerOptions{}), 63 | ReadTimeout: 5 * time.Second, 64 | WriteTimeout: 10 * time.Second, 65 | IdleTimeout: 120 * time.Second, 66 | } 67 | 68 | log.Info("Starting server on port 5000") 69 | 70 | err := server.ListenAndServe() 71 | if err != nil { 72 | log.Error("Unable to start server", "error", err) 73 | os.Exit(1) 74 | } 75 | } 76 | -------------------------------------------------------------------------------- /swagger.yaml: -------------------------------------------------------------------------------- 1 | basePath: / 2 | consumes: 3 | - application/json 4 | definitions: 5 | ServerError: 6 | description: ServerError error message returned by a server 7 | properties: 8 | message: 9 | description: message 10 | type: string 11 | x-go-name: Message 12 | type: object 13 | x-go-package: github.com/Deeptiman/go-cache-kubernetes/database 14 | ValidationError: 15 | description: ValidationError is a collection of validation error messages 16 | properties: 17 | messages: 18 | description: messages 19 | items: 20 | type: string 21 | type: array 22 | x-go-name: Messages 23 | type: object 24 | x-go-package: github.com/Deeptiman/go-cache-kubernetes/database 25 | Employee: 26 | description: Employee defines the structure for an API employee 27 | properties: 28 | id: 29 | description: the id of an employee 30 | format: int 31 | minimum: 100 32 | type: integer 33 | x-go-name: ID 34 | name: 35 | description: the name of an employee 36 | maxLength: 255 37 | type: string 38 | x-go-name: Name 39 | email: 40 | description: the email of an employee 41 | pattern: 'abc@abc.abc' 42 | maxLength: 255 43 | type: string 44 | x-go-name: Email 45 | company: 46 | description: the company name where the employee is working 47 | maxLength: 255 48 | type: string 49 | x-go-name: Company 50 | occupation: 51 | description: the occupation of an employee 52 | maxLength: 255 53 | type: string 54 | x-go-name: Occupation 55 | salary: 56 | description: the salary of an employee 57 | minimum: 9999 58 | type: int 59 | x-go-name: Salary 60 | required: 61 | - id 62 | - name 63 | - email 64 | - company 65 | - occupation 66 | - salary 67 | type: object 68 | x-go-package: github.com/Deeptiman/go-cache-kubernetes/database 69 | Success: 70 | description: Employee created successfully 71 | properties: 72 | message: 73 | description: the employee created successfully 74 | maxLength: 255 75 | type: string 76 | x-go-name: Name 77 | type: object 78 | x-go-package: github.com/Deeptiman/go-cache-kubernetes/database 79 | info: 80 | description: Documentation for Employee API 81 | title: Employee API 82 | version: 1.0.0 83 | paths: 84 | /api: 85 | get: 86 | description: Return a list of employees from the database 87 | operationId: get_all_employees 88 | responses: 89 | "200": 90 | $ref: '#/responses/employeesResponse' 91 | "501": 92 | $ref: '#/responses/errorResponse' 93 | tags: 94 | - employees 95 | post: 96 | description: Create a new employee 97 | operationId: create_employee 98 | parameters: 99 | - description: |- 100 | Employee data structure to Update or Create. 101 | Note: the id field is ignored by update and create operations 102 | in: body 103 | name: Body 104 | required: true 105 | schema: 106 | $ref: '#/definitions/Employee' 107 | responses: 108 | "200": 109 | $ref: '#/responses/successResponse' 110 | "422": 111 | $ref: '#/responses/errorValidation' 112 | "501": 113 | $ref: '#/responses/errorResponse' 114 | tags: 115 | - employees 116 | put: 117 | description: Update a employee details 118 | operationId: update_employee 119 | parameters: 120 | - description: |- 121 | Employee data structure to Update or Create. 122 | Note: the id field is ignored by update and create operations 123 | in: body 124 | name: Body 125 | required: true 126 | schema: 127 | $ref: '#/definitions/Employee' 128 | responses: 129 | "201": 130 | $ref: '#/responses/noContentResponse' 131 | "404": 132 | $ref: '#/responses/errorResponse' 133 | "422": 134 | $ref: '#/responses/errorValidation' 135 | tags: 136 | - employees 137 | /employees/{id}: 138 | delete: 139 | description: Delete an employee details 140 | operationId: delete_employee 141 | parameters: 142 | - description: The id of the employee for which the operation relates 143 | format: int 144 | in: path 145 | name: id 146 | required: true 147 | type: integer 148 | x-go-name: ID 149 | responses: 150 | "201": 151 | $ref: '#/responses/noContentResponse' 152 | "404": 153 | $ref: '#/responses/errorResponse' 154 | "501": 155 | $ref: '#/responses/errorResponse' 156 | tags: 157 | - employees 158 | get: 159 | description: Return a list of employees from the database 160 | operationId: get_employee_by_email 161 | parameters: 162 | - description: Email id of the employee 163 | in: query 164 | name: Email 165 | type: string 166 | responses: 167 | "200": 168 | $ref: '#/responses/employeeResponse' 169 | "404": 170 | $ref: '#/responses/errorResponse' 171 | tags: 172 | - employees 173 | /kafka: 174 | get: 175 | description: Request to reload list of employees using a Kafka message broker 176 | operationId: producer 177 | responses: 178 | "200": 179 | $ref: '#/responses/noContentResponse' 180 | tags: 181 | - employees 182 | produces: 183 | - application/json 184 | responses: 185 | errorResponse: 186 | description: Server error message returned as a string 187 | schema: 188 | $ref: '#/definitions/ServerError' 189 | errorValidation: 190 | description: Validation errors defined as an array of strings 191 | schema: 192 | $ref: '#/definitions/ValidationError' 193 | noContentResponse: 194 | description: No content is returned by this API endpoint 195 | successResponse: 196 | description: Success response defined as strings 197 | schema: 198 | $ref: '#/definitions/Success' 199 | employeeResponse: 200 | description: Data structure representing a single employee 201 | schema: 202 | $ref: '#/definitions/Employee' 203 | employeesResponse: 204 | description: A list of employees 205 | schema: 206 | items: 207 | $ref: '#/definitions/Employee' 208 | type: array 209 | schemes: 210 | - http 211 | swagger: "2.0" 212 | --------------------------------------------------------------------------------