├── .gitignore ├── CODE-OF-CONDUCT.md ├── README.md ├── api ├── Dockerfile ├── docker-compose.yaml ├── go.mod ├── go.sum └── main.go ├── docker-compose.yaml ├── example.env ├── grafana ├── Dockerfile └── docker-compose.yaml ├── loki └── local-config.yaml ├── minio ├── Dockerfile ├── backups │ └── backup.sh ├── docker-compose.yaml └── setup.sh ├── mongodb ├── Dockerfile ├── backups │ └── backup.sh ├── docker-compose.yaml └── init.js └── promtail └── config.yml /.gitignore: -------------------------------------------------------------------------------- 1 | .env -------------------------------------------------------------------------------- /CODE-OF-CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Code of Conduct 2 | 3 | All participants of ShimmyTheDev are expected to abide by our Code of Conduct, both online and during in-person events that are hosted and/or associated with ShimmyTheDev. 4 | 5 | 6 | ### The Pledge 7 | 8 | In the interest of fostering an open and welcoming environment, we pledge to make participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation. 9 | The Standards 10 | 11 | Examples of behaviour that contributes to creating a positive environment include: 12 | 13 | - Using welcoming and inclusive language 14 | - Being respectful of differing viewpoints and experiences 15 | - Gracefully accepting constructive criticism 16 | - Referring to people by their preferred pronouns and using gender-neutral pronouns when uncertain 17 | 18 | Examples of unacceptable behaviour by participants include: 19 | 20 | - Trolling, insulting/derogatory comments, public or private harassment 21 | - Publishing others' private information, such as a physical or electronic address, without explicit permission 22 | - Not being respectful to reasonable communication boundaries, such as 'leave me alone,' 'go away,' or 'I’m not discussing this with you.' 23 | - The usage of sexualised language or imagery and unwelcome sexual attention or advances 24 | - Swearing, usage of strong or disturbing language 25 | - Demonstrating the graphics or any other content you know may be considered disturbing 26 | - Starting and/or participating in arguments related to politics 27 | - Assuming or promoting any kind of inequality including but not limited to: age, body size, disability, ethnicity, gender identity and expression, nationality and race, personal appearance, religion, or sexual identity and orientation 28 | - Drug promotion of any kind 29 | - Attacking personal tastes 30 | - Other conduct which you know could reasonably be considered inappropriate in a professional setting. 31 | 32 | ### Enforcement 33 | 34 | Violations of the Code of Conduct may be reported by sending an email to [shimmythedev@gmail.com](shimmythedev@gmail.com). All reports will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. Further details of specific enforcement policies may be posted separately. 35 | 36 | We hold the right and responsibility to remove comments or other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any members for other behaviours that they deem inappropriate, threatening, offensive, or harmful. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Go docker API stack example 2 | 3 | An example of a a dockerized Go stack for self-hosted purpose. 4 | 5 | #### Services 6 | 7 | - Simple **Go** service that tests logs 8 | - **Grafana**/**loki**/**promtail** Log observability tools 9 | - **MongoDB** - NoSQL Database 10 | - **MinIO** - S3 & Kubernetes Native Object Storage 11 | 12 | --- 13 | 14 | ### Instructions 15 | 16 | ##### Quickstart 17 | 18 | ``` 19 | git clone https://github.com/ShimmyTheDev/go-mongodb-docker-api-stack \ 20 | && cd go-mongodb-docker-api-stack \ 21 | && touch .env \ 22 | && echo -e "MONGODB_DATABASE=database 23 | MONGODB_USER=db_user 24 | MONGODB_PASSWORD=da_password 25 | MONGO_INITDB_ROOT_USERNAME=username 26 | MONGO_INITDB_ROOT_PASSWORD=password 27 | MONGODB_USER=db_user 28 | MONGODB_PASSWORD=db_password 29 | MONGO_INITDB_ROOT_USERNAME=db_init_user 30 | MONGO_INITDB_ROOT_PASSSWORD=db_init_password 31 | MINIO_ACCESS_KEY=minio_access_key 32 | MINIO_SECRET_KEY=minio_secret_key" >> .env \ 33 | && docker compose up -d 34 | ``` 35 | 36 | --- 37 | 38 | ##### Ports 39 | 40 | - **MongoDB** - 27017 41 | - **MinIO** - 9000 42 | - **Grafana** - 3000 43 | - **loki** - 3100 44 | - **API** - 8000 45 | -------------------------------------------------------------------------------- /api/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM golang:1.21-alpine AS build 2 | 3 | WORKDIR /app 4 | 5 | COPY go.mod . 6 | COPY go.sum . 7 | 8 | RUN go mod download 9 | 10 | COPY . . 11 | 12 | RUN go build -o /app-api 13 | 14 | FROM alpine:latest 15 | 16 | WORKDIR /root/ 17 | 18 | COPY --from=build /app-api . 19 | 20 | EXPOSE 8000 21 | 22 | CMD ["./app-api"] 23 | -------------------------------------------------------------------------------- /api/docker-compose.yaml: -------------------------------------------------------------------------------- 1 | services: 2 | api: 3 | build: ../.. 4 | ports: 5 | - "8000:8000" 6 | 7 | -------------------------------------------------------------------------------- /api/go.mod: -------------------------------------------------------------------------------- 1 | module api 2 | 3 | go 1.21 4 | 5 | require ( 6 | github.com/akkuman/logrus-loki-hook v0.0.0-20210518091319-f99fe13c48f5 7 | github.com/sirupsen/logrus v1.9.3 8 | ) 9 | 10 | require ( 11 | github.com/afiskon/promtail-client v0.0.0-20190305142237-506f3f921e9c // indirect 12 | github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect 13 | github.com/golang/protobuf v1.3.2 // indirect 14 | github.com/golang/snappy v0.0.1 // indirect 15 | github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect 16 | github.com/stretchr/testify v1.9.0 // indirect 17 | golang.org/x/sys v0.18.0 // indirect 18 | ) 19 | -------------------------------------------------------------------------------- /api/go.sum: -------------------------------------------------------------------------------- 1 | github.com/afiskon/promtail-client v0.0.0-20190305142237-506f3f921e9c h1:AMDVOKGaiqse4qiRXSzRgpC9DCNTHCx6zpzdtXXrKM4= 2 | github.com/afiskon/promtail-client v0.0.0-20190305142237-506f3f921e9c/go.mod h1:p/7Wos+jcfrnwLqqzJMZ0s323kfVtJPW+HUvAANklVQ= 3 | github.com/akkuman/logrus-loki-hook v0.0.0-20210518091319-f99fe13c48f5 h1:mcdhhXSNtCYAHkfWE2tEmCTfNTzVNYagbCwRFLVhahA= 4 | github.com/akkuman/logrus-loki-hook v0.0.0-20210518091319-f99fe13c48f5/go.mod h1:Zt4S1pDcKNYMRcb5Guyx1HrppigW9ZAN6J9QKks1koc= 5 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 6 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 7 | github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= 8 | github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 9 | github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs= 10 | github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 11 | github.com/golang/snappy v0.0.1 h1:Qgr9rKW7uDUkrbSmQeiDsGa8SjGyCOGtuasMWwvp2P4= 12 | github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= 13 | github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= 14 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 15 | github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= 16 | github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 17 | github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE= 18 | github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ= 19 | github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= 20 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 21 | github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 22 | github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= 23 | github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= 24 | github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg= 25 | github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY= 26 | golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 27 | golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 28 | golang.org/x/sys v0.18.0 h1:DBdB3niSjOA/O0blCZBqDefyWNYveAYMNF1Wum0DYQ4= 29 | golang.org/x/sys v0.18.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= 30 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 31 | gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 32 | gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= 33 | gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 34 | -------------------------------------------------------------------------------- /api/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "time" 5 | 6 | lokihook "github.com/akkuman/logrus-loki-hook" 7 | "github.com/sirupsen/logrus" 8 | ) 9 | 10 | var log = logrus.New() 11 | 12 | func init() { 13 | lokiHookConfig := &lokihook.Config{ 14 | URL: "http://loki:3100/loki/api/v1/push", 15 | LevelName: "severity", 16 | Labels: map[string]string{ 17 | "application": "test", 18 | }, 19 | } 20 | hook, err := lokihook.NewHook(lokiHookConfig) 21 | if err != nil { 22 | log.Error(err) 23 | } else { 24 | log.AddHook(hook) 25 | } 26 | } 27 | 28 | func main() { 29 | log.Info("Starting app") 30 | for { 31 | log.Info("I'm a Test.") 32 | 33 | time.Sleep(5 * time.Second) 34 | } 35 | 36 | } 37 | -------------------------------------------------------------------------------- /docker-compose.yaml: -------------------------------------------------------------------------------- 1 | services: 2 | api: 3 | container_name: API 4 | build: ./api 5 | ports: 6 | - "8000:8000" 7 | environment: 8 | - MONGODB_DATABASE=${MONGODB_DATABASE} 9 | - MONGODB_USER=${MONGODB_USER} 10 | - MONGODB_PASSWORD=${MONGODB_PASSWORD} 11 | volumes: 12 | - ./api:/app 13 | mongodb: 14 | container_name: MongoDB 15 | build: ./mongodb 16 | volumes: 17 | - mongo-data:/data/db 18 | - ./mongodb/backups:/backups 19 | environment: 20 | - MONGO_INITDB_ROOT_USERNAME=${MONGO_INITDB_ROOT_USERNAME} 21 | - MONGO_INITDB_ROOT_PASSWORD=${MONGO_INITDB_ROOT_PASSWORD} 22 | ports: 23 | - "27017:27017" 24 | networks: 25 | - docker-stack 26 | 27 | minio: 28 | container_name: MinIO 29 | build: ./minio 30 | volumes: 31 | - minio-data:/data 32 | - ./minio/backups:/backups 33 | environment: 34 | - MINIO_ROOT_USER=${MINIO_ACCESS_KEY} 35 | - MINIO_ROOT_PASSWORD=${MINIO_SECRET_KEY} 36 | command: server /data 37 | ports: 38 | - "9000:9000" 39 | networks: 40 | - docker-stack 41 | 42 | grafana: 43 | container_name: Grafana 44 | build: ./grafana 45 | volumes: 46 | - grafana-data:/var/lib/grafana 47 | ports: 48 | - "3000:3000" 49 | networks: 50 | - docker-stack 51 | 52 | loki: 53 | container_name: Loki 54 | image: grafana/loki:main 55 | ports: 56 | - "3100:3100" 57 | user: "0" 58 | volumes: 59 | - ./loki:/etc/loki 60 | - loki-data:/var/loki 61 | command: -config.file=/etc/loki/local-config.yaml 62 | networks: 63 | - docker-stack 64 | 65 | promtail: 66 | container_name: Promtail 67 | image: grafana/promtail:main 68 | volumes: 69 | - ./promtail/config.yml:/etc/promtail/config.yml 70 | - /var/log:/var/log 71 | - /var/run/docker.sock:/var/run/docker.sock 72 | command: -config.file=/etc/promtail/config.yml 73 | networks: 74 | - docker-stack 75 | 76 | volumes: 77 | mongo-data: 78 | minio-data: 79 | grafana-data: 80 | loki-data: 81 | 82 | networks: 83 | docker-stack: 84 | driver: bridge -------------------------------------------------------------------------------- /example.env: -------------------------------------------------------------------------------- 1 | MONGODB_DATABASE=database 2 | MONGODB_USER=db_user 3 | MONGODB_PASSWORD=da_password 4 | MONGO_INITDB_ROOT_USERNAME=username 5 | MONGO_INITDB_ROOT_PASSWORD=password 6 | MONGODB_USER=db_user 7 | MONGODB_PASSWORD=db_password 8 | MONGO_INITDB_ROOT_USERNAME=db_init_user 9 | MONGO_INITDB_ROOT_PASSSWORD=db_init_password 10 | MINIO_ACCESS_KEY=minio_access_key 11 | MINIO_SECRET_KEY=minio_secret_key 12 | -------------------------------------------------------------------------------- /grafana/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM grafana/grafana:main -------------------------------------------------------------------------------- /grafana/docker-compose.yaml: -------------------------------------------------------------------------------- 1 | services: 2 | grafana: 3 | image: grafana/grafana:main 4 | container_name: grafana 5 | ports: 6 | - "3000:3000" 7 | volumes: 8 | - grafana-data:/var/lib/grafana 9 | 10 | volumes: 11 | grafana-data: 12 | -------------------------------------------------------------------------------- /loki/local-config.yaml: -------------------------------------------------------------------------------- 1 | auth_enabled: false 2 | 3 | server: 4 | http_listen_port: 3100 5 | grpc_listen_port: 9095 6 | 7 | common: 8 | path_prefix: /var/loki 9 | storage: 10 | filesystem: 11 | chunks_directory: /var/loki/chunks 12 | rules_directory: /var/loki/rules 13 | replication_factor: 1 14 | ring: 15 | kvstore: 16 | store: inmemory 17 | 18 | schema_config: 19 | configs: 20 | - from: 2020-10-24 21 | store: tsdb 22 | object_store: filesystem 23 | schema: v13 24 | index: 25 | prefix: index_ 26 | period: 24h 27 | 28 | compactor: 29 | working_directory: /var/loki/boltdb-shipper-compactor 30 | #shared_store: filesystem 31 | 32 | ruler: 33 | storage: 34 | type: local 35 | rule_path: /tmp/loki/rules-temp 36 | alertmanager_url: http://localhost:9093 37 | -------------------------------------------------------------------------------- /minio/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM minio/minio:latest -------------------------------------------------------------------------------- /minio/backups/backup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | TIMESTAMP=$(date +%F-%H%M) 3 | mc cp /data myminio/shopire-backups/$TIMESTAMP 4 | -------------------------------------------------------------------------------- /minio/docker-compose.yaml: -------------------------------------------------------------------------------- 1 | services: 2 | minio: 3 | image: minio/minio:latest 4 | container_name: minio 5 | ports: 6 | - "9000:9000" 7 | volumes: 8 | - minio-data:/data 9 | - ./backups:/backups 10 | environment: 11 | - MINIO_ACCESS_KEY=${MINIO_ACCESS_KEY} 12 | - MINIO_SECRET_KEY=${MINIO_SECRET_KEY} 13 | command: server /data 14 | 15 | volumes: 16 | minio-data: 17 | -------------------------------------------------------------------------------- /minio/setup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | mc alias set myminio http://minio:9000 $MINIO_ACCESS_KEY $MINIO_SECRET_KEY 3 | mc mb myminio/shopire-backups 4 | -------------------------------------------------------------------------------- /mongodb/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM mongo:latest -------------------------------------------------------------------------------- /mongodb/backups/backup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | TIMESTAMP=$(date +%F-%H%M) 3 | BACKUP_DIR=/backups/$TIMESTAMP 4 | mkdir -p $BACKUP_DIR 5 | mongodump --host mongo1 --out $BACKUP_DIR 6 | -------------------------------------------------------------------------------- /mongodb/docker-compose.yaml: -------------------------------------------------------------------------------- 1 | services: 2 | mongodb: 3 | image: mongo:latest 4 | container_name: mongodb 5 | ports: 6 | - "27017:27017" 7 | volumes: 8 | - mongo-data:/data/db 9 | - ./backups:/backups 10 | environment: 11 | - MONGO_INITDB_ROOT_USERNAME=${MONGO_INITDB_ROOT_USERNAME} 12 | - MONGO_INITDB_ROOT_PASSWORD=${MONGO_INITDB_ROOT_PASSWORD} 13 | 14 | volumes: 15 | mongo-data: 16 | -------------------------------------------------------------------------------- /mongodb/init.js: -------------------------------------------------------------------------------- 1 | rs.initiate({ 2 | _id: "rs0", 3 | members: [ 4 | { _id: 0, host: "mongo1:27017" }, 5 | { _id: 1, host: "mongo2:27017" }, 6 | { _id: 2, host: "mongo3:27017" }, 7 | ], 8 | }); 9 | -------------------------------------------------------------------------------- /promtail/config.yml: -------------------------------------------------------------------------------- 1 | server: 2 | http_listen_port: 9080 3 | grpc_listen_port: 0 4 | 5 | positions: 6 | filename: /tmp/positions.yaml 7 | 8 | clients: 9 | - url: http://loki:3100/loki/api/v1/push 10 | 11 | scrape_configs: 12 | - job_name: system 13 | static_configs: 14 | - targets: 15 | - localhost 16 | labels: 17 | job: varlogs 18 | __path__: /var/log/*.log 19 | - job_name: docker 20 | docker_sd_configs: 21 | - host: unix:///var/run/docker.sock 22 | relabel_configs: 23 | - source_labels: [__meta_docker_container_name] 24 | target_label: container_name 25 | - source_labels: [__meta_docker_container_name] 26 | target_label: job 27 | --------------------------------------------------------------------------------