├── LICENSE ├── README.md ├── docker-compose-app.yml ├── docker-compose-fluent-bit.yml ├── docker-compose-grafana.yml ├── docs └── img │ ├── datasource.png │ └── explore.png └── fluent-bit.conf /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Yash Thakkar 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Docker centralized logging using Fluent Bit, Grafana and Loki 2 | 3 | When running micro-services as containers, monitoring becomes very complex and difficult. That's where Prometheus, Grafana come to the rescue. Prometheus collects the metrics data and Grafana helps us to convert those metrics into beautiful visuals. Grafana allows you to query, visualize, and create an alert on metrics, no matter where they are stored. We can visualize metrics like CPU usage, memory usage, containers count, and much more. But there are few things that we can't visualize like container logs, it needs to be in tabular format with text data. For that, we can setup EFK (Elasticsearch + Fluentd + Kibana) stack, so Fluentd will collect logs from a docker container and forward it to Elasticsearch and then we can search logs using Kibana. 4 | 5 | Grafana team has released Loki, which is inspired by Prometheus to solve this issue. So now, we don't need to manage multiple stacks to monitor the running systems like Grafana and Prometheus to monitor and EFK to check the logs. 6 | 7 | ### Grafana Loki 8 | 9 | Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream. It uses labels from the log data to query. 10 | 11 | ### Fluent Bit 12 | 13 | Fluent Bit is an open-source and multi-platform Log Processor and Forwarder which allows you to collect data/logs from different sources, unify and send them to multiple destinations. It's fully compatible with Docker and Kubernetes environments. 14 | 15 | ### Docker Fluentd logging driver 16 | 17 | The Fluentd logging driver sends container logs to the Fluentd collector as structured log data. Then, users can use any of the various output plugins of Fluentd to write these logs to various destinations. 18 | 19 | We are going to use Fluent Bit to collect the Docker container logs and forward it to Loki and then visualize the logs on Grafana in tabular View. 20 | 21 | ### Setup 22 | 23 | We need to setup grafana, loki and fluent/fluent-bit to collect the Docker container logs using fluentd logging driver. Clone the sample project from [here](https://github.com/thakkaryash94/docker-grafana-loki-fluent-bit-sample). It contains the below files. 24 | 25 | - docker-compose-grafana.yml 26 | - docker-compose-fluent-bit.yml 27 | - fluent-bit.conf 28 | - docker-compose-app.yml 29 | 30 | We can combine all the yml files into one but I like it separated by the service group, more like kubernetes yml files. Let's see, what we have in those files. 31 | 32 | Before running docker services, we need to create an external network `loki` because our services are in different files, so they will communicate in this network. 33 | 34 | ```shell 35 | $ docker network create loki 36 | ``` 37 | 38 | **docker-compose-grafana.yml** 39 | This file contains Grafana, Loki, and renderer services. run `docker-compose -f docker-compose-grafana.yml up -d`. This will start 3 containers, grafana, renderer, and Loki, we will use grafana dashboard for the visualization and loki to collect data from fluent-bit service. Now, go to http://localhost:3000/ and you will be able to access the Grafana Dashboard. 40 | 41 | **docker-compose-fluent-bit.yml** 42 | We will be using `grafana/fluent-bit-plugin-loki:latest` image instead of a fluent-bit image to collect Docker container logs because it contains Loki plugin which will send container logs to Loki service. For that, we need to pass `LOKI_URL` environment variable to the container and also mounting `fluent-bit.conf` as well for custom configuration. 43 | 44 | ```yaml 45 | fluent-bit: 46 | image: grafana/fluent-bit-plugin-loki:latest 47 | container_name: fluent-bit 48 | environment: 49 | - LOKI_URL=http://loki:3100/loki/api/v1/push 50 | volumes: 51 | - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf 52 | 53 | ``` 54 | 55 | **fluent-bit.conf** 56 | 57 | This file contains fluent-bit configuration. Here, for input, we are listening on 0.0.0.0:24224 port and forwarding whatever we are getting to output plugins. We are setting a few Loki configs like LabelKeys, LineFormat, LogLevel, Url. The main key is LabelKeys, using this, we will be able to see the container logs, to make it dynamic, we are setting it so `container_name`, which means when we will be running our services, we need to pass `container_name` in docker-compose file, using that name, we will be able to search and differentiate container logs. We can add as many LabelKeys as we want with a comma(','). 58 | 59 | ``` 60 | [INPUT] 61 | Name forward 62 | Listen 0.0.0.0 63 | Port 24224 64 | [Output] 65 | Name grafana-loki 66 | Match * 67 | Url ${LOKI_URL} 68 | RemoveKeys source 69 | Labels {job="fluent-bit"} 70 | LabelKeys container_name 71 | BatchWait 1 72 | BatchSize 1001024 73 | LineFormat json 74 | LogLevel info 75 | ``` 76 | 77 | Now, let's run it with `docker-compose -f docker-compose-fluent-bit.yml up -d`. This will start fluent-bit container, which will collect the docker container logs and everything that is printed using stdout and forward it to loki service using loki plugin. 78 | 79 | **docker-compose-app.yml** 80 | It contains the actual application/server image service. You can ignore this file but we have to add below config in our server to forward container logs to the fluent-bit container. Important parts of the configuration are container_name, logging. `container_name` is the one we will use to filter the container logs from the Grafana Dashboard. In fluent-address, set your fluent-bit host IP address, if you are running locally, it will be your PC ip address. 81 | 82 | ```yml 83 | container_name: express-app 84 | logging: 85 | driver: fluentd 86 | options: 87 | fluentd-address: FLUENT_BIT_ADDRESS:24224 88 | ``` 89 | 90 | Now, everything is up and running. Let's generate some logs, if you are running docker-compose-app.yml file, then go to http://localhost:4000 and refresh few times, go to http://localhost:4000/test, this will generate some logs. 91 | 92 | So from docker container, logs will be sent to fluent-bit container, which will forward them to the Loki container using the Loki plugin. Now, we need to add Loki in Grafana data source, so that Grafana will be able to fetch the logs from Loki and we will be able to see it on the dashboard. 93 | 94 | #### Setup Grafana Dashboard 95 | 96 | To see the logs on Grafana dashboard, you can follow [YouTube video](https://youtu.be/qE6hEHNH9dE?t=73) or below steps. 97 | 98 | 1. Open the browser and go to http://localhost:3000, use default values `admin` and `admin` for username and password. 99 | 100 | 2. Now, go to http://localhost:3000/datasources and select `Loki` from `Logging and document databases` section. 101 | 102 | 3. Enter `http://loki:3100` in URL under `HTTP` section. We can do this because we are running Loki and Grafana in the same network `loki` else you have to enter host IP address and port here, click on `Save and Test` button from the bottom of the page. 103 | 104 | ![Grafana Data Source](https://raw.githubusercontent.com/thakkaryash94/docker-grafana-loki-fluent-bit-sample/master/docs/img/datasource.png) 105 | 106 | 4. Now, go to 3rd tab `Explore` from the left sidebar or http://localhost:3000/explore, click on `Log Labels` dropdown, here you will see `container_name` and `job` labels, these are same labels that we have mentioned in the `fluent-bit.conf` file with `LabelKeys` key. 107 | 108 | 5. Click on `container_name`, now, you should see our app service container name in the next step else type `{container_name="express-app"}` in the Loki query search. Click on that and that's it, now you should be able to see the container logs, these are the logs that we generated after starting up our app service. 109 | 110 | ![Grafana Explore](https://raw.githubusercontent.com/thakkaryash94/docker-grafana-loki-fluent-bit-sample/master/docs/img/explore.png) 111 | 112 | 113 | Now, we can tweak the view add this to our Grafana Dashboard and that's it. 114 | 115 | So, like this, we have setup fluentd-grafana-loki stack to collect and view the container logs on Grafana Dashboard. 116 | 117 | Below are the links that I have mentioned in the blog, that will help you to setup the stack. 118 | 119 | Links: 120 | 121 | - [GitHub Repo](https://github.com/thakkaryash94/docker-grafana-loki-fluent-bit-sample) 122 | - [Getting started with Grafana Loki - under 4 minutes](https://youtu.be/qE6hEHNH9dE?t=73) 123 | - [Loki GitHub Repo](https://github.com/grafana/loki/tree/master/cmd/fluent-bit) 124 | - [Loki Fluent Bit GitHub Repo](https://github.com/grafana/loki/tree/master/cmd/fluent-bit) 125 | -------------------------------------------------------------------------------- /docker-compose-app.yml: -------------------------------------------------------------------------------- 1 | version: "3" 2 | 3 | services: 4 | app: 5 | image: thakkaryash94/express-example:latest 6 | container_name: express-app 7 | ports: 8 | - "4000:3000" 9 | logging: 10 | driver: fluentd 11 | options: 12 | fluentd-address: FLUENT_BIT_ADDRESS:24224 13 | -------------------------------------------------------------------------------- /docker-compose-fluent-bit.yml: -------------------------------------------------------------------------------- 1 | version: "3" 2 | 3 | services: 4 | fluent-bit: 5 | image: grafana/fluent-bit-plugin-loki:latest 6 | container_name: fluent-bit 7 | environment: 8 | - LOKI_URL=http://loki:3100/loki/api/v1/push 9 | volumes: 10 | - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf 11 | ports: 12 | - "24224:24224" 13 | - "24224:24224/udp" 14 | networks: 15 | - loki 16 | 17 | networks: 18 | loki: 19 | external: true 20 | -------------------------------------------------------------------------------- /docker-compose-grafana.yml: -------------------------------------------------------------------------------- 1 | version: "3" 2 | 3 | services: 4 | loki: 5 | image: grafana/loki:latest 6 | container_name: loki 7 | expose: 8 | - "3100" 9 | networks: 10 | - loki 11 | grafana: 12 | image: grafana/grafana:latest 13 | container_name: grafana 14 | ports: 15 | - "3000:3000" 16 | environment: 17 | GF_RENDERING_SERVER_URL: http://renderer:8081/render 18 | GF_RENDERING_CALLBACK_URL: http://grafana:3000/ 19 | GF_LOG_FILTERS: rendering:debug 20 | networks: 21 | - loki 22 | renderer: 23 | image: grafana/grafana-image-renderer:latest 24 | container_name: grafana-image-renderer 25 | expose: 26 | - "8081" 27 | environment: 28 | ENABLE_METRICS: "true" 29 | networks: 30 | - loki 31 | networks: 32 | loki: 33 | external: true 34 | -------------------------------------------------------------------------------- /docs/img/datasource.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/thakkaryash94/docker-grafana-loki-fluent-bit-sample/9d9c980060cfe6cff8af1c7c25969e023d8984cf/docs/img/datasource.png -------------------------------------------------------------------------------- /docs/img/explore.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/thakkaryash94/docker-grafana-loki-fluent-bit-sample/9d9c980060cfe6cff8af1c7c25969e023d8984cf/docs/img/explore.png -------------------------------------------------------------------------------- /fluent-bit.conf: -------------------------------------------------------------------------------- 1 | [INPUT] 2 | Name forward 3 | Listen 0.0.0.0 4 | Port 24224 5 | [Output] 6 | Name grafana-loki 7 | Match * 8 | Url ${LOKI_URL} 9 | RemoveKeys source 10 | Labels {job="fluent-bit"} 11 | LabelKeys container_name 12 | BatchWait 1s 13 | BatchSize 1001024 14 | LineFormat json 15 | LogLevel info 16 | --------------------------------------------------------------------------------