├── .gitignore ├── README.md ├── arduino ├── esp32.ino └── nodemcu.ino ├── client.py ├── docker-compose.yml ├── img ├── alert.png └── dashboard.png ├── requirements.txt └── www ├── __init__.py └── app.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Playing with Docker, MQTT, Grafana, InfluxDB, Python and Arduino 2 | 3 | I must admit this post is just an excuse to play with Grafana and InfluxDb. InfluxDB is a cool database especially designed to work with time series. And Grafana is one open source tool for time series analytics. I want to build a simple prototype. The idea is: 4 | 5 | * One Arduino device (esp32) emits a MQTT event to a mosquito server. I'll use a potentiometer to emulate one sensor (Imagine for example a temperature sensor instead of potentiometer) 6 | * One Python script will be listening to the MQTT event in my Raspberry Pi and it will persist the value to InfluxDB 7 | * I will monitor the state of the time series given by the potentiometer with Grafana 8 | * I will create an alert in Grafana (for example when the average value within 10 seconds is above a threshold) and I will trigger a webhook when the alert changes its state 9 | * One microservice (a Python flash server) will be listening to the webhook and it will emit a MQTT event depending on the state 10 | * Another Arduino device (one nodemcu) will be listening to this MQTT event and it will activate a led. Red one if the alert is on and green one if the alert is red 11 | 12 | ### Server 13 | As I said before we'll need three servers: 14 | * MQTT server (mosquitto) 15 | * InfluxDB server 16 | * Grafana server 17 | 18 | We'll use Docker. I've got a Docker host running in a Raspberry Pi3. The Raspberry Pi is a ARM device so we need docker images for this architecture. 19 | 20 | ```yml 21 | version: '2' 22 | 23 | services: 24 | mosquitto: 25 | image: pascaldevink/rpi-mosquitto 26 | container_name: moquitto 27 | ports: 28 | - "9001:9001" 29 | - "1883:1883" 30 | restart: always 31 | 32 | influxdb: 33 | image: hypriot/rpi-influxdb 34 | container_name: influxdb 35 | restart: always 36 | environment: 37 | - INFLUXDB_INIT_PWD="password" 38 | - PRE_CREATE_DB="iot" 39 | ports: 40 | - "8083:8083" 41 | - "8086:8086" 42 | volumes: 43 | - ~/docker/rpi-influxdb/data:/data 44 | 45 | grafana: 46 | image: fg2it/grafana-armhf:v4.6.3 47 | container_name: grafana 48 | restart: always 49 | ports: 50 | - "3000:3000" 51 | volumes: 52 | - grafana-db:/var/lib/grafana 53 | - grafana-log:/var/log/grafana 54 | - grafana-conf:/etc/grafana 55 | 56 | volumes: 57 | grafana-db: 58 | driver: local 59 | grafana-log: 60 | driver: local 61 | grafana-conf: 62 | driver: local 63 | ``` 64 | 65 | ### ESP32 66 | The Esp32 part is very simple. We only need to connect our potentiometer to the Esp32. The potentiometer has three pins: Gnd, Signal and Vcc. We'll use the pin 32. 67 | We only need to configure our Wifi network and connect to our MQTT server. 68 | 69 | ```c 70 | #include 71 | #include 72 | 73 | const int potentiometerPin = 32; 74 | 75 | // Wifi configuration 76 | const char* ssid = "my_wifi_ssid"; 77 | const char* password = "my_wifi_password"; 78 | 79 | // MQTT configuration 80 | const char* server = "192.168.1.111"; 81 | const char* topic = "/pot"; 82 | const char* clientName = "com.gonzalo123.esp32"; 83 | 84 | String payload; 85 | 86 | WiFiClient wifiClient; 87 | PubSubClient client(wifiClient); 88 | 89 | void wifiConnect() { 90 | Serial.println(); 91 | Serial.print("Connecting to "); 92 | Serial.println(ssid); 93 | 94 | WiFi.begin(ssid, password); 95 | 96 | while (WiFi.status() != WL_CONNECTED) { 97 | delay(500); 98 | Serial.print("."); 99 | } 100 | Serial.println(""); 101 | Serial.print("WiFi connected."); 102 | Serial.print("IP address: "); 103 | Serial.println(WiFi.localIP()); 104 | } 105 | 106 | void mqttReConnect() { 107 | while (!client.connected()) { 108 | Serial.print("Attempting MQTT connection..."); 109 | if (client.connect(clientName)) { 110 | Serial.println("connected"); 111 | } else { 112 | Serial.print("failed, rc="); 113 | Serial.print(client.state()); 114 | Serial.println(" try again in 5 seconds"); 115 | delay(5000); 116 | } 117 | } 118 | } 119 | 120 | void mqttEmit(String topic, String value) 121 | { 122 | client.publish((char*) topic.c_str(), (char*) value.c_str()); 123 | } 124 | 125 | void setup() { 126 | Serial.begin(115200); 127 | 128 | wifiConnect(); 129 | client.setServer(server, 1883); 130 | delay(1500); 131 | } 132 | 133 | void loop() { 134 | if (!client.connected()) { 135 | mqttReConnect(); 136 | } 137 | int current = (int) ((analogRead(potentiometerPin) * 100) / 4095); 138 | mqttEmit(topic, (String) current); 139 | delay(500); 140 | } 141 | ``` 142 | 143 | ### Mqtt listener 144 | 145 | The esp32 emits an event ("/pot") with the value of the potentiometer. So we're going to create a MQTT listener that listen to mqtt and persits the value to InfluxDB 146 | ```python 147 | import paho.mqtt.client as mqtt 148 | from influxdb import InfluxDBClient 149 | import datetime 150 | import logging 151 | 152 | 153 | def persists(msg): 154 | current_time = datetime.datetime.utcnow().isoformat() 155 | json_body = [ 156 | { 157 | "measurement": "pot", 158 | "tags": {}, 159 | "time": current_time, 160 | "fields": { 161 | "value": int(msg.payload) 162 | } 163 | } 164 | ] 165 | logging.info(json_body) 166 | influx_client.write_points(json_body) 167 | 168 | 169 | logging.basicConfig(level=logging.INFO) 170 | influx_client = InfluxDBClient('docker', 8086, database='iot') 171 | client = mqtt.Client() 172 | 173 | client.on_connect = lambda self, mosq, obj, rc: self.subscribe("/pot") 174 | client.on_message = lambda client, userdata, msg: persists(msg) 175 | 176 | client.connect("docker", 1883, 60) 177 | 178 | client.loop_forever() 179 | ``` 180 | 181 | ### Grafana 182 | In grafana we need to do two things. First create one datasource from our InfluxDB server. It's pretty straightforward to it. 183 | Finally we'll create a dashboard. We only have one time-serie with the value of the potentiometer. I must admit that my dasboard has a lot things that I've created only for fun. 184 | Thats the query that I'm using to plot the main graph 185 | ``` 186 | SELECT last("value") FROM "pot" WHERE time >= now() - 5m GROUP BY time(1s) fill(previous) 187 | ``` 188 | Here we can see the dashboard 189 | ![Dashboard](img/dashboard.png "Dashboard") 190 | 191 | And here my alert configuration: 192 | 193 | ![Alert](img/alert.png "Alert") 194 | 195 | I've also created a notification channel with a webhook. Grafana will use this web hook to notify when the state of alert changes 196 | 197 | ### Webhook listener 198 | Grafana will emit a webhook, so we'll need an REST endpoint to collect the webhook calls. I normally use PHP/Lumen to create REST servers but in this project I'll use Python and Flask. 199 | We need to handle HTTP Basic Auth and emmit a MQTT event. Mqtt is a very simple protocol but it has one very nice feature that fits like hat fits like a glove here. Le me explain it: 200 | Imagine that we've got our system up and running and the state is "ok". Now we connect one device (for example one big red/green lights). Since the "ok" event was fired before we connect the lights, our green light will not be switch on. We need to wait util "alert" event if we want to see any light. That's not cool. 201 | Mqtt allows us to "retain" messages. That means that we can emit messages with "retain" flag to one topic and when we connect one device later to this topic it will receive the message. Here it's exactly what we need. 202 | 203 | ```python 204 | from flask import Flask 205 | from flask import request 206 | from flask_httpauth import HTTPBasicAuth 207 | import paho.mqtt.client as mqtt 208 | import json 209 | 210 | client = mqtt.Client() 211 | 212 | app = Flask(__name__) 213 | auth = HTTPBasicAuth() 214 | 215 | # http basic auth credentials 216 | users = { 217 | "user": "password" 218 | } 219 | 220 | 221 | @auth.get_password 222 | def get_pw(username): 223 | if username in users: 224 | return users.get(username) 225 | return None 226 | 227 | 228 | @app.route('/alert', methods=['POST']) 229 | @auth.login_required 230 | def alert(): 231 | client.connect("docker", 1883, 60) 232 | data = json.loads(request.data.decode('utf-8')) 233 | if data['state'] == 'alerting': 234 | client.publish(topic="/alert", payload="1", retain=True) 235 | elif data['state'] == 'ok': 236 | client.publish(topic="/alert", payload="0", retain=True) 237 | 238 | client.disconnect() 239 | 240 | return "ok" 241 | 242 | 243 | if __name__ == "__main__": 244 | app.run(host='0.0.0.0') 245 | ``` 246 | 247 | ### Nodemcu 248 | 249 | Finally the Nodemcu. This part is similar than the esp32 one. Our leds are in pins 4 and 5. We also need to configure the Wifi and connect to to MQTT server. Nodemcu and esp32 are similar devices but not the same. For example we need to use different libraries to connect to the wifi. 250 | This device will be listening to the MQTT event and trigger on led or another depending on the state 251 | ```c 252 | #include 253 | #include 254 | 255 | const int ledRed = 4; 256 | const int ledGreen = 5; 257 | 258 | // Wifi configuration 259 | const char* ssid = "my_wifi_ssid"; 260 | const char* password = "my_wifi_password"; 261 | 262 | // mqtt configuration 263 | const char* server = "192.168.1.111"; 264 | const char* topic = "/alert"; 265 | const char* clientName = "com.gonzalo123.nodemcu"; 266 | 267 | int value; 268 | int percent; 269 | String payload; 270 | 271 | WiFiClient wifiClient; 272 | PubSubClient client(wifiClient); 273 | 274 | void wifiConnect() { 275 | Serial.println(); 276 | Serial.print("Connecting to "); 277 | Serial.println(ssid); 278 | 279 | WiFi.begin(ssid, password); 280 | 281 | while (WiFi.status() != WL_CONNECTED) { 282 | delay(500); 283 | Serial.print("."); 284 | } 285 | Serial.println(""); 286 | Serial.print("WiFi connected."); 287 | Serial.print("IP address: "); 288 | Serial.println(WiFi.localIP()); 289 | } 290 | 291 | void mqttReConnect() { 292 | while (!client.connected()) { 293 | Serial.print("Attempting MQTT connection..."); 294 | if (client.connect(clientName)) { 295 | Serial.println("connected"); 296 | client.subscribe(topic); 297 | } else { 298 | Serial.print("failed, rc="); 299 | Serial.print(client.state()); 300 | Serial.println(" try again in 5 seconds"); 301 | delay(5000); 302 | } 303 | } 304 | } 305 | 306 | void callback(char* topic, byte* payload, unsigned int length) { 307 | 308 | Serial.print("Message arrived ["); 309 | Serial.print(topic); 310 | 311 | String data; 312 | for (int i = 0; i < length; i++) { 313 | data += (char)payload[i]; 314 | } 315 | cleanLeds(); 316 | int value = data.toInt(); 317 | switch (value) { 318 | case 1: 319 | digitalWrite(ledRed, HIGH); 320 | break; 321 | case 0: 322 | digitalWrite(ledGreen, HIGH); 323 | break; 324 | } 325 | Serial.print("] value:"); 326 | Serial.println((int) value); 327 | } 328 | 329 | void cleanLeds() { 330 | digitalWrite(ledRed, LOW); 331 | digitalWrite(ledGreen, LOW); 332 | } 333 | 334 | void setup() { 335 | Serial.begin(9600); 336 | pinMode(ledRed, OUTPUT); 337 | pinMode(ledGreen, OUTPUT); 338 | cleanLeds(); 339 | Serial.println("start"); 340 | 341 | wifiConnect(); 342 | client.setServer(server, 1883); 343 | client.setCallback(callback); 344 | 345 | delay(1500); 346 | } 347 | 348 | void loop() { 349 | Serial.print("."); 350 | if (!client.connected()) { 351 | mqttReConnect(); 352 | } 353 | 354 | client.loop(); 355 | delay(500); 356 | } 357 | ``` 358 | 359 | [![Playing with Docker, MQTT, Grafana, InfluxDB, Python and Arduino](http://img.youtube.com/vi/T7CzdaEY740/0.jpg)](https://www.youtube.com/watch?v=T7CzdaEY740) -------------------------------------------------------------------------------- /arduino/esp32.ino: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | 4 | const int potentiometerPin = 32; 5 | 6 | // Wifi configuration 7 | const char* ssid = "my_wifi_ssid"; 8 | const char* password = "my_wifi_password"; 9 | 10 | // mqtt configuration 11 | const char* server = "192.168.1.111"; 12 | const char* topic = "/pot"; 13 | const char* clientName = "com.gonzalo123.esp32"; 14 | 15 | String payload; 16 | 17 | WiFiClient wifiClient; 18 | PubSubClient client(wifiClient); 19 | 20 | void wifiConnect() { 21 | Serial.println(); 22 | Serial.print("Connecting to "); 23 | Serial.println(ssid); 24 | 25 | WiFi.begin(ssid, password); 26 | 27 | while (WiFi.status() != WL_CONNECTED) { 28 | delay(500); 29 | Serial.print("."); 30 | } 31 | Serial.println(""); 32 | Serial.print("WiFi connected."); 33 | Serial.print("IP address: "); 34 | Serial.println(WiFi.localIP()); 35 | } 36 | 37 | void mqttReConnect() { 38 | while (!client.connected()) { 39 | Serial.print("Attempting MQTT connection..."); 40 | if (client.connect(clientName)) { 41 | Serial.println("connected"); 42 | } else { 43 | Serial.print("failed, rc="); 44 | Serial.print(client.state()); 45 | Serial.println(" try again in 5 seconds"); 46 | delay(5000); 47 | } 48 | } 49 | } 50 | 51 | void mqttEmit(String topic, String value) 52 | { 53 | client.publish((char*) topic.c_str(), (char*) value.c_str()); 54 | } 55 | 56 | void setup() { 57 | Serial.begin(115200); 58 | 59 | wifiConnect(); 60 | client.setServer(server, 1883); 61 | delay(1500); 62 | } 63 | 64 | void loop() { 65 | if (!client.connected()) { 66 | mqttReConnect(); 67 | } 68 | int current = (int) ((analogRead(potentiometerPin) * 100) / 4095); 69 | mqttEmit(topic, (String) current); 70 | delay(500); 71 | } -------------------------------------------------------------------------------- /arduino/nodemcu.ino: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | 4 | const int ledRed = 4; 5 | const int ledGreen = 5; 6 | 7 | // Wifi configuration 8 | const char* ssid = "my_wifi_ssid"; 9 | const char* password = "my_wifi_password"; 10 | 11 | // mqtt configuration 12 | const char* server = "192.168.1.111"; 13 | const char* topic = "/alert"; 14 | const char* clientName = "com.gonzalo123.nodemcu"; 15 | 16 | int value; 17 | int percent; 18 | String payload; 19 | 20 | WiFiClient wifiClient; 21 | PubSubClient client(wifiClient); 22 | 23 | void wifiConnect() { 24 | Serial.println(); 25 | Serial.print("Connecting to "); 26 | Serial.println(ssid); 27 | 28 | WiFi.begin(ssid, password); 29 | 30 | while (WiFi.status() != WL_CONNECTED) { 31 | delay(500); 32 | Serial.print("."); 33 | } 34 | Serial.println(""); 35 | Serial.print("WiFi connected."); 36 | Serial.print("IP address: "); 37 | Serial.println(WiFi.localIP()); 38 | } 39 | 40 | void mqttReConnect() { 41 | while (!client.connected()) { 42 | Serial.print("Attempting MQTT connection..."); 43 | if (client.connect(clientName)) { 44 | Serial.println("connected"); 45 | client.subscribe(topic); 46 | } else { 47 | Serial.print("failed, rc="); 48 | Serial.print(client.state()); 49 | Serial.println(" try again in 5 seconds"); 50 | delay(5000); 51 | } 52 | } 53 | } 54 | 55 | void callback(char* topic, byte* payload, unsigned int length) { 56 | 57 | Serial.print("Message arrived ["); 58 | Serial.print(topic); 59 | 60 | String data; 61 | for (int i = 0; i < length; i++) { 62 | data += (char)payload[i]; 63 | } 64 | cleanLeds(); 65 | int value = data.toInt(); 66 | switch (value) { 67 | case 1: 68 | digitalWrite(ledRed, HIGH); 69 | break; 70 | case 0: 71 | digitalWrite(ledGreen, HIGH); 72 | break; 73 | } 74 | Serial.print("] value:"); 75 | Serial.println((int) value); 76 | } 77 | 78 | void cleanLeds() { 79 | digitalWrite(ledRed, LOW); 80 | digitalWrite(ledGreen, LOW); 81 | } 82 | 83 | void setup() { 84 | Serial.begin(9600); 85 | pinMode(ledRed, OUTPUT); 86 | pinMode(ledGreen, OUTPUT); 87 | cleanLeds(); 88 | Serial.println("start"); 89 | 90 | wifiConnect(); 91 | client.setServer(server, 1883); 92 | client.setCallback(callback); 93 | 94 | delay(1500); 95 | } 96 | 97 | void loop() { 98 | Serial.print("."); 99 | if (!client.connected()) { 100 | mqttReConnect(); 101 | } 102 | 103 | client.loop(); 104 | delay(500); 105 | } -------------------------------------------------------------------------------- /client.py: -------------------------------------------------------------------------------- 1 | import paho.mqtt.client as mqtt 2 | from influxdb import InfluxDBClient 3 | import datetime 4 | import logging 5 | 6 | 7 | def persists(msg): 8 | current_time = datetime.datetime.utcnow().isoformat() 9 | json_body = [ 10 | { 11 | "measurement": "pot", 12 | "tags": {}, 13 | "time": current_time, 14 | "fields": { 15 | "value": int(msg.payload) 16 | } 17 | } 18 | ] 19 | logging.info(json_body) 20 | influx_client.write_points(json_body) 21 | 22 | 23 | logging.basicConfig(level=logging.INFO) 24 | influx_client = InfluxDBClient('docker', 8086, database='iot') 25 | client = mqtt.Client() 26 | 27 | client.on_connect = lambda self, mosq, obj, rc: self.subscribe("/pot") 28 | client.on_message = lambda client, userdata, msg: persists(msg) 29 | 30 | client.connect("docker", 1883, 60) 31 | 32 | client.loop_forever() 33 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '2' 2 | 3 | services: 4 | mosquitto: 5 | image: pascaldevink/rpi-mosquitto 6 | container_name: moquitto 7 | ports: 8 | - "9001:9001" 9 | - "1883:1883" 10 | restart: always 11 | 12 | influxdb: 13 | image: hypriot/rpi-influxdb 14 | container_name: influxdb 15 | restart: always 16 | environment: 17 | - INFLUXDB_INIT_PWD="password" 18 | - PRE_CREATE_DB="iot" 19 | ports: 20 | - "8083:8083" 21 | - "8086:8086" 22 | volumes: 23 | - ~/docker/rpi-influxdb/data:/data 24 | 25 | grafana: 26 | image: fg2it/grafana-armhf:v4.6.3 27 | container_name: grafana 28 | restart: always 29 | ports: 30 | - "3000:3000" 31 | volumes: 32 | - grafana-db:/var/lib/grafana 33 | - grafana-log:/var/log/grafana 34 | - grafana-conf:/etc/grafana 35 | 36 | volumes: 37 | grafana-db: 38 | driver: local 39 | grafana-log: 40 | driver: local 41 | grafana-conf: 42 | driver: local -------------------------------------------------------------------------------- /img/alert.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gonzalo123/iot.grafana/1585822dde5b2d73cbf076872dc9b7b21dd53ed2/img/alert.png -------------------------------------------------------------------------------- /img/dashboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gonzalo123/iot.grafana/1585822dde5b2d73cbf076872dc9b7b21dd53ed2/img/dashboard.png -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | certifi==2018.1.18 2 | chardet==3.0.4 3 | click==6.7 4 | Flask==1.0 5 | Flask-BasicAuth==0.2.0 6 | idna==2.6 7 | influxdb==5.0.0 8 | itsdangerous==0.24 9 | Jinja2==2.10 10 | MarkupSafe==1.0 11 | paho-mqtt==1.3.1 12 | python-dateutil==2.6.1 13 | pytz==2018.3 14 | requests==2.20.0 15 | six==1.11.0 16 | urllib3==1.24.2 17 | Werkzeug==0.15.3 18 | -------------------------------------------------------------------------------- /www/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gonzalo123/iot.grafana/1585822dde5b2d73cbf076872dc9b7b21dd53ed2/www/__init__.py -------------------------------------------------------------------------------- /www/app.py: -------------------------------------------------------------------------------- 1 | from flask import Flask 2 | from flask import request 3 | from flask_httpauth import HTTPBasicAuth 4 | import paho.mqtt.client as mqtt 5 | import json 6 | 7 | client = mqtt.Client() 8 | 9 | app = Flask(__name__) 10 | auth = HTTPBasicAuth() 11 | 12 | # http basic auth credentials 13 | users = { 14 | "user": "password" 15 | } 16 | 17 | 18 | @auth.get_password 19 | def get_pw(username): 20 | if username in users: 21 | return users.get(username) 22 | return None 23 | 24 | 25 | @app.route('/alert', methods=['POST']) 26 | @auth.login_required 27 | def alert(): 28 | client.connect("docker", 1883, 60) 29 | data = json.loads(request.data.decode('utf-8')) 30 | if data['state'] == 'alerting': 31 | client.publish(topic="/alert", payload="1", retain=True) 32 | elif data['state'] == 'ok': 33 | client.publish(topic="/alert", payload="0", retain=True) 34 | 35 | client.disconnect() 36 | 37 | return "ok" 38 | 39 | 40 | if __name__ == "__main__": 41 | app.run(host='0.0.0.0') 42 | --------------------------------------------------------------------------------