├── Changelog.md ├── License.txt ├── Readme.md ├── SetupDatabase.sql ├── bridge.py ├── bridgeFiltering.py ├── requirements.txt ├── startBridge.sh └── startBridgeFiltering.sh /Changelog.md: -------------------------------------------------------------------------------- 1 | Change history 2 | -------------- 3 | 4 | * **Version 1.0.2.0 (2021-07-25)** : Updated requirements. 5 | * **Version 1.0.1.0 (2020-06-04)** : Removed license year, added first tag. 6 | * **Version 1.0.0.1 (2019-09-29)** : Updated python version, updated requirements. 7 | * **Version 1.0.0.0 (?)** : 1.0 release. -------------------------------------------------------------------------------- /License.txt: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) SeppPenner (https://github.com/SeppPenner) 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /Readme.md: -------------------------------------------------------------------------------- 1 | # MQTT2PostgresBridge 2 | 3 | MQTT2PostgresBridge is a project to connect a locally running broker to a Postgres database and store the messages in a database table. The project was written and tested in Python 3. 4 | 5 | [![Build status](https://ci.appveyor.com/api/projects/status/bhqkj3oacr7jix8x?svg=true)](https://ci.appveyor.com/project/SeppPenner/mqtt2postgresbridge) 6 | [![GitHub issues](https://img.shields.io/github/issues/SeppPenner/MQTT2PostgresBridge.svg)](https://github.com/SeppPenner/MQTT2PostgresBridge/issues) 7 | [![GitHub forks](https://img.shields.io/github/forks/SeppPenner/MQTT2PostgresBridge.svg)](https://github.com/SeppPenner/MQTT2PostgresBridge/network) 8 | [![GitHub stars](https://img.shields.io/github/stars/SeppPenner/MQTT2PostgresBridge.svg)](https://github.com/SeppPenner/MQTT2PostgresBridge/stargazers) 9 | [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://raw.githubusercontent.com/SeppPenner/MQTT2PostgresBridge/master/License.txt) 10 | [![Known Vulnerabilities](https://snyk.io/test/github/SeppPenner/MQTT2PostgresBridge/badge.svg)](https://snyk.io/test/github/SeppPenner/MQTT2PostgresBridge) 11 | 12 | ## Adjust your settings: 13 | 14 | * Adjust the broker to the address you want to use: `broker_source` 15 | * Add your custom filters to `filterMessage()` if you want to filter messages 16 | * Adjust your credentials (uncomment if anonymous): 17 | 18 | ```python 19 | client_source.username_pw_set("mqtt", "IoT") 20 | ``` 21 | 22 | * Adjust your PostgreSQL credentials: 23 | 24 | ```python 25 | DatabaseHostName = 'YourHost' 26 | DatabaseUserName = 'YourUser' 27 | DatabasePassword = 'YourPassword' 28 | DatabasePort = 5432 29 | ``` 30 | 31 | * Run the script `SetupDatabase.sql` on your database to setup the database correctly 32 | 33 | * Add filters to the bridging like described in the `bridgeFiltering.py` file if needed: 34 | 35 | ```python 36 | def filterMessage(payload, topic, qos): 37 | "Filters the messages depending on the configuration for the attributes payload, topic and QoS. 'True' means that the message is not forwarded." 38 | # Examples below: 39 | if(payload == "10 %"): 40 | print('Filtered: payload == "10 %"') 41 | return True 42 | if(topic == "humidity" and qos == 0): 43 | print('Filtered: topic == "humidity" and qos == 0') 44 | return True 45 | if(topic == "temperature" or qos == 2): 46 | print('Filtered: topic == "temperature" or qos == 2') 47 | return True 48 | #Add your filters here 49 | ``` 50 | 51 | ## Setup on the Raspberry Pi 52 | 53 | ```bash 54 | sudo apt-get install python3 55 | sudo apt-get install python3-pip 56 | sudo pip3 install paho-mqtt 57 | sudo pip3 install psycopg2 58 | ``` 59 | 60 | or 61 | 62 | ```bash 63 | sudo apt-get install python3 64 | sudo apt-get install python3-pip 65 | sudo pip3 install -r requirements.txt 66 | ``` 67 | 68 | ## Running the programms: 69 | 70 | ```bash 71 | python3 bridge.py 72 | python3 bridgeFiltering.py 73 | ``` 74 | 75 | ## Installing the latest version of Python (Currently 3.7.4) on the Raspberry Pi: 76 | 77 | https://gist.github.com/SeppPenner/6a5a30ebc8f79936fa136c524417761d 78 | 79 | ## Paho MQTT client documentation 80 | 81 | * https://pypi.org/project/paho-mqtt/ 82 | * https://www.hivemq.com/blog/mqtt-client-library-paho-python 83 | 84 | ## Postgres client documentation 85 | 86 | * http://www.postgresqltutorial.com/postgresql-python/connect/ 87 | * http://www.postgresqltutorial.com/postgresql-python/call-stored-procedures/ 88 | 89 | ## See also 90 | 91 | * [MQTT2AWSS3Bridge](https://github.com/SeppPenner/MQTT2AWSS3Bridge) 92 | * [MQTT2MySQLBridge](https://github.com/SeppPenner/MQTT2MySQLBridge) 93 | * [MQTT2MQTTBridge](https://github.com/SeppPenner/MQTT2MQTTBridge) 94 | 95 | Change history 96 | -------------- 97 | 98 | See the [Changelog](https://github.com/SeppPenner/MQTT2PostgresBridge/blob/master/Changelog.md). -------------------------------------------------------------------------------- /SetupDatabase.sql: -------------------------------------------------------------------------------- 1 | DROP DATABASE mqtt; 2 | CREATE DATABASE mqtt; 3 | USE mqtt; 4 | 5 | CREATE TABLE mqtt ( 6 | id SERIAL PRIMARY KEY, 7 | createdAt TIMESTAMP NOT NULL, 8 | topic TEXT NOT NULL, 9 | message TEXT, 10 | qos NUMERIC(1) NOT NULL 11 | ); 12 | 13 | CREATE FUNCTION InsertIntoMQTTTable(topic TEXT, message TEXT, qos NUMERIC(1)) RETURNS void AS $$ 14 | BEGIN 15 | INSERT INTO mqtt (createdAt, topic, message, qos) 16 | VALUES (CURRENT_TIMESTAMP, topic, message, qos); 17 | END; 18 | $$ LANGUAGE plpgsql; -------------------------------------------------------------------------------- /bridge.py: -------------------------------------------------------------------------------- 1 | import paho.mqtt.client as mqtt 2 | import traceback 3 | import psycopg2 4 | 5 | broker_source = "127.0.0.1" 6 | broker_source_port = 1883 7 | 8 | client_source = mqtt.Client("YourClientId") 9 | client_source.username_pw_set("YourUsername", "YourPassword") 10 | 11 | DatabaseHostName = 'YourHost' 12 | DatabaseUserName = 'YourUser' 13 | DatabasePassword = 'YourPassword' 14 | DatabaseName = 'mqtt' 15 | DatabasePort = 5432 16 | 17 | print("Connecting to database") 18 | connection = psycopg2.connect( 19 | host = DatabaseHostName, 20 | user = DatabaseUserName, 21 | password = DatabasePassword, 22 | database = DatabaseName, 23 | port = DatabasePort 24 | ) 25 | 26 | def insertIntoDatabase(message): 27 | "Inserts the mqtt data into the database" 28 | with connection.cursor() as cursor: 29 | print("Inserting data: " + str(message.topic) + ";" + str(message.payload)[2:][:-1] + ";" + str(message.qos)) 30 | cursor.callproc('InsertIntoMQTTTable', [str(message.topic), str(message.payload)[2:][:-1], int(message.qos)]) 31 | connection.commit() 32 | 33 | def on_message(client, userdata, message): 34 | "Evaluated when a new message is received on a subscribed topic" 35 | print("Received message '" + str(message.payload)[2:][:-1] + "' on topic '" 36 | + message.topic + "' with QoS " + str(message.qos)) 37 | insertIntoDatabase(message) 38 | 39 | def setup(): 40 | "Runs the setup procedure for the client" 41 | print("Setting up the onMessage handler") 42 | client_source.on_message = on_message 43 | print("Connecting to source") 44 | client_source.connect(broker_source, broker_source_port) 45 | client_source.subscribe("#", qos=1) 46 | print("Setup finished, waiting for messages...") 47 | 48 | try: 49 | setup() 50 | client_source.loop_forever() 51 | except Exception as e: 52 | traceback.print_exc() 53 | finally: 54 | connection.close() -------------------------------------------------------------------------------- /bridgeFiltering.py: -------------------------------------------------------------------------------- 1 | import paho.mqtt.client as mqtt 2 | import traceback 3 | import psycopg2 4 | 5 | broker_source = "127.0.0.1" 6 | broker_source_port = 1883 7 | 8 | client_source = mqtt.Client("YourClientId") 9 | client_source.username_pw_set("YourUsername", "YourPassword") 10 | 11 | DatabaseHostName = 'YourHost' 12 | DatabaseUserName = 'YourUser' 13 | DatabasePassword = 'YourPassword' 14 | DatabaseName = 'mqtt' 15 | DatabasePort = 5432 16 | 17 | print("Connecting to database") 18 | connection = psycopg2.connect( 19 | host = DatabaseHostName, 20 | user = DatabaseUserName, 21 | password = DatabasePassword, 22 | database = DatabaseName, 23 | port = DatabasePort 24 | ) 25 | 26 | def insertIntoDatabase(message): 27 | "Inserts the mqtt data into the database" 28 | with connection.cursor() as cursor: 29 | print("Inserting data: " + str(message.topic) + ";" + str(message.payload)[2:][:-1] + ";" + str(message.qos)) 30 | cursor.callproc('InsertIntoMQTTTable', [str(message.topic), str(message.payload)[2:][:-1], int(message.qos)]) 31 | connection.commit() 32 | 33 | def on_message(client, userdata, message): 34 | "Evaluated when a new message is received on a subscribed topic" 35 | print("Received message '" + str(message.payload)[2:][:-1] + "' on topic '" 36 | + message.topic + "' with QoS " + str(message.qos)) 37 | if not filterMessage(str(message.payload), str(message.topic), (message.qos)): 38 | insertIntoDatabase(message) 39 | 40 | def filterMessage(payload, topic, qos): 41 | "Filters the messages depending on the configuration for the attributes payload, topic and QoS. 'True' means that the message is not forwarded." 42 | # Examples below: 43 | if(payload == "10 %"): 44 | print('Filtered: payload == "10 %"') 45 | return True 46 | if(topic == "humidity" and qos == 0): 47 | print('Filtered: topic == "humidity" and qos == 0') 48 | return True 49 | if(topic == "temperature" or qos == 2): 50 | print('Filtered: topic == "temperature" or qos == 2') 51 | return True 52 | #Add your filters here 53 | 54 | def setup(): 55 | "Runs the setup procedure for the client" 56 | print("Setting up the onMessage handler") 57 | client_source.on_message = on_message 58 | print("Connecting to source") 59 | client_source.connect(broker_source, broker_source_port) 60 | client_source.subscribe("#", qos=1) 61 | print("Setup finished, waiting for messages...") 62 | 63 | try: 64 | setup() 65 | client_source.loop_forever() 66 | except Exception as e: 67 | traceback.print_exc() 68 | finally: 69 | connection.close() -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | paho-mqtt==1.6.1 2 | psycopg2==2.9.6 -------------------------------------------------------------------------------- /startBridge.sh: -------------------------------------------------------------------------------- 1 | python3 bridge.py -------------------------------------------------------------------------------- /startBridgeFiltering.sh: -------------------------------------------------------------------------------- 1 | python3 bridgeFiltering.py --------------------------------------------------------------------------------